Harvest

Installed a monitoring application HloginMon to monitor broker server.

Getting below error in HloginMon Logs for Harvest application.  (v 12.5)

Broker server is up.

Not sure what is causing this issue.

Users are not able to login to harvest application (from both harweb & workbench).

Please Help!

REC : Error       Broker             Failed to Respond 2017/10/01 10:45:09
DEL : Error       Broker             Failed to Respond 2017/10/01 10:45:20

The error indicates that the broker failed to respond to a query message of who is logged in from hloginmon to the broker.

You should be getting red flashing records of this error in the matrix display of the utility. Best triage of this

situation is to stop the bkrd.exe process, all hserver processes and the rtserver process then restart the broker.

If that does not remedy the problem then I would suggest to open a Harvest support case.

Thanks Dave. 

I already restarted the broker and it was working fine for a day. 

This issue seems to be sporadic and reappears again after a day or two. Please suggest !!

Already raised a Case with CA and they are not able to find anything from HBroker/Hserver logs 

You can setup HLoginMon to configure it to send you email alerts when the broker fails to respond

even to your smart phone. Since r12.5 that you are using is no longer supported, I would urge your

company to upgrade as soon as feasible to V13. I searched all open Harvest cases and did not find any

for SevenEleven – was this an old case that has been closed?

It would also be wise to make sure there is no network port scanner that is scanning Harvest ports as this could

cause the rtserver to fall into an unstable mode if the socket that it is trying to use is locked by the scanner.

The rtserver is the communication layer of Harvest.

Dave Carmack

Principal Support Engineer

CA Technical Support

214-473-1711 (11711 internal)

Hi Dave,

After installing this application, we got this "failed to respond " log only once. 

After that there was no issues for past 2 weeks. But after uninstalling this monitoring application the issue is reappearing as before on daily basis. 

Kindly help us know how monitoring the broker server and pulling the active user list could fix the issue ? 

Also can you give the exact back ground of this application how it works.

CA case number is 00846666 .

Regarding the version upgrade , our SEI is discussing on and is yet to decide on this.

regards

kannappan S

Can you please provide a screenshot of the application with the error?

HLoginMon must be installed to a windows workstation where the full client (either 32bit or 64bit is installed).

HLoginMon can be installed to any directory – default is C:\HLoginMon.

This utility is designed to connect to the broker using the harvest administrator credentials and basically

makes a hgetusg -cu call to the broker. The returned information is interpreted, sorted, and presented in

a matrix display within HLoginMon.

The call frequency to the broker should occur about every 10 seconds minimum or higher.

This utility has no effect on any other Harvest application.

Dave Carmack

Principal Support Engineer

CA Technical Support

214-473-1711 (11711 internal)

Hi Dave,

Would like to tell you the whole picture of the issue. 

Before installing HloginMon

Symptoms: A user tries to login to Harvest 12.5 using Harweb from a browser, or the Harvest Workbench client from any PC. The login hangs and returns the message “E03060065: Incorrect login” However, the user is putting in the correct credentials.

Debugging: No errors are found in any log and only a restart of Harvest frees up the situation to where users can now login again.

Current remedy: Admin user logs in to broker server (linux) and runs bkrd –shutdown, then bkrd to recycle the Harvest broker and generate a fresh set of HServer processes. Users can immediate login after that and continue their work.

Inorder to triage this issue, we installed HloginMon in client machine (windows), and we got "broker not responding" message only once. we had hdbgctrl option turned on.

There was no sign of above login issue after installation of this application for 2 weeks.

Then we uninstalled it and the above login issue has reappeared on daily basis and we had to restart bkrd to resolve.

Is this coincidental ? or polling the DB for current active users and hdbgctrl logging could fix this above said broker hang issues ?

Regards

kannappan S

HLoginMon has nothing to do with other individual Harvest client logins.

I would recommend that you open a new issue for the login problem.

Be sure to provide the HBroker.arg, HServer.arg, and rtserver.cm files to

the new issue. I would also recommend that you use the hdbgctrl with

logging set to level 5 and record these log in problems, then send the

HBroker and HServer logs to the issue as well (include the userIDs that

fail to login so that we can focus on those while analyzing the server logs).

Dave Carmack

Principal Support Engineer

CA Technical Support

214-473-1711 (11711 internal)

Hi Dave,

I uninstalled this application, and wrote a shell script to pull the list of users from harvest DB (select username from harvestuser) and scheduled it for every 15 minutes in harvest broker using crontab. The issue seems to have disappeared again. Monitored this for 5 days and no issues. When i remove this script, issue reoccurs. This is the same reason why issue dint occur when HloginMon was present and issue repeated when HLoginMon was removed

From this I could conclude below . 

Harvest behavior:

When there Is continuous requests made to Harvest broker to pull some data from database, harvest works well without any issues.

If the harvest broker is idle for a particular period of time (during onsite hours only minimal use of harvest is seen) , this harvest issue repeats which we will have to resolve by recycling Hbroker everytime it occurs. (bkrd -shutdown and bkrd commands)

This behavior is bit surprising as we have only seen highly loaded servers are faulty ones. Any idea why this can happen ?  

Your query against the database is only providing you the users you have assigned to Harvest, the login information

of each user is not maintained in the database, only the Broker maintains a current memory map of current logged in users.

Performing database queries against the database in order to maintain high Broker response has never been introduced

to L1 support before. I have no idea why this is the case. It could be network, database, server or even the clients.

I recommend you open a case with L1 support so they can triage this problem there and not in the community forum.