DX Unified Infrastructure Management

Hi team,

In what scenario we could receive robot inactive alerts even the robots have been moved and showing active on secondary hub. However on primary hub they are still showing inactive and red.

Regards,

IK

How did you move the robots?

I did not move any robot. We have deployed HA probe on Secondary hub.

So first we need to understand when you have  a primary and secondary hub for a robot there is no synchronization between the two hubs on the robot up status. Both operate independently of each other.

in theory the only time the robot should be reporting to the secondary hub is if the primary hub is down.

In which case the hub can not send robot down alerts.

If the hub is hub but the robot can not reach it then an alarm should be sent as there is a problem with the communication from the robot to its primary hub.

Currently this would be working as designed.

We would need more details on why the robot is reporting to the secondary or was moved to the secondary when the primary is up to provide any more insight into your situation.

Hi Gene,

Perhaps, you are right and it looks like sync issue between two hubs. How do i can check and troubleshoot this issue?

Moreover, we did not move any server on another hub. Secondary hub details are configured in the robot.cfg. However besides moving to secondary hub, they generated robot inactive alert also.

Below are the logs sample: First HUB is the primary hub and Secondary HUB is where HA probe is deployed.

Dec 18 02:40:27:312 [9640] Controller: Hub Primary_Server(IP Address) contact established
Dec 18 02:40:33:374 [9640] Controller: restart_all_probes  Restart probes other than hub, controller, spooler
Dec 18 02:40:46:000 [9640] Controller: restart_all_probes  Finished restart probes other than hub, controller, spooler
Dec 18 03:20:22:781 [9640] Controller: hub Primary_Server(IP Address) NO CONTACT (communication error)
Dec 18 03:20:22:781 [9640] Controller: Hub Secondary_Server(IP Address) contact established
Dec 18 03:20:22:781 [9640] Controller: restart_all_probes  Restart probes other than hub, controller, spooler
Dec 18 03:20:35:063 [9640] Controller: restart_all_probes  Finished restart probes other than hub, controller, spooler
Dec 18 03:30:00:070 [9640] Controller: checkin - from PRIMARY, move back (/Domain/server_name/hostname/hub)
Dec 18 03:30:00:070 [9640] Controller: Hub Primary_Server(IP Address) contact established
Dec 18 03:30:09:149 [9640] Controller: restart_all_probes  Restart probes other than hub, controller, spooler
Dec 18 03:30:22:384 [9640] Controller: restart_all_probes  Finished restart probes other than hub, controller, spooler
Dec 18 04:36:18:976 [9640] Controller: hub Primary_Server(IP Address) NO CONTACT (communication error)
Dec 18 04:36:18:976 [9640] Controller: Hub Secondary_Server(IP Address) contact established
Dec 18 04:36:18:991 [9640] Controller: restart_all_probes  Restart probes other than hub, controller, spooler
Dec 18 04:36:32:679 [9640] Controller: restart_all_probes  Finished restart probes other than hub, controller, spooler
Dec 18 05:08:42:169 [9640] Controller: checkin - from PRIMARY, move back (/Domain/server_name/hostname/hub)
Dec 18 05:08:42:185 [9640] Controller: Hub Primary_Server(IP Address) contact established
Dec 18 05:08:45:201 [9640] Controller: restart_all_probes  Restart probes other than hub, controller, spooler
Dec 18 05:08:58:185 [9640] Controller: restart_all_probes  Finished restart probes other than hub, controller, spooler

Hope, this would help you to understand and suggest me something to resolve this issue.

Thanks,

IK

so there seems to be a communication issue between the robots and the their primary hub and the HA hub and its partner.

need to check the network level on this on port 48002. Set the controller loglevel and hub loglevel to 3 and logsize to 5000 and 35000 respectively and check for errors.

The fact that the robots could not communicate with their primary hub is why they failed over to the secondary hub.

the fact that the primary hub is still up and running and the robots are not reporting into it because they can not is why you are getting the errors.

How the fail over is configured for robots within Hubs?

Hi Issac,

I am not sure what you are asking. Do we need to configure robots within hubs?

Regards,

IK

Hi Imran,

  Hope you are setting HA for robot ,if u have two hubs HUB A  as Primary and HUB B as secondary .

As a default while installing robot you will point to Hub A ,if u need a HA for robot add seconday_IP =HUB B .So if hub A is unavailable the robot will move hub B.

Hi Issac08,

In the logs of agents i shared already, it is showing that agent has established connection with the another HUB in HA.

Also we have configured the secondary hub server details in the robot.cfg.

Thanks,

IK

There is no configuration for robot fail over at the hub level.

Fail over for robots  for primary and secondary hubs is done on the robot  level only

Hi Gene,

What do you mean by robot failover at hub level. We have HA probe configured on Secondary server. Do you mean that HA probe is not configured properly?

Also in robot.cfg, we have configured the entry of secondary hub and therefore they are moving on the secondary hub if primary hub is not available.

What do you think I should do?

Thanks & Regards,

IK

The HA probe has nothing to do with robot fail over or messages about robot inactivity.

the only job of the HA probe is to monitor is partner hub and when it is not available activate and deactivate queues and probes. Nothing more.

The robots are completely responsible for failing over between their primary hub and a secondary hub if they can not communicate with the primary hub for some reason.

This will be seen in the individual robot controller logs when set to loglevel 3 or higher.

The robot inactive alarms is handled by the hub that the robot checked into.

When the robot fails to check in after the heart beat is missed the robot alarm will be sent.

This forum is probably not a good solution to finding the root cause. You should open a support case.

Hi Gene,

I just checked and found that when primary is down then robots are moving on secondary however when primary is up then they are showing on primary as well as on secondary hub as well.

What could be the reason? Do i need to reconfigure the secondary hub details again through IM?

Regards,

IK

This is not an instantaneous process.

It is expected on fail back that they would show up in both for a short time.

Some time you will need to log out or close IM or AC to have the UI refresh properly.

this is nothing to be concerned about.

There is nothing on the hub level you can change on this process as this is all handled strictly be each robot.

Yes sometimes you may have to refresh or reopen the IM .