DX Unified Infrastructure Management

One of our pain points with over 1500 robots is what seems to be the unstable behaivour of the robot reporting an IP address as 127.0.0.1 (seems to have increased in frequency since robot ver 5.52) . On Windows servers this requires the restart of the robot AT the server not via the Infrastructure Manager. Nimsoft support indicates best practice is to hard code the IP address but this is unworkable in our environment.  (hmm never seen this documented!) Right now we are required to manually review the IM searching for robots reporting IP address 127.0.0.1 as the status shows okay and no alarms occur. We are unable to communicate to the robot from the IM when in this condition. We could write a script to scan the HUBs looking for this condition but why not fix the issue?  Anyone else see this issue?

I've seen this on a couple of robots, to address it I've just hard coded the IP.  Didn't think too much of it, but maybe there is something lurking here that hasn't manifested itself yet.  Everything points to the robot being more than capable of reporting the correct IP, not that 127 one.  Seems like a 'feature' to me.  Time for a case to Nimsoft perhaps.

I've come across this as well. There are other problems with robots sometimes as well. Sometimes all probes except controller fail and no alarm is created. For this purpose I've created a script that scans my robots once a day, and it catcher all that are not functional.

It would be neat to have this issue fixed, but I'd still have my script running, just to be sure

-jon

Please note that I do have an active case open with CA-Nimsoft on this issue. Thanks all for replying!

Is it possible that you have the first line in the /etc/hosts file set to something like 127.0.0.1    RobotHostName rather than 127.0.0.1   localhost?

thanks for your response - the host files is as follows:

(It is interesting to note we are seeing on Linux systems as well; however we ARE able to restart the robot from Infrastructure Manager - thus (usually) returning to the correct IP address

hosts file on Windows:

(default from Microsoft)

# Copyright (c) 1993-2006 Microsoft Corp......
.....# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

127.0.0.1       localhost
::1             localhost

The only time I have personally seen this occur is on a linux box and the /etc/hosts file has the localhost address set to the hosts fully quailified name rather than

127.0.0.1   localhost

I am not certain what would cause this on a Windows box. I wonder if this correlates to Nimsoft introducing support for IP v6 (controller version >5.50 ?)

No, this issue has existed since long before IPv6 support was added. In my experience it has always been infrequent and inconsistent, so I never pursued any kind of fix in the code. I suspect it might have something to do with Nimsoft starting before or during initialization of the network stack in Windows, but I am not even sure that is possible based on the startup sequence in Windows. The controller might be the victim more than the cause of the problem when this happens, but I would think it could be coded with logic to check if it is usingthe loopback and then try to get the IP again.

Note: Even though you cannot restart the controller via Infrastructure Manager because it will send any requests to 127.0.0.1, you can still send the request to the right server if you know the IP. You can use the PU command; there are command line options to provide an IP address and TCP port rather than a nimaddress.

David,

Has your support case progressed any since posting. I have had this occur recently and was wondering if you made any progress on your support case before I open one myself. 

Thanks

Brandon

Brandon, the case has gone no where. It appears that this is a feature not an issue. Support suggested hard coding the robot IP address (argh).  I will most likely end up writing a script that restarts the robot when the 127.0.0.1 robot IP is encountered. (Keith's idea/suggestion) We are still experiencing this issue primarily with Windows servers with about 5% or the population a week having the issue (out of 1500 servers). We are seeing this even with the latest robot version.

I am interested in any thoughts - feedback regarding this issue.

regards,

Dave

@David - How is support justifying this as a feature and not a bug? To me this prevents end users from being able to administer the robot from IM. A bug. I would open a support case myself but am already finding myself championing other tickets (that are more important to my users) that are getting the "submit an idea for an enhancement "  that I don't want to inundate support with more of my "escalations". 

It would be nice to know how the IP address the robot uses is determine if the IP is not hardcoded in the current version of the robot for both Unix and Linux boxes. Perhaps I will open ticket and pose those questions. Might help provocatively address this issue for us. 

Thanks Keith -- this helps me solve this issue without having to reach out to other administrators in my company to manually log in to servers just to restart the controller.

Just for people's information if you know the hostname or perhaps the FQDN of the server you are looking at then you can ping it in windows command prompt and follow keith's instructions.

We usually use scripts to generate the robot.cfg file but there are often situations where that doesn't work. One workaround that I was really excited about was the use of wildcards in the robotip field, according to the documentation the following should be a valid:

robotip = 10.*.*.*,192.*.*.*,172.*.*.*

In theory, that should make the robot only use valid local addresses such as 10.1.1.50, 192.168.1.50, or 172.1.1.50. Unfortunately, it often does not...

Occasionally, it'll report the following which is very useful:

Specified IP (10.*.*.*) is not recognized as a local ip. No IP change, using 10.1.1.123.

This clearly doesn't seem to be as designed and maybe they did away with that (we use robots v5.70), but at least you can figure out where the problematic server is.  However, more often than not, it reports something like this:

Specified IP (10.*.*.*) is not recognized as a local ip. No IP change, using 169.254.135.187.

Obviously this is completely useless as 169.254.*.* is a local failback address. Whenever the robot sees 127.0.0.1 or 169.254.*.*, there should really be a line of code that says "hmm, that can't be right, let me try again" but so far I've never seen a robot recover from these by itself... The other annoying thing is that clearly there is SOME communication between the broken robot and the hub because somehow, ~magically~, the hub knows that the robot is using 127.0.0.1 or 169.254.x.x....???

Also, finding a robot's IP address in linux should not depend on the hosts file. I'm pretty sure the following command will give you the list of all valid IP addresses on any linux distribution:

/sbin/ifconfig | grep inet | cut -d: -f2 | awk '{ print $1}' | grep -v 127.0.0.1

And btw I've opened cases with Nimsoft about this and my previous post above... Just hoping maybe someone else has thought of something I haven't.

---

tdsnoc wrote:

 

The other annoying thing is that clearly there is SOME communication between the broken robot and the hub because somehow, ~magically~, the hub knows that the robot is using 127.0.0.1 or 169.254.x.x....???

---

The IP address reported to the hub when the robot registers has nothing to do with the IP address of the interface used to send the packets. Therefore, the problems selecting the correct IP address for the robot do not affect the communication that the robot intiiates to the hub. The robot includes the IP address in the payload of the packet when registering with the hub, and the hub relies on this information to initiate communication with the robot. That is the only direction that is affected.

Another place where some kind of sanity-check would be nice.. At least here it's easy to fix.

Vote up this idea:

https://na4.salesforce.com/ideas/viewIdea.apexp?id=08760000000ChsZ

mabye we can get this truley resolved.

I was not able to find a PU command on the HUB and I didn't see anyone post how to hard code the IP address. 

Here is a bat file to cycle through servers and clear the robot's IP address and then restart the robot. 

@ECHO OFF

SET PASSWORD=1234abcd

FOR /F "tokens=*" %%A in (RobotList.txt) do (
pu.exe -u administrator -p %PASSWORD% -H %%A probe_config_set controller controller robotip %%A NULL NULL
pu.exe -u administrator -p %PASSWORD% -H %%A -R
)

PAUSE

Hi ,

 getting this issue for so many windows server in my environment.

 Can i know the cause of this and also if there is anything i can do apart from hardcoding the robot's ip in robot.cfg file ?

thanks

Anmol

The difficulty here is reproducing the problem. From what I can tell from my own research is that this is caused by a race condition between UIM starting the the network going active. The two things that might be worth playing with are setting the service to a delayed start (at least on Windows) or establish a dependency on the network starting.

The blunt alternative is to schedule a daily restart of UIM. At least then you have a restart every 24 hours to fix it.

-Garin

Are you getting loop back IP in windows server ,mostly it will occur in linux machines .Please confirm.

It is maybe 25:1 in favor of occuring on Windows.

I've not done the specific analysis but from the period of time that this has been happening, there's no real OS type/version or UIM version dependency either.

-Garin

Yes ,it will be 25:1 and mostly it occurs in windows clusters which picks private or heartbeat IP address.