DX Unified Infrastructure Management

Expand all | Collapse all

8.5 Upgrade Install Failure on ADE probe startup

  • 1.  8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 05, 2017 12:14 PM

    Great start here. So in my lab ran the 8.5 installation and this is an upgrade from 8.4SP2 to 8.5.

    Stopped the distsrv forwarding ability and then ran the setup. It detected everything correctly and started the upgrade. 34% in it fails with:

    [ An error occurred during installation. View log button then click the CANCEL button to cancel installation. ]

    It doesn't even re-try to fix itself. Awesome. It won't even try to continue after hitting OK. Have to exit the installer. 

     

    Anyway the error was with the attempt to restart the automated_deployment_engine probe:

    2017-01-05 11:54:22,194 DEBUG probe.DistsrvController:getPackageDistributionStatus:643 [Thread-35] - DistsrvPackageDistributionStatus{strJobId=automated_deployment_engine-1483635246444, strJobDescription=NMS installation, strPackageName=automated_deployment_engine, strPackageVersion=, strRobotAddress=/UIM_NYCLAB/UIM-NYCLAB_PriHub/as-nyclab-uim, strStatus=finished, bExpiredFlag=false, nResultCode=0, strResultString=Finished, nAttemptNumber=0, nAttempts=1, nRetryAttempts=0}
    2017-01-05 11:54:22,194 INFO probe.DistsrvController:distributePackageSynchronous:570 [Thread-35] - distStatus.isFinished: true
    2017-01-05 11:54:22,194 INFO impl.UIMServerConfigureController:distributePackageDistsrv:2868 [Thread-35] - distsrvPkgStatus response: DistsrvPackageDistributionStatus{strJobId=automated_deployment_engine-1483635246444, strJobDescription=NMS installation, strPackageName=automated_deployment_engine, strPackageVersion=, strRobotAddress=/UIM_NYCLAB/UIM-NYCLAB_PriHub/as-nyclab-uim, strStatus=finished, bExpiredFlag=false, nResultCode=0, strResultString=Finished, nAttemptNumber=0, nAttempts=1, nRetryAttempts=0}
    2017-01-05 11:54:22,194 INFO impl.UIMServerConfigureController:distributePackageDistsrv:2881 [Thread-35] - Successfully distributed package 'automated_deployment_engine'
    2017-01-05 11:54:22,194 DEBUG pds.PDSController:sendPDSWithAddr:133 [Thread-35] - Sending 'probe_activate' with sid: s5..., timeout: 180000
    2017-01-05 11:54:24,819 ERROR impl.UIMServerConfigureController:run:476 [Thread-35] - NimException caught
    (2) communication error, Received status (2) on response (for sendRcv) for cmd = 'probe_activate'
    at com.nimsoft.nimbus.NimSessionBase.sendRcv(NimSessionBase.java:609)
    at com.nimsoft.nimbus.NimSessionBase.sendRcv(NimSessionBase.java:562)
    at com.nimsoft.nimbus.NimClientSession.send(NimClientSession.java:170)
    at com.nimsoft.nimbus.NimRequest.sendImpersonate(NimRequest.java:263)
    at com.nimsoft.install.nimcommon.pds.PDSController.sendRequest(PDSController.java:213)
    at com.nimsoft.install.nimcommon.pds.PDSController.sendPDSWithAddr(PDSController.java:136)
    at com.nimsoft.install.nimcommon.pds.PDSController.sendPDSWithAddr(PDSController.java:113)
    at com.nimsoft.install.nimcommon.pds.PDSController.sendWithAddr(PDSController.java:108)
    at com.nimsoft.install.nimcommon.probe.ProbeController.activate(ProbeController.java:677)
    at com.nimsoft.install.nimcommon.probe.ProbeController.activate(ProbeController.java:654)
    at com.nimsoft.install.uimserver.action.impl.UIMServerConfigureController.activateProbeCommon(UIMServerConfigureController.java:3006)
    at com.nimsoft.install.uimserver.action.impl.UIMServerConfigureController.configureAde(UIMServerConfigureController.java:1302)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.nimsoft.install.nimcommon.method.NimMethodCallback.invokeMethod(NimMethodCallback.java:532)
    at com.nimsoft.install.nimcommon.method.NimMethodCallback.invokeMethod(NimMethodCallback.java:516)
    at com.nimsoft.install.uimserver.action.impl.UIMServerConfigureController.doConfigure_postNMSStartupPackages(UIMServerConfigureController.java:534)
    at com.nimsoft.install.uimserver.action.impl.UIMServerConfigureController.doConfigure(UIMServerConfigureController.java:490)
    at com.nimsoft.install.uimserver.action.impl.UIMServerConfigureController.run(UIMServerConfigureController.java:474)
    at java.lang.Thread.run(Unknown Source)

     

    Anyway, anyone know what the deal is here?

    I checked the probes running on the pri-hub and at this point only the controller, distsrv, hdb, hub, spooler were running. I now need to stop, restart services and retry...

     



  • 2.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 05, 2017 01:32 PM

    So after exiting the installation and full stop/start of UIM service the 2nd attempt worked this time EXCEPT at the very end the installer said that the trellis probe was expected to be active but was in-active. 

    So finished w/ the installer then did a stop/start on UIM services. Now my primary hub is not starting up. The trellis probe doesn't want to start up.

    Only 11/40 probes start up. The trellis probe is coming up RED and its error is saying:

    Jan 05 00:02:27:453 [attach_socket, trellis] An exception occurred while processing a message from Socket[addr=/192.168.241.53,port=64322,localport=48040].
    Jan 05 00:02:27:454 [attach_socket, trellis] (13) SID has expired, null
    at com.nimsoft.nimbus.NimServerSession$NimServerSessionThread.checkAccessStatus(NimServerSession.java:286)
    at com.nimsoft.nimbus.NimServerSession$NimServerSessionThread.handleMessage(NimServerSession.java:165)
    Jan 05 01:30:00:008 [attach_socket, trellis] An exception occurred while processing a message from Socket[addr=/192.168.241.52,port=54246,localport=48040].
    Jan 05 01:30:00:009 [attach_socket, trellis] (13) SID has expired, null
    Jan 05 06:02:57:450 [attach_socket, trellis] An exception occurred while processing a message from Socket[addr=/192.168.241.53,port=61574,localport=48040].
    Jan 05 06:02:57:450 [attach_socket, trellis] (13) SID has expired, null
    Jan 05 08:00:00:008 [attach_socket, trellis] An exception occurred while processing a message from Socket[addr=/192.168.241.52,port=54336,localport=48040].
    Jan 05 08:00:00:008 [attach_socket, trellis] (13) SID has expired, null

     

    Any suggestions? I've tried already stopping, starting but same issue. Also I'm thinking that on my 1st install attempt, since it failed all the probes were shutdown/deactivated and never re-activated so I have only 11/40 probe running. 

     

    I have a case open but just sharing what I'm going thru on this upgrade attempt from a 8.47 to 8.5. Not fun...



  • 3.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 06, 2017 09:37 AM

    So just following up on this, after the 8.5 install finished I had to stop all services then start UIM Services. Went into IM and had to start up all probes manually. Trellis was still not starting so deactivated it along with the nas and prediction were not starting either. After stopping all 3 then activating they all finally started up normally. 

    Support mentioned that maybe due to the fact that I had IM open so I can view the pri-hub probe status while doing the install could of attributed to the issue. 

    On the 2nd attempt I did have IM closed and the installer worked. 



  • 4.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 06, 2017 01:48 PM

    I was curious if I was going to have the same issue so I tried this update. The UIM update went without issue - completely to my surprise. And it was about 3x as fast as the 8.4 upgrade.  I did note that the Trellis probe took a long time to start - it was green in IM with a PID but no port - for maybe 90 seconds or so - no indication in the logs as to what it was doing.

     

    The original error you posted would be a failure to resolve the NimBus address - that could just be a timing issue. I also have seen cases where the controller and hub stop accepting connections for a short period (seconds to minutes) and during that time you get a connection failure. Try it later and it works "fine".

     

    I also had IM open during the time that the upgrade was running - I can't imagine that would have a negative effect on the upgrade process - if it did the installer should have terminated IM to eliminate the conflict. The vast majority of what IM does when it is just sitting there is listening to the message bus and that's a mostly passive activity.

     

    The thing that I did find is that the UMP install causes more harm to the configurations than previous ones. Obviously you lose the deprecated portlets from the configuration because they no longer run with 8.5 (or 8.4) but in addition, you will likely lose all wasp.cfg configurations and whatever is in the file systems supporting those modifications - So if you have done any re-branding you will have to redo that. Similarly, if you are using HTTPS you'll lose that configuration. And if you are using or modifying the default data sources in the new dashboard tool, you will lose those definitions too. Lesson here is make sure that you have a filesystem level copy of your nimsoft install directory on your UMP server and good notes about all the changes you made along the way so that you can make sure those changes get back into the wasp configuration.

     

    -Garin



  • 5.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 18, 2017 02:16 PM

    Hello all,

     

    Thank you for the feedback! We are presently embarking on going from 8.2 > 8.47> 8.51 so wish us luck!

     

    Nonetheless, did anyone lose any custom dashboards, sites or USM groups?

     

    Additionally, will Trellis probe be the default alarming method in 8.5? I am afraid all our AO profiles and scripts will be rendered useless...

     

    Thanks again for any additional info!

     

    A



  • 6.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 18, 2017 02:23 PM

    So it depends specifically what you mean by "custom dashboards". The legacy flash based dashboard portlet is gone in 8.4 and newer. There is a migration tool that is supposed to migrate your dashboards from the legacy format to the newer HTML5 based tool. I personally found that this tool was worse than useless and wound up recreating all our dashboards from scratch.

     

    There's the typical expected losses when you upgrade wasp - make sure you make copies of everything and have a tool handy (like WinDiff) to compare he old and new files so you can re-apply your modifications.

     

    USM stayed intact though I depend very little on the grouping. I might not actually realize if it were broken.

     

    -Garin



  • 7.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 19, 2017 02:18 PM

    Hi Alberto,

    As far as I know I did not loose any custom dashboards with this. The old dashboard was depreciated but that was depreciated in 8.2 so nothing new depreciating in the portlets as far as I can see in 8.5.

    This is the portlet list in my 8.5 instance:

     

    Also trellis has nothing to do with alarms and your nas AO/Profiles. Those will still all work the same. 



  • 8.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Jan 19, 2017 02:23 PM

    Hi Daniel,

     

    Thanks for the input. Sorry I confused it with ems probe and the new alarm event handling mechanism.

     

    A



  • 9.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Oct 31, 2017 02:29 PM

    Hey all,

    I am having this same issue of ADE probe failure while installing UIM. I have tried 3 times and each time I am getting the same error at 35%. Kindly help

     

    Thanx

    Anmol



  • 10.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Broadcom Employee
    Posted Oct 31, 2017 02:56 PM

    I would set the following values

    in the robot.cfg

    loglevel = 3

    logsize = 5000

    in the hub.cfg

    loglevel = 3

    logsize = 35000

     

    in the ade.cfg

    loglevel = 5

    logsize = 35000

     

    check the logs when the problem hits and check the

    C:\tmp\ca_uim\uimserver_ia_install.log

     

    if the hub and robot are not on 7.91 you might want to try updating these first then run the upgrade.



  • 11.  Re: 8.5 Upgrade Install Failure on ADE probe startup

    Posted Nov 01, 2017 06:03 AM

    Hi Gene,

     

    Actually this is a fresh installation I am doing for my dev env.

     

    thanx