DX Unified Infrastructure Management

Expand all | Collapse all

NMS + UMP 7.60 - wasp does not start

  • 1.  NMS + UMP 7.60 - wasp does not start

    Posted Jul 31, 2014 02:12 PM

    We have finaly decided to upgrade to 64bits NMS system-enviroment. We ccome from 6.20

    We use 3 seperate servers for the main taks. nms, database and ump. NMS and databse is running ok, so it looks, but ump server will not start he wasp probe. It comes withe several error messages:

     

    Wasp:

    jul 31 13:07:07:252 WARN  [pool-1-thread-1, com.nimsoft.nimbus.probe.service.wasp.udm.UDM] Unable to initialize data update from dashboard_engine.
    jul 31 13:07:07:252 WARN  [pool-1-thread-1, com.nimsoft.nimbus.probe.service.wasp.udm.UDM] (7) temporarily out of resources, Received status (7) on response (for sendRcv) for cmd = 'InitializeDataUpdate'

     

    Dashboard_engine:

    jul 31 13:06:03:094 ERROR [Callback_initializeDataUpdate-410, Utilities:89] Error occurred during probe status verification: Probe Name = /eRemote/CentralHub/ump-webserver/wasp Error: (4) not found, Received status (4) on response (for sendRcv) for cmd = 'nametoip' name = '/eRemote/CentralHub/ump-webserver/wasp'

     

    All system have a minimal resource 2 cores and 8gbyte of memory. Java is set toL min1024, max4048, that should be enough I think.

      

    And also:

    As I look in the probe list on our NMS server there is no varaiable_server running. As I understand it is deprectiated. Why is the dashboard_engine on the UMP server aksing for a varaiable_server.

     

    varaiable.PNG

     

    Does anyone have a tip or clue for me why Wasp does not start?

     

    regards

    Ashley Pietersen



  • 2.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Jul 31, 2014 02:29 PM
    Note: I have tested it before although the test was based NMS + UMP both running on same test-server.

    regards
    Ashley Pietersen


  • 3.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Jul 31, 2014 02:30 PM

    The log messages in wasp.log are quite common. We always see those as well whenever wasp is starting. Would be nice if it could tell what kind of resources it was out of, so perhaps we could give it some more, but oh well.

     

    The dashboard_engine.log however is something else. I assume the robot running ump uses the primary hub.

     

    I think I've seen the log message before, but then it was just a matter of things not having started properly yet.

     

    Sure it just didn't take long™ to start? Like 5-10 minutes?

     

    Running the ump on a dedicated robot shouldn't be a problem at all. We do that in all our environments.



  • 4.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Jul 31, 2014 02:46 PM

    Hi Andre,

     

    Yes, UMP is "talking" to the primary hub (CentralHub)

     

    Installing UMP 7.60 goes well until the last phase "Starting UMP" - it is enden with a timeout message about 98%. Looking in the instal_log I do not see anything else then Succeful.

     

    We have Nimsoft for many years running on three seperate machines, and works OK. Installing a new UMP version we often have  this kind of initial trouble unfortunately. Have deinstalled and installed UMP for several time now but can not get it working.

    It uses the same vm-machine we used for ump 6.20. Firewall ports and sort of security issues shoud'nt be a problem.

     

    Ashley



  • 5.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Jul 31, 2014 02:52 PM

    BTW: is the dahboard_server on the primary-hub also deprciated with NMS version 7.x ? Can not find it anymore. Where is the dashboard_engine talking to? What probes on the primary hub do I need to check for problems?



  • 6.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Jul 31, 2014 04:16 PM

    Remarkable:

     

    In the meantime we have stepped back to NMS/UMP 6.20.

     

    looking in the dircetory C:\Program Files (x86)\Nimsoft\probes\service\wasp\work\wasp-engine\localhost on the "old" UMP server we saw this: (see screenshot below)

     

    Lokking in the same directory on the "new" UMP760 this direfctory is empty!

     

     

    wasp-2.PNG



  • 7.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Jul 31, 2014 04:44 PM

    We had kind of a similar issue when we went to 7.5 turned out to be corrupt liferay.  Basically deleting all the liferay tables and letting them get recreated resolved it.  Of course that then makes you do any sort of customization all over again.  Going from 6.x to 7.5 or 6 is going to contain a new liferay version.



  • 8.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Aug 01, 2014 01:10 PM

    Do you have a script or sql-querry for that ? Manually deleting all the Liferay tables makes me some kind of nervous.

     

    regards

    Ashley Pietersen



  • 9.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Aug 04, 2014 11:07 PM

    Here are the tables got it from support.

     

    drop table Account_;
    drop table Address;
    drop table AnnouncementsDelivery;
    drop table AnnouncementsEntry;
    drop table AnnouncementsFlag;
    drop table BlogsEntry;
    drop table BlogsStatsUser;
    drop table BookmarksEntry;
    drop table BookmarksFolder;
    drop table BrowserTracker;
    drop table CalEvent;
    drop table ClassName_;
    drop table Company;
    drop table Contact_;
    drop table Counter;
    drop table Country;
    drop table CyrusUser;
    drop table CyrusVirtual;
    drop table DLFileEntry;
    drop table DLFileRank;
    drop table DLFileShortcut;
    drop table DLFileVersion;
    drop table DLFolder;
    drop table EmailAddress;
    drop table ExpandoColumn;
    drop table ExpandoRow;
    drop table ExpandoTable;
    drop table ExpandoValue;
    drop table Group_;
    drop table Groups_Orgs;
    drop table Groups_Permissions;
    drop table Groups_Roles;
    drop table Groups_UserGroups;
    drop table IGFolder;
    drop table IGImage;
    drop table Image;
    drop table JournalArticle;
    drop table JournalArticleImage;
    drop table JournalArticleResource;
    drop table JournalContentSearch;
    drop table JournalFeed;
    drop table JournalStructure;
    drop table JournalTemplate;
    drop table Layout;
    drop table LayoutSet;
    drop table ListType;
    drop table MBBan;
    drop table MBCategory;
    drop table MBDiscussion;
    drop table MBMailingList;
    drop table MBMessage;
    drop table MBMessageFlag;
    drop table MBStatsUser;
    drop table MBThread;
    drop table MembershipRequest;
    drop table Organization_;
    drop table OrgGroupPermission;
    drop table OrgGroupRole;
    drop table OrgLabor;
    drop table PasswordPolicy;
    drop table PasswordPolicyRel;
    drop table PasswordTracker;
    drop table Permission_;
    drop table Phone;
    drop table PluginSetting;
    drop table PollsChoice;
    drop table PollsQuestion;
    drop table PollsVote;
    drop table Portlet;
    drop table PortletItem;
    drop table PortletPreferences;
    drop table RatingsEntry;
    drop table RatingsStats;
    drop table Region;
    drop table Release_;
    drop table Resource_;
    drop table ResourceAction;
    drop table ResourceCode;
    drop table ResourcePermission;
    drop table Role_;
    drop table Roles_Permissions;
    drop table SCFrameworkVersi_SCProductVers;
    drop table SCFrameworkVersion;
    drop table SCLicense;
    drop table SCLicenses_SCProductEntries;
    drop table SCProductEntry;
    drop table SCProductScreenshot;
    drop table SCProductVersion;
    drop table ServiceComponent;
    drop table Shard;
    drop table ShoppingCart;
    drop table ShoppingCategory;
    drop table ShoppingCoupon;
    drop table ShoppingItem;
    drop table ShoppingItemField;
    drop table ShoppingItemPrice;
    drop table ShoppingOrder;
    drop table ShoppingOrderItem;
    drop table SocialActivity;
    drop table SocialRelation;
    drop table SocialRequest;
    drop table Subscription;
    drop table TasksProposal;
    drop table TasksReview;
    drop table User_;
    drop table UserGroup;
    drop table UserGroupRole;
    drop table UserIdMapper;
    drop table Users_Groups;
    drop table Users_Orgs;
    drop table Users_Permissions;
    drop table Users_Roles;
    drop table Users_UserGroups;
    drop table UserTracker;
    drop table UserTrackerPath;
    drop table VirtualHost;
    drop table Vocabulary;
    drop table WebDAVProps;
    drop table Website;
    drop table WikiNode;
    drop table WikiPage;
    drop table WikiPageResource;
    drop table QUARTZ_JOB_DETAILS;
    drop table QUARTZ_JOB_LISTENERS;
    drop table QUARTZ_TRIGGERS;
    drop table QUARTZ_SIMPLE_TRIGGERS;
    drop table QUARTZ_CRON_TRIGGERS;
    drop table QUARTZ_BLOB_TRIGGERS;
    drop table QUARTZ_TRIGGER_LISTENERS;
    drop table QUARTZ_CALENDARS;
    drop table QUARTZ_PAUSED_TRIGGER_GRPS;
    drop table QUARTZ_FIRED_TRIGGERS;
    drop table QUARTZ_SCHEDULER_STATE;
    drop table QUARTZ_LOCKS;

    drop table AssetCategory;
    drop table AssetCategoryProperty;
    drop table AssetEntries_AssetCategories;
    drop table AssetEntries_AssetTags;
    drop table AssetEntry;
    drop table AssetLink;
    drop table AssetTag;
    drop table AssetTagProperty;
    drop table AssetTagStats;
    drop table AssetVocabulary;
    drop table ClusterGroup;
    drop table LayoutPrototype;
    drop table LayoutSetPrototype;
    drop table Lock_;
    drop table SocialEquityAssetEntry;
    drop table SocialEquityHistory;
    drop table SocialEquityLog;
    drop table SocialEquitySetting;
    drop table SocialEquityUser;
    drop table Team;
    drop table Ticket;
    drop table UserGroupGroupRole;
    drop table Users_Teams;
    drop table WorkflowDefinitionLink;
    drop table WorkflowInstanceLink;
    drop table WSRP_WSRPConsumer;
    drop table WSRP_WSRPConsumerPortlet;
    drop table WSRP_WSRPProducer;



  • 10.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Aug 04, 2014 11:33 PM

    Hi Neal,

     

    Wow that is quit a list and very helpful ! Thanks !  And thanks again !

    And also thanks to support ofcourse !

     

    Best regards

    Ashley Pietersen



  • 11.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 09, 2014 02:56 PM

    We just upgraded UMP from 6.6 to 7.6 ourselves, and the wasp used 30min to start the first time after the upgrade.

    wasp.log just logged the normal resources thing, but portal.log was actually doing stuff. So it took 30min for it to do all the database updates it needed to do (which is what it was doing).



  • 12.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 11, 2014 03:43 AM

    We had issues too with the upgrade - in our case we were running multiple servers and someone restarted one of the older wasp probes accidentally. That resets the "installed version" that liferay tracks and on next start up, it will fail because it can't execute the upgrade again.

     

    See https://na4.salesforce.com/articles/Case_Summary/Applied-Windows-update-to-UMP-servers-and-now-after-a-required-restart-we-are-getting-errors?popup=true

     

    And truncating the Liferay tables is painful. You also lose accounts and any customization users might have done. It essentially winds you back to the start of time.

     



  • 13.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 11, 2014 11:29 AM

    the custom dashboards are converted to html 5 on the wasp start for the first time after the upgrade and this is most likely what has happen here.

     



  • 14.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 13, 2014 05:00 PM

    Still trying and still not working.

     

    Have again prepaired two vm's for upgarde : one nms 760  server w2012R2 2cores/6gbyte mem, and a secons one for UMP 760 on a windows server w2008R2 Datacenter 2Core/6Gbyte mem. Both 100Gbyte HD space.

     

    This is what I get everytime: (no matter how long I wait for any possible database upgrade-scrip to end)  Initial database is a full copy of our prodction database of NMS v6.2.1. (also tried an intermedaite approach by installing 6.50- the first 64bits nms paltform)- no succes either. No detailed messages in any logs that brings me further. Driving me crazy for weeks now...

     

     

    ump-2.JPG 

     



  • 15.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 13, 2014 06:27 PM

    Great an emty wasp log, happy debugging :-(

     

    ump-wasp-log.PNG



  • 16.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 13, 2014 07:39 PM

    A piece of wasp log: note To prevent a memory leak, the ThreadLocal has been forcibly removed.

     

    ThreadLocal with key of type [null] (value [com.sun.xml.bind.v2.runtime.Coordinator$1@4c696ad1]) and a value of type [com.sun.xml.bind.v2.runtime.Coordinator[]] (value [[Lcom.sun.xml.bind.v2.runtime.Coordinator;@1ce3e688]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
    sep 13 18:10:23:807 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearThreadLocalMap() A web application created a ThreadLocal with key of type [scala.concurrent.forkjoin.ForkJoinPool.ThreadSubmitter] (value [scala.concurrent.forkjoin.ForkJoinPool$ThreadSubmitter@c597c22]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@7d1e2b04]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
    sep 13 18:10:23:807 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearThreadLocalMap() A web application created a ThreadLocal with key of type [scala.concurrent.forkjoin.ForkJoinPool.ThreadSubmitter] (value [scala.concurrent.forkjoin.ForkJoinPool$ThreadSubmitter@c597c22]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@479a53c8]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
    sep 13 18:10:24:883 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [attach_clientsession] but has failed to stop it. This is very likely to create a memory leak.
    sep 13 18:10:24:883 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [RedeployManager] but has failed to stop it. This is very likely to create a memory leak.
    sep 13 18:10:25:008 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [RedeployManager] but has failed to stop it. This is very likely to create a memory leak.
    sep 13 18:10:25:211 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [RedeployManager] but has failed to stop it. This is very likely to create a memory leak.
    sep 13 18:10:25:383 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [RedeployManager] but has failed to stop it. This is very likely to create a memory leak.
    sep 13 18:10:25:944 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [RedeployManager] but has failed to stop it. This is very likely to create a memory leak.
    sep 13 18:10:26:662 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearThreadLocalMap() A web application created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@9d87e31]) and a value of type [flex.messaging.io.SerializationContext] (value [flex.messaging.io.SerializationContext@6444c803]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
    sep 13 18:10:27:161 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [RedeployManager] but has failed to stop it. This is very likely to create a memory leak.
    sep 13 18:10:27:302 ERROR [main, com.nimsoft.ui.app.awacs.AwacsLifecycleListener] sendDetailErrorMessage() No detail message broker, so no clients to notify!
    sep 13 18:10:27:302 ERROR [main, com.nimsoft.ui.app.awacs.AwacsLifecycleListener] sendSummaryErrorMessage() No summary message broker, so no clients to notify!
    sep 13 18:10:27:318 ERROR [main, org.apache.catalina.loader.WebappClassLoader] clearReferencesThreads() A web application appears to have started a thread named [RedeployManager] but has failed to stop it. This is very likely to create a memory leak.

     

    great installer! :-)



  • 17.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 14, 2014 09:22 PM

    I am having this exact issue right now with UMP 7.50. Did you ever resolve this?

     



  • 18.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 15, 2014 12:33 AM

    Hi Dblanco,

     

    No could not find any solution. So I followed the suggestion earlier in this post to drop all UMP/Liferay tables from the database and installed UMP again. Now I have a working but half empty UMP environment. It seems this is the best I can get at the moment :-(  Maybe I decide to rebuild the UMP environment completely from scratch, and export/import all my custom dashboards that cost us a lot of time to create.



  • 19.  Re: NMS + UMP 7.60 - wasp does not start

    Posted Sep 28, 2014 03:57 PM

    And now we have this error, somebody knows what to do next?

     

     

     

    sep 28 13:51:09:613 INFO  [main, com.nimsoft.nimbus.NimProbe] ****************[ Starting ]****************
    sep 28 13:51:09:613 INFO  [main, com.nimsoft.nimbus.NimProbe] 7.60


    sep 28 14:16:00:908 ERROR [http-80-3, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/ump-read-only-theme].[default]] invoke() Servlet.service() for servlet default threw exception


    sep 28 14:16:00:909 ERROR [http-80-3, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/ump-read-only-theme].[default]] java.lang.NullPointerException


     at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.getInvokerFilterChain(InvokerFilter.java:184)
     at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:70)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
     at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
     at java.lang.Thread.run(Unknown Source)