AnsweredAssumed Answered

MOM losing connectivity with collectors (Outgoing mailbox is closed)

Question asked by giuseppe01 on Aug 1, 2012
Latest reply on Sep 29, 2014 by Ralstop
Hello Everyone,

I hope you are all enjoying some vacation time this summer. =)

We are facing some recurring issues with our MOM (v9.1.0.2) getting disconnected from collectors (they disappear from the Investigator). This requires an MOM restart.

Below are some error excerpts. I was wondering if anyone has come across these errors.

Thank you very much,
Giuseppe

8/01/12 04:12:18.559 PM GMT+00:00 [ERROR] [PO:main Mailman 4] [Manager.ThreadDump] Authorization for Thread dump not granted
(many of these)

8/01/12 04:13:00.234 PM GMT+00:00 [WARN] [Event Pump Entity] [Manager] com.wily.isengard.message.MessageUndeliverableException
8/01/12 04:13:00.234 PM GMT+00:00 [WARN] [PO:main Mailman 8] [Manager] com.wily.isengard.message.MessageUndeliverableException

8/01/12 04:17:35.869 PM GMT+00:00 [WARN] [Collector phxamspas05.ssd.star@5003] [Manager.Cluster] Waited 15000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.server.beans.console.IConsoleService.ping, v1, []} from address Workstation_391.client_main:263 to service address Server.main:273 from thread Collector phxamspas05.ssd.star@5003 -- We will keep waiting and don't log further messages until we receive the reply or time out
(several of these)

8/01/12 04:18:33.761 PM GMT+00:00 [ERROR] [Collector phxamspas04.ssd.star@5001] [Manager.Cluster] Caught exception trying to get the difference between MOM and this Collector's harvest time: Collector phxamspas04.ssd.star@5001: com.wily.isengard.message.MessageUndeliverableException: Outgoing mailbox is closed. Message cannot be sent
(many of these)

then:

8/01/12 04:18:56.286 PM GMT+00:00 [WARN] [pool-345-thread-1] [Manager] Resource alert data was not retrived from collector:8
com.wily.isengard.message.MessageUndeliverableException
at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:173)
at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:356)
at $Proxy116.getApplicationAgents(Unknown Source)
at com.wily.introscope.application.em.frontend.FrontendApplicationServerBean.doUpdateApplicationAgentMap(FrontendApplicationServerBean.java:449)
at com.wily.introscope.application.em.frontend.FrontendApplicationServerBean.access$9(FrontendApplicationServerBean.java:429)
at com.wily.introscope.application.em.frontend.FrontendApplicationServerBean$1.run(FrontendApplicationServerBean.java:423)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
8/01/12 04:18:56.292 PM GMT+00:00 [WARN] [ClusterManager Async Executor] [Manager] Unable to update load balancing for collector "phxamspas05.ssd.star@5001"
com.wily.isengard.message.MessageUndeliverableException
at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:173)
at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:356)
at $Proxy113.removeCollector(Unknown Source)
at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadBalancer.removeCollector(ClusteredLoadBalancer.java:567)
at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadBalancerBean.collectorsRemoved(ClusteredLoadBalancerBean.java:387)
at com.wily.introscope.spec.server.beans.clusters.ClusterNotification.dataRemoved(ClusterNotification.java:28)
at com.wily.isengard.ongoingquery.AbstractQueryServiceManager$NotifyRemoved.run(AbstractQueryServiceManager.java:438)
at com.wily.isengard.ongoingquery.QueryServiceManager2$1.execute(QueryServiceManager2.java:46)
at com.wily.isengard.ongoingquery.QueryServiceManager2.runNotification(QueryServiceManager2.java:85)
at com.wily.isengard.ongoingquery.AbstractQueryServiceManager.stateRemoved(AbstractQueryServiceManager.java:231)
at com.wily.introscope.server.beans.AOngoingQueriableBean.stateRemoved(AOngoingQueriableBean.java:89)
at com.wily.introscope.server.beans.clusters.ClusterManager.collectorRemoved(ClusterManager.java:238)
at com.wily.introscope.server.beans.clusters.ClusterManager.access$1(ClusterManager.java:222)
at com.wily.introscope.server.beans.clusters.ClusterManager$CollectorRemovedCommand.run(ClusterManager.java:334)
at com.wily.EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:88)
at java.lang.Thread.run(Thread.java:662)


8/01/12 04:18:56.316 PM GMT+00:00 [WARN] [UnknownHub Hub Receive 171] [Manager.Cluster]

8/01/12 04:18:56.331 PM GMT+00:00 [WARN] [master clock] [Manager.Clock] Timeslice processing delayed due to system activity. Combining data from timeslices 89589189 to 89589195
8/01/12 04:18:56.347 PM GMT+00:00 [WARN] [UnknownHub Hub Receive 166] [Manager.Cluster] Stream closed.
8/01/12 04:18:56.350 PM GMT+00:00 [WARN] [UnknownHub Hub Receive 166] [Manager.Cluster] Stream closed.
8/01/12 04:18:56.350 PM GMT+00:00 [WARN] [UnknownHub Hub Receive 166] [Manager.Cluster] Stream closed.

8/01/12 04:18:56.935 PM GMT+00:00 [WARN] [ClusterManager Async Executor] [Manager] Unable to update load balancing for collector "phxamspas03.ssd.star@5002"
com.wily.isengard.message.MessageUndeliverableException: Outgoing mailbox is closed. Message cannot be sent
at com.wily.isengard.postoffice.Mailbox.sendMessage(Mailbox.java:238)
at com.wily.isengard.messageprimitives.service.AAsyncMessageServiceClient.sendRequestAsync(AAsyncMessageServiceClient.java:113)
at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:159)
at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:356)
at $Proxy113.removeCollector(Unknown Source)
at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadBalancer.removeCollector(ClusteredLoadBalancer.java:567)
at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadBalancerBean.collectorsRemoved(ClusteredLoadBalancerBean.java:387)
at com.wily.introscope.spec.server.beans.clusters.ClusterNotification.dataRemoved(ClusterNotification.java:28)
at com.wily.isengard.ongoingquery.AbstractQueryServiceManager$NotifyRemoved.run(AbstractQueryServiceManager.java:438)
at com.wily.isengard.ongoingquery.QueryServiceManager2$1.execute(QueryServiceManager2.java:46)
at com.wily.isengard.ongoingquery.QueryServiceManager2.runNotification(QueryServiceManager2.java:85)
at com.wily.isengard.ongoingquery.AbstractQueryServiceManager.stateRemoved(AbstractQueryServiceManager.java:231)
at com.wily.introscope.server.beans.AOngoingQueriableBean.stateRemoved(AOngoingQueriableBean.java:89)
at com.wily.introscope.server.beans.clusters.ClusterManager.collectorRemoved(ClusterManager.java:238)
at com.wily.introscope.server.beans.clusters.ClusterManager.access$1(ClusterManager.java:222)
at com.wily.introscope.server.beans.clusters.ClusterManager$CollectorRemovedCommand.run(ClusterManager.java:334)
at com.wily.EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:88)
at java.lang.Thread.run(Thread.java:662)
8/01/12 04:18:57.005 PM GMT+00:00 [WARN] [Collector phxamspas04.ssd.star@5001] [Manager.Cluster] Not waiting for response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: {com.wily.introscope.spec.server.beans.clocksync.IClockSyncAgent.getTimeSkew, v1, [com.wily.introscope.spec.server.beans.clocksync.ClockSyncHandshakeCallback@4ae9d]} from address Workstation_1542.client_main:260 to service address Server.main:308 from thread Collector phxamspas04.ssd.star@5001

Outcomes