DX Application Performance Management

  • 1.  [Manager.Cluster] Connection reset

    Posted Mar 13, 2018 05:41 PM

    What can be the cause of these warningarning messages?

     

    3/13/18 04:21:33.214 PM CLST [WARN] [UnknownHub Hub Receive 478] [Manager.Cluster] Connection reset
    3/13/18 04:21:33.214 PM CLST [WARN] [UnknownHub Hub Receive 478] [Manager.Cluster] Connection reset
    3/13/18 04:21:33.214 PM CLST [WARN] [UnknownHub Hub Receive 478] [Manager.Cluster] Connection reset
    3/13/18 04:21:33.214 PM CLST [WARN] [UnknownHub Hub Receive 478] [Manager.Cluster] Connection reset
    3/13/18 04:21:40.578 PM CLST [WARN] [pool-5-thread-1] [Manager.Cluster] [Collector lnxp-caagentes@5001 is sending data slowly to the MOM and will be disconnected now. MOM will try to reconnect to the collector. [TSIndexDiffFromMom : 60 cycles,MaxAllowedTSIndexDiffFromMom :60,044 cycles ]
    3/13/18 04:21:41.112 PM CLST [WARN] [UnknownHub Hub Receive 474] [Manager.Cluster]

    3/13/18 04:21:41.112 PM CLST [WARN] [UnknownHub Hub Receive 474] [Manager.Cluster]

    3/13/18 04:21:41.112 PM CLST [WARN] [UnknownHub Hub Receive 474] [Manager.Cluster]

    3/13/18 04:21:41.112 PM CLST [WARN] [UnknownHub Hub Receive 474] [Manager.Cluster]

    3/13/18 04:21:41.112 PM CLST [WARN] [UnknownHub Hub Receive 474] [Manager.Cluster]

    3/13/18 04:21:41.145 PM CLST [WARN] [UnknownHub Hub Receive 476] [Manager.Cluster]

    3/13/18 04:21:41.145 PM CLST [WARN] [UnknownHub Hub Receive 476] [Manager.Cluster]

    3/13/18 04:21:41.205 PM CLST [WARN] [UnknownHub Hub Receive 476] [Manager.Cluster]

    3/13/18 04:21:43.484 PM CLST [WARN] [pool-5-thread-1] [Manager.Cluster] [Collector lnxp-emcanal@5001 is sending data slowly to the MOM and will be disconnected now. MOM will try to reconnect to the collector. [TSIndexDiffFromMom : 60 cycles,MaxAllowedTSIndexDiffFromMom :61,183 cycles ]
    3/13/18 04:21:43.484 PM CLST [WARN] [pool-5-thread-1] [Manager.Cluster] [Collector lnxp-caagentes2@5001 is sending data slowly to the MOM and will be disconnected now. MOM will try to reconnect to the collector. [TSIndexDiffFromMom : 60 cycles,MaxAllowedTSIndexDiffFromMom :60,363 cycles ]
    3/13/18 04:21:43.484 PM CLST [WARN] [pool-5-thread-1] [Manager.Cluster] [Collector lnxp-emcross@5001 is sending data slowly to the MOM and will be disconnected now. MOM will try to reconnect to the collector. [TSIndexDiffFromMom : 60 cycles,MaxAllowedTSIndexDiffFromMom :61,764 cycles ]
    3/13/18 04:21:43.562 PM CLST [WARN] [UnknownHub Hub Receive 477] [Manager.Cluster] Connection reset
    3/13/18 04:21:43.562 PM CLST [WARN] [UnknownHub Hub Receive 477] [Manager.Cluster] Connection reset
    3/13/18 04:21:43.562 PM CLST [WARN] [UnknownHub Hub Receive 477] [Manager.Cluster] Connection reset
    3/13/18 04:21:43.562 PM CLST [WARN] [UnknownHub Hub Receive 477] [Manager.Cluster] Connection reset



  • 2.  Re: [Manager.Cluster] Connection reset

    Broadcom Employee
    Posted Mar 13, 2018 05:55 PM

    Hi Fernando,

    There could be several possible root causes but in general it indicates a performance problem on the MOM or the collector(s). The fact that it occurs for multiple collectors at same time tends to point at MOM. 

    Some advice here:

    Common Error Messages with Resolution - CA Application Performance Management - 10.7 - CA Technologies Documentation 

    Previous similar community thread here: Wily Collectors missing in CA APM 

    How frequently are these messages occurring in the MOM EM log?

    (For another customer I have seen it recently coincide with MOM is having a long Garbage Collection time (perflog.txt))

     

    Thanks

     

    Lynn



  • 3.  Re: [Manager.Cluster] Connection reset

    Posted Mar 13, 2018 10:03 PM

    Thanks Lynn



  • 4.  Re: [Manager.Cluster] Connection reset
    Best Answer

    Posted Mar 13, 2018 06:42 PM

    If you are taking about the following this is because the collector is busy connecting metrics and sending metrics to MOM.

    3/13/18 04:21:43.484 PM CLST [WARN] [pool-5-thread-1] [Manager.Cluster] [Collector lnxp-emcross@5001 is sending data slowly to the MOM and will be disconnected now. MOM will try to reconnect to the collector.

    check the collector "lnxp-emcross" capacity. 

    1. How many metrics in this collector
    2. How many agent in this collector
    3. Total memory vs heap size

    If following doesn't added in IntrosocpeEnterpriseManager.properties file please add

     

    transport.override.isengard.high.concurrency.pool.max.size=10
    transport.override.isengard.high.concurrency.pool.min.size=10
    transport.outgoingMessageQueueSize=6000

     

     

     



  • 5.  Re: [Manager.Cluster] Connection reset

    Broadcom Employee
    Posted Mar 14, 2018 01:22 PM

    Hi Fernando, I can confirm You that you can solve the issue with performance tuning changes. We had a similar problem 2 years ago: 

     

    https://comm.support.ca.com/csupport/CaseManagement/cases/500a0000012k82nAAA/d

     

     

    Regards,

    Marco