DX Application Performance Management

  • 1.  Collector error message, not in KB

    Posted Sep 28, 2017 07:39 AM

    Hi - is there any detail available as to what this issue may be ? I cant find anything so far in KB.

     

    Happy to raise a support case if required.

     

    Thanks

     

    20170928 120131 BST [WARN] [Manager.BaselineEngine.AutoTrace] AgentThresholdDeliveryService: Timeout trying to process and send autotrace triggers
    20170928 120131 BST [WARN] [Manager.BaselineEngine.AutoTrace] AgentThresholdDeliveryService: attempt to sent autotrace triggers, lost connection for: SuperDomain|slzpsr000000082|JBoss|AS2Remediation_PRODSG01-server03
    com.wily.isengard.message.MessageUndeliverableException: Caller thread interrupted
    at com.wily.isengard.messageprimitives.service.MessageServiceClient.blockOnResponse(MessageServiceClient.java:307)
    at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:163)
    at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:356)
    at com.sun.proxy.$Proxy144.setAutoTracingTriggers(Unknown Source)
    at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl.callAgentTriggerService(AgentThresholdDeliveryServiceImpl.java:523)
    at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl.sendTriggers(AgentThresholdDeliveryServiceImpl.java:485)
    at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl.access$34(AgentThresholdDeliveryServiceImpl.java:431)
    at com.ca.apm.baseline.thresholds.AgentThresholdDeliveryServiceImpl$3.run(AgentThresholdDeliveryServiceImpl.java:397)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)



  • 2.  Re: Collector error message, not in KB
    Best Answer

    Broadcom Employee
    Posted Sep 28, 2017 09:44 AM

    Hi Dave,

     

    There is the "Differential Analysis" feature ( https://docops.ca.com/ca-apm/10-5/en/administrating/manage-metric-data-by-using-management-modules/configure-differential-analysis ) that baselines frontend metrics and then monitors and detects, if a metric exceeds baselined value by some sigma analyses. If value exceeds boundaries, EM tries to set the baseline value as an auto trace threshold for that metric in agent in order to automatically trace all transactions that are above the baseline.

     

    The WARN message means that setting the threshold for the metric in agent failed. Most likely, because agent did not find required metric in its metric pool or there was a connection problem between EM-Agent and therefore cannot set the threshold value.
    The impact is that the auto tracing part of this feature does not work for the metric in question.

     

    From agent perspective things are quite simple, it receives the request from EM with metric name and threshold. It looks it up and sets value. When transactions happen, value is checked and auto tracing is triggered.

     

    Agent does have DEBUG and TRACE logs to track these requests to see what metric EM is setting, where agent is looking for it and, if and why it fails:


    log4j.logger.IntroscopeAgent.IntelligentInstrumentationService=TRACE#com.wily.util.feedback.Log4JSeverityLevel

     

    Suggestions:

     

    1) Check for any possible Agent-EM connection issue, check the Agent logs, use the Agent supportability metrics
    2) Enable TRACE for SI as indicated above, try to correlate the WARN messages with the agent logs and find out if the EM is trying to set threshold incorrectly.
    3) Go to the SuperDomain|slzpsr000000082|JBoss|AS2Remediation_PRODSG01-server03, "traces" tab in the investigator and try to find the failing traces.

     

    If you are unable to identify the root cause, open a support case and attach EM + agent TRACE logs + a backup of your traces db.

     

    I hope this helps,
    Regards,
    Sergio



  • 3.  Re: Collector error message, not in KB

    Posted Sep 28, 2017 10:24 AM

    Thanks Sergio, I will have a look and get back to you