DX Application Performance Management

Hi,

Introscope is causing performance issues. As in whenever Introscope agent is switched on,  it takes 4-5 mins to jump between screens which normally takes10-15 secs.

I tried switching off SQL metrics and JMX metrics with no luck.

Can anyone please suggest what can root cause for this ?

The solution is going to be specific to whichever application framework is being used so a root cause will need investigation.

A common test is to turn off AutoProbe completely in the agent profile - introscope.autoprobe.enable = false

just to check that performance is fine with the agent loaded but no instrumentation happening.

If it is fine with AutoProbe disabled, and you have a lot of time and patience, you can reactivate instrumentation by setting the above property back to true and you can go through toggles-typical.pbd and switch off options one by one to check for any that significantly improve performance.

Indeed if you are already using full instrumentation instead of typical,  you can tell this because you will have something saying ....full.pbl instead of typical.pbl, make sure you use the typical pbl.

Also disable any custom instrumentation, that is pbds that you wrote yourself or you added that were not delivered directly with the agent.

After that it is most useful to review transaction traces to see what the application actually does when you 'jump between screens', it gives a picture of the code involved, and then consider disabling instrumentation of metric types that appear there, perhaps Struts or XML processing if that sort of thing is enabled.

As a general rule it would be useful to look for any metrics that have high values of Responses Per Interval or Concurrent Invocations, it could be something like an EJB that is busy processing whatever the screen takes as input.

If there are calls to external resources through Sockets, there can be a slowdown at this point so you can try disabling Socket Tracing.

We can go on and on suggesting what it could be..

But you should start by reviewing the transaction traces to get a picture of what a jump between screens actually means in code terms.

Hi,

read this blog: Hunting and Eliminating Excessive Overhead (and all other blogs by spyderjacks)!

Ciao,

Guenter

Hi,

As you stated,

• check CPU, GC times and heap to identify resource bottlenecks (might be root cause or symptom!)
• check metric counts and number of invocations for metrics to identify metric explosion and over-instrumentation

I cannot check these stats because Introscope is currently switched off. Is there anything I can check from profile file ? And should be the default values set for the above mentioned metrics ?

Hi,

We are instrumenting JBoss servers and as you stated we are using jboss-full.pbl. I tried searching for jboss-typical.pbl but couldn't find the same.

I checked the logs and found many warnings around transaction tracer as stated below.

[IntroscopeAgent.WilyTransactionStructure] Transaction trace component limit of 5000 reached, recording of any new components will cease for this transaction.

and error around

[ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to Introscope-enable abc/fgh/FaultCodeDaoMoney: Invalid class file

Please help.

Hi,

This is another indication of over-instrumentation. As self-protection CA APM does not track more than 5000 transaction elements. That means that a single transaction contains more than 5000 method calls. And those should be significant method calls, not getters/setters or simple methods. Most of those 5000+ methods will not contribute significantly to the duration and logic of the transaction and therefore should be removed from the configuration by either removing TurnOn directives or adding Skip... directives to the pbds. This is described in the blog I cited above.

The error message indicates that something is wrong with the class file. Either a version mismatch (check the CA APM Compatibility Guides) or a framework/tool is not strictly complying with the Java specification (e.g. JAXB-generated classes).

Ciao,

Guenter

You can check all of the metrics historically even when the agent is not running. You may have to "mount" the agent if it is not visible in the Investigator metric tree.

CPU can be tracked by operating system monitors like Task Manager/Resource Monitor on Windows and top on Linux. GC and heap can be monitored using JConsole or other tools. Just google!

Ciao,

Guenter

Actually that the first statement is not correct. When the unmounted agent is mounted you do not get back the Historical entries under "Custom Metric Agent (Virtual)....|Agents|Host|Process|agent|(.*)Metric Count" nor from SmartTools (Case 183099)