DX Application Performance Management

Document ID:  TEC588643

Last Modified Date:  3/19/2013
Authored By: SergioMorales

• Products
  • CA Application Performance Management
• Components
  • APPLICATION PERFORMANCE MANAGEMENT

Description:Technical Advisory: APM/Introscope 9.1.x Agent Transaction Structure Memory LeakCA Technologies has identified a memory leak issue with CA APM/Introscope 9.1.x agents that can have an operational impact for our customers. This technical advisory describes the affected versions and platforms in more detail, and explains the workarounds or remediation actions that are recommended to prevent this issue from impacting your application environments.Problem DescriptionAPM/Introscope 9.1 introduced a new agent transaction structure along with new tracers that replaces the transaction blame stacks that the old tracers were using in previous releases. This new implementation was designed to optimize computation and processing to improve overall agent performance. Under certain conditions, as described below, objects will remain in memory and are not garbage collected appropriately. Ultimately, when the system runs out of memory, the monitored application will crash or be rendered unusable.Conditions identified:

1. Transaction elements in the WilyTransactionStructure remain in memory and are not garbage collected appropriately even as the structure is aged out over time. Generated heap dumps will show a large number of WilyTransactionElements objects accumulating in the heap over time.
   
   
2. The agent aborts tracing when it encounters an excessive number of errors related to the WilyTransactionStructure and its virtual stack similar to the one below:
   
   [ERROR] [IntroscopeAgent.WilyTransactionStructure] Internal agent error in transaction transition: The stack is corrupted on element com.wily.introscope.agent.blame.VirtualStack$VirtualElement@...
   
   This error occurs with the mixed use of APM 9.1.x tracer definitions together with older tracer definitions. When the agent aborts the trace due to these errors, certain objects do not get garbage collected, leading to a potential memory leak. Heap dumps generated will show a large number of WilyTransactionElements and VirtualStack elements.
   
   
3. Both the NIO and regular socket tracers in APM/Introscope 9.1.x can potentially leak memory over time. When this occurs, the heap dump generated will show a large number of InvocationData objects accumulating in the heap.

Who is impacted?This is a generic agent issue that impacts all application servers and platforms that the CA APM/Introscope agent supports. Below is a list of versions that are affected:

• APM/Introscope Agent versions 9.1.0.0 - 9.1.0.3
  
  
• APM/Introscope Agent version 9.1.1.0

You are not affected by this issue if:You are using APM/Introscope agent versions 9.0.x and below

Solution:This issue is addressed in CA APM/Introscope 9.1.1.1 (GA end of August 2012) and 9.1.0.4 (GA end of September 2012).

Workaround:If a memory leak occurs due to NIO or regular socket tracing as indicated by the large number of InvocationData objects in the heap dump, turning off NIO socket tracing and regular socket tracing or using managed socket tracing will provide a viable workaround.In the interim, the 9.1.x agent offers a "legacy" mode option that will revert to old tracer definitions that leverages the traditional "pre-APM 9.1" transaction blame stacks as opposed to the new transaction structure. Note that this is a supported but deprecated option in the APM 9.1 agent and should only be used temporarily until customers upgrade to the recommended patch. To configure the agent to use this "legacy" option, please follow the steps below:

1. Stop the monitored application
   
   
2. Archive and delete existing log files in the <Agent_Home>/logs directory to prepare for new logs
   
   
3. Back up existing .pbl and .pbd files in the <Agent_Home>/core/config directory
   
   
4. Back up the existing <Agent_Home>/core/config/IntroscopeAgent.profile
   
   
5. Copy the legacy .pbl and .pbd files from the <Agent_Home>/examples/legacy directory to the <Agent Home>/core/configdirectory.
   
   
6. Open <Agent_Home>/core/config/IntroscopeAgent.profile and make the following changes:
   
   
   1. Add a new property: introscope.agent.configuration.old=true
      
      
   2. Update the agent property introscope.autoprobe.directivesFile to point to the appropriate legacy .pbl and/or .pbd files that have been copied over. For example, replace spm.pbl with spm-legacy.pbl
      
      
7. Restart the monitored application

Search the Entire CA APM Knowledge Base