DX Application Performance Management

Hello All,    This thread will be used by the CA Wily product management team to relay product advisories on the APM product lines.         Regards,    Nate Isley  APM Product Manager                 

CA Wily Technical Advisory: Data aggregation in SmartStor causes loss of data
CA Wily has uncovered a potential issue where certain configurations may improperly aggregate data points inside SmartStor. This technical advisory describes the affected versions and platforms in more detail, and explains what remediation actions are recommended.

Problem Description
CA Wily has identified several versions of the Enterprise Manager that do not properly aggregate the first level data tier for SmartStor if the first level storage value is set to any value other than 15 seconds. First level storage frequency for SmartStor is controlled by the tier 1 frequency property in the EnterpriseManager.properties file.
Out of the box, this Enterprise Manager property (introscope.enterprisemanager.smartstor.tier1.frequency) is set to 15 seconds. Customers would have to manually change this value to encounter the issue.

Who is Impacted
 These are the affected Enterprise Manager versions:
o 8.0.x
o 8.1.x
o 8.2.x
o 9.0.x

You are not affected by this issue if:
o You are a CEM only customer
o You run the Enterprise Manager with the default 15 second frequency for the first tier data storage interval:
introscope.enterprisemanager.smartstor.tier1.frequency=15

Solution Timing and Workaround
 This issue is addressed or planned to be addressed in the following Introscope versions:
o Introscope 8.0: this issue is currently planned to be addressed in the 8.0.6 service pack tentatively scheduled for the first week of August
o Introscope 8.1: this issue is currently planned to be addressed in the 8.1.3 service pack tentatively scheduled for the first week of September
o Introscope 8.2: this issue is currently planned to be addressed in the 8.2.4 service pack tentatively scheduled for the first week of November
o Introscope 9.0: this issue is currently planned to be addressed in the next planned 9.0 service pack tentatively scheduled for September

In the meantime, to help avoid further possible data loss, please ensure your Enterprise Manager settings for the aggregation of the first level of data storage is set to 15 seconds.

CA Wily Technical Advisory: ChangeDetector performing work even when disabled in the profile
CA Wily has uncovered a potential issue where ChangeDetector may cause overhead in the application under management even when disabled. This technical advisory describes the affected versions and platforms in more detail, and explains what remediation actions are recommended.

Problem Description
CA Wily has identified several versions of the Agent where overhead is incurred on applications running the Agent when ChangeDetectorAgent.jar is placed in the Agent’s ext directory. In these instances, even if the Agent’s profile has ChangeDetector disabled, there may still be overhead due to actions performed by ChangeDetector.

The exact amount of overhead incurred is very dependent on the application architecture. Certain architectures expose this overhead more prominently than others. The issue first came to light via Customers using Pega applications, and we advise Customers that are monitoring Pega applications with Introscope to please take special note of the affected versions and workarounds.

Who is Impacted
 These are the affected Agent versions:
o 7.x
o 8.0.x
o 8.1.x
o 8.2.0 – 8.2.2

You are not affected by this issue if:
o You are using the Introscope 8.2.3 Agent
o You are using the Introscope 9.0 Agent
o You are a CEM only customer

Solution Timing and Workaround
 This issue is addressed or planned to be addressed in the following Introscope versions:
o Introscope 7.x: We have no dates tentatively scheduled for 7.x service packs. Customers are advised to follow the workaround or upgrade to one of the patched 8.x or 9.0 releases.
o Introscope 8.0: this issue is currently planned to be addressed in the 8.0.6 service pack tentatively scheduled for the first week of August
o Introscope 8.1: this issue is currently planned to be addressed in the 8.1.3 service pack tentatively scheduled for the first week of September
o Introscope 8.2: this issue was addressed in the 8.2.3 service pack release

In the meantime, to help avoid incurring further ChangeDetector related overhead, please remove ChangeDetectorAgent.jar from the Agent’s ext directory if you are not using it.

CA Wily Technical Advisory: FullTTDetailsTracer May Cause Deadlock in Application Server Environments
CA Wily has uncovered a potential issue that may cause a deadlock in an application server environment that is running the Introscope 8.x or 9.0 agent, requiring an application server restart. This technical advisory describes the issue in more detail, and explains what remediation actions are recommended.

Problem Description
The exact circumstances that trigger this problem are dependent on a time sensitive sequence of events involving Stall detection, a Stall, specific method implementations, FullTTDetailsTracer, and the BlameStack. When the deadlock in the agent is triggered, the managed application may further slow down or deadlock, requiring an application server restart.

As the issue is dependent on application developer’s implementation of particular methods, the numbers of customers that will encounter this time sensitive sequence is likely quite low.

Who is Impacted
 Application Servers with Agent deployments that have been customized to utilize the FullTTDetailsTracer. The following versions of the Introscope Agent are affected.
o 8.0.x
o 8.1.x
o 8.2.x
o 9.0

 This issue does not affect the Introscope Enterprise Manager, regardless of the Enterprise Manager deployment type (Standalone or Clustered "MOM").

You are not affected by this issue if:
o You are a CEM only customer
o You have disabled stalls in the IntroscopeAgent.profile (introscope.agent.stalls.enable=false)
o You have not implemented FullTTDetailsTracer in your environment

Solution Timing and Workaround
 This issue is addressed or planned to be addressed in the following Introscope versions:
o Introscope 8.0: this issue is currently planned to be addressed in a future service pack.
o Introscope 8.1: this issue is currently planned to be addressed in the 8.1.3 service pack tentatively scheduled for the first week of September
o Introscope 8.2: this issue is currently planned to be addressed in the 8.2.4 service pack tentatively scheduled for the first week of November
o Introscope 9.0: this issue is currently planned to be addressed in the next planned 9.0 service pack tentatively scheduled for September[/b

In the meantime, there are two workarounds recommended to help protect from the occurrence of this issue.
o Remove the custom .pbd additions which implement the FullTTDetailsTracer tracer
o Disable stalls in the IntroscopeAgent.profile:
introscope.agent.stalls.enable=false

Thanks for the great posts Nate!

Chris

CA Wily Technical Advisory: Agent to EM communication state breakdown may cause memory leak

CA Wily has uncovered a potential issue where the Introscope Agent may fall into a bad communication state and slowly leak memory. This technical advisory describes the affected versions and platforms in more detail and explains what remediation actions are recommended.

Problem Description

CA Wily has identified a scenario where the Agent enters into a state where it believes the Enterprise Manager is up and communicating but it is not. In this state there is at least one Vector that slowly builds up data. This may eventually cause the application under management to run out of memory.
Please note that this problem is exacerbated for environments that run the 8.1 MQ PP as there is an additional ArrayList collection which may add data that it may be unable to release.

Problem Symptoms

Agent Side Memory Levels

If there is a communication breakdown and the Agent ends up experiencing this issue, there will be an observable increase in memory levels and GCs for monitored applications in the Workstation.

Introscope Agent Log File:
The Agent log should have evidence that the Agent is stuck in the problematic communication state. If the 1st message below appears without either of the next two for more than a few minutes, then the Agent has likely entered an indefinite blocking state.

•
Waited 120000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage:.....
-
Got the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessag
-
Enterprise Manager responding too slowly, disconnecting

Who is Impacted
• These are the affected Introscope Agent versions:
o 8.0.0 - 8.0.3, 8.1.0 – 8.1.1, and 8.2.0 – 8.2.1
*Note: Some sub-patches of the versions listed above have intermediary fixes, but our final fix is planned for the delivery vehicles listed below

• The applications under management should not be subject to this issue if:
o You are using the Introscope 8.0.4+ agent
o You are using the Introscope 8.1.2+ agent
o You are using the Introscope 8.2.2+ agent
o You are using the Introscope 9.0+ agent
o You are a CEM only customer

**Note: The timing issues involved here are extraordinarily difficult to encounter and reproduce. We believe 7.x Agents will not encounter this situation due to simpler thread architecture.

Solution Timing and Workaround
• This issue will be addressed in the following Introscope agent versions:
o Introscope 9.0: this issue does not occur with 9.x agents
o Introscope 8.2: there was a change in our 8.2.2.0 release such that this issue no longer occurs
o Introscope 8.1: there was a change in our 8.1.2.0 release such that this issue no longer occurs
o Introscope 8.0: there was a change in our 8.0.4.0 release such that this issue no longer occurs
• In the meantime, if you suspect your Agent is in this bad communications state you may be able to free it by explicitly terminating your Enterprise Manager.

For More Information
If you have any questions or concerns, please contact CA | Wily Technology Technical Support at support@wilytech.com, or at any of the following numbers:
•
Toll-Free U.S.: 1 888 GET WILY ext. 1
•
U.S.: +1 630 505 6966
•
Latin America +55 11 5503 6167
•
Europe: +44 (0)1753 242763
•
Asia-Pacific: +81 3 6868 2300
•
Toll-Free Japan: 0120 974 580
•
Singapore: +65 6432 8641
•
Australia: +61 2 8898 2963
•
Call any Ca Technologies office and ask for xtn 46966
To receive automatic notification of any patches or fixes, please subscribe to Knowledge Base article 498, available at http://support.wilytech.com/cgi-bin/wilytech.cfg/php/enduser/std_adp.php?p_faqid=498.

CA Wily Technical Advisory: v9 Managed Sockets update may cause a memory leak

CA Wily has uncovered a potential issue where the Introscope Agent’s newly updated Managed Sockets feature may leak memory. This technical advisory describes the affected versions and platforms in more detail and explains what remediation actions are recommended.

Note, as we will be issuing a shelf replacement patch, please take special care to pay attention to the 4th digit to differentiate agent builds until our next rollup Service Pack is issued. (i.e. 9.0.5.1 vs. 9.0.6.0).

Problem Description
CA Wily has identified a scenario where the new v9 Managed Socket implementation may fail to clean up references to sockets. Encountering this issue is entirely dependent on how an application developer codes their Java application and in many cases applications will not observe or encounter this scenario.

Problem Symptoms
Agent Side Memory Levels

If the Agent encounters this scenario there will be a constant increase in base memory usage in the JVM. This memory increase will be evident in the Workstation when observing the GC Heap levels of the JVM.

When diagnosing if the issue described in this advisory is the cause for the memory condition, an excessive number of ManagedSocketOutputStreamHighPerformance objects in a Java heap dump would confirm the diagnosis.

Who is Impacted
•These are the affected Introscope Agent versions:
o 9.0.0.x & 9.0.5.0
• The applications under management should not be subject to this issue if:
o You are using the Introscope 7.x.x.x agent
o You are using the Introscope 8.x.x.x agent
o You are a .NET only customer
o You are a CEM only customer

Solution Timing and Workaround
• For the following Introscope agent versions:
o Introscope 9.0.0.x: we plan to address this issue in a shelf replacement patch numbered 9.0.0.2 with planned availability on or around November 9th
o Introscope 9.0.5.0: we plan to address this issue in a shelf replacement patch numbered 9.0.5.1 with planned availability on or around November 9th

• In the meantime, to protect your environment from encountering this issue you may change your 9.x agent to the pre-9 managed socket feature by restarting the JVM after commenting out SocketTracing in the toggles file and un-commenting ManagedSocketTracing. i.e. :
# TurnOn: SocketTracing
# NOTE: Only one of SocketTracing and ManagedSocketTracing….
TurnOn: ManagedSocketTracing
TurnOn: UDPTracing

For More Information I
f you have any questions or concerns, please contact CA | Wily Technology Technical Support at support@wilytech.com, or at any of the following numbers:
• Toll-Free U.S.: 1 888 GET WILY ext. 1
• U.S.: +1 630 505 6966
• Latin America +55 11 5503 6167
• Europe: +44 (0)1753 242763
• Asia-Pacific: +81 3 6868 2300
• Toll-Free Japan: 0120 974 580
• Singapore: +65 6432 8641
• Australia: +61 2 8898 2963
• Call any Ca Technologies office and ask for xtn 46966

To receive automatic notification of any patches or fixes, please subscribe to Knowledge Base article 498, available at http://support.wilytech.com/cgi-bin/wilytech.cfg/php/enduser/std_adp.php?p_faqid=498.

Thank you for making everyone aware of this Nate.

Chris

CA Wily Technical Advisory: CEM stops reporting &
Alerting End User Transaction Incidents after November 7,
2010
CA Wily has uncovered a potential issue where the end user transaction defects do
not get stored in APM database starting from November 7, 2010. This technical
advisory describes the affected versions and platforms in more detail and explains
what remediation actions are recommended.
Problem Description
CA Wily has identified a scenario in the APM product where end user transaction
defects do not get saved to the APM database after November 7, 2010. This scenario
is triggered by the DST change on that day. Thus APM may not provide visibility into
end user affecting problems and may not create and send incidents.
Problem Symptoms
No new defects show up in TESS UI

The defects do not show up because the tables they are stored in do not get created.
The log file has the error messages for not creating tables.
Who is Impacted
 These are the affected APM version:
o 9.0.x
 The applications under management should not be subject to this issue if:
o You are a Introscope only customer
o You are using CEM version 4.x.x
Solution Timing
 This issue is planned to be addressed in next APM service pack targeted for
January 2011.
Workarounds:
Steps: Stop EM, drop ts_defects_20101107,
ts_defect_meta_values_20101107 & ts_tran_comp_details_20101107
and start EM. – Defect related data from 11/07 till the date this
workaround applied is lost. Please contact CA Support for help with
these steps.

Alternative:
Setup a fresh APM database (cemdb) and point EM to that. This will
result in complete loss of data in APM database.

For More Information
If you have any questions or concerns, please contact CA | Wily Technology
Technical Support at support@wilytech.com, or at any of the following numbers:
 Toll-Free U.S.: 1 888 GET WILY ext. 1
 U.S.: +1 630 505 6966
 Latin America +55 11 5503 6167
 Europe: +44 (0)1753 242763
 Asia-Pacific: +81 3 6868 2300
 Toll-Free Japan: 0120 974 580
 Singapore: +65 6432 8641
 Australia: +61 2 8898 2963
 Call any Ca Technologies office and ask for xtn 46966
To receive automatic notification of any patches or fixes, please subscribe to
Knowledge Base at
https://support.ca.com/irj/portal/phpdocs?techDocAccess=N&filePath=7%2F5974%2
Ftechdocs%2FTEC534259.html.

CA Wily Technical Advisory: Native Memory Leak on IBM Java 1.6 when using -javaavent
CA Wily has become aware of a potential issue on IBM Java 1.6 where an application under management may experience severe native memory overhead. This technical advisory describes the affected versions and platforms in more detail, and explains what remediation actions are recommended.

Problem Description
Applications running IBM’s Java 1.6 will experience native severe memory overhead when monitored by the CA Introscope agent using the preferred –javaagent switch. This is due to a native memory leak in the JVM, and applications eventually run out of memory. The memory leak occurs irrespective of the max heap size (-Xmx) setting. The issue also occurs when Autoprobe instrumentation is disabled while still using –javaagent.

The issue is specific to object finalization in the IBM JVM. Details of this issue is available from IBM’s website at http://www-01.ibm.com/support/docview.wss?uid=swg1IZ99243

Who is Impacted
•
This is due to a problem with the underlying JVM, and hence all versions of the Introscope agent are affected.
•
Customers using IBM JDK 1.6 SR9 or below and using the –javaagent switch are impacted.

You are not affected by this issue if:
•
You are using the Introscope agent on platforms other than IBM’s JVM
•
You are using IBM JDK 1.6 SR10 and above
•
You are using the AutoProbe Connector (–Xbootclasspath) agent deployment rather than –javaagent


Solution Timing and Workaround
•
IBM has provided a patch to their JDK which can be obtained at http://www-01.ibm.com/support/docview.wss?uid=swg1IZ99243
•
As a temporary workaround, Introscope customers should use the AutoProbe Connector for IBM and deploy it in the bootstrap classpath using the –Xbootclasspath switch. Note that this is a temporary workaround only and it is recommended that customers obtain IBM’s official patch that resolves this issue.

[size=3]This document is for your informational purposes only. CA assumes no responsibility for the accuracy or completeness of the information. To the extent permitted by applicable law, CA provides this document "as is" without warranty of any kind, including, without limitation, any implied warranties of merchantability, fitness for a particular purpose, or noninfringement. In no event will CA be liable for any loss or damage, direct or indirect, from the use of this document, including, without limitation, lost profits, business interruption, goodwill or lost data, even if CA is expressly advised in advance of the possibility of such damages.[size]

CA Wily Technical Advisory: Agent Memory Leak when using the SQL Agent
CA Wily has become aware of an issue where an application under management may experience severe memory overhead due to a memory leak in the SQL Agent. This technical advisory describes the affected versions and platforms in more detail, and explains what remediation actions are recommended.

Problem Description
Monitored applications using Introscope’s SQL Agent may experience severe memory overhead, sometimes leading to the JVM running out of heap memory. This is due to a memory leak in the SQL Agent, where references to connection counts are appropriately removed, but the backing data structure that stores this information is not maintained appropriately after removal.

Heap dumps will show significant memory occupied by the StatementToConnectionMappingTracer class and all of it being accumulated in multiple instances of a WeakIdentityHashMap data structure. Each of these instances grow to a significant size which contributes to the overhead.

Who is Impacted
This issue affects the SQL Agent all versions of Introscope.
[list]
[*]Introscope Agent 7.x
[*]Introscope Agent 8.0.x
[*]Introscope Agent 8.1.x - 8.2.3
[*]Introscope Agent 9.0.x - 9.0.7
[list]

You are not affected by this issue if:
[list]
[*]You are not using the Introscope SQL Agent
[*]You are using Introscope Agent 8.2.4
[*]Your monitored transactions do not use backend databases heavily. The potential of a problem exists due to the memory leak, but if database calls are monitored sparingly, the symptoms may be rare.
[list]

Solution Timing and Workaround
[list]
[*]This issue is addressed in Introscope 8.2.4, 9.0.8+
[*]As a temporary workaround, Introscope customers can disable the SQL Agent to prevent excessive heap memory usage.
[list]

CA APM Technical Advisory: Memory Leak on Linux and HP-UX Platform Monitors
CA has become aware of an issue that we would like to retroactively highlight for all APM customers. A native memory leak occurs when specific older versions of the Introscope Agent are monitoring a Linux or HP-UX Java Application. This technical advisory describes the affected versions and platforms in detail and explains what action needs to be taken to protect your environment from the occurrence of this issue.

Problem Description
CA APM has identified several versions of the Introscope Agent that were distributed with Linux and HP-UX Platform Monitors that introduce a native memory leak to the managed JVM. This is an extremely slow leak and usually does not cause any noticeable problems under normal usage. In extreme scenarios, when a large number of agents (e.g. 50+) are running on the same machine, the small leak in each agent could add up to have considerable impact on your memory resources.

Who is Impacted
All JVM’s running one of the affect agent versions deployed on Linux and HP-UX are affected. This issue affects the following Introscope Agent versions:
[list]
[*]Introscope Agent 8.0.0.0 - 8.0.2.x
[*]Introscope Agent 8.1.0.0 - 8.1.3.x
[*]Introscope Agent 8.2.0.0 - 8.2.1.x
[*]Introscope Agent 9.0.0.x
[list]
You are not affected by this issue if:
[list]
[*]You are running an Introscope 7.x agent
[*]You are running an Introscope 9.1 agent
[*]You are a CEM only customer
[*]You are using any operating system other than Linux or HP-UX
[list]
Solution Timing and Workaround
This issue is addressed in the following releases of the platform monitors.
Linux Platform Monitor:
[list]
[*]7.2.6.1
[*]8.0.2.11
[*]8.1.3.0
[*]8.2.3.8
[*]8.2.4.0
[*]9.0.5.0
[list]
HP-UX Platform Monitor:
[list]
[*]9.0.8.0
[*]8.2.4.0
[list]
As a temporary workaround, Introscope customers can disable the platform monitors on Linux and HP-UX.

CA APM Technical Advisory: Introscope Enterprise Manager Deadlock Issue
CA has become aware of an issue where an Introscope Enterprise Manager encounters a deadlock. This technical advisory describes the affected versions and platforms in more detail, and explains what remediation actions are recommended.

Problem Description
The Introscope Enterprise Manager occasionally experiences a deadlock. When this occurs, the Enterprise Manager hangs and is inaccessible by users. As a result, users will not be able to connect to or work on the Workstation and the Enterprise Manager must be restarted to remediate the problem. The frequency of occurrence is unsystematic. Some users suffer from this problem daily, while others have only experienced this issue rather infrequently.

Java thread dumps will indicate a deadlock on a LiveMetricSubscriptionManager class. While the Enterprise Manager becomes unresponsive when it encounters this deadlock, performance of the system and the monitored application is not affected as no increased CPU or memory usage is detected when this problem occurs.

Who is Impacted
This issue affects customers with Introscope 9.0.0.0 and above:
[list]
[*]Introscope Enterprise Manager 9.0.0.0 - 9.0.3.0
[*]Introscope Enterprise Manager 9.0.5.0 - 9.0.5.8
[*]Introscope Enterprise Manager 9.0.6.0 - 9.0.6.6
[*]Introscope Enterprise Manager 9.0.7.0 - 9.0.7.1
[list]
You are not affected by this issue if:
[list]
[*]You are using Introscope Enterprise Manager 8.2.4 and below
[*]You are using Introscope Enterprise Manager 9.1.0 and above
[list]
Solution Timing and Workaround
This issue is addressed in Introscope 9.0.6.7, 9.0.7.2, 9.0.8 and 9.1

Are there any latest tidbits or cases on 9.1.0.1 that I should know about? We are currently getting ready for pushing it to prod and already have been evaluating it in non-prod.

Thanks in advance.

Hi All,

I would like to inform you of an important technical advisory concerning CA APM/Introscope 9.1.x. To view the advisory and guidance on the issue, please log in to CA Support Online and access the link below:

[list]
[*]CA APM Technical Advisory: APM/Introscope 9.1.x Agent Transaction Structure Memory Leak
[list]
The technical advisory describes the issue and its affected versions and platforms in more detail; it also provides anticipated dates when the patch will be available and explains the workarounds or remediation actions to use in the meantime to prevent this issue from impacting your application environments.

If you have further questions, please contact CA Support by opening a support ticket or calling 1-800-225-5224.

Regards,
CA APM Product Management

Hi All,

I wanted to provide a brief update to the availability of APM 9.1.1.1 and the fixes to this transaction structure memory leak issue. Please log in to CA Support Online and access the link below to view the updated advisory:
[list]
[*]CA APM Technical Advisory: APM/Introscope 9.1.x Agent Transaction Structure Memory Leak - Updated August 31, 2012
[list]
The technical advisory describes the issue and its affected versions and platforms in more detail; it also provides updated dates for when we expect the patch will be available and explains the workarounds or remediation actions to use in the meantime to prevent this issue from impacting your application environments.
If you have further questions, please contact CA Support by opening a support ticket or calling 1-800-225-5224.

Regards,
CA APM Product Management

Hi all,

Wanted to bring to your attention a product advisory that was just posted to support.ca.com today. A link to the advisory is here. 

Andrew Yeung