Document ID: TEC1063869
Last Modified Date: 9/1/2015
Author: Hallett_German
- Products
- CA Application Performance Management
- Releases
- CA Application Performance Management:Release:10.0
- Components
Summary:
How does the CEM-Introscope integration in APM tie to transaction definitions? They seem to be giving different response times.
Instructions:
What follows is a high-level overview of the CEM-Introscope integration that creates transaction traces. Known gaps, many which were undocumented previously, are included. The information is true as of the above date and in all releases up to APM 10.x unless otherwise noted. This is not a replacement to the step by step approach covered in the documentation.
Functionality Overview
The following comes from the Chapter titled "Configuring CA Introscope to Work with CA CEM" in the APM Configuration and Administration Guide. See the Additional Information section below to access the content.CA CEM meets the needs of two distinct users:
- those responsible for web businesses obtain trend, service level, and success rate reports; and
- those responsible for responding to incidents get real-time visibility into transaction defect details. Introscope provides the industry-leading solution for problem isolation at the application server level.When customers integrate CA CEM and CA Introscope, they obtain a complete end-to-end view of their web application in operation. This solution is CA APM.
There are two types of integration for CA APM:
1. Problem resolution triage - allows you to analyze the root cause of problems by using correlated transaction information. This is the integration that creates transaction traces and is the focus of this document.
2. Customer experience metrics - provides regular updates on volume, errors, and average response times for business transactions. This is what RTTM (Real Time Transaction Monitoring does. It does not create transaction traces.)
Problem Resolution Triage (Transaction Trace) Integration
CA CEM starts a transaction trace session with CA Introscope when an incident is generated for a business transaction. CA CEM has access to a special transaction trace method where it traces by business transaction and time threshold. Note: You can also manually start a transaction trace from the CEM > Incident Overview page.For example, let's trace a business transaction called Update Profile. We will trace all web requests that take longer than 1000 ms (that is, 1 second). This allows the agent to optimize performance because once it identifies that a request is not the Update Profile transaction, no further action is required. Although there might be several transaction trace sessions running, overhead is kept to a minimum.Note: Transaction Trace information for an APM CE (CEM) defect displays on the CA CEM Defect Details page only when the Application Server Time threshold is 15 ms or higher. Note that this threshold is configurable and has a minimum value of 1 ms. This is set though the % of Slow Time Metric from the Setup > Introscope Settings page on the APM CE MOM GUI.
1.1 Transaction Trace Overhead
This is from the documentation and prior cases:
Important! Transaction traces can cause performance degradation on the instrumented application. (If the related slow time defect specification is too low, it is possible to bring down the instrumented application.) Note that these settings are global, not per business transaction.A Transaction Trace session affects overhead from the time it starts until all transactions in process at the end of the session complete. You can specify the execution threshold at the millisecond level, but doing so increases the load on the system.
These Transaction Tracer features reduce the likelihood of trace sessions imposing unacceptable overhead:
* Transaction Trace Session Timeout. A Transaction Trace session times out after a user-defined period so that the Admin user cannot accidentally leave the Transaction Tracer on and negatively affect performance for a sustained period. At the end of the timeout period, the agent stops tracing new transactions and completes tracing for transactions in progress.
* Anti-Flooding Logic. To prevent excessive overhead, agent anti-flooding logic limits the number of transactions traced per 15 second interval to 200. After this limit is exceeded, the agent logs that the anti-flood threshold was exceeded, and does not report Transaction Trace data to the Enterprise Manager until that 15-second period has expired. After the 15-second period expires, the anti-flooding logic resumes reporting.
The CA APM Sizing and Performance Guide has more information about controlling Transaction Trace overhead.
Transaction Tracing is a useful capability of the Java and .NET agents. However, large and frequent Transaction Traces require resources, and incur overhead both on agents and on Enterprise Managers. Follow these best practices for best performance:
1. Limit Transaction Tracing to specific problem transactions.
2. Avoid configuring alerts to generate Transaction Traces when the alerts have low thresholds and alerts trigger frequently.
Note the following:
1. The APM Administrator can control how long a transaction trace runs. A transaction trace can be started and stopped manually from an incident. OR setting Maximum Transaction Trace Session Duration through the Setup > Introscope Settings.
2. A transaction trace is not run for all APM CE transactions. Just for some of those that match the APM CE (CEM) transaction definition and exceed the percent of slow time metric.
In conclusion for this topic, the actual load of a transaction trace on an EM, traces database, and agent will depend on a variety of factors including agent tracer, type of application, and type of transaction. The actual percent of load increase will vary differently for each environment.
Comments: So from the above, you can see that CEM is concerned about web server metrics and Introscope is focused on the application server metrics. (They could be JSP, EJB, Backends,etc. These are internal components that get executed/processed in the application server.) Without the integration, CEM would not "know" about application server components nor Introscope about CEM business transaction components. Only GET (query), Cookie, Request Header and some types of POST values are supported for transaction trace integration with APMCE (CEM). See Known Issues for more details.
1.2 Transaction Traces vs Business Transaction Defects
Let us discuss further the relationship of transaction traces to business transaction defects. We have already noted above that how the two approaches measure things differently. CEM Business Transaction Defects will include the average response time to process a business transaction and associated components. This is the average of the total elapsed time of a transaction whether successful or not, from the first request packet to the last response packet for all transactions for a certain timeframe, as monitored by the TIM. The defect is based on the exceeding of the defect threshold and the matching of the various parameter/value pairs in the business transaction definitions.
For an Introscope transaction trace of a CEM Business Transaction, it is a mix of the business transaction parameter/value pairs plus the application components (such as a servlet.) Therefore a transaction trace will not measure CEM components because it is application server-centric. So, the transactions traces will not add up to the same response time/or measure the same components as the business transactions.So how is the mapping then done between APM and CEM?
Java agents with ServletHeaderDecorator installed add extra information to each HTTP response header. The header information includes the GUID, which is used later to correlate CA Introscope transaction traces with APM CE defects. The .NET agents need HTTPHeaderDecorator installed to provide the same functionality.
There are two types of transaction traces:
1) Those based on identifying APM CE Transactions - See the APM Configuration and Administration Guide for more details.
2) Those based on non-identifying APM CE transactions (Available since 9.1.0.1). This is discussed below.
1.3 How to set up Introscope Agent Non-Identifying Transaction Trace Settings
Go to Business Services > Business Transactions > Introscope Agent Non-Identifying Transaction Trace Settings
Introscope Agent Non-Identifying Transaction Trace Settings:
- By default, CA APM runs traces on identifying transactions and generates metrics for those transactions. CA APM runs transaction traces on request-based transactions. Note: Transaction traces are created based on Request not Response-based transaction definition matching.
- You can enable or disable non-identifying transactions as desired.
- CA APM does not generate metrics for non-identifying transactions.
- Only for slow defects on the business transaction, transaction trace links are included in the Component Timing Information table.
To enable monitoring for non-identifying transactions:
1) In CA CEM, open the Administration page.
2) Click the Business Services tab.
3) Select a business service.
4) Click Introscope Agent Non-Identifying Transaction Trace Settings.
5) Select the desired transactions, and click Enabled and Required.
Note: Mark transactions as Required to retain the selection set when you disable all transactions. Else the selection set will be lost. Entries in the Monitoring column indicate the transaction monitoring status. Also the Introscope Settings set in the APM CE MOM GUI also apply to non-identifying transaction traces.
1.4 Known Issues
The following are known issues with the integration. These are documented in the references provided at the end of this document.
1) Transaction Traces only work on slow time defects.
2) Transaction Traces work for non-web service calls. These include only GET (query), Cookie, Request Header and some types of POST values are supported for transaction trace integration with APMCE (CEM).
3) AJAX, Web service calls (when APMCM is matching on XML elements in the SOAP body), HTTP Response Bodies, and XML are not supported for transaction traces. See http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec604888.aspx for details.
4) Non-identifying transaction traces do not show up in the defect. You have to go to the historical viewer and search by CorID.
5) Response-based parameter/value pairs are not used for transaction matching for APM CE transaction traces.
6) Transaction Traces are created based on request- not response-based transaction definition matching.
7) By design, APM does not perform transaction traces on all CEM transactions. This is due to sampling, transactions not exceeding thresholds, defective transactions, network data issue etc.
Additional Information:
There is no one definitive reference on this topic. Resources that were found helpful are:
- APM On-line Documentation
If running 9.5.3 which is the old documentation format. These cannot be accessed online. That can be downloaded from https://support.ca.com/irj/portal/DocumentationSearch
Documents of interest include:
- APM Sizing and Performance Guide
- APM Configuration and Administration Guide
- APM Workstation Guide
- APM Glossary
Since APM 9.6, all APM documentation is online. Little has changed in these areas so should have
- http://wiki.ca.com/display/APMDEVOPS96/Checklist for Configuring CA CEM -- APM 9.6
- https://wiki.ca.com/display/APMDEVOPS98/Checklist for Configuring CA CEM - APM 10.0
The later releases are provided solely because they are references that are online.
- Knowledgebase Articles
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec604888.aspx -- Unable to get Transaction Traces for defects using Webservices Calls. Transaction traces are missing for Slow Time defects.
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec606601.aspx -- Guidelines for setting effective Transaction Trace Threshold as a % of Slow Time
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec595025.aspx -- About Introscope Agent Overhead
Search the Entire CA APM Knowledge Base
