Here are the KBs for this month. There are eight more
awaiting to be published. I thank my colleagues for taking the time to write
and review these.
APM New KB Articles for August 2014
1. IBM Java 7 agent exception JVMVRFY012 reported
https://comm.support.ca.com/?legacyid=TEC616173
Description: When instrumenting IBM Java 7 based application
with 9.5.5 or 9.6 agent, an exception is reported similar to:
Caused by: java.lang.VerifyError: JVMVRFY012 stack shape
inconsistent;
class=xx/xx/xx/xx/xx/xx/xx/xx/classname
***/******,
method=xx()xx/xx/xx/xx/xx/xx/classname;,pc=5
at java.lang.J9VMInternals.verifyImpl(Native Method)
at java.lang.J9VMInternals.verify(J9VMInternals.java:93)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:170)
at java.lang.Class.forNameImpl(Native Method)
at java.lang.Class.forName(Class.java:181)
2. How does APM CE identify components? Does order of the components matter?
https://comm.support.ca.com/?legacyid=TEC616174
Description: This explains the relationship between the order of components in the APM CE Business Transaction signature and which data
is seen by the TIM. It answers these questions:
Does APM CE need to see the JSPs invoked in the same order as the APM CE Business Transaction signature or does the order not matter ?
When a JSP is called more than once in a single transaction, how does CEM treat this?
Does the order have to be the same always or does it look or the number of times that JSP is called within the transaction?
3. Error message seen during the Postgres database restore step while upgrading to APM 9.6.
https://comm.support.ca.com/?legacyid=TEC616172
Description: NA
4. Installation steps for a APM 9.6 TIM running on the MTP Appliance (CA ADA Multi-Port Monitor)
https://comm.support.ca.com/?legacyid=TEC616111
Description: Although the new 9.6 TIM installer ships now as an RPM package, the 9.6 TIM running on the MTP Appliance still needs to
installed using the MTP Web interface.
5. After upgrading Websphere, four of our twelve agent instances are no longer reporting Frontend or Servlet metrics
https://comm.support.ca.com/?legacyid=TEC615966
Description: After upgrading Websphere from release 7 to 8.5.5, some of the instances on one Server stopped reporting Frontend or Servlet metrics.
The following is the same on both systems:
Agent configuration on each system was not changed during the upgrade.
The Websphere configuration on each system
Each WAS instance serves the same application via a load balancer. It has also been verified that the application is working correctly on all WAS instances
6. A Guide to Solving Common Stats and Defects Aggregation Problems (Update)
https://comm.support.ca.com/?legacyid=TEC610521
Description: Applies to APM 9.1 - 9.6
7. Overriding the default login host for Workstation WebStart
https://comm.support.ca.com/?legacyid=TEC615965
Description: When downloading the Workstation WebStart from the Enterprise Manager, it is auto-populated with a login hostname.
However due to varying network configurations, that hostname may not be valid. is there a way to override the default value to provide a correct hostname?
8. Are synchronizing TIMs versus Introscope Agents
treated differently?
https://comm.support.ca.com/?legacyid=TEC615964
Description: During synchronizing, two different operations are taking place. This document provides more details on each operation.
9. Seeing duplicate Dot Net Agent names in investigator.
https://comm.support.ca.com/?legacyid=TEC615849
Description: Problem: Duplicate Agent names in the investigator appended with %1,%2,%3......
Explanation:
When the Agent restarts it has a new process id.
In Windows, if the Agent about to restart still has a running transaction, then the process will continue analyzing the transaction
and afterwards, the process terminates. However, this does not stop the Windows from creating a new process because of the restart. The result is there are two
processes associated with the same application. In the Investigator, this would be represented as Agent and Agent%1.
On the EM side, Once the original Agent has been terminated completely, then Agent%1 will take the Agent node and Agent%1 node will become
inactive.
On the Agent side, both the agents with the old process Id and the one with the new process Id will have the same name. However, because
the EM already has this name established, the new process Id will have its name appended with "%1". The Agent, recognizes its name without the
"%1" and sends this to the EM to place in the Investigator tree. Thus, the Agent naming feature is active and the Agent name has been properly
retrieved. But users want to stop the EM from creating the Agent with the "%1" appended.
The problem with setting the property "introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds ",
is that it delays the connection until is receives it name. Using the information provided above, we see that the Agent gets its name properly and
relatively quickly. Once the Agent gets its name, it connects to the EM and the EM applies the "%1".
Note: If you try setting this property "introscope.agent.agentAutoRenamingIntervalInMinutes=10 ", this will
not work because the Agent name will not change.
10. How does Deep Inheritance work and what is its Impact?
https://comm.support.ca.com/?legacyid=TEC615845
Description:
For normal instrumentation, the agent only sees one inheritance level on the bytecode level. Let us use the following inheritance
structure as an example:
Class A
|- Class B
|-Class C
|- Class E
Let us assume, that you have a PBD directive which instructs the ProbeBuilder to monitor everything inheriting from "Class A".
On startup, the application sends "Class E" through the Classloader and the ProbeBuilder. At this stage, since the
ProbeBuilder has no idea that "Class E" inherits from "Class A", it would not instrument it.
However, with deep inheritance enabled, when all the other classes are going through the Classloader some time after the startup, it
always checks the complete structure and tries to instrument it using Dynamic Instrumentation. Without Deep Inheritance enabled, this would not happen. This
is because it would not check the full inheritance once a new class comes up. So, in the above example, "Class E" would not be instrumented.
Deep Inheritance Impact
Although the Deep Inheritance feature has played a crucial part in providing a complete monitoring solution, the mechanism of checking the
full inheritance tree and performing Dynamic Instrumentation often comes with a cost.
Due to its described nature, Deep Inheritance causes some overhead, in which on most engines, usually happens at startup. It is
especially significant in frameworks using dynamic proxy classes. For example, frameworks like Spring which are created on the fly, are generally dangerous
for Deep Inheritance. In scenarios like this, the AutoProbe.log can often be used to identify these classes, and a skip of these classes in the PBD could
help addressed the overhead issue.
On the other hand, removing/disabling the Deep Inheritance feature will mean, on most frameworks, having less metrics and more gaps in
transaction traces.
11. CEM - Internal Application Error: JDBC exception on
Hibernate data access
https://comm.support.ca.com/?legacyid=TEC615848
Description: Using the "Group Identification" feature under Business Applications to associate a Group with specific customers, you receive a JDBC error while trying to identify the associated Group using XML.
When creating "User identification" under Business Applications and using XML, it works as expected. When using the same XML for both User ID and Group ID, and trying to "save" the Group ID (XML location), you receive this JDBC error:
Internal Application Error: JDBC exception on Hibernate data access; nested exception is org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update.
12. Which APM releases are supported for MTP 10?
https://comm.support.ca.com/?legacyid=TEC606475
Description: Lists TIM/MTP release compatibility for MTP 10.
13. APM 9.6 Transaction Impact Monitor (TIM) ports and
how to change them post-install.
https://comm.support.ca.com/?legacyid=TEC615847
Description: If needed, APM 9.6 TIM ports for a private httpd web server and the internal communication with that web server can be changed post-install.
14. Frontends missing when monitoring WebSphere 8.5.5
with Introscope 9.1 agent.
https://comm.support.ca.com/?legacyid=TEC615756
Description: What to do if not seeing Frontends reported when monitoring WebSphere 8.5.5 (using IBM JDK 1.6) with an APM 9.1 agent.
You are able to see Servlets and JSPs reported but not Frontends
15. An overview of installation failures on Linux after selecting a new schema for Postgres.
https://comm.support.ca.com/?legacyid=TEC6157596
Description: This document describes a potentially troublesome area in APM installations -- choosing to create a new schema for an
existing Postgres installation. The solution describes the various errors that can be displayed and how to avoid them.
16. EPAgent support for AIX 7.1.
https://comm.support.ca.com/?legacyid=TEC615758
Description: This document clarifies if there is any support for EPagent on AIX 7.1. This is not listed in the Compatibility Guide.
17. A configuration example of the Shell Command Action.
https://comm.support.ca.com/?legacyid=TEC615753
Description: An example of Shell Command Action configuration and use is provided.
18. EM running on a 64-bit Windows 2003 server suddenly becomes
unusable.
https://comm.support.ca.com/?legacyid=TEC615757
Description: PM EM has just been installed as a service on a 64-bit Windows platform. The server is running and does not appear to be heavily utilized but the EM is unusable.
19. Solaris 10-based APM Java agent reports fatal UTF error for "iconv_open()" at start-up
https://comm.support.ca.com/?legacyid=TEC615649
Description:On start up, seeing this this fatal error :
UTF ERROR
["../../../src/solaris/instrument/EncodingSupport_md.c":66]: Failed to complete iconv_open() setup
The environment is a Java Application Server running on Solaris 10 with the APM agent installed
20. Custom tracer only shows Backends on the Application Triage Map when viewed by Frontend but not by Business Service.
https://comm.support.ca.com/?legacyid=TEC615650
Description: The following custom tracer is defined:
SetTracerClassMapping: AppMapCustomBackendTracer com.wily.introscope.appmap.agent.trace.hc2.AppMapMarkerTracer com.wily.introscope.probebuilder.validate.ResourceNameValidator
SetTracerParameter: AppMapCustomBackendTracer nodeType GenericBackend
SetTracerParameter: AppMapCustomBackendTracer nodeLevel MethodClass
SetTracerParameter: AppMapCustomBackendTracer ownerType Application
SetTracerParameter: AppMapCustomBackendTracer boundaryType Backend
SetTracerParameter: AppMapCustomBackendTracer useblame false
SetTracerParameter: AppMapCustomBackendTracer newmode true
SetTracerOrdering: AppMapCustomBackendTracer -1000001
This works correctly when viewing the App Map by Frontend -- the called backends instrumented by this tracer are displayed as expected.
However, when viewing by Business Service those backends are not shown.
21. Running an APM CE (CEM) scheduled report generates database error messages in the Introscope Enterprise Manager log.
https://comm.support.ca.com/?legacyid=TEC615648
Description: After running APM CE daily scheduled reports each morning, there are database errors in the Introscope logs (including key
constraint errors.) The messages in the log look like the following:
2/10/14 06:00:03.238 AM EST [ERROR]
[QuartzScheduler_Worker-8] [org.quartz.core.JobRunShell] Job DEFAULT.jobDetailBean threw an unhandled Exception:
org.springframework.scheduling.quartz.JobMethodInvocationFailedException:
Invocation of method 'pruneData' on target class [class
com.wily.apm.model.pruning.DataPruner] failed; nested exception is org.springframework.dao.DataIntegrityViolationException:
CallableStatementCallback; SQL [{? = call prune_apm_data(?, ?)}]; ERROR: update or delete on table "apm_vertex" violates foreign key constraint "apm_parent_id_fkey" on table "apm_vertex" Detail: Key
(id)=(801) is still referenced from table "apm_vertex"
22. After upgrading to 9.5, we are unable to launch CEM console and NonUniqueResultException errors in MOM log file
https://comm.support.ca.com/?legacyid=TEC615651
Description: After upgrading APM to 9.5.x from 9 with an Oracle APM database, on rare occasions, the following errors appear in the MOM Enterprise Manager log file:
8/12/14 04:47:43.398 PM EEST [ERROR]
[InitializationPool.Thread1] [Manager.com.timestock.tess.util.DbUtils] Select failed for com.timestock.tess.data.objects.
Setting org.hibernate.NonUniqueResultException: query did not return a unique result:2
at org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:813)
at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:804)
at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4533)
at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4486)
at com.timestock.tess.util.DbUtils.getSettingValue(DbUtils.java:8969)
at com.timestock.tess.services.processors.BizEventProcessor.initialize(BizEventProcessor.java:118)
at com.timestock.tess.services.service.ServiceBase.initialize(ServiceBase.java:62)
at com.timestock.tess.services.service.ServicesManager.initialize(ServicesManager.java:123)
at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:87)
at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:1)
at com.timestock.tess.util.ThreadPool$ConcurrentCallableTask.call(ThreadPool.java:218)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
If the environment uses APM CE (CEM) then it may be also not possible to launch the CEM console.
This problem has been reported when upgrading from 9.1.2 to 9.5.2 and from 9.1.2 to 9.5.3. It has not been possible to recreate the problem in CA testing environments
23. How to configure the MOM and Collector when processing VuGen trace files by the APM CE (CEM) Script Recorder.
https://comm.support.ca.com/?legacyid=TEC615512
Description: The APM Transaction Definition Guide provides details on using VuGen Trace files as input to the Script Recorder for generating transaction definitions. However, some undocumented additional steps are needed for the transaction to display in the APM CE UI under the Recording Session tab.
The Guide does not mention using different webserver ports for the MOM and Collector. Failure to do this results in no recording in the
APM CE UI.
24. APM CE(CEM)Analyst Responsibilities
https://comm.support.ca.com/?legacyid=TEC615514
Description: One of the security roles for APM CE reports is that of a CEM Analyst. The security only says the following about this role: has
access to CA CEM reports and views only
The following are suggested APM CE (CEM) Analyst (non-operational, non-administrative) responsibilities. The range and frequency
of these tasks may vary at each site depending on job responsibilities, complexity of environment, type of integration employed, number of users and
support resources, and growth rate of the APM CE-monitored application.
25. What are some of the best practices for APM CE (CEM) Web Server Filters?
https://comm.support.ca.com/?legacyid=TEC615516
Description: This provides a rough framework to follow when implementing web server filters.
26. Wily Technical Advisory: Native Memory Leak on IBM Java 1.6 when using -javaagent (Update)
https://comm.support.ca.com/?legacyid=TEC576360
Description: CA Wily Technical Advisory: Native Memory Leak on IBM Java 1.6 when using -javaagent
27. Information about accessing CEM data (not aggregated) and on how long it is retained after capture.
https://comm.support.ca.com/?legacyid=TEC615518
Description: Three types of raw APM CE (CE) data (Defects, Stats and BtStats) may be accessed via the APM CE (CEM) UI and extracted for reports based on specific data retention rules for each data type.
28. Which version is the WTG Introscope Agent?
https://comm.support.ca.com/?legacyid=TEC615284
Description: Clarifying WTG/Introscope version compatibility.
29. Does SiteMinder Manager and APM TG (WTG) share the same module for Native Data Reporter?
https://comm.support.ca.com/?legacyid=TEC615285
Description: Clarifies Native Data Reporter and the two components that use it -- SiteMinder Memory Manager and WTG.
30. [Japanese] Incorrect value in historical mode
https://support.ca.com/irj/portal/kbtech?docid=TEC000000
-- No Link
Description:
31. Using unsupported characters in a Java class name produce "Invalid Class path" errors in the agent log .
https://comm.support.ca.com/?legacyid=TEC614953
Description: Agent logs are filled with errors as the following:
DATE/TIME [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to Introscope-enable
org/jboss/weldx/transaction/UserTransaction$-11111111$Proxy$_$$_Weld$Proxy$: Invalid class file
DATE/TIME [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to Introscope-enable
org/jboss/weld/security/Principal$-11111111$Proxy$_$$_Weld$Proxy$ Invald class file
32. TIM installation breaks the working TCP Wrapper access control for specific hosts when using SSH and /etc/hosts.allow.
https://comm.support.ca.com/?legacyid=TEC615283
Description: This article discusses a scenario where the TCP Wrapper access control no longer works as configured. This happens usually
these after two installation steps:
The Redhat Enterprise Linux Kickstart image is installed. The TCP Wrapper configuration in /etc/hosts.allow & /etc/hosts.deny works as expected for configured hostnames.
The third-party and tim-complete images are installed. Afterwards, TCP Wrapper access is open to all hosts. This is because the
openSSH version of sshd installed replaced the previous version.