AnsweredAssumed Answered

New APM KBs for August (Many are for 9.6)

Question asked by Hallett_German Employee on Aug 29, 2014

Here are the KBs for this month. There are eight more
awaiting to be published. I thank my colleagues for taking the time to write
and review these.

 

 

APM New KB Articles for August 2014

 

 

1. IBM Java 7 agent exception JVMVRFY012 reported

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC616173&docid=616173&bypass=yes&fromscreen=kbresults

 

 

Description: When instrumenting IBM Java 7 based application
with 9.5.5 or 9.6 agent, an exception is reported similar to:

 

Caused by: java.lang.VerifyError: JVMVRFY012 stack shape
inconsistent;

 

class=xx/xx/xx/xx/xx/xx/xx/xx/classname

***/******,

method=xx()xx/xx/xx/xx/xx/xx/classname;,pc=5

   at java.lang.J9VMInternals.verifyImpl(Native Method)

   at java.lang.J9VMInternals.verify(J9VMInternals.java:93)

   at java.lang.J9VMInternals.initialize(J9VMInternals.java:170)

   at java.lang.Class.forNameImpl(Native Method)

   at java.lang.Class.forName(Class.java:181)

 

2. How does APM CE identify components? Does order of the components matter?

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC616174&docid=616174&bypass=yes&fromscreen=kbresults

 

Description: This explains the relationship between the order of components in the APM CE Business Transaction signature and which data
is seen by the TIM. It answers these questions:

 

Does APM CE need to see the JSPs invoked in the same order as the APM CE Business Transaction signature or does the order not matter ?

 

When a JSP is called more than once in a single transaction, how does CEM treat this?

 

Does the order have to be the same always or does it look or the number of times that JSP is called within the transaction?

 

3. Error message seen during the Postgres database restore step while upgrading to APM 9.6.

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC616172&docid=616172&bypass=yes&fromscreen=kbresults

 

Description: NA

 

 

4. Installation steps for a APM 9.6 TIM running on the  MTP Appliance (CA ADA Multi-Port Monitor)

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC616111&docid=616111&bypass=yes&fromscreen=kbresults

 

Description: Although the new 9.6 TIM installer ships now as an RPM package, the 9.6 TIM running on the MTP Appliance still needs to
installed using the MTP Web interface.

 

5. After upgrading Websphere, four of our twelve agent instances are no longer reporting Frontend or Servlet metrics

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615966&docid=615966&bypass=yes&fromscreen=kbresults

 

Description:  After upgrading Websphere from release 7 to 8.5.5, some of the instances on one Server stopped reporting Frontend or Servlet metrics.

 

 

The following is the same on both systems:

 

Agent configuration on each system was not changed during the upgrade.

 

The Websphere configuration on each system

 

Each WAS instance serves the same application via a load balancer. It has also been verified that the application is working correctly on all WAS instances

 

6. A Guide to Solving Common Stats and Defects Aggregation Problems  (Update)

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC610521&docid=610521&bypass=yes&fromscreen=kbresults

 

Description: Applies to APM 9.1 - 9.6

 

 

7. Overriding the default login host for Workstation WebStart

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615965&docid=615965&bypass=yes&fromscreen=kbresults

 

Description: When downloading the Workstation WebStart from the Enterprise Manager, it is auto-populated with a login hostname.

 

However due to varying network configurations, that hostname may not be valid. is there a way to override the default value to provide a correct hostname?

 

 

8. Are synchronizing TIMs versus Introscope Agents
treated differently?

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615964&docid=615964&bypass=yes&fromscreen=kbresults

 

Description: During synchronizing, two different operations are taking place. This document provides more details on each operation.

 

 

 

9. Seeing duplicate Dot Net Agent names in investigator.

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615849&docid=615849&bypass=yes&fromscreen=kbresults

 

 

Description: Problem: Duplicate Agent names in the investigator appended with %1,%2,%3......

 

 

Explanation:

 

When the Agent restarts it has a new process id.

 

In Windows, if the Agent about to restart still has a running transaction, then the process will continue analyzing the transaction
and afterwards, the process terminates. However, this does not stop the Windows from creating a new process because of the restart. The result is there are two
processes associated with the same application. In the Investigator, this would be represented as Agent and Agent%1.

 

On the EM side, Once the original Agent has been terminated completely, then Agent%1 will take the Agent node and Agent%1 node will become
inactive.

 

 

On the Agent side, both the agents with the old process Id and the one with the new process Id will have the same name. However, because
the EM already has this name established, the new process Id will have its name appended with "%1". The Agent, recognizes its name without the
"%1" and sends this to the EM to place in the Investigator tree. Thus, the Agent naming feature is active and the Agent name has been properly
retrieved. But users want to stop the EM from creating the Agent with the "%1" appended.

 

 

The problem with setting the property "introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds ",
is that it delays the connection until is receives it name. Using the information provided above, we see that the Agent gets its name properly and
relatively quickly. Once the Agent gets its name, it connects to the EM and the EM applies the "%1".

 

 

Note: If you try setting this property "introscope.agent.agentAutoRenamingIntervalInMinutes=10 ", this will
not work because the Agent name will not change.

 

10. How does Deep Inheritance work and what is its Impact?

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615845&docid=615845&bypass=yes&fromscreen=kbresults

 

Description:

For normal instrumentation, the agent only sees one inheritance level on the bytecode level. Let us use the following inheritance
structure as an example:

 

Class A 

|- Class B

|-Class C

|- Class E

 

Let us assume, that you have a PBD directive which instructs the ProbeBuilder to monitor everything inheriting from "Class A".

 

 

On startup, the application sends "Class E" through the Classloader and the ProbeBuilder. At this stage, since the
ProbeBuilder has no idea that "Class E" inherits from "Class A", it would not instrument it.

 

However, with deep inheritance enabled, when all the other classes are going through the Classloader some time after the startup, it
always checks the complete structure and tries to instrument it using Dynamic Instrumentation. Without Deep Inheritance enabled, this would not happen. This
is because it would not check the full inheritance once a new class comes up. So, in the above example, "Class E" would not be instrumented.

 

Deep Inheritance Impact

 

Although the Deep Inheritance feature has played a crucial part in providing a complete monitoring solution, the mechanism of checking the
full inheritance tree and performing Dynamic Instrumentation often comes with a cost.

 

 

Due to its described nature, Deep Inheritance causes some overhead, in which on most engines, usually happens at startup. It is
especially significant in frameworks using dynamic proxy classes. For example,  frameworks like Spring which are created on the fly, are generally dangerous
for Deep Inheritance. In scenarios like this, the AutoProbe.log can often be used to identify these classes, and a skip of these classes in the PBD could
help addressed the overhead issue.

 

 

On the other hand, removing/disabling the Deep Inheritance feature will mean, on most frameworks, having less metrics and more gaps in
transaction traces.

 

 

11. CEM - Internal Application Error: JDBC exception on
Hibernate data access

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615848&docid=615848&bypass=yes&fromscreen=kbresults

 

Description: Using the "Group Identification" feature under Business Applications to associate a Group with specific customers, you receive a JDBC error while trying to identify the associated Group using XML.

 

 

When creating "User identification" under Business Applications and using XML, it works as expected. When using the same XML for both User ID and Group ID, and trying to "save" the Group ID (XML location), you receive this JDBC error:

 

 

Internal Application Error: JDBC exception on Hibernate data access; nested exception is org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update.

 

12. Which APM releases are supported for MTP 10?

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC606475&docid=606475&bypass=yes&fromscreen=kbresults

 

Description: Lists TIM/MTP release compatibility for MTP 10.

 


13. APM 9.6 Transaction Impact Monitor (TIM) ports and
how to change them post-install.

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615847&docid=615847&bypass=yes&fromscreen=kbresults

Description: If needed, APM 9.6 TIM ports for a private  httpd web server and the internal communication with that web server can be  changed post-install.


14. Frontends missing when monitoring WebSphere 8.5.5
with Introscope 9.1 agent.

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615756&docid=615756&bypass=yes&fromscreen=kbresults

 

 

Description: What to do if not seeing Frontends reported when monitoring WebSphere 8.5.5 (using IBM JDK 1.6) with an APM 9.1 agent.

 

You are able to see Servlets and JSPs reported but not Frontends

 

15. An overview of installation failures on Linux after selecting a new schema for Postgres.

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC6157596&docid=615759&bypass=yes&fromscreen=kbresults

 

 

Description: This document describes a potentially troublesome area in APM installations -- choosing to create a new schema for an
existing Postgres installation. The solution describes the various errors that can be displayed and how to avoid them.

 

 

16. EPAgent support for AIX 7.1.

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615758&docid=615758&bypass=yes&fromscreen=kbresults

 

Description: This document clarifies if there is any support for EPagent on AIX 7.1. This is not listed in the Compatibility Guide.

 

 

17. A configuration example of the Shell Command Action.

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615753&docid=615753&bypass=yes&fromscreen=kbresults

 

 

Description: An example of Shell Command Action configuration and use is provided.

 

 

18. EM running on a 64-bit Windows 2003 server suddenly becomes
unusable.

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615757&docid=615757&bypass=yes&fromscreen=kbresults

 

 

Description: PM EM has just been installed as a service on a 64-bit Windows platform. The server is running and does not appear to be heavily utilized but the EM is unusable.

 

 

19. Solaris 10-based APM Java agent reports fatal UTF error for "iconv_open()" at start-up

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615649&docid=615649&bypass=yes&fromscreen=kbresults

 

 

Description:On start up, seeing this this fatal error :

 

 

UTF ERROR
["../../../src/solaris/instrument/EncodingSupport_md.c":66]: Failed to complete iconv_open() setup

 

 

The environment is a Java Application Server running on Solaris 10 with the APM agent installed

 

20. Custom tracer only shows Backends on the Application Triage Map when viewed by Frontend but not by Business Service.

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615650&docid=615650&bypass=yes&fromscreen=kbresults

 

 

Description: The following custom tracer is defined:

SetTracerClassMapping: AppMapCustomBackendTracer com.wily.introscope.appmap.agent.trace.hc2.AppMapMarkerTracer com.wily.introscope.probebuilder.validate.ResourceNameValidator

 

SetTracerParameter: AppMapCustomBackendTracer nodeType  GenericBackend

 

 

SetTracerParameter: AppMapCustomBackendTracer nodeLevel MethodClass

 

 

SetTracerParameter: AppMapCustomBackendTracer ownerType Application

 

 

SetTracerParameter: AppMapCustomBackendTracer boundaryType Backend

 

 

SetTracerParameter: AppMapCustomBackendTracer useblame false

 

 

SetTracerParameter: AppMapCustomBackendTracer newmode true

 

 

SetTracerOrdering: AppMapCustomBackendTracer -1000001

 

 

This works correctly when viewing the App Map by Frontend -- the called backends instrumented by this tracer are displayed as expected.

 

 

However, when viewing by Business Service those backends are not shown.

 

 

 

21. Running an APM CE (CEM) scheduled report generates database error messages in the Introscope Enterprise Manager log.

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615648&docid=615648&bypass=yes&fromscreen=kbresults

 

 

Description: After running APM CE daily scheduled reports each morning, there are database errors in the Introscope logs (including key
constraint errors.) The messages in the log look like the following:

 

 

2/10/14 06:00:03.238 AM EST [ERROR]
[QuartzScheduler_Worker-8] [org.quartz.core.JobRunShell] Job DEFAULT.jobDetailBean threw an unhandled Exception:
org.springframework.scheduling.quartz.JobMethodInvocationFailedException:
Invocation of method 'pruneData' on target class [class
com.wily.apm.model.pruning.DataPruner] failed; nested exception is org.springframework.dao.DataIntegrityViolationException:
CallableStatementCallback; SQL [{? = call prune_apm_data(?, ?)}]; ERROR: update or delete on table "apm_vertex" violates foreign key constraint "apm_parent_id_fkey" on table "apm_vertex" Detail: Key
(id)=(801) is still referenced from table "apm_vertex"

 

 

22. After upgrading to 9.5, we are unable to launch CEM   console and NonUniqueResultException errors in MOM log file

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615651&docid=615651&bypass=yes&fromscreen=kbresults

 

 

Description:  After upgrading APM to 9.5.x from 9 with an Oracle APM database, on rare occasions, the following errors appear in the MOM Enterprise Manager log file:

 

 

8/12/14 04:47:43.398 PM EEST [ERROR]
[InitializationPool.Thread1] [Manager.com.timestock.tess.util.DbUtils] Select failed for com.timestock.tess.data.objects.

Setting  org.hibernate.NonUniqueResultException: query did not return  a unique result:2

 

             
at  org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:813)

 

             
at  org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:804)

 

             
at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4533)

 

             
at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4486)

 

             
at com.timestock.tess.util.DbUtils.getSettingValue(DbUtils.java:8969)

 

 

             
at com.timestock.tess.services.processors.BizEventProcessor.initialize(BizEventProcessor.java:118)

 

              

at com.timestock.tess.services.service.ServiceBase.initialize(ServiceBase.java:62)

 

             
at com.timestock.tess.services.service.ServicesManager.initialize(ServicesManager.java:123)

 

             
at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:87)

 

             
at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:1)

 

             
at  com.timestock.tess.util.ThreadPool$ConcurrentCallableTask.call(ThreadPool.java:218)

 

             
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

 

             
at java.util.concurrent.FutureTask.run(Unknown Source)

 

             
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

 

             
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

 

             
at java.lang.Thread.run(Unknown Source)

 

 

 

 

If the environment uses APM CE (CEM) then it may be also not  possible to launch the CEM console.

 

This problem has been reported when upgrading from 9.1.2 to 9.5.2 and from 9.1.2 to 9.5.3. It has not been possible to recreate the problem in CA testing environments

 

 

 

23. How to configure the MOM and Collector when processing VuGen trace files by the APM CE (CEM) Script Recorder.

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615512&docid=615512&bypass=yes&fromscreen=kbresults

 

 

Description: The APM Transaction Definition Guide provides details on using VuGen Trace files as input to the Script Recorder for generating transaction definitions. However, some undocumented additional steps are needed for the transaction to display in the APM CE UI under the Recording Session tab.

 

 

The Guide does not mention using different webserver ports for the MOM and Collector. Failure to do this results in no recording in the
APM CE UI.

 

 

24. APM CE(CEM)Analyst Responsibilities

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615514&docid=615514&bypass=yes&fromscreen=kbresults

 

Description: One of the security roles for APM CE reports is that of a CEM Analyst. The security only says the following about this role: has
access to CA CEM reports and views only

 

 

The following are suggested APM CE (CEM) Analyst (non-operational, non-administrative) responsibilities. The range and frequency
of these tasks may vary at each site depending on job responsibilities, complexity of environment, type of integration employed, number of users and
support resources, and growth rate of the APM CE-monitored application.

 

 

25. What are some of the best practices for APM CE (CEM) Web Server Filters?

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615516&docid=615516&bypass=yes&fromscreen=kbresults

 

 

Description: This provides a rough framework to follow when implementing web server filters.

 

 

26. Wily Technical Advisory: Native Memory Leak on IBM Java 1.6 when using -javaagent (Update)

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC576360&docid=576360&bypass=yes&fromscreen=kbresults

 

 

Description: CA Wily Technical Advisory: Native Memory Leak on IBM Java 1.6 when using -javaagent

 

 

27. Information about accessing CEM data (not aggregated) and on how long it is retained after capture.

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615518&docid=615518&bypass=yes&fromscreen=kbresults

 

 

Description: Three types of raw APM CE (CE) data (Defects, Stats and BtStats) may be accessed via the APM CE (CEM) UI and extracted for reports based on specific data retention rules for each data type.

 

 

28. Which version is the WTG Introscope Agent?

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615284&docid=615284&bypass=yes&fromscreen=kbresults

 

 

Description: Clarifying WTG/Introscope version compatibility.

 

 

29. Does SiteMinder Manager and APM TG (WTG) share the same module for Native Data Reporter?

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615285&docid=615285&bypass=yes&fromscreen=kbresults

 

 

Description:  Clarifies Native Data Reporter and the two components that use it -- SiteMinder Memory Manager and WTG.

 

 

30. [Japanese] Incorrect value in historical mode

 

 

https://support.ca.com/irj/portal/kbtech?docid=TEC000000
-- No Link

 

 

Description:

 

 

31. Using unsupported characters in a Java class name produce "Invalid Class path" errors in the agent log .

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC614953&docid=614953&bypass=yes&fromscreen=kbresults

 

 

Description:  Agent logs are filled with errors as the following:

 

 

DATE/TIME [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to Introscope-enable

 

 

org/jboss/weldx/transaction/UserTransaction$-11111111$Proxy$_$$_Weld$Proxy$:  Invalid class file

 

 

DATE/TIME [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to Introscope-enable

 

 

org/jboss/weld/security/Principal$-11111111$Proxy$_$$_Weld$Proxy$  Invald class file

 

 

32. TIM installation breaks the working TCP Wrapper access control for specific hosts when using SSH and  /etc/hosts.allow.

 

 

https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC615283&docid=615283&bypass=yes&fromscreen=kbresults

 

 

Description: This article discusses a scenario where the TCP Wrapper access control no longer works as configured. This happens usually
these after two installation steps:

 

 

The Redhat Enterprise Linux Kickstart image is installed. The TCP Wrapper configuration in /etc/hosts.allow & /etc/hosts.deny works as expected for configured hostnames.

 

 

The third-party and tim-complete images are installed. Afterwards, TCP Wrapper access is open to all hosts. This is because the
openSSH version of sshd installed replaced the previous version.

Outcomes