DX Application Performance Management

Hello,

We have the APM 9.7.16 java agents running on WebSphere 8.5.5 recently, I notice in the agent logfile, numerous error messages of "Unable to redefine class ***, we still have agents running on the WAS 7 and the agents logs have no error messages in them, the 8.5.5 agent is running normally however, not sure on how to rectifiy these INFO/Warn messages.

was 8.5.5 agent log shows numerous:

8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.BootstrapInstrumentationManager] Initializing Bootstrap Dynamic Instrumentation.
8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.BootstrapInstrumentationManager] Bootstrap Manager will wait 240 seconds to instrument post main
bootstrap classes.
8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.Agent] Using delegate class =com.wily.introscope.api.instrument.AClassRedefinitionDelegate
8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.Agent] Try and add transformer using java5 API
8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.BootstrapInstrumentationManager] Executing Bootstrap Classes Dynamic Instrumentation.
8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.Agent] Redefinition of class javax.xml.parsers.SAXParser failed because of Unsupported Operation
Exception in Java Instrumentation API.
8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.Agent] Unable to redefine class javax.xml.parsers.SAXParser at java instrument API level
8/07/18 03:54:40 PM PDT [WARN] [IntroscopeAgent.BootstrapInstrumentationManager] The Re-definition for class javax.xml.parsers.SAXParser failed.8/07/18 03:54:40 PM PDT [WARN] [IntroscopeAgent.BootstrapInstrumentationManager] The Re-definition for class java.nio.channels.Selector failed.

8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.BootstrapInstrumentationManager] Processed All Bootstrap Classes.
8/07/18 03:54:40 PM PDT [INFO] [IntroscopeAgent.BootstrapInstrumentationManager] Initialized pre-main Bootstrap Dynamic Instrumentation successfu
lly.

any feedback is welcome,

thanks,

Pat O'Connor

Hi Pat,

Are the APM 9.7.16 java agents running on WebSphere 8.5.5 configured to use AgentNoRedefNoRetrans.jar file instead of the usual Agent.jar per the advice here?:

WebSphere Application Server - CA Application Performance Management - 9.7 - CA Technologies Documentation 

Specifically

• Metrics are not reported for system classes (java.*), NIO (Sockets and Datagrams), and SSL sockets because the class redefinition and retransformation functions are deactivated.
• Deep inheritance and hierarchy support instrumentation features are disabled.

From what I have found  Deep Inheritance needs to be explicitly disabled when using that jar file to stop the "Unable to redefine class" messages i.e. "introscope.autoprobe.deepinheritance.enabled=false" (There is also a user story to add this to the documentation.)

Can you please check?

Not sure why are you are not seeing the same for WAS 7 unless above is already set or the normal Agent.jar is being used?

Thanks

Lynn

Hi Lynn,

Thanks for the feedback, will review today…

Pat

Regards,

Pat O’Connor

Systems Performance Unit

Department of Child Support Services

(916) 464-7064

Hi Lynn,

Yes, since the WAS upgrade from V7 to V8.5.5, we are now using theAgentNoRedefNoRetrans.jar for the Wily java agent.

Thank you for the informational document, in checking the agent logfile, yes, it lists approximately 33 of them, the errors are only present at agent startup:

8/10/18 09:05:22 AM PDT   Unable to redefine class javax.xml.parsers.SAXParser at java instrument API level

8/10/18 09:05:22 AM PDT   The Re-definition for class javax.xml.parsers.SAXParser failed.

8/10/18 09:05:22 AM PDT   Redefinition of class sun.security.ssl.SSLServerSocketImpl failed because of Unsupported Operation E

xception in Java Instrumentation API.

8/10/18 09:05:22 AM PDT   The Re-definition for class java.nio.channels.Selector failed.

8/10/18 09:05:22 AM PDT   Processed All Bootstrap Classes.

8/10/18 09:05:22 AM PDT   Initialized pre-main Bootstrap Dynamic Instrumentation successfully.

For different “class xx”, I see:

Class sun

Class javax

Class java.net

Class com.ibm

In reading futher this “Deep Inheritance Setting” affects PBD directives, if we turn the Deep Inheritance Setting” Off will we loose what we currently have?

We also use the BRTM.pbl as this is our most critical for us for response time, is this affected if we turn off the Deep Inheritance Setting?

I will review further with my co-workers

Regards,

Pat O’Connor

Systems Performance Unit

Department of Child Support Services

(916) 464-7064

In checking a WAS 7 wily java agent log…looks like these were never used/referenced:

WAS V7 AGENT:

ankcsepf2a06[/cse/introscopeAgent971/wily/logs]$ more IntroscopeAgent.cp2p01aOnlineCell_PFT02_Online_p01a_01.log

8/09/18 02:33:34 PM PDT   Initializing Bootstrap Dynamic Instrumentation.

8/09/18 02:33:34 PM PDT   Bootstrap Manager will wait 240 seconds to instrument post main bootstrap class

es.

8/09/18 02:33:34 PM PDT   I/O Socket Client Hosts (introscope.agent.io.socket.client.hosts) set to ""

8/09/18 02:33:34 PM PDT   I/O Socket Client Ports (introscope.agent.io.socket.client.ports) set to ""

8/09/18 02:33:34 PM PDT   I/O Socket Server Ports (introscope.agent.io.socket.server.ports) set to ""

8/09/18 02:33:34 PM PDT   NIO Datagram Client Hosts (introscope.agent.nio.datagram.client.hosts) set to ""

8/09/18 02:33:34 PM PDT   NIO Socket Client Hosts (introscope.agent.nio.socket.client.hosts) set to ""

8/09/18 02:33:34 PM PDT   NIO Datagram Client Ports (introscope.agent.nio.datagram.client.ports) set to ""

8/09/18 02:33:34 PM PDT   NIO Datagram Server Ports (introscope.agent.nio.datagram.server.ports) set to ""

8/09/18 02:33:34 PM PDT   NIO Socket Client Ports (introscope.agent.nio.socket.client.ports) set to ""

8/09/18 02:33:34 PM PDT   NIO Socket Server Ports (introscope.agent.nio.socket.server.ports) set to ""

8/09/18 02:33:34 PM PDT   Configuration property (com.wily.introscope.agent.harvesting.debugmetrics.enabled) set to default value o

f "false"

Regards,

Pat O’Connor

Systems Performance Unit

Department of Child Support Services

(916) 464-7064

Hi Pat,

Replying to both your above posts:

For your WAS 8.5.5 servers:

As Deep inheritance functionality is disabled when using the recommended AgentNoRedefNoRetrans.jar for an IBM JVM you won't currently be getting any functionality from it anyway and the only impact of the agent profile setting "introscope.autoprobe.deepinheritance.enabled=false" should be to stop the log file messages.

Deep inheritance explanation here:

Java Agent Properties - CA Application Performance Management - 9.7 - CA Technologies Documentation 

From what I understand Deep Inheritance disablement should not impact BRTM functionality.

So for your WAS 7 servers you were using the standard Agent.jar?

Hope that helps.

Thanks

Lynn

Lynn,

Good Morning,

Good Information and thank you.

Yes for the WAS 7 we are using the standard Agent.jar file.

I’ll review with my team this week and appreciate your feedback and knowledge share.

Pat

Regards,

Pat O’Connor

Systems Performance Unit

Department of Child Support Services

(916) 464-7064

Dear Team:

I love seeing community posts where several series of questions are asked and answered. Thanks Lynn_Williams for his detailed and helpful responses! 

Lynn,

Ok, I’ve come up with a game plan to test it out.

  1.  Logon to application and select the various tabs to produce output in the access log.

  2.  Tail -f IHS_access.log ( to confirm brtm messages in it…( selectHealthMonitors?WilyCmd=cmdMetrics&hr=)

  3.  Take screen shots of wily java agent console

  4.  Stop EM

  5.  Set the autoprobe line to false

  6.  Start up EM

  7.  Confirm / verify that the “Unable to redefine class” message do not appear.

  8.  Verify brtm functionality.

Will keep you posted,

Pat

Regards,

Pat O’Connor

Systems Performance Unit

Department of Child Support Services

(916) 464-7064

Hi Pat,

For steps 4 & 6 did you mean stop/start the agent?

Also instead of testing "Set the autoprobe line to false" test the suggested "introscope.autoprobe.deepinheritance.enabled=false" to see if the log messages stop?

Thanks

Lynn

Lynn,

Good Morning,

Your correct, restarted java agent instead of EM.

Ok, steps performed…yet still seeing the error messages at agent startup.

8/14/18 01:59:47 PM PDT   Introscope Agent startup complete.

8/14/18 01:59:47 PM PDT   Initializing Bootstrap Dynamic Instrumentation.

8/14/18 01:59:47 PM PDT   Bootstrap Manager will wait 240 seconds to instrument post main bootstrap cl

asses.

8/14/18 01:59:47 PM PDT   Configuration property (introscope.autoprobe.deepinheritance.enabled) set to "false"

8/14/18 01:59:47 PM PDT   Using delegate class =com.wily.introscope.api.instrument.AClassRedefinitionDelegate

8/14/18 01:59:47 PM PDT   Try and add transformer using java5 API

8/14/18 01:59:47 PM PDT   Executing Bootstrap Classes Dynamic Instrumentation.

8/14/18 01:59:47 PM PDT   Redefinition of class javax.xml.parsers.SAXParser failed because of Unsupported Operation Exception in

Java Instrumentation API.

8/14/18 01:59:47 PM PDT   Unable to redefine class javax.xml.parsers.SAXParser at java instrument API level

8/14/18 01:59:47 PM PDT   The Re-definition for class javax.xml.parsers.SAXParser failed.

8/14/18 01:59:47 PM PDT   Redefinition of class sun.security.ssl.SSLServerSocketImpl failed because of Unsupported Operation Exc

eption in Java Instrumentation API.

8/14/18 01:59:47 PM PDT   Unable to redefine class java.nio.channels.Selector at java instrument API level

8/14/18 01:59:47 PM PDT   The Re-definition for class java.nio.channels.Selector failed.

8/14/18 01:59:47 PM PDT   Processed All Bootstrap Classes.

8/14/18 01:59:47 PM PDT   Initialized pre-main Bootstrap Dynamic Instrumentation successfully.

8/14/18 01:59:47 PM PDT   I/O Socket Client Hosts (introscope.agent.io.socket.client.hosts) set to ""

8/14/18 01:59:47 PM PDT   I/O Socket Client Ports (introscope.agent.io.socket.client.ports) set to ""

8/14/18 01:59:47 PM PDT   I/O Socket Server Ports (introscope.agent.io.socket.server.ports) set to ""

8/14/18 01:59:47 PM PDT   NIO Datagram Client Hosts (introscope.agent.nio.datagram.client.hosts) set to ""

8/14/18 01:59:47 PM PDT   NIO Socket Client Hosts (introscope.agent.nio.socket.client.hosts) set to ""

8/14/18 01:59:47 PM PDT   NIO Datagram Client Ports (introscope.agent.nio.datagram.client.ports) set to ""

8/14/18 01:59:47 PM PDT   NIO Datagram Server Ports (introscope.agent.nio.datagram.server.ports) set to ""

8/14/18 01:59:47 PM PDT   NIO Socket Client Ports (introscope.agent.nio.socket.client.ports) set to ""

8/14/18 01:59:47 PM PDT   NIO Socket Server Ports (introscope.agent.nio.socket.server.ports) set to ""

Regards,

Pat O’Connor

Systems Performance Unit

Department of Child Support Services

(916) 464-7064

The NoRedef profile does also disable bootstrap class instrumentation, a quick comparison of profiles shows the following settings set to false in the NoRedef.profile, the above errors are relating to bootstrap instrumentation.

introscope.bootstrapClassesManager.enabled=false

introscope.agent.remoteagentdynamicinstrumentation.enabled=false

introscope.autoprobe.dynamicinstrument.enabled=false

introscope.autoprobe.deepinheritance.enabled=false

introscope.autoprobe.hierarchysupport.enabled=false

Although it can work in some scenarios, the only supported documented configuration with IBM JDK is AgentNoRedefNoRetrans.jar and the NoRedef.profile

Thanks for the feedback David….will review further…

Just want to get rid of the annoying error messages at our java agent for WAS V8.5.5 startup without affecting its functionality!

Pat

Regards,

Pat O’Connor

Systems Performance Unit

Department of Child Support Services

(916) 464-7064

Thanks DavidLewis for the reminder about the IntroscopeAgent.NoRedef.profile having to be used in conjunction with the AgentNoRedefNoRetrans.jar: WebSphere Application Server - CA Application Performance Management - 9.7 - CA Technologies Documentation