DX Application Performance Management

Hi

I have installed a freshly downloaded version of the tomcat agent , and only changed three settings from default

1) set the name to find my MOM

2) set

introscope.agent.jmx.enable=true

and added :
com.wily.use.platform.mbeanserver=true

 

at the request of the Application team.  When they start their tomcat app, I get the correct log files but get this reported every 15 seconds in the IntroscopeAgent.log:

 

10/16/18 10:40:16 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
10/16/18 10:40:31 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
10/16/18 10:40:46 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
10/16/18 10:41:01 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
10/16/18 10:41:16 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException

 

 

Has anyone else seen this as a google search, and search here did not yield anything with that HttpStatusMsgFetcherTracer method?

 

Thanks

Lee

Dear Lee:

Looking at another case having the same error, placing logs in DEBUG should give you more details

Dear Community:

  Can you help Lee out with next steps?

The error comes from httpbackendmetrics.pbd, this was introduced in 10.x to monitor standard HTTP communications out of applications.

The HTTPStatusMsgFetcherTracer maps to URLConnectionStatusMsgTracer

SetTracerClassMapping: URLConnectionStatusMsgTracer com.wily.introscope.agent.trace.backend.http.tracer.HttpStatusMsgFetcherTracer com.wily.introscope.probebuilder.validate.ResourceNameValidator

When we search on that, we get these two calls.

TraceOneMethodWithParametersIfFlagged: HttpURLConnectionBackend getHeaderField(I)Ljava/lang/String; URLConnectionStatusMsgTracer "THIS STRING IS NOT USED"

TraceOneMethodWithParametersIfFlagged: OC4JHTTPClientBackend getHeaderField(I)Ljava/lang/String; URLConnectionStatusMsgTracer "THIS STRING IS NOT USED"

So you can see it is trying to make a getHeaderField call to get the status value

I would assume the value is used to decorate transaction traces with the exact HTTP status code.

Perhaps the method signature is different in the version you are using, or that getHeaderField method is accessible at all.

If you want to stop the error you can comment either of the lines above that start with "Trace", most probably the first one will be enough, unless you know for sure the application uses OC4J/Weblogic HTTPClient library.

It will fairly inevitably lead to a support case to investigate further, as Hal says, DEBUG logs would be useful.

We would need to know what sort of HTTP library is being used in the application to see if there could be a compatibility issue. For example, when this was first released, we supported Apache HTTP client v4, but not v3 and we saw similar issues in the beginning where v3 was being used.

I also can't comment on the metric specifier saying "THIS STRING IS NOT USED" but I assume it doesn't create a metric and only decorates transaction traces.

I tried commenting out one, and then both of these and still continued to get the message every 15 seconds with an additional stack trace dropped in a single time :

10/16/18 12:41:58 PM EDT [INFO] [IntroscopeAgent.Agent] re-loading configuration resource /opt/caapm/wily/core/config/IntroscopeAgent.profile
10/16/18 12:41:58 PM EDT [INFO] [IntroscopeAgent.Agent] re-loading configuration resource /opt/caapm/wily/extensions/Extensions.profile
10/16/18 12:41:58 PM EDT [ERROR] [IntroscopeAgent.Agent] IntervalHeartbeat.execute threw executing: Configuration Watcher
java.lang.NoClassDefFoundError: Could not initialize class com.wily.introscope.agent.intelligent.detection.StackTraceRecorder$SingletonHolder
        at com.wily.introscope.agent.intelligent.detection.StackTraceRecorder.getInstance(StackTraceRecorder.java:66)
        at com.wily.introscope.agent.intelligent.detection.StackTraceRecorder.getExitPointRecorder(StackTraceRecorder.java:124)
        at com.wily.introscope.agent.intelligent.exitpoint.ExitPointDetectionConfiguration.setExitPointDetectionEnabledStatus(ExitPointDetectionConfiguration.java:364)
        at com.wily.introscope.agent.intelligent.exitpoint.ExitPointDetectionConfiguration$1.set(ExitPointDetectionConfiguration.java:100)
        at com.wily.util.properties.hot.ConfigurationProperty.setValue(ConfigurationProperty.java:635)
        at com.wily.util.properties.hot.ConfigurationProperty.setValueAndLog(ConfigurationProperty.java:593)
        at com.wily.util.properties.hot.ConfigurationManager.setProperties(ConfigurationManager.java:627)
        at com.wily.util.properties.hot.ConfigurationManager.processConfiguration(ConfigurationManager.java:241)
        at com.wily.util.properties.hot.ConfigurationManager.onChange(ConfigurationManager.java:224)
        at com.wily.util.ConfigurationWatcher.processContentChange(ConfigurationWatcher.java:97)
        at com.wily.util.ConfigurationWatcher.ITimestampedRunnable_execute(ConfigurationWatcher.java:129)
        at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)
        at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)
        at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)
        at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)
        at java.lang.Thread.run(Thread.java:748)

I will turn on debug and look at the logs also.

The error here is not necessarily related to the first one

Looking at this error:

java.lang.NoClassDefFoundError: Could not initialize class com.wily.introscope.agent.intelligent.detection.StackTraceRecorder$SingletonHolder

It seems we are missing IntelligentInstrumentation.jar, which should be in the core/ext folder of the agent.

This is part of the smart instrumentation feature that adds interesting uninstrumented classes/methods to transaction traces.

It is unlikely this file is missing but possible, otherwise maybe the user that runs Tomcat does not have sufficient permission over the agent folder - read/write for all subfolders to be sure

This error appears after the Extensions.profile has been (re)loaded but configuration for this feature should not be in an extension, it should be part of the basic Java agent itself.

I would suggest to raise a case on this, and, if possible, to share the complete agent folder you are using, so we could run it against a Tomcat app here and see if we get the same errors.

Also, depending on exactly how you downloaded and got the file onto the Tomcat machine, you may want to try downloading again, making sure you transfer in binary mode if using FTP.