DX Application Performance Management

  • 1.  JMX MBean Server Error in Java Agent Log

    Posted Dec 07, 2016 05:01 PM

    I wanted to throw this out there to see if anyone has seen this issue before.  We have a set of Jboss application servers for which we get no jmx metrics.   I set my log level to DEBUG and captured the debug message below.  There is a runtime exception loading the MBean server, specifically a ClassNotFoundException.

     

    My question:  Since this is being logged within the APM java agent log, is it something we may have misconfigured within the agent profile?  I see no analogous errors in the Jboss server log pertaining to JMX mbeans.

     

    Any insight on how to attack this issue would be appreciated.

     

    Thanks

     

     

     

    #######################################################

     

    Log info:

     

    12/07/16 02:03:36 PM EST [INFO] [IntroscopeAgent.Agent] Activating JMX Data Collection

     

    12/07/16 02:03:41 PM EST [INFO] [IntroscopeAgent.Agent] Using the following JMX Primary Keys:

     

    12/07/16 02:03:41 PM EST [INFO] [IntroscopeAgent.Agent] Using the following JMX Filter Keys: jboss*service=invoker*type=pooled,jboss*service=TransactionManager,jboss.j2ee*plugin=pool,jboss*plugin=cache,jboss.jca*service=ManagedConnectionPool,jboss.jca*service=WorkManagerThreadPool,jboss.mq*service=DestinationManager,jboss.mq*service=MessageCache,jboss.mq*service=PersistenceManager,jboss.mq.destination*service=Queue,jboss.mq*service=Topic,jboss.system*service=ThreadPool,jboss.system*type=ServerInfo,jboss.web*type=Cache,jboss.web*type=Manager,jboss.web*type=GlobalRequestProcessor,jboss.web*type=ThreadPool

     

    12/07/16 02:03:41 PM EST [DEBUG] [IntroscopeAgent.Agent] Unable to handle Connection class by name : org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6

     

    12/07/16 02:03:41 PM EST [DEBUG] [IntroscopeAgent.Agent] Unable to handle Connection class by name : org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6

     

    12/07/16 02:03:41 PM EST [INFO] [IntroscopeAgent.Agent] Lookup and use the platform MBeanServer instead of creating a new one                                    (com.wily.use.platform.mbeanserver) set to default value of "false"

     

    12/07/16 02:03:41 PM EST [ERROR] [IntroscopeAgent.Agent] Failed to activate JMX data collection

     

    12/07/16 02:03:41 PM EST [DEBUG] [IntroscopeAgent.Agent] javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class org.jboss.as.jmx.PluggableMBeanServerBuilder: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder

            at javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:503)

            at javax.management.MBeanServerFactory.getNewMBeanServerBuilder(MBeanServerFactory.java:539)

            at javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:316)

            at javax.management.MBeanServerFactory.createMBeanServer(MBeanServerFactory.java:231)

            at com.wily.introscope.agent.jmx.JMXService.IAgentService_startService(JMXService.java:115)

            at com.wily.introscope.agent.service.ServiceAdministrator$ServiceInitializer.startService(ServiceAdministrator.java:550)

            at com.wily.introscope.agent.service.ServiceAdministrator.registerServiceSync(ServiceAdministrator.java:331)

            at com.wily.introscope.agent.webappsupport.AWebAppSupportReflectionDelegatingService.IAgentService_startService(AWebAppSupportReflectionDelegatingService.java:75)

            at com.wily.introscope.agent.service.ServiceAdministrator$ServiceInitializer.startService(ServiceAdministrator.java:550)

            at com.wily.introscope.agent.service.ServiceAdministrator$ServiceInitializer.startServiceSafe(ServiceAdministrator.java:564)

            at com.wily.introscope.agent.service.ServiceAdministrator$ServiceInitializer.IExecutableItem_execute(ServiceAdministrator.java:428)

            at com.wily.util.task.AsynchExecutionQueue$WrapperItem.IExecutableItem_execute(AsynchExecutionQueue.java:414)

            at com.wily.util.task.AsynchExecutionQueue.doTask(AsynchExecutionQueue.java:205)

            at com.wily.util.task.ATask$CoreTask.run(ATask.java:132)

            at java.lang.Thread.run(Thread.java:745)

    Caused by: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder

            at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

            at com.wily.util.extension.JarExtension$AllPermissionsClassLoader.findClass(JarExtension.java:278)

            at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

            at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

            at javax.management.MBeanServerFactory.loadBuilderClass(MBeanServerFactory.java:446)

            at javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:488)

     

    12/07/16 02:08:51 PM EST [WARN] [IntroscopeAgent.Agent] Heartbeat: Agent, Behavior: Introscope JMX Protocol Adaptor has been running for excessively long time (105834 ms).



  • 2.  Re: JMX MBean Server Error in Java Agent Log

    Broadcom Employee
    Posted Dec 07, 2016 05:08 PM

    Hello,

    Can you please advise the full JBoss version and APM agent version

     

    Thanks

     

    Lynn



  • 3.  Re: JMX MBean Server Error in Java Agent Log

    Broadcom Employee
    Posted Dec 07, 2016 05:20 PM

    In the meantime KB TEC1042397 may help with your problem 

    "JMX in Jboss 7.x / jboss eap 6.x"

    Knowledge Base Articles 

    If you are using 9.6 or later agent you can set the new property

    com.wily.introcope.agent.jmx.syncWithAllMbeanServersForcibly=true



  • 4.  Re: JMX MBean Server Error in Java Agent Log

    Posted Dec 08, 2016 08:25 AM

    Thanks Lynn,

     

    We are on jboss-eap-6.4 and APM java agent 10.2.  We found that tech article while first configuring APM for Jboss and the sync forcibly parameter is part of our baseline agent configuration.  So we definitely have that in place.  The agent configuration in-play produces JMX metrics on some of our Jboss instances, yet produces the error in my post on a subset of others.  Therefore, I am wondering if the error in the agent is somehow a misconfiguration in Jboss itself.  I would assume if the agent is misconfigured, we would not get JMX metrics from any Jboss instances, not just a select few.



  • 5.  Re: JMX MBean Server Error in Java Agent Log

    Broadcom Employee
    Posted Dec 08, 2016 10:56 AM

    HI,

    Do you have this property set on the Agent Profile?

    add com.wily.use.platform.mbeanserver=true

    For more information, please refer CA APM 10.2 Java Agent Implementation Guide:

    Configure JBoss to Use the Java Agent

    Please refer Under: Enable the Reporting of JMX Metrics

     

    Thanks

    Yanna



  • 6.  Re: JMX MBean Server Error in Java Agent Log
    Best Answer

    Posted Dec 08, 2016 09:02 AM

    I found the following parameter in this discussion thread:  Exposing JMX datasource metrics in JBOSS 6.4 

    com.wily.use.platform.mbeanserver=true

     

    Right before the Mbean server error, our logs state the following:
    12/07/16 02:03:41 PM EST [INFO] [IntroscopeAgent.Agent] Lookup and use the platform MBeanServer instead of creating a new one (com.wily.use.platform.mbeanserver) set to default value of "false"

     

    Can you elaborate on what 'com.wily.use.platform.mbeanserver=true' does within an agent profile and although we have not needed it with our other, successful jboss instances, would this be something we should try?

     

    Thanks

     



  • 7.  Re: JMX MBean Server Error in Java Agent Log

    Broadcom Employee
    Posted Dec 08, 2016 09:44 AM

    Hi Darren,

     

    In order to see JMX metrics from JBoss in a JConsole by using the JMX remote management server with a platform specific MBeanServer, you would need to add com.wily.use.platform.mbeanserver=true to the agent profile.  

     

    Needing to use this property will vary depending on the application used in JBoss, which would explain why in your other JBoss instances you did not need to set this property.  Its all in how the application exposes JMX.

     

    Thanks,

    Matt



  • 8.  Re: JMX MBean Server Error in Java Agent Log

    Posted Dec 08, 2016 09:58 AM

    Thanks Matt, I will try this and reply back to the thread.



  • 9.  Re: JMX MBean Server Error in Java Agent Log

    Broadcom Employee
    Posted Dec 11, 2016 11:41 AM

    Darren answered his own question on exposing JMX metrics so marking as answered. Branched off to a new question on not seeing any JMX metrics