TEC616589: Agent appears to cause Out of Memory (OOM) exception when instrumenting Websphere BPM

Document created by ChristopherClark Employee on May 26, 2016Last modified by SamCreek on May 26, 2016
Version 2Show Document
  • View in full screen mode

Document ID:  TEC616589

Last Modified Date:  9/5/2014
Author: ChristopherClark

 

  • Products
    • CA Application Performance Management
  • Components
    • INTROSCOPE

 

 

 

Description:

When monitoring WebSphere BPM, every time the project deploys their snapshot to the Process Server they see an OOM core dump to occur. Analysis of this dump shows thread making request to large amount of heap, 2GB+, which causes an OOM exception. Note as memory is never allocated the object does not show on heap dumps.

Solution:

APM engineering team reviewed and determined that the APM Agent itself is not allocating any memory, it is making a call to the stub (copied stack below) and that internally routes the call to the IBM IIOP protocol, which results in the large allocation request.

 

1XMCURTHDINFO  Current thread NULL           ---------------------- 3XMTHREADINFO      "WebContainer : 0" J9VMThread:0x00000000459CD700, j9thread_t:0x000001002CA522B0, java/lang/Thread:0x000000075E9EEE70, state:R,prio=5 3XMJAVALTHREAD      (java/lang/Thread getId:0x111, isDaemon:true) 3XMTHREADINFO1      (native thread ID:0x4C100FB, native priority:0x5,native policy:UNKNOWN) 3XMHEAPALLOC        Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3      Java callstack: 4XESTACKTRACE       at com/ibm/rmi/IOR.read(IOR.java:362(CompiledCode)) 4XESTACKTRACE       at com/ibm/rmi/iiop/CDRReader.read_Object(CDRReader.java:1167) 4XESTACKTRACE       at com/ibm/rmi/iiop/CDRReader.read_abstract_interface(CDRReader.java:1602) 4XESTACKTRACE       at com/ibm/rmi/iiop/CDRReader.read_abstract_interface(CDRReader.java:1591) 4XESTACKTRACE       at com/lombardisoftware/server/ejb/tracking/_APIServicesHome_Stub.getEJBMetaData(_APIServicesHome_Stub.java) 4XESTACKTRACE       at com/wily/introscope/agent/servlethelper/EJBHelper.getEJBMetaData(EJBHelper.java:28) 4XESTACKTRACE       at com/wily/introscope/agent/trace/ejb/hc2/Ejb2StubNameFormatter.findInterfaceName(Ejb2StubNameFormatter.java:40)

 

The problem is known problem with the IBM JVM, see:-

http://www-304.ibm.com/support/docview.wss?uid=swg1PM46698

To overcome either upgrade the JVM or disable the EJB2StubTracing and EJB3StubTracing in toggles-full.pbd or toggles-typical.pbd. Simply add a # character to the TurnOn directive:-

 

 #TurnOn: EJB2StubTracing #TurnOn: EJB3StubTracing 

 



 

 

Search the Entire CA APM Knowledge Base

 

search-kb.jpg

Attachments

    Outcomes