Document ID: TEC616589
Last Modified Date: 9/5/2014
- CA Application Performance Management
When monitoring WebSphere BPM, every time the project deploys their snapshot to the Process Server they see an OOM core dump to occur. Analysis of this dump shows thread making request to large amount of heap, 2GB+, which causes an OOM exception. Note as memory is never allocated the object does not show on heap dumps.
APM engineering team reviewed and determined that the APM Agent itself is not allocating any memory, it is making a call to the stub (copied stack below) and that internally routes the call to the IBM IIOP protocol, which results in the large allocation request.
1XMCURTHDINFO Current thread NULL ---------------------- 3XMTHREADINFO "WebContainer : 0" J9VMThread:0x00000000459CD700, j9thread_t:0x000001002CA522B0, java/lang/Thread:0x000000075E9EEE70, state:R,prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x111, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x4C100FB, native priority:0x5,native policy:UNKNOWN) 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at com/ibm/rmi/IOR.read(IOR.java:362(CompiledCode)) 4XESTACKTRACE at com/ibm/rmi/iiop/CDRReader.read_Object(CDRReader.java:1167) 4XESTACKTRACE at com/ibm/rmi/iiop/CDRReader.read_abstract_interface(CDRReader.java:1602) 4XESTACKTRACE at com/ibm/rmi/iiop/CDRReader.read_abstract_interface(CDRReader.java:1591) 4XESTACKTRACE at com/lombardisoftware/server/ejb/tracking/_APIServicesHome_Stub.getEJBMetaData(_APIServicesHome_Stub.java) 4XESTACKTRACE at com/wily/introscope/agent/servlethelper/EJBHelper.getEJBMetaData(EJBHelper.java:28) 4XESTACKTRACE at com/wily/introscope/agent/trace/ejb/hc2/Ejb2StubNameFormatter.findInterfaceName(Ejb2StubNameFormatter.java:40)
The problem is known problem with the IBM JVM, see:-
To overcome either upgrade the JVM or disable the EJB2StubTracing and EJB3StubTracing in toggles-full.pbd or toggles-typical.pbd. Simply add a # character to the TurnOn directive:-
#TurnOn: EJB2StubTracing #TurnOn: EJB3StubTracing