Shawn_Moore

CA Tuesday Tip: OutOfMemoryError in Clarity

Discussion created by Shawn_Moore Employee on Feb 23, 2011
CA Clarity Tuesday Tip by Shawn Moore, Sr. Principal Support Engineer for 2/22/2011

Hi Folks,

Today's tip is short, but to the point. From time to time we have reports of OutOfMemoryErrors reported on the Clarity App Server. One way a tracking down such errors is to enable a heap dump on reaching an out of memory error. Since this is a fairly advanced topic, consult your application administrator/web server administrator for assistance or implementation of any items below. Such configuration changes should be done on dev/test, as incorrect syntax can result in system outages.

This configuration change can be enabled by using one of the following settings (for the IBM JVM , you may need to adjust these setting as well)

SUN JVM
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\clarity\logs

note: this will not work if using +UseParallelGC (Sun Bug 6452074 makes these 2 settings incompatible )    **This is specific to the 1.4.2 JVM only.

HP JVM
-XX:+HeapDumpOnOutOfMemory -XX:HeapDumpPath=D:\clarity\logs


Once you have enabled heapdumps on OutOfMemoryError conditions, your system will produce a heapdump file in the form of an .phd or .hprof file. Heapdump files are huge but can be easily analyzed with programs such as Eclipse MAT.

The following example shows a common situation where 1.3 GB of the heap was taken by many object instances residing in some jgroup queuing classes. We imported this heap dump into Eclipse MAT.



This above screenshot shows massive usage in jgroups, way beyond anything else in the heap - even more than our object cache. This is a clear indication of a large amounts of multicast traffic resulting in massive heap allocation. (see below for more details)

If you are ever at a loss for what is causing an OutOfMemoryError, please contact CA Technologies Clarity Support. We are here to help you!

Hope you found this one interesting!

-shawn


PS: Here are some important articles regarding known out of memory errors.

TEC490093 - Multicast traffic issue (resolved by IGMP snooping)
TEC525164 - jgroups memory leak

Outcomes