Clarity

Recently we had a couple of Java OutOfMemory errors crop up which we were unable to diagnose sufficiently because our heapdump was not configured correctly.

We had:

-XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\niku\clarity\logs

instead of -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\niku\clarity\logs

so we have changed that in our Sandbox environment for app and bg, restarted services etc.

In order to test if heapdump logs are now being created in the event of an OOM - we thought we would lower the threshold for max heapdump.

Our current config looks like this:

jvmParameters="-Xms2560m -Xmx2560m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\niku\clarity\logs\

if I changed the -Xmx value to something lower, e.g. 200 then this should definitely generate out of memory errors???

When I changed this, and ran admin general upload-config, restarted services nothing seemed to happen. So I rechecked the properties.xml file and now it has a value of 2048, despite me not configuring this?

Has anyone any idea as to how I could carry out my heapdump test better?

I should add that we have v13.1 sp6 installed.

When I try to alter my -Xmx2560 to -Xmx1024 for example, it's as though it cannot be changed to a value below 2048 and defaults to this value even after I have saved, uploaded config and restarted services.

Does anyone know why this may be the case?

Hi cmcn1982,

What is your exact environment, is it Websphere? If so, we have another way of enabling the heap dumps.
If you changed the -Xmx value to 200, this would not be a valid test. When you are using Clarity, some of the JVM heap is used for Clarity threads, and some for caching. Such a low JVM would most likely generate a heap dump, however this heap dump would not represent a real picture of your JVM use on which you could rely on with your configuration. It will basically throw OOM because of lack of memory for a normal Clarity activity.

What I would suggest to you is
1. Confirm your exact application server as per above so we enable the heap dumps with your normally used JVM -Xms2560m -Xmx2560m
2. Wait until new OOM reoccurrence
3. Provide any dumps to us for analyze

Kind Regards
Nika

Hi Nika,

we use apache tomcat 7.0.26.

Currently the Xms and Xmx values are set to 2560. This is in our Sandbox, Dev, Stage and Production environment.

Basically in Sandbox env, I wanted to deliberately lower the Xmx to maybe 1024 in order to see if I could generate an OOM error -> Heap Dumps logs.

Prior to now, when we got OOM errors in Production no Heap Dump logs were being generated and so it was very difficult for us to pinpoint the root cause.

Hi cmcn1982,


Thanks for replying.
OK I see, I thought you actually might be using WAS because of the command (most of Tomcat customers will just restart the services without running the command as it is not required with a restart).

We have a minimum requirement of 2048m for Xmx as per the installation guide:

Maximum Heap
-Xmx<size>
The maximum heap setting determines the most memory that the local operating system gives to the Java VM. The local operating system does not allocate this much memory immediately on startup, but it can do so as the process runs. The minimum is about 2048m (2 GB), even for small installations.

I have just tested and it looks that Clarity will always respect this minimum requirement.

Do not worry you cannot test/force the generation of the dumps: it is sufficient just enabling the dumps exactly as per our instructions:
https://support.ca.com/irj/portal/kbtech?docid=594208

Please note that if you specify a filename, only one dump will be generated as it doesn't override existing dumps.

Please also make sure you have enough disk space.

Then please restart the services and wait for a reoccurrence.


Kind Regards
Nika

Thanks.

We have sufficient free space on the drive where the heap dump logs will hopefully be created.

Our config now looks like this:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\niku\clarity\logs\heapdumps\app

This syntax is correct?

Just an update on this.

I was able to successfully generate a java out of memory error. I got 6 people from our team to perform a large export to Excel at the same time and monitored GC Heap. Within minutes I saw large spikes and after about 20 minutes the heap dump logs were successfully created. Many thanks for your help and advice Nika.

The heap dump log file was very large as expected, almost 3GB in size. You mention 3rd party software which can be used to analyse. Is there any software in particular that is particularly good for this type of analysis? We are just concerned that if this type of error occurs in Production again out of hours, we would like to be in a position to analyse the root cause right away.

Hello cmcn1982,

I am glad this worked out for you.
What we are usually using at Support to analyze the heap dumps is Eclipse MAT available at:
http://www.eclipse.org/mat/downloads.php

For more information on some of the ways to actually analyze the heap dumps, my colleague Aurora has posted some tips that may be quite helpful:
100205303

https://communities.ca.com/web/ca-clarity-global-user-community/message-board/-/message_boards/view_message/100309098?&#p_19

Please also note that if you have an OOM on your Production, you may want to raise a Support case and upload your heap dump for analyze so we help you out with the exact root cause and solution.

Kind Regards
Nika

Just wondering again ...

It is not a secret that

. 6 people from our team to perform a large export to Excel

is guaranteed to create OOM issues.

The course to take is to address the need for large Export to Excel and to satisfy what is needed so that it does not kill Clarity.

Martti K.

One of the approaches would be below -

98302411

NJ

Yes we normally limit the max number of rows which can be exported to Excel to 21000.

We think our initial OOM error in Production was caused by some inexperienced users possibly attempting a large export. But because we didn't have heap dump logs configured correctly we were unable to definitely prove this.

You seem to know how to deal with it even without the dump. :grin:

Martti K.

:grin:

Educated guessing!

Ok, I have the Eclipse MAT software installed on my laptop.

I have copied the heapdump log file locally (3.15GB in size)

When I try to open the file with MAT I get an error along the lines of: "An internal error occurred during:
"Parsing heap dump from **\java_pid6564.hprof'".Java heap space"

So I then checked the Memory Analyser.ini file and saw that the default -Xmx was 1024m.

I changed this value to 3451m but I am still getting the error.

Does this simply mean that my laptop cannot allocate enough memory in order to open the heapdump file?

Second what Josh just said : I too have this setting set to 6GB on my PC. If you don't have enough memory on your laptop, just try using another, more powerful server. Good luck!

Hi,

After analysis what we can do is that find the user who had run / downloded the portlets.Then "as usual" we need to educate the users not to download huge data via portlet as it was not intended for huge downloads :grin:but use BO reporting etc.Portlets are an user friendly option for viewing / analysing /downloading data but we are not allowing to use the functionality fullest level.Users feels restricted.
Technically may be it might not be an easy job but as martti rightly asked it impacts clarity application performance itself.It might be great if this is handled in future release.Any idea of handling it in 13.x?

cheers,
sundar

I also get the same issue due to which my whole Server get down.

What i have done is MAXIMIZE the Heap Size and Check with the LOAD of portlets then i didnt get that problem again