DX Application Performance Management

Hi All,

We have APM 10.7 in our environment hosted on redhat linux 7.4. 

We are getting high memory % due to increase in threaddumps on the on of the collector.

Found maximum of the thread dumps are in waiting state or in aged waiting state.

Kindly suggest me to minimize the memory % issue

Thanks. .

Regards,

Anuranjan Kumar

Hi Anuranjan,

You mean an increase in the actual number of Collector threads rather than thread dumps - correct?

Is it definitely the Collector Java process that is showing the increased memory usage? 

(ps -ef | grep java | grep Introscope_Enterprise_Manager.lax)

The thread and memory usage increase may be expected if more agents are connecting to the Collector and thus it is doing more work.

The maximum heap you have assigned to the Collector Java process in file Introscope_Enterprise_Manager.lax will not get exceeded.

You can view Collector Supportability metrics for memory usage on the WebView/Workstation UI under the Health and GC Heap nodes of *SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual) (collector_name)|Enterprise Manager

Cluster Supportability Metrics - CA Application Performance Management - 10.7 - CA Technologies Documentation 

Hope that helps.

Thanks,

Lynn

Hi Lynn,

It is Collector Java process that is showing the increase in memory usage.

(ps -ef | grep java | grep Introscope_Enterprise_Manager.lax)

 As of now, we are in implementation phase and currently no agents are reporting to it.

Although we have increased the heap size to 4 GB. And we have  observed  that memory is gradually increasing day by day. 

Might be Thread dump issue as logs are pointing

Thanks for quick response

Regards,

Anuranjan Kumar

Before you finish your tests, I would recommend you run your numbers through the sizing utility to ensure you have enough capacity to handle the workload you're anticipating.

https://sizer.ca.com/APMSizingTool/index.html 

Hi Haruhiko, Lynn,

Thanks for the quick response & guidance!!!

I have increased no. of days to 90 days (default 30 days) and found the memory utilization % to be static now.

introscope.enterprisemanager.threaddump.storage.clean.disk.olderthan.days=90 days

Thanks for your support..

Regards,

Anuranjan Kumar

Hi Anuranjan:
Was Lynn's response helpful? Please let us know when you cam

Thanks

Hal German

Dear Anuranjan:

   This is something that may be less impacted in a later release. There may be only such much that can be done on an issue like this using screenshots . Perhaps Lynn has additional insights

Thanks

Hal German

Hi AnuRanjan,

By default the thread dump purge check for older files (default 30 days old per property introscope.enterprisemanager.threaddump.storage.clean.disk.olderthan.days) will occur once per day per property:

introscope.enterprisemanager.threaddump.storage.clean.disk.freq.days=1

So if the logs are only showing the same frequency as that property value then that is expected.

So is Xmx currently set to 4GB in Introscope_Enterprise_Manager.lax?

The top RES value for the process shows higher but that could be explained by other non heap memory usage. 

How is the heap usage looking from the Supportability metrics?

If there are no agents connected to the Collector have you tried another Collector restart to see if the same symptoms occur?

If the problem still persists it is probably best to open a Support Case so we can review log and config files in more detail.

Are you using 10.7 GA or a later HF or SP?

Thanks

Lynn