DX Application Performance Management

[size=4]Hello

We are seeing Memory leak in Wily JDBC hashmap reference:

leak is caused by Wily storing a reference to the JDBC objects in their hashmap (com.wily.util.adt.WeakIdentityHashMap). There is also another reference from com.wily.introscope.agent.trace.jdbc.SQLNameFormatterUtilities.

this is impacting Application performance; has anyone seen this issue?

[size]

Yes, we have also seen this, but may have resolved it after upgrading to 9.1.1.0 (I'm still confirming the fix). What agent version are you using? If I recall correctly, version 9.1.1.0 resolves a bug that sounds very much like what you describe, and so far we've not seen the problem since upgrading.

jakbutler wrote:

Yes, we have also seen this, but may have resolved it after upgrading to 9.1.1.0 (I'm still confirming the fix). What agent version are you using? If I recall correctly, version 9.1.1.0 resolves a bug that sounds very much like what you describe, and so far we've not seen the problem since upgrading.

We are running Version 9.0.7.5

I am on 8.2.3.0...how do I go about getting to 8.2.4 to avoid this issue. Any idea?

This is might be likely related to this advisory - CA Wily Technical Advisory: Agent Memory Leak when using the SQL Agent. This is not really a leak at least not a typical one.

The array is growing depending on how many differrent objects are held. The array content s cleared by a normal GC but the problem is that the array it self is not removed. Where a empty array still consumes memory (the bigger the array the more memory is consumed, because there are still all the "pointers" left referencing "no objects"). A colleague described that type of memory behaivour very well, its like a high wather mark, if the array was once big it stays on that size.

aaronritter wrote:

This is might be likely related to this advisory - CA Wily Technical Advisory: Agent Memory Leak when using the SQL Agent. This is not really a leak at least not a typical one.

The array is growing depending on how many differrent objects are held. The array content s cleared by a normal GC but the problem is that the array it self is not removed. Where a empty array still consumes memory (the bigger the array the more memory is consumed, because there are still all the "pointers" left referencing "no objects"). A colleague described that type of memory behaivour very well, its like a high wather mark, if the array was once big it stays on that size.

We Observed this issue after running a Soak test (48 hrs) on an application and after the test; JVM was utilizing 400MB more than before the test start, took the heap dump and found this issue...

will review the advisory... thanks


ashish

hi ashish,

this is very likely the same thing especial the behaivour you described as this "leak / high wather mark" is related to a loaded environment, and the version you are using is affected by it (This issue is addressed in Introscope 8.2.4, 9.0.8+).

@Tim

you are only affected if you use the SQLAgent and the overhead is as far i understood only high if there are many different SQL metrics (high load with lot of different querries). The only way to indentify if the array is using a lot of memory is by e.g. analyzing a heap dump with MAT

Aaron,
Thanks for the reply. We use SQLAgent and we have a ton of sql queries/updates we monitor. Now and then we catch sql issues as well as index issues that require reorgs and such..so it really helps to see what is causing the slow downs. I will get a Heap Dump
and see if we are affected. I was just trying to see if there was an easy way to upgrade to 8.2.4 from 8.2.3..or is it a complete reinstall?

So how do i go about disabling SQL tracing in 9.0.7.5 version; i have seen that option in 9.1.1 agents but not on 9.0.X version.


ashish

aaronritter wrote:

hi ashish,

this is very likely the same thing especial the behaivour you described as this "leak / high wather mark" is related to a loaded environment, and the version you are using is affected by it (This issue is addressed in Introscope 8.2.4, 9.0.8+).

@Tim

you are only affected if you use the SQLAgent and the overhead is as far i understood only high if there are many different SQL metrics (high load with lot of different querries). The only way to indentify if the array is using a lot of memory is by e.g. analyzing a heap dump with MAT

ashish,

what we did is to completely remove the SQL instrumentation is the following:

remove the files: SQLAgent.jar, sqlagent.pbd, sqlagent-summary-metrics-6.1.pbd sqlagent-6.1.pbd

and modify the pbl accordingly where this pbds are loaded.

this will make sure that not even the jar is loaded. It's probably not the nicest way to do this. But we wanted to make sure that nothing is loaded and if you have a jar in the ext folder its usualy loaded and we didnt know what implications that might have.

the best way solving this would be to upgrade to 9.1.1 as it will give you other benefits too but is a different story again :)

Tim,

the way we upgrade our agent is by replacing the whole agent binary set, but thats only because we have no customizations we would need to take care of. Im not sure how a upgrade works when you want to keep old configuration and if its compatible.

Hello Aaron,
i tried the same yesterday and got an error; under required.pbd that SQLAgent tracer hasn't been declared. i couldn't find where this Required.pbd is... any ideas?

ashish

aaronritter wrote:

ashish,

what we did is to completely remove the SQL instrumentation is the following:

remove the files: SQLAgent.jar, sqlagent.pbd, sqlagent-summary-metrics-6.1.pbd sqlagent-6.1.pbd

and modify the pbl accordingly where this pbds are loaded.

this will make sure that not even the jar is loaded. It's probably not the nicest way to do this. But we wanted to make sure that nothing is loaded and if you have a jar in the ext folder its usualy loaded and we didnt know what implications that might have.

the best way solving this would be to upgrade to 9.1.1 as it will give you other benefits too but is a different story again :)

Tim,

the way we upgrade our agent is by replacing the whole agent binary set, but thats only because we have no customizations we would need to take care of. Im not sure how a upgrade works when you want to keep old configuration and if its compatible.

We are running 9.06 and are relying on the SQL agent heavily right now to diagnose a problem. We seem to fit the scenario perfectly, and indeed have memory issues on that agent.

Are you all saying that there is NO PATCH?? And that our only alternative is to upgrade our entire environment to 9.1 to fix this?

@ashish

sorry, the experience is missing for me with veryion 9.1 Agents we just did this for version 8.2.3.5

in 9 there are different configs for it i found in the bookshelf - The SQL Agent files bookshelf - Java Agent Properties > SQL Agent

@Mac-skye

i belive the only "patch" is to remove the SQL Agent or to upgrade to 8.2.4+ / 9.0.8+. at least thats what the advisory says.

yep i was doing the same (as listed in the bookshelf)... thanks for sharing that link... at least i know that i was heading in right direction.


Ashish