DX Application Performance Management

Hi,

I know there are a lot of docs about it, but still it isn't clear enough for me. Well I've found this in my investigator

tooltip

introscope.enterprisemanager.agent.metrics.limit is active

appears in alert

but in the clamp section I have these configuration

<clamps>
<description>
APM Clamps and Important Events configuration file. Please refer apm-evets-thresholds-config.xsd for schema file.
This configuration configures the pre-defined the clamps and important events. Each clamp/important event
threshold value will be applied for each agent or entire EM depends configuration.
</description>
<clamp id="introscope.enterprisemanager.agent.metrics.limit">
<description>
Most properties here put limits on # of metrics.
The last limits metric data. The metric clamping properties support hot config.
Per Agent limit. Takes into account live and historical metrics.
</description>
<threshold value="55000"/>
</clamp>
<clamp id="introscope.enterprisemanager.transactionevents.storage.max.disk.usage">
<description>
The maximum desired disk usage in MB for the trace storage. If this maximum is exceeded,
then the daily historical trace stores will be deleted starting with the oldest first
until the total historical trace storage size is below this value.
The current days trace store actively storing traces will not be deleted even if
its size exceeds this property value. The size of the trace index is not considered
when determining what historical trace stores to delete.
</description>
<threshold value="1024"/>
</clamp>
<clamp id="introscope.enterprisemanager.metrics.live.limit">
<description>
Per EM limit. Takes into account only live metrics (i.e. currently reporting from Agents)
</description>
<threshold value="500000"/>
</clamp>
<clamp id="introscope.enterprisemanager.metrics.historical.limit">
<description>
Per EM limit. Takes into account metrics with Smartstor data (i.e. live and historical metrics)
</description>
<threshold value="30000000"/>
</clamp>
<clamp id="introscope.enterprisemanager.agent.connection.limit">
<description>
Limits the # of agent connections
</description>
<threshold value="400"/>
</clamp>
<clamp id="introscope.enterprisemanager.disconnected.historical.agent.limit">
<description>
Limits the # of dead/disconnected agents that are mounted state to minimize
the runtime memory of EM.
</description>
<threshold value="400"/>
</clamp>
<clamp id="introscope.enterprisemanager.events.limit">
<description>
Limits # of Events Processed by EM Per Interval
</description>
<threshold value="1250"/>
</clamp>
<clamp id="introscope.enterprisemanager.agent.trace.limit">
<description>
Limits # of Transaction Events Processed Per Interval(Per Agent)
</description>
<threshold value="1000"/>
</clamp>
<clamp id="introscope.enterprisemanager.agent.error.limit">
<description>
Limits # of Error Events Per Interval
</description>
<threshold value="10"/>
</clamp>
<clamp id="introscope.clw.max.users">
<description>
Limits the number of clw users
</description>
<threshold value="500"/>
</clamp>
<clamp id="introscope.workstation.max.users">
<description>
Limits the number of workstation users
</description>
<threshold value="40"/>
</clamp>
<clamp id="introscope.enterprisemanager.collector.cdv.max">
<description>
Limits the number of CDV connections
</description>
<threshold value="5"/>
</clamp>
<clamp id="introscope.enterprisemanager.transaction.discovery.max.nonidentifying.components">
<description>
Transaction Discovery Processor (autogen) Maximum Number of Non-Identifying Components
</description>
<threshold value="50"/>
</clamp>
<clamp id="introscope.enterprisemanager.max.number.domain.configuration.changes">
<description>
Maximum number of domain configuration changes returned by query. 0 is unlimited.
</description>
<threshold value="0"/>
</clamp>
<clamp id="introscope.enterprisemanager.max.transaction.user.groups">
<description>Maximum number of transaction user groups.</description>
<threshold value="5000"/>
</clamp>
<clamp id="introscope.enterprisemanager.max.application.user.rows">
<description>Maximum number of application users retrieved.</description>
<threshold value="1000"/>
</clamp>
</clamps>

but agent metrics are about 13k and the limit is setting in

introscope.enterprisemanager.agent.metrics.limit = 55000

Am I wrong?

I'm not sure if the limit is per agent or is the total for all agents, could you explain it please?

And  the historic metrics

introscope.enterprisemanager.metrics.historical.limit=30000000

So, what metric is out of limit? 

Perhaps 

introscope.enterprisemanager.agent.metrics.limit = 55000 includes Historical Metrics and as historic metric count is over 100k......

Thanks in advance!

osvaldog,

you don't have you worry about anything right now

introscope.enterprisemanager.agent.metrics.limit = 55000 is an agent metrics limit clamp value and right now you have 13K agent metrics. if you have agent historical metrics and live metrics combined with 55+K then the EM will not allowed new metrics for that particular agent.

The 100K historical metrics you are mention is I think from EM historical metrics. you are still very low because 1.2 Million is ok for individual EM. I saw this value up to 2.4 million also.  

Ok, I share your point of view, but metric is clamped = 1, so I don't know why the agent is clamped

Hi Hiko,

Console means workstation or CLW? 

Sorry I haven't seen you message that the clamp is active.

I have some question? most probably agent is reporting for a long time haven't prune from long time.

How long this agent is running?

What version of EM and Agents running in your environment?

You have prune your EMs very 6 months or whenever EM historical metrics are hitting 1.2 million. pruning is a lengthy process the shortcut to clean some historical metrics to fix this clamp go to /data/archive and delete some old .data files.

You need to restart your EM after deleting these files to reflect changes in your EMs.

Osvaldo,

You are currently clamped on the agent side, that is you have more than 55,000 live + historical metrics from the agent.  As Hiko suggested, check your apm status console.

Most likely the agent is overrunning with too many traces.  Check your metric count pie chart to see where the influx of metrics is coming from.

Matt

As Hiko suggest me I've used CLW to investigate clamps agent and I got the agent clamps and its metrics total.

Thanks folk!!

Hi Hiko,

I've changed this propertie in apm-evets-thresholds-config.xml

<clamp id="introscope.enterprisemanager.agent.metrics.limit">
<description>
Most properties here put limits on # of metrics.
The last limits metric data. The metric clamping properties support hot config.
Per Agent limit. Takes into account live and historical metrics.
</description>
<threshold value="55000"/>

<threshold value="60000"/>
</clamp>

but Agent still clamped, I've made this query to EM with CLW

"list active clamps of EM" 

And I've got this response

xxxxxxxx@5001, introscope.enterprisemanager.agent.metrics.limit, Most properties here put limits on # of metrics.
The last limits metric data. The metric clamping properties support hot config.
Per Agent limit. Takes into account live and historical metrics., 55000, 55014, 8:40:45 11/26/17
xxxxxxxx@5001, introscope.enterprisemanager.agent.metrics.limit, Most properties here put limits on # of metrics.
The last limits metric data. The metric clamping properties support hot config.
Per Agent limit. Takes into account live and historical metrics., 55000, 55009, 6:12:30 11/26/17

Do I need restart EM? 

Hi All,
I don't know if you know but this properties needs be adjusted on Collectors too.
Verify if this values are adjusted in collector that this agent are conected and in my opinion, if an agent enters in state "clamped" you need restart this agent to the collector verify again the state.

I hope helped.

Vinícius Campelo

I am experiencing a similar issue and would like to understand the approaches to troubleshooting. I see within the APM console the property however it does not provide any details to what is causing the threshold to be exceeded and where to look next. The only thing I would think to do is to increase the "introscope.enterprisemanager.agent.metrics.limit" in the threshold.xml however I am not sure if that is the best approach.

Is there something on the agent side that I should also be looking into as well?

Thanks!

Sam

Hello Osvaldo,

I was involved in a customer with this behavior.. Agents clamped but reporting less than limit.

My approach to solve it was to perform a SmartStor clean up to get rid of  old (historical) metrics that were adding to the clamp and did not allowed the collector to receive them.

Give it a try to the following doc:

How to clean up the SmartStor database