DX Application Performance Management

How can i test it against the system? what command can i use to take the same value that appear on Investigator?

the top command shows more cpu than the value in the investigator, so i dont know how this value is taken

This metric should be CPU Utilization of Agent. you can find the apm r9.1.x metric under *|Agent Stats|Resources|% CPU Utilization (Host) to verify this. They have the same value curve.

I think you can refer to the thread "Agent Stats > Resources", which link is 100433632

- Tiezheng

CPU Utilization Metrics
Question
What is the difference between the Metrics CPU|Utilization
% (process) and CPU|{Processor number}: Utilization %
(aggregate)?
Answer
CPU|Utilization % (process) describes the percentage of
CPU the JVM is using.
CPU|{Processor number}: Utilization %
(aggregate)describes the total utilization of CPU, including
JVM and non-JVM processes. This is what you would see for
the total load on the CPU using the top or ps commands.
If you want to know the overall CPU utilization on a
4-processor machine, including JVM and non-JVM, this data is
available in Introscope by creating a Metric Grouping and a
Calculator for each processor.

[font=Courier New]Top[font] typically tells you the amount of CPU your process is consuming on a single CPU. According to the Understanding platform monitors section of the CA APM Java Agent Implementation Guide, Introscope uses the below logic for the process CPU calculation:

Utilization % (process) - for the Java Agent process, the percentage of total capacity of all processors this process is using. Regardless of how many processors there are, this metric generates only one number.

This means the metric is for the percentage of total system CPU being used by the process across all CPUs. Since Introscope doesn't store decimal values, it's likely that 2% of one CPU will be less than 1% of all CPUs and therefore display as 0% in Introscope.

the problem is that how can i test it against the system?

i mean if i use top with -p parameter for process the %cpu used is more than the one that appears in the tree, so that is why i ask .... what command can i use to test it against the system

Oscar,

Try dividing the value you see for the process in top by the number of CPUs. Also, if you are on Solaris, use prstat instead of top.

- Jack

why dividing / number of cpus? so then what is bad? the top command or the java form of getting this value?

if the top says java is using 2% of cpu i think that in a 2 cpus processor, java is taking 4% of one cpu and 0% of another so the media is 2% no?

if the top says it is using 2% i suppose that the java should say 2% too no? or why then says 0% ?? (that is what is happening in my machines)

we are on linux we can not use prstat, so i need a command to contrastate the value, or as i have been doing i can obviate this, but i dont want to tell that to the client ......

Oscar,

Let me try to make the example a little clearer.

Facts:
Introscope shows CPU % for a process for all CPUs.
Top shows CPU % for a process for a single CPU.

Scenario:
The top command shows the below. We are watching the [font=Courier New]java[font] command for the [font=Courier New]javauser[font] user name, in [color=#ff0000]red[color] (this is faked data, so don't worry about the load averages and such):

load averages:  2.84,  2.47,  2.65;                    up 32+09:57:26  08:54:56
778 processes: 744 sleeping, 32 zombie, 2 on cpu
CPU states: 86.4% idle,  8.9% user,  4.6% kernel,  0.0% iowait,  0.0% swap
Memory: 128G phys mem, 65G free mem, 32G total swap, 32G free swap

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 [color=#ff0000][font=Courier New]22796 javauser 393   0    0 1383M 1304M cpu/1   27.2H 100.00% java[/font][/color]
 14037 db2inst1 535  59    0   11G   10G sleep  345.8H   3.72% db2sysc
  5119 MGR        1  50    0  297M  230M cpu/2    0:42   1.62% TPDRVR
   715 legacy     1  53    0  308M  181M sleep    0:28   0.36% CFIT
 14425 db2inst7 404  59    0 5304M 3888M sleep   57.5H   0.35% db2sysc
   162 root      96  59    0   23M   10M sleep  171:09   0.26% nscd
 16196 db2inst7   1  52    0   96M   45M sleep    0:00   0.22% db2
 16219 db2inst7   1  54    0   97M   46M sleep    0:00   0.22% db2bp
 16191 javauser   1  48    0 4216K 2728K sleep    0:00   0.22% top
 24240 legacy     1  59    0  311M  191M sleep    0:08   0.21% CFIT
 16226 db2inst4   1  52    0   96M   45M sleep    0:00   0.20% db2
 16235 db2inst4   1  54    0   97M   46M sleep    0:00   0.19% db2bp
 28946 expomon    1  59    0 3160K 1856K sleep    0:00   0.00% top
  1401 root       9  58    0  330M  295M sleep  147:21   0.00% cvd
 23636 root       1 100  -20 2564K 1356K sleep   29:06   0.00% xntpd

The prtdiag command shows the below for the CPU details:

==== Processor Sockets ====================================

Version                          Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz CPU 1
Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz CPU 2

So, according to top, the java process we are monitoring is taking up 100% of one of the CPUs (CPU 1), and according to prtdiag, there are only 2 CPUs. So how much CPU of the whole system is the java process taking? The quick formula I mentioned before could be written as [font=Courier New]a / b = x[font], where [font=Courier New]a[font] is the CPU percentage according to top,[font=Courier New]b[font] is the number of CPUs according to prtdiag, and [font=Courier New]x[font] is the percentage of CPU utlization for the entire server. Based on the above scenario, this becomes:
[font=Courier New]100.00 / 2 = 50.00[font]
Conclusion:
Based on what we know about the [font=Courier New]CPU:Utiliation % (process)[font] metric in Introscope (according to the documentation I referenced earlier), we should expect that it should also be displaying 50%.

so if the top says i am using a 2% cpu and my machine has 4 cpus (for example) it is normal that the introscope says i am using 0% cpu no?

and in order to say that i am using 100% cpu the top must say i am using 400% cpu its ok?

i think it is better explained with that :)
so thank you

Mmmmmm now the client says that the value shown in introscope is much more than the shown in top, so i dont know why it is as it says .......

the top says it is 2%

and the graphic in introscope says it is 0-5-0-5-0-5 so i dont know :S

any idea?

To be honest, I wouldn't try very hard to make [font=Courier New]top[font] match up with Introscope (why are you, by the way?). CPU Utilization calculation is a bit tricky and it is, in effect, a calculated average over time. Basically, it is the time a process utilized one or more processors divided by the elapsed time since the last check (and then divided by the number of processors, in the case of Introscope). If the polling periods for top are even slightly different than those for Introscope (and they are; top defaults to 5 seconds and Introscope, I believe, samples every 7.5 seconds and reports the average every 15 seconds), you're very likely to get different values.

Furthermore, on some systems, [font=Courier New]top[font] is not the most accurate method of getting CPU utilization % in real-time. See the below article for some detail:
https://lopsa.org/node/1495

If you are concerned that Introscope is not showing the correct CPU utilization percentage for the process, I would recommend that you open up a CA Support ticket or use EPAgents to gather CPU details rather than relying on the Introscope metrics.

- Jack

i have oppened the support case :)
but you know more than them :)

a client has asked how it can test the value against something in the system because it has many agents with the same configuration but some agents has different cpu consum so it want to test against the system.

If these are Java agents, I would recommend enabling JMX and using jconsole to see what each JVM has to say for itself regarding CPU %. If each JVM is reporting different utilization percentages from one another, then you've shown that the discrepency in Introscope is to be expected. You're going to have a difficult time using a separate utility (top, ps, prstat, etc.) and getting identical values between them and Introscope, so I wouldn't try to validate that the Introscope metric is correct - that CA's job.

In my opinion, even if you are running two JVMs with the exact same code base and the exact same (or shared) agent configuration, it would not be surprising or worrying if the CPU % were different at times between them; it's the server that controls which program or which thread gets CPU priority at a given time, and the logic it uses will vary depending on everything else going on at that exact moment. It would be more surprising if the CPU % went up uniformly across all agents at the same time, especially if they're on different servers.