DX Application Performance Management

We have this requirement, not sure if anyone of you have worked on something similar before.

Bit of background:

We have an application which uses SOAP over http webservices calls (CXFv3)

APM v10 gives a very neat drill down functionality of the trace view, and the SOA dependency map is pretty cool as well.

But wanted to stretch my luck and find out if the below is possible:

Application A (residing on a linux VM-A) makes a webservice call which goes off to Application B (residing on linux VM-B).

So when I look at the trace for that WS call, I am able to see the amount of time spent at each function within the first leg (within VM-A) of the WS journey.

But then, to look at the second leg of the same WS call, I have to then open the trace metrics of VM-B. 

This is sometimes a bit tedious.

Now my question:

Was wondering if there is a way (through existing functionality within APM v10 OR additional customisation on any APM version) to look at the full journey of a single WS transaction that spans accross multiple JVMs in a single trace view ? Or is that too much to ask ?

Hi Venkit,

Yes cross process tracing feature is designed to do just that and from your description it should be working out of the box, assuming both JVM's A and B have been instrumented by Java Agent and SPM extension to monitor webservices is deployed on both ends. Are you not seeing both traces show up in the Trace view in different swimlanes when selecting JVM A trace?

If yes but you are able to see Webservices metrics(Client\Server) on each JVM individually then it would require further investigation via a support issue.

Regards,

Kulbir.

hi Kulbir,

I am not able to see the trace view in swimlanes for each of the JVMs trace in a seperate lane.

But indivually , I am able to see them and dig down deep - which is good, but just not good enough.

thanks for your response.

Things to check in case correlation is not working:

a) On VM-A, look for "good looking" Backend metric which comes close to your Web Service

b) On VM-B, look for a good looking Frontend metric which comes close to your Web Service or alternative (Pre-APM 10) look for a Web Services|Server metric which looks good.

c) Analyze a transaction Trace on VM-A. You are able to see on the trace your good looking Backend

d) Analyze a Transaction Trace on VM-B. You are able to see on the trace your good looking Frontend

e) You click on the root component of Trace from the VM-B and you can see a property "Caller Txn Trace Id" (maybe similar name) and the property Cross Process Sequence Number (or similar) shows a number higher than 1. For example 1:1 or 1:1:2

f) You still don't see both traces correlated

Here the error analysis:

a) fails: Your WS protocol is not covered by Introscope or SPM / Introscope extension is not installed correctly

b) fails: Your WS protocol is not covered by Introscope or SPM / Introscope extension is not installed correctly

c) fails: Your application does the transaction across multiple Threads and this would become a special case we would have to look into where's the problem (Tibco, Oracle OSB, WAS Process Server, JBoss Fuse, Mule ESB... - we can fix those)

d) fails: Identical problem to c)

e) fails: Something is in between VM-A and VM-B which modifies the HTTP headers / rewrites the message.

f) fails: You are most probably running into a TT Clamp session. Check Transaction Trace clamp configurations and agent log messages.

Hey,

can you write that up in a blog? Maybe with some example screenshots.

Ciao,

Guenter

Hey,

I put it on my list ;-) ...

Cheers

Andreas

hi Andreas,

Did take a bit of time to analyse, but its now working. Thanks for your response.

Perfect Great. Congratulations ;-)

I think Venkit’s problem is the following one, let me know if it is not.

JVM A calls JVM B calls JVM C

CorrelationID1 is used to correlate between JVM A and JVM B

CorrelationID2 is used to correlate between JVM B and JVM C

When he selects a TT from Agent A, he can only see the correlated TT from Agent B, not the one from Agent C

When he selects a TT from Agent C, he can only see the correlated TT from Agent B, not the one from Agent A

Only if he selects the TT from Agent B can he actually see both caller JVM A’s TT and callee JVM C’s TT.

Venkit, is that accurate?

That would have been "c) fails:..."

hi Florian,

You have just highlighted something that could come back to haunt us in the future - currently we dont have that scenario in our application yet. :-)

The issue was that the response to some of the WS request was being cached locally for some reason, and wasnt going off to the destination server which was serving it. As Andreas mentioned, when we started looking out for the correlation ID's on transaction we realised that. And were later able to trigger a proper WS call , and traced that end to end.

There was help from CA support as well in helping us get to the bottom of this.

thanks