DX Application Performance Management

Expand all | Collapse all

Cross Correlation Headers

  • 1.  Cross Correlation Headers

    Posted Oct 11, 2016 10:08 AM

    Can someone fill me in what the actual names of the headers are for cross correlation between JVM's? I see the header decorator in the agent profile talks about the x-wily-info parameter to enable correlation between Introscope and CEM. Is this the same GUID that is being used between JVM's or is there something different? I also know that you can search for a CorGUID parameter in the historical event viewer. Internally, we use multiple types of GUID's for various applications so I'm trying to tease out which ones are APM's. I also see TxnTraceId in traces as well. Is there someplace that details what each of the components are that show up in the transaction trace details?

     

    Thanks!



  • 2.  Re: Cross Correlation Headers

    Broadcom Employee
    Posted Oct 11, 2016 10:56 AM


  • 3.  Re: Cross Correlation Headers

    Broadcom Employee
    Posted Oct 11, 2016 04:06 PM

    Ryan:

       This is not covered in the docs but internal documentation. It looks like we use two fields:

     

    CallerTxnTraceId - The transaction being called. For transaction #1, nothing is called it would be null. For transaction #2, you are calling number 1, so the value would be one. 

     

    TxnTraceId  - Transaction Trace id. So for transaction 1, it is #1, transaction 2 #2. 

     

    Let me know if that helps and matches with what you are seeing.

     

    Thanks 

    Hal German



  • 4.  Re: Cross Correlation Headers

    Posted Oct 11, 2016 04:54 PM
      |   view attached

    Are either of those fields the GUID? Is there a way to know which header is passed to do the cross process correlation?



  • 5.  Re: Cross Correlation Headers

    Broadcom Employee
    Posted Oct 11, 2016 05:27 PM

    Dear Ryan:

        What I can tell you is that there is cross-correlation header that has the GUID. Based on the internal review, I can't get a clear answer if those two fields are the GUID . I have asked internally for someone to respond.

    Failing that, I suggest opening a support case.This is as far as I technically can help you on this. But still doing what I can to get you a response

     

    Thanks

    Hal German



  • 6.  Re: Cross Correlation Headers
    Best Answer

    Posted Oct 11, 2016 05:56 PM

    A user transaction traverses multiple JVM processes/threads. For each JVM process/thread we create a transaction trace wedding cake. These set of wedding cakes all share the same GUID called as "CorProcessData"

    TxnTraceId and CallerTxnTraceId are other GUIDs that help to identify the caller callee relationship between the wedding cake traces. All these guids are sent on the cross process correlation headers  



  • 7.  Re: Cross Correlation Headers

    Broadcom Employee
    Posted Oct 11, 2016 08:16 PM

    In the transaction trace the GUID is called CorProcessData. The http header that is transmitted over the wire is called CorID and included a lot more, e.g. the sequence number.

     

    Example:

    GET /MathProxy/rest/hello/mathcomplex?operation=mean&values=1,2,3,4,5,6 HTTP/1.1

    CorID: B63D31070A82B2D560A6C783CD3209CA,1:1,0,0,,,AgAABClIQgAAAAFGAAAAAQAAABFqYXZhLnV0aWwuSGFzaE1hcAAAAAlIQgAAAAJGAAAAAgAAABBqYXZhLmxhbmcuU3RyaW5nABFBcHBNYXBDYWxsZXJBZ2VudEhCAAAAA0UAAAACAApNYXRoQ2xpZW50SEIAAAAERQAAAAIAEEFwcE1hcENhbGxlclR5cGVIQgAAAAVFAAAAAgAKV2ViU2VydmljZUhCAAAABkUAAAACAA9DYWxsZXJUaW1lc3RhbXBIQgAAAAdFAAAAAgANMTQ3NjIzMTI0NjA4N0hCAAAACEUAAAACABZBcHBNYXBDYWxsZXJNZXRob2ROYW1lSEIAAAAJRQAAAAIAI1dlYlNlcnZpY2UgYXQgaHR0cF8vLzEyNy4wLjAuMV84MDEwSEIAAAAKRQAAAAIAEEFwcE1hcENhbGxlckhvc3RIQgAAAAtFAAAAAgANYXBtLWltbWVyc2lvbkhCAAAADEUAAAACABFVcHN0cmVhbUdVSURDYWNoZUhCAAAADUYAAAADAAAAE2phdmEudXRpbC5BcnJheUxpc3QAAAAKSEIAAAAORQAAAAIAIEI2M0QyN0ZDMEE4MkIyRDU2MEE2Qzc4Mzc2RERDRkQzSEIAAAAPRQAAAAIAIEI2M0QyODhDMEE4MkIyRDU2MEE2Qzc4MzdGRjc0RTk3SEIAAAAQRQAAAAIAIEI2M0QyNzk4MEE4MkIyRDU2MEE2Qzc4MzhCMDVDNDdESEIAAAARRQAAAAIAIEI2M0QyNTkwMEE4MkIyRDU2MEE2Qzc4MzU1Qzg5RkRBSEIAAAASRQAAAAIAIEI2M0QyODBGMEE4MkIyRDU2MEE2Qzc4M0NBNTM3ODRBSEIAAAATRQAAAAIAIEI2M0QyNTk1MEE4MkIyRDU2MEE2Qzc4MzRCRUNBQUM5SEIAAAAURQAAAAIAIEI2M0QyNDJCMEE4MkIyRDU2MEE2Qzc4M0QwMzA3M0JDSEIAAAAVRQAAAAIAIEI2M0QyNzVDMEE4MkIyRDU2MEE2Qzc4MzMyQUU4RTM0SEIAAAAWRQAAAAIAIEI2M0NEREY3MEE4MkIyRDU2MEE2Qzc4MzkyQjQyRUM5SEIAAAAXRQAAAAIAIEI2M0NEQjI3MEE4MkIyRDU2MEE2Qzc4MzcxOTNCMDNGSEIAAAAYRQAAAAIAE0FwcE1hcENhbGxlclByb2Nlc3NIQgAAABlFAAAAAgAGVG9tY2F0SEIAAAAaRQAAAAIAClR4blRyYWNlSWRIQgAAABtFAAAAAgAkQjVDODgyREEwQTgyQjJENTYwQTZDNzgzOTY3QzNENTA0MjI5SEIAAAAcRQAAAAIADkFwcE1hcEFwcE5hbWVzSEIAAAAdRQAAAAMAAAACSEIAAAAeRQAAAAIACk1hdGhDbGllbnRIQAAAAB4=

    Host: 127.0.0.1:8010

    Connection: Keep-Alive

    User-Agent: Apache-HttpClient/4.5 (Java/1.7.0_101)

     

    If I'm not mistaken "B63D31070A82B2D560A6C783CD3209CA" is the CorProcessData, 1;1 the sequence number.

     

    Ciao,

    Guenter



  • 8.  Re: Cross Correlation Headers

    Broadcom Employee
    Posted Oct 11, 2016 05:21 PM

    Chris_Kline CoreyCohen 

       Can you add any more details on this topics? 

     

    Thanks

    Hal German



  • 9.  Re: Cross Correlation Headers

    Posted Oct 11, 2016 05:36 PM
      |   view attached

    So we are trying to get traces that show a transaction traveling across multiple weblogic servers. We have all of the correct configuration enabled in the agent, but still aren’t seeing it. Our initial thought was that an intermediate network appliance was gobbling up the header without passing it on to the next weblogic server, but we tested yesterday by looking at the traffic in between, looking at the headers between all the devices. We determined that the correlation header parameter (what I believe to be x-wily-info, but I’m not sure) aren’t even making it out of weblogic. I’m curious if we know the actual name of the header parameter that we expect so that we can look for it in logs and make sure some weblogic setting isn’t blocking it.

     

    We will be attempting to upgrade the agents to 10.3 from 9.5.1 to see if that makes a difference. I can certainly open a case if that doesn’t work.



  • 10.  Re: Cross Correlation Headers

    Broadcom Employee
    Posted Oct 11, 2016 07:10 PM

    >Can someone fill me in what the actual names of the headers are for cross correlation between JVM's

    Ryan:
    Anand has answered the original question above. So marking as answered. 

     

    >I’m curious if we know the actual name of the header parameter that we expect so that we can look for it in logs >and make sure some weblogic setting isn’t blocking it.

    With the above answer, you should be able to do this.

     

    You are welcome to post follow-up questions if needed

    Thanks

    Hal German

     

     



  • 11.  Re: Cross Correlation Headers

    Posted Oct 12, 2016 04:16 AM

    Hi,

     

    Look at the properties of a Transaction Trace.

     

    All the ones that start with the prefix "Cor" are used to stitch Transaction Traces together. All the other ones are either not related or more cosmetic.

     

    Regs,

     

    Florian.



  • 12.  Re: Cross Correlation Headers

    Posted Oct 13, 2016 10:36 AM

    So I know you are supposed to be able to search in the historical query viewer for certain GUID's using CorGUID:<GUID>

     

    Should I be using the guid Cross Process Data, GUID, Trace ID, or TxnTraceId from the below example? Is there a way to search on any of the other fields if I'm trying to locate specific transaction traces? Or, should there be an actual CorGUID field here? We've been having issues with not seeing the cross process tracing.

     



  • 13.  Re: Cross Correlation Headers

    Posted Oct 14, 2016 02:33 AM

    you will be using Cross Process Data as the guid to search in historical query to find related traces



  • 14.  Re: Cross Correlation Headers

    Broadcom Employee
    Posted Oct 18, 2016 07:20 AM

    TEC1440477 will be published shortly on this topic.