Mark.ODonohue

Siteminder Policy Trace Analysis

Discussion created by Mark.ODonohue Employee on Feb 21, 2012
Latest reply on Jul 26, 2018 by RaviSapare

Software last updated 7-Nov-2018 - new version 679 uploaded.

 

The original article has not been updated since its release, so I will add link to any recent (post Jan-2017) SMTraceAnalysisTool articles here  :

 

Analyzing API Gateway trace logs for performance issues (another tool to use) 

(Nov 2018)

Tech Tip : SMPolicyTraceAnalysis - now result in 10% of the time.  

        (Aug 2017)

SMPolicyTraceAnalysis - now able to graph ODBC transactions from odbctrace.out input 

       (June-2017) 

Siteminder Spiral of Death (or why do I have all these socket 32 errors in my policy server logs) 

      (Sep-2014 - but useful) 

Tech Tip - [PreciseTime] gives better Graphs & Stats with SMTraceAnalysisTool (Jan-2017)

TechTip : two new graphs in SMPolicyTraceAnalysis tool explained.  (Jan-2017)

 

 

Siteminder Policy Trace Analysis Tool

Attached is a java Policy Log analysis tool that we have been using in CA Support for a while now for analysis of various SiteMinder logs.



Currently it works on:

  • smps.log "stats" generated by regularly issuing a : "smpolicysrv -stats" command.
  • Policy Server trace logs (smtracedefault.log) files.
  • Webagent logs (including Normal Webagent WebAgentTrace.log files, and those from Secure Proxy Server, and FWS server)


The program is the evolution of various prior perl and python scripts into a java program which then generates graphs and PDF reports of the results.

 

The graphs in the PDF include :

 

Distribution of ProcessRequest times (table plus graph) :

 

 

Calculation of (real) concurrent thread activity :

 

 


And there are many more options included in the pdf report.

 

Attached are both the java program SMPolicyTraceAnalysis_dist.zip, and the documentation SMPolicyTraceAnalysis_docs.zip showing sample graphs and explanation of how to use the program. And at some time in the near future we also hope to release the src to the java code as well,

 

In general for the SMPolicyTraceTool, at least for most basic uses, how to use the program is fairly self evident.

 

This is an internally generated tool, done by CA Support engineers and subject to the limitations of the disclaimer applied to this discussion group for uploads.

 

A tool like this, developed on a part time basis is never complete, certainly there are bugs, limitations, and also many features we would like to add. But the tool has proved useful internally with CA Support, as it currently is and hope you find it useful as well.

 

Cheers - Mark

Note: In respose to a request, here is a simple basic "howto" video, to show a basic install and run:
http://www.youtube.com/watch?v=LEYhw5g-Yog

I am adding a link to a "beta" or experimental version, it has some good and bad additions, and is not really complete. I am adding a beta, since it has already been delayed, and it will be a while until I get back to it again.
10-Oct-2012 Added beta 298, it has some good speed and bug fixes over the beta 292 release. 20th-Oct-2012 Added beta 319, few fixes and some minor features, 31-Oct, added beta 335, bugs mainly (and removed earlier beta's, not allowed too many attachments). 6-Mar-2013 added beta 349, various fixes.
28-Feb-2014 added updated version (still beta) 2.0.0-432 various fixes, also now reads R12.51+ logs.

26-Oct-2015 added updated version (still beta) 2.0.0-472 various fixes, remembers last directory, cmd line mode (alpha-ish), full list in ChangeLog.txt

.

 

Message was edited by: Mark O'Donohue - added latest binary version (462).

 

Message was edited by: Mark O'Donohue Added version 485 - variety of changes and fixes.

 

Message was edited by: Mark O'Donohue Added version 489 - variety of changes and fixes.

 

Message was edited by: Mark O'Donohue Added version 490 - When split trace into threads, also splits into different

                                                                                                     directories based on type of thread.

Message was edited by: Mark O'Donohue Added version 491 - Fixed bug in parsing [Message] field when broken over 

                                                                                                     multiple lines and has some lines which end in ']' 

                                                                                                     To detect eol for multiline Message now checks for  

                                                                                                     current line end in ']' and next line start with '['  

Message was edited by: Mark O'Donohue Added version 495 - Fixed issue with Stats Report, when smps.log use

                                                                                                     localtime, the first couple of entries are still GMT 

                                                                                                     was throwing off the "throughput" calculation at least.

 

                                                                                                     Fixed issue with state transitions in SrcFile and similar

                                                                                                     graphs where last state still in transaction but is not 

                                                                                                     completed by end of the trace, was giving -ve trans time.

 

Message was edited by: Mark O'Donohue Added version 497 - Fixed issue with AvTransOverTime graph - optimization 

                                                                                                     had made it not collect any data for this metric.

Message was edited by: Mark O'Donohue Added version 498 - Update to read new R12.52 Sp1 C5 Stats format from

                                                                                                     smps.log

Message was edited by: Mark O'Donohue Added version 499 - Added larger readahead buffer, from 30,000 to 50,000

                                                                                                     characters, needed to cope with some SAML requests

                                                                                                     when field value reported in base64 and as hex values. 

                                                                                                     then one line is that long - maybe this should be variable

                                                                                                     that can be set dynamically, normal lines are usually 

                                                                                                     less than 1000 chars.

                                                                                                     Also fixed ca directory query log error, when find long

                                                                                                     transactions, print fmt is same as read format.

Message was edited by: Mark O'Donohue Added version 500 - ca directory query log was not reporting individual 

                                                                                                     graphs for search/modify/etc current open requests 

                                                                                                     fixed - was way it inherited values from template graph.

 

Message was edited by: Mark O'Donohue Added version 501 - fix EnqueueAndProcessMessage to display SrcLine

                                                                                                     graph and table - param had wrong name.

                                                                                                     Make EnqueueAndProcessMessage the first section.

 

                                                                                                     Enqueue And Process graphs the transaction from the

                                                                                                     time it is received on the policy server until the time it

                                                                                                     is finished processing - so it is a better measure of

                                                                                                     of av time the policy server took to process the. 

                                                                                                     message.  The queue wait and actual process time

                                                                                                     are still there, just this one is now the first section.  

Message was edited by: Mark O'Donohue Added version 505 - Add ability to process odbctrace.out files

                                                                                                     Add ability to draw graphs for Agent_Con_Manager for 

                                                                                                     selected count, and the reported throughput and

                                                                                                     capacity

Message was edited by: Mark O'Donohue Added version 506 - Add ability to auto recognise odbctrace(.*).out files

                                                                                                     Fix NullPointer bug when SrcFile was not in input file.

Message was edited by: Mark O'Donohue Added version 588 - Speed improvement about 50% faster now.  Various  

                                                                                                     tests run to see what was effective, embedded db 

                                                                                                     still too slow, pool of threads didnt work, did add reader

                                                                                                     and process or thread - but more threading did not help                                                                                                        fixed (bad) was was doing time checks,  and reduced

                                                                                                     number of counters.  

Message was edited by: Mark O'Donohue Added version 621 - Speed improvement again - now uses 64bit and keeps 

                                                                                                     most data in memory - but 3hr analysis can now take

                                                                                                     20min - so good performance gain.

Message was edited by: Mark O'Donohue Added version 670 - Speed improvement added and testing done on each

                                                                                                     of the other 12 trace analysis report types : 

                                                                                                     PolicyTraceAnalysis,

                                                                                                     PolicyStatsAnalysis, AgentTraceAnalysis,

                                                                                                    SPSAgentTraceAnalysis, 

                                                                                                    IMServerLogAnalysis,

                                                                                                    AsaWebSphereAgentTraceAnalysis,

                                                                                                    AsaWebLogicAgentTraceAnalysis,
                                                                                                    CADirQueryLogAnalysis,

                                                                                                    ODBCTraceAnalysis, OracleLdapAccessAnalysis 

                                                                                                    (missed these three will have to revisit it in a day or so: 

                                                                                                    SDKAgentTraceAnalysis,WWSIAgentTraceAnalysis,

                                                                                                   FWSAgentTraceAnalysis)

Message was edited by: Mark O'Donohue Added version 674 - Tested, WWSI, FWS, dont have trace set for SDK 

                                                                                                     Fixed bug where defaulted to Disk, not Memory.

                                                                                                     Fixed bug with EnqueueAndProcessRequest where 

                                                                                                     if dequeue occurred before enqueue in logs then was

                                                                                                     not counted.  Fixed where counters not reset 

                                                                                                     correctly when run in memory and NullPointerException

                                                                                                     when write to disk.

Message was edited by: Mark O'Donohue Added version 676 - Fixed bug with restart of trace logs - with multithread 

                                                                                                     Fixed detection of smservercommand4 thread 

                                                                                                     there are some different pattern matches. 

Message was edited by: Mark O'Donohue Added version 677 - Read of smps.log files from R12.7 fixed  

Message was edited by: Mark O'Donohue Added version 678 - Added support for processing transactions from layer7

                                                                                                     ssg_0_0.log files.

Message was edited by: Mark O'Donohue Added version 679 - minor changes to layer7 ssg log handling.

                                                                                                     handle service and trace start /end better for

                                                                                                     SrcFile trace analysis.  Also stop duplicate "long"

                                                                                                     transactions (they were reported from both

                                                                                                     ProcessRequest and ServiceRequest). 

Outcomes