The speed with which SMPolicyTraceAnalysis tool runs through trace logs has been improved. The (long) time it takes to process large logs been on my mind for quite a while. A cluster of recent cases with fairly large logs, tipped me into looking at it.
New version 670 : now downloadable from:
Siteminder Policy Trace Analysis
The main speed advantage in the changes comes from taking advantage of the larger memory in the 64bit JVM, and holding all the counters and open transactions in memory. Previously for the 32bit JVM, I had to keep both counters and open transactions in temporary disk files.
There is now a separate thread for reading the trace data and processing the trace data. I tried pool of threads, to process the counters, and parse input lines, but they were slightly slower than just two threads (I suspect there is some bottleneck and more threads could work, but for now the improvement was enough).
I've just used options available in JAVA 7, I did try some of the JAVA 8 pool features, but they did not add any speed, so those are currently commented out.
I did want to store the results in a database, to be able to incrementally add more trace logs, to make it easier to re-generate reports with say different graphs, and to be able to store processed traces from different (say policy ) servers to allow plotting data from multiple machines - but again this was slower - so that is on hold for now.
Here were the improvements I had for my test systems:
Running on Windows:
5.4 gig of logs, takes 3+ hours with older version, now takes less than 20min.
(Windows 7 on Dell Precision M4800 laptop with 32gig of memory and 256 SSD disk, )
Running on Linux,
2 gig of logs takes 44min with older version, and now takes 4min.
(Mint Linux 18. on Dell Precision T7500 with 47 gig of memory and fairly normal 7200 rpm HDD)
The JVM Options:
The run.bat/run.sh now has the following options:
java -XX:+UseLargePages -XX:+UseFastAccessorMethods -XX:+UseParallelGC -jar SMPolicyTraceAnalysis.jar
Occasionally some uses have had trouble with the -XX:+UseLargePages which may require specific permission on win7 machines - removing the options will slow it down a small amount - and you will still get most of the benefits.
Memory Usage:
The above runs show the JVM sitting at using 5gig of memory, the way the JVM works under load however is that it will use memory up to a point where the Garbage Collector is triggered. The exact of memory requiest depends on how long your transactions are, how many are open at once, and how long (ie how long the counters need to count) but the application will generally run quite easily run in 3-4gig of memory.
More JVM Options:
There are more JVM optimisation that could be applied, particularly for java 8, if you want to experiment :
http://docs.oracle.com/javase/7/docs/technotes/tools/windows/java.html
http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html
New version 670 : now downloadable from:
Siteminder Policy Trace Analysis
Cheers - Mark
----
Mark O'Donohue
Snr Principal Support Engineer - Global Customer Success