Helping to debug SSO Policy Server java processes - redirecting stdout/stderr to timestamped file.

Document created by Mark.ODonohue Employee on Jun 5, 2017Last modified by Mark.ODonohue Employee on Oct 13, 2017
Version 5Show Document
  • View in full screen mode

Why do we want to see java stdout: 

Debugging java federation and custom components running in the JVM in the smpolicysrv policy server can be a painful business.    The policy server runs as a service, and redirects and writes to stdout/stderr to null device, effectively throwing them away.

    

The main use for stdout is to see the Stack trace written by the JVM when a process fails (for example OutOfMemoryException, or failed to find class file, or class file is not an acceptable compiled version).  Also a lot of java library components written by third parties and even in java itself, write trace information to stdout (for example the java –verbose, or –Djavax.net.debug=all). So easy access to stdout/stderr often assist with general debugging and monitoring custom components even when there is not a specific problem.

 

Now JVM stdout only gets written when the JVM is started, it is not enough the start the policy server, you also have to call some java component.   The below trace logs were triggered using the javaauthapi-delay.zip example which is also attached. 

 

The method here adds a  "premain" java class to setup java to redirect stdio to a timestamped file.  For completeness, and as an alternative method, we also show how to run the policy server from the cmd line. 

 

 

Solution 1 - Deploy open source RedirectStdio.jar library:  

Attached is a java utility that allows redirecting of the java stdout / stderror of the JVM - that may help capture the stacktrace of the java errors & Exceptions .   Redirection of java stdout/stderr is fairly simple it is done by calling the following java functions:  System.setOut(..); System.setErr(..);  There is also a  "-javaagent" directive that can direct a JVM to run a "premain" function prior to starting the routine.   That is useful as we do not have to change any of the deployed custom auth/az/assertiongenerator code to get access to the java stdout. 

   

Fortunately it is fairly easy to install :

 

a) Deploy the java-redirectstdio.zip

Unzip the attached java-redirectstdio.zip file.   The deployment includes stc code, you may need to rebuild from src, depending on the java version your using (I suspect the build one will work for you, since it is build with java .1.6). But compile is simple, edit build.bat file to point to JDK on policy server, and run the build.bat

 

 

b1-a) Change JVMOptions.txt to add :

There are two methods for setting the parameters to RedirectStdio.jar, the first uses a properties file, which is the more convenient, and allows dynamically enable/disable of the redirect via enable= setting in the .properties file.  The second puts all the parameters on the entry line in the JVMOptions.txt and requires a restart of the policy server for the new parameters to take effect. 

 

This section b1-a) and b1-b) describes the .properties method, section b2) describes the older cmd line option method. 

 

For the SSO Policy server install, we need to change the startup JVM runtime options. These are stored in the JVMOptions.txt file.   We need to add one line, with -javaagent , which contains both the .jar file containing the premain() class file, and then a parameters pointing to the default properties file: 

-javaagent:C:/work/java-redirectstdio/RedirectStdio.jar=C:/work/java-redirectstdio/dist/redirectstdio.properties

Note: If you have spaced in the distribution path, you will need to add %20 as the URL encoded space.   And you may want to change the parameters, for the location of the RedirectStdio.jar file and path to where it writes the properties files are : :  Here is screenshot of my JVMOptions.txt file: 

 

 

 

b1-b) Change redirectstdio.properties file:

The redirectstdio.properties is where parameters to the redirectstdio can be set, the most important ones are "enable=true" which enabled the redirect of java stdout/stderr and prefix=  which determines where the logs are written. 

 

The properties file is checked every 30sec, and if there are any changes it is re-read.  The main use for this is to dynamically be able to turn the logging on & off (via the enable= setting).

 

 

 

b2) Change JVMOptions.txt to add :

This section describes how to put the parameters on the JVMOption.txt cmd line, you probably want to use the previosu method, and skip this setup. 

 

For the SSO Policy server install, we need to change the startup JVM runtime options. These are stored in the JVMOptions.txt file.   We need to add one line, with -javaagent , which contains both the .jar file containing the premain() class file, and then also a list of parameters as per : 

-javaagent:C:/work/java-redirectstdio/RedirectStdio.jar=prefix=C:\\Program%20Files%20(x86)\\CA\\siteminder\\log\\redirect_,rolloverSize=10000,numLogsToRetain=20

Note the %2 0 is used where space is required in the file name.  And you may want to change the parameters, for the logation of the RedirectStdio.jar file and path to where it writes the log files are to be generated. :  Here is screenshot of my JVMOptions.txt file: 

 

c) Restart the smpolicysrv

As soon as some java activity is started log files should appear in the C:\\Program%20Files%20(x86)\\CA\\siteminder\\log\\ directory.

 

And the logs should contain all the stdout log data: 

 

 

 

Solution 2 - Run smpolicysrv from the cmd line: 

One workaround is to run the policy server from the command line.   You can stop the service on windows, or run the stop-all command on linux : 

Then run the smpolicysrv directly from the cmd line: as per: 

As you can see anything written to java stdout (in this case it is from our custom delay auth scheme) is printed to the DOS cmd: 

 

But often that is not convenient, since it: will interrupt current users; can have additional complications when running a service from the command line (such as win2008); often for managed environments a change requests or special approval process is needed, before this can be done, which further delays resolving the problem; and finally a restart may actually clear the exception condition that you are trying to isolate.

 

Clearly it is best if you to run it in your QA environment on one machine before running it on production.  And often for production deployment, if all the PS machines are acting the same, then installing on only one should be sufficient to give some idea of what is being written to the java stdout/stderror log. 

 

(Added -2.zip version, which has a bit more debugging and catch Exception for the bg thread monitoring for changes in the .properties file)

2 people found this helpful

Outcomes