Debugging Java via JMX Console in CA Single Sign On (CA Siteminder)

Document created by Mark.ODonohue Employee on Sep 29, 2014Last modified by kristen.palazzolo on Dec 17, 2016
Version 4Show Document
  • View in full screen mode


The Java JMX Console allows easy diagnosis of JVM problems such as out of memory, and stuck (java) threads.    The following is a quick discussion showing how to enable JMX monitoring of the JVM in the CA Siteminder Policy Server,

 

 

1.1  Enabling JMX monitoring

 

To enable JMX monitoring, you need to pass the following parameters into the JRE runtime.  In Siteminder that is done by editing the JVMOptions.txt file :

 

The options you need to add are :

 

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.port=9004

-Dcom.sun.management.jmxremote.ssl=false

-Dcom.sun.management.jmxremote.authenticate=false

-Djava.rmi.server.hostname=127.0.0.1

 

The above options enable access to JMX for this JVM on host localhost and port 9004, access is enabled from anyone else that is logged onto the same machine.

 

Generally for our purposes that it is enough to secure access - but there are other more advanced access options, including using client X.509 certificates and using UN/ PW.  Which then allow access from remote machines.  Details of all the JMX parameters, including the authentication options can be found here : http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html

 

For the options we have chosen (above) they can then be added anywhere in JVMOptions.txt file :
01-JVMParams.png

 

1.2  Restart the Policy Server


The Policy Server then needs to be restarted, either via the Services or the Policy Management Console, to close any existing JVM and to pickup the new JVMOptions.txt settings.

 

02b- Restart Policy Server.PNG

 

 

1.3 Start the Java JVM

 

Now the tricky part is that Siteminder does not start the Java JVM until there is a request such as a custom Auth or Active Expression that needs the JVM.  Generally that will not be a problem, in a PROD environment as your normal traffic will then start the JVM.

So before we can run the monitor we need to be sure the JVM is running.

 

However in a TEST or DEV DEV environment, you may need to make your own request to start the JVM (I mention this particularly sicne I have caught myself a few times trying to figure out what is wrong, and why is JMX not connection, only to find that the Siteminder JVM hadnt been started).   Below is how I've done it by requesting a page that uses a Java Custom Auth procedure, this then makes sure the JVM is started.

01aa-MakeSureJavaStarted.png

 

1.4  Testing if JMX is enabled

 

You can check if the JVM/ JMX is enabled by checking that the port, 9004 in our case, is being listened on. Via the following :

 

Windows  - netstat -an | find "9004"

Unix         - netstat -an | grep "9004"  

 

That should give a result, when working like the following :

01a-CheckListen.png

Which shows some one is listening on port 9004.

 

 

1.5  Running JConsole

 

Java provides the JConsole program, to run it navigate to the JDK bin directory, and find the jconsole.exe file.   Then double click it to run it.

 

02-RunJConsole-.png

 

1.6  Connecting JConsole to your Siteminder JVM

 

The following connection screen pops up when you start JConsole, it shows under "Local Processes" any java program currently run by the user.   Unfortunately it does not allow connection to "Local Processes" which are running as Services, which is why we needed to use the "Remote Process" connection method.

 

So select the "Remote Process" option, enter the connection entry localhost:9004 and press Connect.

03-ConnectJConsole.png


All going well you should then see the JConsole main screen.

 

 

 

1.6  JConsole Main Screen

 

The JConsole overview screen shows good information about the state of the JVM, including Memory usage, Number of Threads, Number of classes and CPU usage by the JVM.

04-JMX-Overview.png

 

The other tabs in JConsole show detail information.

 

The "Memory" tab shows the status of each of the different pools, as well as initiate a garbage collection run, this allows us to see if the JVM has problems with memory usage, such as slowing down as it exhausts memory.   The "Threads" tab shows the number of threads, it also lists all active threads, and allows the display of the call stack of each thread to be seen by clicking on it - this helps to determine where threads are stuck when there is some conflict for a resource.

05a-Memory.png05b-Threads.png

 

The "VM Summary" shows all of the parameters used to start the JVM and some details of the running OS.  The final tab the "MBeans" tab shows a bit of the under the cover "bean" access methods that are used by JConsole to collect the information.   The MBeans are available via a programmatic interface as well, so it is possible to write a short monitor program that will capture some stats, such as memory values or Jstack style stacktraces.

 

05c-Overview.png05d-Mbeans.png



Needless to say the JVM stats are very helpful when determining the health or otherwise of the JVM running in the Policy Server.


For further information on JConsole & JMX agents the following links will be useful:

http://docs.oracle.com/javase/7/docs/technotes/guides/management/jconsole.html

http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html


I hope you find this article of use.

 

Cheers - Mark

 

PS: I also have steps available for enabling JMX in CA Secure Proxy Server, and I have a JMX client program that uses the JMX interface, to log memory usage and jstack traces to a log file.  This client program can run via a cron/scheduled task to give recorded history of JVM state over time.  Both of these will be posted as community articles in the near future.

4 people found this helpful

Attachments

    Outcomes