Symantec SiteMinder

Introduction 

The purpose of this blog entry is to show all the different types of logs that are available in CA SSO (aka Siteminder) Policy Server.  Tracing information would come in next blog. I will also be referring to some useful utilities commonly used with CA SSO for troubleshooting. 

CA-SSO Policy Server Logging Procedure: 

 The following gives an overview of the major components of Policy Sotre and also shows the name of (all) the logs that can be enabled and where they get their data from:

  Note: I may not be discussing the use all the utilities, but the diagram indicates where they would be used.

Log Files

Depending on the problem you are experiencing, Support may request one or more of the following log files:

Policy Server Logs 

• smps.log: The Policy Server log file records information about the status of the Policy Server and, optionally, configurable levels of auditing information about authentication, authorization, and other events in the Policy Server log file. When the Policy Server is started, its version information and configurations are recorded in the Policy Server log.

• "smpolicysrv -stats" command will show the output on smps.log.

Cron job or Windows Scheduler can be configured to run "smpolicysrv -stats" command to get the policy server statistics.

** Explanation of the fields:

- Msgs = Number of thread pool messages handled
- Waits = # of times Dequeue had to wait for a message before timeout reached
- Misses = # of times waiting thread woke up to find no message and timeout reached
- Max HP Msg = # maximum number of High Priority messages on the queue since the last reset - of stats
- Max NP Msg = # maximum number of Normal Priority messages on the queue since the last reset of stats
- Current Depth = # message in the queue at the time of executing –stats
- Max Depth = # maximum number of messages on the queue since the last reset of stats
- Current High Depth = # High priority messages in the queue at the time of executing –stats
- Current Norm Depth = # Normal priority messages in the queue at the time of executing -stats
- Current Threads = # threads running at the time of executing -stats
- Max Threads = maximum thread number reached

Connections:

- Current = # of agent connections
- Max = maximum # of connections since last reset
- Limit = maximum allowable connections
- Exceeded limit = # of times exceeded the limit

"Busy threads" refers to the number of thread is currently active on stack and processing request.
"Wait" and "missed" are historical record, does not mean much unless an incident is happening, could be an indicator of how much threads are utilized.
"Reset" can either be policy server restarted or admin intentionally flushed the statistics by command line options

• smtracedefault.log: Policy Server Trace log with configuring the Policy Server Profiler. Check the "Enable Profiling" option, then click on the "Configure Settings" button.

      * Config File – config file saved under C:\Program Files\CA\siteminder\config\smtracedefault

** More detail of “Configure the Policy Server Profiler” can be found from document.
https://docops.ca.com/ca-single-sign-on/12-6-01/en/configuring/configure-the-policy-server-profiler

- Configuration Settings: Useful configuration for troubleshooting

-   /Component

Functional group of components that will be logged.

Note: Often all except “Server/Policy_Object_Cache” generates lots of log lines – so often good to leave out.

-   / Data

Items:   <data item: PreciseTime, SrcFile, Function, Pid, Tid, Message  to give better results.>

-   / Filter

      (Expression can be added to exclude items – not widly used but occasionaly helpful)

Siteminder Policy Trace Analysis

https://communities.ca.com/message/97562726

“Siteminder Policy Trace Analysis” is a java Policy Log analysis tool that we have been using in CA Support for a while now for analysis of various SiteMinder logs.

 - Additional tips are available from author's blog post:
Tech Tip - [PreciseTime] gives better Graphs & Stats with SMTraceAnalysisTool 

Other Loggings

• Xtrace – xTrace is an XPSConfig option that captures XPS errors in Policy Store. This option is available from CA SiteMinder Release 12.51.

 Type the number to enable
Need to do “U” to write the entries.

The config file Saves to :

C:\Program Files (x86)\CA\siteminder\config\XPS.cfg

** More detail of xTrace can be found from document.

How to Use xTrace - CA Single Sign-On - 12.8 - CA Technologies Documentation 

• Smaccess.log – audit log

Useful Tools for Troubleshooting

• Wireshark: Wireshark is a free and open source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol.

Usage: Decode SSL, LDAP protocol.
https://www.wireshark.org/ 

• Netstat: netstat (network statistics) is a command-line network utility tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interface (network interface controller or software-defined network interface) and network protocol statistics. 

• Top: Top command displays processor activity of your Linux box and displays tasks managed by kernel in real-time. It'll show processor and memory are being used and other information like running processes.

• Fiddler: Fiddler is an HTTP debugging proxy server application.
  
  - To enable HTTPS traffic decryption:
  Open Fiddler-> Tools-> Fiddler Options-> HTTPS -> Chec ‘Decrypt HTTS traffic’->
  http://www.telerik.com/fiddler

• Debug Diagnostic tools: The Debug Diagnostic Tool (DebugDiag) is designed to assist in troubleshooting issues such as hangs, slow performance, memory leaks or memory fragmentation, and crashes in any user-mode process. 

      https://www.microsoft.com/en-us/download/details.aspx?id=49924

• Process monitor: Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity. 
  
  https://technet.microsoft.com/en-us/sysinternals/processmonitor.aspx

• Strace: strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state. 

        -   Command: strace -p <pid>

• Pstack: pstack attaches to the active processes named by the pids on the command line, and prints out an execution stack trace, including a hint at what the function arguments are.If the process is part of a thread group, then pstack will print out a stack trace for each of the threads in the group.
  If symbols exist in the binary (usually the case unless you have run strip(1)), then symbolic addresses are printed as well.

• Core dumps: Core dump, memory dump, or system dump consists of the recorded state of the working memory of a computer program at a specific time, generally when the program has crashed or otherwise terminated abnormally. (Please refer to ‘Debug Diagnostic tools’ above in case Core Dump file was not generated properly.)
• Pkgapp: Pkg_app is a script that can take a a core/gcore or process id and gather all the libraries required by Sun Support to debug a core/gcore file. (More detail of how to get stack trace with pkgapp can be found from following link- https://communities.ca.com/people/SungHoon_Kim/blog/2016/02/25/collecting-pkgapp-and-how-to-get-the-stack-trace)

Work with Support

If you require assistance from the CA Single Sign-On Support team, there is specific information you can gather and include when opening a Support ticket. Including as much information as possible helps to reduce the amount of time it takes the Support team to resolve the issue.

Note: If you are attaching log files as part of your Support engagement, be sure that the set of files matches. Also, ensure that all the files are from the same time as when the issue occurred.

Have a continuing nice time with your logging. 

Cheers - Gwan

---- Gwan Yu Kim Snr Support Engineer - Global Customer Success