Symantec Access Management

CA Single Sign-On Tech Tip by Stephen McQuiggan, Sr Principal Support Engineer for June 24, 2015

##### Tip Description

Tech tip describes the statistics

Stats switch provides current policy server process runtime statistics such as: pool limit, thread pool message, queue depths for new agent connections (high priority queue), agent API queue depths (normal priority queue), and the number of connections.

smpolicysrv switch: Can be run in a scheduler or cron job for a predefined interval (suggest running every 5 minutes)

NOTE: In a future release this will be configurable/automatic / controlled

• -stats:  This switch produces current server runtime statistics such as thread pool limit, thread pool message, and the number of connections.
• -resetstats: This switch resets the current server runtime statistics without restarting the Policy Server. This switch resets the following counters

Sample output to SMPS.LOG

[sm-Server-02360] Server 'Stats' command received

[sm-Server-01990]===================================================================================

[sm-Server-02000] System Statistics

[sm-Server-02010] Available file descriptors: 16384

[sm-Server-02020] Thread pool limit: 16

[sm-Server-02030] Thread pool: Msgs=317 Waits=317 Misses=0 Max HP Msg=1 Max NP Msg=1 Current Depth=0 Max Depth=1 Current High Depth=0 Current Norm Depth=0 Current Threads=2 Max Threads=2

[sm-Server-02040] Connections: Current=4 Max=4 Limit=4096 Exceeded limit=0

[sm-Server-01990] ===================================================================================

Thread processing of message queue: Return in data the next message on the queue, if the queue is empty wait until there is data available or until specified time. Returns true if message received and false if there was a time expire (or error waiting)

Msgs = # of thread pool messages received

Waits = # of times that a thread went to get a request from the queue to process, but had to wait for one to arrive
Misses = # of times that a thread went to get a request from the queue to process, had to wait for one to arrive, but none came in before the thread went back to sleep (time expired)
Max HP Msg =  maximum number of High Priority messages at one time on the queue since the last reset of stats
Max NP Msg = maximum number of Normal Priority messages at one time on the queue since the last reset of stats
Current Depth = total number of message in the queue at the time of executing –stats
Max Depth = maximum number of total messages on the queue since the last reset of stats
Current High Depth =  Agent connection requests-High priority messages in the queue at the time of executing -stats
Current Norm Depth = Agent API calls/request-Normal priority messages in the queue at the time of executing -stats
Current Threads = # threads running at the time of executing -stats
Max Threads = maximum thread number reached
PriorityRatio = ratio: 1 to N, where N specifies number of Agent Connection requests to serve before serving one Agent API request (deprecated)
Connections:
Current = number of agent connections
Max = maximum number of connections since last reset
Limit = maximum  allowable connections 
Exceeded limit = number of times exceeded the limit

How to tune the number of threads:

Having a “Waits” value that is very close to the “Msgs” value coupled with a low “Misses” value implies that there are enough threads to handle the incoming load.  A high “Misses” value would indicate that you may have too many threads configured. A large “Msgs” value coupled with a low “Waits” value and a low “Misses” value would imply that you may need to increase the number of Worker Threads.

This is a very helpful tip.

However, trying to understand the whole high/low since it is fairly vague and subjective. What is high relative to other numbers? 50%, 70%, etc of total? Is there any specific guidance on total numbers and ranges or anything?

For example, if I have the following output at sampling:

Thread pool: Msgs=250000 Waits=230000 Misses=1000000 Max HP Msg=10 Max NP Msg=700 Current Depth=0 Max Depth=700 Current High Depth=0 Current Norm Depth=0 Current Threads=8 Max Threads=8 Busy Threads=5

Would that number of misses be considered 'high' ? And let's say you have high load only during 'peak' times but very low load / connections during any other times. Wouldn't you need high threads to handle the peak few hours but possibly end up causing high misses then during non-peak hours due to more threads configured than necessary for the majority of time...so over a period of operations that would stack up and seem like a 'high' number of misses - even though the increased threads is necessary for peak loads daily?

Thanks for feedback

You would want to configure the threads based on the highest excepted load +.  The policy server starts all threads at startup, because of this it may be difficult to determine a good thread #value specifically if the site has some high peaks followed by periods of low traffic  - suggest testing in control environment with the highest expected load then adjust as needed

Steve M

An updated version on this blog here : Tech Tip : CA Single Sign-On :: Policy server :: STATS Information (Updated)