Skip navigation
All Places > CA Security > CA Single Sign-On > Blog > 2017 > September

(This is a revised version of : Tech Tip : CA Single Sign-On :: Policy server :: STATS Command )


Please clarify the information listed on running smpolicysrv -stats command.


Policy server : 12.52 SP1 CR6


Here is an example of the 'smpolicysrv -stats' which outputs the smps.log when run:


[3524/4640][Sun Sep 17 2017 23:49:55][CServer.cpp:4747][INFO][sm-Serveor-01990] ===================================================================================
[3524/4640][Sun Sep 17 2017 23:49:55][CServer.cpp:4748][INFO][sm-Server-02000] System Statistics
[3524/4640][Sun Sep 17 2017 23:49:55][CServer.cpp:4765][INFO][sm-Server-02020] Thread pool limit: 70
[3524/4640][Sun Sep 17 2017 23:49:55][CServer.cpp:4787][INFO][sm-Server-02030] Thread pool: Msgs=36630 Throughput=0.033/sec Response Time=209.441ms Wait Time In Queue=0.077ms Max HP Msg=2 Max NP Msg=1 Current Depth=0 Max Depth=2 Current High Depth=0 Current Norm Depth=0 Current Threads=8 Max Threads=8 Busy Threads=0
[3524/4640][Sun Sep 17 2017 23:49:55][CServer.cpp:4795][INFO][sm-Server-02040] Connections: Current=0 Max=23 Limit=256 Exceeded limit=0
[3524/4640][Sun Sep 17 2017 23:49:55][CServer.cpp:4798][INFO][sm-Server-01990] ===================================================================================


Note :

  • stats:  This switch produces current server runtime statistics such as thread pool limit, thread pool message, and the number of connections.
  • resetstats: This switch resets the current server runtime statistics without restarting the Policy Server. This switch resets the following counters


Stats Information

  • Msgs                       : This is the total number of requests that have been removed from the queue.
  • Throughput              : Average number of request processed per second (request/per second)
  • Response Time         : Average response times.
  • Wait Time In Queue  : Average time interval between Enqueue and Dequeue of the messages (request) from the queue
  • Max HP Msg             : Maximum number of High Priority messages on the queue since the last reset of stats
  • Max. NP Msg            : Maximum number of Normal Priority messages on the queue since the last reset of stats
  • Current Depth          : Number of messages in the queue at the time of executing -stats
  • Max Depth               : Maximum number of messages in the queue since the last rest of stats
  • Current High Depth   : Number of High Priority messages in the queue at the time of executing -stats
  • Current Norm Depth : Number of Normal Priority messages in the queue at the time of executing -stats
  • Current Threads       : Number of worker threads running at the time of executing -stats. This includes sum of busy and idle worker threads.
  • Max Threads            : Maximum number of worker thread at any time.
  • Busy Threads           : Number of worker threads actively processing request at the time of executing -stats


Deprecated stats information (available in older releases) : 

  • Waits: This is the number of times that a thread went to get a request from the queue to process, but had to wait for one to arrive.
  • Misses: This is the number of times that a thread went to get a request from the queue to process, had to wait for one to arrive, but none came in before the thread went back to sleep.


  • Current: current number of agent connections
  • Max: Maximum number of agent connections
  • Limit: Maximum allowable agent connections
  • Exceeded Limit: Number of times the connection limit was exceeded.

Exceeded limit notes if the maximum agent connection limit has been reached. If further agent connections are attempted (the connections attempted past the limit will be rejected.


Some relevant registry:

Some of these parameter can be configured by modifying the registry (windows registry or sm.registry file on unix)

32 bit : HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Netegrity\SiteMinder\CurrentVersion\PolicyServer\

64 bit : HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\PolicyServer\


Thread Pool Initial Size

Specifies the minimum number of worker threads that Policy Server opens on startup.

Default: 8


Thread Pool Size

Specifies the maximum number of worker threads that can be opened.

Default: 20


Tcp Max Server Connections

Specifies the maximum number of allowable agent connections.

Default : 256



Number of High Priority Threads. (threads allocated to process high priority handshake messages)

Default : 5

Max : 20

A value less than five or greater than 20 disables the registry key.


Additional Information

Having a "Waits" value that is very close to the "Msgs" value coupled with a low  "Misses" value implies that there are enough threads to handle the incoming load, but there are not too many threads configured. A high "Misses" value would indicate that you may have too many threads configured.


A large "Msgs" value coupled with a low "Waits" value and a low "Misses" value would imply that you may need to increase the number of Worker Threads.


The CA SSO - Secure Proxy Servers releases : R12.51, R12.52, R12.52Sp1 are supported for connecting to some older backend policy server versions.


For SPS released R12.51, R12.52, R12.52 Sp1 the recommended install for SPS is still that the Policy Server release version should be equal or greater than the SPS (or webagent) version.  

But there were some considerations with the above SPS releases, mainly that they let you select an earlier release version of the policy server in the install process, and the net result was that, although not the recommended arrangement, it is a supported configuration.


This above exception is not the case for SPS releases post R12.52 Sp1, (SPS is now called Access Gateway) so Ag releases R12.6 and R12.7 for supported configuration the policy server release must be >= agent version.


A similar situation also exists for CA Sharepoint 2010 Agent.



 Generally SSO agents have a strict regime that a policy server release must be greater than or equal to the webagent release version.   From the R12.52 PSM:

"CA SiteMinder12.52 Policy Server supports Web Agents at a higher CR (cumulative release) number than the Policy Server
provided both are the same level Service Pack."


 So a supported configuration is :

WebAgent R12.51  -> PolicyServer R12.51 (or R12.52, R12.52Sp1 or R12.6 or R12.7) 

And example unsupported configurations are :

WebAgent R12.52 -> PolicyServer R12.51

WebAgent R12.52Sp1 -> PolicyServer R12.52


But within the Service Pack release the CR level does not matter and the WebAgent can have a latter CR release,  so :  

WebAgent R12.52 Cr5 -> PolicyServer R12.52

WebAgent R12.52Sp1 Cr5 -> PolicyServer R12.52Sp1 Cr1

are also supported configurations. 




After install and on running the configure of SPS it gives a choice of what policy server version it is being connected to.  The exact choice depends on the release but are : 

    SPS R12.51 – gives choice of PolicyServer : R6, R12, R12.X

    SPS R12.52 – gives choice of PolicyServer : R6, R12.X, R12.5X

    SPS R12.52Sp1 – gives choice of PolicyServer : R12.X, R12.5X

The choice here determined which affwebservices package is installed into the SPS Tomcat deployment.


Here was the clarification note from Engineering :

Here are the SPS compatibility with Policy Server guidelines, across versions:

1) SPS 12.5 release was geared towards bringing SPS at parity with web-agents ; and to encourage customer-adoption (without forcing Policy Server upgrades), backward compatibility with older Policy Server versions was provided. 

o) This was mentioned in SPS PSM / Release-Notes & explicitly prompted during SPS configuration-screen too.


2) Later versions of SPS, 12.51 & 12.52 continue the configuration screen and support backward compatibility (PS 12.0.x & 12.5 x); though it is not explicitly highlighted in PSM. 

o) However, this compatibility is only for existing/old features & not the new ones. For ex, Session Assurance, support for Active Profile was brought in during 12.52 release and requires both SPS & Policy Server to be 12.52  version. These features are not supported with older PS versions.


Hope this clarifies the compatibility questions.  

Also, going forward, with next version of SPS, plan is not to carry forward this backward compatibility, and only support traditional compatibility (old agents support newer servers, for existing features).


The going forward ended up converting R12.6 and R12.7 which now have a fixed deployment of affwebservices deployed (the same version as per the release) , and do not allow you to select older versions. 


For those R12.5X releases, the screen you are presented with when running SPS configure (ca-sps-config.exe) : is :

For SPS R12.51  & R12.52 



And for R12.52 Sp1 :


In our final post on migrating to a CA SSO solution, we will present six things you need to keep in mind if you want to accelerate the project. There is no standard timeframe for implementation of CA SSO; the speed of implementation is dependent upon a wide range of factors. These factors especially impact implementation timelines:


  • Existing authentication(s): The number and type of authentication and session management mechanisms currently in place to protect web applications affects implementation complexity.
  • User stores: The number and type of existing user stores and migration and synchronization requirements affect project progress. 
  • Custom code requirements: Requirements related to custom authentication schemes, active responses and policies or advanced functionality not supported by the current product can have a negative impact on production timelines.
  • Availability of hardware: Long hardware lead times often have a negative impact on rapidly moving the implementation beyond the development phase.
  • Resource availability: If an organization cannot dedicate the enough resources to provide the necessary info for requirements gathering and help generate designs, the process will be prolonged.
  • Modifications to legacy applications: A company must have developers familiar with the applications being integrated into the architecture and be able to modify applications as necessary. Modifications to legacy applications are sometimes a large undertaking; if the applications aren’t ready when the solution infrastructure is available, the timeline will be affected negatively.


If you are interested in exploring CA SSO implementation options contact


In this guide, we will discuss about the steps performed during web agent initialization.


Then, we will also deep dive into some of the common agent initialization issues and discuss approaches to troubleshoot and resolve theses issues.



  • Web Agent : 12.5 and above
  • OS : ANY 


For this tech tip , we will test on following platform :

  • Web agent version : 12.52 SP1 CR7
  • Web Server : Apache 2.2
  • Web Server OS : RHEL 6.5 64 bit


Web Agent Startup Process 


On the high level the web agent startup process happens in the following order :


  1. Read WebAgent.conf
  2. Locate the path to the SmHost.conf file from WebAgent.conf and read SmHost.conf
  3. Identify the following details from SmHost.conf :
    • Policy server IP ( this policy server is used only for the initial bootstrapping)
    • Shared Secret
    • Trusted Host Name (hostname)
    • Host configuration object (HCO)
      #agentname="<AgentName>, <IPAddress>"

      Figure : WebAgent.conf

      #Add additional bootstrap policy servers here for fault tolerance.

          Figure : SmHost.conf

  4. Establish Agent API connection with the Policy server listed in the SmHost.conf file. This includes 3 way handshake (more details below ).
  5. Read HCO (Host configuration object) from the policy server/policy store.
  6. Establish the Agent API connection with the primary policy server listed in the HCO ( this could be same or different from the bootstrap policy server as listed in SmHost.conf)
  7. Once the connection is established with the primary policy server listed in the HCO , read ACO (Agent configuration object as per the WebAgent.conf)
  8. Initialize agent log/trace file as per the ACO configuration.


Three way agent to policy server handshake


  1. Agent opens a TCP socket connection with the policy server.
  2. Agent sends a Hello message which includes following info among other details :
    • MD5 Hash of shared secret and trusted host name combined (stronger encryption is used if using FIPS only mode)
    • Trusted host name
  3. Policy server validates the shared secret based on the trusted host name passed. It may validate both current and previous shared secret from the policy store against the shared secret sent by agent. If the shared secret validation is successful , policy server sends Hello Reply message which consists of following info among other details :
    • Session Keys
    • New shared secret (optional - this is sent only if the agent currently doesn't have the current shared secret)
    • New shared secret generated time (optional)
  4. Agent sends Hello Confirm message encrypted with the Session Keys previously sent by Policy server.
  5. (Optional ) Agent updates the SmHost.conf file with the new shared secret.


                                             Figure : Three way agent to Policy server handshake


Web Agent Initialization Error Codes :


Code             Meaning                                   
00 00 00 00     Debug version of SiteMinder agent is running.                                   
01 00 00 00     Unable to determine SiteMinder agent configuration file path.                                   
02 00 00 00     Unable to open SiteMinder agent configuration file or file is corrupt.                                  
03 00 00 00     Unable to load SiteMinder host configuration object or host configuration file.                                  
04 00 00 00     Unable to load SiteMinder agent configuration object.                                  
05 00 00 00     Unable to load SiteMinder local agent configuration file or file is corrupt.(EG: Web Server user does not have permissions on the Web Agent repositories & files.)
06 00 00 00     SiteMinder agent has encountered initialization errors and is exiting.                                  
07 00 00 00     SiteMinder agent has encountered initialization errors and will not service requests.                                  
08 00 00 00     SiteMinder agent is not enabled.                                  
09 00 00 00     SiteMinder agent is enabled.                                  
10 00 00 00     DefaultUserName configured for agent cannot logon to the web server. Please provide a new user name or password through central agent configuration or in the local configuration file. The current user name configured is shown below.                                  
11 00 00 00     Secure credential cache has failed to start. The data is the error code. Please check the System events for problems with service startup.                                  
12 00 00 00     SiteMinder agent is running.                                  
13 00 00 00     There was an error allocating memory for the base configuration object.                                  
14 00 00 00     Sm_AgentApi_Init Failed.                                  
15 00 00 00     Failed to Start the LLAWP process.                                  
16 00 00 00     Resource cache failed to initialize.                                  
17 00 00 00     Session cache failed to initialize.                                  
18 00 00 00     Failed to send message to the LLAWP.                                  
19 00 00 00     Failed to initialize the message bus.                                  
20 00 00 00     Failed to initialize the log queue.                                  
21 00 00 00     Failed to initialize the configuration manager.                                  
22 00 00 00     Server already running.                                  
23 00 00 00     Unable to open file.                                  
24 00 00 00     Configuration file path:                                  
25 00 00 00     Failed to send close message to LLAWP.                                  
26 00 00 00     LogonUser failed for specified user shown below.                                  
27 00 00 00     Invalid character found in the server path variable. Make sure that alphanumeric values are used. the invalid character shown below.                                  
28 00 00 00     Message bus already initialized.                                  
29 00 00 00     PID Cache error.                                  
30 00 00 00     Resource cache re-initialized.                                  
31 00 00 00     Session cache re-initialized.                                  
32 00 00 00     Web-agent process is exiting...                                  
ff ff ff ff     unable to get the HostConfigurationObject from any Policy Server                                  


Basic troubleshooting:


  • On UNIX platform, ensure you have sourced the web agent environment script ( before starting the webserver.
    [root@rhl65 webagent]# pwd
    [root@rhl65 webagent]# source ./
    [root@rhl65 webagent]#

    On Windows, the web agent environment variable are set as system environment variable. Ensure that the user running the web server process has access to these system environment variables.

  • Ensure path to the host configuration file (HostConfigObject) is valid in WebAgent.conf. 
  • Ensure the name of the agent configuration object (AgentConfigObject) is valid in WebAgent.conf (This is case sensitive field and need to match against the name of the ACO in the policy store)
  • Ensure that the user under which web server process runs has write permission to SmHost.conf (This is optional requirement. This is required only if the shared secret rollover functionality is used. )
  • Ensure that either defaultagentname or the agentname ACO parameter must be set in the policy store.
  • Ensure that the policy server FQDN/IP is specified in SmHost.conf file. Also ensure that you can ping and resolve the DNS for the specified policy server from the web server host.
  • Ensure that a valid Policy server FQDN/IP is specified in Host configuration object in the policy store. Also ensure that you can ping and resolve the DNS for the specified policy server from the web server host.
  • If there are multiple policy server IP listed in SmHost.conf and HCO , it is usually best to start with just one policy server and comment the remaining servers out. This will help and ease the troubleshooting.
  • On Unix platform, if there are multiple web agent instance running on the same box, ensure that a unique ServerPath is specified for each instance. 
  • Ensure that you can telnet to the Policy server ports from web server
telnet <policyserverIP> <policy server ports>

Test connectivity to all the ports - accounting, authentication , authorization 


Advance troubleshooting:


Following logs will be required to be analyzed for advanced troubleshooting:


Linux :

  • web server error/startup log
  • policy server logs (smps.log) and trace log (smtracedefault.log)

At minimum use following profiler for policy server trace log :

components: AgentFunc, Server/Connection_Management, Server/Policy_Server_General, Tunnel_Service
data: Date, Time, Pid, Tid, TransactionID, SrcFile, Function, User, UserDN, Directory, SessionID, SessionSpec, ErrorValue, ErrorString, Realm, Resource, Action, Rule, Policy, Domain, Message, PreciseTime, ReturnValue, Group, AgentName, AgentType, ObjectClass, DomainOID, SearchKey, ObjectOID, Property, IPAddr, IPPort, AuthStatus, AuthReason, AuthScheme, CertSerial, SubjectDN, IssuerDN, CertDistPt, RealmOID, State, ClusterID, HandleCount, FreeHandleCount, BusyHandleCount, ResponseTime, Throughput, MaxThroughput, MinThroughput, Threshold, TransactionName, Data, HexadecimalData, Query, ActiveExpr, CallDetail, RequestIPAddr, Returns, Expression, Result, CacheHits, CacheSize, RefCount, ExecutionTime, Tenant
version: 1.1
  • strace log from the web server startup ( this will help to identify any issue related to file permission/library dependency/environment variable etc )

    strace -Ff -t -i -v -o strace.log -s 16384 <command to start web server>


strace -Ff -t -i -v -o strace.log -s 16384 apachectl start 
  • tcpdump from webserver

tcpdump -i <network interface> -s 65535 -w <some-file.pcap>
To find the available network interface, you can run following command : ifconfig

[root@rhel65_3 ~]# ifconfig
eth2      Link encap:Ethernet  HWaddr 00:50:56:21:B6:A8 
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::250:56ff:fe21:b6a8/64 Scope:Link
          RX packets:432695 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3238 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:46333929 (44.1 MiB)  TX bytes:926017 (904.3 KiB)

lo        Link encap:Local Loopback 
          inet addr:  Mask:
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:31 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2012 (1.9 KiB)  TX bytes:2012 (1.9 KiB)

[root@rhel65_3 ~]#

then, run the tcpdump command as below :

[root@rhel65_1 Desktop]#  tcpdump -i eth2 -s 65535 -w watopsnetworkcapture.pcap
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
^C244 packets captured
248 packets received by filter
0 packets dropped by kernel
[root@rhel65_1 Desktop]#
  • web agent logs & trace ( note : if there is an agent initialization issue, these logs will most likely not be created as they are created only at the end of init process, however it is good to get this configured just in case )At minimum enable following profiler for web agent trace :
    components:  AgentFramework, HTTPAgent, WebAgent, Agent_Con_Manager
    data: Date, Time, Pid, Tid, SrcFile, Function, ResponseTime, IPAddr, IPPort, AgentName, Resource, User, Threshold, Throughput, MinThroughput, MaxThroughput, HandleCount, BusyHandleCount, FreeHandleCount, State, ClusterID, Message



All the logs from Linux except strace logs are applicable for Solaris/AIX based system.

The Linux's strace equivalent is truss in Solaris.

You can capture truss output from web server startup as below :

truss -a -e -f -D -l -o /tmp/truss.out -rall -wall <command to start webserver>



  • For windows the equivalent of strace is process monitor logs (procmon.exe)

Process Monitor - Windows Sysinternals | Microsoft Docs 

  • TCPDump can also be replaced with the wireshark network capture.

Wireshark · Go Deep. 

  • Event Viewer would also be helpful.


Commons Issues :

Problem : Policy server smps.log shows following error :

[6512/7048][Tue Sep 05 2017 06:41:45][CServer.cpp:2035][ERROR][sm-Tunnel-00010] Bad security handshake attempt. Handshake error: 3154
[6512/7048][Tue Sep 05 2017 06:41:45][CServer.cpp:2046][ERROR][sm-Tunnel-00050] Handshake error: Shared secret incorrect for this client
[6512/7048][Tue Sep 05 2017 06:41:45][CServer.cpp:2207][ERROR][sm-Server-01070] Failed handshake with

The web server log (Apache for our test) shows agent initialization errors:

[04/Sep/2017:23:41:45] [Error] SiteMinder Agent
Unable to load SiteMinder host configuration object or host configuration file.
06 00 00 00
[04/Sep/2017:23:41:45] [Error] SiteMinder Agent
Failed to initialize the configuration manager.
LLAWP unable to get configuration, exiting.
nm: '/etc/httpd/bin/httpd': No such file


No Agent Log/Trace is created.


Cause :

If no changes has been done either in the policy server side or on the agent side since the last working state, then this error indicate a possible change in the hostid (for unix based system) on the web agent side.


On all non-Windows platforms, the agent code used to encrypt and decrypt the shared secret uses a key that is derived from a hard coded value (Web Agent Host Key) combined with the results of calling gethostid() on the platform in question. gethostid() is a standard C Library function that returns a 32-bit long value.

Different UNIX system implements this function differently. For e.g Linux, AIX and solaris , the system implementation for the gethostid() C library function is not the same.
As such, SiteMinder web agent might not be able to decrypt the shared secret generated in one UNIX system when moved to other system. Not only that, if the host ID of the same system changes (due to change in IP, hostname etc ) , the webagent will not be able to decrypt the shared secret which was originally generated on the same system.


Testing :



  • hostname =
  • IP = (in the hosts file)

hostid gives output as a8c00600



Agent initializes fine with the 3 way agent to PS handshake being successful



Now, change the IP address to in the hosts file with everything else remaining the same.

This time hostid command gives a different result : a8c00700



3 way agent to PS handshake now fails with the following error in smps.log :

[6512/7048][Tue Sep 05 2017 06:41:45][CServer.cpp:2035][ERROR][sm-Tunnel-00010] Bad security handshake attempt. Handshake error: 3154
[6512/7048][Tue Sep 05 2017 06:41:45][CServer.cpp:2046][ERROR][sm-Tunnel-00050] Handshake error: Shared secret incorrect for this client
[6512/7048][Tue Sep 05 2017 06:41:45][CServer.cpp:2207][ERROR][sm-Server-01070] Failed handshake with



Also, note in the network capture, the encrypted data in front of the trusted hostname is now different.

If the shared secret+trusted hostname+hostid combination is same, the encrypted data should remain same.


Resolution :

As we saw above, a simple change in the IP address resulted in the change in the hostid in RHEL system. This in turn invalidated the shared secret stored in SmHost.conf. There could be more factor contributing to the change in the hostid which is dependent on the platform being used.


The only way to fix this issue is by re-registering the trusted host or reverting to the previous hostid( reverting to previous IP address in this case).


smreghost -i <policyserver_ip>:44441,44442,44443 -u "siteminder" -p <siteminder super user password> -hn <trustedhost> -hc  <hco> -cf COMPAT -f <Path_To_SmHost.conf> -o



[root@rhl65 bin]# pwd
[root@rhl65 bin]# smreghost -i,44442,44443 -u "siteminder" -p "siteminder" -hn th-rhel65-5 -hc  hco -cf COMPAT -f /opt/CA/webagent/config/SmHost.conf -o
Host Registration written to '/opt/CA/webagent/config/SmHost.conf'.
[root@rhl65 bin]#


If it is expected to have frequent changes in the web server IP address (due to reboot/change in network interface/dns server etc.) , it is recommended to specify a static hostid.


In RHEL you can do this by running command : genhostid. The static hostid is then stored in /etc/hostid file.

(Please refer to your respective OS documentation on how to set static hostid)




To be continued....  






Documented Process for enabling httpclient logging in Access Gateway  


With Access Gateway R12.7 using the documented way to enable httpclient logging does not work. 

The documented method is to change the server.conf entry: 

    httpclientlog ="yes"


But an extra setting is required in Ag R12.7



For Access Gateway R12.7 it needs : 


In server.conf : 

   httpclient ="yes"

This should be the only setting required.

But httpclient logging uses a common logging module, that can use either SimpleLogging, or log4j or java.util.logging.  The common logging module does some checking at startup, and determines which logging mechanism to use.   Prior to R12.7 the default was java.util.logging, but some change in R12.7 has changed the default. 

You can set the default for common logging to java.util.logging via JVM startup property : 



For Access Gateway R12.7 this needs to be set in : 


For Windows: 



Edit the file and add the parameter : 

2017-08-29 12_26_04-SPS R12.52 Sp1Cr02 - VMware Workstation.png


For Unix : 


Edit the file and add the parameter :  

2017-08-29 12_35_19-CentOS 6.5 64-bit-SPS - VMware Workstation.png


It is also raised as an internal Sustaining Engineering ticket, and we expect it will be fixed in a latter CR version.



Raised KD: 

Enable httpclient logging in Agent Gateway 12.7