Problem:
The Advanced Authentication Flow Application installed on the CA Access Gateway (CA Secure Proxy Server) might fail to startup/initialize for number of reasons. In this blog, we will discuss way to troubleshoot these kind of issues with few samples.
Environment:
- CA Access Gateway (formerly CA Secure Proxy Server ) : R12.52SP1 and above
- Policy Server : R12.52SP1 and above
Background:
Logs
The Advanced authentication components on the CA Access gateway comprises of multiple component which generate following logs at the <ARCOT_HOME>/logs directory.
- arcotuds.log—the log file for the UDS services
- CAWebFlowLog.txt—the log file for the authentication flow application
- UIApplog.txt—the log for the UIApp (part of the authentication flow application execution process)
Configuration Files
- arcotcommon.ini (available in <ARCOT_HOME>/conf directory)
The arcotcommon.ini file contains the parameters for database and instance settings for UDS (User Data Service) of Advanced Authentiction server.
For e.g. the primary database
[arcot/db/primarydb]
Datasource.1=CAAdvancedAuthDSN
AppServerConnectionPoolName.1=
URL.1=ca::jdbc:odbc:CAAdvancedAuthDSN
Username.1=admin
The password corresponding to Username.<N> needs to be securely stored in securestore.enc with value of Datasource.<N> as the key. The dbutil utility is used to achieve this.
Note : This datasource must be exists in the odbc.ini file.
- udsserver.ini (available in <ARCOT_HOME>/conf directory)
The udsserver.ini file contains the parameters to set the User Data Service (UDS) log information. The following table lists the log file information of UDS
- odbc.ini (available in <ARCOT_HOME>/odbc directory)
The odbc.ini file contains a list of Data Sources and any properties for each of them. If configured properly it should look like below :
[ODBC Data Sources]
CAAdvancedAuthDSN=SiteMinder Policy Server Wire Protocol
[CAAdvancedAuthDSN]
Driver=/opt/CA/secure-proxy/arcot/lib/libdaproxy.so
HostConfigFile=/opt/CA/secure-proxy/arcot/conf/SmHostFlow.conf
[ODBC]
Trace=0
DATrace=0
DATraceSettingsFile=/opt/CA/secure-proxy/arcot/conf/datracesettings.ini
TraceFile=/opt/CA/secure-proxy/arcot/logs/odbctrace.out
TraceDll=/opt/CA/secure-proxy/arcot/odbc/lib/NStrc27.so
InstallDir=/opt/CA/secure-proxy/arcot/odbc/
Note : The odbc.ini file exists only in case of Unix platform. The corresponding settings is available in Windows in the 32 bit DSN :
Troubleshooting:
The UDS Service (User Data Service) is the core Advanced Authentication component.
The parameters that control logging in this file can be configured by using the udsserver.ini file, which is available in the <ARCOT_HOME>/conf directory.
For troubleshooting any issue related to Session Assurance component on the CA Access Gateway , it is best to start with this UDS log (arcotuds.log) and also it is better to increase the logging level as below in the udsserver.ini file:
Change From:
[arcot/uds/logger]
log4j.rootCategory=ERROR, debuglog
log4j.logger.com.arcot.euds=INFO
log4j.logger.com.arcot.crypto.impl.SecureStoreUtil=INFO
log4j.logger.com.arcot.common.database=INFO
log4j.logger.com.arcot.common.cache=INFO
To
[arcot/uds/logger]
log4j.rootCategory=DEBUG, debuglog
log4j.logger.com.arcot.euds=DEBUG
log4j.logger.com.arcot.crypto.impl.SecureStoreUtil=DEBUG
log4j.logger.com.arcot.common.database=DEBUG
log4j.logger.com.arcot.common.cache=DEBUG
Issue 1: Error on arcotuds.log :
[localhost-startStop-1] : WARN : core.pool.PoolableSocketFactory : Unable to connect to RemoteHost [<PolicyServer_Hostname/FQDN/IP>:7680]
java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method)
Resolution:
This indicates either the "CA RiskMinder" service is down at the Policy server or the port 7680 is not opened between Policy server and CA Access Gateway machines.
Ensure that CA RiskMinder service is up and you can telnet to Policy server on 7680 port from CA Access Gateway server.
Issue 2: Error on arcotuds.log:
2016-08-04 00:08:50,969 PDT : [localhost-startStop-1] : DEBUG : database.ibatis.ArcotiBatisDataSourceFactory : Datasource[1] added successfully to the datasource list
2016-08-04 00:08:50,981 PDT : [localhost-startStop-1] : ERROR : cache.db.CacheRefreshService : Internal error: Unable to read cache refresh state table
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (SiteMinder Agent API initialization failure)
Resolution:
The Advanced Authentication component on the CA Access Gateway communicates with the Policy server on the JDBC/ODBC bridge. The above error usually indicates that the DSN is not fully configured for this purpose.
Check if the SmHostFlow.conf file exists on the <ARCOT_HOME>/conf directory exists.
If it does not exists, re-running the configuration wizard for CA Access Gateway would fix this issue.
Issue 3 : Same error as Symptom 2 but followed by following error in smps.log:
[2332/2396][Thu Aug 04 2016 17:18:58][CServer.cpp:2017][ERROR][sm-Tunnel-00050] Handshake error: Shared secret incorrect for this client
[2332/2396][Thu Aug 04 2016 17:18:58][CServer.cpp:2178][ERROR][sm-Server-01070] Failed handshake with <CA_ACCESS_GATEWAY_IP>:<PORT>
Resolution:
This happens when the SmHostFlow.conf file exists but the shared secret defined within it is no more valid.
To solve, this , you can either copy the SmHost.conf file from <SPS_Install_Directory>/proxy-engine/conf/defaultagent/SmHost.conf to the <ARCOT_HOME>/conf directory and rename to SmHostFlow.conf file (provided the defaultagent is able to initialize without any error)
Alternatively, you can also register a new trusted using following command :
./smreghost -i <PolicyServer_IP>:44441 -u "siteminder" -p <super user password> -hn <trustedhostname> -hc <host_config_object> -cf COMPAT -f <ARCOT_HOME>/conf/SmHostFlow.conf
You can find the smreghost script in <CA_Acess_Gateway_Install_Directory>/agentframework/bin directory
Issue 4 :
Error on arcotuds.log:
2016-08-04 16:18:57,726 PDT : [localhost-startStop-1] : ERROR : cache.db.CacheRefreshService : Internal error: Unable to read cache refresh state table
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory ([DataDirect][ODBC lib] Data source name not found and no default driver specified)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
Error on CAWebFlowLog.txt
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Invalid Oracle URL specified
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1455)
Resolution:
This error indicates that the "CAAdvancedAuthDSN" data source does NOT exists in the <ARCOT_HOME>/odbc/odbc.ini file.
The entry is created if the CA Access Gateway configuration wizard is run successfully.
Alternatively, it can be configured manually as indicated in the sample above. Ensure that all the path are valid.
If the SmHostFlow.conf does not exist in the <ARCOT_HOME>/conf directory , you can apply the resolution from Issue 3 (above).
Issue 5 :
Error on CAWebFlowLog.txt
2016-08-18 17:12:31,208 [FlowExecutor,ajp-bio-9009-exec-3] ERROR - TID[-1] 807259999: Unexpected exception in flow {0}(ID={1}); destroying session
java.lang.RuntimeException: com.arcot.corejsvr.ExceptionWithNC: 808068044: Token ID missing from request.
Resolution:
This error indicates that the cookie is not submitted to the Session Assurance End Point.
This could happen if -
1) Cookie domain of the protected resource and that of Session Assurance End Point do not match.
2) Host only cookie is being used.
To fix this error, ensure that both the protected resource and the Session Assurance End point are using the same cookie domain and also that they are not using Host ONLY cookies.