Mark.ODonohue

Tech Tip : Howto enable Tracing in Access Gateway (fka: Secure Proxy Server)

Blog Post created by Mark.ODonohue Employee on Mar 7, 2017

Introduction 

The purpose of this blog entry is to show how to enable all the different types of trace logs that are available in CA SSO Access Gateway (formerly known as Secure Proxy Server).   I will also be referring to the Access Gateway product as "Ag" in the article -however some of the slides predate the name change so will show up as SPS.  

 

Ag can be used a few different ways, and depending upon what you are using it for will determine what logs you want to enable.   I've split this up into different themes:

  • Ag Logging when used as Reverse Proxy Server
  • Ag Logging when used as Federation Gateway
  • Ag Logging for ProxyUI 
  • Ag Logging for WebServices

 

This log covers the Logginfg when used Ag is used as a Reverse Proxy Server the other logging profiles will be added as a separate documents at a later date. 

 

 

 

Ag Logging when used as a Reverse Proxy Server: 

 

The following gives an overview of the major components of Ag and also shows the name of (all) the logs that can be enabled and where they get their data from:

 

 

 

When used as a reverse proxy server, requests come in from the client, to Apache httpd, get passed to Apache/tomcat and then get forwarded to a backend server for processing.  The backend then completes the request and the data is then passed back to tomcat, to httpd and back to the client.    Note: I wont be discussing the use of fiddler and wireshark, but the diagram indicates where they would be used.

 

In summary we have: 

  • Apache Logs
  • Mod_Jk logs
  • Proxy Engine Logs
  • Web Agent Logs
  • Httpclient Logs

Each of which is covered in the sections below. 

 

 

Apache Logs 

The two major logs for apache httpd are access_log and error_log, these log the interaction with the user <-> httpd process. The httpd.conf entries are :

 

- Access_log  - settings in httpd.conf :  

The formats are defined here : 

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

 

And the rotating logs are set here : 

CustomLog "|'C:/Program Files/CA/secure-proxy/httpd/bin/rotatelogs.exe' 'C:/Program Files/CA/secure-proxy/httpd/logs/access_log' 10M" common

 

Occasionally it is good to supplement what is in the access_log and to get some insight to a problem.  The example above it shows the %{User-Agent}i header, but you can also use that to capture cookies for example:  \"%{SMSESSION}C\" .   The option %T is also useful, since it logs the total time it took Ag to process the request and return the response to the user. 

 

A complete list of LogFormat parameters is available here: 

Tech Note : Enable httpclient logging in Access Gateway 12.7 

 

Note: And one final point to remember, the the access_log entry is written at the END of processing the request.  So if apache httpd crashed then the requests that are currently in flight when the crash happened are NOT logged. 

 

- Error_log  - settings in httpd.conf :  

The formats are defined here : 

# LogLevel: values include: debug, info, notice, warn, error, crit, alert, emerg.
LogLevel warn

 

And the rotating logs are set here : 

ErrorLog "|'C:/Program Files/CA/secure-proxy/httpd/bin/rotatelogs.exe' 'C:/Program Files/CA/secure-proxy/httpd/logs/error_log' 10M"


For debugging, you can raise the LogLevel to debug.   Apache 2.4 also has extra levels trace1 ... trace8, there are needed when you want to trace the raw data packets and SSL handshaking problems between the front end client and the httpd process. So for debugging often we can recommend : 

LogLevel trace8

 

The apache error_log is also good place to find the exact httpd and mod_jk version numbers: 

 

 

Mod_Jk Logs

Mod_jk is the Apache httpd module that forwards requests onto tomcat.  The log settings for it are in httpd.conf : 

 

JkLogFile "|'C:/Program Files/CA/secure-proxy/httpd/bin/rotatelogs.exe' 'C:/Program Files/CA/secure-proxy/httpd/logs/mod_jk.log' 10M"

JkLogLevel error

The log-level parameter describes what detail of logging should occur.   Possible values are : debug  info  error

 

For debug level logging, it is best to also set the JkRequestLogFormat, to display more detail of the transaction:

JkLogLevel debug

JkRequestLogFormat "%w %V %T %m %H %p %U %s"

That will show most of the raw byte data of what is send from httpd -> tomcat and what is returned.  The settings are explained here :  https://tomcat.apache.org/connectors-doc/reference/apache.html 

 

Sample mod_jk.log : 

 

 

Proxy Engine Logs

The proxy engine has two main logs :  

    server.log

    nohup*.out  

These are in secure-proxy/proxy-engine/logs directory by default.   The server.log is the log4j logging for the proxy-engine, and the nohup_<pid>.out log is the redirect of stdout and stderror logs 

 

server.log

Logging level for server.log is set in Tomcat/properties/logger.properties 

log4j.rootCategory=INFO,SvrFileAppender

log4j.rootCategory.ResourceBundle=root

The log level can bec changed to OFF, FATAL, ERROR, WARN, INFO, DEBUG, ALL

 

nohup_<date>_<time>.out log : 

We generally don't change the logging in this one as it logs the stdout/stderr logs from the proxy-engine.  Although one useful tip is adding "-verbose" to the java startup, and then you get the exact .jar file that each class is loaded from in this log.   A new timestamped log is started each time the proxy-engine is started.  The nohup log is good at capturing the stacktrace when Exceptions are thrown in proxy-engine eg: 

 

 

 

Web Agent Logs

 

Ag comes witn the standard WebAgent logs.  These are enabled via the ACO settings as per the normal agent eg: 

Must be enabled, and setup as normal agent ACO parameter: 

 

WebAgent.log 

LogAppend="NO"
LogFile=“YES"
LogFileName=“c:\ca\proxy-engine\logs\WebAgent.log"
LogFileSize="100"

 

 

WebAgentTrace.log 

TraceAppend="NO"
TraceConfigFile=“c:\ca\proxy-engine\conf\defaultagent\SecureProxyTrace.conf"
TraceFile=“YES"
TraceFileName=“c:\ca\proxy-engine\logs\WebAgentTrace.log"
TraceFileSize="100"

 

 

WebAgentTrace.log SecureProxyTrace.conf settings :

The SecureProxyTrace.conf is slightly different to the WebAgentTrace.conf.  It has ProxyAgent as default. 

I also tend to add Agent_Con_Manager, and AgentFunc as components. 

And add data items :  PreciseTime, Function,  and SrcFile as shown below: 

 

WebAgentTrace.log with proxy-rule messages : 

Additionally for the webagenttrace to log the proxy rule evaluation you need to add debug=“yes” to proxy-rules.xml to get additional error messages specific to SPS :

 

 

WebAgentTrace.log examples: 

After setting the above then we endup with normal trace log like: 

 

 

And with Ag specific messages for proxy-rules such as:

 

 

HttpClient Logs

Http client  logs the raw data GET/POST that is sent to the backend and and reply that is received.   So it is good for debugging the interaction with the backend server. 

 

To enable httpclient logging in server.conf set :
           httpclientlog = “yes”

and restart the proxy-engine service. 

 

Note: For Ag R12.7 there is extra setting needed to enable httpclient logging: 

https://communities.ca.com/community/ca-security/ca-single-sign-on/blog/2017/09/01/tech-note-enable-httpclient-logging-in-agent-gateway-127

 

HttpClient / Java SSL Logging

Java has the ability to log the SSL handshake and transfer of data.  This is done by adding  -Djavax.net.debug=all 
to the java runtime startup.  The file this needs to be applied to differs per platform :

For Windows - proxy-engine/conf/SpsProxyEngine.properties
For Unix - proxy-engine/proxyserver.sh

As show below: 

Enable SSL tracing for java:

 

SSL Tracing in the nohup and server.log files : 

These logs then show the SSL handshake, and decrypt/hash of each packet send and received when proxy-engine communicats to the SSL backend: 

 

Have a nice time enjoying your logging. 

 

Cheers - Mark

----
Mark O'Donohue
Snr Principal Support Engineer - Global Customer Success

 

This document is part of a series on Logging in SSO components: 

Tech Tip:How to enable trace logging in SSO (aka Siteminder) Webagent 

Tech Tip : Policy Server Loggings 

Tech Tip : Howto enable Tracing in Access Gateway (fka: Secure Proxy Server) 

Outcomes