Tech Tip:How to enable trace logging in SSO (aka Siteminder) Webagent

Document created by riksa01 Employee on Apr 18, 2017Last modified by Mark.ODonohue on Nov 15, 2017
Version 3Show Document
  • View in full screen mode

Introduction:

Web Agent logging function to monitor the performance of the Web Agent and its communication with the Policy Server.  The following gives an overview of the logging that can be performed on the websever/webagent and where they get their data from:

 

 

In Summary the Log file groups we have are :

  • Web Server Logs (IIS and Apache httpd) 
  • Web Agent Logs (WebAgent.log, WebAgentTrace.log) 
  • SM_TLI_LOG_FILE 

 

And a number of trace tools that are useful.

  • fiddler
  • wireshark
  • netstat
  • strace (Unix)   
  • Sysinternals Process Monitor (Windows)
  • IIS Failed Request tracing (Windows)

 

Below we cover each of the log file groups in some more detail.

 

1. Web Server Logs (IIS and Apache httpd) 

 

1.1  Microsoft IIS web server Logs

 

<need something on normal IIS log> 

 

1.2 Apache httpd web server Logs  

 

The two major logs for apache httpd are access_log and error_log, these log the interaction with the user <-> httpd process. The httpd.conf entries are :

 

- Access_log  - settings in httpd.conf :  

The formats are defined here : 

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

 

Occasionally it is good to supplement what is in the access_log and to get some insight to a problem.  The example above it shows the %{User-Agent}i header, but you can also use that to capture cookies for example:  \"%{SMSESSION}C\" .   The option %T is also useful, since it logs the total time it took Ag to process the request and return the response to the user. 

 

A complete list of LogFormat parameters is available here: 

https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#formats

 

Note: And one final point to remember, the the access_log entry is written at the END of processing the request.  So if apache httpd crashed then the requests that are currently in flight when the crash happened are NOT logged. 

 

- Error_log  - settings in httpd.conf :  

The formats are defined here : 

# LogLevel: values include: debug, info, notice, warn, error, crit, alert, emerg.
LogLevel warn

 

For debugging, you can raise the LogLevel to debug.   Apache 2.4 also has extra levels trace1 ... trace8, there are needed when you want to trace the raw data packets and SSL handshaking problems between the front end client and the httpd process. So for debugging often we can recommend : 

LogLevel trace8

 

The apache error_log is also good place to find the exact httpd and mod_jk version numbers: 

 

For more information refer: https://httpd.apache.org/docs/2.4/logs.html 

 

 

2. Web Agent Logs (WebAgent.log, WebAgentTrace.log) 

 

The Web Agent uses the following log files:

 

2.1 WebAgent.log 

 

Introduction:

Contains program and operational-level errors. One example is when the Web Agent cannot communicate with Policy Server. The level of detail output in this log cannot be customized. Error logs contain the following types of messages:

 

Error messages

Contain program-level errors, which indicate incorrect or abnormal program behavior, or an inability to function as expected due to some external problem, such as a network failure. There are also operational-level errors.

This type of error is a failure that prevents the operation from succeeding, such as opening a file or authenticating a user.

 

Informational messages

Contain messages for the user or administrator that some event has occurred; that is, that a server has started or stopped, or that some action has been taken.

 

Warning messages

Contain warnings for the user or administrator of some condition or event that is unusual or indicative of a potential problem. This does not necessarily mean there is anything wrong.

 

How to enable WebAgent log:

1.Open Agent Configuration Object or local configuration file.

2.Set the value of the LogFile parameter to yes.

3.Specify the full path to the error file, including the file name, in below parameter:

LogFileName

Specifies the full path (including the file name) of the log file.

Default: No

Example: (Windows) web_agent_home\log\WebAgent.log

Example: (UNIX/LInux) /export/iPlanet/servers/https-jsmith/logs/WebAgent.log

Below parameters are optional:

LogAppend

Adds new log information to the end of an existing log file. When this parameter is set to no, the entire log file is rewritten each time logging is invoked.

Default: No

LogFileSize

Specifies the size limit of the log file in megabytes. When the current log file reaches this limit, a new log file is created.Archive or remove the old files manually.

Default: 0 (no rollover)

Example: 80

LogLocalTime

Specifies whether the logs use Greenwich Mean Time (GMT) or local time. To use GMT, change this setting to no. If this parameter does not exist, the default setting is used.

Default: Yes

 

If you use a local configuration file, your settings resemble the following example:

LogFile="yes"

LogFileName="/export/iPlanet/servers/https-myserver/logs/errors.log"

LogAppend="no"

LogFileSize="80"

LogLocalTime="yes"

LogFilesToKeep="20"

 

You can limit the number of log files that an agent keeps. For example, if you want to save disk space on the system that stores your agent logs, you can limit the number of log files using this parameter.

Specifies the number of agent log files that are kept. New log files are created in the following situations:

When the agent starts.

When the size limit of the log file (specified by the value of the LogFileSize parameter) is reached.

Changing the value of this parameter does not automatically delete any existing logs files which exceed the number that you want to keep. For example, If your system has 500 log files stored, and you decide to keep

only 50 of those files, the agent does not delete the other 450 files.

Setting the value of this parameter to zero retains all the log files. Default: 0

Follow these steps:

Archive or delete any existing log files from your system.

Set the value of the LogAppend parameter to no.

Change the value of the LogFilesToKeep parameter to the number of log files that you want to keep.

 

Webagent error log looks like below:

 

 

2.2 WebAgentTrace.log 

 

Trace log contains detailed warning and informational messages, which you can configure. Examples include trace messages and flow state messages.
This file also includes data such as header details and cookie variables. Trace logs contain the following messages:

Trace messages

Provide detailed information about program operation for tracing and/or debugging purposes. Trace messages are ordinarily turned off during normal operation. In contrast to informational, warning, and error messages, trace messages are embedded in the source code and cannot easily be localized. Moreover, trace messages may include significant data in addition to the message itself; for example, the name of the current user or realm.

Note: For Agents on Windows platforms, set the EnableWebAgent parameter to yes to ensure that the Web Agent log gets created. If you leave EnableWebAgent set to no (the default) and set the logging parameters, the Agent log gets created only for Agents on UNIX platforms.

We can specify the location of both the error and trace log files when you configure the Web Agent.

 

Webagent Trace log looks like below:

 

 

How to enable Webagent Trace log:

 

1.Open Agent Configuration Object or local configuration file.

2. Set the TraceFile parameter to yes.

3. Specify the full path to the trace log files in following parameters:

TraceFileName

Specifies the full path to the trace log file.

Default: No default

Limits: Specify the file name in this parameter. Example: web_agent_home\log\trace.log

4. Determine what you want to record in the trace log by reviewing Trace Log Components and Subcomponents, Trace Message Data Fields and Data Field Filters.

Duplicate the default Trace Configuration File. Modify the duplicate file to include the items you want to record. Specify the full path to the duplicate copies of WebAgentTrace.conf file in the following parameter:

TraceConfigFile

Specifies the location of the WebAgentTrace.conf configuration file that determines which components and events to monitor.

Default: No default

Example:web_agent_home\config\WebAgentTrace.conf

(see next section for more details of this file)

 

5. Define the format of the information in your trace log file by setting the following parameters in your Agent Configuration Object or local configuration file:

TraceAppend

Adds new logging information to the end of an existing log file instead of rewriting the entire file each time logging is invoked.

Default: No

TraceFormat

Specifies how the trace file displays the messages. Choose one of the following options:

default—uses square brackets [] to enclose the fields.

fixed—uses fields with a fixed width.

delim—uses a character of your choice to delimit the fields.

xml—uses XML-like tags. A DTD or style sheet is not provided with the Web Agent.

Default: default (square brackets)

TraceDelimiter

Specifies a custom character that separates the fields in the trace file.

Default: No default

Example: |

TraceFileSize

Specifies (in megabytes) the maximum size of a trace file. The Web Agent creates a new file when this limit is reached.

Default: 0 (a new log file is not created)

Example: 20 (MB)

LogLocalTime

Specifies whether the logs use Greenwich Mean Time (GMT) or local time. To use GMT, change this setting to no. If this parameter does not exist, the default setting is used.

Default: Yes

 

Agent should then pickup the changes in some time, as these are dynamic ACO attributes.  For some older agents a restart of the webagent may be required for the logging parameters to take effect. 

 

 

WebAgentTrace.conf settings Trace Log Components and Subcomponents:

 

The WebAgentTrace.conf file determines the content of the trace log. You can control which components and data items appear in your trace log by modifying the settings of the WebAgentTrace.conf file on your web server. The following factors apply when editing the file:

Entries are case-sensitive.

When you specify a component, data field, or filter, the values must match exactly the options in the WebAgentTrace.conf file instructions.

Uncomment the configuration settings lines.

If you modify the WebAgentTrace.conf file before installing a new agent over an existing agent, the file is overwritten. Rename or back up the file first. After the installation, you can integrate your changes into the new file.

 

The CA Single Sign-On Agent can monitor specific CA Single Sign-On components. When you monitor a component, all of the events for that component are recorded in the trace log. Each component has one or more subcomponents that the agent can also monitor. If you do not want the agent to record all of the events for a component, you can specify only those subcomponents you want to monitor instead.

For example, if you want to record only the single sign-on messages for an agent on a web server, you would specify the WebAgent component and the SSO subcomponent.

 

 

Notes: 

 

For components: - the default components are often enough, although the addiiton of AgentFunc and Agent_Con_Manager can be useful.

 

For data: - it is very useful to add PreciseTime, SrcFie, Function, as well as ensuring the Date, Pid, Tid, TransactionID and Messsage eg: 

      data: Date, PreciseTime, Pid, Tid, TransactionID, Function, SrcFile, Message

 

 

The following components and subcomponents are available:

AgentFramework

Records all Agent framework messages. (Applies only to framework agents.) The following subcomponents are available:

Administration,Filter,HighLevelAgent,LowLevelAgent,LowLevelAgentWP

AffiliateAgent

Records web Agent messages related to the 4.x Affiliate Agent, which is part of Federation Security Services, a separately-purchased product. (Applies only to framework agents.) The following subcomponent is available: RequestProcessing

SAMLAgent

Web Agent messages related to the SAML Affiliate Agent. (Applies only to framework agents.) The following subcomponent is available: RequestProcessing

WebAgent

Records all Web Agent log messages. Applies to all Agents except IIS 6.0 or Apache 2.0 Agents. The following subcomponents are available: AgentCore,Cache,authentication,Responses,Management,SSO,Filter

Agent_Functions

Records all Agent API messages. The following subcomponents are available:

Init,UnInit,IsProtected,Login,ChangePassword,Validate,Logout,Authorize,Audit,FreeAttributes,UpdateAttributes,GetSessionVariables,SetSessionVariables,DeleteSessionVariables,Tunnel,GetConfig,DoManagement

Agent_Con_Manager

Records messages related to internal processing of the Agent API. The following subcomponents are available:

RequestHandler,Cluster,Server,WaitQueue,Management,Statistics

 

WebAgentTrace.conf Data Fields:

 The following data fields are available:

Message: Includes the actual trace message

SrcFile: Includes the source file and line number of the trace message

Pid: Includes the process ID

Tid: Includes the thread ID

Date: Includes the date

Time: Includes the time

PreciseTime: Includes the time, including milliseconds

Function: Includes the function in the code containing the trace message

User: Includes the name of the user

Domain: Includes the CA Single Sign-On domain

Realm: Includes the CA Single Sign-On realm

AgentName: Includes the Agent name being used

TransactionID: Includes the transaction ID

DomainOID: Includes the CA Single Sign-On domain OID

IPAddr: Includes the client IP address

RequestIPAddr: Includes the trace file displays the IP of the server where Agent is present

IPPort: Includes the client IP port

CertSerial: Includes the certificate serial number

SubjectDN: Includes the subject DN of the certificate

IssuerDN: Includes the Issuer DN of the certificate

SessionSpec: Includes the CA Single Sign-On session spec

SessionID: Includes the CA Single Sign-On session ID

UserDN: Includes the User DN

Resource: Includes the requested resource

Action: Includes the requested action

RealmOID: Includes the realm OID

ResponseTime: Includes the average response time in milliseconds of the Policy Servers associated with a CA Web Agent or SDK Agent and API application

 

Note:To output the ResponseTime to a trace log, include the component Agent_Con_Manager along with the data field ResponseTime in the WebAgentTrace.conf file or other file specified in the Policy Server Configuration Object (ACO) and restart the Policy Server. The Agent_Con_Manager component, or Agent API Connection Manager, calculates the ResponseTime each time a response is received from a Policy Server and keeps a running average. To locate the ResponseTime in the trace log, search for [PrintStats].

 

Some data fields are shown in the following example:

data:message,date,time,user,agentname,IPAddr

There may not be data for fields in each message, so blank fields my occur. For example, if you select RealmOID as a data field, some trace messages will display the realm's OID while others will not.

 

WebAgentTrace.conf Data Field Filters:

To focus on a specific problem, you can narrow the output of the trace log by specifying a filter based on the value of a data field. For example, if you are having problems with an index.html page, you can filter on resources with an html suffix by specifying Resource:==/html in the trace configuration file. Each filter should be on a separate line in the file.

Filters use the following syntax:

data_field:filter

The following types of filters are available:

== (exact match)

!= (does not equal)

The filters use boolean logic as shown in the following examples:

Action:!=get (all actions except get)

Resource:==/html (all resources ending in /html)

 

Determine the Content of the Trace Log:

 

Limit the Number of Trace Log Files Saved:

You can limit the number of trace logs that a CA Single Sign-On agent keeps. For example, if you want to save disk space on the system that stores your agent logs, you can limit the number of trace logs using the following parameter:

TraceFilesToKeep

Specifies the number of CA Single Sign-On agent trace log files that are kept. New trace logs are created in the following situations:

When the agent starts.

When the size limit of the trace log (specified by the value of the TraceFileSize parameter) is reached.

Changing the value of this parameter does not automatically delete any existing trace logs which exceed the number that you want to keep. For example, If your system has 500 trace logs stored, and you decide to keep only 50 of those files, the agent does not delete the other 450 trace logs.

Setting the value of this parameter to zero retains all the trace logs.

Default: 0

Follow these steps:

Archive or delete any existing trace logs from your system.

Set the value of the TraceAppend parameter to no.

Change the value of the TraceFilesToKeep parameter to the number of trace logs that you want to keep.

 

3. How to Enable Transport Layer Interface (TLI) Logging:

When you want to examine the connections between the agent and the Policy Server, enable transport layer interface logging.

1.Add the following environment variable to your web server.

 SM_TLI_LOG_FILE

2.Specify a directory and log file name for the value of the variable, as shown in the following example:

 directory_name/log_file_name.log

3.Verify that your agent is enabled.

4.Restart your web server. TLI logging is enabled.

<Picture coming>

4. Other useful Logs and Tools for Troubleshooting:

 

4.1 Fiddler:
Fiddler is an HTTP debugging proxy server application,it captures HTTP and HTTPS traffic.
To Enable HTTPS traffic decryption:
Open Fiddler-> Tools-> Fiddler Options-> HTTPS -> Check ‘Decrypt HTTS traffic’->Ok
Download: http://www.telerik.com/fiddler 

 

4.2 Wireshark:
Wireshark is a free and open source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education.
Download: https://www.wireshark.org/ 

 

4.3 Netstat:
Netstat (network statistics) is a command-line network utility tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing),

routing tables,and a number of network interface (network interface controller or software-defined network interface) and network protocol statistics.

Refer: https://en.wikipedia.org/wiki/Netstat 

 

4.4 strace:
strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state. The operation of strace is made possible by the kernel feature known as ptrace.

Refer: https://en.wikipedia.org/wiki/Strace 

 

4.5 Process Monitor:
Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity.

Download: https://technet.microsoft.com/en-us/sysinternals/processmonitor.aspx

 

4.6 Event Viewer:
Event Viewer is a component of Microsoft's Windows NT line of operating systems that lets administrators and users view the event logs on a local or remote machine.

Refer: https://technet.microsoft.com/en-us/library/cc766042(v=ws.11).aspx

 

4.7 IIS Failed request tracing:

Refer: https://www.iis.net/learn/troubleshoot/using-failed-request-tracing 


 

Thanks - Santosh

Santosh Reddy Rikkula - Support Engineer - Global Customer Success

 

This document is part of a series on Logging in SSO components: 

Tech Tip:How to enable trace logging in SSO (aka Siteminder) Webagent 

Tech Tip : Policy Server Loggings 

Tech Tip : Howto enable Tracing in Access Gateway (fka: Secure Proxy Server) 

1 person found this helpful

Attachments

    Outcomes