CA Tuesday Tip: Private Key Overview

Discussion created by Hallett_German Employee on Dec 8, 2012
Latest reply on Oct 22, 2013 by adan_85
CA Wily Tuesday Tip by Hallett German, Sr. Support Engineer for 12/11/2012

Alex Wilkens is a new APM CE Administrator. Her company, Gulch Gears needs to monitor a new web ecommerce application that is important
for company revenue. The definitions are set up and the application starts to be monitored. But the reports and TIM logs are empty. After checking that the network setup is correct, Alex is dumbfounded.

She calls her handy CA Technologies support number. Quickly, they determine that the private keys need to be added to the system. Once done, the TIM logs are filled with HTTP requests and responses and defects/reports are appearing. All is well in the world.

This Tuesday Tip will give an overview of private keys, common issues, and troubleshooting techniques.

Ecommerce applications need to be encrypted to transfer financial transactions in a secure fashion. These use Transport Layer Security (TLS) or Secure Sockets Layer (SSL) protocols for this transmission. This includes using a private key to decode the traffic. The private key may use a variety of formats and cipher suites. APM works with private keys that are in PEM (Privacy-enhanced Electronic Mail) format. Server keys on Apache and Open SSL-supported Web servers are already in this format. Private keys for other web servers need to be converted using the OpenSSL Toolkit to the pkcs12 format (which contains the certificate, public, and private keys). Once done, simply pull out the private key part. This usually starts with BEGIN RSA PRIVATE KEY and terminates with END RSA PRIVATE KEY.

In the APM Configuration and Guide, there are directions to create and obtain private keys for Apache or OpenSSL-supported, Microsoft IIS, and Sun/Netscape web servers. However, this approach works for other servers such as Tomcat and IBM.

A variety of SSL cipher suites are supported: DES, Triple DES, RC4, RC2, and AES. Since 9.0.8, Diffie-Hellman cipheres are recognized but not decoded. There are many web pages that can provide the SSL ciphers being used by publically-facing web servers.

Importing and Migrating Private Keys
There are two ways to import private keys into APM.

Prior to 9.1.2, the only approach was to use the TESS/MOM Administration GUI. This can be found under Setup > HTTPS Settings. Once there, you can enter one or more private keys. Each private key is uniquely identified by IP v4/v6 address or address range and port. It is encrypted and placed on ALL TIMs in the /etc/wily/cem/tim/config/webservers. (Whether the TIM sees traffic from that web server or not.) Private keys are stored with names such as

Note that the GUI only allows one to add a key, update a key (which deletes the old key) or delete all keys. There is not an option to delete one key. (However, the work around is you may go to the above directory and delete a key. You can also move the key to another TIM (on TIM restart, it should be picked up.)

The second approach is documented in APM 9.1.2 Release Notes. A script may used to import multiple keys. This can be on the TIM in /etc/wily/cem/tim/scripts. Then run sslkey_upload.sh with the appropriate arguments.

Troubleshooting technniques and Tools
Some tools and techniques used in debugging APM CE issues are
- Looking at the TIM logs with SSL connections TIM Trace Option is enabled. If you see and open without a session close, there is likely a private key or a network issue.
- Looking at SSL Sessions and see how many were successful. Since 9.0.8 and later, the number of unsupported cipher session are also shown.
- Installing SSLDump to see which web servers are or are not decoding. It can be found at http://www.rtfm.com/ssldump/.
- Checking if the modulus of the SSL certificate and the private key match.

Common SSL issues
Some common SSL issues are:
- Not seeing any SSL sessions at all. This is likely to be the private key or that something is filtering out traffic from the web server such as private keys or the span/tap.
- SSL Decode failures near 100%. This is likely to be the private key. Things to check are if the correct key is used, if the key is still valid, the key has the wrong passphrase, the key entered has extra characters at end of the file. This can also be a sign that the network traffic has lot of out of order packets and other issues.
- Lots of SSL Sessions aging out. This can be changing these TIM settings -- ConnectionTimeoutInSeconds,SslSessionAgeOutCount, and SslSessionAgeOutSeconds

These are the discussion questions for this article:
1. Are you using SSL web applications?
2. Do you have a scalable approach for private key generation and maintenance for APM CE.
3. What APM CE features would you like to see for private keys?