CA Tuesday Tip: Dude, Where is my APM CE Defect/Report?

Discussion created by Hallett_German Employee on Jul 13, 2013
CA Wily Tuesday Tip by Hallett German, Principal Support Engineer for 7/13/2013

Sometimes, defects or reports will not show up. This Tuesday Tip gives you some suggestions on what to check before contacting CA Technologies APM Support.
Step 1: Triage and Problem Determination

First thing is to discover the extent of the problem and which APM and underlying infrastructure components are impacted.
1)Is it something simple as the span port/network tap is misconfigured? So traffic is being filtered out, or two way traffic is not set up. Traffic could also be inadvertently with hardware filters (MTP) or web server filters.
2) Are there issues with private keys so the traffic cannot be decoded? (Previously discussed in an earlier Tuesday Tip)
3) Are files backing up on the TIM?
4) Are the definitions still valid? Check the Save to CSV option under Business Services to see when a transaction last received defects
5) Try to understand the extent of the problem:
- Using Analysis Graphs, figure out:
The date of the last defect or report. Also the scope of the issue for reports (Is it for certain date ranges, Is it for a particular
time frequency such as hourly.)
- See which reports are impacted. (All or just one type of report.)
- Also look for "disappearance patterns" such as hourly data is showing up but not daily. This would show up in reports as data today is showing up but not yesterday.

Step 2: Fixing the issue
Second, research the likely cause of the issue:
1) Check if you have network traffic and private key are valid
- Use tcpdump to check if getting traffic from the desired web servers.
- Check SSL statistics to see if SSL sessions are decoding successfully
2) See if defect and stat files are even being generated at all
- Check the timfiles logs on the TIM and TIM Collector EM Logs to see if defects and stat files are being generated at all. .
3) Defects and Stat Files are being generated but not showing up in APM CEM GUI.
- See if defect and stat files are backing up under /etc/wily/cem/tim/data/out. Look in the defects and stats subdirectories to see how many files are there. (Defects should be processed real-time and have only a few files for a couple of seconds. Only one stats file should typically be present for the previous hour. If files are backed up restart the Stats Aggregation EM.
- Run the appropriate SQL query to check the Last Aggregation Row value in the appropriate stats base table (Such as those in the daily table select max(ts_last_aggregated_row) from ts_st_ts_us_dly;)
- Look for errors after midnight when the stats aggregation has taken place such as
org.postgresql.util.PSQLException: ERROR: relation "ts_st_ts_us_int_xxxxxxxx" or Bad Row
- Note the version that is installed. Check the release notes of later versions (Such as 9.1.2 and higher)
Once having this information and having done the above, please contact Support on next steps.

These are the discussion questions for this article:
1. Are there other techniques that you use when having report issues?
2. Would a way to track files backing up on /etc/wily/cem/data/out be helpful?
3. Which topics would you like to see covered in the future?