DX Application Performance Management

Document ID:  TEC600295

Last Modified Date:  10/9/2013
Author: Hallett_German

• Products
  • CA Application Performance Management
• Components
  • WILY CEM
  • CUSTOMER EXPERIENCE MANAGER

Description:This describes issues caused by the RAM Disk running out of space and what to do about it.
Solution:Looking at the protocolstats files stored at these locations:
/etc/wily/cem/tim/logs/protocolstats
/etc/wily/cem/tim/logs/protocolstats1There are non-zero values for the nospace column. This means that the MTP is dropping packets because there is no space to write them out. The nqnapacapd
should also have an error message showing this.The file system that these packets are written out is the RAMdisk of /nqtmp/tim (in MTP 2,2 and later) and /nqtmp/headers. The area allocated for these tmpfs file systems have been exceeded. By default, it is either 2 GB or 4 GB depending on MTP configuration.There are two fixes for this:

1. Short term strategies
   
   
   • Decrease level of traffic into MTP using hardware filters. (These are configured on the MTP GUI menu -- Administration> Logical Ports
   • Increase RAMdisk space:
     
     
     1. From the web UI, stop the tim and the nqcapd processes. (These are the two processes that access the /nqtmp/tim filesystem). Note: stopping nqcapd
        process is done on the MTP Administration->Processes page.
     2. Putty/ssh into Linux using netqos account and edit /etc/fstab. You will need to prefix the command with sudo to have proper permissions.
     3. Change the following line from 2G to 4G (or appropriate value):
        Old: tmpfs /nqtmp/tim tmpfs size=2G,nr_inodes=100k,mode=0700 0 0
        New: tmpfs /nqtmp/tim tmpfs size=4G,nr_inodes=100k,mode=0700 0 0
     4. Unmount and mount the filesystem:
        sudo umount /nqtmp/tim
        sudo mount /nqtmp/tim
     5. Confirm the new size using the df command
     6. From the web UI restart the TIM process and the nqcapd process.
        
        
2. Long term strategy -- Fix quality of network traffic
   
   The real issue is the quality of network traffic is causing oversized pcap files that reside in the RAMdisk filesystems. Factors impacting network traffic quality include but are not limited to:
   
   
   1. Overdriven Span Ports with high packet loss
   2. Span Port configuration issues such as a high rate of duplicate packets.
   3. Span Ports receiving intermittent asymmetric traffic or one-way traffic for a large number of sessions.
   4. Span Ports seeing the same TCP session from multiple perspectives (e.g. - Load Balancers).
   5. TCP Traffic that violates protocol conventions, such as ignoring the TCP Window size, frequent reuse of TCP Port numbers across multiple sessions, and abuse of TCP RST conventions.
   6. Clear TCP protocol violations such as packets with incorrect Sequence/Acknowledgement values.

Support Engineers may also suggest additional steps based on pcap analysis and after consulting the internal MTP Support documentation.

Search the Entire CA APM Knowledge Base