DX Application Performance Management

CA Wily Tuesday Tip by Hallett German, Principal Support Engineer for 6/8/2013

The MTP or Multi-Port Monitor (formerly known as Multi-Port Collector) provides additional throughput capacity unimagined with a Timsoft. (By the way MTP is an acronym from (M)ul(T)i (P)ort.)

This Tuesday Tip provides a framework to help in determining if the issue is on the MTP or TIM side of the appliance.

[center] What isn't working ?[center]
Here are some common scenarios if the functionality is failing on the MTP side:

- The MTP is not collecting data.
The APM for IM Guide mentions that the Napatech Card Time is different than that of the Network Time Protocol Server. But there could be other scenarios:

* The Napatech card is misconfigured.
* The MTP network connection to a span/tap/aggregator/etc is incorrect.
* There were residual problems from the MTP install/upgrade.

- The MTP is collecting data but running out of space. Or the MTP is restarting frequently
This would be very evident from errors in the logs, seeing a non-zero value
in the nospace column of protocolstats logs, viewing the Ramdisk filespace at 100% by issuing a df command at the MTP console, etc.

The cause of this could be the RAM Disk being too small or the network traffic at a customer is filled with various data integrity issues (such as out of order packets, packet retransmissions) that can be resulting in large packet capture files.

Another cause is the ramdisk is adequate but inadequate hardware filters are in place. This would result in receiving more traffic than actually needed.

Note the recently released TIM Monitor Fieldpack can alert when ramdisk runs out of space as well as other error conditions. It is freely available with As-Is Support.

- The MTP is collecting data but not seeing traffic from a particular server.

The MTP appliance can do filtering at two locations:
The MTP hardware filter (Administration>Logical port) is filtering out traffic from particular ports, VLANs, or IP subnets.
The TESS Web Server filter does not include certain IP addresses so it is excluding traffic. With MTP appliances, it is recommended to do the filtering on the MTP side with the hardware filtesr rather than on the TIM side with the web server filters.

Here are some common scenarios that the data is failing on the TIM side:
- The MTP is collecting data but there are other issues.

As long as the data in the "Forwarded from MTP" column is a non-zero number, MTP is doing its job. Several Tuesday Tips have already been written on common isues on the TIM side. Things to check on MTPs would include:

- If there is SSL traffic, are the ciphers being used by an application supported? Is the private key correct? If bad traffic impacting SSL decoding?

- Are there issues with TIM Plug-ins (SiteMinder, XML, Flex. custom)causing extra load?

- Is the definition configured correctly on the TESS-MOM?

- Is the domainconfig.xml file on the TIM correct?

- Is the TIM communicating correctly with MOM, Collector and database? (Resulting in 4xx/5xx messages in the Events viewer)

These are the discussion questions for this article:
1. Has this been helpful as a first step in triaging MTP vs TIM issues?
2. What is the Ramdisk on your MTP currently set to? Is that the original value?
3. How has the inclusion of network data with APM CE defects helped you in improving application health and customer satisfaction?

Hi Gerha!

I had same problem about NO SPACE with the MTP.. I lost transactions in the CEM but I didnt have packets dropped and the RAM Disk was fine... I dont know what is the cause? where I get about more MTP informations?

Thank you.

Ah the no space issue . You have great timing. I ran into this and just submitted a knowledgebase article on this topic which should be up soon. Here's the essence

Short-term fix

- Reduce traffic into TIM (using hardware Filters)
- Increase RAMdisk size. (Which is 2 or 4 GB) out of the box. Note this is only a bandaid

The real issue is due to your span/tap/aggregator is sending traffic with quality issues. Investigate this with your network team. Duplicate acks and high out of order packets are creating larger packet sizes than the default RAMdisk can hold

I will try to remember to post the KB link when approved.

On the MTP, the MTP docs and the APM for Infrastructure Management are the best public guides on the topic.

Thanks for reading

Sincerely

Hallett German

This is the KB on this issue:
KB on nospace

Hi Hallett German!

Thank you so much! The articule is very interesting and useful.

You're right, the area allocated for the tmpfs file systems have been exceeded wtih 2 GB in a time interval of half an hour approximately is not space to write them out. It is the first time. This had never happened before or is wasn't common.

I suspect also that there was network problem at that time, so How I can show the problem or get the evidence? Is there a log to display problem of traffic?

P.D: I check in the Analysis Menu( MTP GUI) -> Network but I couldn't find high trend traffic.

Thank you!!!

Adan.

Adan:

I suspect also that there was network problem at that time, so How I can show the problem or get the evidence? Is there a log to display problem of traffic?

I am glad you found the KB useful. We are working on increasing our KB content with useful Tech Notes

Network problem I would start with the capd process/TIM logs to see what is being captured and dropped. A packet capture also gives you clues on quality of network traffic. If the issue continues, open a case if needed and we can take a look at it

Thanks
Hal German

Adan,
There is a very good plugin (you will need to request ur SE) to capture MTP statistics, this has already been discussed earlier in this forum, we use this plugin to review information pertaining to thru-put and dropped packets etc.

Apart from that as Hal mentioned, Logs are ur best bet. On the System Status page of your MTP; you will have a following information:

This is real time representation of how ur TIM ports are behaving.

Physical Port Logical Name Packets Received Bytes Received CRC/Align Errors Discarded Duplicates Receive Rate
All 924961760356 482.25 TB 0 431178566425 826.97 Mbps
0 TIM Ports 443161813101 237.77 TB 0 162671790059 380.12 Mbps
1 TIM Ports 481799947255 244.48 TB 0 268506776366 446.85 Mbps





Thanks
Ashish Vashisht
Principal Engineer APM
Enterprise Management Systems
EMC2
Mobile: 207-332-4813
eMail: Ashish.Vashisht@emc.com<mailto:Ashish.Vashisht@emc.com>




From: CA APM Global User Community <CommunityAdmin@communities-mail.ca.com<mailto:CommunityAdmin@communities-mail.ca.com>>
Reply-To: "mb.16496410.101928769@myca-email.ca.com<mailto:mb.16496410.101928769@myca-email.ca.com>" <mb.16496410.101928769@myca-email.ca.com<mailto:mb.16496410.101928769@myca-email.ca.com>>
Date: Wednesday, October 9, 2013 2:15 PM
To: "mb.16496410.101928769@myca-email.ca.com<mailto:mb.16496410.101928769@myca-email.ca.com>" <mb.16496410.101928769@myca-email.ca.com<mailto:mb.16496410.101928769@myca-email.ca.com>>
Subject: [Tuesday's Tips] RE: CA Tuesday Tip: Is the Issue on the MTP or TIM side?

Adan:

I suspect also that there was network problem at that time, so How I can show the problem or get the evidence? Is there a log to display problem of traffic?

I am glad you found the KB useful. We are working on increasing our KB content with useful Tech Notes

Network problem I would start with the capd process/TIM logs to see what is being captured and dropped. A packet capture also gives you clues on quality of network traffic. If the issue continues, open a case if needed and we can take a look at it

Thanks
Hal German
Posted by:gerha02
--
CA Communities Message Boards
101931309
mb.16496410.101928769@myca-email.ca.com<mailto:mb.16496410.101928769@myca-email.ca.com>
https://communities.ca.com

Thanks for your help Ashish and Hal German!!!!

Adan.

Hi Experts

Is their a way we can configure MTP to talk to ADA console on encrypted HTTPS connection.

OR

Can we change MTP web console to open on some other port than 80.

Regards:
GP