DX Application Performance Management

We have been seeing a slow down in our test systems workstation performance in both WorkStation and WebView but were able to mitigate this by deleting the datadependency maps (normally in the <EM HOME>/data) and restarting the collectors. Since we thought we knew how to manage the situation we decided to put it into a small Prooduction system.

The usage in the TEST system is mostly Perfomance Testing and people are watching live data in the investigator or console. In the production system people use it more dynamically trying to solve problems. This difference showed up our first day and with the issue that is being tracked in 21712255 we saw the workstation grind to a halt for all users.

CA has identified the issue and is working on a fix but I thought I would save someone from upgrading right now and then having to go back to 9.1 version like we did...

 

Steve

I haven't had a chance to look at the ticket, but have you tried 9.5.0 or 9.5.1? Does the problem exist there also?

We have customers running on 9.5.0 that don't have this problem.

Yes, I had 9.5.0 in my TEST Cluster (this is very heavy loaded with 1200+ agents on 5 collectors that are mixture 8.x and 9.1.x agents) and set up ABA. However we had some issues there so I upgraded to 9.5.1 (which broke my ABA testing). So I was waiting for 9.5.2....

From what I understand is that the Workstations are constantly asking for the DataDependency maps which forces the MoM to get it from the Collectors. It works OK with one or two users but slows to a crawl to finally not responding as more users are added. From what I understand, this behavior was introduced in 9.5.2....

We have been having problems with the Data Dependency maps since our very first 9.1 implementation..

Techhead49:

Yes, I had 9.5.0 in my TEST Cluster (this is very heavy loaded with 1200+ agents on 5 collectors that are mixture 8.x and 9.1.x agents) and set up ABA. However we had some issues there so I upgraded to 9.5.1 (which broke my ABA testing). So I was waiting for 9.5.2....

From what I understand is that the Workstations are constantly asking for the DataDependency maps which forces the MoM to get it from the Collectors. It works OK with one or two users but slows to a crawl to finally not responding as more users are added. From what I understand, this behavior was introduced in 9.5.2....

We have been having problems with the Data Dependency maps since our very first 9.1 implementation..

 

LynnWilliams:

Techhead49:

Yes, I had 9.5.0 in my TEST Cluster (this is very heavy loaded with 1200+ agents on 5 collectors that are mixture 8.x and 9.1.x agents) and set up ABA. However we had some issues there so I upgraded to 9.5.1 (which broke my ABA testing). So I was waiting for 9.5.2....

From what I understand is that the Workstations are constantly asking for the DataDependency maps which forces the MoM to get it from the Collectors. It works OK with one or two users but slows to a crawl to finally not responding as more users are added. From what I understand, this behavior was introduced in 9.5.2....

We have been having problems with the Data Dependency maps since our very first 9.1 implementation..

 

Hello,

There is a known problem with Workstation slowness/MOM memory usage when using a mix of 8.x & 9.x agents. This was resolved by a new EM property in 9.1.1.1 and later releases

 

com.wily.introscope.soa.dependencymap.normalizecallername.enable=true

 

Do you have that property enabled already on all Collectors? 

 

This KB article has further details: https://comm.support.ca.com/?legacyid=TEC598094

 

Hope this helps

We know about this since we were one of the first companies to report the the issue even though we have found that this is not completely solved. So we have always ran with this setting. This is a new issue and we have received a basic fix from CA that we are testing.
 

Starting with APM 9.5.1.0, you will need to deploy CRC EM in order to use ABA, which is why it broke. Please take a look the ABA guide and CSO about this new requirement.

Did you get much further with support about this yet?

The ABA interface was fixed in 9.5.2 (it even installs all the EM side so all you have to do is install the ABA server).

Yes, we have received a fix from CA and are testing it. I just need to get some logging done to send in to verify the fix.

So far it has made our 9.5.2 TEST environment run very well.

Hi Steve,

We are planning to upgrade our APM Environment from APM 9.5.0.0 to APM 9.5.2.0. You mentioned that CA has identified the issue you were seeing and was working on a fix for it.  How is that going ? Please let us know the status so that we can plan for our upgrade accordingly.

 

Thanks,

Sunil

Steve,

Thanks for the update.

Thanks,

Sunil

So far so good. I just recently sent in the workstation java console log in debug mode and it showed a lot less activity between the workstation and the MoM.

So far, this looks very promising. We will have a another meeting on the topic on Thursday and I beleive with the feedback given the the develope will be able to finalize the fix.

It is one jar file that needs to be replaced plus uping the Workstation Heap of the JNLP to 1024 min 1300 max (we still have a lot of workstations running 32 bit java.

Before moving to 9.5.2 I would check with your pre-sales folks and have them reference CA Support Request 21712255 01 - VERY SLOW WEB VIEW AND WORKSTA

At this point I would say that this is the fastest I have see 9.5 run on this cluster...

Steve,

Thank you very much for the update & the suggestion.

Thanks,

Sunil

Hi

I know this is marked as resolved, but as a follow up question, is the recommendation that we should take this patch to workstation.jar should be deployed as part of the 9.5.2 upgrade ? If so, how can we obtain that. ? We are planning to upgrade all live environments to 9.5.2 at the end of March.

Thanks

Dave,

We have a meeting today with CA that was set at a time so the developer that found the issue could be involved. I hope during this meeting that we find out how it will be packaged and once I know I will post it here.

We too are interested in getting back to 9.5.2 in our Production Clusters since there are a number of elements of the 9.5.2 release for which we were waiting. However, we cannot do that without some productized packaging.

That meeting is at 3:00PM Eastern.

Steve

Hi Steve,

 

What was the result of the meeting?

Hi - is there an update from CA for this issue ? We are looking at deploying 9.5.2 for production in a couple of weeks and we don't know if we need to get hold of a fix or not.

Thanks

Hi Dave,

Here is the info that I have about it:

In 9.1 changes to Investigator folders could trigger a workstation update notification from the EM.  This was changed in 9.5x so that this notification would occur if there was a change in the dependency map.  In the customers case, there were frequent updates or triggers for some reason in the dependency maps (SOA aka SPM), which meant the EM starting spamming the WSs and stopped responding well to requests….

Sorry I did not get back to the group after the meeting but at the end of the meeting it was indicated that they would work on a formal fix (tested and insured that it works into the release cycle). I did not hear back from them until today when I finally asked for a status (based on your query). Here is the answer:

Hi Steve,

Anil is currently testing the hotfix.  You should receive it in a couple of days.

Thanks,

So far the temporary fix they gave us has been working with no issues.

I will let the group know when I receive any communication on the hotfix.

 

Steve

All

I am little bit uncertain as to how to proceed with the 9.5 upgrade at the moment

We are ready to go in production with the 9.5.2 deployment, which is already in test. The upgrade is lined up for Saturday 22nd.

I can see that there is a 9.5.5 available on the download site now.

Do we have a clear statement on what was resolved between the 9.5.2 and 9.5.5 release ? Is the workstation bug fully fixed in 9.5.5 or are there still workarounds needed ?

Is there a way to get the 9.5.2 fix without putting in 9.5.5 - i.e is there a supported hotfix available ?

 

Thanks

dave.richards:

All

I am little bit uncertain as to how to proceed with the 9.5 upgrade at the moment

We are ready to go in production with the 9.5.2 deployment, which is already in test. The upgrade is lined up for Saturday 22nd.

I can see that there is a 9.5.5 available on the download site now.

Do we have a clear statement on what was resolved between the 9.5.2 and 9.5.5 release ? Is the workstation bug fully fixed in 9.5.5 or are there still workarounds needed ?

Is there a way to get the 9.5.2 fix without putting in 9.5.5 - i.e is there a supported hotfix available ?

 

Thanks

Update - I can see that 9.5.5 download is still only for the agents and not the EMs. So I am guessing that at the moment it will have to be 9.5.2 EM install + a hot fix in order to resolve ? Is that correct ?

 

Hi Dave,

The 9.5.5 release is a Java Agent only release that brings full support for Java 7 JREs. The 9.5.5 Agents are compatible with EMs at version 9.5.0 and above.

Workstation issues are not entirely fixed in 9.5.2 (i.e: Workstation should start with Java 7 update 45, but not update 51 and above). I believe 9.5.3 should fix everything, 9.5.3 is planned for March 31st.

The 9.5.5 release notes document will give you a complete list of the fixed defects.

Best regards,

Florian.

Thanks Florian

Are CA recommending we do not deploy 9.5.2 to production and wait intil 9.5.3, or is there a hotfix available?

I have a support case open at the moment and I have been told the file we need to replace is - com.wily.introscope.soa.dependencymap.ui_9.5.0.jar

We can see that this is installed on out 9.5.2 environment in test.

 

I am less worried about the webstart startup issues (re: update 45 as we have a workaround for that using the older JRE), but the issue identified at the start of this thread regarding workstation slowdown concerns us as we have a similar mix of 8.0 and 9.* agents and we have SOA enabled, so I think we could be at risk.

 

Dave,

If the WebStart Workstation is not an issue in your environment, then I think you should be safe using 9.5.2 along with the hotfix for SOA dependency map poor performance.


I did read your support ticket, and it seemed to me that you had properly tested the fix and that the performance of your environment was back to normal.

Regards,

Florian.


From: CA APM/Wily Global User CommunityMessage Boards [mailto:CommunityAdmin@communities-mail.ca.com]
Sent: mercredi 19 mars 2014 10:22
To: mb_message.2307983.111365524@myca-email.ca.com
Subject: [CA APM General Discussion] RE: [CA APM General Discussion] RE: APM 9.5.2 issue


Thanks Florian

Are CA recommending we do not deploy 9.5.2 to production and wait intil 9.5.3, or is there a hotfix available?

I have a support case open at the moment and I have been told the file we need to replace is - com.wily.introscope.soa.dependencymap.ui_9.5.0.jar

We can see that this is installed on out 9.5.2 environment in test.



I am less worried about the webstart startup issues (re: update 45 as we have a workaround for that using the older JRE), but the issue identified at the start of this thread regarding workstation slowdown concerns us as we have a similar mix of 8.0 and 9.* agents and we have SOA enabled, so I think we could be at risk.


Posted by:dave.richards
--
CA Communities Message Boards
111368064
mb_message.2307983.111365524@myca-email.ca.com<mailto:mb_message.2307983.111365524@myca-email.ca.com>
https://communities.ca.com

Thanks Florian, I think we may have some crossed wires here.

At this moment we haven't received or deployed the fix. We are waiting for a copy of the software, but it hasn't been made available to us yet.

We have a 9.5.2 vanilla deoployment in test (no fix applied), and a 9.1.4 build in production.

 

With regard to the webstart issue, we have 2 workarounds at the moemnt - supply the client workstation on request to them, or ask the webstart users to use an older JRE.

 

Sorry I did not get back you you on the final fix for this.

We have been given a hotfix (2nd one I think more formal for distribution) that we are currently testing with good results and at this point I have closed my support ticket.

This was provided to me thru the ticket FTP and I don't think I am supposed to post it here.

You should be able to start a ticket of your own referencing my ticket (CA Support Request 21712255 01) and have the fix supplied to you.

I am planning on putting this fix into the Production Cluster where we first found this issue next weekend.

Steve