AnsweredAssumed Answered

APM 9.5.2 issue

Question asked by Steve_Oak on Feb 21, 2014
Latest reply on Mar 29, 2014 by Steve_Oak

Like • Show 0 Likes0
Comment • 23

We have been seeing a slow down in our test systems workstation performance in both WorkStation and WebView but were able to mitigate this by deleting the datadependency maps (normally in the <EM HOME>/data) and restarting the collectors. Since we thought we knew how to manage the situation we decided to put it into a small Prooduction system.

The usage in the TEST system is mostly Perfomance Testing and people are watching live data in the investigator or console. In the production system people use it more dynamically trying to solve problems. This difference showed up our first day and with the issue that is being tracked in 21712255 we saw the workstation grind to a halt for all users.

CA has identified the issue and is working on a fix but I thought I would save someone from upgrading right now and then having to go back to 9.1 version like we did...

Steve

Steve_Oak Feb 26, 2014 12:54 AM

Correct Answer

So far so good. I just recently sent in the workstation java console log in debug mode and it showed a lot less activity between the workstation and the MoM.

So far, this looks very promising. We will have a another meeting on the topic on Thursday and I beleive with the feedback given the the develope will be able to finalize the fix.

It is one jar file that needs to be replaced plus uping the Workstation Heap of the JNLP to 1024 min 1300 max (we still have a lot of workstations running 32 bit java.

Before moving to 9.5.2 I would check with your pre-sales folks and have them reference CA Support Request 21712255 01 - VERY SLOW WEB VIEW AND WORKSTA

At this point I would say that this is the fastest I have see 9.5 run on this cluster...

See the reply in context

No one else had this question

Outcomes

23 Replies

Hiko_Davis Feb 22, 2014 2:26 AM
Mark Correct
Correct Answer
I haven't had a chance to look at the ticket, but have you tried 9.5.0 or 9.5.1? Does the problem exist there also?

We have customers running on 9.5.0 that don't have this problem.
Like • Show 1 Like1
Actions
- Steve_Oak @ Hiko_Davis on Feb 22, 2014 3:53 AM
  Mark Correct
  Correct Answer
  Yes, I had 9.5.0 in my TEST Cluster (this is very heavy loaded with 1200+ agents on 5 collectors that are mixture 8.x and 9.1.x agents) and set up ABA. However we had some issues there so I upgraded to 9.5.1 (which broke my ABA testing). So I was waiting for 9.5.2....
  
  From what I understand is that the Workstations are constantly asking for the DataDependency maps which forces the MoM to get it from the Collectors. It works OK with one or two users but slows to a crawl to finally not responding as more users are added. From what I understand, this behavior was introduced in 9.5.2....
  
  We have been having problems with the Data Dependency maps since our very first 9.1 implementation..
  Like • Show 0 Likes0
  Actions
  - Lynn_Williams @ Steve_Oak on Feb 23, 2014 11:44 PM
    Mark Correct
    Correct Answer
    
    Techhead49:
    
    Yes, I had 9.5.0 in my TEST Cluster (this is very heavy loaded with 1200+ agents on 5 collectors that are mixture 8.x and 9.1.x agents) and set up ABA. However we had some issues there so I upgraded to 9.5.1 (which broke my ABA testing). So I was waiting for 9.5.2....
    
    From what I understand is that the Workstations are constantly asking for the DataDependency maps which forces the MoM to get it from the Collectors. It works OK with one or two users but slows to a crawl to finally not responding as more users are added. From what I understand, this behavior was introduced in 9.5.2....
    
    We have been having problems with the Data Dependency maps since our very first 9.1 implementation..
    
    Hello,
    
    There is a known problem with Workstation slowness/MOM memory usage when using a mix of 8.x & 9.x agents. This was resolved by a new EM property in 9.1.1.1 and later releases
    
    com.wily.introscope.soa.dependencymap.normalizecallername.enable=true
    
    Do you have that property enabled already on all Collectors?
    
    This KB article has further details: https://support.ca.com/irj/portal/kbtech?docid=598094&searchID=TEC598094
    
    Hope this helps
    Like • Show 2 Likes2
    Actions
    - Steve_Oak @ Lynn_Williams on Feb 25, 2014 8:18 PM
      Mark Correct
      Correct Answer
      
      LynnWilliams:
      
      Techhead49:
      
      Yes, I had 9.5.0 in my TEST Cluster (this is very heavy loaded with 1200+ agents on 5 collectors that are mixture 8.x and 9.1.x agents) and set up ABA. However we had some issues there so I upgraded to 9.5.1 (which broke my ABA testing). So I was waiting for 9.5.2....
      
      From what I understand is that the Workstations are constantly asking for the DataDependency maps which forces the MoM to get it from the Collectors. It works OK with one or two users but slows to a crawl to finally not responding as more users are added. From what I understand, this behavior was introduced in 9.5.2....
      
      We have been having problems with the Data Dependency maps since our very first 9.1 implementation..
      
      Hello,
      
      There is a known problem with Workstation slowness/MOM memory usage when using a mix of 8.x & 9.x agents. This was resolved by a new EM property in 9.1.1.1 and later releases
      
      com.wily.introscope.soa.dependencymap.normalizecallername.enable=true
      
      Do you have that property enabled already on all Collectors?
      
      This KB article has further details: https://support.ca.com/irj/portal/kbtech?docid=598094&searchID=TEC598094
      
      Hope this helps
      
      We know about this since we were one of the first companies to report the the issue even though we have found that this is not completely solved. So we have always ran with this setting. This is a new issue and we have received a basic fix from CA that we are testing.
      
      Like • Show 0 Likes0
      Actions
  - Hiko_Davis @ Steve_Oak on Feb 25, 2014 7:57 PM
    Mark Correct
    Correct Answer
    Starting with APM 9.5.1.0, you will need to deploy CRC EM in order to use ABA, which is why it broke. Please take a look the ABA guide and CSO about this new requirement.
    
    Did you get much further with support about this yet?
    Like • Show 0 Likes0
    Actions
    - Steve_Oak @ Hiko_Davis on Feb 25, 2014 8:21 PM
      Mark Correct
      Correct Answer
      The ABA interface was fixed in 9.5.2 (it even installs all the EM side so all you have to do is install the ABA server).
      
      Yes, we have received a fix from CA and are testing it. I just need to get some logging done to send in to verify the fix.
      
      So far it has made our 9.5.2 TEST environment run very well.
      Like • Show 1 Like1
      Actions
    - sunilhere @ Hiko_Davis on Feb 25, 2014 8:27 PM
      Mark Correct
      Correct Answer
      Hi Steve,
      
      We are planning to upgrade our APM Environment from APM 9.5.0.0 to APM 9.5.2.0. You mentioned that CA has identified the issue you were seeing and was working on a fix for it. How is that going ? Please let us know the status so that we can plan for our upgrade accordingly.
      
      Thanks,
      
      Sunil
      Like • Show 0 Likes0
      Actions
      - sunilhere @ sunilhere on Feb 25, 2014 8:33 PM
        Mark Correct
        Correct Answer
        Steve,
        
        Thanks for the update.
        
        Thanks,
        
        Sunil
        Like • Show 0 Likes0
        Actions
      - Steve_Oak @ sunilhere on Feb 26, 2014 12:54 AM
        Unmark Correct
        Correct Answer
        So far so good. I just recently sent in the workstation java console log in debug mode and it showed a lot less activity between the workstation and the MoM.
        
        So far, this looks very promising. We will have a another meeting on the topic on Thursday and I beleive with the feedback given the the develope will be able to finalize the fix.
        
        It is one jar file that needs to be replaced plus uping the Workstation Heap of the JNLP to 1024 min 1300 max (we still have a lot of workstations running 32 bit java.
        
        Before moving to 9.5.2 I would check with your pre-sales folks and have them reference CA Support Request 21712255 01 - VERY SLOW WEB VIEW AND WORKSTA
        
        At this point I would say that this is the fastest I have see 9.5 run on this cluster...
        Like • Show 1 Like1
        Actions
        sunilhere @ Steve_Oak on Feb 26, 2014 1:26 AM
        Mark Correct
        Correct Answer
        Steve,
        
        Thank you very much for the update & the suggestion.
        
        Thanks,
        
        Sunil
        Like • Show 0 Likes0
        Actions
        dave.richards @ sunilhere on Feb 27, 2014 9:28 AM
        Mark Correct
        Correct Answer
        Hi
        
        I know this is marked as resolved, but as a follow up question, is the recommendation that we should take this patch to workstation.jar should be deployed as part of the 9.5.2 upgrade ? If so, how can we obtain that. ? We are planning to upgrade all live environments to 9.5.2 at the end of March.
        
        Thanks
        Like • Show 0 Likes0
        Actions
        Steve_Oak @ dave.richards on Feb 27, 2014 4:38 PM
        Mark Correct
        Correct Answer
        Dave,
        
        We have a meeting today with CA that was set at a time so the developer that found the issue could be involved. I hope during this meeting that we find out how it will be packaged and once I know I will post it here.
        
        We too are interested in getting back to 9.5.2 in our Production Clusters since there are a number of elements of the 9.5.2 release for which we were waiting. However, we cannot do that without some productized packaging.
        
        That meeting is at 3:00PM Eastern.
        
        Steve
        Like • Show 0 Likes0
        Actions
        bam0382 @ Steve_Oak on Mar 5, 2014 2:21 PM
        Mark Correct
        Correct Answer
        Hi Steve,
        
        What was the result of the meeting?
        Like • Show 0 Likes0
        Actions
        dave.richards @ bam0382 on Mar 10, 2014 2:57 PM
        Mark Correct
        Correct Answer
        Hi - is there an update from CA for this issue ? We are looking at deploying 9.5.2 for production in a couple of weeks and we don't know if we need to get hold of a fix or not.
        
        Thanks
        Like • Show 0 Likes0
        Actions
        bam0382 @ dave.richards on Mar 10, 2014 3:08 PM
        Mark Correct
        Correct Answer
        Hi Dave,
        
        Here is the info that I have about it:
        
        In 9.1 changes to Investigator folders could trigger a workstation update notification from the EM. This was changed in 9.5x so that this notification would occur if there was a change in the dependency map. In the customers case, there were frequent updates or triggers for some reason in the dependency maps (SOA aka SPM), which meant the EM starting spamming the WSs and stopped responding well to requests….
        Like • Show 0 Likes0
        Actions
        Steve_Oak @ dave.richards on Mar 10, 2014 6:09 PM
        Mark Correct
        Correct Answer
        Sorry I did not get back to the group after the meeting but at the end of the meeting it was indicated that they would work on a formal fix (tested and insured that it works into the release cycle). I did not hear back from them until today when I finally asked for a status (based on your query). Here is the answer:
        
        Hi Steve,
        
        Anil is currently testing the hotfix. You should receive it in a couple of days.
        
        Thanks,
        
        So far the temporary fix they gave us has been working with no issues.
        
        I will let the group know when I receive any communication on the hotfix.
        
        Steve
        Like • Show 0 Likes0
        Actions
        dave.richards @ Steve_Oak on Mar 19, 2014 9:01 AM
        Mark Correct
        Correct Answer
        All
        
        I am little bit uncertain as to how to proceed with the 9.5 upgrade at the moment
        
        We are ready to go in production with the 9.5.2 deployment, which is already in test. The upgrade is lined up for Saturday 22nd.
        
        I can see that there is a 9.5.5 available on the download site now.
        
        Do we have a clear statement on what was resolved between the 9.5.2 and 9.5.5 release ? Is the workstation bug fully fixed in 9.5.5 or are there still workarounds needed ?
        
        Is there a way to get the 9.5.2 fix without putting in 9.5.5 - i.e is there a supported hotfix available ?
        
        Thanks
        Like • Show 0 Likes0
        Actions
        dave.richards @ dave.richards on Mar 19, 2014 9:10 AM
        Mark Correct
        Correct Answer
        
        dave.richards:
        
        All
        
        I am little bit uncertain as to how to proceed with the 9.5 upgrade at the moment
        
        We are ready to go in production with the 9.5.2 deployment, which is already in test. The upgrade is lined up for Saturday 22nd.
        
        I can see that there is a 9.5.5 available on the download site now.
        
        Do we have a clear statement on what was resolved between the 9.5.2 and 9.5.5 release ? Is the workstation bug fully fixed in 9.5.5 or are there still workarounds needed ?
        
        Is there a way to get the 9.5.2 fix without putting in 9.5.5 - i.e is there a supported hotfix available ?
        
        Thanks
        
        Update - I can see that 9.5.5 download is still only for the agents and not the EMs. So I am guessing that at the moment it will have to be 9.5.2 EM install + a hot fix in order to resolve ? Is that correct ?
        
        Like • Show 0 Likes0
        Actions
        Florian_Cheval @ dave.richards on Mar 19, 2014 9:14 AM
        Mark Correct
        Correct Answer
        Hi Dave,
        
        The 9.5.5 release is a Java Agent only release that brings full support for Java 7 JREs. The 9.5.5 Agents are compatible with EMs at version 9.5.0 and above.
        
        Workstation issues are not entirely fixed in 9.5.2 (i.e: Workstation should start with Java 7 update 45, but not update 51 and above). I believe 9.5.3 should fix everything, 9.5.3 is planned for March 31st.
        
        The 9.5.5 release notes document will give you a complete list of the fixed defects.
        
        Best regards,
        
        Florian.
        Like • Show 1 Like1
        Actions
        dave.richards @ Florian_Cheval on Mar 19, 2014 9:21 AM
        Mark Correct
        Correct Answer
        Thanks Florian
        
        Are CA recommending we do not deploy 9.5.2 to production and wait intil 9.5.3, or is there a hotfix available?
        
        I have a support case open at the moment and I have been told the file we need to replace is - com.wily.introscope.soa.dependencymap.ui_9.5.0.jar
        
        We can see that this is installed on out 9.5.2 environment in test.
        
        I am less worried about the webstart startup issues (re: update 45 as we have a workaround for that using the older JRE), but the issue identified at the start of this thread regarding workstation slowdown concerns us as we have a similar mix of 8.0 and 9.* agents and we have SOA enabled, so I think we could be at risk.
        
        Like • Show 0 Likes0
        Actions
        Florian_Cheval @ dave.richards on Mar 19, 2014 10:00 AM
        Mark Correct
        Correct Answer
        Dave,
        
        If the WebStart Workstation is not an issue in your environment, then I think you should be safe using 9.5.2 along with the hotfix for SOA dependency map poor performance.
        
        I did read your support ticket, and it seemed to me that you had properly tested the fix and that the performance of your environment was back to normal.
        
        Regards,
        
        Florian.
        
        From: CA APM/Wily Global User CommunityMessage Boards [mailto:CommunityAdmin@communities-mail.ca.com]
        Sent: mercredi 19 mars 2014 10:22
        To: mb_message.2307983.111365524@myca-email.ca.com
        Subject: [CA APM General Discussion] RE: [CA APM General Discussion] RE: APM 9.5.2 issue
        
        Thanks Florian
        
        Are CA recommending we do not deploy 9.5.2 to production and wait intil 9.5.3, or is there a hotfix available?
        
        I have a support case open at the moment and I have been told the file we need to replace is - com.wily.introscope.soa.dependencymap.ui_9.5.0.jar
        
        We can see that this is installed on out 9.5.2 environment in test.
        
        I am less worried about the webstart startup issues (re: update 45 as we have a workaround for that using the older JRE), but the issue identified at the start of this thread regarding workstation slowdown concerns us as we have a similar mix of 8.0 and 9.* agents and we have SOA enabled, so I think we could be at risk.
        
        Posted by:dave.richards
        --
        CA Communities Message Boards
        111368064
        mb_message.2307983.111365524@myca-email.ca.com<mailto:mb_message.2307983.111365524@myca-email.ca.com>
        https://communities.ca.com
        Like • Show 0 Likes0
        Actions
        dave.richards @ dave.richards on Mar 19, 2014 10:06 AM
        Mark Correct
        Correct Answer
        Thanks Florian, I think we may have some crossed wires here.
        
        At this moment we haven't received or deployed the fix. We are waiting for a copy of the software, but it hasn't been made available to us yet.
        
        We have a 9.5.2 vanilla deoployment in test (no fix applied), and a 9.1.4 build in production.
        
        With regard to the webstart issue, we have 2 workarounds at the moemnt - supply the client workstation on request to them, or ask the webstart users to use an older JRE.
        
        Like • Show 0 Likes0
        Actions
        Steve_Oak @ dave.richards on Mar 29, 2014 11:35 AM
        Mark Correct
        Correct Answer
        Sorry I did not get back you you on the final fix for this.
        
        We have been given a hotfix (2nd one I think more formal for distribution) that we are currently testing with good results and at this point I have closed my support ticket.
        
        This was provided to me thru the ticket FTP and I don't think I am supposed to post it here.
        
        You should be able to start a ticket of your own referencing my ticket (CA Support Request 21712255 01) and have the fix supplied to you.
        
        I am planning on putting this fix into the Production Cluster where we first found this issue next weekend.
        
        Steve
        Like • Show 0 Likes0
        Actions

APM 9.5.2 issue

Outcomes

Related Content

Recommended Content