DX NetOps

Hello everybody,

Yes, I have already opened an issue about this, but haven't had a proper answer and our Spectrum system is slowing to almost uselessness, so in parallel with the CA Support issue I'm asking the community if anyone else has seen this.

I upgraded our Spectrum 9.3.3 system to 10.1 on May 1. (Windows 2008 R2, physical server, 16 cores, 24GB memory.) Since then the number of models has more than doubled, and the Spectroserver has spiked at 100% CPU and is at 10GB of memory and growing. I've tried a full system restart, without luck. I've tried stopping both our OneClicks, but the Spectroserver didn't drop usage at all.

There's no significant information in any logs, and since the Spectroserver process is monolithic I can't tell what component or components are giving me trouble.

I did find a reported problem with Junipers, fixed in 10.1.1, but even after putting all our Juniper devices into maintenance nothing changed.

Has anyone else encountered a problem like this, and if so, have you fixed it?

Thank you in advance to anyone one with any useful ideas.

Joe Poutre, BNP Paribas

Check under Chassis Manager->Juniper. See if you have any devices with an abnormally large sub model count (thousands).

Thank you lutelewis.

I had learned about the Junipers, but when I tried putting them all in Maintenance the problem didn't stop. As Jason Meader notes below it is the Juniper devices that are not discovered as Juniper devices but as GnSNMPDev that give Spectrum the worst agita. We have 6 such devices, which once put into maintenance resolved the problem.

Hi Joe,

   I took a look at the dump file you uploaded to your case and it looks like it may be deleting models.  I see in your notes of the case that you mention that the number of models is also increasing drastically.

I believe that this may be the same thing as reported by a couple of other users.  We have found an issue with Juniper Chassis devices constantly reconfiguring.  If the Juniper device is modeled as a GnSNMPDev when it reconfigures Spectrum also creates duplicate modules.  The constant reconfiguring consumes the cpu and the increase in models consumes the memory.

I was waiting for the completed fixes to be available to be able to post about them.   I am currently in the process of posting a tech doc as well about this (TEC1666174 – it isn’t published yet, it will be within the next day or so).

The fixes will be:

10.1.0 - Spectrum_10.01.00.PTF_10.0.025

110.1.1 - Spectrum_10.01.00.PTF_10.1.022

The 10.1.1 fix is ready and available.  The 10.1.0 fix should be ready very soon (possibly today).

Cheers

Jay

Sorry there was a typo and error in the 10.1.1 fix.  These are the correct patches:

10.1.0 - Spectrum_10.01.00.PTF_10.0.025

10.1.1 - Spectrum_10.01.01.PTF_10.1.109

Thank you Jason. I will ask for that patch for 10.1 once it is ready.

Regards,
Joe Poutre
BNP Paribas

Gotta love the timing  ☺  The patch testing completed while I was posting and we’re good to go.  I have uploaded the Spectrum_10.01.00.PTF_10.1.025 to your case for you.

If anyone else needs these patches, please open a case with us and request them (Since they are PTF patches we need to provide them through a case).

Thanks!!

Jay

And if anybody would like the tech doc reference (TEC1666174):

http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1666174.aspx

Thanks again

Jay

It's been a little while since the fix for this issue has been posted, and I'd like to update everyone on this.  If you are monitoring Juniper devices and will be installing either 10.1 or 10.1.1, please open a support case and install the appropriate patch to make sure you do not experience this.  We have seen at a few sites where there have been hundreds of thousands of duplicated modules created causing major SS performance issues.  It would be in everyone's best interest to get the patch installed.  The fix is tentatively scheduled to be included in 10.2.

If you are already on 10.1 or 10.1.1 and have Juniper models, you can use CLI to see if you have hundreds/thousands of duplicate modules.  From Cli run:

./show models | grep JuniperSlot > JuniperSlot_models.txt

Review the txt output and see if you have hundreds/thousands of duplicate modules.  If you do you'll need the patch. I would suggest if you have this problem that you open a case to work with our CA Support team (you'll need to install the patch and then use SSdebug to destroy the duplicates as the bug is causing a looping issue that makes it difficult for OC to process the work/relation activity). 

Cheers

Jay

Hi,

   I'd like to mention that if you have upgraded your version of Spectrum recently (possibly due to the java certificate update needed) that if you have Juniper devices you may run into this problem.  If you previously ran into this and have since installed 10.1.2, you will need a new patch for 10.1.2 (10.1.2 - Spectrum_10.01.02.PTF_10.1.205).

As noted in a previous post, if you are unsure of whether or not you are experiencing this problem, please check the following:

SS cpu will increase

SS memory utilization will increase

Events on the Juniper model will show it is being reconfigured (every minute)

JuniperSlot modules will be duplicated - use CLI --

./show models | grep JuniperSlot > JuniperSlot_models.txt

Review the txt output and see if you have many duplicate modules.  If you do you'll need the patch.

I have updated the tech doc to note this happens with 10.1.2 as well.  If you are experiencing this problem, please open a case as we'll need to provide you the patch and work with you to destroy the duplicate modules.

http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1666174.aspx

Cheers

Jay