CA Service Management

Hi,

Currently got an issue in which we have been experiencing High CPU usage on our secondary servers, with Task Manager reporting service desk processes consuming the CPU.  This is effecting the performance of the application.  We have noticed in the stdlog files the following error messages on both secondary servers.

4988 ERROR        producer.c            5048 Failed attempt to get dob atev:245635094 named in insert notifications

As soon as one of our secondaries stopped generating this error, the CPU consumed dropped, however the other is still generating this error.  We have restarted the application and it is still there.  Does anyone have an idea as to what the message means?

Simon, it says when the attached event time is up, SDM virtdb can't get the atev object on time and then sdm would push it back to anima table. The reason for this could be you have too many records in anima table, or you have some DB difficulty(network, DBMS etc). The thing is, once this happens SDM would need to write to anima table prepare for the next read which will compound to make it worse. You would need to investigate...network connection to DB, DBMS itself, why so many records in anima table if it has too many, etc. If you think you would need a Support case for this, please open one. Thanks _Chi

Thanks Chi, the message we are seeing is referencing an event that doesn't exist, and we have the database and app on the same server so have ruled out network being an issue.

Also the anima process runs on the primary server, yet we're seeing the error messages on the secondary servers.

Hello Simon, please open a case with support for this.

Thank you,

Richard

Alas Richard, this is in our 12.6 environment.

Simon...........

Oh boy - as I am sure you are aware, 12.6 has been end of support for some time.

Do you have any plans on upgrading to a supported release of SDM?

Not my decision i'm afraid....

So we found out what was causing the error message.  We had a service logging closed tickets into the system through an API connection.  Turning this off stopped the error messages from appearing.  However this hasn't fixed out high cpu usage issue on our secondary server. So we have a primary and two secondaries.  both secondaries are identical in every way.  When the application is running the CPU on our #2 secondary is constantly high and constantly using the domsrvr process for sustainable periods of time during business hours.  The #1 secondary isn't doing this at all.  Use experience when logged in via #2 is slow performance.  Anyone got any ideas?  The hardware nor configuration of the application has not change for nearly a year now.

did you check the usage of those 2 respective secondaries?

How do they are balanced? Maybe all sessions going to the same may cause this high usage vs. the other.

As the previous error eventually show you may have too much events to be process or eventually a corrupted anima table.

Did you modified any SLA and/or event recently??

One think you can do is to create another domsrv dedicated for the animator.

Below a copy of old instruction from CA on how to do it. m(newer version have a different procedure)

Add a new domsrvr:

The steps below are a sample, for more details see NX_ROOT/samples/pdmconf/pdm_edit_usage_noted.html
In a command prompt window:

 cd C:\Program Files\CA\Service Desk\samples\pdmconf

and type the command:
 pdm_perl pdm_edit.pl
 That will give the message below:
 Are you editing a pdm_startup file for an NT server? [Y|N] type:Y
 Would you like to continue? [Y|N]Y
 Creating pdm_startup1.dat with pdm_startup.dat Please enter selection: 1 ( for adding domsrvr)
 Again we will get choices: enter x to save. Now go to: C:\Program Files\CA\Service Desk\samples\pdmconf And enter:
 Copy pdm_startup.rmt "..\..\pdmconf\pdm_startup.tpl" (pdm_startup.rmt file should be copied to NX_ROOT\pdmconf\pdm_startup.tpl)

Now run pdm_configure, enter the appropriate choices and do not select "Load Default Data" when prompted.

After the Configuration wizard completes, and the Service Desk Daemon Manager has started, issue the following commands to confirm:
 # now when you will do slstat | find "dom" you should see the domsrvr that was just added:

 domsrvr:01 39 48 Tue May 23 22:54:27 2006

 domsrvr:02 34 47 Tue May 23 22:54:27 2006

Next, we will point the Animator to the newly added dedicated domsrvr: Locate the following two files under C:\Program Files\CA\Service Desk\pdmconf

 i

 pdm_startup.i.tpl

In both of these files
 Search for #define ANIMATOR_NXD(_TAG) and edit the command to:

 command = "$NX_ROOT/bin/animator_nxd -d 'domsrvr:02'"

Here domsrvr:02 is the name of the domsrvr which we got from slstat

Restart the Service Desk Daemon Manager this will complete the process of adding a dedicate domsrvr for Animator.


ADD a DB agent for the anima table in the NX.env

 # NX_VIRTDB_AGENTn defines update agents that manage

 # all updates for a particular table(s).

# These agents are not part of $NX_MAX_DBAGENT count

@NX_VIRTDB_AGENT1=Call_Req

@NX_VIRTDB_AGENT2=Act_Log

@NX_VIRTDB_AGENT3=Change_Request

@NX_VIRTDB_AGENT4=Change_Act_Log

@NX_VIRTDB_AGENT5=Issue

@NX_VIRTDB_AGENT6=Issue_Act_Log

@NX_VIRTDB_AGENT7=Audit_Log, event_log, session_log

@NX_VIRTDB_AGENT8=EBR_FULLTEXT, EBR_FULLTEXT_ADM, BU_TRANS

@NX_VIRTDB_AGENT9=CREATE_INDEX

@NX_VIRTDB_AGENT10=Animator

Save the file and recycle the Service Desk pdm_daemon_manager.

Hope this help.
/J

We've found the issue.  Despite saying there was nothing wrong, we can put it down to the blade that the virtual machine was seated on.  Once we managed to convince them to migrate it to another blade, the CPU dropped like a stone.