CA Service Management

  • 1.  CA Service Desk Manager Performance Problem.

    Posted Mar 08, 2019 08:26 AM
      |   view attached

    Dear Team,

     

    we are running CA SDM 14.1.3 in our organization. but the performance is ca sdm very slow while updating any incident or request tickets. i have also updates SREL_BLOCKS_TIMEOUT 10 and NX_EXPECTED_WEB_RESPONSE.

    but sitll we are getting slow response . when i looked in std.log file in nx root directory. we found this

     

    log file detail.................................................

    03/06 12:40:40.99 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 78908 milliseconds to complete. session id:872229884 login name:11002867 htmpl name:list_cr.htmpl
    03/06 12:40:41.01 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 78299 milliseconds to complete. session id:989654098 login name:11003259 htmpl name:list_cr.htmpl
    03/06 12:40:41.06 del-sdapp-clus spelsrvr 1064 ERROR heat_weather.spl 744 AHD05026:Error creating Activity Log Entry.
    03/06 12:40:41.09 del-sdapp-clus domsrvr 3104 ERROR attr.c 5627 No response to trigger cr.category::category_modified (1 time) after 30 seconds; waiting another 60 seconds
    03/06 12:40:41.09 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 18641 milliseconds to complete. session id:1371943458 login name: htmpl name:
    03/06 12:40:41.14 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 18251 milliseconds to complete. session id:605351945 login name: htmpl name:
    03/06 12:40:41.14 del-sdapp-clus domsrvr 3104 ERROR top_ob.c 4030 Got msg with type set to 0
    03/06 12:40:41.15 del-sdapp-clus spelsrvr 1064 ERROR heat_weather.spl 757 cr::update_ticket_heat_mappingserror during checkin 'AHD03075:Required attribute type is missing from object Request Activity Log'
    03/06 12:40:41.23 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 74659 milliseconds to complete. session id:48303248 login name: htmpl name:
    03/06 12:40:41.59 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 13407 milliseconds to complete. session id:1654953627 login name: htmpl name:
    03/06 12:40:41.90 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 10173 milliseconds to complete. session id:1985183586 login name: htmpl name:
    03/06 12:40:42.09 del-sdapp-clus domsrvr 3104 ERROR factory.c 6555 Cannot resolve multiple indirect references in affected_resource.zsite.name (factory in)
    03/06 12:40:42.23 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 46423 milliseconds to complete. session id:1369718941 login name: htmpl name:
    03/06 12:40:42.26 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 47564 milliseconds to complete. session id:989654098 login name:11003259 htmpl name:list_cr.htmpl
    03/06 12:40:42.34 del-sdapp-clus web:local 4812 SIGNIFICANT genhtml.c 806 Expanding value cache from 178 to 197 for list_cnt.htmpl[1] to accommodate tenant
    03/06 12:40:42.50 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 10940 Session 567446794:0x112BDFE8 login by analyst 11002869 (cnt:9E4A1DC18A62BC48A0B33F6B0034CD13); session count 99
    03/06 12:40:42.51 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 53705 milliseconds to complete. session id:0 login name:11002869 htmpl name:replace_login.htmpl
    03/06 12:40:42.53 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 6563 milliseconds to complete. session id:1632355777 login name:134222 htmpl name:list_cnt.htmpl
    03/06 12:40:42.53 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 34611 milliseconds to complete. session id:989654098 login name:11003259 htmpl name:list_cr.htmpl
    03/06 12:40:42.67 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 3297 milliseconds to complete. session id:1708042825 login name: htmpl name:
    03/06 12:40:42.70 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 131941 milliseconds to complete. session id:505604567 login name:22003749 htmpl name:list_cr.htmpl
    03/06 12:40:42.70 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 107674 milliseconds to complete. session id:505604567 login name:22003749 htmpl name:list_cr.htmpl
    03/06 12:40:42.75 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 105784 milliseconds to complete. session id:1467824666 login name:11002927 htmpl name:list_cr.htmpl
    03/06 12:40:42.87 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 73470 milliseconds to complete. session id:461972887 login name:11002878 htmpl name:list_cr.htmpl
    03/06 12:40:42.89 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 10940 Session 782169415:0x0C62FEE8 login by analyst CMSShift1 (cnt:63543590A0C62C4AA45AC7A2022FBCBF); session count 99
    03/06 12:40:42.89 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 22282 milliseconds to complete. session id:0 login name:CMSShift1 htmpl name:replace_login.htmpl
    03/06 12:40:42.96 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 10940 Session 1310773363:0x0D31BE20 login by analyst 11005950 (cnt:3554D6F6CDBF444D94DE5EF880433200); session count 99
    03/06 12:40:42.96 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 17672 milliseconds to complete. session id:0 login name:11005950 htmpl name:replace_login.htmpl
    03/06 12:40:42.99 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 162097 milliseconds to complete. session id:192658409 login name:11005631 htmpl name:detail_cr_edit.htmpl
    03/06 12:40:42.99 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 155222 milliseconds to complete. session id:872229884 login name:11002867 htmpl name:detail_cr_edit.htmpl
    03/06 12:40:43.01 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 5063 milliseconds to complete. session id:989654098 login name:11003259 htmpl name:list_cr.htmpl
    03/06 12:40:43.01 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 100987 milliseconds to complete. session id:1654953627 login name:11003112 htmpl name:list_cr.htmpl
    03/06 12:40:43.07 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 24297 milliseconds to complete. session id:1275233074 login name:shruti htmpl name:list_cr.htmpl
    03/06 12:40:43.26 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 4025 This request took 3156 milliseconds to complete. session id:0 login name: htmpl name:
    03/06 12:40:43.43 del-sdapp-clus web:local 4812 SIGNIFICANT session.c 8287 Session 11006513-1289174667:0x112bfde0 logoff; id(5CE2B41B7EC95B42A81C0EB9099414D4); session count 98
    03/06 12:40:43.56 del-sdapp-clus web:local 4812 SIGNIFICANT

     

     

    3/06 12:40:44.56 del-sdapp-clus spelsrvr 1064 ERROR heat_weather.spl 84 Received error response from calculate_ticket_heat
    03/06 12:40:44.57 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:538242890 (cr:5269655) for atev:413755435 missed 1 firings, the first at 03/06/2019 12:40:01
    03/06 12:40:44.61 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:538242889 (cr:5269655) for atev:413755425 missed 1 firings, the first at 03/06/2019 12:40:01
    03/06 12:40:44.62 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:538242891 (cr:5269655) for atev:413755433 missed 1 firings, the first at 03/06/2019 12:40:01
    03/06 12:40:44.65 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:535928047 (cr:4910476) for atev:381407945 missed 1 firings, the first at 03/06/2019 12:40:13
    03/06 12:40:44.67 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:536162180 (cr:4583532) for atev:353392859 missed 1 firings, the first at 03/06/2019 12:40:13
    03/06 12:40:44.67 del-sdapp-clus spelsrvr 1064 ERROR heat_weather.spl 744 AHD05026:Error creating Activity Log Entry.
    03/06 12:40:44.68 del-sdapp-clus domsrvr 3104 ERROR top_ob.c 4030 Got msg with type set to 0
    03/06 12:40:44.68 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:535928046 (cr:4910505) for atev:381409878 missed 1 firings, the first at 03/06/2019 12:40:13
    03/06 12:40:44.68 del-sdapp-clus spelsrvr 1064 ERROR heat_weather.spl 757 cr::update_ticket_heat_mappingserror during checkin 'AHD03075:Required attribute type is missing from object Request Activity Log'
    03/06 12:40:44.70 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:535928057 (cr:4414923) for atev:338680956 missed 1 firings, the first at 03/06/2019 12:40:14
    03/06 12:40:44.71 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:535928059 (cr:4583532) for atev:353392873 missed 1 firings, the first at 03/06/2019 12:40:15
    03/06 12:40:44.73 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308659 (cr:2907164) for atev:198786401 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.75 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308665 (cr:3031743) for atev:210406266 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.76 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308669 (cr:2901576) for atev:198257858 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.76 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308676 (cr:3103262) for atev:217052833 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.79 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308670 (cr:2854966) for atev:193901967 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.79 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:535928132 (cr:4949575) for atev:384895295 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.81 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308662 (cr:3031668) for atev:210398479 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.81 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308668 (cr:2096305) for atev:123572974 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.82 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308660 (cr:3031850) for atev:210415581 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.82 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308666 (cr:2989082) for atev:206426359 missed 1 firings, the first at 03/06/2019 12:40:27
    03/06 12:40:44.84 del-sdapp-clus animator_nxd 5272 ERROR animator_nxd.c 2471 Recurring animator ANI:534308673 (cr:3031985) for atev:210428965 missed 1 firings, the first at 03/06/2019 12:40:27

     

    Also i have attached  a log file.

     

    Please help me ASAP.

     

     

     

    Regards

    Dheeraj

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    Attachment(s)

    zip
    stdlog.2.zip   100 KB 1 version


  • 2.  Re: CA Service Desk Manager Performance Problem.

    Posted Mar 08, 2019 11:57 AM

    Hi Dheeraj.
    Can you post some general details about your setup ,like
    Which Db,what is installed where , server hw specs,what components are you using, sdm, pa, catalog, xFlow, elastic search.etc.,which setup , conventional or advanced availability, which http server, how many concurrent users
    Thanks and regards
    ....Michael



  • 3.  Re: CA Service Desk Manager Performance Problem.

    Posted Mar 10, 2019 12:59 AM

    First of all thanks for reply Michael.

     

    1) Server Hardware for Data Base --- Intel (R) XEON CPU E5-2620 V3 @2.40 (2 Processor) Ram 192 GB.

    OS window server 2012 R2 Standard 64 bit.

    2) DATA BASE SQL SERVER 2008 MDB.

     

    2) Server Hardware for APPLICATION --- Intel (R) XEON CPU E5-2620 V3 @2.40 (2 Processor) Ram 192 GB.

    But application on Hyper V 

    3) CA Servicedesk  14.1.3 on Conventional Configuration.

    4) IT PAM for Work Flow.

    5) 200 Concurrent Users.

     

    Regards

    Dheeraj



  • 4.  Re: CA Service Desk Manager Performance Problem.

    Posted Mar 11, 2019 02:54 AM

    Hello Dheeraj,

     

    First up - and this is to everyone - it is a good idea to sanitise any log data that is uploaded by CA Communities. Although you shouldn't come to any harm by doing so, I always think it is a good idea to replace items such as hostnames with text such as "MY_SERVER" for example.

     

    Second, performance problems can be very difficult to troubleshoot remotely. We'll try our best here. But if you have a serious issue, it may be worth calling in a CA Business Partner or CA Services to run a healthcheck and at the same time, upgrade your system to a supported version such as ITSM 17.1. Someone onsite with the whole picture is in a much better position to provide advice on the whole solution, rather than troubleshooting spot issues as they arise.

     

     

    Moving onto the troubleshooting.

     

    Your starting point is to define the issue.

    Your definition so far is "ITSM 14.1.03 is very slow while updating any Incident or Request tickets."

    This calls out for clarification. For example:

    * What does "very slow" mean? 30 seconds? Three minutes?

    * What about other ticket types such as Change Orders, Problems or Issues? 

    * What about other parts of the system, such as Administration, Search, Configuration Items etc?

    * Does this impact all users or only some?

    * Does it occur at all times or only some times?

    * When did this issue start? Has it been growing in impact or constant? Was it every working?

     

    These are just examples, but a very good place to get a guide on defining the issue is here:

    How to Identify Performance Problems in CA SDM - CA Service Management - 17.1 - CA Technologies Documentation 

     

    You should also define the environment.

    * How many servers are there? You say "Conventional Configuration" which often implies more than one server (Primary server and Secondary servers), and yet only mention one SDM server. So is there only the one SDM server?

    * Is CA PAM on its own server?

    * How heavy is the CA PAM usage? Are there are a lot of Web Service calls? (If so, stand up a secondary for this purpose.)

    * How many concurrent Analyst users do you have at peak? (Run pdm_webstat).

    * Are all of the servers physical, and not virtual?

    * How many domsrvr and webengine pairs do you have? (At 200 concurrent users, you should probably have a minimum of two pairs)

    * Is your system Event heavy? (If so, dedicate a domsrvr to the Animator process.)

    * Is there a test system? Does it see similar issues?

    * Why is this system not on ITSM 17.1 and when will it be moving there?

    * Is the environment showing any bottlenecks, such as a CPU core at maximum against an SDM process, memory reaching 2Gb, or a network or hard disk bottleneck? (I would guess that a CPU is maxed, based on the animator processes missing their firings and the long running messages in the stdlogs.)

    * Are there any other CA products, like CA Catalog installed?

    * How is the authentication done?

     

    Look for common known issues.

    See here to start:

    CA SDM Performance Problems - Quick Checklist - CA Service Management - 17.1 - CA Technologies Documentation 

    And the Known issues for 14.1 for all point releases, starting with your version here:

    Known Issues - 14.1.03 - CA Service Management - 14.1 - CA Technologies Documentation 

     

    Some things that spring to mind from what you've said so far.

     

    1) SREL_BLOCKS_TIMEOUT is typically set to 30 to begin with, not 10. See:

    CA SDM Performance Problems - Quick Checklist - CA Service Management - 17.1 - CA Technologies Documentation 

    and

    The usage of NX_SREL_BLOCKS_TIMEOUT to improve Per - CA Knowledge 

     

    2) You've got 200 users probably pointing to one domsrvr. Add an additional domsrvr and webengine pair (or two) to spread the load, and leave the original domsrvr to handle the singleton processes, like Animator.

    Rule of thumb is 150 - 300 users pointing to a domsrvr/webengine pair, but this is entirely dependent on load. A high Event system would have a lower number of users compared to a low Event system.

     

    3) Is CA PAM on its own box? If not, it probably should be. Also, check the Web Services load. These will entries with the process name "sda" in the stdlog, but there are other ways to check. Start with CA PAM itself.

     

    4) What is the volume of tickets and other data in the system? Both the absolute number already stored, and the daily/monthly generation rate. If these numbers are large, do you have Archive and Purge running to remove old/excess data?

    5) When was this last working, and did it fail suddenly or slowly over time? The former might point to a sudden change, like a customisation being introduced. The latter might point to something like a growth in ticket data and usage.

     

    6) Start testing a move to a certified ITSM 17.1 environment, and use this as a good opportunity to review your system setup and perform a health check. Now may be a good time to move to a primary/secondary or even an Advanced Availability configuration, for example. But only a familiarity with your environment and business needs would inform which are the better courses of action.

     

    That should probably start your investigation. You don't have to go point-by-point through each part of the checklist, BUT you should check each part of the checklist and focus on the items that seem likeliest to be a cause.

     

    Best case: Your system has simply grown over time, and needs additional resources such as extra domsrvr/webengine pairs, or a dedicated Web Services server.

    Worst case: You've encountered a specific bug (possibly overlaid with the causes above) which needs specific troubleshooting techniques, such as log analysis, to identify. 

     

     

    And if in doubt, please just give us a little more information and we'll see if we can advise on those points.

     

    Thanks, Kyle_R.