Layer7 API Management

  • 1.  Intermittent Slow Performance CA API gateway 9.2

    Posted Feb 11, 2019 06:31 AM

    Hi All,

     

    We are doing performance testing and experiencing intermittent slow response time in CA API Gateway 9.2.

    We are trying approximately 25 API calls per seconds and policy is very light weight.

    Every now and then, few transaction takes more then 15 seconds. Average transaction processing time is still within 1 seconds. it appears that CA server halts for few seconds and then starts processing everything fine again.

     

    Appreciate your help/pointer for this issue.



  • 2.  Re: Intermittent Slow Performance CA API gateway 9.2

    Broadcom Employee
    Posted Feb 11, 2019 01:06 PM

    Hi ,

     

    Without details about the environment it would be difficult to say anything about it maybe it is better to raise a support case so we can look into the details . 

     

    Regards 

    Dirk



  • 3.  Re: Intermittent Slow Performance CA API gateway 9.2

    Broadcom Employee
    Posted Feb 11, 2019 05:38 PM

    Never see the tps as low as 25, out test servers are not strong, but still be able to reach 1000 tps for simple calls.

    As Dirk said, it's very dependent on the environment,

    - is it software gateway, or appliance gateway

    - standalone server or gateway cluster (how many nodes)

    - what kind of (physical) server the gateway is running on (I hope it's not running on a laptop)

    - how many cpu/mem of the server

    - audits are not recommended for performance test and production, is it disabled

    - if the audits are enabled, what's the size of the ssg database

    - any external connection for the test APIs (ldap, backend apis, etc.)

    - what's the front end response time and back end response time

    ...



  • 4.  Re: Intermittent Slow Performance CA API gateway 9.2

    Broadcom Employee
    Posted Feb 11, 2019 05:52 PM

    Running the Data Collection Tool, which gives thread dumps is good way to see what is the bottleneck, and where all the worker threads are caught in a load test : 

    Data Collection Tool - CA API Gateway - 9.4 - CA Technologies Documentation 

     

    Enabling trace logging can also help -

    Analyzing API Gateway trace logs for performance issues (another tool to use) 

     

    it can slow transactions (although as Mark_HE says for 25 trans/sec is fairly slow) - but it can help identify what step is taking the time - and also help identify those bumps, where you see increased 15sec response time.

     

    Then after identifying the bottleneck, and hopefully being able to fix it you can turn tracing off again.  

     

    Analyzing API Gateway trace logs for performance issues (another tool to use)

    Cheers - Mark



  • 5.  Re: Intermittent Slow Performance CA API gateway 9.2

    Posted Feb 12, 2019 05:15 AM

    Thanks everyone for the reply.Upon further investigation, we found that majority of time within long running transaction is being taken by assertion which sends outbound JMS message. 

    everytime API is invoked, we are publishing audit message in JMS queue and intermittently it's taking long time to publish.

    We are using TIBCO EMS servers and there aren't any error/warning in EMS server logs.

    Only thing i have observed in EMS logs is that, API gateway creates connection and closes connection every few mins. I had expect it to open connection once and use it for entire run.  In JMS transport setting , we are able to specify number of sessions but there is no option for connection pool (We are using CA 9.2).

     

    Please let me know if anyone has come across similar issue.



  • 6.  Re: Intermittent Slow Performance CA API gateway 9.2

    Broadcom Employee
    Posted Feb 12, 2019 11:23 PM

    I cannot find connect pool for jms connection on gateway 9.2, but I can find it on gateway 9.3



  • 7.  Re: Intermittent Slow Performance CA API gateway 9.2

    Broadcom Employee
    Posted Feb 22, 2019 10:06 AM

    Hi

    Did the answers on this thread answered your question? If it did please mark it as the right answer.
    When your question is not answered or you still have additional questions please let us know.

    With Kind Regards
    Dirk