Service Virtualization

Hi,

I have created virtual service for Performance testing purpose and deployed in PERF VSE. Virtual Service simple one having one request and response pair. Web Application is under performance test which uses Virtual Service in back end. Service will be hit by 60 TPS. Virtual Service is responding properly for all transactions.

But when i checked "Virtual Service Response Time" Report from portal, I could see Spikes intermittently during the load test. Not able to find any error or info regarding why Spikes occurred. Below is an example.

In the above graph, During 8hr run, few spikes occurred. 

How to identify what caused these spikes in response times and what might be the reasons. 

We have high configuration server, 24 core processor and 64 GM RAM.

Thank You

Rajasekhar

Some random thoughts on this:

• You show the response time graph, What does the number of transactions per second graph show aroound those period? Just your average 60 tps or does it also show spikes?
• JVM garbage collection is something that could cause JVM to temporarily do much less work/processing and subsequently let the average response time spike. You mention your system has 64GB RAM but how much memory has been configured for the VSE process?
• If your average transactions per second graph is steady, and the JVM garbage collection is not the cause, then you would have to know what is happening to your system resources at those particular periods of time. Is some other process kciking in and consuming CPU and/or Memory?

Cheers,

Danny 

I have also experienced similar behaviour during performance tests- Do you have anything special in your virtual service like match scripts or custom coding?

Thank you Jamesdunn and Danny,

Added my comments. Could you please check and advice.

• You show the response time graph, What does the number of transactions per second graph show aroound those period? Just your average 60 tps or does it also show spikes?

       Average transactions per second graph is steady during the spike time. 

• JVM garbage collection is something that could cause JVM to temporarily do much less work/processing and subsequently let the average response time spike. You mention your system has 64GB RAM but how much memory has been configured for the VSE process?

        8 GB Memory is configured for VSE process. And i am not sure how to check is there any JVM garbage                 collection activity occurred during that time.

• If your average transactions per second graph is steady, and the JVM garbage collection is not the cause, then you would have to know what is happening to your system resources at those particular periods of time. Is some other process kciking in and consuming CPU and/or Memory?

        I have consulted Admin team and got confirmation that no other process kicked in during that time. 

         I have seen this kind of behaviour very frequently during performance test runs. I could not see any details in logs also.

I have also experienced similar behaviour during performance tests- Do you have anything special in your virtual service like match scripts or custom coding?

This particular service doesn't contain any thing like match script or Custom Coding. Can you please let me know, if virtual service contains any Match Script or Custom Coding, will it affect the performance.

Than you

Rajasekhar

I am getting similar behaviour with some of our services and it's also causing issues:

Response times:

Transactions per Second: