Service Virtualization

Hi Everyone,

I have created the virtual service and Performance test team is using the virtual service for their performance testing. After the test is completed, there are some internal server errors. I have configured the think time as per the SLA defined to that particular service. Path finder is set to off.

What could be the reason for this?

J_NeSmith

Thanks

Nikhil

The capacity is set to 100 

Hi Kokumar4, we need to do some narrowing down to try to help you with an answer.

What indicators did you use to determine the capacity of 100 was a proper setting?

How many other Virtual Services are active during this load test? If any were running, do these services have a high capacity setting?

Within Portal, what was the highest TPS rate achieved by the service during the load test?

What is the response SLA (Think Time) set to?

Can you share information about the 'internal server errors' you refer to? 

- Perhaps, some log file excerpts are available.

Can you provide some additional information about the transactions in the load test ( avg. payload size, # TPS or virtual users targeted, VS protocol, etc.)

Can you offer information about the VSM/VSI under test

- Number operations / responses

- Any operations with a high number of incoming request arguments 

- Does the virtual service under test have customizations (added steps, custom dphs, database I/O, Match Scripts, etc.)

Can you provide some high level server configuration information?

- OS, # CPUs, # Cores per CPU  <== CPUs and Cores affect the maximum capacity setting

- What other services are running on the server where the VSE service is running

- What is the heap allocation for the VSE (vmoptions)

- Did you make any changes to the OOTB GC (vmoptions)

Thanks J_NeSmith 

Please find my comments inline. However we have ran another test after sometime and the results were as expected. 

What indicators did you use to determine the capacity of 100 was a proper setting?
<Nikhil> Earlier the Capacity was set to 1, Default one. We thought this was the reason so changed it to 100 thinking that is the maximum transactions that can come to virtual service in a shot.

How many other Virtual Services are active during this load test? If any were running, do these services have a high capacity setting?
<Nikhil>There are around 60 virtual services are running but those are of completley differnt virtual services, They do not have any interaction with this virtual service.

Within Portal, what was the highest TPS rate achieved by the service during the load test?
<Nikhil> For other services - the maximum TPS was 150

What is the response SLA (Think Time) set to?
<Nikhil> The SLA is 3000 milli seconds - which is given in the Virtual Service Image

Can you share information about the 'internal server errors' you refer to?

- Perhaps, some log file excerpts are available.

<Nikhil>
Unexpected Exception occured
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.7.0_55]

Can you provide some additional information about the transactions in the load test ( avg. payload size, # TPS or virtual users targeted, VS protocol, etc.)
<Nikhil> In 10 mins of load test - there were 6600 passed transactions and roughly about 360 internal server error transactions.
The maximum TPS for this service is 14

Can you offer information about the VSM/VSI under test

- Number operations / responses
- <Nikhil>There are completley 79 differernt responses configured

- Any operations with a high number of incoming request arguments
- <Nikhil>Not exactly

- Does the virtual service under test have customizations (added steps, custom dphs, database I/O, Match Scripts, etc.)
<Nikhil>No, there is no extra customizations done to this service.

Can you provide some high level server configuration information?

- OS, # CPUs, # Cores per CPU <== CPUs and Cores affect the maximum capacity setting

- What other services are running on the server where the VSE service is running
<Nikhil>There are other services running but no body was doing any testing on those services

- What is the heap allocation for the VSE (vmoptions)
- <Nikhil>6gb

- Did you make any changes to the OOTB GC (vmoptions)
<Nikhil>No

- Yes for the Other performance test, that was resulting us in higher CPU Utilization.

Thank you for the details.

What version of DevTest are you using?

Is the VSE running on Linux?

Thank you joe.gunnells

yeah my VSE is running on some Unix box. 

and i am using devtest 10.1.0 version of Devtest

I do not have a definitive answer that is guaranteed to solve this issue. It might be best to open a ticket to allow CA Support to help you out.

If you try the below, my suggestion would be to try one setting at a time to determine the impact. Too often, teams make mass changes without determining which settings yield desired results. The settings below may or may not be detrimental to the environment.  

Linux Machine where VSE running:

- Check the # of sockets (i.e., ulimit). Many times a Linux environment will have a limited number of sockets available.

- If you know the VSE PID, you may be able to see the limit using:

cat /proc/<PID>/limits

You may need to increase the number of available sockets:

ulimit -n 11000  (you likely need to be admin to do this; and 11000 is just an example)

logging.properties file:

- make sure the log4j.rootCategory=WARN or higher. Logging can be detrimental to performance.

local.properties (you might try this): 

vse.socket.reuse=true (allow environment to reuse sockets, if it can)

lisa.net.timeout.ms=60000 (increase timeout / expiration)

lisa.pathfinder.on=false (turns off Application Insight overhead)

lisa.CycleExecHistory.buffer.size=2  (reduces the amount of history the VSE worker thread tries to keep)

Again, keep in mind, the above should be carefully tried and tested. CA does not know the exact configuration of your environment so the above are generic items that may improve performance and reduce issues.