Service Virtualization

Hi,

How frequently DevTest components(Registry, VSE, Simulator and coordinator) and Virtual Services deployed on VSE can be restarted?

What would be the impact or Advantages & Disadvantages, if I restart Virtual Services deployed on VSE everyday ?

Thanks,

Narasimha

Interesting question.  Would be interested to see posts here from customer users of DevTest and their experience.

My initial reaction is why do you want to do this? In other words, are you seeing some unexpected performance issues, spikes in memory or CPU utilization, or you need to clean up resources that a customization might be using such as the Shared Model Map?

I have worked with customers that restarted their DevTest environments as part of their weekly maintenance windows. Traditionally, this is not an issue as most customer teams are out of the systems, the downtime is scheduled, and the teams are notified via various communications mechanisms. Conversely, I have seen a few customers run services for weeks to months at a time with no significant issues.

In the Continuous Integration / Continuous Delivery space, I have seen customers use things like Docker to spin up environments just long enough to run tests. In these situations, the services themselves are just-in-time and only live for the duration of the test(s).

One disadvantage of stopping/starting services is the potential impact to consumer applications if they send a request while the service is down. When this happens, a negative perception about service availability begins to develop.

Hi Joel J_NeSmith

Due to following issues, I want to restart the VS(Daily) and components(Weekly).

- Few stubs are going to offline mode with DB connectivity errors

Ex: I have total 15 VS deployed on VSE and out of these 15 only 2 VS are behaving like this i.e going to offline mode with DB connectivity issues if the stubs are idle for 4 to 5 hours.

- Performance issues: Services are taking long time to respond, If I don't restart VSE at least once in a week

Ex: There is a VS which took 15 seconds to respond before VSE restart (In this case I have not restarted VSE for one month). Post VSE restart response was sent back in 1 to 2 seconds. I have observed same behaviour multiple times

So I have created a monitor which auto Stops and Starts the virtual services. Once this activity is done, automatic email will be triggered with the status of virtual services post restart. This activity will be done at 7:00 AM and this time no one will be invoking the virtual services.

I just want to know, will these activities impact performance of DevTest Components.

Thanks,

Narasimha.

Your approach should not impact the performance negatively. It seems that this is actually helping your environment. It appears, however, that stop/starts are solving a symptom not the root cause of the issue.

In your posts, you report that services slow down (1-2 second response times increase to 15+ second response times), service connections to the DB start dropping, etc. These symptoms make me wonder about VSE memory consumption, CPU utilization spiking to 100%, possibly heap dumps occurring, what other software is running on the server, and more.

By design, the steps in a service do not maintain a persistent connection to the DBMS. So, connection drops are indicative that something else happening in the server or the network when the step attempts to make a connection to the DBMS. 

If possible, try to obtain reports that show the health of the server when the services slow down? Look for things like memory consumption at the point of the slow down, CPU utilization by server process, etc.  How much memory is consumed by each of the DevTest server components (Registry, Portal, VSE, etc.) and how much free memory is left on the machine.

If you have any customizations in the services such as custom JARs, scripting, etc., are these assets experiencing issues, throwing exceptions into the logs, consuming memory, leaking memory, etc.

In the vse.log file, what do you see the VSE reporting in terms of CPU and memory consumption? 

The vse.log displays that might be helpful are the ones that show the running services and consumption stats. For example:

<datetime removed> [Event Sink Thread Pool Thread 1] INFO com.itko.lisa.coordinator.VirtualServiceEnvironmentImpl - newimage: status=running, capacity=1 current tx/sec: 0 peak tx/sec: 0 errors: 0
<datetime removed> [Event Sink Thread Pool Thread 1] INFO com.itko.lisa.coordinator.VirtualServiceEnvironmentImpl - Memory used 192mb, allocated 631mb, max 910mb (21%) Our cpu usage 0%, system cpu used 33% GC time 0%

Hi Joel,

CPU utilization is not more than 60% any time

VSE, Simulator, Coordinator and other components heap memory utilization is about 60%

All services are running absolutely fine for 14 to 20 hours a day. Testing team will not be using these virtual services after certain point of time in a day and from that time if services are idle for more than 3 to 4 hours then VS is going to offline status because of DB connectivity issues.

Services which are going offline are Out of box features to DevTest Solutions and does the following activities.

-          Testing team will insert records into DB

-          Customised virtual service will read records from DB, creates a request and updates the DB record

-          Above process repeats with a sleep time of 20 seconds

So after some time while querying the DB, DB is throwing some connection error.

Thanks,

Narasimha.