Service Virtualization

We are facing lot of issues Virtual service goes down during Friday and weekend night  and never send a alert or restart the service if it is ended or stop. This is causing testing impacts. 

Please note that The  option to restart a virtual service is selecting the 2 check boxes while deploying the
service, is already checked.

but still virtual services are not being started automatically and sending any alerts. Tool should be able to send alerts and restart the VSI if it stops.  Is there any way it if tool can set up automatic trigger to restart services & send alert. Currently we are restarting manually offshore time to prevent offshore testing impact. Please advise steps and screen sho. 

Thanks!

Given the scenario you are describing and assuming that you have checked the auto restart when the services were initially deployed, I believe you may be trying to solve a symptom and not the root cause.  Your services are shutting down for a reason.  Normally, when the services shut down like this, they do so that a developer can look at why they are shutting down.  Do you have any output in the log files indicating why they are shutting down?

Check out the use of the lisa.vse.max.hard.errors property here:  Local Properties File - DevTest Solutions - 9.5 - CA Technologies Documentation 

The lisa.vse.max.hard.errors property will let you set a limit (default is 3) for the number of times a service shuts down before an automatic restart is not attempted.  I would use caution about setting this number to unlimited (-1) or disabling the feature.  As I said, I believe there is a different issue that needs to be solved for in your environment.   

For example, what is the failing VSM's Base Path?  If it is only a "/" and your environment uses port scanning software, I would look in the VSE or service log to see if malformed transactions are causing your VSM to stop.  I have seen this occur at other installations.  The service shut down, in this example, was resolved by tightening the Base Path on the VSM.

Thanks Joel,

LISA service below & few others  stop running & fails over the weekend into Monday morning when LIVE url gets down due to Live applications refresh or something else. looks like when Live application brings down application in backend Url for DB refresh then virtualize  service stop. After  DB refresh application restart the Live service, then LISA service does not know how to start start this service

Please see issues and errors application is getting.   I come in the morning and had to restart services manually .

Issues:

Friday 7/8 - 404 error from Get Pricing Rules response

Monday 8/1 - The Get Pricing Rules service is encountering an issue. 404 Not FoundNot FoundThe requested URL /myca/loyalty/us/pricingservice/v1/LoyaltyPricingInfoServiceWS was not found on this server

Monday 8/15 - The Get Pricing service is returning a 404 error. 

https://slog-view-qa.iseatz.com/get/e448950822a93895b693b19fb67fb200/620ced4d-94d7-4175-b5d2-2f7e200fc9c2/supplier/get_profile/get_pricing_rules_response.xml

404 Not FoundNot FoundThe requested URL /myca/loyalty/us/pricingservice/v1/LoyaltyPricingInfoServiceWS was not found on this server.

LISA URL

http://qcentral917.intra.aexp.com:5555/myca/loyalty/us/pricingservice/v1/LoyaltyPricingInfoServiceWS

So backend base url is :/myca/loyalty/us/pricingservice/v1/LoyaltyPricingInfoServiceWS

as long as backend base url is up, there is no issue. but if backend base url down due to DB refresh or something else then LISA should be able to send atleast an alert to users that service has been stopped.

Can this alert be customized in the tool?

I am not sure why a 404 from the Live service would not be returned to the SUT on the Responder step.  Rather it seems to be causing your VSM to stop.

Some potential things to consider:

- If you are running on the later versions of DevTest check out the different Fail Over modes.  If the Live backend is down, the service can branch to provide a virtual response.

- Look to see if you could branch the If Environment Error assertion on Live Invocation to another step in the model.

Perhaps, add a step that sends an email anytime the Environment Error assertion fires.  (Look under Steps, Utilities, Send Email step).

 or  

- If your VSM traps an environment error, add some logic to change the lisa.vse.execution.mode of the service to "Most Efficient" to cause it to always go virtual.  Challenge here is that someone would have to go in and revert the execution mode back to "Live" or "Failover" or whatever it was originally set to at deployment.

- Or, branch to a step that both changes the execution mode and sends an email using the Send Mail step.

- Implement a Live System shutdown procedure to include an automation process that changes your VSM.  For example, when the Live System shuts down, an automation script sets the service to virtual mode (See DevTest Rest API).  When the Live System comes back up, an automation script sets the service back to "whatever" mode it is supposed to be running in. 

Thanks Joel,

We do not want service to virtual mode when Live System is down as back-end account will not work only Virtual Account will work. We need to virtual service to restart automatically when there is red in error column on Server console . Can Lisa server team check . Can Server console have restart option when it sees red in error column.  

Hi Joel,

For now, I've configured Property in Dev Test System Properties to 

lisa.vse.max.hard.errors=101

on my local Lisa workstation (8.0.1)

Please provide the steps to deploy property change to Lisa server?

would registry start on Lisa server  pick up local change?  

Thanks!

Make the change to local props on the server or servers where VSE is running.  Then, stop / start your VSE service.

The change should be applied to the local.properties file that the VSE Service consumes when it starts.  I do not believe that you have to restart any other DevTest services. 

So instead of making change this  to Dev Test System Properties , we 

We requested lisa server team. That is solution provided by CA .

Please update Property  to  lisa.vse.max.hard.errors=101  Please check out this use of  the lisa.vse.max.hard.errors property in LISA API below : :  Local Properties File - DevTest Solutions - 9.5 - CA Technologies Documentation 

Please see the response below from LIsa server team.

 "SMC Registry is used by other teams as well who belongs to same portfolio. It is not dedicated to your team alone. local.properties file change will effect other teams as well. So, we don't think it is correct" .

Is there any other solution to resolve this issue.

Agreeing with Joel, from our point of view, it does not make sense to keep a virtual service running that causes a lot of errors. This behavior affects other virtual services sharing the same VSE unnecessarily. If you want to have a different behavior (different configuration) of the VSE than the other teams you may want to ask your DevTest (fka LISA) server team for a separate VSE service. There are several options to execute additional VSEs on the same or on different servers connecting to the same registry.

As Joel recommended, it seems to be the best way to analyze and identify the root cause why the virtual service fails, and then try to add some logic to it to harden it against this failure. 

On the missing alert for a stopped virtual service I recommend to create an idea in our community. These ideas are reviewed frequently for adding new features to the product.

We found the LISA server logs below after service shut down on LISA console. do we know why this error happen on server? which directory is this? and how to fix it. I tested service works fine on local. These issues are happening on lisa console only.
/opt/ca/lisa/logs/LISA8_SMC_Func/lads/A728A7027C6111E680F6FA163ED3CA7C


-------------------------------------------------------------------- | Message: Sourced file: inline evaluation of: ``import com.ca.sv.devtest.util.GenerateString; package amex; import com.itko.lis . . . '' : Typed variable declaration : Method Invocation Workbook.getWorkbook : at Line: 17 : in file: inline evaluation of: ``import com.ca.sv.devtest.util.GenerateString; package amex; import com.itko.lis . . . '' : Workbook .getWorkbook ( new File ( "/opt/ca/lisa/logs/LISA8_SMC_Func/lads/A728A7027C6111E680F6FA163ED3CA7C/Lisa_New/Data /CTT_online_dt/Online_LoyaltyPricing/LoyaltyPricingInfoResponse.xls" ) ) Target exception: java.io.FileNotFoundException: /opt/ca/lisa/logs/LISA8_SMC_Func/lads/A728A7027C6111E680F6FA163ED3CA7C/ Lisa_New/Data/CTT_online_dt/Online_LoyaltyPricing/LoyaltyPricingInfoResponse.xls (No such file or directory)

--------------------------------------------------------------------------------------------------------------------------------------------------------

I've posed idea on missing alert. 

Did you notice the blank (White space character) in the two paths? It is at different places in the paths. Is this correct or a typo or copy/paste problem?

Please find attached logs for Path on LISA console.

above attachments are the logs on Lisa console after service shut down below.

From my point of view the exception says that the custom java script step (Beanshell) fails to create an Excel file in the specified path. I'd suggest to start troubleshooting this error, such as to check if the path is correct (exists and has the proper access rights), the file system has some space left, the Linux/Unix system has some inodes left for file creation, etc. Once you know the reason of this exception you may then be able to harden the script and work around this failure.