Anand_AK_Kameswaran

Q&A from Recent WebCast on Performance Testing with CA Service Virtualization

Blog Post created by Anand_AK_Kameswaran Employee on Apr 29, 2016

Recently I did a webcast on how CA Service Virtualization could be used for API performance testing. At the end of the event we received a number of great questions. So many that we ran out of time and could not get to them all.

 

With that, I’ve gone ahead and answered the questions from the event in this blog.  For those of you who missed the event and may want to watch you can view it at: http://bit.ly/1NaZmve.

Questions in the order answered below include:

 

  • Question 1: Is service virtualization the same thing as application virtualization?

  • Question 2: What is the difference between mocking a service, stubbing a service and service virtualization?

  • Question 3: What type of testing should service virtualization be used for: System Tests (ST), System Integration Tests (SIT), or User Acceptance Tests (UAT)?

  • Question 4: Typically external services have some sort of inherent wait time associated with network travel time; can we simulate this since the solution will most likely be in the same network as our System Under Test (SUT)?

  • Question 5: Can we replicate network latency/bandwidth limitation scenarios using service virtualization?

  • Question 6: If my actual application is not available in one environment, can I virtualize that application and make it available in the required environment?

  • Question 7: We have many developers that test against our externally facing APIs can we replace those with a virtual service on our end? Will the developer notice any difference between the real service and the virtual one?

  • Question 8: How does service virtualization factor into 3rd party testing?

    Question 9: What third party services can you emulate with your virtual services: Facebook, Twitter etc.?

  • Question 10: Is there a difference in the way service virtualization handles a virtualized service vs. a real service?

    Question 11: What are the requirements for an application to create a virtual service? Does an application ever need to be rearchitected in order to create a virtual service?

  • Question 12: How do you map the speed of the virtualized system to compare it against the performance of the real system?

    Question 13:  Can you create fluke error and edge conditions for performance testing with service virtualization?

  • Question 14: Is service virtualization beneficial for an IT ecosystem where all the services are available in all the environments 24X7, and there are no third party services?

  • Question 15: Is CA Service Virtualization supported in any public/private cloud servers?

  • Question 16: How can I try CA Service Virtualization to test Microservices?

  • Question 17: Can we virtualize RDBMS databases using service virtualization?

  • Question 18: What are some of the future enhancements planned in CA Service Virtualization?

 

Question 1: Is service virtualization the same thing as application virtualization?
Answer:
Service Virtualization does simulate the back-end of an application, so at a certain level they are the same. One thing to keep in mind is we are mocking, stubbing specific testing, and data scenarios, but we are not actually performing the actual business logic. Service virtualization does cool things like magic strings, magic dates and we have the ability to give dynamic responses. It's not the business logic of the backend system that we are testing.  We are testing the interactions between different layers of the application. But it’s good enough. It’s more than good enough. The first time I got introduced to the solution I was working with a client in the field and we were showing them the capability. After we were done I remember the developer coming up to me and saying something must have been wrong because they had put in a transaction that the virtual service had not seen yet and we had to explain that we can give dynamic responses.

 

Question 2: What is the difference between mocking a service, stubbing a service and service virtualization?
Answer:
Stubs provide pre-canned answers in support of a test and are deterministic at the time you write them.  Generally, mocks allow a data scenario or set of expectations to be initialized at run time. Both of these require development resources to write and generally are closely coupled with the actual code. Virtual services enable you to accomplish both mocking and stubbing.  In contrast, virtual services are created by recording actual services, by contract, or by being provided with sample requests and responses. They are inherently more dynamic and don’t require a developer to write them. Virtual services also already come with a deployment framework making creating them and consuming them a much more seamless operation.

 

Question 3: What type of testing should service virtualization be used for: System Tests (ST), System Integration Tests (SIT), or User Acceptance Tests (UAT)?
Answer:
Integration testing is usually where most clients start using service virtualization. System testing comes in many times during prototyping. The best practice for UAT testing is to use production systems. We do have some clients that simulate some third party calls during UAT. Regardless, the nice thing is with service virtualization most of the defects have been found before you get to UAT because you’ve already run comprehensive system, integration and performance tests.

 

Question 4: Typically external services have some sort of inherent wait time associated with network travel time; can we simulate this since the solution will most likely be in the same network as our System Under Test (SUT)?
Answer:
This is a very common requirement for any performance test because a virtual service can respond instantaneously.  To address this we have created what we call “think time”. It exists at both the service and transaction level. When we record a service we observe the think time, so if we see the response takes 15 milliseconds, then that is how long the virtual service will also take to respond. From there we can then scale times up or down. For example, we can slow things down for the first five minutes while we’re waiting for all the classes to preload. If we also know the performance decreases by 1/3 we can simulate that kind of start time, or we can create a spike for garbage collection every 10 or 15 minutes. We not only have the ability to control response times for the test run, but also at the transaction level, and at the component level. This becomes especially important when working in agile environments with multiple teams working on multiple pieces. In addition, if you are a CA APM customer and you’ve been collecting your performance data from production you can feed that into CA Service Virtualization and make you test system perform just like the real thing.

 

Question 5: Can cwe replicate network latency/bandwidth limitation scenarios using service virtualization?

Answer: Yes. Using think time you can introduce delays and specific timing to the responses to simulate network latency. The think time can be randomly selected from a range. 

 

 

Question 6: If my actual application is not available in one environment, can I virtualize that application and make it available in the required environment?
Answer:
Yes. That is much of what service virtualization does. A good example of this is how we virtualize third party services. Third party services are not even within your network, there not available. We can still capture the needed information using code contracts like WSDLS, or we can use Request and Response pairs. Service virtualization is all about removing obstacles, virtualizing what is not available. Virtual services can be deployed in various Virtual Service Environments (VSEs) so developers and testers have everything they need when they need it. 

 

Question 7: We have many developers that test against our externally facing APIs can we replace those with a virtual service on our end? Will the developer notice any difference between the real service and the virtual one?

Answer: Testing against external APIs is a bread and butter use case for CA Service Virtualization.  External APIs are a common constraint and SV focuses on eliminating constraints.  In general when used for testing, developers don’t need to know that they are using a virtual service vs. the real thing. However, virtual services don’t replicate the full business logic of a service. There are edge cases where it might be noticed.  Virtual services are generally also much easier to provide developers access to and for them to consume. This helps eliminate reduce wait times and test windows and enables developers to be more productive.

 

Question 8: How does service virtualization factor into 3rd party testing?
Answer:
The first challenge with 3rd party testing is connectivity. How do you get access to the 3rd party service? The second challenge is it not always free to test the 3rd party service. Even though it may only be fractions of a penny when it comes to performance and regression tests that are being run in an agile development cycle costs can add up quick.

 

Question 9: What third party services can you emulate with your virtual services: Facebook, Twitter etc.?

Answer: We focus on protocols.  Virtual Services have 2 layers of protocols.  Transport protocols handle things on the wire and support TCP, HTTP, MQ, JMS, etc.  On top of that, you have Data protocols that interpret the message formats such as SOAP, REST, JSON etc.  You can mix and match those as needed to support a third party API.  Most third party API’s will conform to one or more of these standards and can be virtualized out of the box. 

 

Question 10: Is there a difference in the way service virtualization handles a virtualized service vs. a real service?
Answer:
We’ve had that capability to switch between real and virtualized services built into the product for a while. You can set-up all your connections to go thru the virtual service, and then the virtual service can send it to a live service. Within a test environment, you can tradeoff between a virtual service and a live service if you have reasons for doing so. The ability to switch between live and virtual services also plays into what we call image validation. Sometimes live services change over times. It is important to periodically look at the comparison of what you see when you run a test against a virtual service vs. a live service to see what has changed over time.  This way we you know how services evolve over time. Our API Management product can also help you accomplish this. All of our virtualized services can be built to know where the live services sit.

 

Question 11: What are the requirements for an application to create a virtual service? Does an application ever need to be rearchitected in order to create a virtual service?
Answer:
The requirements are having two components that talk over a wire. Service virtualization has the ability to follow the conversation at a Packet Capture (PCAP) level. We can do straight network captures at an HTTP level or try to use whatever else you have available. The product is powerful in how it records and captures and should not require you to rearchitect an application in order to create a virtual service. We’re agnostic when it comes to how an application is coded. If there are two components that talk over a wire somewhere, we should be able to create a virtual service. We also have another technology we leverage called Opaque Data Processing (ODP). This can be used to capture a legacy protocol, like the one that is 15 years old, no one ever touches, it’s going over the wire but you don’t know what it is. With ODP, we can learn it well enough to give you a simulated response, one that will be good enough for you to test with.

 

Question 12: How do you map the speed of the virtualized system to compare it against the performance of the real system?
Answer:
One scenario is if you use CA Application Performance Manager (APM) you can capture it that way. If you’re pulling the data from another tool and then exporting it, we can set up the mappings based on the API call themselves. Bottom line, we do have a way for you to build out the mappings but may vary based on the tool you’re using. 

 

Question 13:  Can you create fluke error and edge conditions for performance testing with service virtualization?
Answer:
You can do this with service virtualization a few ways. For example, with service virtualization you have absolute control of the data scenarios. You can serve negative cases. This is especially useful when running performance tests. Using CA Continuous Application Insight, you can actually capture a bad transaction and then recreate it within service virtualization. By observing the chaos you can then recreate it more efficiently than tasking testers to recreate.

 

Question 14: Is service virtualization beneficial for an IT ecosystem where all the services are available in all the environments 24X7 and there are no third party services?
Answer:
Yes, there are still benefits of using service virtualization in highly available environments. Error conditions are one example. If you have all live systems, error conditions can present a challenge. Another area to look at would be race conditions. It’s very difficult to speed up or slow down systems in production so that you can see how the system under test would react. An additional advantage of CA Service Virtualization is related to maintenance costs. Virtual service environments are less costly than real environments. In fact, a recent study from Forrester showed them to be a lot less expensive. And then there is of course the issue of testing systems that don’t yet exist.

 

Question 15: Is CA Service Virtualization supported in any public/private cloud servers?
Answer:
Yes. CA Service Virtualization runs on the Microsoft Azure platform today. The Azure offering is designed for developers who want to work on a small project or just try out service virtualization. We have other cloud offerings in production. We also have some use cases with Docker and containers. We do see many customers moving to hybrid clouds, where they provide virtual services to users via a cloud architecture.

 

Question 16: How can I try CA Service Virtualization to test Microservices ?

Answer: The easiest way is to try CA Service Virtualization is by signing up for the Azure 30 day free trial. With the Azure instance everything is installed and ready to go.

 

Question 17: Can we virtualize RDBMS databases using service virtualization?

Answer: We can virtualize databases using JDBC virtualization.  However, that is an edge case.  CA also has a dedicated Test Data Management solution that is usually a better fit than databases. Particularly if you are interested in the performance of the database itself or testing data changes, CA Test Data Management will better solve those use cases.

 

 

Question 18: What are some of the future enhancements planned in CA Service Virtualization?

Answer: For the next major release, we are working on a full migration to the web based portal to provide a single unified experience for our users.  In addition, we are working on seamless integration with CA Test Data Management and CA API Management to provide customers with even better coverage, integrating data scenarios and security with the virtual services. We’ve also started work on the DevTest Catalog which enables organizations to share the virtual services more easily across cross-functional teams and throughout the SDLC.

Outcomes