Business decisions are run by numbers, and these numbers are often compared over time. One of the most important values for making decisions is Return on Investment, or ROI, and it can be a very powerful tool to track value.

SV is valuable because it simulates unavailable systems and applications across the software development lifecycle, allowing developers and testers to work in parallel for faster application delivery, higher quality, and improved reliability. SV greatly reduces costs and accelerates time to market, thereby improving overall investment return. OK, so how do we prove the value of SV using numbers so we can make better business decisions?

Have you heard the phrase “back of the envelope” or “back of the napkin” before? Sure, you have. It’s a way of rounding out numbers and estimating so you can quickly determine a value. It’s useful because it’s fast, easy, and fairly accurate. We’ll use this method to calculate ROI for SV in our example below.

Engineering cost assumptions

Let’s start by calculating the average cost of one engineering hour. You can of course calculate separate software developer or QA engineer costs, but I’ll combine them for simplicity. We can assume their average salary combined is $100,000 and they work eight hours per day. We all know there are 365 days per a year, but excluding weekends and holidays there are only about 250 working days per year (don’t get me started with the Gregorian calendar, leap years, and working nights and weekends to launch a major release). The rest of the time you get to spend with friends and family, mowing the lawn, and ferrying your children from one game to the next. Done! Not too bad, huh?

So far, we have:

Bug fixing costs

OK, so, what about the average cost to fix a defect prior to release? How’d you get that number? Well, there’s no “right” answer. There are several studies and published reports and of course we have recent industry surveys. The numbers vary between sources but not by as much as you might expect.

Obviously, the cost of a bug goes up based on how far down the Software Development Lifecycle (SDLC) the bug is found since it must go back to the beginning to be fixed, hence the “shift left” saying that is all the rage these days. Interestingly, the cost per bug also goes up as quality improves since less bugs are found (but this is not a reason to skimp on quality!). And don’t forget that some bugs take longer or more people to fix than others, so you see this isn’t cut and dry.

OK, then. What do we know? Well, I’m glad you asked! There’s a widely-cited study from the Systems Sciences Institute at IBM claiming that errors cost about seven times as much to fix if found in production compared to pre-release testing (Dawson, Burrell, Rahim, & Brewster, 2010). Barry Boehm explains in his book Software Engineering Economics that an error in the wild is somewhere around 10 times costlier to fix than if found during testing (Boehm, 1981).

Furthermore, CA’s research into public sources has developed an industry average cost ratio per defect at each stage of testing.

This gives us an average of being eight times costlier to fix a bug in production than during testing.

Putting real money behind our example, if a bug found before release takes one engineer 10 hours to fix ($50/hr x 10 hours), the cost to fix the error is $500. If this same error were found at a customer site, it could cost $4,000 to fix. This is because troubleshooting takes time from customers and multiple technical support, development, and quality assurance engineers as it restarts the SDLC.

Let’s add this to our list:

Are you starting to see why testing early and often is so important? It’s more cost effective to test sooner rather than later!

Putting it all together

Now, after just two rough estimates, we can compile a simple ROI assessment! Let’s use SV to “shift left” and assume we found half the bugs reported in production before release when using SV compare to not using SV. We’ll add to our assumptions that we release a new software version quarterly and calculate the total savings per year. We might also say that we will find more bugs pre-release because of more thorough testing and increasing the number of test cases, coverage percentage, or otherwise, but we’re trying to be simplistic. Use your own numbers and try a few different iterations and see what you come up with! Are you with me?

Our assumptions:

Wow!!! Can that be? We reduced costs in defect remediation due to shifting left, improved staff productivity due to accessibility, and reduced overall time to market due to earlier release!

Did we really save the company $175,000 last year?

YES! Yes, we did! Roughly. Now go show your manager and ask him or her to hire two more developers with the savings so you can improve your ROI even further. Multiply the savings by the number of years you’re using SV. Don’t forget to subtract hardware and third-party licensing costs. Happy saving!

Please note, any ROI analysis is only as accurate as the data you put in. Check out our SV Benefits Calculator or use the ROI Tool in the DevTest Enterprise Dashboard to run your own ROI analysis. You can also contact your CA account representative for a more thorough analysis by our specialized team.

DISCLAIMER:  

The numbers/percentages used to calculate savings are representative of data found across CA Service Virtualization customers and various industry studies/reports. The values expressed are not a guarantee of achievable results and will vary depending upon your current infrastructure, people, and processes.

REFERENCES

Boehm, B. W. (1981). Software Engineering Economics. Prentice-Hall.

Dawson, M., Burrell, D., Rahim, E., & Brewster, S. (2010). Integrating Software Assurance into the Software Development Life Cycle (SDLC). Journal of Information Systems Technology and Planning (Vol. 3).