ROI in Software Testing and Automation

There’s an annoying misconception about the test automation ROI that’s been haunting the QA industry. The misconception is that by its very nature, software testing doesn’t have any ROI. Quite literally, some folks (including people from SmartBear) argue that the concept of ROI doesn’t really apply to software QA.

If, for some odd reason, you believe this too, allow me to opine as to why you’re wrong:)

The discount fallacy and the software testing ROI

SmartBear illustrates its idea of ROI not working for QA by likening the software testing ROI to shopping discounts. According to the SmartBear’s Dawn Haynes, discounts create a sense of cost-saving because, well, stuff costs less (Yeah, science!). But in reality, spending less is still spending. So the ROI of buying discounted goods is still negative, the more so if the discount prompts you to buy stuff you don’t need.

Software testing is about cost-saving, too, but just like the glazed doughnuts at your local supermarket, testing costs money. Moreover, testing doesn’t immediately cause revenue growth (unless you’re an outsourcing firm that sells software testing services).

So if there’s no immediate return on investment, wouldn’t it be pointless (unfair, even) to talk about ROI in the context of software testing? That’s exactly how Dawn Haynes sees it.

The problem with this line of thinking is that negative ROI is still ROI. If you absolutely need to spend money on something, spending 30% less puts you in a better place financially. And let’s face it, you absolutely need to invest in testing software. There’s also a pretty obvious causal relationship between testing software efficiently and getting better quarterly results.

The ROI of software testing and the ROI of insurance

QASymphony and Matthew Heusser claim it’s better to look at the software testing ROI the same way we look at insurance. We’re still spending money, so technically, the ROI is still negative. Yet, these costs represent an investment into the mitigation of expensive risks.

Note we’re not only talking about low-probability scenarios where bugs cost millions in lost revenue, litigation, and missed opportunities for VC funding. There’s a ton of high-probability, commonplace risks, as well.

A bug-ridden release inevitably raises the cost of future development of all software that interfaces with the released functionality or feature. Software defects inadvertently impede product maintenance and slow your team down. Besides, post-release defects typically take a lot longer to fix, than bugs caught early in development.

All of these issues are inherent vices of software development, and testing is the only form of insurance that can remedy them. This makes testing a must-include point when calculating the ROI of software development.

Factors behind the test automation ROI

Generally speaking, the software testing ROI is a part of the bigger picture that is the ROI of software development. When calculating the ROI of a particular software testing initiative, you’ll want to analyze the return across multiple dimensions:

• The cost and speed of shipping new features (e.g. changes in the number of story points finished per period of ROI).
• Changes in the cost of code maintenance.
• QA team productivity. This includes changes in test coverage, number of new bugs, number of post-delivery bug fixes, number of test runs per day or week, etc.

It’s easy to see how all of these can impact your bottom line. Needless to say, automation promises positive changes across each of these dimensions. Speaking of the “investment” in ROI, most automation solutions introduce quite a few factors to watch. Let’s briefly look at these factors.

Hidden costs of free tools

Test automation never comes for free, even with free tools. Open-source frameworks like Selenium and Watir are only accessible to programmers whose rates are high. Finding hirable automation engineers may take weeks, also delaying the return on your investment into automation.

The time it takes developers to write unit and integration tests also puts pressure on your budget. If you’re going codeless, free automation platforms like Katalon require paid support from professionals. All in all, automation requires skills, and paying skilled people for the a job is always a considerable investment.

The cost of learning

Whichever testing infrastructure you’re using, newcomers to your team will take time to learn the ins and outs. This is especially relevant for in-house frameworks that many companies still use in 2018. Learning the infrastructure of large software testing platforms is just as complicated which is why enterprise software often requires certification.

Test development costs

In the long term, software testing helps decrease the development costs. That said, someone needs to write automated tests too.

According to Apriorit, writing tests for a simple application takes about 30–35% of the development time for the same application, GUI testing included. In the case of a more complex distributed application, you’ll be looking at 35–50% of development time.

The development of a mobile app takes 3–8 months for a team of 2–5 people. Getting a basic test coverage of the same app can take between 3–4 weeks to 4 months. Obviously, this is just a starting point, and you’ll need more tests as you add new features. All of these require continuous maintenance too.

Test maintenance costs

Maintenance is less painful on the unit level where tests are more robust. However, the higher-the-level you get, the more fragile your test suites are. The maintenance hell becomes a real threat on the GUI and e2e level if you’re hand-coding your tests and don’t take proactive measures to keep the complexity at bay.

A decade ago the fragility of UI tests was one of the reasons behind the industry-wide rule of keeping UI lean. Strangely enough, you can still hear this from some developers. But the reality is GUIs have grown a lot more complex and feature-rich, and will continue to

the quality of the GUI and UX have become a crucial factor driving conversions. As a result, thorough UI and end-to-end testing have become must-have nowadays.

So how do you ensure both high quality and high ROI of UI and end-to-end testing? Having a good tool certainly helps…

Increasing the UI testing automation ROI

Handwritten UI tests are notorious for being high-maintenance. What’s worse, they’ve never been famous for their effectiveness.

Selenium locators are the main culprit in both cases. Maintenance-wise, all but ID-based locators break with the slightest tweaks introduced to the UI. On the effectiveness side, locators only allow for narrow-scope tests that cannot cover a user journey, even if we’re talking about the simplest happy paths. So, basically, we’re talking about a low return on high investment here.

It’s these issues that made us ditch hand-coding and consider visual testing as a more efficient alternative. Automated visual testing has become the core functionality of our internal tool, so let’s see how it fixes the ROI of UI testing.

• The scope of visual tests is the complete UI state. If a defect triggers a UI change that falls outside of the user journey under tests, WebDriver will ignore the bug. In the meantime, visual testing will catch the bug by detecting the UI change. Ultimately, this means wider test scope and fewer unnoticed bugs.
• Visual testing simulates vision, doesn’t just match pixels. Good visual testing algorithms can recognize the DOM underlying the UI. They can also single out separate UI elements and compare them instead of comparing whole-screen screenshots. This approach decreases false positives while keeping the comparison pixel-perfect. Again, this functionality increases the number of caught bugs, thereby boosting QA productivity.
• Intelligent test creation. Our tools uses record/playback for test creation, yet the algorithm is more sophisticated than simplistic record/playback tools of the past. When recording a test, we can automatically recognize the DOM and content, ignore dynamic UI regions, and determine optimal waits.
• No auto-generated code. Auto-generated junk code that old record-playback tools produce was a productivity killer. To eliminate this issue, we manage our tests via a user-friendly yet feature-rich dashboard. The available functionality includes the creation, editing (codeless or via Selenium and JavaScript code), and removal of tests and test steps.
• Low entrance threshold. With most of the functionality available in a codeless format, our UI automation solution is accessible to non-coders. The makes our manual testers 10x more efficient than most Selenium engineers. Also, it takes newcomers to our team only hours, not days, to find their way about the UI testing.

All of these make our testing efforts more productive compared to the Selenium-based process we had a few years ago. Namely, automating 20 test cases for a web application now takes us 1/10 of the time we’d spend hand-coding the same test with Selenium.

If you find this interesting, the functionality described above is now available as a standalone product, Screenster. The tool is web-based, with an option to run on premise. We offer flexible subscription plans with no hidden cost, and the demo functionality available for free. You can try Screenster by clicking the button below.