Automated regression testing in Agile: why, what, and how?

Automated regression testing in Agile

There’s a paradox surrounding automated regression testing of the UI in Agile. On the one hand, everyone tells you how important it is. On the other, it doesn’t seem to work right in far too many Agile teams.

Web application teams are the most susceptible to this. UI regression testing automation is something that very few companies do, even though visual bugs are incredibly common. So why does this happen and what can you do about it?

Why is automated regression testing so challenging in Agile?

The three cornerstone challenges of software testing in Agile are ROI, ROI, and — you guessed it — ROI. In the brave new world of agile product development, everyone frowns on large upfront investment of time and effort. The problem is you typically have to spend a whole lot of both time and effort to automate regression testing. More so for UI regression testing.

In addition to logic and user interactions, UI regression testing has to cover CSS bugs. Issues of this sort are hard to trace, and they require visual testing across screen sizes and browsers. Coincidentally, most UI testing frameworks lack a good default mechanism for visual testing. This brings us to another question: what can you do to fit automated regression testing into the framework of Agile?

What can you do to make automated regression testing of the UI economically feasible?

There are two strategies that agile teams use to automate regression testing on the UI level while keeping things cost-efficient.

  1. Automation of happy paths with hand-coded tests. This strategy boils down to test case prioritization in regression testing. It dictates that testers only automate those parts of the UI which stakeholders deem crucial for functionality and/or conversions.
  2. End-to-end regression testing automation with record-playback IDEs. In case with this approach, teams use automation tools to cover a wider scope of potential issues on the UI level.

Let’s take a closer look at how companies use these two strategies to automate regression testing of the UI.

Happy paths / test case prioritization with handwritten tests

When you code tests manually, you often end up with insufficient coverage. Real-life UI testing poses too much of a scope of work, and i simply takes too long to automate and maintain hundreds of coded tests.

So if covering the complete UI isn’t an option, focusing on the most critical aspect of the UI is a strategy that you can use. This strategy is as good as your test engineers and project managers are. Still, even with the best developers on-board, the more tests you write, the more overhead you get.

With happy paths, solid prioritization skills are paramount: stakeholders will need to agree on what parts of the UI require automated regression testing the most. This is something that you must do in the early stages of project development. Besides, your team has to decide whether to include tests into the DONE criteria, when to write tests, and how often to revise them. Answers to all of these questions will differ across teams and projects.

If you are working in a TDD environment, your automated regression testing suites can reuse some of the tests written by your UI developers. Testers can tackle visual bugs with screenshot comparison tools like Wraith or PhantomCSS. This will provide you with basic visual testing functionality — alas, at the cost of an increased maintenance burden.

Codeless automation of UI regression testing

The caveat with test prioritization is that it leaves parts of the UI compromised. In theory, you can fill the gaps with manual regression testing, but few teams have enough time to do this on a regular basis. Besides, there’s a wholly different option. If hand-coded tests can’t cover the whole UI, why not use a low-code or a codeless framework?

The codeless approach to automated regression testing is, essentially, as good as your record-playback IDE is. Automating regression testing with a record-playback IDE is something a manual tester without technical skills can do, and it excludes programming-related expenses from the equation. Besides, the creation of codeless tests takes much less time, making project management simpler.

Finding the right record-playback IDE isn’t always easy, though. In the worst-case scenario, your tool will only support editing tests via auto-generated code, which is a one-way ticket to maintenance hell. Luckily, more modern solutions allow for 100% codeless test editing.

There’s a whole lot of other features — aside from codeless test editing — that record-playback tools offer to streamline automated regression testing. Let’s take Screenster as an example of a modern solution for automated regression testing of the UI. We’ll look at the features that we think you should expect from every solution for UI regression testing automation.

Automatic verification of every UI element

Verifying everything on the web page has obvious advantages over handpicking the important elements. Full verification involves a lot of intricate manipulations with the WebDriver, but in case with Screenster, they happen under the hood. Instead of hundreds of lines of code, you get intuitive visual baselines and dashboards.

The idea of verifying everything is important because of the scope of UI testing and the unpredictability of CSS bugs. Where Selenium tests would leave most of the UI untested, leaking visual bugs, Screenster detects every single issue in the content that matters.

Self-healing locators for simpler maintenance

Broken locators constitute one of the top-3 reasons why regression tests break as your project grows. To make automated regression testing a simpler matter, Screenster employs a smart failover mechanism when dealing with locators.

When recording a UI test, the platform captures and stores all possible locators for each element. Whenever a locator gets broken by a UI change, Screenster automatically substitutes broken locators with the ones that remain intact.

Automatic timeouts

Selenium Waits are infamously hard to deal with en masse. Given that UI regression testing typically constitutes of hundreds of test suits, automatic handling of waits seems like a good idea. This is exactly what Screenster offers. When automation regression testing for web pages, the platform calculates optimal waiting time on a per-element basis.

Intelligent handling of dynamic UI regions

Some code-based screenshot comparison tools have a basic functionality for dealing with dynamic region like timestamps, userpics, or banner ads. When working with Wraith or Gemini, you can manually “blackout” particular regions by specifying its coordinates. Screenster can do the same, but with less input.

Instead of making you tinker with on-screen coordinates, the platform automatically detects dynamic UI regions during the optimization of a recorded test. During the further stages of automated regression testing, you can also create new ignored regions by clicking on an element in a UI screenshot.

Collaboration functionality

Being a team effort, automated regression testing requires collaboration-friendly tools. In case with a UI regression testing platform, this would typically imply a shared portal where teams can access and edit tests. This is exactly what Screenster offers, complete with color coded project and test health reports.

Multiple testing modes

Given how different QA automation teams can be in terms of size, skillset, and process coordination, flexibility is always great. As far as flexibility is concerned, Screenster can run the same tests in three modes allowing you to prioritize thoroughness or speed of test execution.

  • The most thorough testing algorithm is called visual verification. In this mode, Screenster executes pixel-perfect visual comparison for every screenshot. Being the most precise out of the three options, visual verification is the recommended mode for pre-release runs of automated regression testing.
  • In the content verification mode, the platform places a greater emphasis on content updates while decreasing the precision of visual testing. Content verification runs faster than visual verification, which makes it more suitable for everyday test runs.
  • Minimal verification is a mode where Screenster focuses on user interactions with the UI. While doing so, the platform drops visual comparison for all test steps except for the first and the last one. The fastest out of the three modes, minimal verification is optimal for quick smoke testing sessions.

When it comes to Agile, simplicity often translates into greater ROI. With Screenster, UI testing becomes simple and cost efficient enough for teams to actually perform it. But does this mean that this solution is going to work for your project? There’s a great way to find out! Try our demo and automate a UI test for your website or web app.

Want to try Screenster on the cloud?

Try Online
Automated regression testing in Agile: why, what, and how? was last modified: September 6th, 2018 by Ilya Goncharov
Thoughts on “Automated regression testing in Agile: why, what, and how?”
  1. In Agile, automated regression testing can be a pain, that’s true, but you should definitely go outside of happy paths automation. I will try Screenster for my website. I really hopes it does more than Selenium IDE or Ghost Inspector.

  2. whenever I hear about regression testing in agile, people mostly talk about unit vs e2e testing, which is weird. why is there a “vs”? i think a tool like this one has a potential to bridge the gap between the two

  3. Okay, so Screenster runs a second comparison while the new test is still fresh to detect a rapidly-changing dynamic regions. This will work for timestamps (e.g. “five seconds ago — a minute ago”), or ads. But what if the timestamp doesn’t change this rapidly (e.g. “today” — “yesterday”)?

  4. @Carl Thompson
    Screenster will detect this difference during a later test run. You will be able to add an ignore region to it. It’s a manual process (for now), but it’s as simple as two mouse clicks.

  5. Okay, there’s actually a statistic that i saw somewhere that says 75% of cases of functional testing are manual testing. In Agile, automated regression testing is something that everyone wants, and a must have. But if it’s a “from-scratch” app or a small “yet-to-become-VC-backed” startup, you won’t have the money/people/time to write all those test cases. You’ll simply get someone run manual UI tests (or outsource the testing to India/Eastern Europe/Phillipines/etc).

  6. Two reasons why companies are not automation their UI tests are:

    Decision fatigue. Do I test my React application UI logic with Jest or Karma, do I add a UI testing, and what should I use, Selenium or something with syntactic sugar on top of Webdriver? Or maybe, should I try one of these “new record-playback” tools, and if yes, how do I support my cases in five years?
    Job Security. With Selenium, I have a team of testers who can potentially become developers or evolve as technical specialists.

  7. Codeless automation is really the holy grail of regression testing in Agile. If you could have manual testers produce robust robust UI tests with a decent coverage, why would anyone want to write code? But here’s the thing, we all saw QTP, Sahi, Ranorex, TestComplete, TestPlant — and everyone still writes their tests? If your tool works — or will work — better that the aforementioned tool, sign me up!

  8. This really is a decent automated regression testing tool, but you really need to do something about how it looks

  9. Agile regression testing should be about speed, which kind of welcomes automation. Truth be told, I always felt king of skeptical about commercial tools of this sort, but seeing how they improve makes me wonder if these will oust Selenium in the future

  10. Looks like Selenium IDE on steroids, which is kinda cool. Would be great to see a record-playback tool that I could actually use in production

  11. An automated regression testing tool of this sort would do god for larger web app-like websites, like ecommerce catalogues. I’m sure you guys will make it

Leave a Reply

Your email address will not be published. Required fields are marked *

WordPress Image Lightbox Plugin