What is end-to-end testing?September 25, 2018
What is end-to-end testing?
End-to-end testing (e2e testing) is a software quality assurance methodology that ensures correct functioning and performance of applications in production-like scenarios. This methodology checks if an application performs as designed on all levels and across all subsystems. The scope of end-to-end testing encompasses the application in its entirety, as well as its integration with external interfaces and outside applications.
Okay, but doesn’t this ring a bell? Whenever someone talks about testing software in its entirety, doesn’t it make you think of system testing? As a matter of fact, novice testers tend to confuse end-to-end testing and system testing due to their seemingly similar scopes. This said, the two couldn’t be more different.
End-to-end testing versus system testing
Comparing end-to-end testing and system testing, the latter actually has a narrower focus. For this reason, it makes more sense to look at what system testing covers and proceed to what’s added in end-to-end testing.
|System Testing||End-to-End Testing|
|Validates the software system but doesn’t look outside of it.||Validates interconnected systems and external interfaces.|
|Looks at system features (viewing each feature as a subsystem).||Looks at the UI/GUI, backend and hardware to cover all functionality that the user experiences when using an application|
|Follows integration testing.||Follows system testing.|
|Is easier to automate (though manual system testing is still a widespread practice).||Manual end-to-end testing is more common due to the need of dealing with external interfaces.|
Basically, system testing precedes end-to-end testing. It isolates a system and checks whether the functionality of the system meets the product requirements. E2E testing also looks at how separate features interoperate as parts of a product.
With end-to-end testing, you go one step up the pyramid. You test at the complete functionality of the product against life-like scenarios that mirror product requirements. Tests of these sort involve interaction with databases, hardware, and networks. By doing this, e2e tests explore system dependencies of various origin and assure correct operation and error-free data flow across all applications layers.
Okay, I guess all of this is beginning to sound too high-level at this point. So what exactly does the end-to-end process look like?
How to do end-to-end testing?
The columns in this table should serve as a starting point for QA teams planning out and performing end-to-end testing. Each case in end-to-end testing covers a single functionality, which is why tests reflect separate user-facing functions.
|1. Building user functions →||2. Building conditions →||3. Test cases development|
|Listing features of all systems and subsystems||Listing conditions for each user function||Building test cases for each condition|
|Analysis of actions, input and output data|
|Analysis of nature and relations between user functions|
Seems clear and simple so far, right? Let’s see how this applies to a real-world application.
End-to-end test example: signing in to Google
A complete end-to-end testing suite encompasses everything that a user can do with your application. A single e2e test, in its turn, verifies a single functionality as perceived by the user. With this in mind, signing in to your Google account seems like a good example of an end-to-end test. In the test below, the user will navigate through two pages and complete four interactions with the UI.
Page #1: Input valid email, click Next.
Page #2: Input valid password, click Next.
When testing these four steps, and end-to-end test will check if anything prevents a user from successfully completing the interaction with the web app. This can include a whole bunch of things.
- The forms and buttons need to render on the page and send the correct data to the server.
- The web application needs to process user actions and server responses correctly.
- The UI need to maintain its visual integrity throughout the user journey.
- The sign-in menus should be accessible via the right URL.
- The completion all steps in the user journey should redirect the user to another correct URL.
Notice how things can go wrong on both the front and the back end? Another thing to note is that this e2e test remains UI-driven. It’s easy to imagine someone using Selenium or Protractor to automate tests like this one.
Speaking of Selenium and Protractor, these two are arguably the most popular end-to-end testing tools in web UI development. Cypress is yet another framework that’s gaining traction, along with TestCafe and commercial tools like TestComplete.
So if there’s no shortage of e2e testing tools, what is it that makes QA teams struggle with automated end-to-end testing? Why do so many companies still rely on manual testing to get the job done?
Knowing about end-to-end testing best practices is great, but have you actually seen anyone follow them? There’s a huge difference between how teams of QAs should do end-to-end testing on paper and how they actually do it in real life. Let’s explore the challenges of end-to-end testing automation and the strategies companies use to overcome these challenges.
End-to-end testing on paper and in real life
On paper, product managers, requirements managers, and testers will compile detailed lists of conditions for every user function. On paper, QA automation teams will invest weeks into writing multiple tests for each feature of each system. In real life, no one has time for this.
Real-life end-to-end testing mostly boils down to manual testers reproducing *some* user stories. Sure, there are TDD and unit tests to rule out potentially critical issues at lower levels, which actually makes this “selective end-to-end testing” acceptable. But not everything gets caught! Depending on how solid your engineering and manual QAs are, the end users will see a few or a painful number of bugs. And the cycle repeats itself with each new release.
In the best case scenario, testers will also reuse suites of hand-coded functional and UI tests to automate happy paths at the system level. These are no substitute for automated end-to-end testing at full scale, but hey, that’s pretty much all you get with handwritten tests. Let’s take a look at another example of e2e testing.
End-to-end testing example: Gmail
Manual or automated, end-to-end testing mostly reproduces how a user interacts with an application. A (somewhat simplified) case of an end-to-end test for Gmail will include following steps:
- Launching a browser instance.
- Using the browser address bar to type in the URL of the Gmail login page; waiting for the login page to load and open.
- Logging into Gmail account by typing the user credentials into the forms available on the page.
- Accessing the inbox and verifying that content and presentation are correct.
- Opening Read and Unread emails and verifying that content and presentation are correct.
- Logging out of Gmail.
Naturally, you can automate these steps with Selenium or other frameworks. You’d normally have similar test cases for other simple actions like reading, sending and receiving emails, reporting spam, etc. But doesn’t this look a bit like system testing?
The problem is, actually writing e2e tests that cover everything is simply impossible. For this reason, testers normally focus their effort on writing tests for the core of the user journey. Real-life tests will often ignore content and presentation because the GUI is too much to cover. As a consequence, teams still have to run occasional manual end-to-end testing sessions to verify that the UI and content are fine.
Yet another option is to automate end-to-end testing with a codeless tool…
Automated end-to-end testing, codeless style
Automation engineers are often sceptical about record-playback IDEs, which is partly justified. Old-school record-playback platforms of the 2000s were too cumbersome, and offered too little in the way of functionality. True, tools like Sahi or Ranorex spared you from having to handwrite tests. On the flip side, editing auto-generated code they produce is more painful than writing quality code from scratch.
Luckily, modern-day solutions don’t make you fumble through auto-generated code whenever you’re editing a test. The cloud-based automation platforms don’t depend on testers’ programming skills. Let’s see what our own product Screenster has to offer in the way of automated end-to-end testing.
Codeless end-to-end tests that work for real projects
Being a codeless solution, Screenster gives you the full power of WebDriver while abstracting away its complexity. Both test creation and test editing are carried out via a neat user-friendly dashboard, and the tool has a virtually non-existent learning curve.
Eliminating the need to hand-code your tests increases the productivity of automated end-to-end testing. With Screenster, a manual tester can automate an end-to-end test in under 3 minutes. Editing a test is just as fast and simple. Here’s how it works for a Gmail test similar to the one we’ve mentioned earlier:
Smart baselines with image, CSS, and content comparison
When recording a test, Screenster verifies every on-page element. It generates separate baselines for UI elements and content, and it captures the DOM structure of every UI component. When comparing visuals, the tool filters out visual noise with anti-aliasing, and captures the most minute layout shifts and visual bugs.
Self-healing tests with Smart locators
When analyzing the DOM structure of a UI, Screenster stores multiple locators for every element. In case of a change in the DOM structure or CSS breaks a locator, the platform automatically substitutes it with another one. By automatically restoring locators in this manner, the tool effectively reduces the time you need to spend on test maintenance.
Automatic timeouts for asynchronous events and server interaction
Communication with a server is integral to end-to-end testing, which makes text execution delays necessary. Instead of making you handcraft explicit and implicit waits, Screenster determines optimal wait time for every element. Again, this reduces the maintenance footprint of your test suites.
Running tests in a continuous integration environment has become the default way of doing automated end-to-end testing. With the support for Jenkins, TeamCity, Travis, and Bamboo, Screenster meets the requirements of modern-day testing. CI support is implemented via a proprietary plugin that you can download from our website.
Like any other automation tool that belongs in 2017, Screenster is cloud-based. There’s a shared browser-based environment for the collaborative management of end-to-end testing suites, as well as a cloud server. This setup eliminates the need to instal and tinker with separate modules. Besides, you can opt for running your tests on a server installed on premise if this setup suits the nature of your project better.
Bottom line: automated end-to-end testing done right
The scope of end-to-end testing poses a major challenge for test automation engineers which is why having the right tool is so important. Screenster streamlines the automation of end-to-end testing to a point where it becomes cost-effective and simple enough for any company.
One way to see if Screenster can live up to this claim is to try our demo. Try to automate a simple test for your website and you’ll see that hand-coded tests are a thing of the past.
The idea of fully UI-driven end-to-end testing makes sense — is you can come up with the right test cases that exercise the system outside of the UI.
Your section on what is end-to-end testing mentions that it tests functionality and performance across all levels and subsystems. How can you do it then with what’s basically a UI testing tool?
I’d say that much of it depends on how you construct your end-to-end testing cases. A UI-centric test can include a user session that exercises the server side, third-party APIs, random data, etc
What’s up with you using Gmail for most of your examples ? (most of the end-to-end testing demos on youtube are with Gmail, for instance)
It’s just a good example. Gmail is a complex web application with lots of things going on under the hood. Besides, Gmail is a web application that most people can access. So, basically, anyone can do the same testing routine and see how Screenster works for them
Not sure if I’m going to want to use your tool for end to end testing, but it looks like a workable solution for UI testing. It works for my website and I’m curious to see how it’s going to handle something more complex.
You’re defining end to end testing the same way the Protractor website defines it. Were still talking about UI testing here, right?
Great post but needs a section on end to end testing best practices. Also, would be great to see a comparison on horizontal and vertical end-to-end testing.
Horizontal e2e testing (on UI and integration level): login, add to shopping cart, checkout, perform a transaction;
Vetical e2e testing: automating/running tests across all levels: from Unit, through Integration, to UI.
Would be awesome to see e2e testing compared to system testing and UAT. I know, an experience engineer wouldn’t really confuse the three, but then again, an experienced engineer wouldn’t confuse e2e and system testing. As for beginners, I witnessed quite a few cases when folks were unable to tell the difference
Looks like a good solution for e2e testing of a feature that’s already in production
Interesting approach to end to end automation. Easy to see how this can work