CI for CSS regression testing with Screenster

Now that the CI integration plugin has been successfully rolled out, it is time to recall how it all began… Ready for a tale about defeating a horde of CSS regression testing challenges with the help of a Jenkins server and a visual QA automation tool?

Why did we need CSS regression tests on our CI

Our team has realized the necessity of automating the visual regression testing process last winter, when our other product AjaxSwing gained a significant number of new users, who were actively requesting improvements. A new version was released each week, sometimes even more often. AjaxSwing generates web UI for desktop applications so you can imagine how much visual UI verification we had to accomplish each time a build was ready. It just seemed never-ending. We had unit tests in our continuous integration environment running against every new build, but they didn’t help with verifying the visual aspects of the UI such as CSS, formatting and layouts. That’s when our QA team decided it is time to try our new automation testing tool Screenster in real action, despite it being in its early Beta at the time.

visual regression testing continuous integration

We upgraded our CI environment to the following state:

  1. A test server with AjaxSwing repository on GitLab
  2. Jenkins server running on the same test server
  3. A separate Screenster server with the last stable version of Screenster installed there — typically it is the latest released build
  4. Automated CSS regression tests recorded on this stable server which perform visual verification of our AjaxSwing apps belonging to the regression scope.

We are running visual regression testing jobs for AjaxSwing either manually or by pre-set time — normally it is 3 — 5 times a day. A job includes the following operations:

  1. Pulls all changes from a selected AjaxSwing branch to the repository on the test server
  2. Prepares a build based on this branch
  3. Launches CSS regression tests on the stable Screenster server
  4. Sends a message with the test status to a person assigned to visual regression testing task after every finished test run.

Visual regression testing visual baseline CSS

The example above demonstrates how Screenster reports visual bugs in web UI. A screenshot captured during the latest run around 16:31 (see the time on the left page) contains differences from the baseline screenshot captured during the first run, around 16:23 (on the right page). These differences are marked with red color in order to alert a tester, who monitors the test results, about detected bugs.

Screenster saves the day!

Thanks to Screenster, today we have 98% fewer cases of our AjaxSwing users reporting visual issues caused by our changes. Moreover, we have recently started testing Screenster itself with Screenster! Visual regression testing process is the same here: a special job on Jenkins server creates a Screenster build with latest changes. After that, a test located at the stable server emulates user actions against this fresh Screenster including:

  • Recording a visual regression test
  • Running a test
  • Editing a test (adding & deleting steps, adding ignore regions an approving steps)
  • Overriding a test with new parameters (with the help of Screenster’s Extend feature)

This is a good example how Screenster, a CSS regression testing tool, well compatible with CI, enhances the web development team’s productivity! And what is your experience with automated visual tests on CI? Leave your comments and questions below!

Want to try Screenster on the cloud?

Try Online

 

CI for CSS regression testing with Screenster was last modified: September 6th, 2018 by Michael Tomara
Thoughts on “CI for CSS regression testing with Screenster”
  1. All I have to say is, CSS is painfull. If this method allows for adding CSS regression testing to CI, I’m in

  2. What CI tools does Screenster support? I’m specifically interested in TeamCity because I’ll probably be working with this one in the next few months

  3. Hi George!
    Screenster supports TeamCity, Jenkins, Bamboo, and Travis. Basically, we tried to make sure you can use Screenster with all go-to solutions for CI

  4. Speaking of CSS regression testing, wouldn’t it be a more sane approach to just compare the calculated styles for each element ? This is legit question because I haven’t seen anyone try it.

  5. You really can limit a CSS test to comparing the text contents of the .css files because there are rendering differences. Also, you’ll find people rewriting CSS really often at the early stages of development, so tests of this sort would be kind of pointless.

  6. I get a feeling that Jenkins has long become an industry standard and every testing tool should support it.

Leave a Reply

Your email address will not be published. Required fields are marked *

WordPress Image Lightbox Plugin