Automatic Visual Regression Testing on WordPress Themes

So the other day I discovered a plugin update broke one of my theme demo websites. The layout on this particular demo page was probably broken for a good week.

How can I stop this happening again? Automatic Visual Regression Testing!

This basically means “take screenshots of web pages and report any visual differences”. Cool hey!

Now there’s a bucket of tools out there to help you do this.

• browserstack
• selenium
• wraith
• nightwatchjs
• backstopjs
• screener.io
• shoov
• and many more

Well I tried a few and ended up using NighwatchJS with Selenuim and the chrome/gecko drivers.  I really wanted to use browserstack for the cross platform factor, however couldn’t easily access full page screenshots via their API. Hopefully that becomes easier down the track.

Here’s the end result, it integrated into my theme development pipeline quite nicely.

1) Nightwatch will tell Selenuim to spin up a bunch of headless browsers in Xvfb at various resolutions and capture the output. Here we’re capturing 9 pages on the demo website at 3 different resolutions:

2) When it runs again it will detect any visual changes and flag them for review. Here it is highlighting background graphic changes and some text changes half way down the second page:

3) Script Output:

4) Test configuration:

{
  "urls": {
    "https://verge-blog-demo.themes.dtbaker.net": {
      "/": "Home",
      "/sample-page-2/": "Pages",
      "/about/": "About",
      "/shop/": "Shop",
      "/product/faux-plant-brushy/": "Product",
      "/blog/": "Blog",
      "/2017/01/15/outdoor-photography/": "Post",
      "/contact/": "Contact",
      "/category/interior-design/": "Category"
    }
  }
}

The code will land up on github once I’ve cleaned it up a bit more.