Setup Image Comparison Service of WDIO to make UI Testing more impactful

In the previous article, we talked about WebdriverIO. A tool that has become popular very fast in the field of test automation due to its powerful advantages. In this article, we will talk about one of its advantages that has advanced our test automation suite. We have an application that helps our clients develop stunning websites. Whenever a new feature is developed that may impact the UI of the site, WebdriverIO can help us test the change in the UI of the sites. WebdriverIO provides a service called as wdio-image-comparison-service, which performs pixel-by-pixel image comparison.

We already have a wdio-based automation project that does end-to-end functional testing. We expanded it to screenshot test all the pages for thousands of sites. We explored various tools for image comparison. We then discovered that WebdriverIO furthermore offers an image comparison tool. Then we began looking into how we could incorporate this "wdio-image-comparison-service" into our already-running automation project.

Set-up

Referring to the official doc for wdio-image-comparison-service, we installed the module:

npm install --save wdio-image-comparison-service

We added the below options to the Test runner services in wdio.conf.js file:

services: [
  'chromedriver',
  [
    'image-comparison',
    // The options
    {
      // Some options, see the docs for more
      baselineFolder: join(process.cwd(), './tests/'),
      formatImageName: '{tag}-{logName}-{width}x{height}',
      screenshotPath: join(process.cwd(), '.tmp/'),
      savePerInstance: true,
      autoSaveBaseline: true,
      blockOutStatusBar: true,
      blockOutToolBar: true,
      disableCSSAnimation: false,
      ignoreColors: true,
      ignoreTransparentPixel: true,
    },
  ],
],

Above is the default setup. With this default setup, we thought that we had completed configuring the setup and the next step was to create a spec file to compare screenshots.

1. browser.saveScreen("homepage") function saves the screenshot as a baseline image with the name homepage.png.
2. browser.checkScreen("homepage") function saves the screenshot of the current page and compares it with the baseline image having the same name i.e. homepage.png.

However, we encountered an issue with the default setup. When we attempted to create a spec to compare screenshots of the two different doodles, it did not detect any mismatches, and the spec passed. We will explain this issue in more detail below:

We created a spec file called image_compare.js as follows:

describe("Image Comparison", () => {
  it("saves the baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
    await browser.saveScreen("homepage");
  });

  it("compares with baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");

    const result = await browser.checkScreen("homepage");

    expect(result).toEqual(0);
  });
});

In the above spec, we were comparing screenshots of the Teacher's Day - 2021 homepage and Teacher's Day - 2022 homepage. And we were expecting some mismatch percentage. But the spec got passed. When we investigated, we found that wdio-image-comparison-service saved the baseline image under .tmp folder. The .tmp folder has actual as a subfolder. Under the actual folder, it saved the baseline image.

In our case, when we ran the spec, it overrode Teacher's Day - 2021 homepage.png with Teacher's Day - 2022 homepage.png inside .tmp/ folder. Inside ./tests/ folder also, it saved Teacher's Day - 2022 homepage.png. When we investigated more on this, then we got to know that internally this package uses fs-extra plugin and it has limited Windows support. But, running the same script on Mac or Linux did not result in any issues or failures. We also found out that this is an existing issue. We have also added a comment there about what could be a workaround.

Workaround

As we previously said, the baseline picture is saved by 'wdio-image-comparison-service' by default under the '.tmp' folder. In 'wdio.conf.js', we removed the following code from the'services' section:

baselineFolder: join(process.cwd(), './tests/'),
screenshotPath: join(process.cwd(), '.tmp/'),

Then, we specified the testOptions in the it block where screenshots are compared.

const testOptions = {
  actualFolder: join(process.cwd(), './.tmp/checkActual'),
  baselineFolder: join(process.cwd(), './.tmp/actual'),
  diffFolder: join(process.cwd(), './.tmp/testDiff'),
  returnAllCompareData: true,
};

• checkActual: The current image will be saved in this folder. It is called as actualFolder.
• actual: The baseline image will be saved in this folder. It is called as baselineFolder.
• diffFolder: This image gets created with all the mismatches highlighted after the two screenshots are compared.
• returnAllCompareData: By default, check methods of this service returns only mismatch percentage. When this option is passed as true, then it returns the current image filename, folders that contain the path of actual, baseline, and diff, and misMatchPercentage.

After these modifications, the updated code is as follows:

wdio.conf.js

services: [
  'chromedriver',
  [
    'image-comparison',
    // The options
    {
      // Some options, see the docs for more
      savePerInstance: true,
      autoSaveBaseline: true,
      blockOutStatusBar: true,
      blockOutToolBar: true,
      disableCSSAnimation: false,
      ignoreColors: true,
      ignoreTransparentPixel: true,
    },
  ],
],

image_compare.js

import { join } from 'path';

describe("Image Comparison", () => {
  it("saves the baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
    await browser.saveScreen("homepage");
  });

  it("compares with baseline image", async () => {
    await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");

    const testOptions = {
      actualFolder: join(process.cwd(), './.tmp/checkActual'),
      baselineFolder: join(process.cwd(), './.tmp/actual'),
      diffFolder: join(process.cwd(), './.tmp/testDiff'),
      returnAllCompareData: true,
    };

    const result = await browser.checkScreen("homepage", testOptions);

    expect(result).toEqual(0);
  });
});

When we passed testOptions as a parameter to the checkScreen function, then wdio easily identified where our baseline images are saved and with which image to compare.

Now, if you run the above spec, then it will properly do the comparison, highlight the mismatch, and return the mismatch percentage.

Teacher's Day - 2021	Teacher's Day - 2022
Image Difference

The wdio-image-comparison-service is not just limited to capturing and comparing screens, it can also compare the full page screen, and elements, and also checks if the website supports the keyboard's TAB key. Please refer to wdio-image-comparison-service options.

Mobile Screenshot Testing

If screenshot testing for mobile devices is needed, then just resizing your browser is never a good idea because this module compares visuals of what the end user is going to see. Wdio Devtools can be helpful in such cases. Wdio Devtools is in itself a distinct, extensive subject that offers additional benefits.

We just need to pass service as devtools in wdio.conf.js file. We can emulate a specific device by await browser.emulateDevice('iPhone X'). Please refer to Device emulation.

We have created a small demo project wdio-image-comparison-demo that you can clone for better understanding. You can explore other options available for wdio-image-comparison-service.

References:

• Image Comparison (Visual Regression Testing) Service
• wdio-image-comparison-service options
• wdio-image-comparison-service issue
• Wdio Devtools Device Emulation