Nico Vogel

Posted on Sep 14, 2023

Building Visual Regression Testing for Components with Storybook and BackstopJS

#tutorial #webdev #testing

Reviewing changes to UI components can be a time-consuming task, especially if you want to ensure that everything looks just right.
Traditionally, this process involves running the application locally to manually inspect each alteration, a meticulous and sometimes arduous endeavor.
Wouldn't it be great if you could automatically see which components changed in a pull request without having to run the application locally?
Visual regression testing can help with just that.

What are Visual Regression Tests?

Visual regression tests are a way to detect visual changes in your application's user interface.
They allow you to easily identify any visual changes introduced by code changes without running it locally.

Here's a step-by-step overview of how visual regression tests can work:

Screenshot Components from the Main Branch: You take screenshots of your application's UI components as they appear in the main branch.
Screenshot Components from the Pull Request: Next, you capture screenshots of the same UI components from the pull request branch.
Compare Screenshots: A tool compares these screenshots pixel by pixel, highlighting any differences between them.
Generate a Report: Finally, you receive a report that clearly shows any visual changes, making it easy to review and address them.

Now, let's explore how you can custom-build your visual regression testing setup using Storybook, Playwright, and BackstopJS.

Create Your Own Visual Regression Testing Tool

This blog assumes that you have Storybook already setup and ready to use and that you also use the addon-docs plugin.

GitHub Link: If you're interested in the code right away, you can find it at NicoVogel/nx-lit-vite.

Please note that the code in the repository is a bit more oriented towards the lit framework, while the code presented here is more generalized.

Steps to Create Your Visual Regression Testing Tool

1. Export Stories JSON from Storybook

To automate the process, you need to export a stories.json file from Storybook.
If you are using Storybook version 7 or later, you can enable the buildStoriesJson feature.
For earlier versions, use the sb extract command.

// .storybook/main.ts (Storybook v7+)
export default {
  // ...
  features: {
    buildStoriesJson: true,
  },
  // ...
};

You have two options regarding the stories.json file.
Either you check it into your repository, or you generate it as part of your build process.

Before Storybook introduced the buildStoriesJson feature, it was a good idea to check the file in, to safe time.
However, now it just comes down to personal preference.

Because the sb extract command takes a while to

Note: Make sure to include the stories.json file in your repository, especially for older Storybook versions.

2. Write a Selector Extraction Script

BackstopJS uses scenarios to identify what parts of a page should be captured in a screenshot.
Each scenario requires a URL and a selector.
So, we need to extract the selectors from the stories.json file.

The URL is simply the address visited to take the screenshot, while the selector is a CSS selector that identifies the element to capture in the screenshot.

In case of Storybook, all individual component showcases are rendered in an iframe.
This page is hosted at /iframe.html and which component is shown is determined by the query parameter.

The selector code is for Storybook version 7. For other versions, you need to inspect the Storybook iframe.html page to find the suitable selector.

// apps/visual-regression/src/main.ts

import storyJSON from './assets/stories.json';

function constructScenarios(stories: typeof storyJSON, baseUrl: string) {
  console.assert(baseUrl.endsWith('/'), 'baseUrl should not end with a /', baseUrl);

  return Object.values(stories.stories)
    .filter(
      story =>
        // since version 7, the docs are an overview page and as we are only interested in the stories, we filter them out
        story.tags.includes('story') &&
        // remove blacklisted stories (e.g. loading spinner, what ever is not static)
        !config.user.blacklistStories.includes(story.id)
    )
    .map(entry => {
      const {title, name, id} = entry;
      return {
        ...config.backstopCapture.scenarioDefault,
        // backstop waits for this to appear, before taking the screenshot
        readySelector: '#root-inner',
        label: `${title} ${name}`,
        // custom url is required to isolate the storybook component in its own frame
        url: `${baseUrl}/iframe.html?viewMode=docs&id=${id}`,
        selectors: ['#root-inner'],
        misMatchThreshold: config.user.misMatchThresholdInPercentage,
      } satisfies Scenario;
    })
    .filter(defined);
}

function defined<T>(x: T | undefined): x is T {
  return Boolean(x);
}

// apps/visual-regression/src/config.ts
export const config = {
  user: {
    blacklist: ['LoadingSpinner'],
    misMatchThresholdInPercentage: 0.1,
  },
  backstopCapture: {
    scenarioDefault: {
      selectorExpansion: true,
      requireSameDimensions: true,
    },
  },
};

3. Configure Backstop to Use Playwright

Configure BackstopJS to use Playwright and pass along the parsed scenarios.

The console.log statements make it easier to reason about problems in CI.
Even though, I did not experience any problems in CI, I still left them in.

// apps/visual-regression/src/main.ts

import {Config, Scenario} from 'backstopjs';

async function main(backstopExecutionMode: string) {
  //...

  const scenarios = constructScenarios(storyJSON, location);
  console.log(
    'scenarios',
    scenarios.map(({label, url, selectors}) => ({label, url, selectors}))
  );

  const config = constructBackstopConfig(scenarios);
  console.log('backstop config', config);

  // ...
}

function constructBackstopConfig(scenarios: Scenario[]): Config {
  // checkout the file `apps/visual-regression/src/config.ts` for the full config
  const {base: engine_scripts, ...scripts} = config.backstopCapture.scripts;
  return {
    ...config.backstopCapture.browser,
    ...scripts,
    id: config.backstopCapture.id,
    viewports: config.backstopCapture.viewPorts,
    paths: {
      ...config.backstopCapture.locations,
      engine_scripts,
    },
    scenarios,
  };
}

4. Run Backstop

Run BackstopJS with reference in the case of the main branch and with test in the case of a pull request.

// apps/visual-regression/src/main.ts
import backstop, {Config, Scenario} from 'backstopjs';

const [, , backstopExecutionMode] = process.argv;
main(backstopExecutionMode);

async function main(backstopExecutionMode: string) {
  const {action, label, location} = getAction(backstopExecutionMode);
  console.log(`task: '${action}' with location: '${location}'`);

  const scenarios = constructScenarios(storyJSON, location);
  console.log(
    'scenarios',
    scenarios.map(({label, url, selectors}) => ({label, url, selectors}))
  );

  const config = constructBackstopConfig(scenarios);
  console.log('backstop config', config);

  backstop(action, {
    config,
  })
    .then(() => {
      console.log(`${label} completed`);
    })

    .catch(err => {
      console.log(`${label} failed, because: `, err);
      process.exit(1);
    });
}

function getAction(mode: string) {
  switch (mode) {
    case 'reference':
      return {
        action: 'reference',
        label: 'referencing',
        location: `http://localhost:${config.buildProcess.locations.referenceServePort}`,
      } as const;
    case 'test':
    default:
      return {
        action: 'test',
        label: 'testing',
        location: `http://localhost:${config.buildProcess.locations.changedServePort}`,
      } as const;
  }
}

5. Define Scripts to Run Everything

Now that you have the code to run BackstopJS, define the default scripts to run it.
This example uses Nx, but you can adapt it to your own setup.

// apps/visual-regression/project.json
{
  // nx does not know that we rely on this project, so we need to tell it
  "implicitDependencies": ["@nx-lit-vite/source"],
  "targets": {
    "reference": {
      "executor": "nx:run-commands",
      "options": {
        "command": "pnpm reference",
        "cwd": "apps/visual-regression"
      },
      // Ensure that the Storybook build is finished before running visual regression tests
      "dependsOn": ["^vr-reference"],
      "configurations": {
        "ci": {
          "command": "pnpm reference:backstop"
        }
      }
    },
    "check-changed": {
      "executor": "nx:run-commands",
      "options": {
        "command": "pnpm check-changed",
        "cwd": "apps/visual-regression"
      },
      "dependsOn": ["^vr-changed"],
      "configurations": {
        "ci": {
          "command": "pnpm check-changed:backstop"
        }
      }
    },
    "extract-stories": {
      "executor": "nx:run-commands",
      "options": {
        "command": "cp dist/storybook/@nx-lit-vite/source/stories.json apps/visual-regression/src/assets/stories.json"
      },
      "dependsOn": ["^build-storybook"]
    }
  }
}

// project.json (wherever you define your Storybook build)
{
  "name": "@nx-lit-vite/source",
  "targets": {
    "build-storybook": {
      "executor": "@nx/storybook:build",
      "outputs": ["{options.outputDir}"],
      "options": {
        "outputDir": "dist/storybook/@nx-lit-vite/source",
        "configDir": "./.storybook"
      },
      "configurations": {
        "ci": {
          "quiet": true
        }
      }
    },
    "vr-reference": {
      "executor": "nx:run-commands",
      "outputs": ["{options.outputDir}"],
      "options": {
        // you can override configurations via flags, if they are named the same
        // so here, we override the outputDir wit a custom output
        "command": "pnpm exec nx build-storybook @nx-lit-vite/source --configuration=ci --outputDir=/tmp/reference --skip-nx-cache",
        "outputDir": "tmp/reference"
      }
    },
    "vr-changed": {
      "executor": "nx:run-commands",
      "outputs": ["{options.outputDir}"],
      "options": {
        // skipping the nx cache to be really sure that we have the latest changes
        "command": "pnpm exec nx build-storybook @nx-lit-vite/source --configuration=ci --outputDir=/tmp/changed --skip-nx-cache",
        "outputDir": "tmp/changed"
      }
    }
  }
}

// apps/visual-regression/package.json
{
  "scripts": {
    "reference": "concurrently --success first --kill-others npm:reference:backstop npm:reference:serve",
    // the timeout is especially important in CI, in case something goes wrong.
    // otherwise `wait-on` will never fail
    "reference:backstop": "wait-on http://localhost:8080 --timeout 3000 && ts-node src/main.ts reference",
    "reference:serve": "http-server /tmp/reference --port 8080",
    "check-changed": "concurrently --success first --kill-others npm:check-changed:backstop npm:check-changed:serve",
    // using ts-node to skip the need for a build step
    "check-changed:backstop": "wait-on http://localhost:8081 --timeout 3000 &&  ts-node src/main.ts test",
    "check-changed:serve": "http-server /tmp/changed --port 8081",
    // if you want to see the local output
    "serve-result": "http-server /tmp/backstop_data --port 8080"
  }
}

The setup allows for two different modes: running in CI (assuming Storybook is already hosted) and running locally.
To set up these scripts, you'll need dependencies like concurrently, http-server, wait-on, ts-node, backstopjs, playwright, @types/backstopjs, typescript, and more.
You can install them with the following pnpm command:

pnpm add -D concurrently http-server wait-on ts-node backstopjs playwright @types/backstopjs typescript

Make sure backstopjs and playwright versions are compatible.

6. Set Up GitHub Actions

To automate the process, set up GitHub Actions to run the visual regression tests automatically with each pull request.
The following steps are involved:

Checkout the main branch.
Install dependencies.
Build Storybook.
Host Storybook.
Run BackstopJS reference.
Checkout the pull request branch.
Install dependencies.
Build Storybook.
Host Storybook.
Run BackstopJS test.
Upload the report to GitHub.

name: Visual Regression

on:
  pull_request:
    types: [opened, synchronize]

permissions:
  contents: read
  checks: write # junit-report

jobs:
  backstop:
    name: Build target and reference and test
    runs-on: ubuntu-latest
    env:
      # BackstopJS installs puppeteer and it wants to install chrome.
      # But we already use playwright for that, so we don't need it.
      PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: true
    steps:
      - name: Checkout Reference 🛎️
        uses: actions/checkout@v3.5.3
        with:
          fetch-depth: 0
          ref: ${{ github.base_ref }}
      - name: Setup pnpm ⚙️
        run: |
          npm i -g pnpm@8.6.12
      - name: Install for Reference 📦
        run: |
          pnpm install
      - name: Host Reference
        uses: Eun/http-server-action@v1.0.10
        with:
          directory: /tmp/reference
          port: 8080
          no-cache: true
          index-files: |
            ["index.html"]
      - name: Host Changed
        uses: Eun/http-server-action@v1.0.10
        with:
          directory: /tmp/changed
          port: 8081
          no-cache: true
          index-files: |
            ["index.html"]
      - name: Visual Regression Reference 📖
        run: |
          pnpm exec nx run visual-regression:reference --configuration=ci
      - name: Checkout Target 🛎️
        uses: actions/checkout@v3.5.3
        with:
          ref: ${{ github.head_ref }}
      - name: Install for Target📦
        run: |
          pnpm install
      - name: Visual Regression Target 📖
        run: |
          pnpm exec nx run visual-regression:check-changed --configuration=ci
      - name: Post report
        uses: mikepenz/action-junit-report@v3.8.0
        id: post-report
        with:
          report_paths: /tmp/backstop_data/ci_report/*.xml
          fail_on_failure: true

Visual Regression Output Look and Feel

Let's explore the output generated by this setup and understand what it provides.
After executing the reference and check-changed scripts, you'll find the report stored in the /tmp/backstop_data/ directory.
To view it locally, use the serve-result script included in the visual regression project's package.json.

Upon opening the html_report folder, you'll see an overview of visual regression tests results:

When you click on a specific test, you'll focus on that test and get a convenient scrubber tool that allows you to compare the two screenshots:

With this simple report, you can quickly assess the visual changes introduced by a pull request.

Potential Improvements

As you continue to refine your visual regression testing setup, consider these potential improvements:

Cache the Reference: Cache the reference screenshots from the main branch to speed up the testing process.
Ensure you have cancellation and restart logic in place to rerun PRs whenever the reference changes.
Upload Reports to Storage: Instead of just uploading junit reports to GitHub, consider using a storage solution to store reports and make them accessible to reviewers.
Don't forget to clean up old reports after a certain period, e.g., 30 days.
Host the Report: Host the report online so that reviewers do not need to run them locally to see the changes.
This can be an alternative or complementary approach to uploading reports to storage.
Action to Create stories.json PR: If you check-in the stories.json file, you can create an action that automatically creates a PR whenever the file changes, ensuring that the stories.json file is always up-to-date.
Consider Theme Variants: If your application supports multiple themes, you may need to capture screenshots for each theme.

Key Learnings

Throughout this process, several key learnings emerged:

Selectors: Using #root-selector ensures that everything is captured in the screenshot.
While you can create more specific selectors to capture only the component itself, it adds complexity without providing additional value.
Storybook Hosting: Building and hosting Storybook for efficient visual regression testing is crucial.
Using the serve command is considerably slower.
Visual Regression as a Pull Request Gate: Visual regression tests are valuable but can't be mandatory gates.
Sometimes, visual changes are intended and expected.
It's just another tool to help you review changes.
Hosting in GitHub Actions: Running a simple HTTP server with npx http-server doesn't work in GitHub Actions, so use predefined actions for hosting.
Breaking Changes with Version Bumps: Be aware that updating Storybook, BackstopJS, or Playwright may break your visual regression tests.
For example, when updating Storybook from version 6 to 7, the selectors changed, which broke the visual regression tests.
It was an easy fix, but something to keep in mind when building your own tooling.
Another aspect here is that BackstopJS uses playwright, so you need to find a compatible version.
Blacklisting: In some cases, you may want to blacklist certain components, such as loading spinners or non-static elements.
Playwright Version: Ensure that the Playwright version is compatible with the BackstopJS version.

Conclusion

Building your own visual regression testing setup is not as challenging as it may seem.
Once established, you can reuse it across multiple projects with only minimal adjustments.
The result is an easier pull request review process and increased confidence in the visual integrity of your applications.
Additionally, it makes it easy to see visual changes by using the slider feature in the report.

Top comments (1)

Greg Bergé • Dec 28 '23

Very interesting! Have you considered using Argos for visual testing?

DEV Community