DEV Community: Nico Vogel

Building Visual Regression Testing for Components with Storybook and BackstopJS

Nico Vogel — Thu, 14 Sep 2023 06:31:16 +0000

Reviewing changes to UI components can be a time-consuming task, especially if you want to ensure that everything looks just right.
Traditionally, this process involves running the application locally to manually inspect each alteration, a meticulous and sometimes arduous endeavor.
Wouldn't it be great if you could automatically see which components changed in a pull request without having to run the application locally?
Visual regression testing can help with just that.

What are Visual Regression Tests?

Visual regression tests are a way to detect visual changes in your application's user interface.
They allow you to easily identify any visual changes introduced by code changes without running it locally.

Here's a step-by-step overview of how visual regression tests can work:

Screenshot Components from the Main Branch: You take screenshots of your application's UI components as they appear in the main branch.
Screenshot Components from the Pull Request: Next, you capture screenshots of the same UI components from the pull request branch.
Compare Screenshots: A tool compares these screenshots pixel by pixel, highlighting any differences between them.
Generate a Report: Finally, you receive a report that clearly shows any visual changes, making it easy to review and address them.

Now, let's explore how you can custom-build your visual regression testing setup using Storybook, Playwright, and BackstopJS.

Create Your Own Visual Regression Testing Tool

This blog assumes that you have Storybook already setup and ready to use and that you also use the addon-docs plugin.

GitHub Link: If you're interested in the code right away, you can find it at NicoVogel/nx-lit-vite.

Please note that the code in the repository is a bit more oriented towards the lit framework, while the code presented here is more generalized.

Steps to Create Your Visual Regression Testing Tool

1. Export Stories JSON from Storybook

To automate the process, you need to export a stories.json file from Storybook.
If you are using Storybook version 7 or later, you can enable the buildStoriesJson feature.
For earlier versions, use the sb extract command.



// .storybook/main.ts (Storybook v7+)
export default {
  // ...
  features: {
    buildStoriesJson: true,
  },
  // ...
};

You have two options regarding the stories.json file.
Either you check it into your repository, or you generate it as part of your build process.

Before Storybook introduced the buildStoriesJson feature, it was a good idea to check the file in, to safe time.
However, now it just comes down to personal preference.

Because the sb extract command takes a while to

Note: Make sure to include the stories.json file in your repository, especially for older Storybook versions.

2. Write a Selector Extraction Script

BackstopJS uses scenarios to identify what parts of a page should be captured in a screenshot.
Each scenario requires a URL and a selector.
So, we need to extract the selectors from the stories.json file.

The URL is simply the address visited to take the screenshot, while the selector is a CSS selector that identifies the element to capture in the screenshot.

In case of Storybook, all individual component showcases are rendered in an iframe.
This page is hosted at /iframe.html and which component is shown is determined by the query parameter.

The selector code is for Storybook version 7. For other versions, you need to inspect the Storybook iframe.html page to find the suitable selector.



// apps/visual-regression/src/main.ts

import storyJSON from './assets/stories.json';

function constructScenarios(stories: typeof storyJSON, baseUrl: string) {
  console.assert(baseUrl.endsWith('/'), 'baseUrl should not end with a /', baseUrl);

  return Object.values(stories.stories)
    .filter(
      story =>
        // since version 7, the docs are an overview page and as we are only interested in the stories, we filter them out
        story.tags.includes('story') &&
        // remove blacklisted stories (e.g. loading spinner, what ever is not static)
        !config.user.blacklistStories.includes(story.id)
    )
    .map(entry => {
      const {title, name, id} = entry;
      return {
        ...config.backstopCapture.scenarioDefault,
        // backstop waits for this to appear, before taking the screenshot
        readySelector: '#root-inner',
        label: `${title} ${name}`,
        // custom url is required to isolate the storybook component in its own frame
        url: `${baseUrl}/iframe.html?viewMode=docs&id=${id}`,
        selectors: ['#root-inner'],
        misMatchThreshold: config.user.misMatchThresholdInPercentage,
      } satisfies Scenario;
    })
    .filter(defined);
}

function defined<T>(x: T | undefined): x is T {
  return Boolean(x);
}

// apps/visual-regression/src/config.ts
export const config = {
  user: {
    blacklist: ['LoadingSpinner'],
    misMatchThresholdInPercentage: 0.1,
  },
  backstopCapture: {
    scenarioDefault: {
      selectorExpansion: true,
      requireSameDimensions: true,
    },
  },
};

3. Configure Backstop to Use Playwright

Configure BackstopJS to use Playwright and pass along the parsed scenarios.

The console.log statements make it easier to reason about problems in CI.
Even though, I did not experience any problems in CI, I still left them in.



// apps/visual-regression/src/main.ts

import {Config, Scenario} from 'backstopjs';

async function main(backstopExecutionMode: string) {
  //...

  const scenarios = constructScenarios(storyJSON, location);
  console.log(
    'scenarios',
    scenarios.map(({label, url, selectors}) => ({label, url, selectors}))
  );

  const config = constructBackstopConfig(scenarios);
  console.log('backstop config', config);

  // ...
}

function constructBackstopConfig(scenarios: Scenario[]): Config {
  // checkout the file `apps/visual-regression/src/config.ts` for the full config
  const {base: engine_scripts, ...scripts} = config.backstopCapture.scripts;
  return {
    ...config.backstopCapture.browser,
    ...scripts,
    id: config.backstopCapture.id,
    viewports: config.backstopCapture.viewPorts,
    paths: {
      ...config.backstopCapture.locations,
      engine_scripts,
    },
    scenarios,
  };
}

4. Run Backstop

Run BackstopJS with reference in the case of the main branch and with test in the case of a pull request.



// apps/visual-regression/src/main.ts
import backstop, {Config, Scenario} from 'backstopjs';

const [, , backstopExecutionMode] = process.argv;
main(backstopExecutionMode);

async function main(backstopExecutionMode: string) {
  const {action, label, location} = getAction(backstopExecutionMode);
  console.log(`task: '${action}' with location: '${location}'`);

  const scenarios = constructScenarios(storyJSON, location);
  console.log(
    'scenarios',
    scenarios.map(({label, url, selectors}) => ({label, url, selectors}))
  );

  const config = constructBackstopConfig(scenarios);
  console.log('backstop config', config);

  backstop(action, {
    config,
  })
    .then(() => {
      console.log(`${label} completed`);
    })

    .catch(err => {
      console.log(`${label} failed, because: `, err);
      process.exit(1);
    });
}

function getAction(mode: string) {
  switch (mode) {
    case 'reference':
      return {
        action: 'reference',
        label: 'referencing',
        location: `http://localhost:${config.buildProcess.locations.referenceServePort}`,
      } as const;
    case 'test':
    default:
      return {
        action: 'test',
        label: 'testing',
        location: `http://localhost:${config.buildProcess.locations.changedServePort}`,
      } as const;
  }
}

5. Define Scripts to Run Everything

Now that you have the code to run BackstopJS, define the default scripts to run it.
This example uses Nx, but you can adapt it to your own setup.



// apps/visual-regression/project.json
{
  // nx does not know that we rely on this project, so we need to tell it
  "implicitDependencies": ["@nx-lit-vite/source"],
  "targets": {
    "reference": {
      "executor": "nx:run-commands",
      "options": {
        "command": "pnpm reference",
        "cwd": "apps/visual-regression"
      },
      // Ensure that the Storybook build is finished before running visual regression tests
      "dependsOn": ["^vr-reference"],
      "configurations": {
        "ci": {
          "command": "pnpm reference:backstop"
        }
      }
    },
    "check-changed": {
      "executor": "nx:run-commands",
      "options": {
        "command": "pnpm check-changed",
        "cwd": "apps/visual-regression"
      },
      "dependsOn": ["^vr-changed"],
      "configurations": {
        "ci": {
          "command": "pnpm check-changed:backstop"
        }
      }
    },
    "extract-stories": {
      "executor": "nx:run-commands",
      "options": {
        "command": "cp dist/storybook/@nx-lit-vite/source/stories.json apps/visual-regression/src/assets/stories.json"
      },
      "dependsOn": ["^build-storybook"]
    }
  }
}

// project.json (wherever you define your Storybook build)
{
  "name": "@nx-lit-vite/source",
  "targets": {
    "build-storybook": {
      "executor": "@nx/storybook:build",
      "outputs": ["{options.outputDir}"],
      "options": {
        "outputDir": "dist/storybook/@nx-lit-vite/source",
        "configDir": "./.storybook"
      },
      "configurations": {
        "ci": {
          "quiet": true
        }
      }
    },
    "vr-reference": {
      "executor": "nx:run-commands",
      "outputs": ["{options.outputDir}"],
      "options": {
        // you can override configurations via flags, if they are named the same
        // so here, we override the outputDir wit a custom output
        "command": "pnpm exec nx build-storybook @nx-lit-vite/source --configuration=ci --outputDir=/tmp/reference --skip-nx-cache",
        "outputDir": "tmp/reference"
      }
    },
    "vr-changed": {
      "executor": "nx:run-commands",
      "outputs": ["{options.outputDir}"],
      "options": {
        // skipping the nx cache to be really sure that we have the latest changes
        "command": "pnpm exec nx build-storybook @nx-lit-vite/source --configuration=ci --outputDir=/tmp/changed --skip-nx-cache",
        "outputDir": "tmp/changed"
      }
    }
  }
}

// apps/visual-regression/package.json
{
  "scripts": {
    "reference": "concurrently --success first --kill-others npm:reference:backstop npm:reference:serve",
    // the timeout is especially important in CI, in case something goes wrong.
    // otherwise `wait-on` will never fail
    "reference:backstop": "wait-on http://localhost:8080 --timeout 3000 && ts-node src/main.ts reference",
    "reference:serve": "http-server /tmp/reference --port 8080",
    "check-changed": "concurrently --success first --kill-others npm:check-changed:backstop npm:check-changed:serve",
    // using ts-node to skip the need for a build step
    "check-changed:backstop": "wait-on http://localhost:8081 --timeout 3000 &&  ts-node src/main.ts test",
    "check-changed:serve": "http-server /tmp/changed --port 8081",
    // if you want to see the local output
    "serve-result": "http-server /tmp/backstop_data --port 8080"
  }
}

The setup allows for two different modes: running in CI (assuming Storybook is already hosted) and running locally.
To set up these scripts, you'll need dependencies like concurrently, http-server, wait-on, ts-node, backstopjs, playwright, @types/backstopjs, typescript, and more.
You can install them with the following pnpm command:



pnpm add -D concurrently http-server wait-on ts-node backstopjs playwright @types/backstopjs typescript

Make sure backstopjs and playwright versions are compatible.

6. Set Up GitHub Actions

To automate the process, set up GitHub Actions to run the visual regression tests automatically with each pull request.
The following steps are involved:

Checkout the main branch.
Install dependencies.
Build Storybook.
Host Storybook.
Run BackstopJS reference.
Checkout the pull request branch.
Install dependencies.
Build Storybook.
Host Storybook.
Run BackstopJS test.
Upload the report to GitHub.



name: Visual Regression

on:

  pull_request:

    types: [opened, synchronize]

permissions:

  contents: read

  checks: write # junit-report

jobs:

  backstop:

    name: Build target and reference and test

    runs-on: ubuntu-latest

    env:

      # BackstopJS installs puppeteer and it wants to install chrome.

      # But we already use playwright for that, so we don't need it.

      PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: true

    steps:

      - name: Checkout Reference 🛎️

        uses: actions/checkout@v3.5.3

        with:

          fetch-depth: 0

          ref: ${{ github.base_ref }}

      - name: Setup pnpm ⚙️

        run: |

          npm i -g pnpm@8.6.12

      - name: Install for Reference 📦

        run: |

          pnpm install

      - name: Host Reference

        uses: Eun/http-server-action@v1.0.10

        with:

          directory: /tmp/reference

          port: 8080

          no-cache: true

          index-files: |

            ["index.html"]

      - name: Host Changed

        uses: Eun/http-server-action@v1.0.10

        with:

          directory: /tmp/changed

          port: 8081

          no-cache: true

          index-files: |

            ["index.html"]

      - name: Visual Regression Reference 📖

        run: |

          pnpm exec nx run visual-regression:reference --configuration=ci

      - name: Checkout Target 🛎️

        uses: actions/checkout@v3.5.3

        with:

          ref: ${{ github.head_ref }}

      - name: Install for Target📦

        run: |

          pnpm install

      - name: Visual Regression Target 📖

        run: |

          pnpm exec nx run visual-regression:check-changed --configuration=ci

      - name: Post report

        uses: mikepenz/action-junit-report@v3.8.0

        id: post-report

        with:

          report_paths: /tmp/backstop_data/ci_report/*.xml

          fail_on_failure: true

Visual Regression Output Look and Feel

Let's explore the output generated by this setup and understand what it provides.
After executing the reference and check-changed scripts, you'll find the report stored in the /tmp/backstop_data/ directory.
To view it locally, use the serve-result script included in the visual regression project's package.json.

Upon opening the html_report folder, you'll see an overview of visual regression tests results:

When you click on a specific test, you'll focus on that test and get a convenient scrubber tool that allows you to compare the two screenshots:

With this simple report, you can quickly assess the visual changes introduced by a pull request.

Potential Improvements

As you continue to refine your visual regression testing setup, consider these potential improvements:

Cache the Reference: Cache the reference screenshots from the main branch to speed up the testing process.
Ensure you have cancellation and restart logic in place to rerun PRs whenever the reference changes.
Upload Reports to Storage: Instead of just uploading junit reports to GitHub, consider using a storage solution to store reports and make them accessible to reviewers.
Don't forget to clean up old reports after a certain period, e.g., 30 days.
Host the Report: Host the report online so that reviewers do not need to run them locally to see the changes.
This can be an alternative or complementary approach to uploading reports to storage.
Action to Create stories.json PR: If you check-in the stories.json file, you can create an action that automatically creates a PR whenever the file changes, ensuring that the stories.json file is always up-to-date.
Consider Theme Variants: If your application supports multiple themes, you may need to capture screenshots for each theme.

Key Learnings

Throughout this process, several key learnings emerged:

Selectors: Using #root-selector ensures that everything is captured in the screenshot.
While you can create more specific selectors to capture only the component itself, it adds complexity without providing additional value.
Storybook Hosting: Building and hosting Storybook for efficient visual regression testing is crucial.
Using the serve command is considerably slower.
Visual Regression as a Pull Request Gate: Visual regression tests are valuable but can't be mandatory gates.
Sometimes, visual changes are intended and expected.
It's just another tool to help you review changes.
Hosting in GitHub Actions: Running a simple HTTP server with npx http-server doesn't work in GitHub Actions, so use predefined actions for hosting.
Breaking Changes with Version Bumps: Be aware that updating Storybook, BackstopJS, or Playwright may break your visual regression tests.
For example, when updating Storybook from version 6 to 7, the selectors changed, which broke the visual regression tests.
It was an easy fix, but something to keep in mind when building your own tooling.
Another aspect here is that BackstopJS uses playwright, so you need to find a compatible version.
Blacklisting: In some cases, you may want to blacklist certain components, such as loading spinners or non-static elements.
Playwright Version: Ensure that the Playwright version is compatible with the BackstopJS version.

Conclusion

Building your own visual regression testing setup is not as challenging as it may seem.
Once established, you can reuse it across multiple projects with only minimal adjustments.
The result is an easier pull request review process and increased confidence in the visual integrity of your applications.
Additionally, it makes it easy to see visual changes by using the slider feature in the report.

Wrapping Libraries for Enhanced Flexibility: A Practical Approach

Nico Vogel — Fri, 08 Sep 2023 14:36:54 +0000

In the realm of software development, third-party libraries are invaluable tools for accelerating projects.
However, these libraries might not perfectly align with evolving requirements.
Enter the concept of wrapping libraries — an approach that adds a layer of control and adaptability.

In this blog, we'll delve into library wrapping, its benefits, challenges, and provide an example.
Let's explore how wrapping libraries empowers you to meet dynamic demands in the coding world.

Code can be found here: NicoVogel/logger-wrapper

What Does it Mean to Wrap Libraries You Use?

At its core, library wrapping involves crafting a bridge between your application code and external libraries.
This bridge, often referred to as an interface, encapsulates the library's functionality, and your code exclusively interacts with this interface.
Here's a breakdown of the process:

Interface Creation: Design an interface that exposes only the features you require from the library.
Test Your Interface: Develop a suite of tests for your interface to ensure that it behaves as expected.
Selective Usage: In your application code, use the interface exclusively to interact with the library.
Library Usage: The actual library is integrated into the implementation of your interface. \

Unveiling the Benefits

Abstraction of Complexity: Wrapping libraries allows you to abstract away unnecessary complexity.
Your code interacts with a streamlined interface, making it easier to work with the library's features without being overwhelmed by its intricacies.
Flexible Library Swapping: Should you need to switch libraries due to changes in requirements or the library's status, your application is less affected because you isolated all the library interaction to a single point inside the interface implementation.
This is especially useful when dealing with unmaintained libraries or when you want to avoid vendor lock-in.
Test-Driven Stability: By testing your interface extensively, you ensure that your application's behavior remains consistent even if you decide to upgrade the library to a new major version.
For this to work, you can't mock the library.
Feature Enhancement: You can extend your interface with additional features that the original library might lack, tailoring the library's capabilities to your exact needs.

What Are the The Roadblocks to Be Aware Of

As always, there are no silver bullets in software engineering.

Increased Code Complexity: Creating and maintaining an interface requires additional code, potentially leading to increased complexity.
Abstraction Challenges: Balancing abstraction and practicality can be challenging.
Over-abstracting may lead to convoluted code, while under-abstracting might not provide the desired benefits like better generalization, less code duplication and isolation.

Don't underestimate the second downside.
If you have never been a library/API maintainer, this can be incredibly hard.

Real-World Case: Creating a Logger Wrapper

In my current project we have specific requirements for logging and we could not find a library that met them outright.
The requirements are:

Logging needs to happen in frontend and backend.
A log level defines what should be logged
A logger hierarchy allows to increase/decrease logging in only parts of the application.
Frontend logs are send to the backend.
Frontend logs need to be accessible in the UI
The backend stores frontend logs in a file.
The backend logs are shown in the terminal and stored in a file.

As usual, we did not need to meet all the requirements from the start, but it just grew over time.
The general development flow was in essence:

We started with ts-log (after only a short search)
As more requirements came in, we wrapped ts-log in an interface to add more features
The more we added, the uglier the wrapper became and we searched again for a library. This time going with pino. We massively simplified the wrapper and there was no change at all for our application code.
Again, pino does not support everything we need out of the box, so we added some more capabilities in.

This should give you a taste as to how a wrapper can form naturally over time.
The code can be found here: NicoVogel/logger-wrapper

The code does not reflect the full logger we use in production, but should give you a nice idea about how to wrap a library.

Don't Wrap Everything

It's important to be selective when deciding whether to wrap a library or not.
Consider the following decision tree:

Likelihood of Library Swap: If switching libraries is foreseeable, creating a wrapper is a wise move.
Missing Features: If the library lacks essential features, consider creating a wrapper to address these gaps (The alternative is to not wrap it fully, but just provide a supporting class).
Library Support: If the library was not updated for quite some time or is not used by many, consider creating a wrapper.
Evading Vendor Lock-in: If you don't want a vendor lock-in, then you have to create a wrapper. But ask really hard why this is required (e.g. the database that we might change in the future, but never will).

Remember, in the end the decision to wrap a library depends on your project requirements and team expertise.

Conclusion

Wrapping libraries through the creation of interfaces offers a practical approach to enhance flexibility and adaptability in your software projects.
By abstracting away unnecessary complexities, tailoring functionality to your needs, and isolating potential library changes, you ensure a more stable and maintainable codebase.
However, it's important to strike a balance and avoid over-abstracting.
The decision to wrap a library should be based on careful consideration of factors such as potential library switches, missing features, and library support.

Onboarding New Team Members Simplified with Dev Containers

Nico Vogel — Thu, 17 Aug 2023 20:46:23 +0000

Onboarding New Team Members Simplified with Dev Containers

Ever grappled with the challenges of onboarding new team members? Initializing a development environment can often be a long-winded affair, especially when one needs to adjust to configurations from prior projects. What if I told you there's a strategy to alleviate such pain points and ensure a uniform development environment across diverse operating systems, including Linux, Windows, and Mac?

In recent times, several IDEs have incorporated support for Dev Containers, and in this piece, I'll elaborate on this concept using Visual Studio Code.

Understanding Dev Containers

For those seeking a deep dive, Visual Studio Code's official documentation provides a comprehensive guide on Dev Containers and the various configuration possibilities. Check out the documentation here.

In a nutshell, here's how Dev Containers work:

Define your environment via a Dockerfile (optional) and devcontainer.json.
Build and start the container via VS Code.
Integrate the repository within the live container (automatic).
A server instance is launched in the container by VS Code (automatic).
The local VS Code instance connects to this server instance (automatic).

Why Use Dev Containers?

As you may already have noticed, defining environments using Dockerfile's has become a de facto standard for production services.
We can now do the same for our development environments, defining them in code to gain:

Consistency Assured: With Dev Containers, uniformity across the team's development environment is a given, dispelling discrepancies in configuration.
Effortless Rollbacks: Bumps on the road? Simply rebuild the current or revert to a former container state and resume your development.
Streamlined Maintenance: The upkeep and update of the development environment are rendered straightforward and less prone to errors with Dev Containers.

Tangible Benefits Gained

By adopting Dev Containers in my current project, we managed to reduce the onboarding for a new developer to:

Permission for the git repository access.
Having Git and Docker installed on their local machine.

We've managed to trim down the setup to approximately 2 minutes of hands-on work.
This is succeeded by about 10 minutes of building time for the Docker Image, with an added 3 minutes for the installation of code dependencies within the container.

So, for your project, we can guarantee a working setup within 15 minutes.

This can be sped up further by pre-building the image and storing it in a repository.

The Flip Side: Potential Limitations

While Dev Containers offer numerous benefits, it's essential to acknowledge the potential downsides:

Overhead: Owing to the containerization layer, Dev Containers introduce some overhead.
Restricted File Access: On Windows, gaining access to files within the container is not straightforward (depending on your setup).
Intermittent Rebuilding: On rare occasions, a container rebuild becomes necessary for seamless functioning.

We could not isolate the root cause yet, but as this occurs so rarely, we did not invest more resources in solving it.

Setting Up Dev Containers

To get started with a simplified Dev Container setup for my current project, you can use this template NicoVogel/dev-container.

Template Setup At A Glance

To grant you a brief overview of what the template includes:

Use of pnpm for improved performance.
playwright for end-to-end testing.
Enabled "docker in docker" via socket mount, allowing Docker to be used within the development environment.

To leverage the full potential of pnpm, you need to combine it with Linux's filesystem instead of Windows' ntfs.
Dependencies are mostly small files, and Linux is just far superior for this workload.
Therefore, we have opted to clone our code into the Linux's filesystem and mount it into the Dev Container.

This also speeds up git.

For Windows users, two potential routes are:

WSL2: Establish a personalized Linux and incorporate the repository therein.
- + Easy to access files later on
- - Setup guide for Linux required
Clone Repository into Container Volume: Use a feature from the Dev Containers extension to clone it directly into a docker volume.
- + No setup needed whatsoever
- - You can access the files in the file explorer, but you need to find them first via docker commands.

Depending on the skill sets of your developers, one or the other may make more sense for you.

Features Omitted from the Template

The main feature omitted is the artifactory integration.
You can also extend your container setup to:

Store the Dev Container image in Artifactory
- This includes proxy settings and custom company certificates.
Mirror npm packages in Artifactory.
Provide a login script for Artifactory after startup.

The scripts/setup.py is the login script for Artifactory, in case anyone needs it.

Tailoring The Template

To customize the template for your specific needs, follow these steps:

Duplicate the template folder, renaming it .devcontainer, ensuring its location is at your project's root.
For those with a proxy, modify the parameters in .devcontainer/devcontainer.env. If not, simply clear the file contents.
Set a base image that contains your company's certificates (if you have any) and adjust the FROM directive in .devcontainer/Dockerfile. Alternatively, you can use any Ubuntu base image.
Update the list of extensions that should be preinstalled automatically in devcontainer/devcontainer.json.

With these adjustments in place, you're ready to use the VS Code command Dev Containers: Reopen in Container.

In Retrospect

Dev Containers, as described using Visual Studio Code, offer a solution to simplify onboarding and ensure a consistent development environment across different operating systems.
By defining the Dev Environment as Code, Dev Containers guarantee consistency, easy rollback, and simplified maintenance.
This leads to a reproducible setup time and significantly reduced onboarding efforts.