DEV Community

Cover image for Holistic Visual Regression Testing
Dimitri Ivashchuk
Dimitri Ivashchuk

Posted on • Originally published at lost-pixel.com

Holistic Visual Regression Testing

Introduction to Holistic Visual Testing πŸ–ΌοΈ

As front-end development continues to evolve, it's more important than ever to have a robust testing strategy in place to ensure that your applications look and function as expected. One of the most effective ways to do this is through visual regression testing, which allows you to catch and fix visual bugs before they reach production.

Holistic visual regression testing is an approach that aims to test the entire application, including both layout and interactions, as a whole.

In this example, we will set up a complex visual testing suite for our Next.js application. Here is the list of tools we will use:

  • Storybook

  • Playwright

  • GitHub actions

  • Lost Pixel Platform

As the final result, we want to make sure that the following parts of our applications are looking the way they are intended to look:

  • Individual components - visual unit tests

  • Pages - visual integration tests

  • Results of playwright/cypress tests - visual e2e tests

Lost Pixel makes it easy to compose all of this in a single place to make the management of your visual testing a breeze. Here is the final version of the code for the whole integration:

Here is the final version of the code on GitHub

Setting up Next application πŸ”Ό

We will clone the app-playground repo from Next.js, it has rudimentary parts of the actual application, and that's precisely what we need.

git clone https://github.com/vercel/app-playground.git

pnpm install

pnpm dev
Enter fullscreen mode Exit fullscreen mode

image

Visual unit tests πŸ–ΌοΈ

We will use Storybook in combination with Lost Pixel to make sure our app components are looking the way we expect them to look.

Let's install Storybook:

npx storybook@next init && pnpm install postcss-loader style-loader @storybook/addon-postcss
Enter fullscreen mode Exit fullscreen mode

Inside .storybook/main.js we need to make Storybook & Tailwind.css play well with each other :D

const path = require('path');
module.exports = {
  stories: ['../stories/**/*.mdx', '../stories/**/*.stories.@(js|jsx|ts|tsx)'],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-interactions',
    {
      name: '@storybook/addon-postcss',
      options: {
        cssLoaderOptions: {
          // When you have splitted your css over multiple files
          // and use @import('./other-styles.css')
          importLoaders: 1,
        },
        postcssLoaderOptions: {
          // When using postCSS 8
          implementation: require('postcss'),
        },
      },
    },
  ],
  framework: {
    name: '@storybook/nextjs',
    options: {},
  },
  docs: {
    autodocs: 'tag',
  },
};
Enter fullscreen mode Exit fullscreen mode

Inside .storybook/preview.js we need to import tailwind styles on top of the file:

import '../styles/globals.css';

export const parameters = {
  actions: { argTypesRegex: '^on[A-Z].*' },
  controls: {
    matchers: {
      color: /(background|color)$/i,
      date: /Date$/,
    },
  },
};
Enter fullscreen mode Exit fullscreen mode

Inside stories/Button.stories.tsx let's write our first story that we will use in tests:

import type { Meta, StoryObj } from '@storybook/react';
import React from 'react';

import Button from 'ui/Button';

// More on how to set up stories at: https://storybook.js.org/docs/7.0/react/writing-stories/introduction
const meta: Meta<typeof Button> = {
  title: 'Example/Button',
  component: Button,
};

export default meta;
type Story = StoryObj<typeof Button>;

export const Default: Story = {
  render: () => <Button kind="default">Default</Button>,
};

export const Error: Story = {
  render: () => <Button kind="error">Error</Button>,
};
Enter fullscreen mode Exit fullscreen mode

Now we run our storybook to see our components displayed in an isolated manner:

npm run storybook
Enter fullscreen mode Exit fullscreen mode

image

Our stories will serve as the basis for the unit visual tests. Let's integrate this first step with Lost Pixel Platform to see how it will work there.

Lost Pixel Platform βš™οΈ

You can use the quickstart documentation from lost-pixel.com to set up the platform to work with your repositories via installing the official Lost Pixel GitHub app. In this guide, we will focus on setting up the right configuration, assuming you have done the platform setup already!

You can use Lost Pixel Onboarding to easily set up the project by copying & pasting the respective files.

In our lostpixel.config.ts at the root of our project, let's define the tests for Storybook

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },

  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
Enter fullscreen mode Exit fullscreen mode

In our .github/workflows/vrt.yml at the root of our project let's define our CI/CD job:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
Enter fullscreen mode Exit fullscreen mode

With this simple setup, we are already covering the visual unit tests πŸ™ŒπŸΌ

image

Let's make it really holistic now!

Visual integration tests πŸ–ΌοΈ ↔️ πŸ–ΌοΈ

Sometimes we care not only about how our components will be used in isolation but also about how they will look on the page when they are combined with a plethora of other components & styles. Let's explore how we can easily expand our initial setup to monitor our pages continuously!

Let's extend our lostpixel.config.ts to include our page shots.

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
Enter fullscreen mode Exit fullscreen mode

We also need to extend .github/workflows/vrt.yml to build & run our Next.js app so we can access pages in Lost Pixel:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
Enter fullscreen mode Exit fullscreen mode

Now we are talking πŸš€ Our visual tests are covering not only unit parts with individual components but also the integration part where we are making the screenshots for the whole pages.

image

Let's make it an ultimate setup with a magic flavour of the feature of Lost Pixel called custom shots πŸͺ„βœ¨βœ¨βœ¨

Visual E2E tests πŸ–ΌοΈ βš™οΈ πŸ–ΌοΈ βš™οΈ πŸ–ΌοΈ

What if I told you that on any given CI run, you can create the screenshots using your logic & they will be available immediately on Lost Pixel Platform for the visual regression testing? Enter custom shots πŸ“¦

We will be using Playwright to interact with the page & make sure that after some set of interactions, we are always looking at the same UI.

npm install --save-dev @playwright/test
Enter fullscreen mode Exit fullscreen mode

Let's also extend our package.json with test:e2e command:

...
"scripts": {
  "dev": "next dev",
  "build": "next build",
  "start": "next start",
  "test:e2e": "playwright test",
}
Enter fullscreen mode Exit fullscreen mode

Now we will create our E2E test that will not do much, but will give you a good understanding of how you could integrate Lost Pixel into any test like this!

In e2e/example.spec.ts :

import { test } from '@playwright/test';

test('lost-pixel e2e', async ({ page }) => {
  // Perform some action on the page, like clicking the button.
  // πŸ“’ http://172.17.0.1:3000 is already fit for GitHub action runtime
  await page.goto('http://172.17.0.1:3000/context');
  await page.click('data-test-id=context-click-counter');
  await page.click('data-test-id=context-click-counter');
  // Make a shot for Lost Pixel to test
  // πŸ“’ Path is arbitrary, but I advise you to store all of the custom shots in a single folder, e.g. lost-pixel. 
  await page.screenshot({ path: 'lost-pixel/a.png', fullPage: true });
});
Enter fullscreen mode Exit fullscreen mode

Our ContextClickCounter component gets a new data-test-id, so the code above works:

      <button
        // Here is the important part
        data-test-id="context-click-counter"
        onClick={() => setCount(count + 1)}
        className="rounded-lg bg-gray-700 px-3 py-1 text-sm font-medium tabular-nums text-gray-100 hover:bg-gray-500 hover:text-white"
      >
        {count} Clicks
      </button>
Enter fullscreen mode Exit fullscreen mode

Now we are ready to integrate our E2E with Lost Pixel! Let's extend our lostpixel.config.ts to include our custom shots.

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  customShots: {
    // πŸ“’ currentShotsPath should be matching the one above from playwright test
    currentShotsPath: './lost-pixel',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
Enter fullscreen mode Exit fullscreen mode

And finally, we extend our GitHub Action declaration file by running Playwright tests:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Playwright tests
        run: npx playwright install --with-deps && npm run test:e2e

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
Enter fullscreen mode Exit fullscreen mode

After Lost Pixel Platform runs this time, you should see something similar to this:

image

After interacting with our page & clicking the button two times, we made a screenshot of a page to test visually. Sometimes it's very useful because apart from asserting the presence or absence of some elements on the page that E2E tests traditionally rely on, the layout could be broken after a particular interaction and here visual regression tests will catch this for us.

Summary πŸ”Ό

We have built a holistic visual testing framework over an existing Next.js application using the tools suitable for every task: Storybook for visual unit tests, Pages for visual integration tests & Playwright for visual E2E tests. We have use GitHub Actions to make sure our setup runs continuously on every push. Finally, we have used the Lost Pixel Platform to manage our visual tests in one central place & notify us whenever visual regression is found.

image

If you liked the read and enjoyed setting up your visual tests, you might as well like our Visual Regression Testing Discord. There you can find help with any issues, chat about your setup & discuss the good practices for visual tests.

Top comments (0)