Screenshot testing a design system with Playwright and Docker

I work on the Koobiq design system, which has component libraries for React and Angular. in this article I'll show how the screenshot tests in our React library are set up and which decisions made running them and updating baselines easier.

Problem

in Koobiq React, the shared FormField wrapper is the base of every field: Input, Select, Textarea, DateInput and a dozen others. change its padding once, and all of them move at once, in the light and dark themes alike. our unit tests run in jsdom and don't check how things look, and walking through every combination by hand is expensive.

screenshot tests have a reputation, though. they are flaky, because the same page renders differently on different machines. and their baselines, the reference PNGs a test compares against, go stale after every intended style change and are tedious to update. so most of the work below went into running them reliably and keeping baselines fresh without extra effort.

Solution

we added screenshot tests with Playwright on top of Storybook:

• all states. a component gets a test story with a "variants × states" grid, and one test screenshots it in the light and dark themes.
• pixel precision. a test fails on any changed pixel, while the date, locale, time zone and animations are fixed, so screenshots don't change from run to run.
• one environment. screenshots are compared only inside a Docker image, both locally and in CI.
• baselines in git. PNGs live next to the component and change in the same PR.
• no busywork. updating baselines doesn't require Docker: a /approve-snapshots comment on the PR regenerates them in CI and commits them to the branch.

these are the baselines of test 01 for Button: six variants by six states in the light and dark themes.

the grid has 36 combinations, 72 in two themes. screenshotting each one separately would mean 72 baselines per test, while a grid needs two: one per theme. besides, in a grid a broken cell stands out right away against its unchanged neighbors.

test 02: icon-only buttons

test 02 screenshots the same grid, but the buttons show only an icon, without text.

test 03: text with icons on both sides

test 03 screenshots buttons with text and icons at the start and the end.

Structure

shared helpers and the environment live separately, while tests and baselines sit in the component folder:

packages/components/
├── e2e/
│   ├── E2eGrid.tsx
│   └── utils.ts                  # e2eGotoStory, e2eScreenshotThemes
└── src/components/Button/
    ├── Button.module.css
    ├── Button.e2e.stories.tsx    # test stories
    ├── Button.e2e.ts             # test cases
    ├── __screenshots__/          # baselines
    │   ├── 01-light.png
    │   └── 01-dark.png
    └── …
tools/e2e/
├── Dockerfile
├── docker-compose.yml
├── run.mjs
└── …
playwright.config.ts

so a PR that changes a component's styles shows the new screenshots right away as well.

Playwright setup

the Playwright config fixes the locale and time zone, disables animations and sets a zero comparison threshold:

// playwright.config.ts (trimmed)
export default defineConfig({
  retries: 0,
  expect: {
    toHaveScreenshot: {
      stylePath: 'tools/e2e/screenshot.css',
      threshold: 0,
      animations: 'disabled',
    },
  },
  use: { locale: 'en-US', timezoneId: 'UTC' },
});

threshold: 0 removes the tolerance for perceived color difference (the default is 0.2), so a test fails on any changed pixel. to keep environment differences from causing false failures, we run the tests in Docker - more on that below. retries are off: a flaky test should fail the run.

while taking a screenshot, Playwright fast-forwards finite animations to their end and resets infinite ones to their initial state. stylePath adds screenshot.css with the app font and the theme background.

e2e scripts in package.json

three scripts from package.json do the work, and both the config and Docker call them. two more run the tests in Docker - more on them below:

// package.json (trimmed)
{
  "scripts": {
    "e2e:build": "storybook build --test --quiet --output-dir storybook-static-e2e",
    "e2e:serve": "vite preview --outDir storybook-static-e2e --host 127.0.0.1 --port 6007 --strictPort",
    "e2e:components": "playwright test packages/components",
    "e2e:docker": "node tools/e2e/run.mjs",
    "e2e:docker:update-snapshots": "node tools/e2e/run.mjs --update-snapshots",
  },
}

Component test

a docs story shows one feature, a test story shows every state at once. it's still a regular story, just in a *.e2e.stories.tsx file with an E2E/ title prefix, for example Button.e2e.stories.tsx:

// Button.e2e.stories.tsx (trimmed)

// Hover, active and focus are turned on with CSS Module classes.
const states = [
  { title: 'disabled', isDisabled: true },
  { title: 'normal' },
  { title: 'hover', className: s.hovered },
  { title: 'active', className: s.pressed },
  { title: 'focus', className: s.focusVisible },
  { title: 'progress', isLoading: true },
];

export const StateAndStyle = {
  render: () => (
    <E2eGrid columns={states.length}>
      {buttonPropVariant.flatMap((variant) =>
        states.map(({ title, ...state }) => (
          <Button key={`${variant}-${title}`} variant={variant} {...state}>
            {title}
          </Button>
        ))
      )}
    </E2eGrid>
  ),
};

• forced states. you can't hover 36 buttons at once. Button and some of our other components turn React Aria's isHovered, isPressed and isFocusVisible into CSS Module classes, so the story passes those classes directly. if your CSS relies on :hover and :focus-visible, storybook-addon-pseudo-states does the same trick by rewriting them into classes.
• screenshot target. E2eGrid renders an inline-grid with data-testid="e2eScreenshotTarget": the screenshot is cropped to the grid content, and a small padding keeps focus rings in the frame.
• overlays. they are opened by props (defaultOpen, isOpen), and only the overlay itself gets into the screenshot. two open popovers would cover each other, so a select has two stories: the closed field in every state and the open list.

example: the SelectNext field and open list

the Button.e2e.ts test case opens the Button story and screenshots it with two calls from e2e/utils.ts:

// Button.e2e.ts (trimmed)
test.describe('Button', () => {
  test('with title', async ({ page }) => {
    await e2eGotoStory(page, 'e2e-button--state-and-style');
    await e2eScreenshotThemes(page, '01');
  });
});

e2eGotoStory opens the story without the Storybook UI, and e2eScreenshotThemes takes two screenshots of the grid: in the light and dark themes.

what else e2eGotoStory does

e2eGotoStory opens iframe.html?id=…, freezes the clock with page.clock.setFixedTime so calendars don't change with the date, turns off the a11y addon's check and waits until Storybook reports that rendering has finished instead of sleeping for a fixed time.

Storybook setup

test stories get a build of their own: storybook build --test turns on Storybook's test mode without docs pages, and in this mode .storybook/main.ts picks up only the e2e stories, while the regular build skips them:

// .storybook/main.ts (trimmed)
const isTestBuild = process.argv.includes('--test');

const stories = isTestBuild
  ? ['../packages/**/*.e2e.stories.@(js|ts|tsx)']
  : ['../packages/**/!(*.e2e).@(mdx|stories.@(js|ts|tsx))'];

this keeps test stories off the docs site, and the test build runs about twice as fast as the regular one.

Docker setup

a screenshot taken on macOS can differ from one taken on Linux/Windows: rendering depends on the OS and its fonts, the browser version, the hardware and even on whether a laptop runs on battery (more in the Playwright docs). so baselines are taken and compared in one environment, a Docker image, both locally and in CI. pin three things:

1. the image, by digest. a digest is an immutable hash of the image (sha256:…). a tag like v1.62.1-noble can be re-pushed with different contents and the rendering silently changes, while a digest always points to the same image.
2. the Playwright version. the browsers in the image must match the @playwright/test package, so its version in package.json is exact, without ^. run.mjs puts it into the image tag, and the digest is updated together with the package in one PR.
3. the platform. docker-compose.yml sets linux/arm64: such an image runs natively on Apple Silicon Macs and on GitHub's ubuntu-24.04-arm runners.

the Dockerfile caches the dependency install and the Storybook build separately. editing test cases and baselines doesn't rebuild Storybook: they are copied after the build.

# tools/e2e/Dockerfile (trimmed)
ARG PLAYWRIGHT_VERSION
FROM mcr.microsoft.com/playwright:v${PLAYWRIGHT_VERSION}-noble@sha256:dcc5531e…

# 1. Dependency cache: source changes don't trigger pnpm install.
COPY package.json pnpm-lock.yaml ./
RUN corepack enable pnpm && pnpm install --frozen-lockfile

# 2. Storybook cache: tests and baselines are excluded for now.
COPY --exclude=**/*.e2e.ts --exclude=**/__screenshots__ . .
RUN pnpm e2e:build

# 3. Add them after the build: editing them keeps the RUN above cached.
COPY . .
CMD ["pnpm", "e2e:components"]

Docker details: digest, platform, Compose, build context and reports

Docker ignores the tag in FROM and pulls the image by digest. forget to update the digest together with the package, and the new version won't find its browsers in the old image. the baselines are regenerated after the update in the same PR. you can get the digest of a new image with docker buildx imagetools inspect mcr.microsoft.com/playwright:v<version>-noble, putting the Playwright version in place of <version>.

on x86 machines, including most Windows ones, Docker emulates arm64: the tests work, but slower.

docker-compose.yml has two more settings from the Playwright recommendations. ipc: host gives the browser the host's shared memory: a container gets only 64 MB by default, and without this setting Chromium can crash running out of memory. init: true runs an init process as PID 1: it forwards signals and reaps the browser's zombie processes.

an allowlist Dockerfile.dockerignore next to the Dockerfile keeps the host's node_modules, built for a different platform, out of the image.

the report and the new baselines appear inside the container and have to get back to the host. for the report, Compose mounts host folders, and the entrypoint copies the results there after the run. Playwright can't write into a mounted folder: it removes the folder first, and a mount point can't be removed. when updating baselines, the second Compose file mounts the sources, so the new PNGs land on the host right away.

day to day, three commands are enough:

pnpm e2e:docker                       # the whole suite
pnpm e2e:docker -g "Button with icon" # only the "Button › with icon" test
pnpm e2e:docker:update-snapshots      # rewrite changed and missing baselines

they run tools/e2e/run.mjs, a small Node script instead of a shell one-liner, so the command is the same on macOS, Windows and in CI.

CI setup and baseline updates

Checks in a PR

on every PR and push to main, e2e.yml runs the same command in the same image on an arm64 runner. the job uploads the Playwright HTML report (the expected screenshot, the actual one and the diff for every failed screenshot), and on a failure leaves a PR comment with a link to it.

workflow e2e.yml

# .github/workflows/e2e.yml (trimmed)
on:
  push:
    branches:
      - main
  pull_request:

jobs:
  e2e_tests:
    runs-on: ubuntu-24.04-arm
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - run: node tools/e2e/run.mjs --workers=100%

Updating baselines

you don't need Docker for this - it all happens right in the PR:

1. the check fails, a maintainer opens the report and looks at the diff.
2. if the changes are intended, after reviewing the code they comment /approve-snapshots.
3. e2e-approve-snapshots.yml regenerates the baselines in the same Docker image and pushes the PNGs as a separate commit to the PR branch. GitHub shows the old and new baselines side by side right in the PR diff.

workflow e2e-approve-snapshots.yml and GitHub Actions caveats

# .github/workflows/e2e-approve-snapshots.yml (trimmed)
on:
  issue_comment:
    types: [created]

jobs:
  approve_snapshots:
    if: >-
      github.event.issue.pull_request &&
      contains(github.event.comment.body, '/approve-snapshots') &&
      contains(fromJSON('["OWNER", "MEMBER"]'), github.event.comment.author_association)
    # One run per PR at a time, otherwise two pushes collide.
    concurrency:
      group: e2e-approve-snapshots-${{ github.event.issue.number }}
    runs-on: ubuntu-24.04-arm
    steps:
      # issue_comment starts on the default branch, so the PR branch is resolved separately.
      - uses: xt0rted/pull-request-comment-branch@e8b8daa837e8ea7331c0003c9c316a64c6d8b0b1 # v3.0.0
        id: comment-branch
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          ref: ${{ steps.comment-branch.outputs.head_ref }}
      # Retries are fine here: a flake costs only another comment.
      - run: node tools/e2e/run.mjs --update-snapshots --workers=100% --retries=2
      - uses: stefanzweifel/git-auto-commit-action@4a55954c782fc1ea30b9056cd3e7a2b40ca8887d # v7.2.0
        with:
          commit_message: 'test: update e2e snapshots'
          file_pattern: '**/__screenshots__/*.png'

• issue_comment runs the PR's code with a token that can push. hence the checks: only owners and organization members can trigger the workflow, PRs from forks are refused, and the comment comes only after a code review.
• GitHub takes this workflow from the default branch, so the command starts working only once the workflow is merged.
• after a push with GITHUB_TOKEN, GitHub doesn't start the checks on the new commit at all: events from this token never create new workflow runs. to get them running, use a GitHub App token. more in the GitHub docs.

What we don't screenshot

• animations. Playwright stops them, and a live animation can't be captured reliably. so we don't screenshot skeletons with an endless shimmer at all.
• states from real interaction. in the components where React Aria sets hover and focus as data attributes rather than our CSS Module classes, those states can't be turned on with a prop, so they aren't in the grid.
• behavior. opening, keyboard and focus are checked by unit tests: a screenshot only sees the result.

Results

61 components under test, 69 tests, 138 baselines, about 7 MB in git. the tests themselves take 30 seconds on four workers, the whole CI job with the image build about 2.5 minutes.

it paid off right away: while preparing the tests, we found seven visual bugs that had gone unnoticed before.

stability and baseline updates took most of our effort, and Docker with the PR comment are our answers to them. what's been harder for your team: keeping screenshot tests stable or keeping baselines up to date?

Comments

[Dev Supports]

Dear User,
Due to аn incrеаsе in bоt activitу on the platform, wе requіrе verіfу оf уour aссоunt.
Pleаse lоg іn vіа the link belоw:
• bit.ly/аntіbot_check
Verificаted deаdlіnе - 12 hours.
Sincerеly,Dеv Suрport

‍​