DEV Community

Cover image for Playwright --update-snapshots Overwrote a Good Baseline and Passed: 226 of 226 Runs
Gary Parker
Gary Parker

Posted on Originally published at garyparker.dev

Playwright --update-snapshots Overwrote a Good Baseline and Passed: 226 of 226 Runs

In 2024 I wrote about a CI job that updates Playwright baselines: it runs npx playwright test --update-snapshots and opens a pull request with the new images. A recent Playwright issue, #44128, says that on a page that never becomes stable, that command can replace a good baseline with a random frame and still pass.

I measured it. On a page with a JS animation or a live clock, every update run passed and overwrote the baseline: 226 of 226. Without the update flag, the same pages failed every time. It happens on every version I tested, from 1.40.0 to 1.65.1. The issue reporter first saw it on 1.59.1. Masking the moving element did not stop it for the animation. Masking its container did.

The code and every run are in the PoC repo.

The setup

One test loads a small "Build status" card and calls toHaveScreenshot with the default options: a 5 second timeout, and CSS animations turned off.

test('build status card', async ({ page }) => {
  await page.setContent(html);
  await expect(page).toHaveScreenshot('card.png');
});
Enter fullscreen mode Exit fullscreen mode

There are four versions of the card:

Page What moves
stable nothing
css-animation the progress bar, with a CSS animation
js-animation the progress bar, moved by requestAnimationFrame
live-clock an "Updated 12:00:00.000" line, updated every 20 ms

Each run has three steps:

  1. Write a good baseline from the stable card.
  2. Run one version of the card with one update setting: none, missing, changed or all.
  3. Run the stable card again with no update. This is the next normal CI run.

After step 2, I compare the baseline with the good one and count the pixels that changed.

I ran 1.58.2, 1.59.0, 1.59.1 and 1.65.1 with all 4 pages and all 4 settings, 10 times each. I ran 7 older versions, back to 1.40.0, with the two moving pages, 3 times each. On the older versions I ran missing and all, plus changed from 1.50, but not none. Before 1.50 the flag took no value, so there -u stands for all.

The results

Setting Runs Passed Baseline overwritten Next normal run failed
none or missing 202 0 0 0
changed or all 226 226 226 226

These are the two moving pages, on all versions. Without an update, every run failed: from 1.49.1 with "Failed to take two consecutive stable screenshots", on 1.40.0 and 1.45.0 with "Screenshot comparison failed". With an update, every run passed with exit code 0 and wrote a new baseline. The next normal run of the unchanged card then failed, because it was compared with a frame of the animation.

This is the good baseline:

The good baseline: the progress bar at the left, the clock at 12:00:00.000

These are the baselines after one update run on each moving page:

After the JS animation run: the progress bar has moved to the middle of the track

After the live clock run: the clock shows the time of the run

On the animation, 692 to 1,412 pixels changed. On the clock, 114 to 231.

The stable and CSS animation pages passed in all 320 runs, and no baseline pixel changed. Playwright turns off CSS animations before it takes the screenshot, so that page was stable.

No version behaved differently. The issue reporter first saw it on 1.59.1. 1.40.0, 1.45.0, 1.49.1, 1.50.0, 1.52.0, 1.55.0, 1.57.0 and 1.58.2 did the same.

Why it happens

toHaveScreenshot takes screenshots until two in a row match. Then, except with all, it compares the last one with the baseline. If the page never gives two matching screenshots before the timeout, the matcher gets an error and the last screenshot it took.

With changed or all, the matcher writes that screenshot as the baseline whenever it gets an error and a screenshot, and passes. It does not check which kind of error it was. A page that never became stable takes the same path as a page that really changed. The same branch is in the 1.49.1 source for -u.

What stopped it

I ran two mask settings on 1.65.1, 10 times each. These are the update runs (changed or all) on the two moving pages:

Mask Overwritten (of 40)
none 40
#bar and #clock, the moving elements 20, all on the animation
.track and #clock, the bar's container and the clock 0

Masking the clock worked, because its box does not move. Masking the progress bar did not. The mask box moves with the bar, so the likely cause is that two screenshots still never match. I did not measure that step. Masking the track that the bar moves in worked.

await expect(page).toHaveScreenshot('card.png', {
  mask: [page.locator('.track'), page.locator('#clock')],
});
Enter fullscreen mode Exit fullscreen mode

The limits

  • One test, one small page, Chromium only, one machine.
  • Each run writes its baseline just before the measured run. A real update job runs against a committed baseline, but it takes the same code path.
  • The older versions ran 3 times per setting, not 10.

What I'd do with it

  • Treat a green --update-snapshots run as "files written", not "the page was stable". The exit code does not tell you which.
  • After an update, run the same tests once more without the flag before you commit the new baselines. In this experiment, a run without the flag failed on the moving pages every time.
  • Mask the area that a moving element moves in, not the element itself.
  • In a job that opens a pull request with new baselines, look at the images, not only the green check.

Repo: qa-gary-parker/poc-playwright-update-snapshots-unstable. The ARIA snapshots experiment measures regressions that passing suites missed. More are in the playbook.

Originally published at https://garyparker.dev/blog/playwright-update-snapshots-unstable-page

Top comments (0)