Your visual test failed: is it the page or the capture?

#testing #devops #ci

A visual test that cannot tell “the page changed” from “the capture is not
reproducible” gets muted within a week. Keeping those apart is the whole job — and it is
deterministic by construction: same inputs, same verdict.

Two reasons, opposite responses

The page changedBlock the release. That is what the test is for.
The page did not change, the capture is not reproducibleAnimation mid-frame,
blinking caret, timestamp, ad slot, font fallback, scrollbar. Fix the capture, not the
UI.

A gate that cannot separate them becomes noise and someone turns it off. No AI in the loop:
same input, same verdict.

The four verdicts (and what to do with each)

PASSWithin the pixel budget and the region limit. Ship.
FAILBudget exceeded, with regions listed as x/y/w/h. Look at
diff.png and the region crops first.
STALEThe baseline does not belong to this build (git HEAD, mask
config or the PNG itself changed). Re-baseline deliberately or unstage the change — never
“just re-run”.
ERRORUnreadable PNG or size mismatch. That is a capture bug (viewport changed?),
not a regression.

STALE exists because the classic failure of visual testing is not flakiness — it is
a baseline quietly re-generated until the test went green. Tying the baseline to git HEAD plus a hash of the mask config makes re-baselining a deliberate, reviewable act.

Exit codes are the contract

`0 PASS   1 FAIL   2 STALE   3 ERROR

python scripts/visual_gate.py fingerprint baseline/home.png \
    --url https://app.example/home --viewport 1280x720 \
    --config baseline/home.masks.json

python scripts/visual_gate.py compare baseline/home.png actual/home.png \
    --meta baseline/home.png.meta.json --masks baseline/home.masks.json \
    --max-diff 0.001 --max-regions 1 --bundle artifacts/home`

With --json you get the verdict, the diff percentage, the region count and every
region's bounding box — enough for CI to publish what moved instead of “the test
failed”.

Budgets that survive contact with reality

--max-diff is a safety net, not the primary signalAntialiasing and
font hinting shift a few thousand pixels with zero real change; a real component move shifts tens
of thousands. Start at 0.001 (0.1%) and tighten once masks are in place.
--max-regions is the signal that catches small, meaningful changesA
wrong colour on a 60×40 button is ~0.26% of a 1280×720 page, but exactly one region. Budget
1.
--tolerancePer-channel delta (default 0) for capture noise on
gradients. If you need more than ~8, the capture is the problem.
MasksRemove whole rectangles from the comparison and are reported
(masked_pixels) — nobody hides a regression inside one without it showing in the
report.

Determinism checklist (before blaming the gate)

Fixed viewport and DPR 1DPR scales the layout and invalidates every
baseline.
Disable animationprefers-reduced-motion: reduce and
*{animation:none!important;transition:none!important}.
Wait for fonts and networkdocument.fonts.ready and network idle — a
fallback font diffs the whole page.
Hide the caret, the scrollbar and the dev overlayNone of it is page content, and
all of it moves pixels.
Freeze time and randomnessIf the UI renders date/time or
Math.random, freeze it before capturing.
Mask only what is genuinely dynamicAd slot, avatar of the day, live counter, A/B
slot.
Capture in the same container as CIA different font stack means different pixels
everywhere.

Evidence: the gate is tested in both directions

`15 tests passing — pytest -q  (28.9s)
  identical screenshots            -> PASS, diff 0.00000%, 0 regions
  one 60x40 block changed          -> FAIL, 1 region at the exact painted coordinates
  same change inside a mask        -> PASS, masked_pixels >= the masked area
  change outside the mask          -> FAIL (a mask must not hide the rest of the page)
  two changes with --max-regions 1 -> FAIL; with --max-regions 2 -> PASS
  stale baseline (git HEAD differs)-> STALE, exit 2
  mask config changed              -> STALE
  640x480 vs 1280x720              -> ERROR, "size mismatch" (never a silent PASS)
  decoder vs PIL + ImageMagick     -> identical pixels across 6 encodings
  16-bit capture                   -> high bytes equal the 8-bit rendering
  failure bundle                   -> diff.png re-decodes, red area >= the changed region`

The two-way test is the point: a gate tested only against “no change” is decoration. Both
directions are asserted — including the case that bites in production: an optimised baseline
(optipng, pngquant, ImageMagick) that a naive decoder refuses.

And the PNG decoder is stdlib (zlib + struct) — no PIL, OpenCV or ImageMagick on the runner: a CI
gate that needs a C library on every machine breaks on the machine that matters.

The pitfalls that cost the most

Baseline committed without its .meta.jsonWithout it the gate cannot
detect staleness. Commit both.
Re-baselining to make CI greenExactly what the STALE verdict exists to
prevent: the diff is in the bundle, review it before writing a new baseline.
Masking more than ~10% of the pageYou stopped testing that page. Prefer freezing the
widget over masking it.
Comparing different viewports or DPRsYields ERROR by design: it is a
capture bug, not a regression.
A PASS on a page you never capturedsweep reports
MISSING and fails the run — an empty actual directory is not “all good”.
Running the gate against a dirty treeFor release decisions, fingerprint the
baseline at a commit and compare only when CI is on that same ref.

Start now

The Visual Regression Gates skill ships the stdlib gate, 15 tests, the Playwright capture
template, the mask format and both guides (capture determinism, mask/fingerprint strategy).

See the Visual Regression Gates skill
See all skills

Want it wired into your pipeline? Guided setup from
R$ 1.500, with the first cycle run together.

← all skills

DEV Community