Part 2 of a series about building VisTest, an open-source visual regression library for pytest and Playwright. Part 1 was about why I started it. This one is about the month I spent fooling myself with a perfect score.
TL;DR
- My comparison engine scored 54/54 on my own benchmark: 0 false failures, 0 missed changes. Playwright's
toHaveScreenshot()scored 38/54, pixelmatch 34/54. - The number was real. It was also close to worthless, because the benchmark was drawn by the same library, by the same person, with the same assumptions that set the engine's thresholds.
- Along the way I found four ways a benchmark lies: a score that depended on the installed OpenCV, an engine that printed a shift it never drew, a test case that never existed, and a JPEG case that passed because of swapped color channels.
- On one real Chromium page, the same day, the picture looked completely different.
The perfect score
By late September VisTest had a benchmark I was proud of. A table, generated by a script, with every tool running on its default settings:
| Tool | False failures | Missed changes | Correct |
|---|---|---|---|
| VisTest | 0/32 | 0/22 | 54/54 |
Playwright 1.63 toHaveScreenshot()
|
14/32 | 2/22 | 38/54 |
| pixelmatch 7.2.0 | 18/32 | 2/22 | 34/54 |
Thirty-two pairs where nothing meaningful changed (sensor noise, sub-pixel text, a JPEG re-encode, a scroll bar) and the test should stay green. Twenty-two pairs with a real change (a price, a button, a header color) where it should go red. My engine got every single one right.
If I had stopped there, this would be a launch post.
Where the ruler came from
The corpus came from a script, tests/synthetic.py. It draws a fake shop page with OpenCV: a header, a product card, a "Buy now" button, a promo banner. Then it draws variations of it: add noise, re-encode as JPEG, change the price, repaint the header. 27 cases, each drawn on two OpenCV versions, 54 pairs.
Here is the baseline page from that corpus, unedited:
Look at the heading. It is a row of question marks. The script asks cv2.putText to write "Оформление заказа" ("Checkout" in Russian), and putText can't draw Cyrillic. So every page in my benchmark had a heading no browser would ever render, and nobody noticed. Not me, not the tests, not the engine.
That should have been the first hint. Here is the loop I was actually in:
The page is drawn by OpenCV. The engine is built on OpenCV. Its thresholds were tuned on this exact corpus. Then the corpus tells me the thresholds are right. A perfect score is the expected outcome of that loop, not evidence.
I didn't see it as a loop at the time. I saw it one crack at a time.
Crack 1: the score depended on which OpenCV you installed
What I thought. On one machine the engine scored 26/27 on a raster; on another, 25/27. Same code. So the engine had to be non-deterministic somewhere, and I spent two rounds of debugging inside the comparison cascade looking for it.
What it was. I finally hashed every intermediate array of the cascade on both machines. The very first array differed: the input images. The corpus was drawn on every run, and OpenCV 5 rasterizes text differently from OpenCV 4. All 54 images of the corpus differed between versions. 1.91% of the pixels on the base page, up to 222 levels in a channel. In 62% of the differing pixels, OpenCV 5 drew the stroke lighter. The text was physically thinner.
The engine was fine. The question changed under it.
And there was a second layer: the "26 vs 25" comparison itself had been recorded on an older version of the engine. On the current code nobody could reproduce it. I had been chasing a number from memory.
What changed. The corpus is now frozen on disk as PNG files, both rasters side by side as separate cases (54ad866, 17 Sep). Redrawing it is a deliberate command, never a side effect of running the benchmark. And a rule I still follow: a number without the commit it was measured on is a memory, not a fact.
Crack 2: the engine printed a shift it never drew
VisTest doesn't just say "these pixels differ". When it decides a difference is rendering noise, it says why: "the baseline moved +0.30 px reproduces 100% of the changed pixels." That sentence is the whole point of the tool. If I call something noise, I owe you the proof.
What I thought. The engine shifts the baseline by a fraction of a pixel, sees if that reproduces the change, and reports the shift.
What it was. On OpenCV 4.x, cv2.warpAffine with linear interpolation works in fixed point with 5 fractional bits. Before touching a single pixel, it rounds the shift to a 1/32 px grid. Ask for +0.30 px and you get +0.3125.
So the report printed +0.30, a precision that OpenCV version doesn't have. OpenCV 5 applies the shift as given, which means the same pair of PNGs was explained with different numbers on the two versions. A second note in the same file did the same thing: it reported dx=+0.25 dy=+0.19 while 4.x had actually drawn 8/32 and 6/32. While I was in there, I found one more: several places converted to 8-bit with astype(np.uint8), which truncates. That's a systematic bias of half a level downwards in the grey channel that every mask is computed on.
An explanation that is wrong in the last digit it prints is worse than no explanation.
What changed. Every sub-pixel shift now goes through one module, core/warp.py: bilinear interpolation written out in numpy, one rounding rule (half to even), and a test that swaps in a rounding backend to check that both code paths report what was actually drawn. After that, the detailed benchmark table is identical on OpenCV 4.14 and 5.0: all 54 pairs, every metric, not just the verdicts.
Then I froze the engine's answers too, in metrics.json, checked on every run. The first time it ran on another machine, it went red in the fourth decimal of a percentage. Verdicts identical. My file had stored more precision than the number has. So I measured: 25 environments, three numpy versions, two Pythons, both OpenCV majors. The version axis didn't move the numbers at all. What moved them was which SIMD kernels OpenCV picks for a Gaussian blur. Now every tolerance in that file comes with the measurement it was derived from, and the failure message prints it, so a red line can tell you whether it looks like a new machine or a changed engine (4caabd4).
Crack 3: a test case that never existed
After the freeze, the benchmark had two misses left. According to my plan, they were "header color" and "Buy now → Buy now!".
What I thought. Two cases, two different weaknesses. I wrote both into the brief for the next step.
What it was. The first thing that step did, before changing any code, was look up both cases in the corpus manifest. "Buy now!" wasn't there. Not in the manifest, not in the generator. The closest real case was "button shrunk" ("Buy now" → "Buy"), and it passed. The name had been born in a report, I copied it into my plan, it lived there for about a month, and then it went into a task. At no step did anyone check it against the files.
The two misses were one case on two rasters: "header color".
What changed. The rule about commits got a sibling: a task that refers to a test case starts by finding that case in the manifest. Case names outlive the code in reports, and they lie more convincingly than numbers. And "measure first, don't touch code" became the first step of every engine task, because it's the step that caught this.
The miss that was real
The header color case is my favorite pair in the whole corpus.
The header is repainted. The color difference is ΔE00 6.7 on every one of its 57,600 pixels. For scale: around 2.3 is "just noticeable", so a person sees this immediately. pixelmatch reports 0 differing pixels. Playwright reports 0 differing pixels. And my engine passed it too.
Why mine missed it: the engine wants color and structure to agree before it calls something a change. Structure is measured with SSIM, and SSIM subtracts the mean before comparing, so it's blind to a uniform shift of color by design. The only place where structure changed was the header's bottom edge, a 900×2 line, and a morphological opening removed that.
The obvious fix was "the opening must never erase a component completely". I measured it first: it would have flagged 170 noise components on 12 of the 32 noise pairs. Worse, it would have made the header case fail because of the 2-pixel edge, not because of the color. That's a pass for the wrong reason, which breaks the first time the layout changes. Rejected.
The fix that went in is a third way past the consensus: a large, flat recolor (1483ae4, 25 Sep). A connected area counts as a real change when it covers at least 1.25% of the frame and its color difference is uniform (σ/μ ≤ 0.09). Both numbers sit in the middle of a measured gap. The largest uniform noise was 0.31% of the frame (a scroll bar), the smallest header 5.2%. The most uniform noise had 0.18 unevenness, the least uniform header 0.05.
One more rule from that week. I was not allowed to relabel the case. Calling it "ambiguous" would have turned 52/54 into 54/54 by a decision of the interested party, and devalued the benchmark I had just spent a week making checkable.
So: 54/54. Fair and square. On my corpus.
Crack 4: the JPEG case passed because of swapped channels
The same day, 25 September, the project went through a full code review, and the headline finding was one sentence: 54/54 describes the corpus, not the engine.
The first proof was the JPEG case. The corpus has a "JPEG at quality 75" pair labeled as noise, and the engine has a rule to recognize JPEG artifacts: it re-encodes the baseline at several qualities and checks if that reproduces the difference. The generator's re-encode looks like this:
def recompress(img, quality=82):
ok, buf = cv2.imencode(".jpg", img, [int(cv2.IMWRITE_JPEG_QUALITY), quality])
return cv2.imdecode(buf, cv2.IMREAD_COLOR) if ok else img
OpenCV's encoder expects BGR. The generator and the engine's detector both handed it an RGB array. The same mistake on both sides, so they agreed perfectly with each other. Encode the same frame the right way, with OpenCV and the correct channel order or with Pillow on its defaults, and at quality 75 the engine fails with 4 regions. The JPEG rule had learned to recognize my generator's artifact, not JPEG.
A symmetric bug is invisible to a benchmark built by the same hands. That's the whole article in one sentence.
(That rule still exists. On the real-browser corpus from the next article it fires on 0 of 462 pairs, so it no longer decides anything. It's on my list.)
Then I opened a real browser
The second proof was one typical admin page in real Chromium. 21 CSS changes with labels I wrote down in advance: a link color, an underline, a border radius, a button color. Plus 4 rendering variations that really differ between machines: LCD text with sub-pixel positioning, text-rendering, the page shifted by a fraction of a pixel and by 1 px.
| Tool, setting | False failures (of 4) | Missed (of 21) |
|---|---|---|
| VisTest, default | 3 | 7 |
| Playwright, default | 4 | 9 |
Playwright, threshold: 0.05
|
4 | 1 |
It's a probe, not a benchmark: one page, one browser, mutations I picked. It doesn't prove VisTest is worse. It proves that "0/32 vs 14/32" doesn't survive contact with a real renderer in either direction, and that my engine had blind spots the synthetic corpus could never show:
- The anti-aliasing filter judged only lightness. A link going from
#2563ebto#7c3aedpassed as "strokes slightly heavier". - The morphological opening that saved text erased lone thin lines. A new 1 px underline under a link passed. Playwright caught it.
- A button repainted up to ΔE00 ≈ 9 passed, because the new flat-recolor rule only starts at 1.25% of the frame. On a full-page screenshot the same button is even harder to see.
Five other rendering variations (a restart, two hinting modes, GPU rasterization, headless shell vs full Chromium) gave 0 differing pixels. On a pinned environment, there was no noise to filter at all. My engine had been built to explain noise that a real browser, set up properly, mostly doesn't produce, and it was missing changes that a real browser does.
What I changed in how I work
The code fixes are in the repo. These are the habits that came out of that month, and they matter more:
- A ruler drawn by the hand that set the thresholds measures self-persuasion. The benchmark has to be produced by something that doesn't know about the engine. For a screenshot tool, that means a real browser.
- Every number gets a date and a commit. Without one, it's a memory.
- Every case name gets checked against the files before anyone works on it.
- Labels don't move to improve a score. If one ever changes, the changelog says the number went up because of relabeling, not because of the engine.
- Determinism is tested, not assumed. The engine's answers to the corpus are frozen and replayed on every merge, with measured tolerances.
- Explanations must be exact. If the tool says "moved by +0.30 px", it moved by exactly that.
Where it is now
VisTest is on PyPI and on GitHub under Apache-2.0. It's one function in your pytest tests:
pip install "vistest[browser]"
playwright install chromium
from vistest import expect_screenshot
def test_home(page):
page.goto("http://localhost:8000")
expect_screenshot(page, "home.png")
When a check fails, it doesn't print a percentage. It tells you what changed, "fill: #3478f6 → #1d4ed8, color difference 14.5", and shows it:
The engine you install today isn't the one from this article. It was rebuilt on top of the corpus from a real browser, and the claim I'm willing to make now is narrower than "54/54": on the same renderer, it catches changes Playwright misses, with no more false failures. How I built that corpus, and how badly the engine lost on it at first, is the next article.
If you work with screenshot tests, I'd love for you to try it on your own failures. There's a command for exactly that: vistest bench <folder> takes your existing baseline/actual pairs, including Playwright's layout, and shows where VisTest and Playwright disagree. Tell me where it's wrong.
⭐ github.com/marshlw/vistest: code, the frozen corpus, the benchmark scripts and the full development journal, mistakes included.
Next: "A corpus from a real browser, and a cold shower."
Top comments (0)