The most expensive bugs we have shipped were invisible to every automated check. The build was green. The tests passed. And the page was wrong: a headline wrapped into nonsense on mobile, a section rendered behind another, a form that looked fine and submitted nothing.
So we adopted a rule that sounds primitive and works: no change is done until someone has looked at a screenshot of the deployed page. Not the dev server. The real thing, both viewports, at full resolution.
Making evidence cheap
for (const vp of [{ w: 1440, h: 900 }, { w: 390, h: 844 }]) {
const page = await browser.newPage({ viewport: vp });
page.on('console', (m) => errors.push(m)); // console is part of the picture
await page.goto(URL, { waitUntil: 'networkidle' });
await page.screenshot({ path: shot(vp), fullPage: true });
}
Capture is one command per project, so there is no excuse to skip it. Timestamped captures accumulate into the cheapest regression archive ever built.
The machine reads first
This year we put a vision model in front of the human. Every capture pair goes to a model with a deliberately narrow brief: report text overflow, truncation, overlapping elements, broken alignment and contrast failures, with pixel regions, and answer NONE if the page is clean. No taste, no compliments, defects only. It catches roughly 70% of mechanical failures before a person looks, and it never gets bored on capture number forty.
The human pass still decides, and it has rules of its own: open captures at 1:1 because overflow hides at fit-to-window zoom, read the rendered copy aloud because screenshots find the typos your editor did not, and treat a clean-looking page with console errors as dirty.
The generalization
Database features are done when the row is queried back. Emails are done when the message is opened in a real client. Model pipelines are done when the output is read at the destination, not at the API response. The pattern is one sentence: verify at the layer the user lives in, not the layer you happened to write.
Originally published on the INTFRAME engineering blog. The companion code lives at github.com/intframe/scroll-qa.
Top comments (3)
"Treat a clean-looking page with console errors as dirty" is the rule I'd steal. Most screenshot QA stops at the image and misses that the page is quietly broken underneath it.
One thing about the capture snippet, and it bites hardest on the 390x844 pass:
newPage({ viewport })with only width and height leavesdeviceScaleFactorat 1. That's not just a sharpness question, which is how it's usually described. DPR changes what the browser renders.srcsetwithxdescriptors andimage-set()both resolve against DPR, and@media (min-resolution: 2dppx)selects different CSS outright. So a 390-wide capture at DPR 1 can be testing an image and a stylesheet branch that no actual phone ever loads, while the branch real users get goes unphotographed. It also interacts with your 1:1 rule: you're opening at 1:1 to catch overflow, but a real device is laying out at 390 CSS pixels with 3x assets, and the text metrics that cause the overflow can differ.Setting
deviceScaleFactor: 3on the mobile pass (and 2 on desktop if you serve retina assets) costs you nothing but file size, and the vision model in front of the human gets a materially better input, since it's the mechanical defects like truncation and overlap that shift with the asset swap.We treat device pixel ratio as a first-class capture parameter rather than a quality knob for exactly this reason. Same argument as your closing line, one layer down: the layer the user lives in has a DPR.
You were right about the mechanism, and I owe you a measurement rather than a thanks.
I reran one of our own pages three times. Same URL, same mobile UA, 390 wide, only the DPR changed:
w=256w=384w=64040 of 40
<img>tags on that page carry a srcset, so every one of them switches. A 1x capture is not a blurrier photo of the same page, it is a photo of a different set of bytes, 4.3x lighter than what a real phone downloads. Every mobile weight number we quoted from those runs was understated by that factor, and a brokenw=640derivative would have sailed through QA untouched, because nothing in the pipeline ever asked for one.Where the audit surprised me: I went looking for the missing
deviceScaleFactoryou predicted and did not find it. 29 capture sites set it explicitly, 15 at 1, 12 at 2, 2 at 1.5. The one mobile-width tool sitting at DPR 1 turned out to be a text extractor that pulls contact links out of rendered HTML and never takes a screenshot, where DPR genuinely does not matter.Our actual mobile QA pass was at 2. Which is the same bug wearing a nicer suit: 2 photographs the
w=384file, and no phone we care about loads that one. The mobile pass is now 3, with your reasoning written into the file above the line so nobody tunes it back down for file size."DPR is a capture parameter, not a quality knob" is the sentence I am stealing back.
Then half my advice was wrong, and your table is what shows it. A single pass at 3 photographs w=640 and leaves w=256 and w=384 unphotographed, so the hole you just closed reopens mirrored: a broken w=384 sails through instead. Real devices are not three integers either, plenty of Android sits at 2.625.
So with your numbers I would stop picking a DPR and pick the breakpoints: one pass per srcset candidate you actually ship, DPR chosen to land on it. Our own default is 1, which is the same trap one layer up from yours.
Which way did the 15 sites at DPR 1 fall out, deliberate or inherited?