My visual regression check reported zero pixels changed across 28 screenshots.
Every one of those pages had a bright red background injected into it.
28 screenshots compared
0 px changed
PASS
Three separate defects were queuing up behind that zero. The check had been green since the day it was written, which is exactly what made it worthless.
The thing that changed
Test-driven development asks you to write a failing test before the code. Most engineers have never worked that way, and the profession has largely made peace with that. Tests get written after, or alongside, the suite goes green, and life continues.
That peace held because writing a test by hand carried some verification as a side effect. You had to imagine the failure in order to type the assertion. You pictured the thing breaking, decided what breaking looked like, and wrote that down. Nobody called it a methodology, but it was doing real work.
An agent generating forty tests from an implementation that already passes does none of that. It reads what the code does and asserts that the code does it. Every assertion is true at the moment it is written, by construction, and none of them has ever distinguished a working system from a broken one.
The output is a suite that raises coverage while verifying nothing. It is harder to catch than a missing test, because it looks reviewed. Agents produce well-named, well-shaped, arrange-act-assert tests. They look like the work of someone careful.
What the red step was actually for
The ordering in TDD exists to deliver one guarantee: that a human being has personally watched this assertion distinguish broken from working. Writing the test first is simply the most convenient moment to collect that guarantee, because the code does not exist yet and red is free.
Which means the guarantee is separable from the ordering. You can write the code first, the way nearly everyone and every agent actually works, and still collect it. You just have to pay for it afterwards, deliberately, instead of getting it for free.
The pattern: Red Once
No assertion is trusted until you have watched it fail, for the reason you think it fails.
Performed at the moment a test is authored, not retroactively across an existing suite.
1. Break the code under test
One change, inside the code under test. Invert a condition. Return a constant. Delete the line that does the work. Rename the column the query selects.
Editing the test instead proves nothing, because a test can always be made to fail by breaking the test.
2. Run that one test and read the failure
It has to go red, and the message has to name the thing you broke.
$ pytest tests/test_balance.py::test_excludes_body_only -x
FAILED assert 4200 == 3900
where 4200 = invoice_total(session)
1 failed in 0.41s
3. Restore, re-run, and write down what you broke
Green again, and the mutation recorded next to the test. One line is enough.
def test_excludes_body_only(session):
# red-once: returned the unfiltered total from
# invoice_total() -> 4200 != 3900. 2026-10-13.
assert invoice_total(session) == 3900
This is the step everybody skips, and it is the one that turns a private act of diligence into something the next person can repeat.
Where it sits
Red for the right reason
A test that fails is not automatically a test that works. It can go red because an import blew up, a fixture collapsed, a migration had not run, a network call timed out, or a mock went stale. All of those produce red, and none of them tells you anything about your assertion.
So step 2 has two conditions, and the second one is the real one: it failed, and the failure names the thing you broke. If you inverted a filter and the test reports a connection error, you have learned nothing and you are not finished.
This is the same error that produced the red page and the zero. The check passed for a reason that had nothing to do with the pages matching: the diff library painted unchanged pixels as faded grey, the test read colour channels for any value at all, so nearly every pixel registered as unstable and the mask built to ignore instability grew to cover the entire page. A correct answer arrived by a route that could never have produced a wrong one.
What was hiding behind the zero
The harness was wrong three ways at once, and all three were invisible to a green run.
- The mask ate the page. The diff library paints unchanged pixels as faded greyscale rather than leaving them transparent. The comparison tested colour channels for any value, so almost every pixel counted as changed, and the instability mask covered everything. Reading the alpha channel instead fixed it.
- A footer that was never photographed. Screenshots captured the viewport. One of the footers sat four thousand pixels down the page.
- Two sides on different frames. The CSS animations were keyed to page load, so the comparison was never made at the same moment in the animation, and the drift was absorbed as noise.
There was a fourth problem in the threshold itself. It was expressed as a ratio of total pixels, which scales with page height, so the longer the page the more a real regression could hide inside an acceptable percentage. Absolute pixel counts, set just above a measured noise floor, do not have that property.
All four surfaced within an hour of injecting a regression the check was supposed to catch and refusing to accept green.
Make the agent do it
This is the part that makes the discipline survive contact with how software is now written. An agent that can write a test can also break the code, run the test, and show you the failure. It is a strictly mechanical sequence, which is the kind of work agents are good at.
Add it to your agent instructions, and require the evidence rather than the assurance:
After writing a test, prove it discriminates:
1. make one change to the code under test that
should break it
2. run that test alone, capture the output
3. confirm the failure names what you changed
4. revert, re-run, confirm green
Paste both outputs. Do not describe them.
The captured failure is the artifact. It goes in the pull request next to the diff, and it changes what a reviewer is looking at. A recorded failure is a receipt, and a reviewer can check it in seconds.
| The old review question | The question that survives agents |
|---|---|
| Is there a test for this? | Has this test ever failed? |
| Answerable by looking. Satisfied by anything shaped like a test. | Answerable only with evidence, and the evidence is cheap to produce and impossible to fake by accident. |
What it costs, and where it stops
- About a minute per test. Break, run one test, read, revert, note. One case, start to finish.
- It is not retroactive. Nobody is going to go back through four thousand existing tests. Do it at authoring time, and do it for any test standing guard over something whose failure would be expensive or silent.
- One mutant is enough. The goal is to establish that the assertion discriminates at all, not to prove it catches every possible defect.
- It does not replace the tools. Mutation testing frameworks do this exhaustively and automatically. If you can run one, run it.
On the name and the idea
None of the mechanism here is new. Deliberately breaking code to confirm a test notices is the manual form of mutation testing, which has been studied since the 1970s and has mature tooling: Stryker, PIT, mutmut.
What changed is the cost of skipping it. For decades this was a nicety, because hand-writing a test did some of the work incidentally. Generated tests do none of that work, so the step everyone used to skip is now the one holding the suite up. It needed a short name and a place in the workflow.
Write the code first if you want, but never trust an assertion you have not watched fail for the right reason. That is all of Red Once.
Top comments (2)
Your second condition is the one that would have caught ours. We had two checks that had never been red.
One compared a screenshot before and after a swipe on a real iPhone to decide whether the feed moved to the next video. It was green on swipes that bounced back, because the video was playing and the frame changed without the page changing. Same shape as your mask: a right answer by a route that couldn't give a wrong one.
The other was a session recycle that was supposed to fire after 60 minutes. It read process age with
ps -o etimes=, which Linux has and macOS doesn't, so on the Mac it never got an age and never fired. It looked like a safety and did nothing.What I took from it is close to your step 1, but for actions instead of code: do the thing wrong on purpose. A flick we know is too slow to page (on TikTok ours bounced at 295 points a second) has to come back as "did not advance" before the check gets trusted.
How do you record the mutation when the thing under test is a device and not a function? A line next to the test works for code. I haven't found the equivalent for a gesture.
The line moves out of the source and into the run, because for a device the thing it describes goes stale without anyone touching the code.
For a function the mutant is temporary: invert, watch it go red, revert. If you're driving an app you don't own, the stimulus is the only input you control, so that's where the mutant goes, and it stays there. You already have the number, so commit 295 as a standing control. The check then carries two: a flick past the threshold that has to come back "advanced", and 295 that has to come back "did not advance". Its verdict only counts on a run where both landed right. Your screenshot differ fails that on day one.
The record is a row, because the number is only true for one device on one build:
Your recycle belongs to a different class. An empty value was read as "not old enough", so the check never reached a verdict. On macOS that keyword exits 1 with nothing on stdout, so capturing output without checking exit status is what kept it quiet. Could-not-measure needs to fail the run in its own right.