Intro
There's a pitch behind every "AI writes your tests too" workflow: more coverage, less manual toil, a safety net that used to take a sprint now takes minutes.
The pitch skips over what that safety net is actually made of. When the same model writes the implementation and the test suite, you haven't added a second, independent check. You've asked one reviewer to grade its own homework and handed you the green checkmark as if someone else had signed off.
The blind spot loop
A model reasons about a function once, forms an implicit set of assumptions (input shapes, timezone handling, what counts as "empty"), and writes the implementation against those assumptions. Ask the same model to write tests for that function, and it doesn't re-derive correct behavior from scratch. It writes tests against the same assumptions it just used to write the code. If it assumed dates always arrive as ISO strings in UTC, the implementation assumes that, and the tests assume it too. The suite goes green. The assumption is still wrong.
Tests that pass for the wrong reason
(Illustrative, not a specific case, but recognizable to anyone who's shipped an AI-generated suite.) Picture a discount-calculation function where the model assumes quantities are always positive integers. The implementation skips a negative-quantity check. The generated tests exercise 1, 5, and 100, because those are the "normal" values a model reaching for plausible test data will reach for. Nothing ever asks what happens at -1 or 0, because neither pass, the code or the tests, ever considered them worth asking about. Coverage tooling reports 100% on this function. The bug ships anyway.
Coverage becomes a false signal
High line or branch coverage from an AI-authored suite tells you the code paths were exercised, not that the right inputs exercised them. A suite can hit every line of a function and still never send it a null, an empty array, a duplicate key, or a value at a type boundary, if the author, human or model, never imagined those as possibilities in the first place. Coverage percentage was never built to detect a shared blind spot. It just counts what got tried.
What to keep human, and what to hand to AI
The fix isn't "stop using AI for tests." It's separating the two jobs testing actually does:
- Defining correct behavior (the assertions) should come from a human who understands the spec or the business rule the function is supposed to honor, decided independently of however the implementation happens to work.
- Generating volume and variety (mock data, randomized inputs, edge-case permutations) is exactly what AI is good at, and doing it well doesn't require the model to have written the implementation.
- Human review time is best spent on boundary conditions specifically: zero, negative, empty, duplicate, malformed, concurrent, because that's statistically where both human and AI implementations tend to fail first.
This isn't really a testing problem. It's a correlated-error problem wearing a green checkmark. Two independent reviewers catch different mistakes because they're independent. One reviewer checking its own work twice catches the same mistakes it already missed, twice.
Where's your line? Do you let AI touch your test assertions at all, or only the scaffolding, mocks, fixtures, input generation, around them?
Top comments (5)
The cheapest defence I know against this costs about a minute, and I only started using it after being caught by exactly your loop.
I wrote a function that splits work across streams, and a test for it. Both green. Then, on a whim, I replaced the algorithm with a deliberately wrong one - plain block-splitting instead of the interleaving it was supposed to do. The test stayed green. It had rebuilt the same computation to check the computation, so it agreed with whatever the implementation did. Written by one mind from one understanding, exactly as you describe, and the fact that the mind was mine rather than a model's changed nothing.
So the rule here now is: after writing a guard, break the thing it guards and watch it go red. A test nobody has ever seen fail is not evidence, whoever wrote it. It also picks the fight your coverage number cannot - coverage tells you the line ran, the counter-check tells you the assertion bites.
On your question, my line ended up somewhere slightly different from where you drew it. It is not about which side writes the assertion. It is that unit tests only ever prove the code does what its author meant. Last night I wrote a detector plus its tests, all green, and then ran it against real stored data: it found nothing, because the keys I had chosen were not the kind of key that can collide. No test would have caught that. Only the data did.
Mutation testing by hand, basically, and yeah it's the cheapest reality check going. I've started doing something similar: after a test passes I go break the invariant it claims to enforce and make sure it actually screams. If it doesn't, the assertion was decorative. Cheap, fast, embarrassing how often it catches something.
Your second point is the one that really lands though. My whole post was about correlated authorship, but you're pointing at a bigger version: correlated worldview. The author (human or model) picks the test inputs from the same mental model that shaped the code, so the tests can only probe the space the author already imagined. Real data doesn't care what you imagined. I think that's why staging replays and shadow traffic end up finding stuff no suite ever does, the inputs weren't curated by anyone with a stake in the code being right.
"Real data doesn't care what you imagined" holds one storey up too, and it caught me the same week.
I built a detector for a specific defect and measured it on eleven known-good pairs: median score 0.966, eight of eleven above 0.9. On that table alone I would have shipped it. Then I ran the same thing on pairs with no relation at all - the control I nearly skipped, because the first number already looked settled. Median 0.818, five of eleven above 0.9. At the threshold that made the first table look good, precision was 62 %.
Nothing about the eleven real pairs was wrong. I had picked which half of the data to look at, out of the same mental model that built the detector. Your loop, one level up: the measurement inherits the blind spot as readily as the test does.
The counter-check at that level is naming the number before you run it. A different detector I tried the next day found nothing at all in 735 real entries - and "nothing" is the dangerous shape, because it reads as "there was nothing to find" rather than "this design cannot fire". It was the second one. Writing "≥ 5 of 11 known cases, at most 5 false positives" beforehand turns a zero into a refutation instead of a shrug.
That is also the gap your replay argument does not close, and it is worth naming next to it: shadow traffic surfaces inputs you did not imagine, but it cannot surface an input that never occurs. A null result nobody predicted in advance is invisible to both a suite and a replay.
Yeah, this is the version I should have written. You've basically got three nested layers of the same failure: the code inherits the author's model, the tests inherit the code's model, and the evaluation inherits the author's model of what's worth measuring. Replay fixes layer two, sometimes. It does nothing for layer three because the metric itself is still curated by the person who wants the thing to work. Selection of the eval set is doing the same job selection of test inputs did one floor down.
Pre-registering the number is the right move, and it's underused outside of research contexts. ≥ 5 of 11, at most 5 FPs is a claim that can lose. "Median 0.966 on known-goods" is a claim that already won by the time you wrote it down. And your null-result point is the sharpest part: a silent detector is indistinguishable from a well-behaved world unless you said in advance what a firing one should have looked like. I'm going to steal that framing, "a zero is only a refutation if someone predicted a non-zero," for the follow-up.
Three layers is the right count, and there is a fourth underneath it that I only found by accident last night: the assumption that one run is a measurement.
A restart made me run the same condition three times with identical inputs - same tasks, same harness, and a control arm with no memory at all. It solved 15, 15 and 19 of 30. Four cells of movement with nothing changed.
Which means the pre-registered number needs a companion: how big is the number that would have shown up anyway? The cheap version is running the control twice before believing a single delta. One extra run, and it tells you whether you have a result or a coin.