DEV Community

Nikolay Chernev
Nikolay Chernev

Posted on

100% Coverage, 4% Mutation Score: The Gap That Explains Why Your AI Tests Pass Everything

#testing #ai #qa #mutationtesting

100% Coverage, 4% Mutation Score: The Gap That Explains Why Your AI Tests Pass Everything

One team ran mutation testing on an AI-generated suite that reported 100% line coverage. Every statement executed. The mutation score — the share of deliberately injected bugs the tests actually caught — came back at 4%. Shiplight's write-up walks through exactly how that happens, and it's worth sitting with the number: 96% of injected defects slipped straight through a "fully covered" suite.

That gap is the whole problem with trusting an AI's own tests, and it's sharper than the usual "AI hallucinates" complaint. Coverage tells you code ran. It says nothing about whether anything was actually checked.

Why coverage lies

Line coverage counts a statement as tested the moment it executes inside a test function — even if the assertion that follows is expect(result).not.toBeNull() on a function that was never going to return null. Thoughtworks' Technology Radar flags this directly: high coverage percentages routinely mask logically hollow tests, especially generated ones.

There's a structural reason AI-written tests drift this way. When the same model writes the implementation and the test, both inherit the same blind spots. One practitioner account (HackerNoon) describes a counter that increments whether or not a request actually succeeds — and a test suite that treats "counter went up" as proof the request worked, because that's the assumption baked into the code it was copying. The bug and the test agree with each other. Nobody disagrees, so nothing goes red.

Mutation testing is the counter-check: inject a small fault (flip a comparison, drop a line, change a constant) and rerun the suite. If nothing fails, that mutant "survived" — and a surviving mutant means some class of real bug could ship with every test green. In one informal report, 20 AI-written tests against a Laravel controller reported full endpoint coverage but killed only 68 of 95 injected mutants — 27 bugs that full coverage didn't catch.

A reader already found the sharper version of this

Last week I wrote about telling a real bug from a model failure. A reader, Foundry, left a comment describing something worse than a weak assertion: a check that is structurally incapable of failing.

Their example: an automated mobile check hashed a screenshot before and after a swipe to confirm the feed advanced. On a playing video, the hash changes every single frame — so the check passes whether or not the swipe did anything at all. A second check read process age with ps -o etimes=, which doesn't exist on macOS; the command failed silently, the age never crossed the threshold, and the rule never fired, ever. Neither check had ever gone red, so neither got flagged for review. Their question is the one worth stealing: could this check pass if the thing I care about did not happen?

That's mutation testing in one sentence, applied by hand instead of with a tool.

What to actually do about it

You don't need a mutation-testing framework to start. Before trusting any AI-written test — yours or a teammate's — ask it Foundry's question directly: if I introduce the bug this test is supposed to catch, does it fail? If you can't answer that without running it, that's the review item, not "does it pass."

The free lab I maintain is a runnable version of exactly this comparison. node run-lab.js weak generates AI tests against deliberately broken code — they pass 3/3. node run-lab.js verify-candidate runs an independent verification suite against the same code — it fails, and it catches all 4 seeded bugs. The difference between those two runs is the entire argument of this post, reproducible in about two minutes:

Free lab: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

If this is a problem you deal with weekly and you want the fuller method — how to tell a real failure from a model failure, how to build an independent verification layer, worked examples — that's the subject of the book:

Don't trust it — run it.

Top comments (0)