The agent finished in about four minutes. Ninety-one tests, all green, 100% line coverage on a service I had been meaning to test properly for a year.
I felt great for roughly one day. Then a null in a config field took the thing down in staging, and I went looking for the test that should have caught it.
The test existed. It passed. It asserted that a mock I had written returned the value I had told the mock to return.
AI-generated unit tests are extremely good at the thing we measure and extremely bad at the thing we actually want. Those are not the same thing, and coverage has been hiding the gap for twenty years. AI just made it free to exploit.
TL;DR
- Coverage measures which lines ran, not which lines were checked. An agent optimizing for coverage will execute every branch and assert almost nothing meaningful.
- In my run on a mid-size Python service: 91 AI-written tests, 100% line coverage, 96% branch coverage, and a mutation score of about 31%. Two thirds of deliberately broken code still passed the suite.
- The four killers: tests that assert the mock, tests written from the implementation instead of the spec, happy-path-only inputs, and snapshot tests that freeze current behavior as correct.
- The fix that worked: run mutation testing, then hand the surviving mutants back to the agent as the target. Twelve more tests took the score to roughly 68%.
- Agents are great at machine-checkable goals. Give them a real one instead of coverage.
What happens when you let AI write your unit tests?
You get a suite that runs everything and verifies almost nothing. Here is the actual experiment.
I took an internal service, roughly 4k lines: request validation, a pricing calculator, a retry wrapper around a payments API, and some config parsing. I deleted the sad existing test suite and pointed an agent at the repo with a simple instruction: write unit tests, get coverage up.
It did. 91 tests. Green. 100% line coverage.
Then I ran mutation testing on it, which is the only test-quality metric I still trust. Mutation score: about 31%. Meaning if you randomly break the source code, roughly two out of three breakages sail past the entire suite without a single red test.
For calibration, my hand-written suites on similar code usually land in the 60-75% range and I do not consider those good.
Why do AI-generated unit tests get 100% coverage but catch no bugs?
Because coverage is a metric about execution and the agent is optimizing exactly that. Four specific failure patterns showed up over and over in my suite.
1. The tautology test. The single most common shape:
def test_fetch_user_returns_user(mocker):
mock_repo = mocker.Mock()
mock_repo.get.return_value = {"id": 7, "name": "Ada"}
svc = UserService(mock_repo)
result = svc.fetch_user(7)
assert result == {"id": 7, "name": "Ada"}
That test passes if fetch_user is return self.repo.get(id). It also passes if fetch_user does that plus deletes your database. The expected value is right there in the setup. You wrote the answer on one line and checked it three lines later. This is a mock reflecting your own face back at you, and it will be green forever.
2. Tests written from the implementation, not the spec. This one is genuinely dangerous. The agent reads your function to decide what "expected" means. If your error path returns None where it should raise, the AI test asserts None. Congratulations, the bug is now load-bearing, documented, and defended by CI. The next person who fixes it correctly gets a red build and assumes they broke something.
Human juniors do this too. The difference is that an agent does it 91 times in four minutes and never gets a bad feeling about it.
3. Happy-path gravity. Bugs live at boundaries: < versus <=, empty list, zero, the 32nd day, the timezone-naive datetime. Agents test the middle of the domain, because the middle is what the docstring describes. My suite had eleven tests for the pricing calculator and not one passed a quantity of zero.
4. Snapshot tests as crime scene photography. A few functions got approval tests where the expected output was whatever the code produced at generation time. A snapshot written by something that has never seen the correct output is not a test. It is a photograph of the current state, with an alarm that goes off only if the state changes. That is useful for refactors and worthless for correctness, and mixing the two in one suite is how you get a team that stops believing red builds.
There is a fifth, subtler one: over-mocking. The agent mocked my own internal modules with enthusiasm, which means every seam where things actually break in production (serialization, DB constraints, clock behavior) was replaced with a polite stub that always cooperates.
What is mutation testing and why does it expose AI tests?
Mutation testing deliberately breaks your source code and checks whether your tests notice. Flip > to >=. Change + to -. Replace a return value with None. Delete a line. Each of those is a "mutant." If the suite still passes, the mutant survived, and your tests would not have caught that bug. Mutation score is killed mutants divided by total.
Coverage asks: did this line run?
Mutation asks: would anyone notice if this line were wrong?
Only the second question is the one you actually care about, and it is the exact question AI-generated unit tests are worst at answering. Tools: mutmut or cosmic-ray for Python, Stryker for JS/TS, PIT for Java. Run it on one module first, not the repo, because it is slow by nature (it reruns your suite once per mutant).
How do you get AI to write tests that actually catch bugs?
Change the target. The agent will hit whatever you point it at, so stop pointing it at coverage. Five rules that moved my score from 31% to about 68%:
Feed it the spec, not the file. Paste the ticket, the docstring, the API contract, the invariant in plain English. Then say explicitly: do not read the implementation to decide expected values. This one change kills failure mode #2 outright.
Make it write the failing test first. Every time you fix a bug, have the agent produce a test that fails on the pre-fix code and passes after. If it cannot make the test go red on the old code, it did not understand the bug, and you just learned that in ten seconds instead of at the next incident.
Ban mocks of your own code. Mock the network, the clock, and the credit card processor. Not your own functions. Put it in the prompt, put it in CLAUDE.md or your rules file, and reject the diff when it shows up anyway.
Feed it the survivors. This is the whole trick. Run mutation testing, take each surviving mutant as a small diff, and hand it over: "here is a change to the source; write a test that fails when this change is applied." That is a concrete, machine-checkable target, and agents are outstanding at concrete machine-checkable targets. It converts a vague quality goal into a pass/fail loop they can grind on.
Ask for one property-based test per module. hypothesis, fast-check, whatever your stack has. Agents are decent at stating invariants ("total is never negative", "encode then decode is identity") and terrible at inventing nasty inputs. Property testing splits the labor exactly along that line: the model writes the invariant, the framework finds the input that breaks it.
The three questions I ask every AI-written test
Fast review filter, takes about five seconds per test:
- If I delete the assertion, does the test still pass by itself? Then it is a smoke test. Fine, but name it that and stop counting it.
- Does the expected value appear anywhere in the setup? Tautology. Delete or rewrite.
- Would this test fail if I inverted one comparison in the function? If you cannot answer instantly, that function has no real test.
So are AI-generated unit tests worth it?
Yes, with a different job description. Agents are fast and tireless at the boring 60% of testing: fixture scaffolding, parametrized input tables, mock plumbing, converting one good test into twelve variations, and grinding against a mutation-survivor list until it is empty. What they cannot do is decide what "correct" means, because they infer it from the code you already wrote. The honest summary of my experiment: AI-generated unit tests gave me 100% coverage and a 31% mutation score, which means the suite proved my code runs and proved nothing about whether it works. Coverage was always a bad metric. The difference now is that faking it costs four minutes instead of a week, so if coverage is still the number on your dashboard, you are not measuring your tests anymore. You are measuring your token budget.
What is your mutation score? Run it on one module and post the number. I suspect a lot of green dashboards are about to get interesting.
Top comments (1)
The tautology test is the one I'd bet shows up in almost every AI-
generated suite, "write the answer in the mock setup, check it three
lines later" is such a natural pattern for an agent optimizing for a
passing test that it barely looks wrong on review. It only becomes
obviously worthless once you frame it as "does the expected value
appear anywhere in the setup," which is a genuinely good five-second
filter.
Feeding surviving mutants back as the target is the sharpest fix here
though, it reframes "write a better test" (vague, no feedback signal)
into "write a test that fails on this specific diff" (concrete,
checkable), which is exactly the shape of problem agents are actually
good at. That's the same lesson as a lot of agent-tooling posts lately:
agents don't fail at reasoning, they fail when nothing in the loop
tells them what "good" actually means.
31% to 68% via five concrete rules, rather than "use a smarter model,"
matches the pattern too. The gap was structural, not capability.