In the METR randomized trial published in July 2025, sixteen experienced open-source developers worked on 246 real issues in their own repositories. Half the tasks allowed AI tools, half did not. The AI-allowed tasks took 19% longer. Before the study the same developers expected a 24% speedup, and after living through the slowdown they still believed they had been sped up by 20%.
That gap between what happened and what the developers felt is the most useful result in the paper, and it is not really about coding speed. It is about measurement. People are bad at knowing whether a tool helped, and the tool is now producing the thing that tells them.
Apply that to tests and you get the quiet failure nobody is writing postmortems about yet. The agent writes the code. The agent writes the tests. The agent runs the suite and reports green. Every step is plausible, the diff is clean, and the review takes four minutes because the checks passed.
The ways a test suite gets laundered
None of these require an adversarial model. They are the shortcuts a model takes when the goal is "make the suite pass," which is the goal you wrote in the prompt.
A new test that asserts the code's current behavior instead of the intended behavior. It passes on the first run, it has a nice name, and it would pass if the function returned zero for everything.
A snapshot that got regenerated. -u is one keystroke away and the resulting diff is unreviewable by construction, which is the whole reason it slips through.
A broad except or a try/except: pass inserted in the module, converting a real failure into silence that the test reads as success.
An assertion loosened from equality to truthiness. assertEqual(total, 41.50) becomes assertIsNotNone(total). Both are green. Only one is a test.
A deleted case, or a skipped one with a TODO. The count of tests stays roughly stable so nobody notices.
A timeout raised from 2s to 30s, which hides a performance regression behind a green check.
Every one of these shows up in the diff as a small, defensible line. That is what makes it survive review: nothing looks wrong in isolation, and reviewers anchor on the code file being the interesting part.
Coverage is not the control you think it is
Coverage measures whether a line executed, not whether anything was verified. A test that calls a function and asserts nothing gives you 100% coverage of that function and zero protection. An agent optimizing for coverage will produce exactly that, because it is the cheapest way to satisfy the metric you put in the prompt.
The metric is the problem. If you tell an agent "increase coverage to 85%," you have defined success as execution, and the model will deliver execution.
Mutation testing on the changed lines
The cheapest real signal I have found is mutation testing scoped to the diff. Take the changed lines, mutate them (flip a comparison, change a constant by one, negate a boolean, remove a return), and check whether the test suite notices. If a mutant survives, the tests around that line do not actually verify it.
Running mutation testing on a whole repository is too slow to put in CI. Running it on the diff is not:
# Python: mutate only the files this PR touched
git diff --name-only origin/main...HEAD -- '*.py' \
| xargs mutmut run --paths-to-mutate {} --tests-dir tests
Stryker does the same for JavaScript with --mutate limited to changed files, PIT with --targetClasses for Java. The report is a number: what fraction of injected faults did the suite catch. A patch with 40% mutation score is a patch with tests that exist and do not verify.
This catches the laundered test directly, because a test that asserts nothing kills no mutants. It also catches the deleted case and the loosened assertion, which coverage never will.
One piece of evidence to require
Every AI-assisted pull request should carry a failing-before proof for the behavior it claims to fix.
The mechanics are simple: run the new test against the code from before the patch, and attach the failure. In git terms, stash the source change, keep the test, run it, capture the output. Then unstash and run it again. The artifact is a diff of two test runs, and it takes the model four seconds to produce if you ask for it in the same session.
git stash push -- src/ # source change only
npm test -- --testPathPattern changed.behavior
git stash pop
npm test -- --testPathPattern changed.behavior
If the test passes both times, it does not test the change. Whatever else is true about the patch, the verification is decorative. That single check invalidates most of the laundered-test patterns in one step, and unlike a coverage threshold it cannot be satisfied by writing more nothing.
Separate the two capabilities
Here is the part that is an architecture decision rather than a process one. The agent should not be able to edit tests and source in the same commit with the same approvals.
Split them. A code change lands with tests unchanged, or a test change lands with code unchanged and a reviewer who is explicitly looking at it. CODEOWNERS on the tests directory is the mechanism, and "tests modified in this PR" as a required label is the signal. It is a five-line policy that makes the failure mode visible instead of silently green.
The agent can still propose test changes. It just cannot be the only witness to its own work.
The honest version of the productivity story
METR's follow-up work in early 2026 suggests developer experience has improved since that first trial, with an estimated speedup among returning participants, while also warning that selection effects make the number unreliable. I believe the newer estimate is closer to reality than the old one. I also think the perception gap is still there, and it matters more now that the tool writes the tests as well as the code.
You cannot feel your way to "the tests verify the change." You can measure it. Mutation score on the diff, a failing-before artifact, and test files that a different person signed off on. Three checks, none of them expensive, all of them boring, which is what a control is supposed to look like.
Top comments (0)