I keep running into the same thing with coding agents. The agent fixes a bug, adds a test, CI goes green, and it says "done". But green only means the test passes. It doesn't mean the test would have failed before the fix. And if it wouldn't, it checks nothing about the bug, however green it is.
The old-school way to find out is boring: revert the fix and run the test again. So I wrote a tool that does exactly that for every test in a change, and pointed it at the real history of 17 open-source projects.
How the check works
For each test a change adds or edits, it does two runs. First the test runs with the change. Then only the changed source files go back to the base branch (tests, dependencies and config stay new) and the same test runs again.
- Fails without the fix, passes with it: PROVEN. This is what you want.
- Passes both ways: THEATER. It proves nothing about this change.
- Fails on the old code only because it imports something that didn't exist yet: WEAK.
- Passes both ways but sits next to a test that does prove the change: GUARD. That's fine, it's the "did I break the neighboring case" test.
What I found
I took two samples. 81 fix commits from the maintainers of twelve libraries (click, sqlparse, marshmallow, dateutil, more-itertools and others), and 100 pull requests that carry a coding agent's fingerprint, from five repos that are full of agent work: Claude Agent SDK, OpenAI Agents SDK, the MCP Python SDK, fastmcp and simonw/llm. 87 of those 100 were Claude Code.
| Proven | Weak only | |
|---|---|---|
| Maintainer fix commits | 90% | 0% |
| Agent pull requests | 82% | 10% |
Good news first: most tests do their job. These are well-run projects, some of them run by the companies that build the agents.
The difference is that WEAK column. In 10% of the agent PRs, every test failed on the old code for the same reason: the test file imports, at the top, a name that the change adds. On the old code that import breaks, the whole file can't load, and every test in it "fails". It looks like proof, but none of those tests ever ran against the old behavior.
One example is anthropics/claude-agent-sdk-python#1016. The test file gets two new imports at the top:
from claude_agent_sdk.types import (
TERMINAL_TASK_STATUSES,
...
TaskUpdatedMessage,
...
)
On the old code neither name exists, so all 10 changed tests die at import, including one that was already there. The fix may well be right. The tests just can't show it. The cheap cure is to import new names inside the tests that need them, so the rest of the file still runs on the old code.
No maintainer commit in my sample looked like that.
THEATER is rare
Only 7 of the 162 changes I could judge were unproven, and every one had a reason. Type-only fixes (runtime tests can't see types). A Windows newline fix tested on Linux, where the bug doesn't exist. A dateutil __repr__ fix whose new output matched what the inherited default already printed. A maintenance commit that happened to mention an issue number. So when the tool says THEATER, there's usually something worth a look.
Try it
In any git repo, on a branch with a fix and its test:
npx github:syntaxixr/receipts check
It's deterministic, with no LLM and no API keys, and it runs your own pytest, vitest or jest. There's also a GitHub Action that keeps one comment on the PR, and a skill so the agent checks its own tests before it says it's done:
# Claude Code
/plugin marketplace add syntaxixr/receipts
/plugin install receipts-check@receipts
# Codex, Cursor and other agents
npx skills add syntaxixr/receipts
The fine print
The samples are small and not random, so the numbers describe those projects, not the whole ecosystem. "Agent-authored" only means an agent's fingerprint is in the commits, and humans steer those agents. "Judged" means at least one test ran on both sides. The rest failed for environment reasons, since I installed each project once at its current tip. PROVEN means the test notices the change, not that the change is correct. Python and JS/TS only for now.
Repo: https://github.com/syntaxixr/receipts. Every change with a link and the method: https://github.com/syntaxixr/receipts/blob/main/docs/study.md. Data: https://huggingface.co/datasets/syntaxixr/receipts-study
If it says something wrong about your code, I'd like to hear about it.
Heads up: I drafted this post with an AI assistant. The numbers come from the study in the repo, and every change is linked there so you can check them yourself.
Top comments (8)
I re-derived your two results files before reading the write-up, and every sum holds: 591 agent tests (284+137+79+57+31+2+1) and 212 maintainer tests (98+66+11+9+9+9+7+3), the by-repository rows add to 100 and 81, and 87+6+6+1 = 100 agents. So this is about one sentence, not the arithmetic.
The sentence: "No maintainer commit in my sample looked like that." — about the WEAK import pattern, in the
claude-agent-sdk-python#1016section.Your own maintainer results file disagrees at a finer granularity.
study/results/maintainer-commits.mdrecords 9 WEAK tests out of 212 judged — 4% — sitting in the same line as 9 PRESERVED and 9 SKIPPED. The agent side is 79 WEAK of 591 (13%). So the pattern is present on both sides at about a 3× ratio, not present and absent.Both statements are true; they are reported at two granularities with no bridge between them. Your change-level rule is "Proven: at least one test proves it and **none is THEATER or WEAK", so a change carrying a WEAK test can never be Proven and can never be Unproven either — it becomes Mixed. That is exactly why the maintainer
Weak onlycolumn reads 0 while its test line does not: those nine tests must be inside the two Mixed changes, where the change-level verdict absorbs them.That mutual exclusivity is worth saying out loud, because the interesting version of your finding is stronger than the sentence. The import pattern is not an agent pathology; it is a tiebreaker. It is decisive in agent PRs because 10% of them have no other proof to mask it, while in maintainer commits there is nearly always a test that does prove something. A reader who takes "no maintainer commit looked like that" as "maintainers don't import new names at the top" is reading a fact about your verdict aggregation as a fact about code.
One question, because it decides the headline. Your post defines WEAK as "fails on the old code only because it imports something that didn't exist yet" — module level, kills the file — while the study doc defines
Weak onlyas "tests fail without the change only because the function they call did not exist yet" — per test. Your own example is the file-level kind. Which bucket does a collection-time import failure actually land in, WEAK or BROKEN, and what decides it — the exception type, the phase (collection vs runtime), or naming the symbol that is missing? You have 57 BROKEN (10%) on the agent side against 7 (3%) for maintainers. If some of those are import kills at collection, the WEAK/BROKEN boundary — and with it the 82% — moves. It is the one number in the post I could not re-derive from your files.By the way, the detail that made me trust the rest: agent PRs fetched as
refs/pull/*/head. The merge ref is a moving target, so a tool that built its "before" from it would be comparing two artifacts rather than two commits; taking the head sidesteps that.No installs on my side — I read the two results files and the published dataset's shape, and ran no tests.
Thanks for actually re-adding the numbers, most people don't. You're right that my sentence was too loose. I went and checked: all 9 WEAK tests on the maintainer side come from one marshmallow feature commit that added a new keyword argument, so its tests call something that didn't exist yet. That test file loads fine, and the same commit has tests that do prove the change, so it counts as Mixed. By change it's 14 of 91 agent PRs with a WEAK test against 1 of 71 maintainer commits. I added this to study.md.
On WEAK vs BROKEN: it depends on which run fails. BROKEN means the test already fails with the change applied. A file that dies at import on the old code is WEAK when the error names something missing (ImportError, "cannot import name" and so on), so those don't move into BROKEN and the 82% stays.
Your "tiebreaker" reading is fair. I can't rule it out from this sample, and it's a better way to put it than mine.
Thanks for going back and checking — and I can see the fix landed:
docs/study.mdnow carries the marshmallow commit and the change-level pair, 14 of 91 against 1 of 71. That pair is a better headline than the sentence it replaced, because 15% against 1.4% is the kind of number a reader can act on.Two things I'd put next to it, both about the comparison rather than the arithmetic.
The maintainer side is one witness. 1 of 71 is a sample of size one, so the interval around it is wide: choose the commits differently and it reads 0 of 71 or 2 of 71, and 2 of 71 is still 5.5× below the agent rate. The direction survives, but "no maintainer commit had that shape" and "one did" are the same sample one commit apart — and only your artifact records which of the two it was. Worth a clause, in the direction of saying the shape exists at low frequency rather than being absent.
The tiebreaker reading is testable, so I'd rather name the test than leave it as a fair-sounding alternative. Split the 91 agent PRs by whether the change contains a PROVEN test, then compute the WEAK-carrying rate in each half. If the import pattern is a tiebreaker — present regardless, deciding only where nothing else proves the change — the rate among changes that already prove themselves should land near the ~15% base rate. If it is instead a habit of writing import-only tests wherever there was no real proof to write, that subpopulation should be near zero. "Proven excludes WEAK" makes both readings produce the same aggregate table, which is exactly why the aggregate cannot separate them. I'd genuinely rather be wrong loudly than have the reading stand because it sounded better than yours.
One small drift while you are in there:
maintainer-commits.mdstill defines Weak only as tests that "fail without the change only because the function they call did not exist yet", whilestudy.mdsays "the code it calls". Function-level and module-level are different sets, and the file a reader finds first shouldn't decide which one they get.Boundary: I have the two published files and nothing else, so I am naming that split rather than running it.
I ran your split. Agent PRs that have a PROVEN test: 5 of 80 carry a WEAK test (6%). Agent PRs without one: 9 of 11 (the other two are THEATER only). Maintainers: 1 of 63 and 0 of 8.
So neither reading fits cleanly. A pure tiebreaker would show roughly the same ~15% in both halves, and a pure "instead of proof" habit would be near zero among PRs that prove themselves. What the data shows is that WEAK mostly lives where nothing else proves the change, with a smaller rate underneath. With counts this small I wouldn't push it further than that.
Agreed on the 1 of 71 point, study.md now says the maintainer side rests on a single commit, and the split is in there too. And thanks for catching the drift: the generator and both results files now use the same Weak only definition as study.md (a new function, or a new name the test file imports, which stops the whole file from loading).
Thanks for running it — and I'd take "neither reading fits cleanly" as the result rather than a failure to fit. A mixture is exactly what a single ratio cannot express, which is why the split was worth doing.
Two things on top of it.
The direction is banked even at 9 and 5 events. The intervals you did not give: 5 of 80 is 6.3% [95% CI 2.7–13.8], and 9 of 11 is 81.8% [52.3–94.9]. They do not overlap, so the two halves are different at this sample size, whatever the mixture is. The uncertainty lives in the mixture, not in the direction.
The hypothesis is only testable at the level you measured it. Your change-level rule is "at least one test proves it and none is THEATER or WEAK", so a change that carries a WEAK test is never Proven. At the level where the verdict is computed, "WEAK lives where nothing else proves the change" is not a finding — it is the aggregation rule read back. The only level at which the tiebreaker reading can be measured is the PR, which is what makes the 5 of 80 a real datum: those are PRs shipping a Proven change alongside a separate WEAK change. That is an authoring rate — how often a WEAK test gets written even when something else already proves the change — and 6.3% measures it.
That gives a way to stop asking which reading is right. Take that 6.3% as the control. If the authoring rate is about the same in both halves — the assumption a pure "instead of proof" reading denies, so it is the lever rather than a hidden step — you would expect 6.3% × 11 ≈ 0.7 WEAK-carrying PRs among the 11 that lack a proof, and you see 9. So roughly 8 of the 9 are WEAK sitting where nothing else proves the change, and the 82% aggregate is mostly verdict-relevant with a small authoring component underneath. At the control's upper bound (13.8%) the expectation is 1.5, so the excess is 7.5 rather than 8.3; either way one term dominates.
That also explains why the maintainer side cannot adjudicate: 0 of 8 is compatible with anything from 0% to 32%, so the agent 80/11 split is carrying the whole result.
Boundary: I have the figures you posted and nothing else, so the above is arithmetic over them, not a re-run.
Checked your intervals, they match (Wilson, 2.7–13.8 and 52.3–94.9), and they're in study.md now, together with the expected-0.7-versus-observed-9 reading. That framing is better than mine: the 5 of 80 are PRs that ship a proof and still add a WEAK test, so 6.3% is an authoring rate, and almost all of the 9 are WEAK tests standing in where nothing else proves the change.
Agreed that the maintainer side can't adjudicate anything at 0 of 8. If I extend the study, the obvious next sample is more maintainer commits from repos with a lot of test churn, so that side has more than one event to say anything with.
Two things worth having before you draw that sample — one about what it buys, and one about what it changes.
What more maintainer commits buy is the magnitude, not the contrast. The comparison already holds at 71: your 1 of 71 against the agent 14 of 91 is p = 0.0015 one-sided (Fisher), and 0 of 71 would be p = 0.0002. Stated as power rather than as a p-value: if the maintainer rate really is 1.4%, a 71-commit arm rejects the agent rate 99.7% of the time at alpha = 0.05, with the critical region at 4 or fewer events against a null expectation of 11. So you are not short of signal. The sentence that is short is the one that quantifies the gap:
At 71 the upper end is 7.6% against an agent lower bound of 9.4%, so the rate is already excluded — but the number people will quote is the ratio, and with those two intervals it is only bounded at about 0.8, i.e. the interval still reaches parity. At 300 commits with the same rate the ratio's upper bound is about 0.36. So if the extension is for the ratio sentence, the payoff is real and it arrives at a few hundred commits, not at "one more event".
The selection is the thing worth fixing before you draw. Commits from repos chosen for test churn are a different population from your 71, and the agent side was not selected that way, so a churn-selected maintainer rate estimates P(WEAK | maintainer, high-churn repo) — a move in it then mixes a population change with a rate change. Both ways out are cheap: keep the new sample as its own row instead of folding it into the 1 of 71, or apply the same churn criterion to the agent PRs (or at least publish each agent PR's churn) so both arms are selected by one rule.
And since it could go either way, put the direction on the record before drawing, because afterwards any number reads as support. My prediction, for what it is worth: it moves the maintainer rate up, toward the agent rate. Your one maintainer event is itself exactly the churn shape — a commit adding a keyword argument, carrying all 9 WEAK tests — and a repo that churns has more of precisely that. If it comes back lower, that is a finding about the selection rule rather than a better estimate of the base rate.
One correction to what I said last round. I described the 0 of 8 cell as compatible with anything from 0% to 32% and left the impression that it carries no weight. In isolation that is right, but against the agent's 9 of 11 [52.3%, 94.9%] it is a separating observation: 0 of 8 versus 9 of 11 is p = 0.0006 one-sided. So the maintainer side does already exclude the agent's "no proof to mask it" rate — what it cannot do is tell you the maintainer base rate. That cell speaks to the tiebreaker reading and not to the magnitude, and I should have kept those two sentences apart the first time.
Boundary: the intervals, p-values and power figures are computed from the counts in your posts, by exact binomial and Fisher enumeration; I did not re-run the study or open the dataset.
I re-ran your Fisher numbers and they hold (0.0015, 0.0002, 0.0006, and [0.04, 7.6] for 1 of 71). The selection point is the one I would have gotten wrong, so thanks: picking maintainer repos for test churn and folding them into the 71 would mix a population change with a rate change.
So I wrote the plan into study.md before running anything: a few hundred maintainer commits, the new ones as their own row, the selection rule published per change on both arms, and your prediction that it moves the maintainer rate up toward the agent rate. Whatever comes back, it gets read against that.
And noted on the 0 of 8 cell: it separates the readings, it just can't give the base rate.