Last week I published a study of 181 real code changes, and one sentence in it was wrong. A reader found it by re-adding my numbers. This is what they found, what I changed, and what is still open.
The sentence
I'd written that in 10% of agent-written pull requests every test failed on the old code only because the test file imports a name the change adds (so the file can't even load), and that "no maintainer commit in my sample looked like that."
The reader re-derived my results files before reading the write-up. Every sum held. Then they pointed at a line I'd skipped: the maintainer file lists 9 tests with the verdict WEAK out of 212 (4%), while the agent file lists 79 out of 591 (13%). So the pattern isn't absent on the maintainer side.
I went and looked. All nine come from one marshmallow feature commit that added a new keyword argument. Its test file loads fine, and the same commit also has tests that do prove the change. So the sentence was about "every test is WEAK", and the fix was to say what the maintainer data actually contains. By change, 14 of 91 agent PRs contain a WEAK test, against 1 of 71 maintainer commits.
The split that made it more interesting
The reader's next question: does the import pattern show up even when something else already proves the change, or only when nothing does? Splitting the 91 agent PRs by whether they have a proving test:
- With a proving test: 5 of 80 carry a WEAK test (6.3%, 95% interval 2.7 to 13.8).
- Without one: 9 of 11 (81.8%, interval 52.3 to 94.9).
If WEAK tests were written at the same 6.3% rate everywhere, you'd expect about 0.7 of the 11 to carry one. There are 9. So most of it sits exactly where nothing else proves the change. The reader also pointed out the catch: my change-level rule never calls a change Proven if it carries a WEAK test, so part of that is the aggregation rule read back. The 5 of 80 is the only number that isn't, because those PRs ship a proof and still add a WEAK test. That's an authoring rate, and it's small but not zero.
I checked their statistics myself (Wilson intervals, Fisher exact tests). They hold. The maintainer side is a different story: 1 of 71 is one event, so it can't tell anyone the base rate, however clean the contrast looks.
What changed
- The study doc now says all of the above, with the intervals, and one definition of "weak only" everywhere (my generator and two results files disagreed with the doc).
- Receipts, the tool, now prints how each PROVEN test failed on the old code. A different reader, on Reddit, said "it went red" isn't enough: you want to see that the test reached its assertion, not that it died on an import. The tool knew that and wasn't showing it.
- I wrote down the next step before running it: a few hundred more maintainer commits, as their own row, with one selection rule applied to both sides. The reader predicted the maintainer rate would move up toward the agent rate, because the one maintainer event is itself a churn-shaped commit. That prediction is in the doc now, so whatever comes back gets read against it.
What I took from it
Two things. One: re-derive someone's sums before you argue with their sentence, because the sentence is where it's loose. Two: a nice headline number ("10% of agent PRs") can be a fact about your verdict rules as much as about the code, and the only way I found that out was a reader who wasn't invested in the answer.
The tool is open source and free: https://github.com/syntaxixr/receipts. The study with every change linked: https://github.com/syntaxixr/receipts/blob/main/docs/study.md
Heads up: I drafted this post with an AI assistant. The numbers are from the study files in the repo and from the comment thread under the original post.
Top comments (0)