Two people left long, technical comments on the posts I published this week. One of them asked for an experiment I should have run before publishing anything at all.
The original number was 10/10: take ten real CVE fix commits, revert each one, and ask our security agent to find what was reintroduced. It found all ten. I wrote a whole post about why that number is weak, and listed four reasons. A reader pointed out the fifth, with a design:
For each of the ten cases, revert a commit of about the same size in the same files that fixed nothing security related, and run the reviewer with the same settings. If it reports a vulnerability on many of those, ten out of ten says the reviewer notices removed checks rather than the bug.
He is right, and the reason cuts deeper than "we forgot a control". A reverted fix is a diff that deletes a guard. If the reviewer is pattern matching on a guard was removed, then any reverted commit that removes a guard gets flagged, and the ground truth (this was a CVE) is doing none of the work. Without the control, 10/10 cannot distinguish "finds vulnerabilities" from "notices deletions".
The design
Ten control commits, one per original case, chosen to be as close as possible to the fix it replaces:
- same repository, and where the file had enough history, the same source file;
- an already merged commit with a non-security subject (no CVE, no security wording, no revert);
- comparable diff size;
- same treatment as the original ten: check out the commit,
git revert --no-commit, restoredocs/tests, commit, then run only the security agent onHEAD~1with no PR description and no commit message.
| Case | Fix (lines / date) | Control commit | Lines | Date | What it actually changed |
|---|---|---|---|---|---|
| c01 | 264 / 2026-11-03 | c709a748 |
42 | 2023-12-20 | Dropped support for GDAL 2.4 |
| c02 | 72 / 2026-11-03 | 77d0c9d5 |
37 | 2026-06-18 | Admin displays database defaults as empty |
| c03 | 87 / 2026-11-03 | a0f2e29e |
50 | 2026-11-12 | Deprecated passing table names to createcachetable
|
| c04 | 84 / 2026-11-03 | d3954c98 |
52 | 2026-11-16 | Re-raise RecursionError as ValueError
|
| c05 | 176 / 2026-11-03 | 7732f942 |
117 | 2026-02-10 | Made the multipart parser class pluggable |
| c06 | 110 / 2026-10-06 | be6a149d |
62 | 2026-09-07 |
ModelFormSet respects custom Form.add_prefix()
|
| c07 | 154 / 2026-10-06 | 9ea5f4fa |
163 | 2026-07-24 | Typos and grammar in docs and docstrings |
| c08 | 55 / 2026-08-04 | 4b2b4bf0 |
35 | 2026-03-18 | Admin boolean icons for related fields |
| c09 | 97 / 2020-11-24 | 58b16c88 |
81 | 2026-07-15 | Extract duplicated setFormDataHeaders
|
| c10 | 29 / 2021-02-20 | 0cec2257 |
38 | 2019-10-16 |
_.isEqual and circular references |
This is not a clean matched experiment, and I would rather say so here than have someone find it later. Three controls are years away from their fix in repository time (c01 is 2023 code against a 2026 fix, c10 is 2019 against 2021, c09 is six years later). Two controls are mostly docs or docstrings, which are trivially quiet. c03 and c04 are dated after their fix. c01's control is a sixth of the size of its fix. For files with almost no history there was no same-file, same-size, same-year commit that was not itself a security fix, so "nearest available" was the honest ceiling.
What the control showed
All ten controls eventually completed. The axios case timed out twice on the provider; after a cooling-off period a third attempt succeeded, so it is included. I am keeping that detail because "ten clean runs" would be a nicer sentence and a less true one.
| Complete silence | 7 / 10 |
| Commented | 3 / 10 |
| Total comments | 3 |
| All three graded as noise for a security review | 3 |
So the cheap version of the criticism does not hold: the reviewer was not flagging every reverted commit. It stayed quiet on seven of ten ordinary reverts, against 10/10 on the CVE reverts. Something in the security patches is being detected.
But the three comments it did produce are more interesting than a clean zero would have been, because all three are the same failure mode. None is a random hallucination. Each describes the reverted change accurately, then promotes an ordinary robustness bug to a security vulnerability.
d3954c98(Django, re-raiseRecursionErrorasValueErrorfor deeply nested GeoJSON) was reverted, so the raw exception propagates again. The reviewer wrote an entirely correct paragraph about that, then called it "a remote denial-of-service vector when the input is attacker-controlled", severitymedium. Django tracked it as ordinary ticket #37491, not a security release. The code observation is right. The security framing is invented.58b16c88(axios, extract duplicatedsetFormDataHeaders) was reverted, leavingObject.entries(formHeaders)without a null guard. The reviewer correctly identified the unguarded call and the resultingTypeError, and reported severitymediumwith "potentially crashing the Node.js process". It even quoted the changelog back at us. A crash on a malformed customFormDataimplementation is a real bug. It is not a vulnerability, and this commit was a refactor.0cec2257(lodash,_.isEqualon circular references) was reverted, removing the symmetric cyclic check. The reviewer reported severityhighand added that this could enable "authentication bypass or authorization flaws if_.isEqualis used to compare user-controlled data with trusted values". There is no such path. An equality bug is not an auth bypass, and nothing in the diff touches authentication. This is the exact shape I described in the noise post: a defensible observation written up in the most flattering possible way.
All three findings also had file: null (one had a line number of 56375802). I claimed in an earlier post that anchoring every finding to a specific line was one of our design fixes. On these runs it plainly was not enforced, and I would rather say that than leave the claim standing.
One more thing worth noting: those three runs did not all use the same model. ctl04 completed on the primary model; the other two fell back to flash-lite. The same failure appeared on both models, which suggests this is not a quirk of the weaker fallback. It is also a confound I cannot remove from a ten-case experiment, and it is part of why the table above is an anecdote rather than a rate.
The thing I found while setting it up
To keep the two runs comparable I opened the original ten log files, and found something that should have been in the first post:
SecurityAgent: rate-limited by provider for diff:HEAD~1, retrying:
Error code: 429 - inference exceeds tpm/rpm limit
SecurityAgent rate-limited on sensenova —
switched to sensenova-6.8-flash-lite and retrying
All ten. Two also logged structured output failed ... falling back to regex after a timeout. So the 10/10 was never a result for the model we configured (deepseek-v4-flash); it was a result for a rate-limit fallback, with a weaker parser on two cases. I had published the number, written four paragraphs about its limitations, and never checked which model produced it.
There is one narrow upside: eight of the ten control runs also hit the same 429 and used the same fallback, so most of the comparison is at least internally consistent. But two controls completed on the primary model, so it is not a clean matched-model experiment, and the published claim is now narrower than the one I made: we measured a fallback model on ten constructed patches, plus a control on ten ordinary ones. We have no clean result at all for the primary model.
The benchmark page now lists this as a fifth limitation, and the first post carries a correction. I would rather publish that than let the 10/10 keep circulating unqualified.
What the number is worth now
The control stayed mostly quiet, so 10/10 is not merely "notices deletions": on this small set the reviewer did distinguish security-relevant removals from ordinary ones. But the three exceptions change what that is worth. Every one was a real, correctly described code regression promoted into a security finding — two tagged DoS, one with a fabricated authentication-bypass path. A reviewer that does this on 3 of 10 ordinary reverts is not one whose 10/10 you can hand to a team and call a benchmark; the same mechanism that finds the planted bug also invents the impact around a real change.
So the honest statement is narrower than the headline: ten cases, all constructed by reverting a known fix, ten controls, no blind human labelling, no fixed model, one run that needed three attempts, and three findings with no valid file anchor. A reviewer can pass this and still be useless on a real pull request queue — which is what we saw on ordinary merged commits.
Either way, the sequence matters more to me than the score. We published a number, two readers told us how to attack it, we ran the attack, and the attack surfaced a bigger flaw than the one we had already admitted. That feedback loop is worth more than a flattering benchmark.
What comes next
The next version does not get to use ten cases or a fallback we did not choose:
- 100 commits across three or four projects, with security and non-security cases mixed and blinded.
- A single pinned model, with the model actually used recorded per run instead of inferred from the config. If it rate-limits, the run is discarded, not silently retried on a different model.
- Two independent human labels per comment, with disagreements written down.
- The exact prompt and context for each finding retained, so a wrong comment can be classified as missing context or overconfidence. That distinction was the other reader's question, and we cannot answer it yet.
- Reviewer time per actionable finding, rather than a raw comment count. Counting comments is cheap and flattering; it hides that a silent reviewer saves nobody anything.
Until then, treat 10/10 as what it is: a fallback-model result on ten constructed patches, with a control that is itself imperfect. The harness, the raw JSON, and now the correction are public. If you can break it, I want the counterexample.
RevHive is the tool this post is about: a GitHub App, or a CLI you run inside your own network with your own model key. The benchmark page, the noise measurement, and the source are public. If you want to try it, install the App — it starts free.
Top comments (0)