DEV Community

jian soul
jian soul

Posted on Originally published at revhive.souljian.cn

Does the PR description leak the answer key? We ran the benchmark three ways.

The first post in this series included a checklist for anyone evaluating a code reviewer, and one item keeps coming up in the comments:

Ask whether the eval strips the answer key (release notes, tests, commit messages). If it doesn't, you are measuring the model's ability to read.

It is the right question, and we owed a direct answer for our own benchmark. So we audited the prompt path, reran all ten CVE cases under three conditions, and found a claim on our benchmark page that needed correcting. Here is everything, including the part where we were wrong.

What the reviewer actually reads

The CLI takes a diff reference, runs git diff, and hands the raw diff text to the agents. Commit messages are never read: there is no code path from git log to the prompt. We verified this two ways — by reading the prompt-construction path (run_from_diff builds the text from git diff output only), and empirically: identical diffs produce identical reviews whether or not a commit message exists.

So for commit messages, the answer was clean. It did not stay clean.

The correction: we claimed we stripped tests, and we did not

The benchmark page has said the harness "fetches the fix commit, reverts it, strips docs and tests, runs the CLI". We went back and checked the script that produced the published numbers. It does not strip anything. The revert commit contains every change the fix made, including the test hunks — and reverted test code often carries descriptive names. In case c02, the deleted tests sit in files named tests/admin_utils/… exercising exactly the password-hash path. That is answer-key material sitting inside the diff, in the published runs, with a claim on the page saying otherwise.

We have corrected the page. And because the honest move is not just to admit the leak but to measure what it does, we reran all ten cases three ways.

The three conditions

  1. Arm A — as published. Full revert diff (test hunks included), no commit message, no PR description. This replicates the conditions behind every number we have published so far.
  2. Arm B — the description a real reviewer would see. Same full diff, plus the original fix commit's message injected as a PR description — complete with CVE numbers. This is the strongest possible answer key, and also the most realistic: a human reviewer on a real PR reads the description next to the diff.
  3. Arm C — the strictest key-free condition. The diff with all test and docs hunks removed, so the reviewer sees only the source-code change. Nothing but the vulnerable code itself.

Same ten reverted CVE fixes (the original cases), same model for every run (agnes-3.0-flash — one model, recorded per run, no fallback), same agent set, one run per case, serialised. To make context injection possible we shipped a small --pr-description flag, which is also just a real feature: reviewers on GitHub see the description.

Results

Case What was reverted A: full diff B: + PR description C: stripped diff
c01 GDAL file/network access restriction ✓ (2) ✓ (3) ✓ (3)
c02 Password hash disclosure ✗ (0) ✓ (3, critical) ✓ (2, high)
c03 Empty vs missing Vary header cache keys ✓ (1) ✓ (1) ✓ (3)
c04 Unescaped log messages ✓ (2) ✓ (3) ✓ (1)
c05 Unbounded multipart memory ✓ (4) ✓ (3) ✓ (3)
c06 Forged formset actions ✓ (2) ✓ (1) ✓ (2)
c07 Language-variant parser memory exhaustion ~ (3) ~ (1) ~ (1)
c08 Unvalidated URL rendering (XSS) ✓ (3) ✓ (2) ✓ (1)
c09 axios SSRF via redirects ✓ (5) ✓ (2) ✓ (3)
c10 lodash template command injection ✓ (2) ✓ (2) ✓ (2)
Total 8 clear / 24 findings 9 clear / 21 findings 9 clear / 21 findings

✓ = the core vulnerability of that case was reported. ~ = the reviewer found a real bug in the same function or surrounding logic, but not the DoS itself (c07: every arm found adjacent defects — an unvalidated language-code path, a reset_cache mis-call — none named the unbounded cache growth). Counts are total findings including noise, not hits.

What this says

The leak did not create the score — removing it made the score better. Under the strictest key-free condition the reviewer caught every case the leaky condition caught, plus the one it missed: c02, the password-hash disclosure, went from zero findings to two high-severity findings on the stripped diff. The plausible mechanism is mundane — the full diff carries hundreds of deleted test lines that dilute the actual change, and the stripped version is a smaller, more focused text.

PR descriptions help, and they do not turn the model into a parrot. Arm B saw the original fix messages with CVE numbers — the loudest answer key we could construct — and the total finding count went down from 24 to 21 while c02 flipped to a critical hit. The model used the description as context, not as an answer to copy. That matters because real PR review always happens next to a description, and now the CLI supports it: revhive review -d HEAD~1 --pr-description "...".

c02 is the case that keeps teaching. It was the miss in the noise measurement's sibling experiment, it was the miss in the first rerun, and it is the case most sensitive to context. A small diff (2K tokens, 6 files), a subtle information-exposure path — exactly the shape that a reviewer working from a bare diff is most likely to walk past. We do not have enough runs to say the stripped condition reliably beats the full condition; we have one observation that it did on the hardest case.

Corrections issued

  • The benchmark page no longer says the harness strips docs and tests. The published runs reviewed the full revert diff, test hunks included.
  • The first post's method description carries the same correction.
  • All numbers published so far remain as published — with the note that they were produced under the leaky condition, which the three-way rerun suggests made them conservative, not inflated. That is a lucky break we did not design, and we are not claiming it as a virtue.

Honest limits of this experiment

  • Ten cases, one model, one run per condition. Any cell can be a fluctuation.
  • Label matching ("did it report the core vulnerability?") is our judgment on named findings; the raw JSON for all thirty runs is in the repository under benchmark-runs/.
  • Arm B's descriptions came from the fix commits, which are better written than most real PR descriptions. A vague real-world description would likely help less.
  • c07 remains unresolved in every condition, which we take as the honest boundary of what this reviewer does with subtle resource-exhaustion paths.

The checklist item that started this is now answered for our tool, in public, with the correction attached. The next open item from the comments is the 100-commit double-labelled run, and that one is bigger than a Saturday. The harness, all thirty raw run files, and the diff of every correction are public. If you can break it, we want the counterexample.

RevHive is the tool this post is about: a GitHub App, or a CLI you run inside your own network with your own model key. Longer version with the full tables: the answer-key audit.

Top comments (0)