<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: RevHive</title>
    <description>The latest articles on DEV Community by RevHive (@revhive).</description>
    <link>https://dev.to/revhive</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4256055%2Fc15983e1-2ff6-4119-9880-66fddfc15fbe.png</url>
      <title>DEV Community: RevHive</title>
      <link>https://dev.to/revhive</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/revhive"/>
    <language>en</language>
    <item>
      <title>Same reviewer, three models: is the noise sampling or model choice?</title>
      <dc:creator>RevHive</dc:creator>
      <pubDate>Sat, 21 Nov 2026 09:02:42 +0000</pubDate>
      <link>https://dev.to/revhive/same-reviewer-three-models-is-the-noise-sampling-or-model-choice-5cgf</link>
      <guid>https://dev.to/revhive/same-reviewer-three-models-is-the-noise-sampling-or-model-choice-5cgf</guid>
      <description>&lt;p&gt;The &lt;a href="https://revhive.souljian.cn/blog/ordinary-prs-noise.html" rel="noopener noreferrer"&gt;multi-pass experiment&lt;/a&gt; ended with a confession: nineteen commits on one model cannot tell you whether the ±21-point cross-pass spread we measured is a property of that model, of the provider, or of the task. The next cell we could afford was the 10-CVE suite — the same ten revert-diffs from the original benchmark — run on three models with everything else held constant.&lt;/p&gt;

&lt;h2&gt;
  
  
  The design, including the flag we had to build
&lt;/h2&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Three models, same pipeline (two agents, same prompts, same reconciliation code), one pass per cell:


  - `agnes-3.0-flash` — the model behind every number we have published so far
  - `agnes-2.5-pro` — same provider, a reasoning model
  - `sensenova-ds` (DeepSeek V4.1 via a free gateway) — a different provider entirely


One design problem surfaced immediately. Our CLI silently rotates to a smaller model when the primary hits a rate limit — the right behavior for production uptime, and exactly wrong for a benchmark, where a substituted model's answer pollutes the cell it lands in. (It polluted our *first* benchmark that way, before we knew to look.) So we built `--fallback none`: the named model's answer, or a recorded failure. Never a substitution.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;h2&gt;
  
  
  Results: every model missed a different case
&lt;/h2&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Hit = the report names the core vulnerability the revert reintroduced (judged on finding title, description, and anchored file path; the three disagreement cases were verified manually). Raw JSON for all thirty runs is in the repository under `benchmark-runs/2026-11-21-3model/`.

| case | agnes-3.0-flash | agnes-2.5-pro | sensenova-ds |
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;|---|---|---|---|&lt;br&gt;
| c01 SSRF / local file read | hit | hit | hit |&lt;br&gt;
| c02 password hash disclosure | hit | hit | hit |&lt;br&gt;
| c03 empty vs missing headers | hit | hit | hit |&lt;br&gt;
| c04 unescaped log messages | hit | hit | hit |&lt;br&gt;
| c05 unbounded multipart memory | hit | hit | hit |&lt;br&gt;
| c06 forged formset actions | hit | hit | hit |&lt;br&gt;
| c07 translation parser memory | hit | hit | miss |&lt;br&gt;
| c08 unvalidated URL rendering | hit | hit | hit |&lt;br&gt;
| c09 axios SSRF | hit | miss | hit |&lt;br&gt;
| c10 template command injection | hit | hit | hit |&lt;br&gt;
| total | 10/10 | 9/10 | 9/10 |&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;The headline is not that one model won. It is that **the two misses do not overlap**. `agnes-2.5-pro` returned risk 0 and zero findings on c09 — silent on a reintroduced SSRF. `sensenova-ds` missed c07 entirely. No single model is strictly better; an ensemble voting 2-of-3 would have caught all ten. We are not claiming that as a feature — it is a hypothesis the next cell has to earn.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;h2&gt;
  
  
  The case that flipped
&lt;/h2&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;The original benchmark reported agnes-3.0-flash at 8/10, missing c02 (password hash disclosure). Today, the same model on the same revert returns a CRITICAL anchored at `django/contrib/admin/utils.py:435` describing exactly that vulnerability.

Attribution honesty: we cannot credit a single change. Since that first run the security prompt gained the evidence-cap rubric, findings gained real file anchoring, and severity reconciliation moved from prose to code. Anchoring is post-processing, so it did not make the model *see* the bug — but prompt changes and plain sampling noise are both live candidates, and one run cannot separate them. What we can say: the published 8/10 is no longer the current number, and the published number will be re-baselined multi-pass before anyone treats either as a rate.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;h2&gt;
  
  
  The fail-closed gate met real data
&lt;/h2&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;On `sensenova-ds`, the free tier's TPM limit kept killing the LogicAgent — in 6 of 10 runs. Under the old pipeline, those runs would have shipped a verdict computed from half the checks. Instead, every one of them carries a withheld risk score: findings listed, verdict refused, reason in the summary. Six withheld reports are the clearest demonstration we have that the gate does its job on real provider failures — a partial review cannot masquerade as a clean one.

Practical read: the free tier is workable for *evaluation* (the SecurityAgent completed all ten cases and found 9/10 vulnerabilities), but production-shaped reviews need a provider that finishes what it starts. That is why our BYOK table still puts paid DeepSeek in the recommended slot.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;h2&gt;
  
  
  Which noise is bigger?
&lt;/h2&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Risk-score spread, median (raw JSON behind every number):


  - Across three models, same case, same pass: **median 30** (n=3 cases where all three models scored; the withheld runs are excluded, not zeroed)
  - Across passes, same model, same case: **median 21** (n=19 commits, the multi-pass cell)
  - Across two models of the same provider, same case: **median 14** (n=9)


Reading it: crossing providers moves the verdict about as much as re-rolling the same model. Swapping models inside one provider moves it less than either. Neither number is a substitute for the other — **model choice does not buy you stability, and multi-pass does not buy you a model that sees c09**. If your CI gates on a single-run score from a single model, you are stacking both noise sources and calling the result a verdict.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;h2&gt;
  
  
  One correction in our own tooling
&lt;/h2&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Our first automated pass of the hit matrix scored one case wrong: the matching script compared keywords against finding titles and descriptions but not anchored file paths, so a correct critical on `django/utils/translation/trans_real.py` read as a miss. We caught it while verifying the disagreement cases, fixed the script, and re-ran the matrix before publishing anything. We mention it because it is the same failure mode we criticize in vendor benchmarks — an eval script that quietly mislabels results — and it happened to us on a Saturday afternoon with the publish button in reach.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;h2&gt;
  
  
  Honest limits
&lt;/h2&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;  - One pass per cell. The multi-pass post showed single-run numbers carry ±21 points of sampling noise; these hit rates are one sample, not a rate. The 10/10 will get a multi-pass re-baseline before we print it on the benchmark page.
  - Hit judgment is ours, on named findings, verified manually only for the disagreement cases.
  - CVE-diff shapes: these are reverts of real security fixes — well-formed, security-flavored diffs. Ordinary merged PRs behave differently, as the noise post showed.
  - `agnes-2.5-pro` is a reasoning model: slower per call, and its behavior under the pipeline differs in ways a single cell does not fully expose.


Open items from the comments, in order: the 100-commit double-labelled run, and a multi-pass re-baseline of this cell. The harness, all thirty raw runs, and the corrected analysis script are public. If you can break the matrix, we want the counterexample.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>codereview</category>
      <category>machinelearning</category>
      <category>programming</category>
    </item>
    <item>
      <title>Does the PR description leak the answer key? We ran the benchmark three ways.</title>
      <dc:creator>RevHive</dc:creator>
      <pubDate>Sat, 21 Nov 2026 04:09:26 +0000</pubDate>
      <link>https://dev.to/revhive/does-the-pr-description-leak-the-answer-key-we-ran-the-benchmark-three-ways-ho6</link>
      <guid>https://dev.to/revhive/does-the-pr-description-leak-the-answer-key-we-ran-the-benchmark-three-ways-ho6</guid>
      <description>&lt;p&gt;The first post in this series included a checklist for anyone evaluating a code reviewer, and one item keeps coming up in the comments:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Ask whether the eval strips the answer key (release notes, tests, commit messages). If it doesn't, you are measuring the model's ability to read.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It is the right question, and we owed a direct answer for our own benchmark. So we audited the prompt path, reran all ten CVE cases under three conditions, and found a claim on our benchmark page that needed correcting. Here is everything, including the part where we were wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the reviewer actually reads
&lt;/h2&gt;

&lt;p&gt;The CLI takes a diff reference, runs &lt;code&gt;git diff&lt;/code&gt;, and hands the raw diff text to the agents. Commit messages are never read: there is no code path from &lt;code&gt;git log&lt;/code&gt; to the prompt. We verified this two ways — by reading the prompt-construction path (&lt;code&gt;run_from_diff&lt;/code&gt; builds the text from &lt;code&gt;git diff&lt;/code&gt; output only), and empirically: identical diffs produce identical reviews whether or not a commit message exists.&lt;/p&gt;

&lt;p&gt;So for commit messages, the answer was clean. It did not stay clean.&lt;/p&gt;

&lt;h2&gt;
  
  
  The correction: we claimed we stripped tests, and we did not
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://revhive.souljian.cn/benchmark.html" rel="noopener noreferrer"&gt;benchmark page&lt;/a&gt; has said the harness "fetches the fix commit, reverts it, &lt;strong&gt;strips docs and tests&lt;/strong&gt;, runs the CLI". We went back and checked the script that produced the published numbers. It does not strip anything. The revert commit contains every change the fix made, including the test hunks — and reverted test code often carries descriptive names. In case c02, the deleted tests sit in files named &lt;code&gt;tests/admin_utils/…&lt;/code&gt; exercising exactly the password-hash path. That is answer-key material sitting inside the diff, in the published runs, with a claim on the page saying otherwise.&lt;/p&gt;

&lt;p&gt;We have corrected the page. And because the honest move is not just to admit the leak but to measure what it does, we reran all ten cases three ways.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three conditions
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Arm A — as published.&lt;/strong&gt; Full revert diff (test hunks included), no commit message, no PR description. This replicates the conditions behind every number we have published so far.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Arm B — the description a real reviewer would see.&lt;/strong&gt; Same full diff, plus the original fix commit's message injected as a PR description — complete with CVE numbers. This is the strongest possible answer key, and also the most realistic: a human reviewer on a real PR reads the description next to the diff.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Arm C — the strictest key-free condition.&lt;/strong&gt; The diff with all test and docs hunks removed, so the reviewer sees only the source-code change. Nothing but the vulnerable code itself.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Same ten reverted CVE fixes (&lt;a href="https://dev.to/revhive/our-ai-reviewer-caught-10-out-of-10-known-vulnerabilities-here-is-why-i-told-people-to-ignore-that-222k"&gt;the original cases&lt;/a&gt;), same model for every run (&lt;code&gt;agnes-3.0-flash&lt;/code&gt; — one model, recorded per run, no fallback), same agent set, one run per case, serialised. To make context injection possible we shipped a small &lt;code&gt;--pr-description&lt;/code&gt; flag, which is also just a real feature: reviewers on GitHub see the description.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;What was reverted&lt;/th&gt;
&lt;th&gt;A: full diff&lt;/th&gt;
&lt;th&gt;B: + PR description&lt;/th&gt;
&lt;th&gt;C: stripped diff&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;c01&lt;/td&gt;
&lt;td&gt;GDAL file/network access restriction&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c02&lt;/td&gt;
&lt;td&gt;Password hash disclosure&lt;/td&gt;
&lt;td&gt;✗ &lt;strong&gt;(0)&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✓ &lt;strong&gt;(3, critical)&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✓ &lt;strong&gt;(2, high)&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c03&lt;/td&gt;
&lt;td&gt;Empty vs missing Vary header cache keys&lt;/td&gt;
&lt;td&gt;✓ (1)&lt;/td&gt;
&lt;td&gt;✓ (1)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c04&lt;/td&gt;
&lt;td&gt;Unescaped log messages&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;td&gt;✓ (1)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c05&lt;/td&gt;
&lt;td&gt;Unbounded multipart memory&lt;/td&gt;
&lt;td&gt;✓ (4)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c06&lt;/td&gt;
&lt;td&gt;Forged formset actions&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;td&gt;✓ (1)&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c07&lt;/td&gt;
&lt;td&gt;Language-variant parser memory exhaustion&lt;/td&gt;
&lt;td&gt;~ (3)&lt;/td&gt;
&lt;td&gt;~ (1)&lt;/td&gt;
&lt;td&gt;~ (1)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c08&lt;/td&gt;
&lt;td&gt;Unvalidated URL rendering (XSS)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;td&gt;✓ (1)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c09&lt;/td&gt;
&lt;td&gt;axios SSRF via redirects&lt;/td&gt;
&lt;td&gt;✓ (5)&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;td&gt;✓ (3)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c10&lt;/td&gt;
&lt;td&gt;lodash template command injection&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;td&gt;✓ (2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;8 clear&lt;/strong&gt; / 24 findings&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;9 clear&lt;/strong&gt; / 21 findings&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;9 clear&lt;/strong&gt; / 21 findings&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;✓ = the core vulnerability of that case was reported. ~ = the reviewer found a real bug in the same function or surrounding logic, but not the DoS itself (c07: every arm found adjacent defects — an unvalidated language-code path, a &lt;code&gt;reset_cache&lt;/code&gt; mis-call — none named the unbounded cache growth). Counts are total findings including noise, not hits.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this says
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The leak did not create the score — removing it made the score better.&lt;/strong&gt; Under the strictest key-free condition the reviewer caught every case the leaky condition caught, plus the one it missed: c02, the password-hash disclosure, went from zero findings to two high-severity findings on the stripped diff. The plausible mechanism is mundane — the full diff carries hundreds of deleted test lines that dilute the actual change, and the stripped version is a smaller, more focused text.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;PR descriptions help, and they do not turn the model into a parrot.&lt;/strong&gt; Arm B saw the original fix messages with CVE numbers — the loudest answer key we could construct — and the total finding count went &lt;em&gt;down&lt;/em&gt; from 24 to 21 while c02 flipped to a critical hit. The model used the description as context, not as an answer to copy. That matters because real PR review always happens next to a description, and now the CLI supports it: &lt;code&gt;revhive review -d HEAD~1 --pr-description "..."&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;c02 is the case that keeps teaching.&lt;/strong&gt; It was the miss in the &lt;a href="https://dev.to/revhive/we-ran-our-ai-reviewer-on-20-ordinary-merged-commits-it-commented-4-times-c15"&gt;noise measurement&lt;/a&gt;'s sibling experiment, it was the miss in the first rerun, and it is the case most sensitive to context. A small diff (2K tokens, 6 files), a subtle information-exposure path — exactly the shape that a reviewer working from a bare diff is most likely to walk past. We do not have enough runs to say the stripped condition reliably beats the full condition; we have one observation that it did on the hardest case.&lt;/p&gt;

&lt;h2&gt;
  
  
  Corrections issued
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The benchmark page no longer says the harness strips docs and tests. The published runs reviewed the full revert diff, test hunks included.&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://dev.to/revhive/our-ai-reviewer-caught-10-out-of-10-known-vulnerabilities-here-is-why-i-told-people-to-ignore-that-222k"&gt;first post&lt;/a&gt;'s method description carries the same correction.&lt;/li&gt;
&lt;li&gt;All numbers published so far remain as published — with the note that they were produced under the leaky condition, which the three-way rerun suggests made them &lt;em&gt;conservative&lt;/em&gt;, not inflated. That is a lucky break we did not design, and we are not claiming it as a virtue.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Honest limits of this experiment
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Ten cases, one model, one run per condition. Any cell can be a fluctuation.&lt;/li&gt;
&lt;li&gt;Label matching ("did it report the core vulnerability?") is our judgment on named findings; the raw JSON for all thirty runs is in the repository under &lt;code&gt;benchmark-runs/&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Arm B's descriptions came from the fix commits, which are better written than most real PR descriptions. A vague real-world description would likely help less.&lt;/li&gt;
&lt;li&gt;c07 remains unresolved in every condition, which we take as the honest boundary of what this reviewer does with subtle resource-exhaustion paths.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The checklist item that started this is now answered for our tool, in public, with the correction attached. The next open item from the comments is the 100-commit double-labelled run, and that one is bigger than a Saturday. The harness, all thirty raw run files, and the diff of every correction are public. If you can break it, we want the counterexample.&lt;/p&gt;

&lt;p&gt;RevHive is the tool this post is about: a &lt;a href="https://github.com/apps/revhive-review/installations/new" rel="noopener noreferrer"&gt;GitHub App&lt;/a&gt;, or a CLI you run inside your own network with your own model key. Longer version with the full tables: &lt;a href="https://revhive.souljian.cn/blog/answer-key-leakage.html" rel="noopener noreferrer"&gt;the answer-key audit&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>codequality</category>
      <category>security</category>
      <category>opensource</category>
    </item>
    <item>
      <title>We ran the control a reader asked for. It also turned out we had benchmarked the wrong model.</title>
      <dc:creator>RevHive</dc:creator>
      <pubDate>Sat, 21 Nov 2026 02:43:28 +0000</pubDate>
      <link>https://dev.to/revhive/we-ran-the-control-a-reader-asked-for-it-also-turned-out-we-had-benchmarked-the-wrong-model-13bb</link>
      <guid>https://dev.to/revhive/we-ran-the-control-a-reader-asked-for-it-also-turned-out-we-had-benchmarked-the-wrong-model-13bb</guid>
      <description>&lt;p&gt;Two people left long, technical comments on the posts I published this week. One of them asked for an experiment I should have run before publishing anything at all.&lt;/p&gt;

&lt;p&gt;The original number was 10/10: take ten real CVE fix commits, revert each one, and ask our security agent to find what was reintroduced. It found all ten. I wrote a whole post about why that number is weak, and listed four reasons. A reader pointed out the fifth, with a design:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;For each of the ten cases, revert a commit of about the same size in the same files that fixed nothing security related, and run the reviewer with the same settings. If it reports a vulnerability on many of those, ten out of ten says the reviewer notices removed checks rather than the bug.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;He is right, and the reason cuts deeper than "we forgot a control". A reverted fix is a diff that deletes a guard. If the reviewer is pattern matching on &lt;em&gt;a guard was removed&lt;/em&gt;, then any reverted commit that removes a guard gets flagged, and the ground truth (this was a CVE) is doing none of the work. Without the control, 10/10 cannot distinguish "finds vulnerabilities" from "notices deletions".&lt;/p&gt;

&lt;h2&gt;
  
  
  The design
&lt;/h2&gt;

&lt;p&gt;Ten control commits, one per original case, chosen to be as close as possible to the fix it replaces:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;same repository, and where the file had enough history, the &lt;strong&gt;same source file&lt;/strong&gt;;&lt;/li&gt;
&lt;li&gt;an &lt;strong&gt;already merged commit&lt;/strong&gt; with a non-security subject (no CVE, no security wording, no revert);&lt;/li&gt;
&lt;li&gt;comparable diff size;&lt;/li&gt;
&lt;li&gt;same treatment as the original ten: check out the commit, &lt;code&gt;git revert --no-commit&lt;/code&gt;, restore &lt;code&gt;docs&lt;/code&gt;/&lt;code&gt;tests&lt;/code&gt;, commit, then run only the security agent on &lt;code&gt;HEAD~1&lt;/code&gt; with no PR description and no commit message.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;Fix (lines / date)&lt;/th&gt;
&lt;th&gt;Control commit&lt;/th&gt;
&lt;th&gt;Lines&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;What it actually changed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;c01&lt;/td&gt;
&lt;td&gt;264 / 2026-11-03&lt;/td&gt;
&lt;td&gt;&lt;code&gt;c709a748&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;42&lt;/td&gt;
&lt;td&gt;2023-12-20&lt;/td&gt;
&lt;td&gt;Dropped support for GDAL 2.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c02&lt;/td&gt;
&lt;td&gt;72 / 2026-11-03&lt;/td&gt;
&lt;td&gt;&lt;code&gt;77d0c9d5&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;37&lt;/td&gt;
&lt;td&gt;2026-06-18&lt;/td&gt;
&lt;td&gt;Admin displays database defaults as empty&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c03&lt;/td&gt;
&lt;td&gt;87 / 2026-11-03&lt;/td&gt;
&lt;td&gt;&lt;code&gt;a0f2e29e&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;2026-11-12&lt;/td&gt;
&lt;td&gt;Deprecated passing table names to &lt;code&gt;createcachetable&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c04&lt;/td&gt;
&lt;td&gt;84 / 2026-11-03&lt;/td&gt;
&lt;td&gt;&lt;code&gt;d3954c98&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;td&gt;2026-11-16&lt;/td&gt;
&lt;td&gt;Re-raise &lt;code&gt;RecursionError&lt;/code&gt; as &lt;code&gt;ValueError&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c05&lt;/td&gt;
&lt;td&gt;176 / 2026-11-03&lt;/td&gt;
&lt;td&gt;&lt;code&gt;7732f942&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;117&lt;/td&gt;
&lt;td&gt;2026-02-10&lt;/td&gt;
&lt;td&gt;Made the multipart parser class pluggable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c06&lt;/td&gt;
&lt;td&gt;110 / 2026-10-06&lt;/td&gt;
&lt;td&gt;&lt;code&gt;be6a149d&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;62&lt;/td&gt;
&lt;td&gt;2026-09-07&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ModelFormSet&lt;/code&gt; respects custom &lt;code&gt;Form.add_prefix()&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c07&lt;/td&gt;
&lt;td&gt;154 / 2026-10-06&lt;/td&gt;
&lt;td&gt;&lt;code&gt;9ea5f4fa&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;163&lt;/td&gt;
&lt;td&gt;2026-07-24&lt;/td&gt;
&lt;td&gt;Typos and grammar in docs and docstrings&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c08&lt;/td&gt;
&lt;td&gt;55 / 2026-08-04&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4b2b4bf0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;35&lt;/td&gt;
&lt;td&gt;2026-03-18&lt;/td&gt;
&lt;td&gt;Admin boolean icons for related fields&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c09&lt;/td&gt;
&lt;td&gt;97 / 2020-11-24&lt;/td&gt;
&lt;td&gt;&lt;code&gt;58b16c88&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;81&lt;/td&gt;
&lt;td&gt;2026-07-15&lt;/td&gt;
&lt;td&gt;Extract duplicated &lt;code&gt;setFormDataHeaders&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c10&lt;/td&gt;
&lt;td&gt;29 / 2021-02-20&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0cec2257&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;td&gt;2019-10-16&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;_.isEqual&lt;/code&gt; and circular references&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is not a clean matched experiment, and I would rather say so here than have someone find it later. Three controls are years away from their fix in repository time (c01 is 2023 code against a 2026 fix, c10 is 2019 against 2021, c09 is six years later). Two controls are mostly docs or docstrings, which are trivially quiet. c03 and c04 are dated &lt;em&gt;after&lt;/em&gt; their fix. c01's control is a sixth of the size of its fix. For files with almost no history there was no same-file, same-size, same-year commit that was not itself a security fix, so "nearest available" was the honest ceiling.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the control showed
&lt;/h2&gt;

&lt;p&gt;All ten controls eventually completed. The axios case timed out twice on the provider; after a cooling-off period a third attempt succeeded, so it is included. I am keeping that detail because "ten clean runs" would be a nicer sentence and a less true one.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Complete silence&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7 / 10&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Commented&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3 / 10&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Total comments&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;All three graded as noise for a security review&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So the cheap version of the criticism does not hold: the reviewer was not flagging every reverted commit. It stayed quiet on seven of ten ordinary reverts, against 10/10 on the CVE reverts. Something in the security patches is being detected.&lt;/p&gt;

&lt;p&gt;But the three comments it did produce are more interesting than a clean zero would have been, because all three are the same failure mode. None is a random hallucination. Each describes the reverted change accurately, then promotes an ordinary robustness bug to a security vulnerability.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;d3954c98&lt;/code&gt;&lt;/strong&gt; (Django, re-raise &lt;code&gt;RecursionError&lt;/code&gt; as &lt;code&gt;ValueError&lt;/code&gt; for deeply nested GeoJSON) was reverted, so the raw exception propagates again. The reviewer wrote an entirely correct paragraph about that, then called it &lt;em&gt;"a remote denial-of-service vector when the input is attacker-controlled"&lt;/em&gt;, severity &lt;code&gt;medium&lt;/code&gt;. Django tracked it as ordinary ticket #37491, not a security release. The code observation is right. The security framing is invented.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;58b16c88&lt;/code&gt;&lt;/strong&gt; (axios, extract duplicated &lt;code&gt;setFormDataHeaders&lt;/code&gt;) was reverted, leaving &lt;code&gt;Object.entries(formHeaders)&lt;/code&gt; without a null guard. The reviewer correctly identified the unguarded call and the resulting &lt;code&gt;TypeError&lt;/code&gt;, and reported severity &lt;code&gt;medium&lt;/code&gt; with &lt;em&gt;"potentially crashing the Node.js process"&lt;/em&gt;. It even quoted the changelog back at us. A crash on a malformed custom &lt;code&gt;FormData&lt;/code&gt; implementation is a real bug. It is not a vulnerability, and this commit was a refactor.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;0cec2257&lt;/code&gt;&lt;/strong&gt; (lodash, &lt;code&gt;_.isEqual&lt;/code&gt; on circular references) was reverted, removing the symmetric cyclic check. The reviewer reported severity &lt;code&gt;high&lt;/code&gt; and added that this could enable &lt;em&gt;"authentication bypass or authorization flaws if &lt;code&gt;_.isEqual&lt;/code&gt; is used to compare user-controlled data with trusted values"&lt;/em&gt;. There is no such path. An equality bug is not an auth bypass, and nothing in the diff touches authentication. This is the exact shape I described in the noise post: a defensible observation written up in the most flattering possible way.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;All three findings also had &lt;code&gt;file: null&lt;/code&gt; (one had a line number of &lt;code&gt;56375802&lt;/code&gt;). I claimed in an earlier post that anchoring every finding to a specific line was one of our design fixes. On these runs it plainly was not enforced, and I would rather say that than leave the claim standing.&lt;/p&gt;

&lt;p&gt;One more thing worth noting: those three runs did &lt;strong&gt;not&lt;/strong&gt; all use the same model. &lt;code&gt;ctl04&lt;/code&gt; completed on the primary model; the other two fell back to flash-lite. The same failure appeared on both models, which suggests this is not a quirk of the weaker fallback. It is also a confound I cannot remove from a ten-case experiment, and it is part of why the table above is an anecdote rather than a rate.&lt;/p&gt;

&lt;h2&gt;
  
  
  The thing I found while setting it up
&lt;/h2&gt;

&lt;p&gt;To keep the two runs comparable I opened the original ten log files, and found something that should have been in the first post:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SecurityAgent: rate-limited by provider for diff:HEAD~1, retrying:
  Error code: 429 - inference exceeds tpm/rpm limit
SecurityAgent rate-limited on sensenova —
  switched to sensenova-6.8-flash-lite and retrying
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All ten. Two also logged &lt;code&gt;structured output failed ... falling back to regex&lt;/code&gt; after a timeout. So the 10/10 was never a result for the model we configured (&lt;code&gt;deepseek-v4-flash&lt;/code&gt;); it was a result for a rate-limit fallback, with a weaker parser on two cases. I had published the number, written four paragraphs about its limitations, and never checked which model produced it.&lt;/p&gt;

&lt;p&gt;There is one narrow upside: eight of the ten control runs also hit the same 429 and used the same fallback, so most of the comparison is at least &lt;em&gt;internally&lt;/em&gt; consistent. But two controls completed on the primary model, so it is not a clean matched-model experiment, and the published claim is now narrower than the one I made: we measured a fallback model on ten constructed patches, plus a control on ten ordinary ones. We have no clean result at all for the primary model.&lt;/p&gt;

&lt;p&gt;The benchmark page now lists this as a fifth limitation, and the first post carries a correction. I would rather publish that than let the 10/10 keep circulating unqualified.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the number is worth now
&lt;/h2&gt;

&lt;p&gt;The control stayed mostly quiet, so 10/10 is not merely "notices deletions": on this small set the reviewer did distinguish security-relevant removals from ordinary ones. But the three exceptions change what that is worth. Every one was a real, correctly described code regression promoted into a security finding — two tagged DoS, one with a fabricated authentication-bypass path. A reviewer that does this on 3 of 10 ordinary reverts is not one whose 10/10 you can hand to a team and call a benchmark; the same mechanism that finds the planted bug also invents the impact around a real change.&lt;/p&gt;

&lt;p&gt;So the honest statement is narrower than the headline: ten cases, all constructed by reverting a known fix, ten controls, no blind human labelling, no fixed model, one run that needed three attempts, and three findings with no valid file anchor. A reviewer can pass this and still be useless on a real pull request queue — which is what we saw on ordinary merged commits.&lt;/p&gt;

&lt;p&gt;Either way, the sequence matters more to me than the score. We published a number, two readers told us how to attack it, we ran the attack, and the attack surfaced a bigger flaw than the one we had already admitted. That feedback loop is worth more than a flattering benchmark.&lt;/p&gt;

&lt;h2&gt;
  
  
  What comes next
&lt;/h2&gt;

&lt;p&gt;The next version does not get to use ten cases or a fallback we did not choose:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;100 commits across three or four projects, with security and non-security cases mixed and blinded.&lt;/li&gt;
&lt;li&gt;A single pinned model, with the &lt;strong&gt;model actually used&lt;/strong&gt; recorded per run instead of inferred from the config. If it rate-limits, the run is discarded, not silently retried on a different model.&lt;/li&gt;
&lt;li&gt;Two independent human labels per comment, with disagreements written down.&lt;/li&gt;
&lt;li&gt;The exact prompt and context for each finding retained, so a wrong comment can be classified as missing context or overconfidence. That distinction was the other reader's question, and we cannot answer it yet.&lt;/li&gt;
&lt;li&gt;Reviewer time per actionable finding, rather than a raw comment count. Counting comments is cheap and flattering; it hides that a silent reviewer saves nobody anything.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Until then, treat 10/10 as what it is: a fallback-model result on ten constructed patches, with a control that is itself imperfect. The harness, the raw JSON, and now the correction are public. If you can break it, I want the counterexample.&lt;/p&gt;




&lt;p&gt;RevHive is the tool this post is about: a GitHub App, or a CLI you run inside your own network with your own model key. The &lt;a href="https://revhive.souljian.cn/benchmark.html" rel="noopener noreferrer"&gt;benchmark page&lt;/a&gt;, the &lt;a href="https://revhive.souljian.cn/blog/ordinary-prs-noise.html" rel="noopener noreferrer"&gt;noise measurement&lt;/a&gt;, and the &lt;a href="https://github.com/jovian-zhibai/RevHive" rel="noopener noreferrer"&gt;source&lt;/a&gt; are public. If you want to try it, &lt;a href="https://github.com/apps/revhive-review/installations/new" rel="noopener noreferrer"&gt;install the App&lt;/a&gt; — it starts free.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>codequality</category>
      <category>testing</category>
      <category>opensource</category>
    </item>
    <item>
      <title>We ran our AI reviewer on 20 ordinary merged commits. It commented 4 times.</title>
      <dc:creator>RevHive</dc:creator>
      <pubDate>Fri, 20 Nov 2026 17:04:24 +0000</pubDate>
      <link>https://dev.to/revhive/we-ran-our-ai-reviewer-on-20-ordinary-merged-commits-it-commented-4-times-c15</link>
      <guid>https://dev.to/revhive/we-ran-our-ai-reviewer-on-20-ordinary-merged-commits-it-commented-4-times-c15</guid>
      <description>&lt;p&gt;Last week I published a benchmark where our reviewer caught 10 out of 10 re-introduced vulnerabilities, and then spent four paragraphs explaining why that number is close to worthless. The main reason was that it measured recall and never precision. A tool that finds every planted bug and also comments on clean code is not half-good, it is unusable.&lt;/p&gt;

&lt;p&gt;So here is the missing half. Twenty ordinary commits, nothing planted, and every comment read by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  The samples
&lt;/h2&gt;

&lt;p&gt;Twenty commits from &lt;code&gt;psf/requests&lt;/code&gt;, in reverse chronological order, filtered to those that touch &lt;code&gt;src/requests/**.py&lt;/code&gt; and are not pure type-annotation or CI churn. All twenty were merged by maintainers. None of them re-introduces a vulnerability.&lt;/p&gt;

&lt;p&gt;For each one: check out the commit, run the security agent against &lt;code&gt;HEAD~1&lt;/code&gt; with no PR description and no hints, and record what it says. The security agent alone, a small model, one pass each. Twenty out of twenty completed, no failures.&lt;/p&gt;

&lt;h2&gt;
  
  
  The result
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Count&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Commits reviewed&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Commits that received a comment&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Total comments&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Comments I would call noise&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Seventeen of the twenty commits got complete silence. That is the part that matters most: on most real changes the reviewer has nothing to say, which is the correct answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it said when it did speak
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;a044b02&lt;/code&gt; — "Move DigestAuth hash algorithms to use &lt;code&gt;usedforsecurity=False&lt;/code&gt;" (5 lines changed), 2 comments, both noise.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The commit adds &lt;code&gt;usedforsecurity=False&lt;/code&gt; to the &lt;code&gt;hashlib&lt;/code&gt; calls in HTTP Digest auth. That flag exists so the code keeps working in FIPS-restricted environments, where MD5 and SHA1 are otherwise blocked. It is a deliberate, well-understood annotation: &lt;em&gt;this hash is part of a wire protocol, not a security primitive&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;The reviewer reported a &lt;strong&gt;high&lt;/strong&gt; severity finding titled "FIPS security bypass via usedforsecurity=False", and a &lt;strong&gt;medium&lt;/strong&gt; one about weak hash for cnonce generation. Both restart from "MD5 is weak", which is true and has been true since 2004, and neither engages with what the diff does. The maintainers added the flag &lt;em&gt;because&lt;/em&gt; the hash is not a security control. The reviewer read the flag as a security control being disabled.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;fd62809&lt;/code&gt; — "Preserve leading slashes in request path_url" (2 lines changed), 1 comment, overstated.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The reviewer reported &lt;strong&gt;high&lt;/strong&gt;: removing &lt;code&gt;//&lt;/code&gt;-to-&lt;code&gt;/&lt;/code&gt; normalization may enable request smuggling, because some proxies resolve &lt;code&gt;//evil.com/path&lt;/code&gt; as a network-path reference. There is a real debate in that area, and the finding is not stupid. But the change was made on purpose, after discussion, and "may enable request smuggling" is a much stronger claim than the evidence in a two-line diff supports.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;a4f9a59&lt;/code&gt; — "Port bpo-39057 to Requests" (proxy bypass fix), 1 comment, real but marginal.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The finding: if &lt;code&gt;no_proxy&lt;/code&gt; contains an entry of only dots, &lt;code&gt;host.lstrip(".")&lt;/code&gt; produces an empty string and the suffix match degenerates. That is a genuine edge case. It is also the kind of thing that will never happen in a real deployment, which is presumably why it was rated low.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure mode is one thing, repeated
&lt;/h2&gt;

&lt;p&gt;All three false positives have the same shape: &lt;strong&gt;the diff contains something that looks like a security control being removed or disabled, and the reviewer reacts to the shape without recovering the intent.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;usedforsecurity=False&lt;/code&gt; reads as "we are telling the crypto library to skip security". &lt;code&gt;//&lt;/code&gt; normalization being removed reads as "we stopped sanitising the path". Both are locally true statements about the text and both are wrong about the change, because the reason lives outside the diff: in the FIPS requirement, in the linked issue, in the discussion that decided the trade-off.&lt;/p&gt;

&lt;p&gt;Which is the mirror image of the problem I wrote about last time. A reverted security fix is easy to catch precisely because the &lt;em&gt;reason&lt;/em&gt; is obvious from the diff (a guard clause disappeared and nothing replaced it). Real commits that touch security surfaces are hard because the reason is usually absent from the diff and the shape looks identical to a regression.&lt;/p&gt;

&lt;p&gt;An LLM reviewer with no PR description is being asked to tell those two apart from the patch alone. Most of the time it can't, so the honest design goal is not "understand intent", it is "say less when intent is missing".&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does and does not support
&lt;/h2&gt;

&lt;p&gt;What it supports: on ordinary changes this reviewer is quiet. Fifteen percent of commits got a comment, and the absolute cost of that is low. Four comments across twenty commits means roughly one extra thing to read per five PRs, not a wall of noise.&lt;/p&gt;

&lt;p&gt;What it does not support: a precision &lt;em&gt;rate&lt;/em&gt;. Four comments is not a sample. I also judged "noise" myself, and I know what the diffs were supposed to do, which a reviewer in CI would not.&lt;/p&gt;

&lt;p&gt;Three caveats worth stating plainly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One project, one language.&lt;/strong&gt; &lt;code&gt;requests&lt;/code&gt; is Python, small, and unusually well reviewed. A codebase with more security-sensitive surface will produce more comments, not fewer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No PR description, by design.&lt;/strong&gt; That is the harshest setting. In CI the reviewer usually gets a title and a description, which should reduce these false positives. It can also increase them, if the description hands the model a thesis to confirm.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Security agent only.&lt;/strong&gt; The logic and performance agents were not running. A full review is noisier, and I have no honest number for it yet.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What changed in the product
&lt;/h2&gt;

&lt;p&gt;The two findings above are the reason the reviewer now has a confidence floor and only reports findings anchored to a specific line with a concrete failure path. "MD5 is weak" has no failure path, so it does not get to speak. "This path is used for auth, therefore disabling FIPS checks on it is a bypass" has no failure path either, because the premise is wrong.&lt;/p&gt;

&lt;p&gt;Recall went down when we added that. On the twenty commits it would have cost us nothing, since none of the four comments were actionable. On the ten vulnerability cases it cost us nothing either. That is the trade I want: a reviewer that is wrong less often, at the price of being quiet more often.&lt;/p&gt;

&lt;p&gt;The next version of this needs a hundred commits across three or four projects, with PR descriptions included, and two people labelling the comments independently. Four comments is an anecdote with a table, and I would rather say so than round it into a percentage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; a reader asked for the control this post cannot provide: the same experiment with the CVE reverts replaced by ordinary ones. It is now published: &lt;a href="https://dev.to/revhive/we-ran-the-control-a-reader-asked-for-it-also-turned-out-we-had-benchmarked-the-wrong-model-13bb"&gt;ten ordinary reverts beside the ten CVE reverts&lt;/a&gt;. Seven stayed silent, three produced confident security findings about real but non-security bugs — the same shape as the four comments above, now with a control to compare against.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>codequality</category>
      <category>testing</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Our AI reviewer caught 10 out of 10 known vulnerabilities. Here is why I told people to ignore that number.</title>
      <dc:creator>RevHive</dc:creator>
      <pubDate>Fri, 20 Nov 2026 16:20:54 +0000</pubDate>
      <link>https://dev.to/revhive/our-ai-reviewer-caught-10-out-of-10-known-vulnerabilities-here-is-why-i-told-people-to-ignore-that-222k</link>
      <guid>https://dev.to/revhive/our-ai-reviewer-caught-10-out-of-10-known-vulnerabilities-here-is-why-i-told-people-to-ignore-that-222k</guid>
      <description>&lt;p&gt;Every AI code review tool has the same demo. Someone opens a PR with a planted bug, the bot comments on line 42, everyone nods. It is theatre. The only question that matters is whether the thing works on &lt;em&gt;your&lt;/em&gt; code, on a Tuesday, when nothing was planted.&lt;/p&gt;

&lt;p&gt;So when we built our own reviewer, we did the obvious thing: we took ten commits that fixed real CVEs in widely used projects, reverted each one so the working tree contained the vulnerable code again, and asked the tool to review the diff with no hints. Then we published the result — including a section explaining why the result is close to meaningless.&lt;/p&gt;

&lt;p&gt;We got 10/10. And that is the least interesting sentence in this post.&lt;/p&gt;

&lt;h2&gt;
  
  
  The method, because a number without a method is marketing
&lt;/h2&gt;

&lt;p&gt;For each case:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Take a commit that fixed a real vulnerability in a widely used project (Django, axios, lodash).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;git revert --no-commit&lt;/code&gt; it, so the diff looks like somebody just introduced the problem.&lt;/li&gt;
&lt;li&gt;
&lt;del&gt;Strip documentation and test changes from the patch.&lt;/del&gt; &lt;strong&gt;Correction (2026-11-21):&lt;/strong&gt; this step described what we intended the harness to do, but the script that produced the published numbers does not strip anything — test hunks stayed in the diff. We reran all ten cases three ways (full diff / +PR description / stripped diff): the stripped condition scored the same or better, including catching the case the published runs missed. Details: &lt;a href="https://revhive.souljian.cn/blog/answer-key-leakage.html" rel="noopener noreferrer"&gt;the answer-key audit&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Run the reviewer on &lt;code&gt;HEAD~1&lt;/code&gt; with the security agent only. No repository context, no PR description, no hints.&lt;/li&gt;
&lt;li&gt;A case counts as caught when a finding names the vulnerable file and describes the right class of bug.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Ten cases, one pass each, security agent alone, a small and cheap model, 2026-11-20.&lt;/p&gt;

&lt;p&gt;Two of the ten had to be thrown out and re-run. In the first attempt the patch still contained the release notes, and the reviewer quoted the CVE title back at us, verbatim. That is not review, that is OCR with confidence. It was the first hint that our benchmark was measuring something other than what we thought.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a perfect score is a warning sign
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A reverted fix is suspicious by construction.&lt;/strong&gt; Reverting a security fix removes a length check, a validation call, a sanitizer invocation. The diff says "I deleted the guard clause" in plain sight. Any reviewer that looks at the hunk will notice, because the shape of a security fix is itself a tell. Real bugs are not introduced by deleting guards. They are introduced by accident, at 11pm, inside a 400-line feature that was mostly about something else — and that is a fundamentally harder detection problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It measures recall and never precision.&lt;/strong&gt; Nothing in these ten runs says how often the reviewer comments on clean code. Most PRs are fine. A tool that finds every planted bug and also comments on a third of all normal PRs is not half-good, it is unusable, because every false positive costs a human the minutes it takes to read it, decide it is wrong, and move on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ten cases is an anecdote with a table.&lt;/strong&gt; Add two cases and the headline moves twenty points. There is no confidence interval in a sample that size, and no honest way to present one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Severity labels are unreliable.&lt;/strong&gt; One case — password hash disclosure to view-only admin users — was rated &lt;em&gt;low&lt;/em&gt; by the model. Django shipped it as a security release. The finding text was correct; the colour was not. If your workflow is "only look at critical and high", a mislabelled severity is functionally a miss even though the tool technically saw the bug.&lt;/p&gt;

&lt;p&gt;All four of these are written on the public benchmark page, above the fold, next to the 10/10. Specifically: &lt;em&gt;"A perfect score here is a sign that the benchmark is weak, not that the tool is perfect."&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What happened on real pull requests
&lt;/h2&gt;

&lt;p&gt;Then we ran the same reviewer over ordinary PRs, the kind where nothing is wrong. It produced a confident, plausible-sounding comment on roughly a third of them.&lt;/p&gt;

&lt;p&gt;Some were real. Most were pattern matching on style, or a fabricated failure mode that the surrounding code already handled three lines down. Each one cost a human a few minutes of "wait, is this actually broken?" — which is exactly the review time that is supposed to be the thing we are saving.&lt;/p&gt;

&lt;p&gt;That reframes the design problem. The question is not &lt;em&gt;how do we find more bugs&lt;/em&gt;. It is &lt;em&gt;when should the reviewer stay quiet&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Three things changed as a result:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Anchor every finding to a line and a failure path.&lt;/strong&gt; If the model cannot name the input that triggers the problem and the consequence that follows, it does not get to speak. This cut findings by more than half and made the remaining ones readable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat silence as a correct answer.&lt;/strong&gt; "No comment" is the right output for most diffs, and it needs to be an explicit, cheap, first-class outcome rather than an accident of the prompt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Demote severity to a hint.&lt;/strong&gt; The text of the finding is the signal. The label is a sorting convenience that is wrong often enough that no workflow should gate on it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Recall went down. Review time went down more. That trade is the entire product.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why publish the weakness next to the strength
&lt;/h2&gt;

&lt;p&gt;Because the failure mode of this category is not bad models. It is a demo culture where every tool shows a screenshot of a planted bug being found, and nobody shows the false positive rate. If we publish only the 10/10 we become another screenshot.&lt;/p&gt;

&lt;p&gt;There is also a selfish reason. A benchmark you are willing to describe, with the method written down and the harness shipped as a shell script, is one other people can attack. Ours is a shell script that fetches a fix commit, reverts it, strips docs and tests, runs the CLI, and diffs against the files the fix touched. It is not pretty and it is not peer reviewed. That is precisely why it is written down: so the next version can be better, and so someone can tell us where it is wrong.&lt;/p&gt;

&lt;p&gt;The next version of the page needs twenty ordinary pull requests and a false positive count. Until it has them, treat 10/10 as a floor on capability, not a rate of quality.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you are evaluating a reviewer
&lt;/h2&gt;

&lt;p&gt;A short list, whether you use ours or anyone else's:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ask for the &lt;strong&gt;false positive rate on clean PRs&lt;/strong&gt;, not the number of bugs found. Almost nobody has this number, and the ones who do will tell you an unflattering one.&lt;/li&gt;
&lt;li&gt;Ask whether the eval &lt;strong&gt;strips the answer key&lt;/strong&gt; (release notes, tests, commit messages). If it doesn't, you are measuring the model's ability to read.&lt;/li&gt;
&lt;li&gt;Ignore severity labels in the demo and read the finding text. Ask yourself whether you would have shipped the code after reading it.&lt;/li&gt;
&lt;li&gt;Ask how many cases. If the answer is under thirty, it is a story, not a benchmark.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We published ours at 10/10 with four paragraphs explaining why that does not count for much. If you want to break it, the harness and the raw runs are in the repo, and I would genuinely like the counterexample.&lt;/p&gt;




&lt;p&gt;RevHive is the tool this post is about: a GitHub App, or a CLI you run inside your own network with your own model key. The benchmark page and the harness are public, including the four limitations:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ten reverted fixes, method written down: &lt;a href="https://revhive.souljian.cn/benchmark.html" rel="noopener noreferrer"&gt;https://revhive.souljian.cn/benchmark.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Source and the review harness: &lt;a href="https://github.com/jovian-zhibai/RevHive" rel="noopener noreferrer"&gt;https://github.com/jovian-zhibai/RevHive&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you have a better way to measure a reviewer's noise ratio, I want to read it — that is the next thing we are building.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Update 2:&lt;/strong&gt; I went back through the raw run logs to set up the matched control a reader asked for, and found something I should have caught before publishing. All ten cases hit &lt;code&gt;429 inference exceeds tpm/rpm limit&lt;/code&gt; on the primary model and were retried on the fallback, &lt;code&gt;sensenova-6.8-flash-lite&lt;/code&gt;. Two of them also timed out on structured output and were parsed by regex instead. So the 10/10 belongs to a fallback model with a weaker parser, not to the model we configured. It does not change the four arguments above — if anything it makes the number less meaningful — but it was wrong to publish it without saying so. The benchmark page now lists this as a fifth limitation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; that follow-up is published. We ran the same reviewer on &lt;a href="https://dev.to/revhive/we-ran-our-ai-reviewer-on-20-ordinary-merged-commits-it-commented-4-times-c15"&gt;20 ordinary merged commits from psf/requests&lt;/a&gt;. Seventeen got complete silence, three got four comments, and we read every one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Update 3:&lt;/strong&gt; the matched control is now published: &lt;a href="https://dev.to/revhive/we-ran-the-control-a-reader-asked-for-it-also-turned-out-we-had-benchmarked-the-wrong-model-13bb"&gt;ten ordinary reverts beside the ten CVE reverts&lt;/a&gt;. Seven stayed silent, three produced confident security findings about real but non-security bugs, and every finding had no valid file anchor. Reading the logs to build it also produced the correction above — all ten original runs used the fallback model, so both sides of the comparison share that flaw.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Update 2 (2026-11-21):&lt;/strong&gt; we audited this eval for answer-key leakage after a reader's checklist asked the right question. Findings: commit messages never enter the prompt (verified in code and in paired runs); the "strips docs and tests" claim above was wrong for the published runs and is now corrected; a three-way rerun (full diff / + PR description / stripped diff) shows the key-free condition matched or beat the leaky one — c02, the case the published runs missed, was caught under both the stripped-diff and PR-description conditions. Full tables and all thirty raw run files: &lt;a href="https://revhive.souljian.cn/blog/answer-key-leakage.html" rel="noopener noreferrer"&gt;Does the PR description leak the answer key?&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>codequality</category>
      <category>security</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
