Why developers mute AI code reviewers (and what I changed to stop it)

Every team I talked to had the same story about AI code review bots: they installed one, it posted 25 comments on the first PR, half of them were wrong, and within two weeks everyone was scrolling past it.

The bot wasn't useless. It was noisy, and noise teaches people to ignore you. A reviewer that's right 50% of the time is worse than no reviewer, because it trains your team to skip the 50% that matters.

I've been building Audit Bench Ai (https://auditbenchai.com), an AI reviewer for pull requests, and most of the work hasn't been "make the AI smarter". It's been "make the AI shut up unless it can prove something". Here's what actually moved the needle.

1. Give the model the code that calls your change

Most review bots see the diff and nothing else. But the bugs that hurt are rarely inside the diff. They're in what the diff breaks:

// The PR changes this...
export function refundOrder(orderId: string, amountCents: number) { ... }

// ...but the bug is in a file the PR never touched,
// still passing dollars instead of cents.
await refundOrder(order.id, order.total);

So for every changed function, we find where it's called elsewhere in the repository and include short excerpts of those callers, plus related test files. The model can now say "this breaks processReturn() in returns.ts" instead of "consider validating the input".

That one change made findings far more specific, and specific findings get fixed.

2. Make every finding show its evidence

We require each finding to include:

• the exact line,
• when it breaks: the concrete input or sequence that triggers it ("a retried refund is sent twice"),
• and a fix.

If the model can't describe when it breaks, it usually isn't a real bug. Asking for the failure scenario filters out a surprising amount of "this could potentially maybe be an issue" padding.

3. Keep severity and confidence separate, then calibrate

Models love to call things "critical". So we ask for two separate numbers: severity (how bad if real) and confidence (how sure it's real). After the model answers, deterministic rules decide what gets posted:

for (const f of findings) {
  if (f.confidence < 0.4) continue; // a guess isn't a defect
  if ((f.severity === 'critical' || f.severity === 'high') && f.confidence < 0.6) {
    f.severity = 'medium'; // an unproven claim can't block a merge
  }
  if (f.category === 'Maintainability' || f.category === 'Architecture') {
    f.severity = lower(f.severity, 'medium'); // style is never a merge blocker
  }
  // ...then merge duplicates on overlapping lines, keeping the more confident one
}

Because these rules are plain code, the same model output always publishes the same way, and teams can tune the thresholds if they want a stricter or quieter reviewer.

4. Never say the same thing twice

The fastest way to get muted: post the same comment on every push. When you push a fix for one issue, the bot shouldn't repeat the other four you already saw.

We match new findings against comments already on the PR by file and kind of problem, not line number (code moves, line numbers lie):

• still open from an earlier review → not posted again
• dismissed by the team → not posted again
• marked fixed but detected again → posted, flagged as reappeared

The summary simply says "3 not repeated: still open from an earlier review".

5. Let developers argue back

This was the change I didn't expect to matter so much. When a developer replies "this can't be null, the caller already checks it", the reviewer goes back to the code at the latest commit and answers honestly:

• ✅ Resolved: the fix is in
• 👍 Dismissed: you're right, not an issue here
• ↘️ Downgraded: real, but less serious than first stated
• 🔁 Still applies: with the evidence from the code

It judges the reply against the code, not against how confident the reply sounds. And if a team 👎s the same kind of finding a couple of times in a repo, it turns that into a rule and stops raising it there.

Developers trust a reviewer that can admit it was wrong.

6. Do the cheap work first

AI calls cost money, and cost decides whether you can run a reviewer on every PR. So the free work runs first: linters, type checks, Semgrep, secret scanning and complexity analysis. Then only the changed code (plus its callers and tests) goes to the model, one call per file, with results cached so unchanged code is never reviewed twice.

Did it work?

On an internal benchmark of 25 seeded cases (16 with a planted bug, 9 clean), the current version caught 15 of the 16 bugs and raised zero false alarms on the 9 clean cases. In the push-back tests, it withdrew, downgraded or held its ground correctly in all 3.

It's a small benchmark, and real PRs are messier, which is why the reviewer also tracks precision per team: of the findings a team reacted to, how many did they accept? That's the number I watch now, not "findings per PR".

So far 40 teams use it and it has reviewed 220 pull requests. Still early, and I'm learning something from every false positive.

What I'd tell anyone building (or choosing) an AI reviewer

1. Measure precision, not volume. Fewer, correct comments beat many plausible ones.
2. Context beats model size. Show the callers and the tests.
3. Require a failure scenario for every finding.
4. Never repeat yourself across pushes.
5. Let people push back, and actually listen.

If you want to see what this looks like on a real PR, Audit Bench Ai is a GitHub App (also GitLab, Bitbucket, VS Code and a CLI), and you can run a free scan on any public repo without signing up.

I'd genuinely love to hear: what made you mute an AI reviewer, or keep one?

---

I wrote this post with help from an AI assistant; the product, numbers and lessons are from building Audit Bench Ai.