Operations teams learned this lesson a decade ago and wrote it on the wall: a pager that cries wolf doesn’t get silenced; it gets ignored, and then the real page scrolls by unread. AI code review is now relearning it, one speculative comment at a time. False positives are the #1 complaint against every tool in the category, and the damage isn’t the wasted minute per dismissal: it’s what the dismissals do to the reader.
The short answer: Noise doesn’t just dilute review; it retrains the reviewer. Each dismissed false positive lowers the attention paid to the next comment, until the bot’s genuine catches get the same reflexive skim as its nonsense. A reviewer, human or machine, is only as useful as its credibility, and credibility is a budget that bad comments spend.
The mechanism, not the moral
When a bot leaves nine comments on a PR and two matter, the author must spend their own judgment to find the two, which is precisely the work the bot was bought to reduce. Worse, the triage is repetitive: the same speculative concerns recur on every PR (“consider handling the case where this config is undefined”; it can’t be), so developers build the only rational defense: a mental filter labeled probably nothing. The team that built one popular reviewer documented this exact arc: they had to engineer nitpicking out of their own bot because noisy comments were burying the signal ones. We saw the same dynamic with human nitpicks; machines just produce them at scale.
Tuning for trust
- Deterministic first. Anything a linter or type checker can decide should never reach the AI layer: it’s free precision, and it shrinks the surface the bot can be wrong about.
- Narrow the bot’s beat. Security-sensitive paths, error handling, concurrency: places where a speculative flag is worth reading. Silence elsewhere is a feature.
- Demand confidence, suppress hedging. Most tools accept custom instructions; “only comment when you can name the failing input” filters out the twenty-speculative-reasons genre.
- Measure dismissal rate per bot, visibly. What gets measured gets tuned; a bot nobody audits drifts noisy.
- One reviewer’s voice, not three. Stacking multiple AI reviewers multiplies noise faster than coverage: their false positives don’t overlap; their true positives mostly do.
Protect the scarce resource
The thing alert fatigue actually destroys is the thing review runs on: reviewer attention. Every unit spent dismissing speculation is a unit not spent on the core files of the change. That’s the lens we’d apply to any addition to the review loop, our own product included: does it add attention (by making reading faster, by triaging what matters first) or does it tax attention and call the tax a feature? Tools that can’t answer that cleanly end up in the mental spam folder, no matter what their benchmark said.
Frequently asked questions
What dismissal rate means a review bot is too noisy?
There’s no published standard, but the ops world’s alerting experience suggests the bar is high: once a clear majority of a bot’s comments get dismissed without action, developers stop reading them carefully, at which point even its valid findings underperform. Track resolved-vs-dismissed per bot comment and treat a sustained slide as an incident in your review process.
Should we just turn the AI reviewer off?
Not necessarily: narrow it. Most tools accept path filters, severity thresholds, and custom instructions. A bot that only speaks on security-relevant paths and hard-error patterns, and stays silent elsewhere, keeps its credibility. The kill criterion is simple: if after tuning the team still skims past it, it’s costing more attention than it saves.
Top comments (0)