Our AI code reviewer problem wasn't that it was wrong. It was that it wouldn't shut up

I've been thinking about a problem with AI code review that I don't see discussed enough: a reviewer can be technically correct and still make the whole review experience worse.

Here's the PR that made this click for me. Small change, maybe 200 lines, nothing exotic. New validation path, a couple helper functions. The kind of thing that used to get a comment or two and an approve.

The AI reviewer came back with something like a dozen comments. A naming suggestion. A "this null check might be redundant" note. A theoretical race condition that would only show up under conditions the code was never going to hit in prod. A "you could extract this into a function" suggestion on a function that was already about ten lines long.

And here's the annoying part most of these weren't wrong. The variable name genuinely could've been clearer. The null check probably was redundant. Individually, I could defend almost every comment on that list.

But there was one comment in there about an actual edge case in the validation logic that genuinely mattered. And it was sitting in the middle of eleven other comments that didn't. The dev opening the PR skimmed most of it, dismissed a few without really reading them, and nearly skipped the important one too — because by comment seven or eight, he'd already stopped expecting any of it to be worth his time.

Nobody wanted "more thorough." They wanted to know what actually mattered.

That's the distinction I think gets missed constantly: "is this technically an issue" and "is this worth interrupting someone for" are completely different questions, and most tooling in this space only ever answers the first one.

The assumption baked into a lot of AI review tools is basically: if finding more issues is good, finding even more must be better. But review was never just an issue-detection problem, it's a signal-to-noise problem, and honestly kind of a trust problem too. Once a reviewer starts treating a PR like a checklist to maximize, it stops mattering how accurate any individual comment is — the aggregate experience is what people remember, and the aggregate experience was "annoying."

That frustration eventually became one of the reasons I started working on Codzee. Not going to turn this into a pitch — the short version is the goal isn't "find everything," it's figuring out what actually deserves a developer's attention. Still early, still figuring a lot of it out.

Genuinely curious how other people think about this though:

• How many AI review comments do you actually want to see on a normal PR? Is there a number where it tips from "helpful" to "annoying" for you?
• Do you find yourself tuning out AI-generated comments after a tool has burned you with noise a few times?
• Where's your personal line between a useful warning and something that's just noise dressed up as a finding?
• Would you rather a reviewer miss a low-confidence issue entirely, or flag it anyway and let you decide?
• What actually makes you trust an automated reviewer, versus just tolerate it?

Not trying to manufacture a debate here, I'm actually unsure about some of these myself — like the miss-vs-flag question, I go back and forth on what the right default even is. Curious what this sub thinks.

Comments

[Bert Shim]

What changed is that the cost of leaving a comment went to zero. A human who noticed all twelve of those would still only write three, because the other nine aren't worth the goodwill. That brake was doing more than I realised.

Running several agents, the ones I keep paying attention to are the ones that stay quiet on small stuff. Not smarter, just quieter, and that turns out to be most of it.

No idea how you tune that without also losing the occasional nitpick that was load-bearing.