A developer added an AI reviewer to a small Node and React project expecting an easy win.
At first, the comments looked useful. Then the reviewer started repeating style complaints, commenting on code that had already changed, and missing a misplaced null check that crashed the application in staging.
The team still had to perform a complete human review.
That experience, shared in a public DevOps discussion, captures the real question engineering leaders should ask before adding an AI reviewer to every pull request:
Did the reviewer remove work from the team, or did it create another thing the team had to review?
The problem is not that AI review never works
Developers report genuinely useful results too.
In one Experienced Developers discussion, engineers described AI reviewers catching privacy leaks, incorrect data-flow assumptions, and logic errors that human reviewers had missed. In the same discussion, another engineer said their review bot was useful but produced plausible, inaccurate comments about one-third of the time.
These are anecdotes, not a benchmark. But together they explain why the debate feels confused. AI review is not simply good or bad. Its value depends on the codebase, the context available to the reviewer, the kind of issue being reviewed, and how much verification its output requires.
A tool can catch one subtle bug and still make the overall review process slower. It can also say nothing on several pull requests and then save a team from a serious failure. Counting comments cannot distinguish between those outcomes.
Comment volume measures activity, not value
GitHub says Copilot code review has completed more than 60 million reviews. Its definition of a good review has changed as that volume has grown. The team says it moved from optimizing for thoroughness to optimizing for accuracy, signal, and speed.
GitHub reports actionable feedback in 71% of Copilot reviews. In the other 29%, the reviewer says nothing. That silence is intentional: if there is no consequential finding, manufacturing a comment only adds noise.
GitHub has also added severity labels and grouped similar comments so developers do not have to process the same suggestion repeatedly on a large pull request. The product change reflects an important design constraint: every review comment spends human attention.
A comment is not free because a machine wrote it. Someone still has to read it, understand it, check it against the code, decide whether it is correct, and either act on it or dismiss it.
The test is net review work
Imagine an AI reviewer leaves eight comments. Three lead to useful fixes. Two are correct but trivial. Three are wrong and require investigation before the team can dismiss them.
Was that a successful review?
The answer depends on what happened to the team's workload. If the three useful findings avoided an incident or removed a difficult debugging session, the review may have been valuable. If the comments repeated checks a linter already performed and delayed approval, it was not.
For a pilot, track three outcomes:
- Resolved findings: how often a comment leads to a code change before merge.
- Dismissal rate: how often a developer dismisses, rewrites, or disproves the finding.
- Human review time: whether the team spends less time getting a pull request to an informed approval.
A fourth number, comment count, can help explain the other three. It should never be the target.
Remove the predictable work before asking AI to reason
Many review comments should not require an AI model at all. A swallowed exception, unsafe type assertion, debug statement, unresolved placeholder, dead export, or agreed complexity limit has a repeatable answer. The same code should receive the same verdict every run.
Run those checks before contextual review. That keeps predictable findings out of the AI reviewer's prompt and out of the human reviewer's queue. It also gives the team an enforceable baseline instead of asking a probabilistic reviewer to remember every rule on every pull request.
Then use AI where context matters: whether a change matches the issue, whether a migration missed a call site, whether the data flow creates an unexpected privacy risk, or whether the code conflicts with a local architectural decision.
Humans still own product intent, operational risk, and the decision to merge. The full workflow is covered in our guide to automated code review for AI-generated code.
A quiet reviewer can be a good reviewer
Engineering teams do not need a bot that proves it was present on every pull request. They need fewer repeatable problems reaching review, useful context when something consequential is wrong, and enough silence to trust that a comment deserves attention.
Before rolling an AI reviewer across every repository, test it on real pull requests. Measure the findings your team accepts, the findings it has to correct, and the time people actually spend reviewing. Keep it only if the total burden falls.
To detect repeatable AI-code quality findings before either an AI or a human reviews the pull request, run:
npx aislop@latest scan
aislop provides that deterministic first pass without using an LLM at runtime: the same code receives the same verdict on every run.
Top comments (0)