An LLM reviewer earns its place when it catches the boring stuff a human skims past — missing null checks, a swallowed error, a test that asserts nothing — and stays quiet otherwise. It becomes a liability the moment it posts twelve comments per pull request, half of them restating what the diff already says. The difference is almost entirely in how you scope and gate it, not in which vendor you pick.
I've run LLM review on a few repos now, both hosted tools and a homegrown GitHub Action. The teams that kept it running past week two all did the same thing: they made the bot advisory, narrowed what it was allowed to comment on, and treated every false positive as a config bug to fix rather than noise to tolerate.
What can an LLM actually catch that linters miss?
Linters and type checkers already own the deterministic layer: formatting, unused variables, obvious type mismatches. An LLM is worth adding only for the fuzzy layer above that — the reasoning a static rule can't encode.
In practice the useful hits fall into a few buckets: error handling that looks fine but swallows or misclassifies failures, off-by-one and boundary logic in new code, tests that run green without actually asserting the behavior they claim to, and security-adjacent smells like unparameterized queries or secrets pasted into config. It's also genuinely good at "this function's name no longer matches what it does" — the kind of drift a human reviewer stops noticing after the third file.
What it's bad at is anything requiring repo-wide context it can't see. Ask it "does this break an existing caller two directories over" and it will confidently guess, because the diff is all it was handed. That guess is where most of the noise comes from.
Takeaway: Point the LLM at judgment calls a linter can't make, and keep it away from questions that need context outside the diff.
How do you keep it from flooding every PR?
The single highest-leverage setting is a comment budget. Cap it — three or four inline comments per PR, maximum. Forcing the model to rank and pick its strongest findings does more for signal quality than any prompt tuning, because it turns "list everything you notice" into "what are the two things most worth a human's attention."
Beyond the cap, four rules that consistently reduced complaints:
- Advisory, never blocking. The bot posts comments; it does not set a required status check. A required LLM gate that hallucinates once will get muted by the whole team within a day.
- Only comment on changed lines. Reviewing the full file invites the model to relitigate code nobody touched in this PR.
- Suppress style opinions entirely. If your formatter runs in CI, the LLM has no business mentioning naming or whitespace. Say so explicitly in the prompt.
- Skip trivial diffs. Lockfile bumps, generated code, and pure renames don't need an opinion.
Takeaway: A three-comment budget on changed lines only, with no power to block, is what separates a reviewer people keep from one they mute.
Which tool should you reach for?
You have three broad options, and the right one depends on how much control you need versus how fast you want it running.
| Option | Setup effort | Control over behavior | Cost model | Best when |
|---|---|---|---|---|
| Hosted service (CodeRabbit, Qodo Merge, Ellipsis) | Install app, done | Config file, limited | Per-seat or per-repo subscription | You want it live this afternoon |
| Copilot code review | Native to GitHub if you have Copilot | Low, GitHub-controlled | Bundled with Copilot | You're already paying for Copilot |
| Custom GitHub Action + LLM API | Half a day of YAML and prompt work | Total | Per-token API usage | You need domain-specific rules or data control |
Hosted tools like CodeRabbit and Qodo Merge (the successor to the open-source PR-Agent) get you a capable reviewer with almost no work, and they handle diff chunking and comment threading for you. The trade-off is that you're tuning behavior through their knobs, and your code passes through their infrastructure — which may or may not clear your compliance bar.
A custom Action costs you a morning but gives you the prompt, the budget, and the model choice outright. As of mid-2026 the general-purpose frontier models from Anthropic and OpenAI are all strong enough at code reasoning that model choice matters less than prompt discipline. Every one of these tools will occasionally suggest a "fix" that's wrong or that reintroduces a bug — the honest limitation across the whole category is that you cannot trust a suggestion without reading it.
Takeaway: Start hosted to prove the value, move to a custom Action only when you hit a behavior or data-residency wall you can't configure around.
What does a minimal custom setup look like?
If you go the custom route, the shape is small. A workflow that triggers on PRs, pulls the diff, sends it to an API with a tightly scoped prompt, and posts back the top findings. Here's the skeleton of the trigger:
name: llm-review
on:
pull_request:
types: [opened, synchronize]
jobs:
review:
runs-on: ubuntu-latest
permissions:
pull-requests: write # to post review comments
contents: read
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Get PR diff
run: git diff origin/${{ github.base_ref }}...HEAD > pr.diff
- name: Run review
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
run: python review.py pr.diff
The prompt is where the discipline lives. The instruction that mattered most in my testing was making the model justify silence — telling it that returning zero comments is a valid and often correct answer:
SYSTEM = """You review a git diff for a pull request.
Report ONLY: likely bugs, swallowed errors, missing edge-case handling,
tests that don't assert their stated behavior, and security issues on
CHANGED lines. Ignore style, naming, and formatting.
Return at most 3 findings, ranked by severity. If nothing meets the bar,
return an empty list — that is the correct answer for most PRs.
Each finding: {file, line, one-sentence issue, suggested fix}.
Respond as JSON only."""
Parse that JSON, drop anything not on a changed line, and post the survivors as review comments. Requesting structured JSON instead of free-form prose is what makes the "changed lines only" and budget filters enforceable in code rather than hoped for in the prompt.
Takeaway: Ask for ranked JSON with an explicit empty-list option, and enforce your filters after the model responds, not just inside the prompt.
How do you know it's actually helping?
Two numbers tell you almost everything. Track the ratio of bot comments that a human marks resolved-as-useful versus dismissed, and track how often the bot comments at all. A healthy reviewer stays quiet on most PRs and lands a real catch when it does speak. If it's commenting on every PR, your bar is too low. If nobody ever acts on its comments, turn it off — a muted bot is worse than no bot, because it trains the team to ignore an automated reviewer's voice entirely.
Give it a two-week trial with an explicit kill switch, and tell the team upfront that noisy comments are bugs to report, not etiquette to endure. That framing keeps people filing config fixes instead of quietly resenting the thing.
Takeaway: If fewer than a third of its comments get acted on, the problem is your scoping, not the model.
Bottom line
Solo developers and small teams should start with a hosted tool — CodeRabbit or Qodo Merge — set to advisory-only, because the setup cost is near zero and you'll learn what's worth catching before you invest in tuning. Teams with domain-specific rules, strict data-residency needs, or a strong opinion about comment volume should build a custom GitHub Action so the budget and scope live in code they control. Whatever you pick, keep it non-blocking, cap it to three or four comments, and restrict it to changed lines. The goal isn't a bot that reviews like a senior engineer — it's one that reliably catches the tedious mistakes so your humans can spend their attention on design.
Top comments (0)