DEV Community

Cover image for How I built an AI code reviewer that knows when to shut up
phi_blankslate
phi_blankslate

Posted on

How I built an AI code reviewer that knows when to shut up

#webdev #ai #programming #llm

Every AI code reviewer I tried had the same problem: it wouldn't stop talking.
Rename this. Add a comment here. Consider extracting that. By the third file
you've stopped reading, and a tool you've stopped reading is worse than no
tool — it's a tool that will hide a real bug from you inside a wall of
suggestions.

So I built one with the opposite rule: say nothing unless you found something
worth saying. That sounds like a prompt engineering problem. It isn't. The
model will happily agree to be concise and then hand you fourteen findings
anyway. Every constraint that actually held up in production is a constraint
I enforce in application code, after the model has already spoken.

Here's what that looks like, including three bugs I only found by pointing
the thing at real pull requests and a real credit card.

The problem: nitpick fatigue is a trust problem, not a UX problem

A reviewer you've muted is worse than no reviewer, because now there's a wall
of suggestions for a real bug to hide behind. This matters most for solo
developers and small teams — the people who don't already have an enterprise
code-review bundle sitting on top of their existing tools. If the review
output is noisy, they turn it off in week one and never come back.

Architecture in one line 

GitHub webhook → diff fetch → Claude → structured findings → ranker → inline comments
Enter fullscreen mode Exit fullscreen mode

Every stage after "Claude" exists to decide what not to show you.

Structured output, then rank it yourself

The model returns JSON findings, each with { path, line, severity, message },
where severity is one of four values: BUG | WARN | NIT | PRAISE. The
severity string coming back from the model is validated against that
whitelist — an invalid value gets the finding dropped rather than trusted.
Then everything is stable-sorted by severity:

const severityRank: Record<ReviewComment["severity"], number> = {
  BUG: 0,
  WARN: 1,
  NIT: 2,
  PRAISE: 3,
};
const rankedComments = sanitizedComments
  .map((comment, index) => ({ comment, index }))
  .sort((a, b) => {
    const rankDiff = severityRank[a.comment.severity] - severityRank[b.comment.severity];
    if (rankDiff !== 0) return rankDiff;
    return a.index - b.index; // same-severity ties keep the model's own order
  })
  .map((entry) => entry.comment);
Enter fullscreen mode Exit fullscreen mode

Stable sort matters here: within the same severity, findings keep the order
the model produced them in, instead of being silently reshuffled.

The cap is a constant, not a request 

const comments = rankedComments.slice(0, MAX_COMMENTS);
const suppressedCount = sanitizedComments.length - comments.length;
Enter fullscreen mode Exit fullscreen mode

MAX_COMMENTS is a constant, applied by slice after the sort — not a line
in the prompt asking the model to "please be concise." A prompt is a request
the model can ignore on a bad day; a slice() cannot.

What gets cut isn't dropped silently. suppressedCount flows into the review
body:

body += `🔇 ${result.suppressedCount} lower-priority remark${result.suppressedCount !== 1 ? "s" : ""} suppressed to keep this review focused.\n\n`;
Enter fullscreen mode Exit fullscreen mode

Why disclose the count instead of just trimming quietly? Because "quiet
reviewer" and "reviewer that missed it" look identical from the outside
unless something tells you which one you're looking at.

Fair pushback on this design, and I don't have a clean answer: on a PR with
more than 8 genuine bugs, the cap works against you. Sorting bugs to the
front means you at least see the worst of it first, but "the cap doesn't
hide real bugs" is not a guarantee I'm willing to write — only that severity
ordering makes it less likely.

PRAISE is not decoration

There's a fourth severity that isn't a problem at all — a slot reserved for
"this was a good change." A review that's only ever negative gets the same
treatment as a chatty one: people stop opening it.

Position anchoring, and the fallback that keeps a finding alive

GitHub's inline PR comments only land if the position matches the actual
diff hunk. If a finding's line doesn't anchor, the naive move is to drop it.
Instead, the handler falls back to a single top-level comment that lists
everything, formatted findings included:

try {
  await createReview(/* ...inline comments... */);
} catch (reviewError) {
  console.error("Inline review failed, falling back to issue comment:", reviewError);
  await postPRComment(octokit, owner, repo, prNumber, fallbackCommentBody(reviewResult));
}
Enter fullscreen mode Exit fullscreen mode

A formatting mismatch shouldn't be able to delete a finding.

Bug #1 — the webhook that succeeded and failed at the same time

GitHub wants a 2XX response within roughly 10 seconds of a webhook delivery.
Generating a review — fetch the diff, call the model, post the comments —
routinely takes longer than that. Doing all of it synchronously meant GitHub
logged the delivery as failed while my function kept running and posted the
comments anyway. The delivery log said "failed." The PR said otherwise. Both
were technically correct, which made it maddening to debug.

The fix: verify the signature, return 200 immediately, and do the actual
work in the background.

case "pull_request":
  waitUntil(
    handlePullRequestEvent(payload).catch((error) => {
      console.error("PR review background processing failed:", { deliveryId }, error);
    })
  );
  break;
Enter fullscreen mode Exit fullscreen mode

Bug #2 — the silent fail that going async created

Async fixed the timeout problem and introduced a worse one: if the
background work dies partway through, the review row just sits at
"pending" forever. Not visible on the dashboard as an error. Not counted
against quota. GitHub already has its 200. Nothing anywhere tells you a
review didn't happen — that's the definition of a silent failure.

The fix was to stop depending on platform defaults and make the ceiling
explicit:

// Give the background work (PR file fetch + Claude review + GitHub post)
// enough time. Leaving this unset silently inherits Vercel's default,
// and on timeout the review sits at "pending" with no way to detect it.
export const maxDuration = 120;
Enter fullscreen mode Exit fullscreen mode

Going async doesn't remove failure, it changes what failure looks like.
Anything that runs outside the request/response cycle needs its own
explicit way of surfacing "this didn't finish" — a timeout, a status you
can query, something. Silent pending isn't good enough.

Bug #3 — the subscription that said "Renews" after being cancelled

This one wasn't a code review bug, it was a billing bug, and I only found it
because I ran my own Stripe checkout on the live account. Cancelled a trial
from the customer portal. Stripe's own dashboard confirmed "cancels
[date]." My app's dashboard kept saying "Renews."

The webhook handler was trusting cancel_at_period_end as the single source
of truth for "is this cancelling":

cancelAtPeriodEnd: sub.cancel_at_period_end,  // before the fix
Enter fullscreen mode Exit fullscreen mode

Pulling the actual webhook payload in the Stripe dashboard showed the real
shape of the event: cancel_at_period_end stayed false through the whole
cancellation, while cancel_at flipped from null to a real timestamp.
Current Stripe subscription behavior resolves an end-of-period cancellation
directly to a cancel_at timestamp rather than only flipping the boolean.
The fix reads both:

// cancel_at_period_end alone isn't reliable here — cancellation can resolve
// straight to a cancel_at timestamp instead. Check both, keep the boolean
// path for backward compatibility.
cancelAtPeriodEnd: sub.cancel_at_period_end || sub.cancel_at != null,
Enter fullscreen mode Exit fullscreen mode

Two things I took from this: don't trust a single boolean field to represent
a state transition without checking the real payload first, and billing
paths get tested with a real card, not a happy-path assumption about what
the API returns. Six lines to fix, but shipped as-is it would have told
every cancelling customer a lie about their own subscription.

Try it

DevReview reviews GitHub pull requests and ranks findings BUG → WARN → NIT →
PRAISE, hard-capped at 8 comments per review with the suppressed count
disclosed rather than hidden. Free tier is $0 forever — 15 reviews/month on
one repo, no card involved. Pro is $9/user/month with a 14-day trial (card
required at checkout, first charge on day 15).

https://getdevreview.com

I'd genuinely like to know where the 8-comment cap is the wrong call —
tell me if you try it.

Top comments (1)

Collapse
 
jo-do profile image
Jo Do •

The suppression count is a useful honesty signal, but I would make the cap conditional on severity rather than absolute. Keep the eight-comment budget for WARN/NIT/PRAISE, while allowing BUG findings past the cap into a collapsed top-level section. That preserves attention without silently turning the ninth real defect into telemetry.

The more interesting metric is not total comments per review but accepted findings per unit of reviewer attention. Track which comments lead to a fix, dismissal, or mute, then tune the cap per repository. A team that fixes six of eight findings has a different noise threshold from one that dismisses seven.