What an AI code reviewer actually catches on real PRs — and the day mine cried '20 critical issues'

#showdev #ai #devtools #buildinpublic

I spent the last few months building an AI code reviewer, solo. It's now live on
the GitHub Marketplace, which feels like a good moment to write the honest version
of what these tools catch, what they don't, and the embarrassing bug that almost
made mine useless.

The problem that started it

AI coding assistants made everyone faster — and quietly broke code review.

More code, more PRs, landing faster than any lead can review in depth. When a
stack of PRs is piling up, humans skim. And that's exactly when the off-by-one,
the unhandled null, or the API key someone pasted "just to test" slips through.

The volume outgrew the time anyone has to review it carefully. So I built
MicroReview — a bot that reviews every PR, flags likely bugs and hardcoded
secrets as inline comments, and gives the whole PR a single 0–100 risk score
so you know at a glance which ones are safe to merge and which need real eyes.

What it actually catches

The genuinely useful stuff is the "obvious in hindsight" class of bug:

A flipped conditional that inverts your auth check.
total = items.map(i => i.price).reduce(sum) / items.length — that's the average, not the sum. Every customer gets undercharged.
const apiKey = "sk_live_..." committed straight into a service file.

None of these are clever. They're the things a tired reviewer rubber-stamps on a
Friday. An AI that never gets tired and never skips the boring files is a genuinely
good second pair of eyes for exactly this.

What it does NOT catch (and anyone claiming otherwise is hand-waving)

Architecture. It reviews the diff, not the design. It has no idea if your abstraction is right.
Whole-codebase understanding. Context limits are real. On a huge diff it reviews the most relevant hunks, not literally everything.
Intent. It can't know that the "bug" is actually a deliberate workaround.

It's a second reviewer, not a replacement for one. Selling it as more than that is
how you lose developers' trust in the first five minutes.

The day it cried "20 critical issues"

Here's the bug that almost tanked the whole thing.

Early on, I let the model assign severity — critical | warning | info — and
never told it what those meant. So it did what LLMs do when everything feels
important: it slapped critical on everything. A perfectly normal PR would come
back with "🔴 20 critical issues."

That's fatal for a review tool. The entire value is signal-to-noise. If everything
is critical, developers get alert fatigue and stop trusting the score — which is
the one thing you're selling.

The fix wasn't fancier AI. It was a severity rubric in the prompt:

critical is ONLY for real production harm — security vulns, data loss, crashes
on a common path, broken core functionality. A typical PR has zero or one
critical issue. If you're marking many things critical, downgrade to warning.

Overnight, reviews went from noisy walls of red to a believable handful of
findings. Restraint, not cleverness, is what makes an AI reviewer usable.

Two decisions I'd defend

Diff-only. It only ever sends the changed lines to the model — never your whole repo, history, or branches. Better for trust, and it makes each review cost a fraction of a cent instead of dollars.
Per-repo pricing, not per-seat. The value scales with how many codebases you want guarded, not how many people are on the team. A 20-dev team reviewing 2 repos shouldn't pay for 20 seats.

Try it (no signup)

You can paste any function or diff and get a real review in a few seconds, no
account needed: https://microreview.dev/sandbox

And it's now on the GitHub Marketplace if you want it running on your PRs
automatically.

I'm building this in the open and genuinely want the hard feedback: does a single
0–100 risk score per PR actually help you triage, or is it noise on top of the
inline comments? Tell me — that answer shapes what I build next.