Do you need to read every line of AI-generated code to trust it? Not necessarily. You need to check five things instead: the intent behind a change, its acceptance criteria, the invariants it can't break, the decisions made along the way, and evidence that it works.
At Aviator, that shift let the same nine engineers merge 4× more pull requests, cut median time to merge from 2.1 days to 3.4 hours, and drop reverts by 40%. A human still signs off on every PR.
The insight behind it: most of what a reviewer writes on a PR is either a rule they've written before or a decision. Rules should be checked by a machine, every time. Decisions need a person, and a diff is the worst place to make them.
The problem: code review doesn't scale with AI
Our code flipped from human-written with AI help to AI-generated with human help. Weekly merged PRs went from about 100 to more than 400. Our median AI-generated PR reached 640 lines, well past 400 lines, widely considered the limit where bug detection falls off.
Catching bugs was never the main point of the code review anyway. In Microsoft's 2013 study (Bacchelli & Bird), only 14% of review comments were about defects. Review is how engineers share knowledge and keep their mental model of the system current, and that's the part worth protecting.
What didn't work
- An AI reviewer. On one of my PRs it opened 16 of 18 threads, mostly nits, and buried the two human comments that mattered.
- A four-hour review SLA. First reviews got faster, and approvals got emptier, as reviewers skimmed to hit the clock.
- A self-review skill tuned to our conventions. Better findings, but it still read the diff after the fact, and its reasoning died in a chat window.
All three tried to read the diff better. But the diff is the least important part of code review anyway.
What worked: Verify intent, not lines
Manual review used to do several jobs at once: catch bugs, enforce team conventions, settle design decisions, and keep everyone's understanding of the system current. We split those jobs up and gave each one a better home.
Argue before the PR exists. Our review agent cross-examines the author's agent, running on a different model, and every disagreement is written down. The agents settle most of them with a stated reason; a human gets only the ones they couldn't.
Capture intent and acceptance criteria. Not as an up-front spec, but captured from the coding session with the agent. Every time an engineer corrects the agent or answers its question, a criterion is made. We used to throw those away when the session ended.
Codify repeated comments as invariants. We mined a year of review history: 11,400 comments became 212 candidate rules and 64 approved invariants, each with a scope, a verification method, provenance, and an owner. One example: "Every config change writes an audit-log entry, only when something changed." Where possible, an agent compiles invariants into ast-grep rules that run in shadow beside the model until they agree. Nineteen now run deterministically, in 0.3 seconds.
Prove every criterion with evidence. Each criterion becomes a scenario. An agent spins up a preview environment, drives it like a user and records screenshots, requests and logs. A green CI check only says the tests you thought of passed.
Debate the decisions. The PR lists every decision, where it came from and who settled it. Reviewers skim what's settled, settle what the agents couldn't, reopen anything they'd call differently, and talk it through with the author.
Own the rules. Engineers stop reading every line. They own the rules that check every line, and keep their understanding of the system by reading intent, evidence, and decisions.
The results
- PRs merged per week: 4×, with the same nine engineers
- Median time to merge: 2.1 days → 3.4 hours
- Human threads per PR: 6.2 → 1.3
- Reverts per 100 PRs: down 40%
Review work didn't disappear. It moved to refining invariants, debating decisions, and improving verification. Adopting this framework takes time: month one is double work, by month two invariants catch enough that reviews shrink, and by month three you read intent and evidence, not code.
Start tomorrow
Mine your last 1,000 review comments, take the 20 that repeat most, and write them as checks. Every code review comment your team repeats is a guardrail you haven't written.
Top comments (0)