Every comment you write twice should become a check, a helper, or a rule for your AI reviewer.
I asked my coding agent to read a year of my code review comments and tell me what I keep saying. The answer was humbling: the same race in a cache, the same log line inside a loop, the same metric tagged with a user ID, PR after PR. I had been fixing PRs, not the reasons the mistakes kept coming back.
If agents write a growing share of your team’s code, that habit gets expensive fast. Here’s how I turned those repeats into checks, helpers and a lean AI review skill, and how to decide which agent-written PRs still need a human.
A year of review comments, and the same ones kept coming back.
Why now
Zarar Siddiqi’s The System Around the Code cites Faros AI data: teams that use AI a lot merged 98% more PRs, but PRs took 91% longer to review. Writing code got faster, and review became the bottleneck. The Claude Code team at Anthropic describes the same shift in How the Claude Code team does code review: changes landing per day in Anthropic’s monorepo went from about 2,500 in May to more than 30,000 in September.
Siddiqi has a line I couldn’t shake: “Every mistake should only happen once.” So I checked how many of mine had happened more than once.
Every repeat needs an exit plan
Not every repeat comment belongs in your AI reviewer. It’s an LLM, so it can miss things, and every rule you add competes for its attention. So before I teach it anything, I move out everything a tool or shared code can handle. A comment I write twice has three possible exits, and only the last one is the AI reviewer.
Automate it, when a tool can find the mistake. For example, if you tag a metric with a user ID, your monitoring system creates a separate data series for every user. Nothing breaks, but the bill keeps growing. A small lint rule that fails the build on this kind of tag stops the mistake before anyone reviews the PR.
Design it out, when no tool can find the mistake, but shared code can prevent it. Write the correct code once, in a shared function, and let everyone use it. A cache stampede is a good example. When a popular cache entry expires, many requests miss it at the same time and all of them go to the database. No lint rule can see this in the code. One shared GetOrCreate function that handles it correctly means nobody writes this logic by hand again, and my review comment becomes “use GetOrCreate”.
Teach it, when finding the mistake needs judgment. Can this change show one customer’s data to another customer? Does the new test really check anything? A tool can’t answer these questions, so they go into the instructions of the AI reviewer.
The Claude Code team does the same thing. When their AI reviewer keeps finding the same problem, they fix it in one of three places: the reviewer’s instructions, a lint rule, or a REVIEW.md file that the code owners write. They also share a tip that works for any AI reviewer: ask it to show the file and line that prove the problem, so it can’t just guess.
Lint your skill
Every lesson I add to the skill competes for the model’s attention, and a reviewer that flags everything gets ignored, human or not. Noise costs even more when the author is an agent. A person waves off a nit in seconds, while an agent fixes it, “which means another commit and another round of review.” One of the Claude Code team’s PRs went through more than fifty rounds overnight. Their fix is a cap: after three rounds, the reviewer asks the author to step back and reconsider the approach.
To keep the skill lean, I borrowed from Andrej Karpathy’s LLM wiki pattern:
- Keep one short core file with the rules for every PR. Put the rest in one file per area, like data-model.md or concurrency.md. The core file tells the reviewer which one to read, so it only reads the rules that fit the change.
- Each rule is the why plus one example.
- The skill is a waiting room. In Siddiqi’s words, instructions are advice and checks are law. Once a lesson is clearly detectable, it graduates into a check and leaves.
- A lint pass every quarter removes rules that contradict each other, rules the code has outgrown, and rules that caught nothing.
Who still has to read it
If the AI reviewer passes an agent-written PR, does a person still need to approve it? Siddiqi’s answer is change lanes: the type of change decides the type of review. The Claude Code team enforces it. A classifier picks how many people must approve each change: none for docs or additive changes behind a flag, one for ordinary product changes, two for the riskiest paths. CI workflow files always need a human. Three more rules caught my eye:
- An agent’s approval never fills a human’s slot.
- Whoever prompted the agent counts as one of the required reviewers, as if they had typed the code.
- An approval covers one commit. If the agent pushes again, someone approves again.
The second one is where I’d start. The person who prompted the agent owns the change, so they should be able to say what it’s supposed to do. A short Intent Header on the PR does that, and gives the approver something to check the diff against.
You don’t need a classifier like Anthropic’s to start. Two GitHub settings do most of the job. A CODEOWNERS file says who must approve changes to sensitive folders. A branch protection rule cancels old approvals when someone pushes a new commit, so an agent can’t change the code after a person approved it.
Review the reviewer
New repeat comments keep showing up, so run the whole process again every three months: read your recent comments, find the repeats, and give each one an exit.
Then check that the reviewer still finds what matters. The Claude Code team collected old PRs that caused incidents and runs the reviewer on them every day, to check that it still finds those bugs. Each new incident adds another PR to the set, and every change to the reviewer is tested against all of them. They also track what happens to each comment the reviewer leaves: the author fixed it, said it was wrong, said the code was like that on purpose, or decided it didn’t matter.
The Bottom Line
Before you teach an AI reviewer, review your own reviews. Give every repeat comment an exit: automate what a tool can find, design out what shared code can prevent, and teach the AI reviewer only what needs judgment, in short instructions. Let the type of change decide who still reads the PR, and whether anyone needs to. Every mistake should only happen once, including the ones you keep catching in review.
P.S. A short version of the prompt I used, if you want to run it on your own history (read-only):
Analyze my PR review comments (user: <MY_USERNAME>) from the last 12 months, read-only.
Group them by the underlying issue, not the wording, and rank the themes by the number of PRs.
For the top 10, give a short generic name, what I usually ask for, two anonymized examples,
and which exit fits best: automate it (lint rule, analyzer or test), design it out (shared helper),
or teach it to the AI reviewer. Don't include anything confidential. If you're unsure about a grouping, say so.
Top comments (0)