The first sign that our code review process was broken was a PR that went eleven rounds. Eleven. It was a 40-line change. What was actually holding it up? Nobody could tell. CodeRabbit had posted 34 comments. Copilot had posted another 12. The human reviewer had left a dozen more. Somewhere in that pile were two comments that had to be addressed before merge. The rest were style, philosophy, nice-to-haves. No one on the team could tell them apart.
The fix was not less AI. It was a shared vocabulary.
I standardized on Conventional Comments — a five-prefix labeling spec that has been sitting around since 2019, and which most teams have never bothered to enforce. Once every reviewer (human or bot) tagged their comments with issue:, suggestion:, nitpick:, question:, or praise:, the loops collapsed. In the 12 weeks after rollout, average review rounds per PR dropped from 4.5 to 1.8. Same reviewers. Same bots. Same reviewer noise level, mostly. Just labeled.
Here's what actually made it stick.
Why "please label your comments" fails
The first thing I tried was writing "please use Conventional Comments" in our CONTRIBUTING.md. This did nothing. Reviewers who cared already did it; reviewers who didn't kept dropping bare "you should probably..." comments with no prefix.
Asking politely is not a control. It's a wish.
Sorting garbage is the reference case here. Cities that ask residents to sort recyclables voluntarily get low compliance. Cities that give you a bin with a slot shaped for cans get high compliance. The lesson: change the shape of the container, not the effort required of the person. Applied to code review, that meant making the AI reviewer output labels by default and treating human unlabeled comments as second-class.
The five prefixes we actually use
Conventional Comments defines seven prefixes. We collapsed to five. The two we cut (chore and todo) were sources of ambiguity nobody wanted.
| Prefix | Meaning | Blocks merge? |
|---|---|---|
issue: |
Must be fixed. Reviewer must explain why and propose a fix. | Yes |
suggestion: |
Would improve the code, but merging without is fine. | No |
nitpick: |
Small stylistic thing. Non-blocking. | No |
question: |
Asking for intent. Author must reply, not necessarily change. | Requires reply |
praise: |
Calls out a pattern worth keeping. | No |
The rule that mattered most: issue: must include the why and a fix suggestion. Bare issue: this is wrong gets rejected as malformed. This is the block that keeps issue: from becoming a synonym for nitpick:, which is how these systems usually die.
Wiring the labels into CodeRabbit
Asking humans to label is optional. Asking bots to label is a config change. This is where we got most of the wins.
.coderabbit.yaml:
reviews:
profile: default
auto_review:
enabled: true
path_instructions:
- path: "**"
instructions: |
Every review comment MUST start with one of:
praise:, issue:, suggestion:, nitpick:, question:
Rules:
- issue: is reserved for blockers. Include (1) what is wrong,
(2) why it is wrong, (3) a specific fix. No bare issue: statements.
- suggestion: for improvements the author may skip.
- nitpick: for style-only, must be a single line.
- question: for unclear intent. Do not use to imply a change.
- praise: use liberally on genuinely good patterns.
Never use warning:, note:, or unlabeled bullets.
CodeRabbit obeys this about 95% of the time in my logs. When it slips (usually on partial-context reviews), a follow-up prompt "please retag your comments using our conventions" fixes it. That is worth keeping in your prompt library.
Wiring the labels into Copilot chat / PR review
GitHub Copilot's PR review does not accept per-repo instructions the same way CodeRabbit does. The pattern that worked: add a paragraph to .github/copilot-instructions.md and a matching one to AGENTS.md. Copilot picks up both.
# .github/copilot-instructions.md
## Review comment format
When posting a review comment on a pull request, prefix it with:
issue:, suggestion:, nitpick:, question:, or praise:.
issue: must include the reason and a proposed fix. Non-blocking
comments should use suggestion: or nitpick:.
Copilot follows this less reliably than CodeRabbit — maybe 70% of comments come back labeled. The remaining 30% we handle at the human-review layer: reviewers reply "please retag as issue: or suggestion:" and Copilot self-corrects.
The gate that made it a rule, not a hope
The last step is the reason the numbers moved. We added a CI check that fails if any PR has an unresolved issue: comment when merge is attempted.
.github/workflows/review-gate.yml:
name: Review Gate
on:
pull_request:
types: [labeled, synchronize]
jobs:
block-on-open-issues:
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@v7
with:
script: |
const {data: comments} = await github.rest.pulls.listReviewComments({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.pull_request.number
});
const open = comments.filter(c =>
/^\s*issue:/i.test(c.body) && c.in_reply_to_id === undefined
);
if (open.length > 0) {
core.setFailed(`${open.length} unresolved issue: comments`);
}
suggestion: and nitpick: never trigger the gate. Only issue:. Which means reviewers now think carefully about whether their comment is really an issue before they type the word. Self-labeling under pressure is more accurate than self-labeling in the abstract.
The numbers, one month and three months in
We measured PR review rounds — defined as any push to the branch after the initial review is requested, before merge. Same set of contributors, same repos, over 12 weeks.
| Window | Median rounds / PR | % of PRs merged in ≤ 2 rounds |
|---|---|---|
| 8 weeks before rollout | 4.5 | 22% |
| Week 1 after rollout | 3.1 | 41% |
| Week 4 after rollout | 2.2 | 63% |
| Week 12 after rollout | 1.8 | 74% |
What changed was not that we got fewer comments. Comments per PR stayed at around 30–40. What changed was that authors could immediately tell which 2–3 comments they needed to act on, and merge with the rest open. Blocker density on a comment went from "unknown, read all of them carefully" to "greppable, act on issue: only."
The number I trust the most from that table is the last column. 63% of PRs merging in two or fewer rounds after a month is the metric that predicts developer sanity. Before the change we were at 22%.
What Conventional Comments does not fix
The framing does not fix bad reviewers. If your CodeRabbit is producing 30 misguided comments per PR, tagging them nitpick: will not save you — it will just make it faster to ignore them, which is a different kind of failure. If your human reviewers use issue: to mean "I would have done it differently," you have a culture problem that no vocabulary will paper over.
What it does fix is the coordination cost of ignoring comments. That cost was the majority of my team's review overhead. Cutting it in half was worth the two-hour config change.
Book CTA
The three-layer model this fits into (hooks / AI / human) — where the AI reviewer's job is to be labeled and cheap, the hooks layer catches everything mechanical, and the human layer does not touch anything either of the other two can — is written up in the book AI Code Review as a Harness: The Three-Layer Model. Ch. 7 goes deep on Conventional Comments; ch. 9 covers the exact CodeRabbit config we ship in every repo now; ch. 12 covers autoFixable — the label we added to signal "AI, don't comment, just push a fix."
If your PR review process feels loud, this is the cheapest lever I know. Label the comments. Gate on issue:. Everything else is noise you can safely learn to skim.
Top comments (0)