Mayur Agnani

Posted on Oct 20 Originally published at theaicommit.com

How to Stop a Coding Agent and an AI Reviewer From Looping on One Pull Request

#ai #testing #devops #productivity

Engineer's view. This is "retry until green" on a flaky test suite. The job stops at the first green run, so noise decides when it ends, and one green run proves little. An agent and an AI reviewer stop the same way: at the first review with nothing to fix. It costs rounds and missed bugs.

TL;DR. An agent and an AI reviewer stop looping only when a review finds nothing to fix. Let only new, important problems on changed lines block, and stop after two clean reviews in a row.

Explain Like I'm 5

You ask a friend to check your story for mistakes. Every time they find something, you fix it and hand it back. Some of their notes are real mistakes. Some are just "I'd use a different word." Each fix gives them a fresh page to find new little things on. So you go back and forth all afternoon. Then one time they glance and say "looks fine," and you stop. But a glance can miss a real mistake. Better rules: only fix real mistakes, and stop when they've read it carefully twice and found nothing.

The Problem

You have shipped this loop before, in CI.

A test suite has a few flaky tests. Someone adds a retry: run the suite until it passes, up to five times. Two things go wrong. Builds take longer, because the flaky tests keep failing for no reason. And a real failure that only shows up half the time now gets through. One lucky green run is enough. The stop rule was "the first green run," and noise decided when that came.

A coding agent working with an AI reviewer runs the same loop. The reviewer reads the pull request and leaves comments. The agent fixes every one, because to an agent a comment is an instruction. Each fix is a new commit, so the reviewer runs again. The loop ends at the first review that finds nothing to fix.

Anthropic's Claude Code team, where Claude writes about 90% of the code, measured this. Before they changed their rules, 8.9% of internal pull requests needed five or more review rounds. One pull request ran more than fifty rounds overnight. About 59% of what the reviewer kept flagging in later rounds was code that was already there before the change.

The long tail costs CI time and attention. The quieter cost is the one the flaky suite taught you: a single clean review is weak evidence when the reviewer misses half the bugs on any one pass.

Decide what is allowed to block, and decide when the loop may stop. Block only new, important findings on lines the change touched. Stop after two clean reviews in a row. After three fix rounds, hand the pull request to a person.

The Fix: Let Only Real Findings Block, and Require Two Clean Passes

New to how an agent works through a task? Start at AI basics → The agent loop.

Why does the loop run so long?

Because the stop rule counts every comment. A review pass is one full read of the pull request by the reviewer. A fix round is one commit the agent makes in answer to it. The loop stops at the first pass with nothing to fix.

Three kinds of comment keep that from happening. Nits are new on every pass, because the reviewer samples them fresh. Findings in old code reappear, because the old code is still there. And the same finding can come back in new words. The agent fixes all of them, and each fix starts another round.

Why not just cap the rounds, or re-review only the new lines?

Both cut rounds, and both lose bugs. Each full pass is another independent look at the real bugs the last pass missed. In the simulation in code_example.py, the reviewer's recall is 50%: it catches each real bug half the time per pass. Cutting the noise but stopping at one clean pass let 24.4% of bugs escape. The noisy loop let 13.1% escape. Its extra rounds were accidental second looks.

So what should block, and when should it stop?

Only a blocking finding starts a fix round. That means new, tagged important, and on a line the pull request changed. A pre-existing finding goes on a list once. A finding is identified by its file, line, and rule, so a repeat is dropped. A clean pass must happen twice in a row before the gate passes. The second clean pass costs a review, but no agent commit and no CI run.

What This Means for You

When this matters. You run a coding agent that answers review comments on its own. That might be the Claude Code GitHub Action, Copilot's coding agent with Copilot review, or your own loop. It matters most on large pull requests. The Claude Code team found most pull requests over 1,000 lines get findings, while most small ones come back clean.

How it affects you. The first thing you notice is CI cost and slow merges. The thing you don't notice is escapes. Loosening the loop to save rounds can quietly double the bugs that get through. The agent's fix commits are the expensive part. Each one reruns CI, and each one can add a bug of its own. In the simulation, the two-clean-pass gate cut agent fix rounds from 2.07 to 1.01 per pull request. No pull request needed five rounds. Escapes stayed at 13.1%, the same as the noisy loop. The cost: 13.4% of pull requests went to a person after three rounds.

What to do about it. Today, count how many review rounds your last twenty agent pull requests took. Look for rounds where the agent fixed only nits or old code; those are the rounds the gate removes. Then write down what is allowed to block. Change the stop rule last, and only together with the blocking rule.

Implementing It

The change.

Three roles touch this. Whoever configures the reviewer decides what it tags important. Whoever owns CI runs the gate. Whoever writes the agent's instructions stops it from fixing what does not block.

Role 1, the reviewer's configuration. If you use Claude's managed code review, put this in REVIEW.md at the repo root. Other reviewers take the same text as part of their prompt:

## What counts as Important here
- A bug on a line this pull request changed, with file:line evidence.
- Not style, naming or cleanup. Tag those Nit.
- Code the pull request did not touch: tag Pre-existing, never Important.
Start your summary with the count of Important findings.

Ask the reviewer to emit its findings as JSON: path, line, rule, severity. The gate reads that. Keep the rule names stable from pass to pass, because the gate matches repeats on them.

Role 2, CI. The gate is a required status check on every push. It keeps a small state file per pull request, so it can count clean passes and fix rounds across pushes:

def decide(self, findings):
    blocking = [f for f in self.new(findings)
                if f.severity == "important" and (f.path, f.line) in self.changed]
    if not blocking:
        self.clean_streak += 1
        return "pass" if self.clean_streak >= REQUIRED_CLEAN else "review"
    self.clean_streak = 0
    if self.fix_rounds >= MAX_FIX_ROUNDS:
        return "escalate"
    self.fix_rounds += 1
    return "fix"

self.new() drops any finding whose path, line, and rule were already seen. On "review", run the reviewer again without waiting for a commit. On "escalate", fail the check and request a human reviewer. Get the changed lines from git diff --unified=0 against the merge base, so a rebase does not count old lines as new. Store the state as a hidden comment on the pull request or in the CI cache. Either one survives between pushes.

Role 3, the agent's instructions. Add this to CLAUDE.md, or to whatever file your agent reads:

When you answer review comments, fix only findings the review gate lists as blocking.
Do not fix Nit or Pre-existing findings in this pull request. Do not edit lines
outside the diff to satisfy a comment.

Without this, the agent fixes nits anyway, and every nit still costs a commit.

How you know it worked.

Before you switch, record three numbers over twenty pull requests: fix rounds per pull request, the share needing five or more rounds, and reverts within a week of merge. After two weeks, rounds should fall and the five-round tail should vanish. Escalations should stay near one in ten; far above that, the important tag is too loose. Reverts must not rise. If they do, you have only moved the work, so raise the required clean passes before anything else.

When a Two-Clean-Pass Gate Is the Wrong Tool

When two passes are not independent. The second pass helps only if it can miss different things than the first. A reviewer at temperature zero, with the same prompt and the same context, may return the same answer twice. Then the second clean pass proves nothing. Vary something between passes: a second model, a different focus, or parallel reviewers that vote.

When a person writes the code. A human author pushes back on nits and rarely fixes all of them. The loop that this gate controls mostly doesn't form.

When the change is small and low risk. A docs fix or a flagged, additive change can pass on one clean review. Match the passes to the risk, as the Claude Code team does with its review tiers.

When nobody reads the escalations. A gate that sends 13% of pull requests to a person needs a person. Otherwise those pull requests just sit, and engineers learn to route around the check.

Before you adopt it, ask:

Do my reviewer's passes actually differ from each other?
Who picks up a pull request after three fix rounds?
Do I know my revert rate today?

Glossary

review pass — one full read of the pull request by the AI reviewer.
fix round — one commit the agent makes in answer to a review pass.
blocking finding — a new finding, tagged important, on a line the pull request changed.
pre-existing finding — a finding in code the pull request did not touch; listed, never blocking.
clean pass — a review pass with no blocking findings.
recall — the share of real bugs one review pass catches.

DEV Community