An AI code reviewer that remembers its findings still needs a stop condition

Wonil Cho — Thu, 10 Sep 2026 08:00:58 +0000

An automated code reviewer can get stuck in a loop that looks productive:

It finds three problems.
An agent fixes two and changes something nearby.
The next review finds one old problem again, plus two new ones.
Repeat until somebody gives up.

The failure is not just that the model is imperfect. It is that the loop has no explicit end state.

I built an optional review runner in frontier-simplify to make that failure visible and bounded. It is a local maintainer tool, not a hosted service and not a merge gate. The runner keeps concrete findings across review rounds, checks later repairs against those findings, and gives a human the evidence when the automatic budget is exhausted.

Remembering findings is the first requirement

A second review should not behave as though the first review never happened.

The runner stores the original findings and carries them into the next attempt. If the target branch changes, history is rewritten, or the supplied context changes, it performs a fresh scope review without resetting the PR's overall attempt budget. Identical inputs reuse the most recent attempt rather than spending another model call on the same failure.

That produces a more useful question on the second pass: was this particular concern repaired? It is not just another open-ended request to inspect the whole pull request.

Three is a backstop, not a magic number

I chose three attempts as a practical safety mechanism, not because I have evidence that three reviews is the correct amount for every pull request.

A cap prevents an unattended agent from turning a small review into an unbounded process. But reaching the cap does not mean the code is unsafe, and staying below it does not mean the code is safe. It means automated review has reached the point where it should stop spending more model calls and preserve what it knows for a person.

The runner returns a distinct human-handoff result when the budget ends. The handoff includes the original findings, the latest review, and the evidence needed to see what changed. It does not approve, merge, push, or post on anyone's behalf.

The hard part is still scope

A useful comment on the project pointed out a limitation: attempt limits cannot decide whether a finding belongs in the current task. A reviewer needs some definition of what the PR is meant to change.

I do not want to answer that by creating another permanent process document or requiring a human approval step for every PR. My current direction is narrower:

derive a provisional scope from the PR description, linked issue, diff, and tests;
require each finding to point to the evidence that makes it relevant;
mark the review as under-specified when that evidence is missing, instead of silently expanding the task;
keep the attempt limit as a backstop for ambiguous or badly specified work.

That is deliberately not presented as solved. If a model writes the scope and then judges its own work against it, it can rationalize the very expansion the guard is supposed to prevent. The design question is how much independent evidence is enough before a finding stays in the loop.

Running it

The review runner is a separate optional skill. It runs from a trusted local maintainer host outside the repository being reviewed:

export REVIEW_CODEX_MODEL=gpt-6-astra
skills/frontier-simplify-review/scripts/review-pr.sh "$CONSUMER_REPO" "$PR_NUMBER" auto codex
skills/frontier-simplify-review/scripts/review-pr.sh "$CONSUMER_REPO" "$PR_NUMBER" status codex

It has a default 1,800-second executor timeout. A timeout, interrupt, or termination signal preserves the failed attempt instead of quietly losing it. A status check reads evidence without starting a model run.

The main point is not the number three. It is making the loop accountable: every repeated review should either resolve an earlier finding, produce new evidence, or stop and say why it cannot decide.

The source, install instructions, local self-tests, and raw benchmark limits are in MongLong0214/frontier-simplify.

DEV Community: Wonil Cho

An AI code reviewer that remembers its findings still needs a stop condition

Remembering findings is the first requirement

Three is a backstop, not a magic number

The hard part is still scope

Running it