Review Fatigue, Not Model Quality, Is Your Agent-PR Bottleneck

Agent-generated pull requests are cheap to produce and expensive to read. The scarce resource is not tokens, it is the twenty minutes a human spends deciding whether a diff is safe. Free model access lowers the cost of the second reader, but it does nothing about the first one.

So the useful question is not "which model reviews best?" It is "which parts of this diff deserve human attention at all?" Below is a two-stage gate that answers that with a deterministic pass first and a model pass second, plus a table for what to trust, test, or revert.

The attention budget is fixed; the diff volume is not

Every agent PR forces the same four costs, and only one of them is technical:

1. Re-reading code the agent restated from context.
2. Re-running checks that already passed on a different commit.
3. Deciding whether a passing test actually verifies the new behaviour.
4. Writing down why you approved it, so the next reviewer does not repeat the work.

Stage 1 kills costs 1 and 2 mechanically. Stage 2 attacks cost 3. Nothing automates cost 4, which is why the decision record matters more than the verdict.

Stage 1: deterministic triage before any model sees the diff

This pass uses git diff only. No network, no model, no judgement calls. Each rule maps to a bucket, not a score, because scores invite arguing with the tool instead of reading the code.

#!/usr/bin/env python3
"""gate.py - stage 1 triage for an agent-generated diff.

Usage: python gate.py <base_sha> > triage.json
Deterministic. No network. No model.
"""
import json, re, subprocess, sys

TEST_HINT = re.compile(r"(^|/)(tests?|__tests__|spec)/|(_test\.|\.test\.|_spec\.)")
LOCKFILES = {"package-lock.json", "poetry.lock", "pnpm-lock.yaml", "Cargo.lock", "uv.lock"}
MANIFESTS = {"package.json", "pyproject.toml", "Cargo.toml", "requirements.txt"}

def git(*args):
    return subprocess.run(("git",) + args, capture_output=True, text=True, check=True).stdout

def numstat(base):
    out = {}
    for line in git("diff", "--numstat", base).splitlines():
        add, rem, path = line.split("\t")
        out[path] = {"added": add, "removed": rem}
    return out

def removed(base, path):
    d = git("diff", "-U0", base, "--", path)
    return [l[1:] for l in d.splitlines() if l.startswith("-") and not l.startswith("---")]

def triage(base):
    findings, stats = [], numstat(base)
    lock = [p for p in stats if p.rsplit("/", 1)[-1] in LOCKFILES]
    man = [p for p in stats if p.rsplit("/", 1)[-1] in MANIFESTS]
    if lock and not man:
        findings.append({"rule": "lock-without-manifest", "bucket": "revert",
                         "why": "dependency tree moved without a declared change", "files": lock})
    for path in stats:
        if TEST_HINT.search(path):
            gone = [l for l in removed(base, path) if "assert" in l]
            if gone:
                findings.append({"rule": "assertion-removed", "bucket": "test",
                                 "why": "confirm assertions were replaced, not dropped",
                                 "files": [path], "sample": gone[:3]})
    wf = [p for p in stats if p.startswith(".github/workflows/")]
    if wf:
        body = "\n".join(git("diff", base, "--", p) for p in wf)
        if "permissions:" in body or "pull_request_target" in body:
            findings.append({"rule": "ci-privilege", "bucket": "revert",
                             "why": "workflow privilege changed inside a feature PR", "files": wf})
    return {"base": base, "files": len(stats), "findings": findings}

if __name__ == "__main__":
    print(json.dumps(triage(sys.argv[1]), indent=2))

Run it against the PR's merge base, not against main, or you will diff unrelated commits that landed since the branch was cut.

BASE=$(git merge-base origin/main HEAD)
python gate.py "$BASE" | tee triage.json

The output is intentionally boring: file count plus zero or more findings. A revert finding means stop — the diff is touching something outside the stated task. A test finding means the checks are suspect even when they are green.

Stage 2: a second reader with a rubric and a citation requirement

A model reviewer is only useful if you constrain what it can say. Three constraints do most of the work:

• Advisory only. It cannot approve. It can only add findings to the queue.
• Cite or drop. Every finding must name a file and a line that exists in the diff's added lines. Anything else is discarded.
• Cap the volume. Seven findings maximum. Beyond that, humans stop reading and start skimming.

The rubric is short on purpose. Long prompts produce hedged prose; short prompts produce claims you can check.

You are reviewing the added lines of a pull request written by an autonomous agent.
Report only defects that a human could confirm by reading the diff.
For each finding output: file, line, claim (one sentence), evidence (quoted added line).
Do not comment on style, naming, or test coverage.
Do not speculate about code you cannot see.
If you find nothing, output an empty list.

Filtering the response is where most implementations fail, because uncited findings read exactly like cited ones:

def changed_lines(base):
    lines, chunks = set(), git("diff", "-U0", base).split("diff --git ")[1:]
    for chunk in chunks:
        head, _, rest = chunk.partition("\n")
        path = head.split(" b/")[-1]
        for l in rest.splitlines():
            m = re.match(r"@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@", l)
            if m:
                start, count = int(m.group(1)), int(m.group(2) or 1)
                lines.update((path, n) for n in range(start, start + count))
    return lines

def keep_cited(findings, changed, cap=7):
    kept = [f for f in findings if (f.get("file"), f.get("line")) in changed]
    return kept[:cap], len(findings) - len(kept)

Log both numbers: kept and dropped. A rising drop rate means the rubric drifted, or the prompt is being fed files the diff never touched.

The decision table

Stage 1 signal	Stage 2 says	Action
No findings, one file, tests added	nothing	Trust, record the base SHA
No findings	cites a real added line	Test that line first, then decide
assertion-removed	nothing	Test; green checks are not evidence
lock-without-manifest	anything	Revert
ci-privilege	anything	Revert
Multiple files outside the stated task	anything	Revert and re-scope the prompt

The table is the deliverable. Tools change; the mapping from evidence to action should not.

Where free model access and a free server actually fit

Stage 1 runs anywhere. Stage 2 needs an endpoint, and that is the only place a hosted tier matters.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding agent project, and the operator states that its free tier includes model access (a 10-million-token allowance) and a free server option. Treat those as current, checkable terms rather than guarantees: allowances, model line-ups, and server availability change, so read the project's own page before you wire anything into CI. What is verifiable today is the shape of the integration — any OpenAI-compatible chat endpoint plugs into stage 2 behind a base URL you configure, which means your gate does not depend on this or any specific vendor.

A free server is useful for a narrower reason than it sounds: you want stage 1 running on every PR, on a machine that is not a reviewer's laptop. A rule that only runs when someone remembers to run it is a suggestion.

If you want to measure whether a second reader earns its place before paying for one, the free tier is the cheapest way to collect that data — check the terms first.

Limitations and who should skip this

• Low volume. Under roughly five agent PRs a week, the harness costs more attention than it saves. Read the diffs.
• Policy-constrained codebases. If sending source to a hosted model is prohibited, stage 2 is off the table. Stage 1 still works.
• Intent-shaped bugs. Neither stage knows what the task was supposed to be. A diff can be internally consistent and still solve the wrong problem.
• Rule drift. Every rule in stage 1 encodes a past incident. Review the rule set quarterly or it becomes folklore.
• Green is not proof. Stage 1 flags weakened tests; it cannot tell you the remaining test tests the new behaviour.

Four numbers to collect after two weeks

1. Findings accepted per PR.
2. Findings dropped for missing citations.
3. Median time from PR open to first decision.
4. Reverts traced to a revert bucket finding.

If (3) does not move, the gate is decoration. If (2) climbs, fix the prompt before adding rules.

The point of the two-stage gate is not to trust agents less. It is to spend human attention only on the diffs where a human reading actually changes the outcome.