DEV Community: Casey Chen

Plant Bugs Before You Trust the Reviewer: Calibrating a Cheap Model on Agent-Generated PRs

Casey Chen — Tue, 27 Oct 2026 13:37:51 +0000

A second model reading your agent's diff is not a reviewer until you have measured its miss rate. Everything below is the harness I use to decide whether a cheap — or free-tier — model earns a slot in the review queue, and what to do when it does not.

That distinction matters because the failure mode is invisible. A reviewer that catches one real bug is memorable. A reviewer that silently walks past five is just latency. You cannot observe false negatives in production, so you have to manufacture them.

The trap: "it looks competent" is not a measurement

Hand a 200-line agent diff to any mid-sized model and you will get something plausible back — structured findings, confident prose, a line number. Plausibility is cheap. What you actually need to know is:

Recall per defect class. Does it catch swallowed exceptions but miss weakened assertions?
False-positive rate on clean diffs. How many of its findings are noise you have to dismiss?
Stability. Same diff, three runs — same answer?

None of those are answerable from a demo. They are answerable from planted defects.

Step 1: Build a small calibration set

Six to ten diffs is enough. Pull them from merged agent PRs in your own repo, and require two properties:

They merged, so you are grading detection, not arbitration.
They are diverse — at least three different files and three different agent sessions.

Keep two clean diffs that never contained a defect. You need those for the false-positive count.

Step 2: Seed defects deterministically

Pick four classes you actually care about. Mine are error handling, test integrity, config defaults, and added validation. Then express each seed as a literal find/replace against a scratch checkout, so the whole set is reproducible and reviewable in a PR of its own.

# seed_defects.py — plant known defects into a scratch copy of a merged PR
import json, pathlib, shutil, subprocess, sys

SEEDS = json.loads(pathlib.Path("seeds.json").read_text())

def apply(repo: pathlib.Path, seed):
    p = repo / seed["file"]
    src = p.read_text()
    if seed["find"] not in src:
        # anchors drift. fail loudly instead of scoring against a phantom
        raise SystemExit(f"anchor missing for {seed['id']} — reseed this PR")
    p.write_text(src.replace(seed["find"], seed["replace"], 1))

def main(sha: str, workdir: str):
    out = pathlib.Path(workdir)
    shutil.rmtree(out, ignore_errors=True)
    subprocess.run(["git", "clone", "--quiet", ".", str(out)], check=True)
    subprocess.run(["git", "-C", str(out), "checkout", "--quiet", sha], check=True)
    for seed in SEEDS:
        apply(out, seed)
    truth = [{"id": s["id"], "class": s["class"], "file": s["file"],
              "anchor": s["replace"].splitlines()[0].strip()} for s in SEEDS]
    # kept outside the scratch checkout; the reviewer never sees it
    pathlib.Path(workdir + "-truth.json").write_text(json.dumps(truth, indent=2))
    print(f"seeded {len(truth)} defects into {out}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

A seeds.json entry is three lines of JSON:

{ "id": "swallow-1", "class": "error-handling", "file": "app/client.py",
  "find": "    except TimeoutError:\n        raise",
  "replace": "    except TimeoutError:\n        pass" }

Line numbers are derived at scoring time from the anchor, not hard-coded — a rebase should not invalidate your calibration set.

Step 3: Give the reviewer a contract, not a conversation

Free-form review output cannot be scored. Constrain it to JSON with a fixed class vocabulary, and pass the diff, not the whole repo:

# reviewer.py — strict-JSON second reviewer over a diff
import json, os, subprocess, urllib.request

BASE  = os.environ["REVIEW_BASE_URL"]   # OpenAI-compatible endpoint
MODEL = os.environ["REVIEW_MODEL"]      # record this in the results file

CONTRACT = (
    "Review this pull request diff. Report only changes you believe are defective.\n"
    'Return JSON: {"findings":[{"file":str,"line":int,'
    '"class":"error-handling|test-integrity|config-default|validation","why":str}]}\n'
    "An empty list is a valid answer. No prose, no markdown."
)

def ask(diff_text: str):
    payload = json.dumps({
        "model": MODEL, "temperature": 0,
        "messages": [{"role": "system", "content": CONTRACT},
                     {"role": "user", "content": diff_text[:12000]}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ.get('REVIEW_API_KEY','unused')}"})
    with urllib.request.urlopen(req, timeout=120) as r:
        content = json.loads(r.read())["choices"][0]["message"]["content"]
    return json.loads(content)["findings"]

Two details that matter more than they look. temperature: 0 is not determinism, so run each diff three times and score the median. And truncating at 12k characters is a real limitation, not a bug — it tells you the harness only measures what fits.

Step 4: Score recall by class, then count noise

# score.py — per-class recall against planted defects
def score(findings, truth, tol=3):
    hit = {t["id"] for t in truth
           if any(f["file"] == t["file"] and abs(f["line"] - t["line"]) <= tol
                  for f in findings)}
    per_class = {}
    for t in truth:
        per_class.setdefault(t["class"], []).append(t["id"] in hit)
    recall = {c: sum(v) / len(v) for c, v in per_class.items()}
    return recall, len(hit) / len(truth)

A ±3-line tolerance is deliberate: the reviewer is allowed to point at the right neighbourhood. You are not grading formatting. You are grading whether it noticed.

Then run the same script on your two clean diffs and count findings. That number — findings per clean PR — is your review tax. Anything above two per PR and you are doing more arbitration work than you saved.

Step 5: Turn the numbers into a gate, not a vibe

Recall on error-handling and config-default	Findings per clean PR	Verdict
≥ 0.8	≤ 1	Trust as triage. Its findings choose which tests you write; a human still signs off.
0.5 – 0.8	≤ 2	Test-only. Convert findings into failing tests, ignore its verdicts entirely.
< 0.5, or any class at 0	any	Human-only. The queue is cheaper without it.

Fill in your own numbers. A recall figure from my repository tells you nothing about yours — different defect mix, different language, different prompt.

Where free model access actually changes this loop

This harness is call-hungry: 8 diffs × 3 repeats × 3 prompt variants is 72 requests before you have changed a single line of the contract. That iteration cost, not the review cost, is what stops most teams from calibrating anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option (both operator-supplied availability claims; quotas and capacity change, so confirm the current terms on the project itself rather than trusting this paragraph). In practice that removes two specific frictions here: the harness can be run to completion without metering calls mid-iteration, and the free server option means the seeding and scoring scripts run from a hosted box instead of tying up a laptop for an hour.

One rule I would keep regardless of price: the endpoint name is not a model version. Write REVIEW_MODEL and the endpoint's reported version into your results file. When either changes, recall can move and your gate is stale until you re-run step 4.

Limitations, stated plainly

The harness measures detection of seeded classes, not novel bugs. A model can score 0.9 here and still be useless on a race condition.
Small models degrade on cross-file invariants. Any diff touching more than about eight files is out of scope for this loop.
Truncated diffs produce silent false negatives, which is the worst kind of number to put in a table.
Three runs and a median is statistics-flavoured, not statistics.

Who should skip this

If you see fewer than ten agent-generated PRs a week, human review is cheaper than maintaining a seed set — and a stale seed set is worse than none, because it produces confident numbers about code you no longer write. Skip it too if the PR needs an approval stamp for compliance: this harness never approves anything, and no calibration result should be paraphrased into one.

If you want to try the five steps before committing, the free model access and free server option are enough to run the whole loop end to end — start with the clean diffs and the false-positive count, because that is the number that decides whether the rest is worth your afternoon.

A Claim Ledger for Agent PRs: What to Trust, What to Test, What to Revert

Casey Chen — Mon, 26 Oct 2026 12:14:09 +0000

Liquid syntax error: Unknown tag 'endraw'

Review the Contract, Not the Diff: Auditing Agent-Generated PRs Hunk by Hunk

Casey Chen — Tue, 20 Oct 2026 10:16:59 +0000

Most agent-authored PRs are not hard to read. They are hard to bound. The code is usually fine line by line; what you cannot see is the scope the agent silently negotiated with itself while writing it. So you spend forty minutes reconstructing intent that was never written down, then approve because the tests are green.

There is a cheaper order of operations. Write the scope down before the agent runs, commit it next to the task, and let a deterministic script tell you which hunks deserve your attention. The script does not judge correctness. It tells you where correctness is even worth arguing about.

The failure mode: silent scope expansion

A typical agent PR says "handle 429 responses without infinite retry." The diff touches:

src/http/client.py — expected, that is the task.
src/config/defaults.py — not expected. A default changed from 3 attempts to 1.
tests/http/test_client.py — six assertions replaced by one assert response is not None.

No single line is obviously wrong. That is the problem. You are being asked to review a decision that was never argued for, mixed into a diff where most hunks are mechanical, and the review UI renders all of it with the same visual weight.

Reviewers then pick one of two bad strategies: read every hunk with equal attention (slow, burns out by the third PR of the day), or skim and trust the tests (fast, and exactly how assertion weakening gets merged). Both are workarounds for missing metadata.

Step 1: commit the scope contract

scope.json lives next to the task description, in the same PR, before any code exists.

{
  "task": "handle 429 from provider without infinite retry",
  "allowed_paths": ["src/http/*", "tests/http/*"],
  "forbidden_paths": ["src/config/*", ".github/*", "pyproject.toml"],
  "allow_delete": false,
  "max_files": 8,
  "max_changed_lines": 250,
  "human_required_if": ["out_of_scope", "deps", "ci", "skip_added", "assertion_weakened"]
}

Three rules make this file useful rather than decorative:

Deny wins over allow. src/http/* allowed and src/config/* forbidden means a config edit is a finding, not a judgment call.
Budget the size, not just the paths. An agent that needs 40 files for a 5-file task is telling you something about the task, not about its speed.
allow_delete: false by default. Deletions are the cheapest way for any diff — human or generated — to make failing tests pass.

Step 2: generate a review packet

The packet script is intentionally boring: stdlib only, reads git, no network, no model calls. Determinism is the point. A classifier that changes its answer between runs cannot gate a merge.

#!/usr/bin/env python3
"""review_packet.py - classify an agent PR diff against scope.json.

Exit 0 = no human-required signal. Exit 1 = human review required.
Globbing is fnmatch-simple; see limitations below.
"""
import argparse, fnmatch, json, re, subprocess, sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@ ?(.*)$")
MECHANICAL = re.compile(r"^\s*(?:import\s|from\s+\S+\s+import\s|#|//|/\*|\*|$)")
SKIP_ADDED = re.compile(r"(?:pytest\.mark\.skip|\.skip\(|xit\(|xdescribe\(|t\.Skip\()")
WEAKENED = re.compile(r"(?:assert\s+True|assert\s+[\w.]+\s+is\s+not\s+None|except\s+Exception\s*:\s*pass)")

RISK = {
    "deps":  ["requirements*.txt", "pyproject.toml", "package.json", "package-lock.json", "go.mod"],
    "ci":    [".github/workflows/*", ".gitlab-ci.yml", "Dockerfile*", "Makefile"],
    "db":    ["*/migrations/*", "*.sql"],
    "infra": ["terraform/*", "*.tf", "helm/*"],
}
TESTS = ["tests/*", "*/tests/*", "test_*.py", "*_test.go", "*.test.ts", "*.spec.ts"]


def git(*args):
    return subprocess.run(("git",) + args, capture_output=True, text=True, check=True).stdout


def matches(path, globs):
    return any(fnmatch.fnmatch(path, g) for g in globs)


def files_changed(base):
    out = git("diff", "--numstat", "--no-renames", f"{base}...HEAD")
    rows = []
    for line in out.splitlines():
        add, dele, path = line.split("\t")
        rows.append({"path": path, "add": int(add or 0), "del": int(dele or 0)})
    return rows


def hunks(base, path):
    out = git("diff", "--no-color", "-U0", "--no-renames", f"{base}...HEAD", "--", path)
    start, header, body = None, "", []
    for line in out.splitlines():
        m = HUNK.match(line)
        if m:
            if start is not None:
                yield {"line": start, "header": header, "body": body}
            start, header, body = int(m.group(1)), m.group(3), []
        elif start is not None and line[:1] in "+-" and not line.startswith(("+++", "---")):
            body.append(line)
    if start is not None:
        yield {"line": start, "header": header, "body": body}


def label(path, hunk, contract):
    added = [l[1:] for l in hunk["body"] if l.startswith("+")]
    text = "\n".join(l[1:] for l in hunk["body"])
    if matches(path, contract["forbidden_paths"]):
        return "out_of_scope"
    for kind, globs in RISK.items():
        if matches(path, globs):
            return kind
    if any(SKIP_ADDED.search(l) for l in added):
        return "skip_added"
    if WEAKENED.search(text):
        return "assertion_weakened"
    if not matches(path, contract["allowed_paths"]):
        return "out_of_scope"
    if all(MECHANICAL.match(l) for l in added):
        return "mechanical"
    return "semantic"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--contract", default="scope.json")
    ap.add_argument("--format", choices=["json", "markdown"], default="markdown")
    args = ap.parse_args()

    contract = json.load(open(args.contract))
    rows = files_changed(args.base)
    gate = set(contract["human_required_if"])

    findings, changed = [], sum(r["add"] + r["del"] for r in rows)
    for f in rows:
        for h in hunks(args.base, f["path"]):
            findings.append({"kind": label(f["path"], h, contract), "path": f["path"],
                             "line": h["line"], "header": h["header"]})

    human = [f for f in findings if f["kind"] in gate]
    if len(rows) > contract["max_files"] or changed > contract["max_changed_lines"]:
        human.append({"kind": "over_budget", "path": "-", "line": 0,
                      "header": f"{len(rows)} files, {changed} lines"})

    if args.format == "json":
        print(json.dumps({"task": contract["task"], "human_required": human,
                          "all": findings}, indent=2))
    else:
        print(f"# Review packet - {contract['task']}")
        print(f"changed: {len(rows)} files, {changed} lines\n")
        print(f"## human required ({len(human)})")
        for f in human:
            print(f"- [{f['kind']}] {f['path']}:{f['line']}  {f['header']}")
        print("\n## mechanical, in scope")
        for f in findings:
            if f["kind"] == "mechanical":
                print(f"- {f['path']}:{f['line']}")

    return 1 if human else 0


if __name__ == "__main__":
    sys.exit(main())

Run it against the PR branch, not your working tree:

python scripts/review_packet.py --base origin/main --contract scope.json > packet.md
echo "$?"   # 1 means: a human has to open this, and the packet says where

Output for the 429 example, trimmed:

# Review packet - handle 429 from provider without infinite retry
changed: 4 files, 75 lines

## human required (3)
- [out_of_scope] src/config/defaults.py:12  -retries = 3 +retries = 1
- [assertion_weakened] tests/http/test_client.py:88  def test_retry_budget()
- [skip_added] tests/http/test_client.py:140  def test_gives_up_after_limits()

## mechanical, in scope
- src/http/client.py:3

What the packet lets you skip

mechanical is the only label that earns a fast pass. Import churn, comment-only edits, formatting inside allowed paths. Skim it, run the suite, move on. Everything else gets read.

semantic inside scope is where the real work sits: control flow, error handling, new defaults, changed numeric literals. These hunks need the new test to fail on the base commit before you believe it. There is no packet signal that substitutes for that check.

What the packet forces you to test

Packet signal	First-read trust	Minimum evidence	Default action
`mechanical`, in scope	yes	CI green	merge
`semantic`, in scope, real tests added	partial	new test fails on base	review the hunk
`assertion_weakened`	no	stash test change, suite result	send test change back
`skip_added`	no	reason for the skip, in writing	send back
`out_of_scope`	no	—	re-scope the PR or split it
`deps` / `ci` / `db` / `infra`	no	lockfile diff, clean-cache build	separate PR
`over_budget`	no	explanation of the size	split

The right-hand column is deliberately mechanical. It exists so you do not have to re-derive a policy at 6pm with eleven tabs open.

Where a model actually helps — and where it must not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The classifier above labels shape: which file, which hunk, which risk class. It cannot say why src/config/defaults.py was touched. That is a job for a language model, and it is a narrow job: draft one sentence per out-of-scope hunk explaining the likely reason, so a human can accept or reject the explanation in seconds instead of re-reading the diff.

That note goes in a notes field. It never touches kind, never flips the exit code, and never suppresses a human required row. Practically: run the deterministic packet first, then call a model with the packet plus the out-of-scope hunks only. You get the cheap tokens spent on the ambiguous 5% instead of on a whole-diff summary nobody reads.

This is where free model access matters for the workflow. A gate that costs money per invocation gets disabled the week someone notices the bill; a gate that runs on a free tier keeps running. The operator states the current offer includes 10M free tokens and a free server option — treat that as a snapshot of a promotional offer, not a permanent SLA, and check the live terms before you make a pipeline depend on it. If you want to host the packet endpoint as an internal webhook instead of a laptop script, the payload is tiny (paths, line counts, hunk headers), which is exactly the shape a small free server can serve.

One hard constraint: do not paste diffs from private repositories into any hosted endpoint without reading its retention terms. Running the model call locally against free access keeps the diff on your machine. That is the configuration I would pick first, and the one I would keep as the default.

Limitations, including cases where this is the wrong tool

The hunk classifier is syntactic. Whole-file renames, vendored code, generated code, and reformat-only commits will be mislabeled. If your PRs routinely include git mv plus edits, --no-renames will report deletions you did not make.
Globbing is fnmatch, not gitignore. **/migrations/* does not match nested paths. Replace matches() with a pathspec-based check if your tree is deep.
It only works if the contract comes first. Retrofitting scope.json onto a diff that already exists turns the gate into rationalization: everything gets declared in scope, and the packet politely agrees.
In-scope but wrong is invisible. A retry loop inside src/http/client.py is correctly labeled semantic and still loops forever. The packet routes attention; it does not supply judgment.
Skip this if you have no CI runner that can execute git against the PR base, or your agent already writes directly to a protected branch. In that case the missing control is branch protection, not a review script. And if no one on the team will read a packet, adding one is theater with extra steps.

Monday, in order

Write scope.json for the next agent task before the agent starts. Keep it under ten lines.
Run the packet against your last five merged agent PRs. Look only at the human required section. When something there was merged without discussion, that is your baseline risk.
Add one CI job that fails closed on exit code 1. Never make it advisory on day one; advisory gates decay into comments.
Only then add model notes for ambiguous hunks — after the deterministic part has been trusted for a week.

The review time you save is real, but the durable win is different: every merged agent PR now carries a written statement of what it was allowed to touch. The next reviewer does not have to guess, and neither does the audit.

If you want to try this end to end, copy review_packet.py and scope.json first — they work with no account and no network. Add the model notes when the packet has already taught you which hunks are worth a second pair of eyes.

Route the Diff Before You Read It: A File-Level Triage Script for Agent-Generated PRs

Casey Chen — Mon, 19 Oct 2026 08:57:48 +0000

Most agent-generated PRs are not reviewed badly because reviewers are lazy. They are reviewed badly because reviewers read in the wrong order: the summary first, then the hunks top to bottom, and the manifest or CI change at the bottom of the diff where attention is already gone.

Fix the order and the review gets cheaper. The file list is a smaller, more truthful artifact than the summary the agent wrote about itself, so route the review from the file list.

The failure surface is in the file list

In my experience triaging agent PRs, the risky changes cluster in a handful of paths, and they are almost never the ones the agent highlights:

Lockfiles and build output edited directly, because the agent "fixed" a resolution error by hand.
CI workflow files, where a permission, trigger, or secret reference changed by one line.
Config and env defaults, where a new key silently changes runtime behavior.
The tests themselves, where the oracle moved to meet the code.
Ordinary source, which is usually the safest part and the part reviewers spend 80% of their time on.

That last bullet is the whole problem. Review effort is inversely correlated with risk.

A triage script you can run in one command

The script below reads git diff --name-status between two refs, matches each path against an ordered rule list, and prints a Markdown table with a bucket and the evidence the PR must carry. First match wins, so specific patterns go above general ones.

#!/usr/bin/env python3
"""triage.py - bucket every file in a PR diff before anyone reads a hunk.

    python triage.py origin/main HEAD
    python triage.py origin/main HEAD --fail-on REVERT
"""
import argparse
import fnmatch
import subprocess
import sys

# glob, bucket, reason, evidence the PR must carry
RULES = [
    ("package-lock.json",     "REVERT", "lockfile edited by hand",      "matching package.json diff"),
    ("*.lock",                "REVERT", "lockfile edited by hand",      "matching manifest diff"),
    ("*.min.js",              "REVERT", "build output committed",       "source file + build command"),
    ("dist/*",                "REVERT", "build output committed",       "source file + build command"),
    (".github/workflows/*",   "TEST",   "permissions/triggers/secrets", "before and after permissions block"),
    ("Dockerfile",            "TEST",   "base image and entrypoint",    "built image + start command"),
    ("*.env*",                "TEST",   "config surface",               "every key added, with default"),
    ("*migrations/*",         "TEST",   "irreversible state change",    "forward run and rollback run"),
    ("test_*.py",             "TEST",   "the oracle itself changed",    "test fails on the parent commit"),
    ("tests/*",               "TEST",   "the oracle itself changed",    "test fails on the parent commit"),
    ("*.py",                  "TRUST",  "ordinary source",              "existing unit tests still green"),
    ("*.ts",                  "TRUST",  "ordinary source",              "existing unit tests still green"),
]

LARGE_DIFF_LINES = 400


def changed(base, head):
    out = subprocess.run(
        ["git", "diff", "--name-status", "-M", f"{base}...{head}"],
        capture_output=True, text=True, check=True,
    ).stdout
    rows = []
    for line in out.splitlines():
        parts = line.split("\t")
        if len(parts) >= 2:
            rows.append((parts[0], parts[-1]))
    return rows


def numstat(base, head):
    out = subprocess.run(
        ["git", "diff", "--numstat", f"{base}...{head}"],
        capture_output=True, text=True, check=True,
    ).stdout
    counts = {}
    for line in out.splitlines():
        added, removed, path = line.split("\t")
        if added.isdigit() and removed.isdigit():
            counts[path] = int(added) + int(removed)
    return counts


def classify(path, size):
    for pattern, bucket, reason, evidence in RULES:
        if fnmatch.fnmatch(path, pattern) or fnmatch.fnmatch(path.rsplit("/", 1)[-1], pattern):
            if size > LARGE_DIFF_LINES and bucket == "TRUST":
                return "TEST", f"{reason}; {size} changed lines", "read the hunks in order"
            return bucket, reason, evidence
    return "TRUST", "no rule matched", "existing tests still green"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("base")
    ap.add_argument("head")
    ap.add_argument("--fail-on", choices=["REVERT", "TEST"], default=None)
    args = ap.parse_args()

    sizes = numstat(args.base, args.head)
    order = {"REVERT": 0, "TEST": 1, "TRUST": 2}
    rows = []
    for status, path in changed(args.base, args.head):
        bucket, reason, evidence = classify(path, sizes.get(path, 0))
        rows.append((order[bucket], bucket, status, path, reason, evidence))
    rows.sort()

    print("| bucket | status | path | why | evidence required |")
    print("|---|---|---|---|---|")
    for _, bucket, status, path, reason, evidence in rows:
        print(f"| {bucket} | {status} | `{path}` | {reason} | {evidence} |")

    counts = {b: sum(1 for r in rows if r[1] == b) for b in order}
    print(f"\nREVERT={counts['REVERT']} TEST={counts['TEST']} TRUST={counts['TRUST']}")

    if args.fail_on and counts[args.fail_on]:
        sys.exit(1)


if __name__ == "__main__":
    main()

Run it against the PR head:

git fetch origin pull/4123/head:pr-4123
python triage.py origin/main pr-4123 --fail-on REVERT

Example output shape (illustrative, not a measurement from a specific repo):

| bucket | status | path                       | why                        | evidence required              |
| REVERT | M      | package-lock.json          | lockfile edited by hand    | matching package.json diff     |
| TEST   | M      | .github/workflows/test.yml | permissions/triggers       | before and after permissions   |
| TEST   | M      | tests/test_billing.py      | the oracle itself changed  | test fails on the parent commit|
| TRUST  | M      | src/billing/retry.py       | ordinary source            | existing unit tests green      |

REVERT=1 TEST=2 TRUST=1

The decision table

Bucket	What to trust	What to test	Revert trigger
REVERT	Nothing. Treat the file as unreviewable until a generator produced it.	Regenerate from the manifest (`npm ci`, `poetry lock --no-update`) and diff the result against the PR.	Any hand-edited lockfile, committed build output, or vendored blob with no source.
TEST	The intent, not the effect.	Run the thing the file controls: CI on a throwaway branch, the container locally, the migration forward and back.	A permission widened, a trigger added on `pull_request_target`, a new env key with a hardcoded default, a test weakened in the same commit as the fix.
TRUST	The shape of the change, subject to green tests.	Existing suite plus one new assertion that would fail on the parent commit.	Logic that only passes because a TEST-bucket file changed.

Reading order: REVERT, then TEST, then TRUST

Clear REVERT first. These are five-minute decisions with binary outcomes. A hand-edited lockfile is not a discussion; ask for a regenerate commit.
Spend the real time on TEST. This is where agent PRs actually break: one line in a workflow file, one default in a config loader.
Skim TRUST. If the tests are green and the file is ordinary application code, the marginal value of a line-by-line read is low. Read the function signatures and the deleted lines, not every hunk.

This inverts the default behavior, where reviewers start at the top of a diff that is usually sorted alphabetically and exhaust themselves on src/ before reaching .github/.

Three checks before you read any hunk text

BASE=origin/main; HEAD=pr-4123

# 1. Lockfile moved without the manifest, or the reverse
 git diff --name-only $BASE...$HEAD -- 'package.json' 'package-lock.json' 'pyproject.toml' 'poetry.lock'

# 2. CI permission and trigger delta
git diff $BASE...$HEAD -- .github/workflows/ | grep -E '^[+-].*(permissions|pull_request_target|secrets\.|runs-on)'

# 3. Config and credential surface
git diff $BASE...$HEAD -- '*.env*' '*.yaml' '*.yml' | grep -E '^\+.*(KEY|TOKEN|SECRET|PASSWORD)'

If check 2 prints a line starting with +, stop and read that hunk before anything else. A single added permissions: write-all outranks every code improvement in the PR.

Where free model access and a free server actually fit

Triage is deterministic work, so it is a poor place to spend a paid model call. It is a reasonable place to spend a free one.

Two narrow uses, both of which keep the model out of the approval path:

Drafting review comments from the bucket table only. Pass the script's output — paths, buckets, reasons, no source code — and ask for one comment per REVERT entry. The model never sees the diff, so it cannot hallucinate an approval.
Running the script itself. Triage is a cron job or a webhook handler, not an interactive tool. That is exactly the workload a small always-on box handles well.

The MonkeyCode open-source project advertises free model access and a free server option. At the time of writing the operator states the free allotment is 10M tokens plus a free server; those terms are time-sensitive, so confirm them in the repository before you design a pipeline around them. For this workflow the free model is enough because the model is only formatting a table that triage.py already produced. It is not asked to judge the PR.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A minimal webhook handler that runs the script and posts the table as a PR comment is about 30 lines; the interesting part is the shell call, not the transport:

python triage.py "$BASE" "$HEAD" --fail-on REVERT > triage.md || echo "BLOCKED" >> triage.md
gh pr comment "$PR" --body-file triage.md

Fail the check on REVERT and the PR cannot merge until a human regenerates the lockfile. That is a review gate, not a model judgment.

Limitations and who should skip this

Generated-code-heavy repos. If half the tree is codegen output, the rule list fills with noise and every PR trips REVERT. Match on the generator config instead of the output paths.
Monorepos without ownership boundaries. The script has no idea which team owns which path. Add CODEOWNERS parsing before you trust the TRUST bucket.
Teams without a green test suite. The TRUST bucket assumes tests are a usable oracle. If they are not, everything is TEST and the triage buys you nothing.
PRs that are genuinely architectural. A 40-file refactor does not need bucketing; it needs a design review with a human in the room.

If you want to try the free-model and free-server path, start by running triage.py on your last ten merged agent PRs and see whether the REVERT bucket would have caught anything. That answer is more useful than any review checklist, including this one.

Review Fatigue, Not Model Quality, Is Your Agent-PR Bottleneck

Casey Chen — Sun, 18 Oct 2026 07:34:58 +0000

Review Fatigue, Not Model Quality, Is Your Agent-PR Bottleneck

Agent-generated pull requests are cheap to produce and expensive to read. The scarce resource is not tokens, it is the twenty minutes a human spends deciding whether a diff is safe. Free model access lowers the cost of the second reader, but it does nothing about the first one.

So the useful question is not "which model reviews best?" It is "which parts of this diff deserve human attention at all?" Below is a two-stage gate that answers that with a deterministic pass first and a model pass second, plus a table for what to trust, test, or revert.

The attention budget is fixed; the diff volume is not

Every agent PR forces the same four costs, and only one of them is technical:

Re-reading code the agent restated from context.
Re-running checks that already passed on a different commit.
Deciding whether a passing test actually verifies the new behaviour.
Writing down why you approved it, so the next reviewer does not repeat the work.

Stage 1 kills costs 1 and 2 mechanically. Stage 2 attacks cost 3. Nothing automates cost 4, which is why the decision record matters more than the verdict.

Stage 1: deterministic triage before any model sees the diff

This pass uses git diff only. No network, no model, no judgement calls. Each rule maps to a bucket, not a score, because scores invite arguing with the tool instead of reading the code.

#!/usr/bin/env python3
"""gate.py - stage 1 triage for an agent-generated diff.

Usage: python gate.py <base_sha> > triage.json
Deterministic. No network. No model.
"""
import json, re, subprocess, sys

TEST_HINT = re.compile(r"(^|/)(tests?|__tests__|spec)/|(_test\.|\.test\.|_spec\.)")
LOCKFILES = {"package-lock.json", "poetry.lock", "pnpm-lock.yaml", "Cargo.lock", "uv.lock"}
MANIFESTS = {"package.json", "pyproject.toml", "Cargo.toml", "requirements.txt"}

def git(*args):
    return subprocess.run(("git",) + args, capture_output=True, text=True, check=True).stdout

def numstat(base):
    out = {}
    for line in git("diff", "--numstat", base).splitlines():
        add, rem, path = line.split("\t")
        out[path] = {"added": add, "removed": rem}
    return out

def removed(base, path):
    d = git("diff", "-U0", base, "--", path)
    return [l[1:] for l in d.splitlines() if l.startswith("-") and not l.startswith("---")]

def triage(base):
    findings, stats = [], numstat(base)
    lock = [p for p in stats if p.rsplit("/", 1)[-1] in LOCKFILES]
    man = [p for p in stats if p.rsplit("/", 1)[-1] in MANIFESTS]
    if lock and not man:
        findings.append({"rule": "lock-without-manifest", "bucket": "revert",
                         "why": "dependency tree moved without a declared change", "files": lock})
    for path in stats:
        if TEST_HINT.search(path):
            gone = [l for l in removed(base, path) if "assert" in l]
            if gone:
                findings.append({"rule": "assertion-removed", "bucket": "test",
                                 "why": "confirm assertions were replaced, not dropped",
                                 "files": [path], "sample": gone[:3]})
    wf = [p for p in stats if p.startswith(".github/workflows/")]
    if wf:
        body = "\n".join(git("diff", base, "--", p) for p in wf)
        if "permissions:" in body or "pull_request_target" in body:
            findings.append({"rule": "ci-privilege", "bucket": "revert",
                             "why": "workflow privilege changed inside a feature PR", "files": wf})
    return {"base": base, "files": len(stats), "findings": findings}

if __name__ == "__main__":
    print(json.dumps(triage(sys.argv[1]), indent=2))

Run it against the PR's merge base, not against main, or you will diff unrelated commits that landed since the branch was cut.

BASE=$(git merge-base origin/main HEAD)
python gate.py "$BASE" | tee triage.json

The output is intentionally boring: file count plus zero or more findings. A revert finding means stop — the diff is touching something outside the stated task. A test finding means the checks are suspect even when they are green.

Stage 2: a second reader with a rubric and a citation requirement

A model reviewer is only useful if you constrain what it can say. Three constraints do most of the work:

Advisory only. It cannot approve. It can only add findings to the queue.
Cite or drop. Every finding must name a file and a line that exists in the diff's added lines. Anything else is discarded.
Cap the volume. Seven findings maximum. Beyond that, humans stop reading and start skimming.

The rubric is short on purpose. Long prompts produce hedged prose; short prompts produce claims you can check.

You are reviewing the added lines of a pull request written by an autonomous agent.
Report only defects that a human could confirm by reading the diff.
For each finding output: file, line, claim (one sentence), evidence (quoted added line).
Do not comment on style, naming, or test coverage.
Do not speculate about code you cannot see.
If you find nothing, output an empty list.

Filtering the response is where most implementations fail, because uncited findings read exactly like cited ones:

def changed_lines(base):
    lines, chunks = set(), git("diff", "-U0", base).split("diff --git ")[1:]
    for chunk in chunks:
        head, _, rest = chunk.partition("\n")
        path = head.split(" b/")[-1]
        for l in rest.splitlines():
            m = re.match(r"@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@", l)
            if m:
                start, count = int(m.group(1)), int(m.group(2) or 1)
                lines.update((path, n) for n in range(start, start + count))
    return lines

def keep_cited(findings, changed, cap=7):
    kept = [f for f in findings if (f.get("file"), f.get("line")) in changed]
    return kept[:cap], len(findings) - len(kept)

Log both numbers: kept and dropped. A rising drop rate means the rubric drifted, or the prompt is being fed files the diff never touched.

The decision table

Stage 1 signal	Stage 2 says	Action
No findings, one file, tests added	nothing	Trust, record the base SHA
No findings	cites a real added line	Test that line first, then decide
`assertion-removed`	nothing	Test; green checks are not evidence
`lock-without-manifest`	anything	Revert
`ci-privilege`	anything	Revert
Multiple files outside the stated task	anything	Revert and re-scope the prompt

The table is the deliverable. Tools change; the mapping from evidence to action should not.

Where free model access and a free server actually fit

Stage 1 runs anywhere. Stage 2 needs an endpoint, and that is the only place a hosted tier matters.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding agent project, and the operator states that its free tier includes model access (a 10-million-token allowance) and a free server option. Treat those as current, checkable terms rather than guarantees: allowances, model line-ups, and server availability change, so read the project's own page before you wire anything into CI. What is verifiable today is the shape of the integration — any OpenAI-compatible chat endpoint plugs into stage 2 behind a base URL you configure, which means your gate does not depend on this or any specific vendor.

A free server is useful for a narrower reason than it sounds: you want stage 1 running on every PR, on a machine that is not a reviewer's laptop. A rule that only runs when someone remembers to run it is a suggestion.

If you want to measure whether a second reader earns its place before paying for one, the free tier is the cheapest way to collect that data — check the terms first.

Limitations and who should skip this

Low volume. Under roughly five agent PRs a week, the harness costs more attention than it saves. Read the diffs.
Policy-constrained codebases. If sending source to a hosted model is prohibited, stage 2 is off the table. Stage 1 still works.
Intent-shaped bugs. Neither stage knows what the task was supposed to be. A diff can be internally consistent and still solve the wrong problem.
Rule drift. Every rule in stage 1 encodes a past incident. Review the rule set quarterly or it becomes folklore.
Green is not proof. Stage 1 flags weakened tests; it cannot tell you the remaining test tests the new behaviour.

Four numbers to collect after two weeks

Findings accepted per PR.
Findings dropped for missing citations.
Median time from PR open to first decision.
Reverts traced to a revert bucket finding.

If (3) does not move, the gate is decoration. If (2) climbs, fix the prompt before adding rules.

The point of the two-stage gate is not to trust agents less. It is to spend human attention only on the diffs where a human reading actually changes the outcome.

Rehearse the Revert Before You Merge: A Review Gate for Agent-Generated PRs

Casey Chen — Fri, 16 Oct 2026 05:39:10 +0000

An agent opened a pull request. CI is green, the diff is 400 lines of plausible code, and your calendar says you have fifteen minutes. The instinct is to read it top to bottom and ask whether it is correct.

That is the second most useful question. The first one is cheaper: if this is wrong at 02:00, what does undo cost?

This week's front page had several posts about AI tooling improving while the surrounding software did not. Review capacity is part of that surrounding software. It did not grow. The count of machine-written diffs per reviewer did. A process that only measures correctness will fall behind quietly, and the failure mode is not a bad merge — it is a merge nobody can back out.

Revert cost is a review metric, not an incident detail

Revert cost has three parts, and only the first is about git.

Structural cost: after main has moved, does reverting this commit apply cleanly, or does it collide with everything that landed since?
State cost: what did the change write outside the repository? Rows, objects, published artifacts, webhooks already delivered, caches keyed on the old shape.
Blast cost: how many callers depend on the new behavior, and did any of them ship in the same week?

Add those three and you have a revert budget. The rule I use: if the revert budget is larger than the cost of fixing forward, the PR must ship with its own undo, written and tested, in the same diff. Not a plan in a comment. A command someone can run at 02:00.

Assume the test-diff and retry-policy gates already ran. This is the gate that runs after them.

Three buckets for agent diffs

Merge-now

Pure additions: new function, new tests, no writes outside the process, no new dependency. If the classifier below files the diff here and the tests fail before the change and pass after, there is nothing left to rehearse.

Rehearse-first

Additive but stateful: schema changes, lockfile bumps, CI workflow edits, environment or secret surface, feature-flag defaults. These usually merge, but only after someone has run the undo once. The rehearsal is the review.

Human-only

Destructive or externally visible: backfills, DELETE or DROP, anything that publishes to a registry, sends a webhook, or changes billing and permissions. Agent-generated diffs land here more often than their tone suggests, because models write down-migrations far less reliably than up-migrations. Do not merge these on a green pipeline alone.

The rehearsal loop

Two rehearsals, both cheap. The first models git. The second models state.

Step 1: structural rehearsal

Does the PR still revert after the rest of the team merged their work? Replay the PR on its merge base, replay what landed since, then try to undo just the PR.

#!/usr/bin/env bash
# revert-rehearsal.sh - does this PR still undo cleanly after main moved?
set -euo pipefail

PR_COMMIT="${1:?usage: revert-rehearsal.sh <pr-commit> <merge-base>}"
BASE="${2:?usage: revert-rehearsal.sh <pr-commit> <merge-base>}"
SCRATCH="$(mktemp -d)"

git worktree add --detach "$SCRATCH" "$BASE"
cd "$SCRATCH"

git cherry-pick -x "$PR_COMMIT"
PR_REPLAY="$(git rev-parse HEAD)"

# replay whatever landed on main after the PR branch point
git cherry-pick -x "$BASE"..origin/main

if git revert --no-commit "$PR_REPLAY"; then
  echo "STRUCTURAL: clean revert"
  git revert --quit
else
  echo "STRUCTURAL: conflict, the undo needs a human"
  git revert --abort
fi

cd - >/dev/null
git worktree remove --force "$SCRATCH"

Two outputs matter. A conflict during the second cherry-pick means the PR is stale against main, which is its own review finding. A conflict during the revert means the undo is a manual job, so the merge should be scheduled, not squeezed in.

Step 2: state rehearsal

Run the down-path on a scratch host with a scratch database, then compare shape before and after. The example below is Alembic, but any migration tool with a down command fits.

set -euo pipefail

up()   { alembic upgrade head; }
down() { alembic downgrade -1; }

up
psql -tAc 'select count(*) from users' > after_up.txt

down
psql -tAc 'select count(*) from users' > after_down.txt

if diff -q after_up.txt after_down.txt >/dev/null; then
  echo "STATE: shape restored"
else
  echo "STATE: down-path changed the shape, review the migration"
fi

This proves shape, not identity. A down-path can restore the column count and still lose every value in it. If the migration writes real data, the rehearsal belongs on a snapshot, and no snapshot means no merge.

Step 3: triage the diff before you read it

A regex classifier will not understand your code, and that is fine. It only has to tell you which of the three buckets to open first.

#!/usr/bin/env python3
"""Heuristic revert-friction classifier. Triage aid, not an approval gate."""
import re
import subprocess
import sys

RULES = [
    ('human-only', '^[+-].*(DROP|TRUNCATE|DELETE FROM)', 'destructive SQL'),
    ('human-only', '^[+].*(npm publish|docker push|put_object|send_webhook)', 'publishes outside the repo'),
    ('rehearse-first', '^[+].*(CREATE TABLE|ALTER TABLE|CREATE INDEX)', 'schema change'),
    ('rehearse-first', '^[+].*(workflows/|Dockerfile|package-lock.json|poetry.lock)', 'build or CI surface'),
    ('rehearse-first', '^[+].*(os.environ|secrets.)', 'env or secret surface'),
    ('merge-now', '^[+].*(def test_|test_|.spec.ts)', 'test-only addition'),
]

def classify(patch):
    buckets = {}
    for line in patch.splitlines():
        for bucket, pattern, label in RULES:
            if re.search(pattern, line):
                buckets.setdefault(bucket, []).append(label)
    return buckets

if __name__ == '__main__':
    diff = subprocess.run(
        ['git', 'diff', '--no-color', sys.argv[1]],
        capture_output=True, text=True, check=True,
    ).stdout
    for bucket, labels in classify(diff).items():
        print(f'{bucket}: {sorted(set(labels))}')

A line can land in more than one bucket, and the human-only rules win when it does. Run it as python classify.py origin/main...HEAD on a copied diff, never as a merge gate.

A decision table you can argue with

Diff surface	Revert cost	Rehearsal	Verdict
New pure function plus tests	seconds	none	Merge
Column added with a default, no backfill	minutes	down then up on scratch	Merge after the down-path runs clean
Backfill over existing rows	hours, and data loss is possible	needs a snapshot	Block until the snapshot exists
CI workflow edit	minutes, but it affects every branch	revert on a scratch branch, run once	Merge, then watch the next scheduled run
Lockfile bump	minutes	revert, reinstall, run tests	Merge if the reverted lockfile installs
Anything that already delivered an event	not reversible by git	none available	Split the PR: merge the interface, hold the side effect

Print that table, tape it to the review checklist, and delete any row that stops being true for your stack.

Where the scratch environment comes from

The state rehearsal needs a host you are willing to break. Running a half-applied migration on a shared staging box is a worse incident than the bug you were reviewing, and requesting a sandbox from a platform team is usually the step that makes people skip the rehearsal entirely.

I ran the rehearsal above on a throwaway host from MonkeyCode's free server option and used its free model access for the agent half of the loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two things I can state, because the operator's current announcement says so: there is a free token allowance listed at 10 million tokens and a free server option, and both were enough for this workflow, where the jobs are small and bursty — one up/down migration cycle, one reverted test run. Two things I will not state: hardware, uptime, or retention, or that the numbers hold next month. Check the current terms before you put anything real on a free tier.

Where the free tier genuinely helps is the cost of trying. A rehearsal host you have to ask permission for is a rehearsal you skip. Where it does not help: compliance-scoped data, customer PII, or anything you would not put on a shared machine. And a free model will write you a confident down-migration; treat that as a draft, not as evidence. Run it.

If you want to try the loop before wiring a paid runner, the free tier is there, and nothing above depends on it.

Limitations, and who should skip this

The classifier is regex over diff lines. It misses DDL generated at runtime by an ORM, infrastructure state changes, and writes performed by a queue consumer in another repository.
The structural rehearsal models git conflicts, not runtime behavior. Caches, feature flags, and eventually consistent readers mean a clean revert can still be wrong for an hour.
The state rehearsal proves shape, not fidelity. Column counts and nullability can return while the values inside them do not.
Reverting is sometimes the wrong answer. If a forward fix is ten minutes and a down-migration plus re-up is two hours, fix forward and write down why.

Skip this approach if your migrations are owned by a platform team you cannot run against a scratch database, if the workload touches regulated data, or if you have no scratch environment at all. In that last case, build the sandbox first — the rehearsal is worth little without somewhere safe to fail.

You do not need all three steps on every PR. Run the classifier on every agent diff, run the structural rehearsal when the diff touches shared files, and run the state rehearsal whenever the diff writes anything. The review question is not only whether the change is right. It is whether the change is right, and how cheap being wrong is.

The Test That Can't Fail: Reading the Test Diff of an Agent-Generated PR

Casey Chen — Thu, 15 Oct 2026 11:36:04 +0000

The Test That Can't Fail: Reading the Test Diff of an Agent-Generated PR

Agent-generated PRs fail review in a specific way. The production code looks reasonable, the CI is green, and the new tests prove nothing — because they mock the one function the PR changed.

Strip the tests out and the diff is usually small. That is the tell. When 200 lines of test text surround 15 lines of behavior, the volume is doing rhetorical work, not verification work.

This post is about reviewing that test diff. Not the architecture, not the naming, not whether the agent's approach is elegant. Just one question: if the behavior broke tomorrow, would any of these new tests turn red?

Why the test diff is the least trustworthy part

Production code and test code in an agent PR are written by the same model, in the same session, from the same misunderstanding. If the agent believes a function returns a normalized path, both the implementation and the assertion will encode that belief. Reading them together feels like corroboration. It is actually one claim repeated twice.

So calibrate trust by executability, not by volume:

Lines that execute in production — a new branch, a changed constant, a new call site. You can test these.
Lines that describe behavior — tests, docstrings, comments, PR-body summaries, changelog entries, assert True placeholders, eval scripts that were never run. These are claims.

The failure mode is double counting. The agent asserts X in the PR body, asserts X in a test name, and you count three pieces of evidence where there is one.

A 15-minute diff autopsy

Do this before reading the implementation in detail. Order matters.

Split the diff by kind. Separate production files from test files from docs/lockfiles.
Write down one sentence describing the behavior the production hunk claims to add.
Find the assertion that would fail if that behavior were absent. Not the test that passes when it is present — the one that fails when it is missing.
Break the production line on purpose in a scratch copy and rerun only the new tests.
Revert anything you could not make fail. A test that survives a broken implementation is not a test.

# 1. what actually changed, by kind
git diff --stat "${BASE:-origin/main}"...HEAD -- ':!tests' ':!*_test.*'
git diff --name-only "${BASE:-origin/main}"...HEAD -- 'tests/*' '*_test.*'

# 2. did the new tests ever pass on their own?
python -m pytest -q $(git diff --name-only "$BASE"...HEAD -- 'tests/*' | tr '\n' ' ')

Step 4 is the one people skip, because it is repetitive. Make it scriptable.

The artifact: a mutation smoke test for the review thread

This is a blunt, regex-level mutation loop. It takes the production files touched by the PR, mutates numeric literals and booleans, and reruns only the new test files. Any mutation that leaves the suite green is a survivor — a behavior the new tests cannot detect.

This script has not been executed by the article author. Treat the paths as placeholders and run it in a throwaway worktree, never in your working tree.

# mutate_smoke.py — survivors are untested behavior, not proof of a bug.
import pathlib, re, shutil, subprocess, sys, tempfile

TARGETS = sys.argv[1:6]          # production files from the diff
NEW_TESTS = sys.argv[6:]         # only the test files the PR added/changed

PATTERNS = [
    (r"\b(\d+)\b", lambda m: str(int(m.group(1)) + 1)),  # n -> n+1
    (r"\bTrue\b",  lambda m: "False"),
    (r"\bFalse\b", lambda m: "True"),
]

def sites(src):
    for pattern, replace in PATTERNS:
        for match in re.finditer(pattern, src):
            yield match.start(), match.end(), replace(match)

survivors, killed = [], 0

with tempfile.TemporaryDirectory() as tmp:
    work = pathlib.Path(tmp) / "repo"
    shutil.copytree(".", work, ignore=shutil.ignore_patterns(".git", "__pycache__"))

    for target in TARGETS:
        original = (work / target).read_text()
        for start, end, replacement in sites(original):
            mutated = original[:start] + replacement + original[end:]
            (work / target).write_text(mutated)
            run = subprocess.run(
                ["python", "-m", "pytest", "-q", *NEW_TESTS],
                cwd=work, capture_output=True, text=True,
            )
            if run.returncode == 0:
                survivors.append(f"SURVIVED {target}:{start} -> {replacement!r}")
            else:
                killed += 1
            (work / target).write_text(original)

print(f"killed={killed} survived={len(survivors)}")
print("\n".join(survivors) or "every mutation was caught")

Read the output like a reviewer, not a metric. A surviving +1 on a retry count is a real gap. A surviving digit inside a string literal is noise — the regex does not parse Python. Cap the run at roughly 20 mutations per PR; beyond that, the loop stops being something a human will actually do.

What to trust, what to revert, what to test

Diff element	Default action	Evidence that flips the default
New production branch	Test it, then trust	Mutation survives → revert or add a test
Retry / timeout / limit constant	Revert	A test that fails when the value is out of range
Test that mocks the function under review	Revert	Rewritten to exercise a real boundary
New dependency pin	Revert	Lockfile diff plus a linked upstream changelog
Prompt string	Treat as data, never as policy	Server-side validation exists independent of it
Tool/schema registration	Test a denied call	Executor denies by default
PR-body "verified" claim	Ignore	A copy-pasteable command with its output
Changelog / doc edits	Collapse to one paragraph	Claim traceable to a code line

Two rules keep the table honest. First, a revert is not a rejection — it is a request for the smallest diff that carries its own evidence. Second, never merge because the tests are green; merge because you broke the code and the tests noticed.

Where a free re-run fits

Steps 1–4 are free for pure code. They stop being free when the mutation loop calls a model.

If the PR touches a model call, the survivor check needs a live endpoint, and a review pass can quietly become a metered task. That is the specific gap MonkeyCode covers in this workflow: per the operator it provides free model access and a free server option, which makes the reproduce-and-re-run half of review cheap enough to do per PR instead of per quarter.

The operator states the free tier currently includes a 10M-token allowance. Confirm the current terms on the project page before you plan a review cadence around it — allowances change, and this article should not be your source of record. MonkeyCode is presented as an open-source project; check the repository's own documentation for what is actually released.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Three constraints apply, and they are not negotiable:

A re-run only counts as evidence if someone else can repeat it. Paste the command and the raw output into the review thread. Otherwise it is an anecdote with a green checkmark.
Bound the re-run. Use a scratch credential and a scratch server, never the production key, for the review loop.
A free endpoint is not an allowlist. Cheapness is not a security control, and it is not a reason to merge.

Limitations and who should skip this

Cost scales as mutants × test runtime. On a fat config module you will generate survivor spam and learn nothing. Scope to files the PR touched.
It detects tests that cannot fail — not behavior that is missing. A feature with no test at all produces zero survivors. The autopsy steps still matter more.
Regex mutation is deliberately crude. It edits string literals and comments. Review the list; do not paste the count into a dashboard.
Skip it if your CI already runs mutation testing, if the PR is docs-only or a pure dependency bump, or if the changed tests require a live third-party service you cannot isolate.
Skip it if you are the only reviewer and there is no second person to hand the survivor list to. The artifact is a communication device. Without a reader, it is just latency.

The point is not to make agents write better tests. It is to make the tests stop being decoration. Break the code, watch for red, and revert whatever stayed green.

If you want to try the re-run half of this loop, MonkeyCode's free model access and free server option are aimed at exactly this unglamorous work — the review discipline still has to come from you.

A 429 Loop Is Not Resilience: Reviewing Agent PRs That Retry Model Calls

Casey Chen — Tue, 13 Oct 2026 03:17:43 +0000

An agent pull request that wraps model calls in retries is a capacity change, not a reliability patch. Trust the error classification and the stop condition. Revert unbounded loops, retries on non-idempotent writes, and any comment that treats a shared server as spare capacity. Prove the budget with a local fault injector before the branch is allowed to touch a hosted endpoint.

Read the loop before the summary

Agent summaries promise resilience. The diff says otherwise when a new function calls the client inside a loop. Start with the loop bound, the status set, and whether the call writes state. Those three facts decide the review.

The initial call counts as attempt 1. A helper that retries three times after a failure sends four calls. Spell that in the test name so a later edit cannot widen the budget silently.

Look for four smells before reading the rest of the PR body.

A loop with no ceiling, or a ceiling copied from a comment rather than a named constant with a test.
A bare exception handler, or a retry on every status below 500, including 401, 403, and 422.
Sleep on the request thread, with no jitter and no respect for Retry-After.
The same wrapper around reads and writes, with no idempotency key.

Label every hunk

Use one label per hunk. Do not average the patch into a single pass.

Diff signal	Label	Why
Typed timeout, no retry added	Trust	Bounds wait time without multiplying calls
Retry only on connection reset, cap 2, read-only	Prove	The cap is necessary, not sufficient
Unbounded loop or a high cap with no test	Revert	Unjustified amplification
Retry on 400, 401, 403, 404, 409, 422	Revert	Deterministic client failures will not heal
Retry on 429 with jitter and a hard stop	Prove	Needs a clock-injection test
Retry around mail, deploy, or billing POST	Revert	Replay is a second effect
Idempotency key plus a replay test	Prove	The key is untrusted without the test
Hardcoded quota or a free-tier capacity claim	Revert	Unverified capacity is not an input
Test that only mocks the final 200	Revert the test	It hides the policy

A green unit test that stubs the second attempt as success does not move a row from Prove to Trust. The label column is the merge gate.

429 Too Many Requests is specified by RFC 6585. Retry-After belongs to HTTP semantics in RFC 9110. Treat other 4xx codes as non-retryable unless the spec for that code says the request can be repeated safely.

A classifier you can run locally

The module below is a review aid, not a client SDK and not a benchmark. It has no network calls. It targets Python 3.10 or newer.

The cap of 3 is a merge heuristic for interactive calls, not a vendor limit. Nothing here was executed for this article.

from dataclasses import dataclass

NON_RETRYABLE = frozenset({400, 401, 403, 404, 409, 422})
TRANSIENT = frozenset({408, 425, 429, 500, 502, 503, 504})

@dataclass(frozen=True)
class Attempt:
    status: int | None
    attempt: int
    max_attempts: int
    idempotent: bool
    writes_side_effect: bool
    honors_retry_after: bool

def label(a: Attempt) -> str:
    if a.max_attempts < 1 or a.max_attempts > 3:
        return 'revert'
    if a.writes_side_effect and not a.idempotent:
        return 'revert'
    if a.status in NON_RETRYABLE:
        return 'revert'
    if a.status == 429 and not a.honors_retry_after:
        return 'prove'
    if a.status in TRANSIENT or a.status is None:
        if a.attempt >= a.max_attempts:
            return 'trust_stop'
        return 'prove'
    if a.status is not None and 200 <= a.status < 300:
        return 'trust'
    return 'prove'

if __name__ == '__main__':
    samples = [
        Attempt(429, 1, 3, True, False, False),
        Attempt(401, 1, 2, True, False, True),
        Attempt(503, 2, 2, True, False, True),
        Attempt(503, 1, 8, False, True, True),
    ]
    for s in samples:
        print(s.status, label(s))

Expected lines are 429 prove, 401 revert, 503 trust_stop, and 503 revert. If those four labels are not obvious from the diff under review, the helper is not ready.

Pair the classifier with five local cases. Keep every case on a stub. None of them should open a socket.

Inject 401 and assert a single attempt.
Inject 429 with a two-second Retry-After and assert one bounded delay, not a tight loop.
Inject 503 until the cap and assert stop, counting the original call.
Mark a POST as a side effect without an idempotency key and assert refusal.
Advance a fake clock. Do not call time.sleep in CI.

class FakeClock:
    def __init__(self) -> None:
        self.delays: list[float] = []

    def sleep(self, seconds: float) -> None:
        if seconds < 0 or seconds > 30:
            raise AssertionError(seconds)
        self.delays.append(seconds)

That clock is an unexecuted proposal. Wire it into the helper under review. A production sleeper that ignores the injected function fails case 5 by construction.

Idempotency key, not a comment

A comment that says a call is safe to retry is not a key. Trust a stable key derived from the job id and the step name. Revert a key built with a fresh random value inside the loop. A new value per attempt defeats deduplication on the server.

def idempotency_key(job_id: str, step: str) -> str:
    # Unexecuted proposal: same key on every attempt of this step.
    return 'job-%s-%s' % (job_id, step)

Cancellation between attempts

A loop that checks an abort flag only before the first call will keep going after the worker is cancelled. Trust a check between attempts. Revert a helper that cannot observe shutdown. On a shared server, a cancelled job that continues is wasted capacity and a stale-write risk.

Commands for the pass

rg is convenient. git grep -n is the fallback when rg is absent. Replace the base ref with the PR base.

git diff --unified=5 origin/main -- '*.py' '*.ts' '*.go' | rg -n 'retry|backoff|max_retries|Retry-After|while True|tenacity'
rg -n 'time.sleep' tests

The second search is a smell detector, not a proof. A clean result still needs the five cases. A hit on time.sleep in tests means the clock is real, so the job can stall under a 429.

Suggested comment to paste and edit, not a record of a specific review. Trim it to the hunk you are looking at.

This wrapper multiplies calls. Cap reads at 2 attempts after you count the original call, refuse retry when the call writes state, and honor Retry-After on 429. Inject failures with a fake clock. A job that loops against a hosted server to see what happens should not be accepted.

Where a free model path changes the gate

Some teams let an agent draft this patch on a hosted assistant, then point CI at a free server instead of a stub. That choice does not relax the table.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option are the only availability claims used here. This article does not state a token quota, a model list, hardware, uptime, or a duration. Those figures were not verified, and they are the wrong kind of constant to paste into a retry helper. If a diff comment cites a specific free allowance as the reason retries are safe, revert the justification and ask for a primary doc link in the PR.

Keep the hosted path in one place. Use free model access, if it is already available, to draft the classifier and the five tests. Run the tests on a local stub.

After they pass, one canary call can confirm auth and URL shape. It cannot tell you whether backoff terminates.

Do not parallelize that canary. Do not loop it to find a ceiling. A shared free server has neighbors.

It is not a load-test target. A 429 from it is a stop sign for this review, not a defect to hammer through.

Limitations

Status codes are a partial taxonomy. Timeouts, reset streams, and truncated JSON need separate branches. This classifier does not cover them.

Retry-After may arrive as a delta or as an HTTP date. The sample does not parse either form.

The cap of 3 is a review heuristic for interactive calls, not a capacity study. Batch jobs need a ceiling taken from your own queue metrics. This protocol will not tell you whether a model answer is correct. It only decides whether a retry policy is safe to land.

Skip the approach when the call sends mail, charges a card, or mutates production without an idempotency key already under test. Skip it if the only oracle available is the hosted server. Also skip it when the change needs a formal error budget. A table is a gate, not an SLO.

If MonkeyCode is already how the draft gets written, keep the free server outside the retry loop and spend one capped canary only after the local labels match. Read the live terms before anyone writes a number into the patch.

A Schema Pass Is Not an Execution Grant: Reviewing Agent PRs That Add Tools

Casey Chen — Sun, 11 Oct 2026 22:20:26 +0000

Merge the parser. Hold the grant. An agent pull request that adds function calling is three changes in one diff: a schema, a policy, and a side effect. Schema checks can land like any other contract. Policy text and a live handler cannot ride along just because a model emitted valid JSON.

This piece uses a synthetic fixture, not a production incident report. No latency, cost, or pass-rate figures are claimed. The commands are local. The tests below are unexecuted examples, labeled so a reviewer can run them instead of trusting this draft.

What the diff usually hides

Agent PRs in this class often look small. A registry gains one tool. A test checks that one sample payload validates. The handler is short, usually added so the task can "finish."

That shape is the risk. Validation answers whether an object matched a schema. Execution answers whether this process may perform an action. Those questions fail independently.

Watch for these signals in the same commit:

A tool name that is a vague verb, such as run_task or apply_fix
A description that hands judgment to the model ("do what is needed")
A parameter typed as an unconstrained string
A handler that can reach a shell, an HTTP client, or the filesystem
A config default that enables the tool outside tests

Review each signal on its own. A green happy-path test does not clear the bundle.

Synthetic diff under review

The module below is a constructed review fixture. It is not a recommended implementation, and the handler does not start a process.

# synth/agent_tools.py — fixture, not production code
from dataclasses import dataclass
from typing import Any, Callable

@dataclass
class Tool:
    name: str
    description: str
    parameters: dict
    handler: Callable[[dict], Any]

def proposed_tools() -> list[Tool]:
    return [
        Tool(
            name="run_task",
            description="Run whatever is needed to finish the user task",
            parameters={
                "type": "object",
                "properties": {
                    "command": {"type": "string"},
                    "url": {"type": "string"},
                },
                "required": ["command"],
            },
            handler=lambda args: args["command"],
        )
    ]

Returning the string is deliberate. The review question is whether this shape may become an executor, not how to invoke one.

What can land with the parser

Trust only hunks a reviewer can re-run without a model and without a network.

The schema document, if a test pins the tool name, required keys, and types.
A pure function that accepts or rejects an argument object before any handler runs.
A dry-run result that records the intended action and performs none.
Renames, comments, or tighter types, once those tests exist.

A passing schema test is evidence about parsing. It is not evidence about least privilege.

What to split out of the PR

Split or drop any hunk that does the following:

Turns the new tool on from a default constructor, production config, or shared fixture
Forwards a model-supplied string into a shell, a query API, or a URL fetch
Uses the description string as the only constraint
Adds a catch-all tool next to a narrow one "for flexibility"
Copies raw arguments into logs, metric labels, or outbound errors
Asserts on live assistant prose instead of the gate's return value

This is not a rejection of tool calling. It is a rejection of merging the grant with the parser.

What the replacement must prove

The replacement PR needs three artifacts: an allowlist expressed as data, a gate that rejects unknown actions before the handler, and tests that call a fake runner only.

Gate

# synth/grants.py
ALLOWED = {
    "format_python": ("ruff", "format"),
    "show_version": ("ruff", "--version"),
}

class GrantRejected(Exception):
    pass

def gate(action: str, extra: list[str] | None = None) -> tuple[str, ...]:
    if action not in ALLOWED:
        raise GrantRejected(action)
    if extra:
        raise GrantRejected("extra arguments are out of policy")
    return ALLOWED[action]

def dry_run(action: str) -> dict:
    argv = gate(action)
    return {"action": action, "argv": argv, "executed": False}

ruff is a stand-in binary name. A real repository should list its own fixed argv pairs. Python 3.11+ is assumed for list[str] | None.

Tests

# tests/test_tool_grant.py
import pytest
from synth.grants import GrantRejected, dry_run, gate

def test_known_action_is_a_fixed_argv():
    assert gate("show_version") == ("ruff", "--version")

def test_dry_run_does_not_execute():
    report = dry_run("format_python")
    assert report["executed"] is False
    assert report["argv"][0] == "ruff"

@pytest.mark.parametrize("action", ["run_task", "bash", "curl", ""])
def test_unknown_action_is_rejected(action):
    with pytest.raises(GrantRejected):
        gate(action)

def test_extra_arguments_do_not_widen_the_grant():
    with pytest.raises(GrantRejected):
        gate("show_version", extra=["--unsafe"])

git diff --stat
git diff -U3 -- synth/agent_tools.py synth/grants.py
python -m pytest tests/test_tool_grant.py -q

Intended outcome of this unexecuted fixture: known actions return a fixed argv, dry-run sets executed to false, unknown actions raise, and extra arguments do not widen the grant. This draft does not report an executed pytest session. If a later edit appends extra onto argv, the last test is the signal. A model transcript is not.

Decision table

Diff signal	Bucket	Merge action
JSON Schema for a named tool, types pinned by tests	Parser	Land after the local test run
Description narrowed to one verb and one object	Policy text	Keep as documentation only
Unconstrained `command` or `url` string	Grant	Split out
Import of a shell, HTTP, or filesystem helper in the handler	Side effect	Require dry-run before any call
Config or constructor default enables the tool	Deploy surface	Remove the default
Test depends on a live model or a shared remote host	Oracle	Replace with the fake runner
Allowlist data plus rejection tests	Proof	Re-merge the grant alone

Apply the table per hunk, not per pull request. One PR can contain a trustworthy parser and three hunks that should not land.

Review order on a mid-sized agent diff

Use this order when the diff is past a couple of screens. It keeps the grant from hiding inside formatting noise.

List files touched. Separate schema, config, handler, and tests.
Read config and constructors before descriptions. A default-on flag is a deploy, even if the schema looks narrow.
Search the handler for process, network, and filesystem calls. If any exist, demand a dry-run path in the same replacement.
Confirm tests import the gate, not a hosted client.
Write the review comment against the table row, and name the hunk.

Comment template:

Hunk: synth/agent_tools.py, tool run_task
Table row: unconstrained command string / grant
Decision: split out. Parser tests may land.
Replacement required: allowlist entry, gate rejection test, dry-run flag.
Not accepted as evidence: assistant prose, a green call to a shared host.

That comment is enough for a second reviewer to repeat the check. It does not depend on which model drafted the original patch.

Where optional hosted access fits

The gate does not need a hosted model. Review quality comes from the allowlist and the tests.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator-supplied context for this draft is limited to two availability claims: free model access and a free server option. Model names, quotas, hardware, duration, and benchmarks are omitted on purpose. They were not verified against a primary source here, and they change. Read current product documentation before use.

Optional sequence, after the local tests exist:

Keep tests/test_tool_grant.py as the merge bar.
Use free model access only to draft extra rejected-action names. A human deletes vague, duplicated, or out-of-scope cases before commit.
Use a free server only as a manual replay host for one recorded tool-call trace, on scratch input, outside CI.

Do not add that server to required fixtures. Do not fail the build when the free option is slow, capped, or withdrawn. A replay can suggest a missing rejection name. It cannot approve a handler.

Suggested note after a manual replay:

Replay host: free server option, manual, not a CI dependency.
Docs checked on: YYYY-MM-DD (reviewer fills the date).
Merge decision: tests/test_tool_grant.py, not the replay transcript.

If the docs are unavailable, skip the replay. Local tests still stand. If the access terms fit repository policy, those two free options are enough to draft rejection cases and replay one trace. Cite the docs date in the review comment. Do not cite a remembered allowance.

Limitations

The allowlist is only as careful as the binaries it names. This fixture does not cover tools loaded after startup, or a second agent that mints schemas at runtime. Description text can still mislead a model when the gate is correct. Enforcement belongs in gate, not in the prompt.

No count is offered for how often agent PRs mix these hunks. The table is a procedure, not a prevalence study. Teams bound by a data-processing addendum, a contractual uptime clause, or a ban on third-party inference should not send traces to a free server at all.

Who should skip this

Skip the hosted replay when the repository cannot accept an external processor, even for a scratch trace. Skip the grant split only when the change truly has no side effects, which is uncommon once a handler exists. Do not return this PR to the same agent to "fix the tests" unattended. The agent that widened the grant is a poor judge of whether the grant should exist.

Local tests remain the bar. Hosted drafting and replay stay optional, time-bounded, and disposable.

Model Snapshots Fail as Oracles: A Three-Bucket Review for Agent Diffs

Casey Chen — Sat, 10 Oct 2026 18:58:59 +0000

An agent pull request that deletes a local fixture and commits a model transcript has not added a test. It has stored one observation. Keep the intent to cover the parser. Revert the live call, the golden file, and any silent reroute. Prove the behavior with a fixture the repository owns.

That rule does not depend on which assistant drafted the patch. Review the diff. The generator's confidence is not evidence.

What landed in the branch

The shape is usually small. A file such as tests/golden/answer.json appears. A helper builds a client from a base URL. The assertion becomes equality against the new file. CI changes just enough for the job to reach an external inference host.

Sometimes a second hunk hides in the client. On timeout, quota, or HTTP 429, the code assigns a different base URL and sends the same payload again. That is not error handling. It is a second deployment, selected at runtime, with no review of which data leaves the process.

Read network edges before you read the PR prose. A required check that opens a socket will fail for reasons that are not regressions in your code. It also makes every fork run a client of an external host.

Trust, revert, prove

Split every hunk into three buckets before you write a comment. Short diffs still need the split. Length is a weak signal.

Trust lines that remain true when the network is off:

A pure parser for the completion payload.
The short list of fields the application actually reads.
A regression expressed as bytes in the repo, written by a reviewer or generated from a schema you control.

Revert lines that bind the merge gate, or production, to an environment you do not control:

HTTP clients in the default test path.
Checked-in transcripts used as equality oracles.
An except block that swaps in another base URL.
Logs of raw prompts, tool outputs, or Authorization values.
Hard-coded quotas, model identifiers, regions, or comments that say the host is permanently free.
Shell helpers that build a command string and pass shell=True.

Prove with two jobs. The contract job uses a local cassette and is required. The smoke job hits an external host only when someone dispatches it. Smoke may check a status code. It must not check exact prose.

Why equality against model text fails

Three properties of the design break the oracle. No benchmark is required to see them.

Sampling and server-side changes make byte-identical output unstable. A green run today does not pin the server you will get next week. The transcript can also echo the prompt. If the agent copied a credential or a customer string into the request, the golden file publishes it.

So the file is evidence of a past call, closer to a log than to a specification. Logs are not contracts. Merge the parser if it is sound. Delete the captured answer.

Latency fields fail the same way. A cassette that asserts elapsed_ms < 800 couples your build to someone else's queue. Drop timing, token-usage, and provider metadata from any file you intend to keep.

Cassette rules

A cassette is a contract only if a reviewer can explain every field. Keep three ideas in the file: a synthetic input label, the payload fragment the parser accepts, and the expected structure. Drop chain-of-thought text and raw tool traces. Those values change without your application code changing, and some of them are sensitive even when the prompt looked harmless.

Name the file after the behavior, not after the host or the date. parse_completion.json survives a server change. free_host_2026_10_10.json does not. If the agent used a timestamped name, rename it in the same review. The filename is part of the claim.

Worksheet

Fill the action from the patch, not from the description.

Question	Fail signal	Action
Does required CI open a socket?	`httpx`, `requests`, `curl`, or a client import under `tests/` without a skip mark	Move it out of the default job
Is model prose committed?	New JSON or text under `tests/` or `fixtures/` whose body is a completion	Revert; add a hand-written cassette
Can cache keys collide across tenants?	Key derived only from the prompt text	Revert, or include tenant id and contract version
Does an error change the host?	`except` assigns a second base URL	Revert silent failover; default the flag off
Did secrets enter the diff?	`.env`, `Bearer`, `sk-`, full prompt dumps	Revert; rotate if the commit is already public
Does the PR state a quota or model name it cannot cite?	Token counts, hardware, permanence	Delete the claim until a primary source exists

An external inference option is not an allowlist, a capacity plan, or an oracle. If the patch treats it as any of those, that hunk is a revert.

Commands

These commands are a proposed review sequence. They are not output captured from a named repository.

git diff --stat origin/main...HEAD
git diff origin/main...HEAD -- tests .github docker-compose.yml
rg -n -i "httpx|requests|curl|base_url|subprocess|shell=True|golden" tests .github
rg -n -i "authorization|api_key|bearer |sk-|prompt" -g "!**/node_modules/**"

Open new files under tests/ first. The risky call is often in a helper with a harmless name, such as client_for_tests or load_golden.

If the search hits shell=True, stop on that hunk. A shell command is concatenation. Unquoted expansion splits on spaces and interprets metacharacters. Prefer a fixed argument list and shell=False, or revert the helper. A note that the agent already ran the script is not a review outcome.

Proposed harness

The Python below is a sketch for reviewers to demand. It has not been executed for this article. It is not a benchmark, and the helper probe_status is a placeholder.

import json
import os
from pathlib import Path

import pytest

CASSETTE = Path(__file__).parent / "cassettes" / "parse_completion.json"

def parse_completion(payload: dict) -> dict:
    choice = payload["choices"][0]["message"]["content"]
    return {"text": choice, "truncated": False}

def test_parser_reads_contract_fields_only():
    payload = json.loads(CASSETTE.read_text(encoding="utf-8"))
    assert parse_completion(payload)["text"] == "cassette-text"

@pytest.mark.smoke
def test_optional_reachability_smoke():
    pytest.importorskip("httpx")
    base = os.environ.get("SMOKE_BASE_URL")
    if not base:
        pytest.skip("SMOKE_BASE_URL unset; smoke is opt-in")
    status = probe_status(base)  # proposal: GET {base}/models or /health
    assert status == 200

The cassette should contain boring synthetic text. Do not paste a live completion into it. Exact prose assertions belong only on that local file, because you can change the file in the same pull request as the parser.

Pair the tests with a workflow fragment. This fragment is also unexecuted.

jobs:
  contract:
    steps:
      - run: pytest -m "not smoke"
  smoke:
    if: github.event_name == 'workflow_dispatch'
    steps:
      - run: pytest -m smoke
        env:
          SMOKE_BASE_URL: ${{ vars.SMOKE_BASE_URL }}

Manual dispatch is the point. An external host on the automatic path becomes a flaky required check. It is also an unbounded client of shared capacity whenever the workflow retries.

What not to multiply

Retries, matrix builds, and parallel smoke jobs multiply calls. A retry policy that was reasonable for a paid primary can be a load spike against a shared server. Keep smoke to one probe and no retry loop.

If you need throughput numbers, that is a different test. It needs an owner, a concurrency cap, and a stop time. This worksheet does not define that test, and a green status code does not supply those numbers.

Cache refills have the same shape as a bad golden file. A job that warms a cache by replaying production prompts at an external host is a data export. Refuse it unless every prompt is synthetic and the cache key includes a tenant boundary plus a contract version.

Using a free model path without promoting it

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode enters this review in two narrow places. Free model access can help a reviewer draft the comment or a synthetic cassette, after which a human still reads the diff. A free server option can be the value of SMOKE_BASE_URL on the manual job.

That is the full scope used here. This draft does not name models, token quotas, hardware, duration, or permanence, because those details were not supplied as citable sources. If a generated README claims a specific allowance, do not publish the figure. Ask for a primary document first. Availability is not a capacity plan.

Delete the product name and the worksheet still holds. Observations are not contracts. Required checks should not depend on an external inference host.

Who should skip this

Skip the split if you cannot keep fixtures free of customer data. Synthetic prompts only. If a real prompt is already in the branch, reverting that file is the first action. Rotate any credential that sits beside it.

Skip it if CI cannot exclude marks. A smoke test that still runs on every pull request is the original bug with a new name. Fix the workflow filter before discussing hosts.

Skip it when the main change is authorization, tenancy, or a tool that has side effects. Those patches need a threat model. A cassette will not catch a confused-deputy bug in a shell tool, and a status code will not show which files the tool touched.

Streaming responses also sit outside the sketch. Partial buffers, disconnects, and reassembled tokens need their own assertions. Do not freeze one stream mid-flight and call that file the spec.

Comment shape

Write the three buckets in the review. A usable comment looks like this:

Trust the parser and the field list in the application module.
Revert the golden transcript, the default base URL in tests, and the failover assignment.
Prove behavior with the cassette on the required job. Leave external reachability on manual dispatch.

If a free model path and a free server are already available to the team, attach them only to that manual job, after the required check is local. The merge decision stays on the cassette either way.

A Free Endpoint Is Not an Allowlist: Reviewing Agent PRs That Add Fetch

Casey Chen — Fri, 09 Oct 2026 15:34:09 +0000

An agent pull request that switches a client onto a free model server is three changes, not one. The client binding, the live smoke, and any new outbound tool each move data across a trust boundary. Review them separately. Trust a hunk only when a local test proves the bound. Revert the live path when that proof is missing.

The specimen in this note is constructed for review practice. It is not a report of a merged repository, a production incident, or a measured product limit.

What the specimen changes

The diff adds four surfaces that look unrelated in a file list:

agent/client.py defaults base_url to a hosted model endpoint and reads a key from the environment.
agent/tools/fetch.py registers a tool that issues GET for a URL chosen by the model.
.github/workflows/model-smoke.yml runs the client on push and pull_request.
tests/test_client.py expects a live completion instead of a stub.

Each file can be justified in isolation. The combination sends prompt text, tool traffic, and CI identity onto a network path the repository did not have. That combination is the review subject.

A short diff stat is not evidence of a small risk. Forty lines that open egress outweigh four hundred lines of renaming.

Gate table

Use the table as a merge gate. A row that fails its proof column blocks the PR even when the other rows look tidy.

Hunk	Trust only if	Revert if	Proof
Client base URL	Default is empty or loopback, and hosted URLs arrive from outside config	Library code defaults to a hosted endpoint	Construct the client with an empty environment and assert no socket use
Credential read	The key stays in the runner secret store	A key, `.env`, or Authorization value is in the diff	Search the diff for `API_KEY=`, `sk-`, and `Authorization`
Fetch tool	Allowlist is executable code, and redirects are off	Any URL, `file:`, or link-local target is accepted	Table tests for allow and deny, including redirects
CI smoke	Trigger is `workflow_dispatch` only, prompt is canned, timeout is set	Trigger is push, pull_request, or schedule	Read the `on:` block; run tests with the network stubbed
Tests	Default suite makes no network call	Passing requires a live server	Run the suite with sockets disabled

Review order

Read in this order so a client rename cannot hide the permission change.

Tool schema and HTTP helper.
Workflow triggers, then steps.
Tests, looking for live hosts and HTTP libraries.
Client defaults last.

A clean client does not redeem an open tool. A green live test does not redeem an open tool either. The live test is the policy you are trying to avoid making mandatory.

Commands

git diff --stat origin/main...HEAD
git diff origin/main...HEAD -- agent/tools .github/workflows tests
rg -n "httpx|requests\\.|base_url|workflow_dispatch|pull_request" agent .github tests

If the search hits both a fetch tool and a workflow trigger, treat the diff as one policy change. The model identifier string is the least useful line in the file.

Proposed allowlist

The module below is a proposed review gate for the specimen. It is not a benchmark, and it was not executed for this article.

from urllib.parse import urlparse

ALLOWED_HOSTS = frozenset({"docs.example.com", "static.example.com"})

class EgressDenied(Exception):
    pass

def assert_fetch_allowed(url: str) -> str:
    parsed = urlparse(url)
    if parsed.scheme != "https":
        raise EgressDenied(f"scheme not allowed: {parsed.scheme!r}")
    host = (parsed.hostname or "").lower()
    if host not in ALLOWED_HOSTS:
        raise EgressDenied(f"host not allowed: {host!r}")
    if parsed.username or parsed.password:
        raise EgressDenied("credentials in URL")
    return parsed.geturl()

The matching tests are the artifact a reviewer can ask the agent to keep. They fail closed.

import pytest

@pytest.mark.parametrize("url", [
    "https://docs.example.com/guide",
    "https://static.example.com/a.png",
])
def test_allows_known_hosts(url):
    assert assert_fetch_allowed(url) == url

@pytest.mark.parametrize("url", [
    "http://docs.example.com/guide",
    "https://user:pw@docs.example.com/x",
    "https://169.254.169.254/latest/meta-data/",
    "https://evil.example/docs.example.com",
    "file:///etc/passwd",
    "https://docs.example.com.evil.example/x",
])
def test_denies_unlisted_targets(url):
    with pytest.raises(EgressDenied):
        assert_fetch_allowed(url)

Two gaps belong in the review comment, not in a follow-up ticket that never lands. The function does not follow redirects, so the HTTP client must disable redirect following or re-check every hop. The function does not pin DNS, so an allowed name can still resolve somewhere unexpected. If the team cannot accept both gaps, remove the tool.

Comments are not the gate

A docstring that says not to fetch internal hosts is not an allowlist. A PR description that calls the smoke safe because the server is free is not a trigger restriction. Ask for the test file. If the agent updates the comment and leaves the tool open, revert the tool.

Where a free server is allowed to appear

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For this draft, MonkeyCode is relevant only as an operator-supplied option: a free model access path and a free server option are availability claims, not measured limits. This note does not state model names, token quotas, hardware, duration, uptime, or price. Those facts change, and none of them were verified here. Check the current primary terms before use. A free path is still an external processor of whatever you send.

Keep the hosted call inside all four constraints:

Default client target stays empty or loopback. Hosted configuration is injected, not committed.
The only workflow trigger is manual, with an explicit boolean input.
The prompt file is canned. It contains no source, no secrets, and no user data.
The job stores status, not raw prompt or completion text. Timeout is short. Retries stay at zero until a separate review.

# Proposed fragment for review. Not a deployed workflow.
on:
  workflow_dispatch:
    inputs:
      run_live_smoke:
        description: Run one canned call against the hosted endpoint
        required: true
        type: boolean
jobs:
  smoke:
    if: ${{ inputs.run_live_smoke == true }}
    timeout-minutes: 5
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Canned prompt only
        run: python -m agent.smoke --prompt-file ci/canned_prompt.txt

Revert on: push and on: pull_request from the specimen. A free endpoint still receives the prompt, still shares capacity, and still fails for reasons that have nothing to do with the change under review. Making that call required on every push converts a convenience into a merge dependency.

If the free server is down, skip the smoke. Do not fall back to an unbounded fetch, and do not fall back to a second hosted vendor inside the same unreviewed hunk. Fallback is a new data flow. It needs its own row in the gate table.

Failures a green unit test can hide

The allowlist is strict, and the same PR appends a host for a demo. Review allowlist edits like auth edits. A one-line host addition is a permission grant.
The smoke is manual, and the step still prints the response body. Log retention is a copy. Print a status code and a byte length, not the text.
The application allowlist is correct, and HTTPS_PROXY is set in the runner. Proxy configuration can move bytes after your check. Fix the runner environment, or deny the run when a proxy is present.
A retry wrapper lands beside the smoke. Five retries are five sends. The smoke job should set the retry count to zero.
The test suite mocks assert_fetch_allowed itself. That test proves the mock, not the gate. Call the real function.

Who should skip this split

Do not use the manual-smoke pattern when any item below is true.

Prompts can contain customer data, credentials, or unreleased source. Keep that traffic off hosted endpoints, free or paid.
Nobody on the reviewing team can read the tool schema. An unread schema is an unread permission. Close the tool until a human can name every argument.
Redirects cannot be disabled, and the host list cannot be kept explicit. Delete the fetch tool. A warning comment is not a substitute.
The goal is a latency or quality number. This protocol does not produce one. It produces a pass or fail on egress and on trigger scope.

Approval check

Approve only when four facts are visible in the diff: a local or empty client default, deny-by-default fetch tests, no live call on push or pull request, and no secret material. The hosted free-server option can remain as a manual canned smoke after those four are true. If one is missing, land the allowlist and revert the live path.

When a similar agent PR is already in the queue, run the deny cases against a stub before enabling any hosted smoke. The endpoint that answers is not the review. The bytes you allow to leave are the review.

Merge the Logger, Revert the Payload: Reviewing Agent PRs That Record Prompts

Casey Chen — Thu, 08 Oct 2026 10:55:07 +0000

Merge the logger. Revert the payload. Prove redaction with an offline test before any model call.

Agent pull requests often add tracing in the same diff as a new tool. The logger name looks harmless. The attributes do not. A span that stores messages, tool_args, or Authorization is an export, not a debug aid. Treat that export as the change under review.

Public DEV discussion in early October 2026 clusters around agent experiments and model behavior. Those posts are topic signals, not evidence for this review. The rule below does not depend on any of them. It depends on what the diff writes to disk, to a collector, or to a third-party endpoint.

This note uses a synthetic pull request. Nothing here is a measured incident, a benchmark, or a claim about a named production system.

The diff under review

The proposed change does four things:

It wraps tool execution in an OpenTelemetry-style span helper.
It attaches the full chat messages array and raw tool arguments as span attributes.
It adds GET /debug/last-trace so a reviewer can “see what the model saw.”
It points a fixture at a shared model server so the new path “stays realistic.”

The first item can be a real improvement. Items 2 through 4 change the trust boundary. Review them separately, even when they arrive in one commit.

A shortened form of the risky helper, labeled as a proposal and not as shipped code:

# Synthetic bad pattern. Do not copy into a service.
def run_tool(span, tool_name, tool_args, messages, headers):
    span.set_attribute("tool.name", tool_name)
    span.set_attribute("tool.args", tool_args)          # may include tokens
    span.set_attribute("llm.messages", messages)        # user content export
    span.set_attribute("http.authorization", headers.get("Authorization"))
    return tool_registry[tool_name](**tool_args)

Short attributes such as tool name, status, latency, and token counts are reviewable. Unbounded payloads are not.

What to trust

Trust only what you can check on the branch without a network call.

The span helper is pure enough to unit test: given a dict, it returns attributes, and it does not perform I/O itself.
Attribute keys are a fixed allowlist, not **payload spread into the span.
String values have an explicit max length, and over-limit values are dropped or hashed, not truncated into a still-readable secret.
The debug route is absent, or it is gated by an admin auth check that already exists in this codebase and is covered by a test.
Tests use a fake transport. They do not require DNS, a vendor key, or a shared server.

A green CI run is not evidence here. If the new test calls a live endpoint, the green result only proves that the endpoint answered. It does not prove that logs are safe.

What to revert

Revert these even if the rest of the PR is useful:

Default-on capture of messages, prompt, tool_args, tool_result, request bodies, or response bodies.
Header attributes, especially Authorization, Cookie, Set-Cookie, and x-api-key.
A debug route that returns the last prompt, the last tool payload, or a trace ID mapped to raw content.
Fixture or CI configuration that calls a free or shared model server to assert behavior.
A “temporary” sample rate of 1.0 on a path that still records content. Sampling does not fix an export. It only reduces how often the export happens.

Keep the logger skeleton if the allowlist is sound. Revert the payload attachment in the same review. Do not leave a TODO that re-enables full capture behind an environment flag defaulting to true.

What to prove

Prove three claims before merge: sensitive keys never become attributes, nested encodings do not sneak through a shallow check, and the test fails closed when the helper changes.

Offline redaction check

The following is an unexecuted review artifact. It is a proposal for the PR branch, not a result from a run on 8 October 2026.

import json

DENY_KEYS = {
    "messages", "prompt", "tool_args", "tool_result",
    "authorization", "api_key", "cookie", "set-cookie",
}

def flatten_keys(value, prefix=""):
    keys = set()
    if isinstance(value, dict):
        for key, child in value.items():
            path = f"{prefix}.{key}" if prefix else str(key)
            keys.add(str(key).lower())
            keys |= flatten_keys(child, path)
    elif isinstance(value, list):
        for child in value:
            keys |= flatten_keys(child, prefix)
    elif isinstance(value, str):
        stripped = value.strip()
        if stripped.startswith("{") or stripped.startswith("["):
            try:
                keys |= flatten_keys(json.loads(stripped), prefix)
            except json.JSONDecodeError:
                pass
    return keys

def assert_span_safe(attributes: dict) -> None:
    found = flatten_keys(attributes) & DENY_KEYS
    if found:
        raise AssertionError(f"span exports sensitive keys: {sorted(found)}")

Add one case per deny key, plus one case where the payload is a JSON string inside a “safe” field such as note. A shallow key in attributes check misses that case. This helper still misses base64 and homoglyph keys. Say that in the test module docstring so the next reviewer does not treat the denylist as complete.

Commands for the review

Run these from the PR branch. They are inspection commands, not a deploy.

git diff --unified=3 origin/main...HEAD -- '*.py' '*.ts' '*.yml'
rg -n -i "messages|prompt|tool_args|authorization|api_key|last-trace" \
  --glob '!**/node_modules/**' --glob '!**/.venv/**'
rg -n "set_attribute\(|span\.setAttribute\(" -g '!**/vendor/**'

Then classify each hit. A key in a test fixture that expects rejection is fine. The same key in a production helper is a revert.

Decision table

Diff signal	Trust	Revert	Prove
Span around tool call, name + status + latency only	Yes, if helper is pure	No	Unit test the attribute dict
Full `messages` or tool args on the span	No	Yes	Denylist test must fail on current patch, pass after revert
Token-count integer, no text	Yes, after type check	No	Assert value is `int` and bounded
`/debug/last-trace` without auth	No	Yes	Route test expects 404 or 401, never 200 with a body
Fixture URL set to a shared model host	No	Yes	Test suite passes with network disabled
Env flag `LOG_PROMPTS=true` by default	No	Yes	Default config test expects false or absent

If a row is “Revert,” do not spend the review on naming, formatting, or model choice in that hunk. Fix the boundary first.

Where a free model endpoint fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project. The outreach brief for this article states free model access, a free server option, and a free allowance on the order of 10 million tokens. This draft does not independently verify model names, hardware, duration, rate limits, or whether that allowance is still current on 8 October 2026. Confirm the live terms before you plan any review budget around them. Do not treat the number as a permanent quota.

Use that access only after the offline test is red. A disposable server is an optional manual repro for a scrubbed, synthetic prompt. It is not a test dependency, not a fixture host, and not a place to replay production traces. The earlier failure mode, calling a free server from a fixture and calling the result a contract, still stands. This workflow inverts it: the contract is the local assertion. The server, if used at all, comes later and stays out of CI.

A practical order:

Land the denylist test and watch it fail on the unsafe helper.
Remove payload attributes and the debug route. Watch the test pass with the network off.
If you still need to see a live tool round-trip, send one synthetic prompt from a local script, outside CI, to a free server whose current terms you just checked.
Record latency or error class in the PR comment if you want. Do not paste the prompt, the tool args, or the response body into the comment.

If the free server is down, slow, or quota-exhausted, the review does not block. The merge gate is the offline test.

Before you spend a live call on an agent diff, read the current MonkeyCode free-access terms and keep that server out of the test process.

Who should not use this workflow

Skip the live-repro step, and skip any third-party model endpoint, when any of these are true:

Policy forbids sending even synthetic prompts to an external service, or the data class is unknown.
The payload under review may contain customer content, credentials, or regulated data. Do not “sanitize and retry” on a hunch. Revert the export and review locally.
You need a quota, model list, or uptime guarantee. None of those are established in this article.
The PR’s real risk is authorization logic, schema compatibility, or retry amplification. Those need their own reviews. A redaction test will not catch them.

The denylist is also the wrong tool for a team that has not agreed which keys are sensitive. Agree the key set in the review, commit it next to the helper, and fail the build when the set and the helper diverge.

Review outcome

Approve the tracing shape when attributes are small, typed, and allowlisted. Request changes when the diff records prompts, tool payloads, or headers, or when a test needs a shared server to pass. The logger can ship the same day. The payload should not ship at all.

Limitations are part of the outcome. This check is key-oriented. It will not catch every encoding, every side channel, or every future attribute name. Re-run the rg pass on the next agent PR that touches telemetry. Agent diffs repeat this pattern under new file names.