Plant Bugs Before You Trust the Reviewer: Calibrating a Cheap Model on Agent-Generated PRs

A second model reading your agent's diff is not a reviewer until you have measured its miss rate. Everything below is the harness I use to decide whether a cheap — or free-tier — model earns a slot in the review queue, and what to do when it does not.

That distinction matters because the failure mode is invisible. A reviewer that catches one real bug is memorable. A reviewer that silently walks past five is just latency. You cannot observe false negatives in production, so you have to manufacture them.

The trap: "it looks competent" is not a measurement

Hand a 200-line agent diff to any mid-sized model and you will get something plausible back — structured findings, confident prose, a line number. Plausibility is cheap. What you actually need to know is:

• Recall per defect class. Does it catch swallowed exceptions but miss weakened assertions?
• False-positive rate on clean diffs. How many of its findings are noise you have to dismiss?
• Stability. Same diff, three runs — same answer?

None of those are answerable from a demo. They are answerable from planted defects.

Step 1: Build a small calibration set

Six to ten diffs is enough. Pull them from merged agent PRs in your own repo, and require two properties:

1. They merged, so you are grading detection, not arbitration.
2. They are diverse — at least three different files and three different agent sessions.

Keep two clean diffs that never contained a defect. You need those for the false-positive count.

Step 2: Seed defects deterministically

Pick four classes you actually care about. Mine are error handling, test integrity, config defaults, and added validation. Then express each seed as a literal find/replace against a scratch checkout, so the whole set is reproducible and reviewable in a PR of its own.

# seed_defects.py — plant known defects into a scratch copy of a merged PR
import json, pathlib, shutil, subprocess, sys

SEEDS = json.loads(pathlib.Path("seeds.json").read_text())

def apply(repo: pathlib.Path, seed):
    p = repo / seed["file"]
    src = p.read_text()
    if seed["find"] not in src:
        # anchors drift. fail loudly instead of scoring against a phantom
        raise SystemExit(f"anchor missing for {seed['id']} — reseed this PR")
    p.write_text(src.replace(seed["find"], seed["replace"], 1))

def main(sha: str, workdir: str):
    out = pathlib.Path(workdir)
    shutil.rmtree(out, ignore_errors=True)
    subprocess.run(["git", "clone", "--quiet", ".", str(out)], check=True)
    subprocess.run(["git", "-C", str(out), "checkout", "--quiet", sha], check=True)
    for seed in SEEDS:
        apply(out, seed)
    truth = [{"id": s["id"], "class": s["class"], "file": s["file"],
              "anchor": s["replace"].splitlines()[0].strip()} for s in SEEDS]
    # kept outside the scratch checkout; the reviewer never sees it
    pathlib.Path(workdir + "-truth.json").write_text(json.dumps(truth, indent=2))
    print(f"seeded {len(truth)} defects into {out}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

A seeds.json entry is three lines of JSON:

{ "id": "swallow-1", "class": "error-handling", "file": "app/client.py",
  "find": "    except TimeoutError:\n        raise",
  "replace": "    except TimeoutError:\n        pass" }

Line numbers are derived at scoring time from the anchor, not hard-coded — a rebase should not invalidate your calibration set.

Step 3: Give the reviewer a contract, not a conversation

Free-form review output cannot be scored. Constrain it to JSON with a fixed class vocabulary, and pass the diff, not the whole repo:

# reviewer.py — strict-JSON second reviewer over a diff
import json, os, subprocess, urllib.request

BASE  = os.environ["REVIEW_BASE_URL"]   # OpenAI-compatible endpoint
MODEL = os.environ["REVIEW_MODEL"]      # record this in the results file

CONTRACT = (
    "Review this pull request diff. Report only changes you believe are defective.\n"
    'Return JSON: {"findings":[{"file":str,"line":int,'
    '"class":"error-handling|test-integrity|config-default|validation","why":str}]}\n'
    "An empty list is a valid answer. No prose, no markdown."
)

def ask(diff_text: str):
    payload = json.dumps({
        "model": MODEL, "temperature": 0,
        "messages": [{"role": "system", "content": CONTRACT},
                     {"role": "user", "content": diff_text[:12000]}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ.get('REVIEW_API_KEY','unused')}"})
    with urllib.request.urlopen(req, timeout=120) as r:
        content = json.loads(r.read())["choices"][0]["message"]["content"]
    return json.loads(content)["findings"]

Two details that matter more than they look. temperature: 0 is not determinism, so run each diff three times and score the median. And truncating at 12k characters is a real limitation, not a bug — it tells you the harness only measures what fits.

Step 4: Score recall by class, then count noise

# score.py — per-class recall against planted defects
def score(findings, truth, tol=3):
    hit = {t["id"] for t in truth
           if any(f["file"] == t["file"] and abs(f["line"] - t["line"]) <= tol
                  for f in findings)}
    per_class = {}
    for t in truth:
        per_class.setdefault(t["class"], []).append(t["id"] in hit)
    recall = {c: sum(v) / len(v) for c, v in per_class.items()}
    return recall, len(hit) / len(truth)

A ±3-line tolerance is deliberate: the reviewer is allowed to point at the right neighbourhood. You are not grading formatting. You are grading whether it noticed.

Then run the same script on your two clean diffs and count findings. That number — findings per clean PR — is your review tax. Anything above two per PR and you are doing more arbitration work than you saved.

Step 5: Turn the numbers into a gate, not a vibe

Recall on error-handling and config-default	Findings per clean PR	Verdict
≥ 0.8	≤ 1	Trust as triage. Its findings choose which tests you write; a human still signs off.
0.5 – 0.8	≤ 2	Test-only. Convert findings into failing tests, ignore its verdicts entirely.
< 0.5, or any class at 0	any	Human-only. The queue is cheaper without it.

Fill in your own numbers. A recall figure from my repository tells you nothing about yours — different defect mix, different language, different prompt.

Where free model access actually changes this loop

This harness is call-hungry: 8 diffs × 3 repeats × 3 prompt variants is 72 requests before you have changed a single line of the contract. That iteration cost, not the review cost, is what stops most teams from calibrating anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option (both operator-supplied availability claims; quotas and capacity change, so confirm the current terms on the project itself rather than trusting this paragraph). In practice that removes two specific frictions here: the harness can be run to completion without metering calls mid-iteration, and the free server option means the seeding and scoring scripts run from a hosted box instead of tying up a laptop for an hour.

One rule I would keep regardless of price: the endpoint name is not a model version. Write REVIEW_MODEL and the endpoint's reported version into your results file. When either changes, recall can move and your gate is stale until you re-run step 4.

Limitations, stated plainly

• The harness measures detection of seeded classes, not novel bugs. A model can score 0.9 here and still be useless on a race condition.
• Small models degrade on cross-file invariants. Any diff touching more than about eight files is out of scope for this loop.
• Truncated diffs produce silent false negatives, which is the worst kind of number to put in a table.
• Three runs and a median is statistics-flavoured, not statistics.

Who should skip this

If you see fewer than ten agent-generated PRs a week, human review is cheaper than maintaining a seed set — and a stale seed set is worse than none, because it produces confident numbers about code you no longer write. Skip it too if the PR needs an approval stamp for compliance: this harness never approves anything, and no calibration result should be paraphrased into one.

If you want to try the five steps before committing, the free model access and free server option are enough to run the whole loop end to end — start with the clean diffs and the false-positive count, because that is the number that decides whether the rest is worth your afternoon.