Synthetic PRs Are the Missing Calibration Harness for AI-Era Reviewers

A PR lands on Monday. It has a green CI badge, a clean diff, and no explanation. Three reviewers approve it before lunch. The bug ships on Thursday. Nobody knows who missed it.

This is the new normal. AI tools write more code, so humans get promoted to reviewers. But nobody promoted the reviewer's judgment. Nobody tested it. The result is a gate that looks busy and feels unsafe.

Review quality is trainable. You just need known defects and honest feedback. Synthetic PRs give you both.

The untested gate

Most teams measure code coverage, deploy frequency, and incident count. They do not measure whether a reviewer can spot a real problem.

Real reviews have no ground truth. If a reviewer approves a bad PR, you discover it weeks later, and by then the evidence is buried in an incident postmortem. If a reviewer rejects a good PR, you rarely notice at all.

You need a controlled experiment. You need to know exactly which defect is inside a PR, and you need to see whether the reviewer finds it.

Synthetic PRs are that experiment.

One loop, three artifacts

The loop is small: find a real function, inject one defect, ask a teammate to review it, then score the review. Run it weekly. Plot the trend.

Three artifacts make this reproducible:

1. A generator that produces seeded tasks.
2. A report format for reviewers.
3. A scorer that turns reports into precision and recall.

The whole loop runs on a free stack. MonkeyCode is an open-source project whose free tier includes free models for generating tasks and a free server for running the pipeline. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Check the project docs before relying on quotas. Free tiers change without ceremony.

Step 1: Build a seeded task generator

The generator takes one function from your repo and asks a free model for a variant with exactly one behavioral change. The change keeps the same signature and style. The output is stored with a ground-truth file.

# seed_review_task.py — adapt the model call to your provider's SDK
import json
import os
import random
from pathlib import Path

ENDPOINT = os.getenv("MODEL_ENDPOINT")  # e.g. MonkeyCode free model endpoint
API_KEY = os.getenv("MODEL_API_KEY")
MODEL = os.getenv("MODEL_NAME")

SYSTEM_PROMPT = """You create code review exercises.
Given a Python function, produce a modified version that:
- changes behavior in exactly one place
- keeps the same function signature and style
- adds no comments
- looks plausible at first glance
Output only Python code, no explanation."""

def pick_target(root: Path):
    candidates = list(root.rglob("*.py"))
    return random.choice(candidates)

def generate_task(source: Path):
    prompt = f"{SYSTEM_PROMPT}\n\nOriginal:\n```
{% endraw %}
python\n{source.read_text()}\n
{% raw %}
```"
    # Call the model client here. Store the result and the defect metadata.
    mutant_code = call_model(prompt)  # placeholder; implement per provider docs
    task_id = f"task_{random.randrange(100000)}"
    meta = {"task_id": task_id, "source": str(source), "defects": [{"line": None}]}
    Path(f"tasks/{task_id}.diff").write_text(mutant_code)
    Path(f"tasks/{task_id}.meta.json").write_text(json.dumps(meta, indent=2))

The metadata file is your ground truth. Fill in the exact line and defect category before the task goes to a reviewer.

Run this on a free server via cron on Friday afternoon. Monday morning, the tasks are ready.

Step 2: Give reviewers one YAML output

A synthetic review is only useful if the reviewer records what they found in a machine-readable format.

# review.yaml
task_id: task_48231
verdict: change_requested
found_defects:
  - file: src/pricing.py
    line: 41
    category: off_by_one
nice_to_have:
  - "Add a regression test for the boundary."

Ask reviewers to spend no more than twenty minutes per task. Tell them it is a routine review. Do not reveal the seeded defect until after scoring.

Step 3: Score precision and recall

The scorer is a small Python script that compares the reviewer's report against the metadata you wrote in step one.

# score_review.py
import json
import sys

def score(report_path: str, meta_path: str) -> dict:
    report = json.loads(open(report_path).read())
    meta = json.loads(open(meta_path).read())

    found = {(d["file"], d["line"]) for d in report.get("found_defects", [])}
    expected = {(d["file"], d["line"]) for d in meta["defects"]}

    true_positives = len(found & expected)
    precision = true_positives / len(found) if found else 0.0
    recall = true_positives / len(expected) if expected else 0.0

    return {
        "task_id": meta["task_id"],
        "precision": round(precision, 2),
        "recall": round(recall, 2),
        "found": len(found),
        "expected": len(expected),
    }

if __name__ == "__main__":
    print(json.dumps(score(sys.argv[1], sys.argv[2]), indent=2))

Precision measures false alarms. Recall measures missed defects. Both matter.

Step 4: Act on calibration zones

One score means nothing. A quarterly trend means everything.

Zone	Recall	Precision	What to do
Steady gate	>= 0.9	>= 0.8	Keep reviewing; add harder tasks
Silent misser	< 0.6	>= 0.8	Add checklist + pair review
Noise generator	>= 0.7	< 0.5	Clarify defect definitions
Random clicker	< 0.5	< 0.5	Retrain before solo approvals

A reviewer in the bottom two zones should not be the sole approver on AI-generated PRs. That is the real value of the harness: it stops unsafe approvals before they ship.

Limitations and who should skip this

Synthetic defects are not the same as production incidents. They miss domain rules, integration failures, and design flaws. A high score does not certify a reviewer; it only says they noticed the planted problem.

Do not feed proprietary code to an external model endpoint if your compliance rules forbid it. Self-host or use an internal model when needed.

This loop also fails if you reuse tasks. Reviewers share notes, and a known task stops measuring judgment. Rotate functions and refresh the pool weekly.

Skip this entirely if your team has no written review culture. The script measures review skill, not team politics. If approvals are already theater, the scorer will only prove what you already avoid.

Start small

Pick one function. Inject one defect. Give it to one reviewer. Score one report. Do that for three weeks before building anything bigger.

Free models give you the generation side. A free server gives you the schedule. MonkeyCode's open-source free tier covers both; read its docs first and decide whether the loop fits your stack.

Reviewing is the new coding. Start measuring it.