Golden Patches: Test Your AI Reviewer Before It Tests Others

An AI review gate is software. Software needs regression tests. Most deployed gates have none.

Maintainers tune prompts until a demo pull request looks right. That is anecdote, not evidence. A golden set replaces anecdote with numbers. It is a small corpus of past patches with known outcomes.

The four-phase contribution workflow

Every open-source pull request follows the same skeleton. Maintainers or contributors reproduce the failure. Someone patches the code. Test automation runs the suite. A review gate judges the diff. Free models enter at the last phase. They only help when the first three phases are recorded.

1. Reproduce

Capture the failing test before any fix exists. The reproducer becomes the reviewer's ground truth. A review without a failing test is guesswork.

2. Patch

Apply the diff to a clean checkout. Never judge a diff from memory. The gate needs the real file state. Otherwise it reviews context instead of code.

3. Test

Run the complete suite on the patched tree. A green suite means consistent behavior. It does not mean correct behavior. Inverted logic can pass every test and still be wrong.

4. Review

Send the patch and the reproducer to a free-model reviewer. Ask one question: whether the patch fixes the failure without adding regressions. Record the verdict as pass or flag.

Why the gate itself needs tests

Free models change quietly between releases. Prompt behavior drifts as well. The gate that caught real bugs last month can flood the queue with noise this month. The fix is a golden set built from project history. Merged and never reverted means good. Reverted or closed for a bug means bad. Label each patch once. Reuse it for every future prompt. The gate and the model are separate moving parts. The corpus tests them together.

The harness

#!/usr/bin/env python3
"""golden_set.py: regression harness for a PR review gate."""

import json
import subprocess

def classify(review_output: str) -> str:
    """Map reviewer output to a gate verdict."""
    return "flag" if "FLAG" in review_output else "pass"

def run_gate(patch_path: str, context: str) -> str:
    """Run the repository's real review command."""
    cmd = ["/repo/bin/review_gate.sh", patch_path]
    if context:
        cmd += ["--context", context]
    return subprocess.check_output(cmd, text=True).strip()

def evaluate(cases: list[dict]) -> dict:
    tp = fp = tn = fn = 0
    for case in cases:
        verdict = classify(run_gate(case["patch"], case.get("context", "")))
        if verdict == "flag":
            if case["truth"] == "bad":
                tp += 1
            else:
                fp += 1
        else:
            if case["truth"] == "good":
                tn += 1
            else:
                fn += 1
    return {
        "true_positive": tp,
        "false_positive": fp,
        "true_negative": tn,
        "false_negative": fn,
        "precision": tp / (tp + fp) if tp + fp else 0,
        "recall": tp / (tp + fn) if tp + fn else 0,
    }

if __name__ == "__main__":
    cases = json.load(open("golden.json"))
    print(json.dumps(evaluate(cases), indent=2))

The script expects one small JSON file. truth is the label from history. context is any recorded failure information. The corpus below shows the shape.

{
  "cases": [
    {
      "id": "gh-482",
      "patch": "examples/gh-482.diff",
      "truth": "bad",
      "context": "Sets the retry timeout to 5s. The issue reports hangs at 30s."
    },
    {
      "id": "gh-491",
      "patch": "examples/gh-491.diff",
      "truth": "good",
      "context": "Parses the user-agent header before auth."
    }
  ]
}

Precision is the share of flagged patches that are truly bad. Recall is the share of bad patches that got flagged. High recall with low precision equals noise. High precision with low recall equals silence. Report both numbers after every run.

Start with ten cases. Grow to fifty as the project matures. Label from the merge record, never from memory.

Build a golden set in five steps

1. Collect 20 to 50 patches from merged pull request history.
2. Label each patch good or bad from the merge record.
3. Run python3 golden_set.py after every prompt change.
4. Reject any prompt that increases false positives.
5. Schedule a weekly cron job and archive every report.

Treat the script as a starting point. Adjust the command and the classification rule for your gate. Verify it once against a known case before trusting the output. Each run returns four counts and two ratios. Archive the report next to the commit.

Where a free-model gate helps or hurts

Pull request type	Gate value	Why
Small fix plus failing test	High	The reproducer anchors the verdict
Dependency bump	Medium	Semantic drift is easy to miss
Large refactor	Low	The diff tells only part of the story
Generated diff	Low	The verdict adds no real signal
Patch with private design context	Low	The model lacks the rationale

Use the table as a triage filter. Route promising patches to the gate. Route the rest straight to human review.

Who should not use this workflow

Avoid it when the repository has no test suite. Avoid it when project code cannot leave the organization. Avoid it when maintainers will not read verdicts. An unread review gate is theater. Free model behavior can also change without notice. The golden set catches drift after the fact. It cannot predict the next change.

A cheap place to start

A golden set is storage, not magic. It turns model updates from surprises into reports. The harness is product-agnostic and fits any command-line reviewer.

MonkeyCode's current offer includes 10 million free tokens and a free server, as of this writing. That combination runs the weekly harness for a small corpus without cloud setup. It is a practical first step for maintainers who want evidence before automation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Clone the script. Feed it your own merged history. The first report takes less than an hour. Then the gate earns its place in CI.