Your AI Reviewer Needs a Baseline: A Zero-Cost Patch Audit Loop

Reviewing is the new bottleneck. Generated code passes through more reviews than ever, and the reviewer's own accuracy stays unmeasured.

A model that generates plausible code is only useful when its review catches the defects that matter. That property is measurable. It is rarely measured. The current round of AI discussion has plenty of opinions about the reviewer's new role and almost no numbers attached to it.

This article walks through a reproducible patch audit loop. It runs on a free server, spends tokens from a free allowance, and produces a score that tells a solo founder when to trust the AI review and when to read the diff alone.

The Review Debt Problem

Generated code lands faster than reviewed code. The backlog of unmerged patches grows, and the person doing the review — usually the same person who wrote the prompt — has less context than the model had when it generated the patch.

The result is a trust gap. A reviewer that misses a null check on line 41 is not a minor flaw; it is a shipped bug. Treat the reviewer as a component, and components get test suites.

What a Reviewer Audit Should Measure

A useful audit tracks five numbers:

Metric	Definition	Target
Catch rate	seeded bugs the reviewer flags	as high as possible
Precision	flagged issues that are real	avoid alert fatigue
Patch coverage	files that receive at least one comment	broad, not concentrated
Latency	minutes per review run	fits a nightly schedule
Token cost	tokens consumed per full run	fits the allowance

Catch rate alone is not enough. A reviewer that flags every line has a perfect catch rate and useless precision. The audit needs both numbers.

A Minimal Harness in 50 Lines

The harness does three things. It extracts a diff between two commits, sends the diff to whatever review command the tool exposes, and compares the output against a ground-truth case file curated from your own git history.

# reviewer_audit.py
import argparse
import json
import subprocess
from pathlib import Path


def diff_for(repo: str, base: str, head: str) -> str:
    cmd = ["git", "diff", base, head, "--", "*.py", "*.js", "*.ts"]
    proc = subprocess.run(cmd, cwd=repo, capture_output=True, text=True, check=True)
    return proc.stdout


def run_review(patch: str, review_cmd: str) -> str:
    """Wire this to the CLI your review tool exposes."""
    proc = subprocess.run(
        review_cmd.split(), input=patch, capture_output=True, text=True
    )
    return proc.stdout


def judge(text: str, case: dict) -> dict:
    if not case.get("bug"):
        # Clean control case: any output counts as a false positive.
        return {"caught": None, "false_positive": bool(text.strip())}
    text = text.lower()
    bug_hit = case["bug"].lower() in text
    place_hit = case["reporter"].split(":")[0] in text
    return {"caught": bug_hit and place_hit, "false_positive": None}


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--repo", required=True)
    parser.add_argument("--cases", required=True)
    parser.add_argument("--review-cmd", required=True)
    args = parser.parse_args()

    cases = json.loads(Path(args.cases).read_text())
    results = []
    for case in cases:
        patch = diff_for(args.repo, case["base"], case["head"])
        output = run_review(patch, args.review_cmd)
        results.append({"name": case["name"], **judge(output, case)})

    bugged = [r for r in results if r["caught"] is not None]
    clean = [r for r in results if r["caught"] is None]
    caught = sum(r["caught"] is True for r in bugged)
    false_positives = sum(r["false_positive"] is True for r in clean)

    print(json.dumps(results, indent=2))
    if bugged:
        print(f"catch rate: {caught}/{len(bugged)} ({caught / len(bugged):.0%})")
    else:
        print("catch rate: no bugged cases")
    if clean:
        print(f"false positives: {false_positives}/{len(clean)}")
    else:
        print("false positives: no clean cases")


if __name__ == "__main__":
    main()

The case file maps real history to expected findings:

[
  {
    "name": "null-attributes-in-parser",
    "base": "a1b2c3",
    "head": "d4e5f6",
    "bug": "attributes can be None",
    "reporter": "parser.py:41"
  },
  {
    "name": "clean-config-refactor",
    "base": "f7a8b9",
    "head": "c0d1e2",
    "bug": ""
  }
]

Step 1 — Build the Ground-Truth Set

1. Pick five to ten commits that fixed real bugs. Use the parent of the fix as base and the fix commit as head; the issue title or regression test name provides the bug phrase.
2. Record the file where the fix landed as reporter. The judge requires both the bug phrase and the file name, which filters out vague "something looks wrong" output.
3. Add two clean commits that change no behavior. These control cases measure false positives.
4. Run the harness once and inspect every result manually. The first run validates the case set, not the reviewer.

Ground truth comes from history, not from the model. That keeps the audit honest.

Step 2 — Schedule It on a Free Server

An audit that runs once is a checkpoint. An audit that runs weekly is a regression test, and weekly runs need a host that is always on and costs nothing.

The open-source MonkeyCode project covers that part of the equation. Its free server option hosts scheduled jobs without a paid VM, and its free model access absorbs the repeated review calls. The 10,000,000-token free allowance, documented earlier on this account in the token-ledger post, fits a weekly baseline on a small repository.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A single cron entry is enough:

0 6 * * 1 cd /srv/patch-audit && python3 reviewer_audit.py --repo /srv/app --cases cases.json --review-cmd "/usr/local/bin/review-cli" >> audit.log 2>&1

The harness only requires a text-in, text-out command. CLI names differ between tools; the contract does not change.

Step 3 — Read the Numbers Like an Engineer

Catch rate	Precision	Action
80% or higher	high	approve routine generated patches, read the rest
50–80%	high	review everything it misses, watch for a pattern
below 50%	low	stop using that reviewer for the repository

The pattern matters more than the headline number. If the reviewer always misses database migrations, add a migration category and re-test the same cases after configuration changes.

Accept the limits. A reviewer with a known miss profile is safer than a reviewer with an unknown one.

Limitations and Who Should Skip This Workflow

The harness measures only what the case set seeds. Five trivial cases produce a meaningless score; the cases must be real fixes with real failure modes.

The workflow needs git history with fix commits. A greenfield project cannot build a ground-truth set until a few bugs have shipped and been fixed.

Token allowances and free servers have constraints. The 10,000,000-token allowance is real but finite: weekly runs on a small repo fit, hourly runs on a monorepo do not. The free server option is a starting point, not a guarantee of production uptime.

Skip the workflow in three situations: when the team ships fewer than ten patches per week, when a human reads every diff anyway, or when compliance requires a traceable approval chain. An audit loop adds process, and process is only worth its cost when review volume is high.

The Point of a Baseline

Ship today, keep the bill at zero, accept the limits.

A baseline does not make the AI reviewer good. It makes the reviewer predictable, and predictability is what lets a solo founder merge a generated patch at 6 a.m. and still sleep.

Anyone who wants to run this loop against a real repository can use the same free model access and free server option in MonkeyCode.