Your AI Reviewer Is Untested Code. Fix That First.

You merged an AI reviewer last month. Did it ship with tests? Most AI reviewers ship untested — that is the hole in the workflow.

DEV ran this debate this week. Everyone got promoted to reviewer. Nobody tested the reviewer.

Here is my position. Review comments are code. Untested code.

You do not merge untested code. Why trust untested feedback? Fix the feedback first.

The fix is small. Twenty PRs, one script, a scorecard. This post builds all three.

Run the experiment with free model access and an always-on box. MonkeyCode, the open-source project behind this outreach, offers both: a free 10M-token model allowance and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Start with a golden set

A golden set is history with labels. Take twenty merged PRs. Keep the human comments that caught real bugs.

Label each finding:

• critical — would have caused a bug or a rework
• improvement — real and useful, but not blocking
• nit — style, taste, formatting
• ignore everything else

Here is one case:

[
  {
    "pr": "fix fee-calc divide-by-zero",
    "diff": "- return total / count;\n+ return count == 0 ? 0 : total / count;",
    "labels": {
      "critical": ["Guard count == 0 before dividing"],
      "improvement": [],
      "nit": ["Add a comment explaining the fallback"]
    }
  }
]

The diff is tiny. The lesson is not. Twenty of these give your reviewer a baseline.

Collect the PRs without the pain

Export review history once. Reuse the set until the codebase shifts enough to rot it.

gh pr list --state merged --limit 20 --json number,title
gh pr diff 1234 > golden/1234.diff

Paste the human review thread next to each diff. Label it once. Audit the labels before you trust the scores.

Build the harness

The harness replays each diff through your reviewer. Then it compares the output against the golden set.

# review_harness.py — score an AI reviewer against a golden set
import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["MODEL_API_KEY"],
    base_url=os.environ.get("MODEL_BASE_URL"),
)

REVIEW_PROMPT = """You are a senior code reviewer.
Given a diff, list findings as critical, improvement, or nit.
Skip praise, skip generic advice, skip LGTM."""

GENERIC_PHRASES = [
    "consider adding",
    "looks good",
    "lgtm",
    "maybe refactor",
    "great work",
]

def review(diff: str) -> str:
    completion = client.chat.completions.create(
        model=os.environ.get("REVIEW_MODEL", "your-model"),
        messages=[
            {"role": "user", "content": REVIEW_PROMPT + "\n\n" + diff}
        ],
    )
    return completion.choices[0].message.content or ""

def score_raw(raw: str, golden: list[str]) -> dict:
    lowered = raw.lower()
    hits = [g for g in golden if g.lower() in lowered]
    recall = len(hits) / max(len(golden), 1)
    generic = sum(p in lowered for p in GENERIC_PHRASES)
    return {
        "recall": round(recall, 2),
        "generic_hits": generic,
        "word_count": len(raw.split()),
        "found": hits,
    }

if __name__ == "__main__":
    cases = json.load(open("golden_set.json"))
    report = []
    for case in cases:
        raw = review(case["diff"])
        truth = case["labels"]["critical"] + case["labels"]["improvement"]
        report.append({"pr": case["pr"], **score_raw(raw, truth)})
    print(json.dumps(report, indent=2))

It reports three numbers:

• recall — how many real findings the reviewer reproduced
• generic hits — how many empty phrases leaked into the output
• word count — confidence without signal

The script uses the OpenAI SDK contract. If your model access speaks that contract, it runs unchanged. If not, patch review(). That function is the entire integration point.

Read the scorecard, not the vibes

Vibes fail. Scores do not. Use these thresholds as a starting point:

Recall	Generic hits	Verdict
< 0.3	any	Drop the reviewer. It is noise with formatting.
0.3–0.6	> 2	Keep it in suggest mode. Require human confirmation.
>= 0.6	<= 1	Let it post findings. Keep a revert path.

Under 0.3 recall? Your reviewer is polite, confident, and useless. It burns tokens and attention. Remove it.

Between 0.3 and 0.6? Every finding needs a human confirm. That is a workflow, not an autopilot.

Above 0.6 with low generic noise? Let it post directly. Keep a blame-free culture and a fast revert path.

Thresholds are yours to tune. The point is that you tune them.

Schedule it like a regression test

A one-off score is a snapshot. A scheduled score is a regression test.

30 9 * * 1 cd /opt/review-harness && python review_harness.py > scorecard.md

Run it every Monday. Diff the scorecard against last week. Dropping recall means your prompt or your codebase drifted. Catch drift in review, not in production.

This is where the free server option fits. It gives the harness a permanent home. No laptop, no "I forgot to run it". Weekly artifacts instead.

Honest limitations

This is a smoke test, not a benchmark.

• Twenty PRs is a small sample. Triple it when you can.
• Phrase matching misses semantic hits. A reworded finding counts as a miss.
• Garbage labels become garbage scores. Audit your labels.
• No prompt here is sacred. The harness exists to test yours.

Who should skip this? Solo developers with tiny PR volume. Teams whose review comments are mostly nits. Teams without saved review history. There is no baseline to score.

Close

AI review does not need more trust. It needs measurement.

Start with one golden set. Run the harness once. Let the score surprise you. That surprise is cheaper than a production incident.

Measurement is the cheapest insurance you will ever buy. Your future reviewers will thank you.