Your AI Code Reviewer Is Untested: 45 Lines to Score It Before Merge Rights

Every developer is a reviewer now. The reviewer itself? No test suite.

Picture the scene. A PR lands. The AI reviewer comments "Looks good, the null check is handled." The null check was planted in that diff to be caught. The team merges.

I built a 45-line harness that stops that scene. It scores any OpenAI-compatible endpoint against seeded bugs. If the reviewer misses too many, it exits 1. Fail-closed. No merge rights without evidence.

The harness is endpoint-agnostic. I pointed it at MonkeyCode's open-source project, which offers free model access and a free server option — no credit card needed to run the experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why test a reviewer like code

A reviewer is a tool. Tools get tests. Your linter has fixtures. Your formatter has snapshots. Your AI reviewer gets merge buttons.

The failure is quiet. The review is confident, formatted, and wrong. Nobody checks whether the reviewer can see the bugs that matter to your team.

I wanted three numbers before trusting any review: precision, recall, determinism. This harness gives me the first two. A saved transcript gives me the third.

The fixture: one diff, one planted bug

A fixture is a folder with two files:

fixtures/
  01-null-check/
    diff.patch
    bugs.json
  02-missing-await/
    diff.patch
    bugs.json

diff.patch is a real diff from your repo with one bug seeded in. bugs.json is the ground truth:

[{"file": "app.py", "line": 12, "issue": "calls .strip() on a value that can be None"}]

Five fixtures minimum. Ten is better. Use your own diffs, not tutorial examples. Your team's bug patterns are the ones that matter.

The harness

Python, one dependency (httpx), 45 lines:

#!/usr/bin/env python3
"""Score an AI code reviewer on seeded bugs. Exit 1 = fail closed."""
import json, os, sys
from pathlib import Path

import httpx

ENDPOINT = os.getenv("REVIEW_ENDPOINT")
API_KEY = os.getenv("REVIEW_API_KEY")
FIXTURES = Path(os.getenv("FIXTURES_DIR", "./fixtures"))
RECALL_FLOOR = float(os.getenv("RECALL_FLOOR", "0.8"))
PRECISION_FLOOR = float(os.getenv("PRECISION_FLOOR", "0.5"))

PROMPT = """Review this diff. Reply with JSON only:
[{"file": "...", "line": 0, "severity": "error", "issue": "..."}]
Report real bugs only. No style nits. Empty array if clean.

DIFF:
{diff}"""

def ask(diff: str) -> list:
    r = httpx.post(
        ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"messages": [{"role": "user", "content": PROMPT.format(diff=diff)}]},
        timeout=120,
    )
    r.raise_for_status()
    return json.loads(r.json()["choices"][0]["message"]["content"])

def main() -> int:
    fixtures = sorted(FIXTURES.iterdir())
    if not fixtures:
        print("FAIL gate 1: no fixtures")
        return 1
    tp = fp = fn = 0
    for fx in fixtures:
        diff = (fx / "diff.patch").read_text()
        truth = {(b["file"], b["line"]) for b in json.loads((fx / "bugs.json").read_text())}
        found = {(f["file"], f["line"]) for f in ask(diff)}
        tp += len(truth & found)
        fp += len(found - truth)
        fn += len(truth - found)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    print(f"precision={precision:.2f} recall={recall:.2f}")
    if recall < RECALL_FLOOR or precision < PRECISION_FLOOR:
        print("FAIL: below floor. Keep review human-only.")
        return 1
    print("PASS: reviewer may comment, never approve.")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Run it:

export REVIEW_ENDPOINT="<your-compatible-endpoint>"
export REVIEW_API_KEY="$KEY"
python test_reviewer.py   # 0 = pass, 1 = fail closed

Why 45 lines? Because a reviewer test that needs a framework never gets run. This one runs anywhere Python exists.

Six gates before merge rights

Copy these into your repo. Each gate produces one artifact.

1. Fixture coverage. At least five diffs with planted bugs. Artifact: the fixtures/ folder, committed.
2. Recall floor. Catch at least 80% of planted bugs. A reviewer that misses a planted null-check is doing vibes, not review.
3. Precision budget. At least 50% of findings must be real. Below that, the team learns to ignore the bot. Ignored bots are worse than no bot.
4. Determinism sample. Run one fixture three times. Findings should be mostly stable. Save the transcripts.
5. Fail-closed wiring. Timeout, bad JSON, or HTTP 500 means exit 1. The bot comments "unreviewed," not a green check.
6. Override log. Record every human "dismiss." If two weeks of overrides show the bot is noise, turn it off. That's the exit criterion.

Where MonkeyCode fits

The harness does not care which endpoint it points at. That is the point.

I used MonkeyCode's free model access and free server option because both fit a solo-builder budget. The free tier currently includes a 10M token allowance, enough for several hundred fixture runs. No version numbers, no benchmark claims — the harness measures what your diffs actually trigger.

Free endpoints are shared. They can be slow. The 120-second timeout handles that: when the endpoint dies, the harness fails closed. That is a feature, not a workaround.

Who should skip this

Skip this checklist if any of these describe you:

• No CI at all. Run the harness locally, but don't wire it to merge.
• You want AI to replace human review. This only tests whether it can comment.
• Your team dismisses every bot finding. Fix the culture first. A faster bot won't help.

Limitations

Line-based matching is crude. The reviewer says "line 12 is null-unsafe." You match on file plus line. Real reviewers describe bugs in prose. Fine. This fixture is a canary, not a full benchmark.

Free tiers change. Quotas move. The harness re-measures every run. That's why it exists.

Run it before you trust the next review. Expect the first run to hurt. That's the harness earning its keep.

What bug class does your team miss most in review? I'm building fixture number six around that one.