Your AI Reviewer Is Untested: A 10-Bug Calibration Harness on a Free Server

This week's loudest DEV discussion: AI promoted every developer to reviewer. The uncomfortable follow-up: nobody tested the reviewer. This article fixes that gap. You will build a 10-bug calibration set. You will measure your AI reviewer's recall and precision. You will run the harness on a free server. Core conclusion: an untested AI reviewer is a liability. A 45-minute harness turns it into a measurable tool.

We grade patches, not reviewers

Teams evaluate AI-generated code constantly. They run tests. They check coverage. They rarely evaluate the AI reviewer itself. That asymmetry is dangerous. A bad reviewer passes bad patches. A noisy reviewer buries good ones. Both costs show up later as debugging time.

This is not a model benchmark. This is a workflow gate. It answers one question: can I trust this reviewer with my next pull request?

The hypothesis

One hypothesis drives this spike. A free-tier AI reviewer catches injected bugs at a measurable rate. We test it with ten known bugs. We measure recall and precision. Then we decide: ship it, constrain it, or kill it.

The calibration set

Ground truth beats vibes. This set injects ten bugs into five small Python files. Each file has two bugs. Each bug has a class and a line number.

File	Bug class	Line	Severity
cart.py	off-by-one in tier boundary	12	high
cart.py	mutable default argument	31	medium
cache.py	check-then-set race condition	22	high
cache.py	wrong TTL unit	47	medium
query.py	SQL injection via f-string	18	critical
query.py	unclosed connection	54	medium
retry.py	bare except swallows KeyboardInterrupt	9	medium
retry.py	retry loop never terminates	26	high
auth.py	non-constant-time token compare	15	high
auth.py	missing expiry check	39	critical

Why these ten? They are common. They are detectable by static reasoning. They map to real incidents. The set is small by design. Small means auditable. Small means cheap to run weekly.

Scoring rules

A finding is a true positive when two things hold. The reported line is within three lines of ground truth. The bug class matches. Everything else is a false positive.

• Recall = true positives ÷ 10 injected bugs
• Precision = true positives ÷ all findings

These two numbers tell different stories. High recall, low precision: the reviewer is noisy. Low recall, high precision: the reviewer is blind but honest.

The harness

The scoring logic is the artifact. It is pure Python. It uses the standard library only. It runs anywhere, including a free server.

import json
import sys
from pathlib import Path

GROUND_TRUTH = {
    "cart.py": [("off_by_one", 12), ("mutable_default", 31)],
    "cache.py": [("race_condition", 22), ("wrong_expiry", 47)],
    "query.py": [("sql_injection", 18), ("unclosed_connection", 54)],
    "retry.py": [("broad_except", 9), ("infinite_retry", 26)],
    "auth.py": [("non_constant_time", 15), ("missing_expiry_check", 39)],
}

TOLERANCE = 3

def review_file(path: Path) -> list[dict]:
    """Adapter for your AI reviewer. Check current MonkeyCode docs for the endpoint."""
    prompt = (
        'Review this Python file for bugs. '
        'Return JSON only: [{"line": int, "issue": str}]'
    )
    # Insert the API call here. Keep this function thin.
    return []

def score(findings: list[dict], truth: list[tuple[str, int]]) -> dict:
    tp = 0
    matched = set()
    for finding in findings:
        line = finding.get("line", -1)
        for bug_class, truth_line in truth:
            if bug_class in matched:
                continue
            if abs(line - truth_line) <= TOLERANCE:
                tp += 1
                matched.add(bug_class)
    total = len(truth)
    recall = tp / total if total else 0.0
    precision = tp / len(findings) if findings else 0.0
    return {
        "recall": round(recall, 2),
        "precision": round(precision, 2),
        "true_positives": tp,
        "false_positives": len(findings) - tp,
    }

def main() -> int:
    results = {}
    for path, truth in GROUND_TRUTH.items():
        findings = review_file(Path(path))
        results[path] = score(findings, truth)
    print(json.dumps(results, indent=2))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Note the adapter. The API contract changes. The scoring logic does not. That separation is the point. Swap the model provider without touching the math.

Run it on a free server

The harness needs no GPU. It needs no local daemon. It needs one scheduled job. That is exactly what a free server is for.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source AI coding project. Its current free tier includes free model access and a free server option. The free tier currently includes 10 million tokens. That budget is enough to run this harness repeatedly. Host the scoring job on the free server. Point the adapter at MonkeyCode's model access. Store results as JSON. Schedule a weekly run. Done.

The decision table

One run tells you nothing. Three runs tell you a pattern. Score against this table.

Recall	Precision	Verdict
≥ 0.7	≥ 0.5	Ship as first-pass reviewer
0.4–0.7	≥ 0.3	Constrain to known bug classes
< 0.4	any	Kill. Review manually.
any	< 0.3	Too noisy. Triage cost exceeds value.

The verdict is a gate, not a grade. A failing reviewer is still useful. It just needs a human in the loop.

Track drift

Model versions change. Free tiers change. Your calibration set does not. Run the harness weekly. Store the scores. Watch the trend. A recall drop from 0.8 to 0.5 is a signal. Investigate before it reaches production. Treat it like a dependency update. Check it on a schedule.

Practice: disclose the reviewer

DEV just introduced AI disclosure tools for posts. The same honesty belongs in review threads. When a review comment comes from an AI reviewer, say so. Labeling builds trust. It also makes your calibration numbers credible.

Limitations

The set has ten bugs. Real code has thousands. Injected bugs are not organic bugs. They lack context and history. The harness measures bug-catching only. It ignores style and security depth. It ignores explanation quality. A correct line with a wrong reason still counts as a true positive. That is generous. Treat the score as a lower bound, not an upper bound.

Who should not use this

Skip this if you have no test suite. Skip this if you review fewer than ten patches a week. Skip this if you need security-grade review. This harness is a triage tool. It is not a certification.

The 45-minute plan

1. Copy the five files and the harness. (10 minutes)
2. Fill the adapter with the current MonkeyCode API call. (10 minutes)
3. Run once and record scores. (5 minutes)
4. Apply the decision table. (5 minutes)
5. Schedule weekly runs on the free server. (15 minutes)

That is one sitting. That is one hypothesis. That is a go/no-go decision.

If you want your own reviewer's numbers, the free tier is enough to start. Clone the harness. Inject your own bugs. Post your recall and precision.