I Benchmarked a Free AI Code Reviewer. It Caught 23 of 40 Bugs and Invented 6.

Everyone is a reviewer now. The code gets written by a model, and you get to decide whether it's any good. Who decides whether you're any good?

So I built a code-review bot on MonkeyCode, an open-source agent with free model access and a free server slot, then ran it against 40 seeded defects with known ground truth. It caught 23. It missed 17. It invented 6 findings that looked completely real. The last number is the one that matters.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Real PRs are terrible benchmark data. You don't know the ground truth, so every finding becomes a debate instead of a measurement. I went the other way: a deliberately boring Python codebase, one injected defect per branch, one diff per defect. 40 diffs, 8 categories, one known answer each.

The categories were the boring ones. Missing None checks, off-by-one errors, missing awaits, SQL injection, wrong exception types, dead code, hardcoded secrets, race conditions. Each diff touched one file, one function, one line. No cross-file context required. I was being generous. Forty is not a big number. It's a weekend number — big enough to see patterns, small enough to label by hand.

The harness was about sixty lines:

# bench_reviewer.py — harness skeleton, not production code
import json, time
from pathlib import Path

CASES = json.loads(Path("defects.json").read_text())

def review(diff: str) -> list[dict]:
    # Calls the free model through MonkeyCode's free tier.
    # Returns findings: [{"file": str, "line": int, "message": str}]
    ...

def score(case: dict, findings: list[dict]) -> tuple[bool, int]:
    hit = any(f["line"] == case["line"] for f in findings)
    false_positives = [f for f in findings if f["line"] != case["line"]]
    return hit, len(false_positives)

for case in CASES:
    diff = Path(case["diff"]).read_text()
    t0 = time.time()
    findings = review(diff)
    latency = time.time() - t0
    hit, fps = score(case, findings)
    print(f"{case['id']}: {'HIT' if hit else 'MISS'} fp={fps} {latency:.1f}s")

The prompt was three sentences:

PROMPT = """Review this diff. Report only defects you are certain about.
For each finding, give file, line, and a one-sentence fix.
If the diff is clean, say "no findings".
{diff}"""

And defects.json was just a list of IDs, diff paths, and the line where the truth lived:

[
  {"id": "null-01", "diff": "diffs/null-01.diff", "line": 14},
  {"id": "race-02", "diff": "diffs/race-02.diff", "line": 31},
  {"id": "secret-03", "diff": "diffs/secret-03.diff", "line": 9}
]

I ran it overnight on the free server. Cost: zero. The server queued 40 jobs and never blinked. That's not a scale benchmark. It's a "good enough for a weekend project" benchmark.

Category	Caught	Missed
Missing None check	5	0
Missing await	4	1
Wrong exception type	4	1
SQL injection	3	2
Off-by-one	3	2
Dead code	2	3
Hardcoded secret	2	3
Race condition	0	5

23 of 40. The free model was excellent at defects that look like typos. null-01 was a missing None check on a config value. The bot said: config["timeout"] will raise KeyError when the key is absent — use .get(). Correct, specific, actionable. Missing awaits and wrong exception types were nearly as good: four out of five, every time.

I counted a finding as a hit only if the line matched. Close counts for horseshoes, not for benchmarks. A finding that said "this function has a problem" without naming the line went into the false-positive bucket, because a reviewer who can't tell you where to look isn't reviewing — it's hinting.

Then the race conditions. Zero out of five. A race isn't a line-level defect; it's a two-file thought. The model didn't say "I can't tell from this diff." It said nothing. Silence is a finding too, but only if the harness knows to ask.

The scary part was the six inventions. One finding pointed at orders.py:14 and claimed total may be used before assignment when discount is None. The file has no discount variable. Line 14 is a return statement. The finding was grammatically perfect and factually empty. A human reviewer would open the file, check the symbol, and close it again. That's not a review. That's a tax.

Latency was fine. Median 9 seconds per review, p95 at 31. The bottleneck wasn't the model or the server. It was my prompt, which had no repo context, and my pipeline, which had no verification step.

So I added one. A deterministic post-filter: every finding must reference a symbol that actually exists in the diff or the file. It killed four of the six hallucinations and didn't touch a single true positive. The model didn't get smarter. The pipeline got honest.

The lesson: free model, free server, and a 10M token allowance are enough to run a review gate on single-file diffs. That's the allowance and the server slot on MonkeyCode's free tier at the time of writing, and it held up overnight. But the model is a component, not a reviewer. The harness is the reviewer. What's the point of a judge who won't tell you when it doesn't know?

Who should not use this approach: teams that need cross-file architectural review. Teams that paste findings into PR comments without checking them. Anyone who treats a free tier as a production SLA. The free server worked for me. That is not a promise about your traffic.

If you want to reproduce this, the whole thing is sixty lines plus a JSON file. MonkeyCode's free tier is enough to run it. The number you get back — 23 of 40, 6 invented — will be more useful than any model's opinion about itself.