Most AI review setups cannot tell you whether the reviewer is any good. This post shows a 100-line harness that scores AI code review comments against seeded bugs, built on MonkeyCode's free model access and free server option. The entire experiment costs nothing but a weekend.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Why test the reviewer?
Recent developer discussions keep circling the same gap: AI promoted every developer to reviewer, but nobody tested the reviewer. A PR review comment looks plausible even when it misses the actual bug. The cheapest way to measure review quality is a fixture-based harness.
You plant a known bug in a diff, run the AI reviewer, and check whether its comments mention the bug. Repeat with ten diffs and you get a signal. That signal becomes even more useful when you tweak prompts and watch the score move.
MonkeyCode is an open-source AI coding toolkit. The project ships a server that you can run locally or use through their free server option. As of August 2026, the free tier includes 10 million tokens, which is plenty for a small review-scoring experiment.
What I built
A Python script called review_scorer.py and a folder named prs/. Each file in prs/ is a JSON fixture with three fields:
{
"id": "pr-007",
"diff": "--- a/auth.py\n+++ b/auth.py\n@@ -10,7 +10,7 @@ def login(user):\n- if token is None:\n+ if token = None:\n raise ValueError(\"missing token\")",
"expected_flags": ["assignment in condition", "token handling"]
}
The script sends the diff to the MonkeyCode review server, collects the comments, and checks which seeded issues were mentioned.
The scoring core
Three functions do the heavy lifting:
import json
import os
import requests
from pathlib import Path
def run_review(diff: str, endpoint: str) -> list[str]:
response = requests.post(
endpoint,
json={"prompt": build_prompt(diff)},
timeout=int(os.getenv("REVIEW_TIMEOUT", "60"))
)
response.raise_for_status()
return response.json()["comments"]
def score_comments(comments: list[str], expected: list[str]) -> tuple[int, int]:
hits = sum(1 for e in expected if any(e.lower() in c.lower() for c in comments))
return hits, len(expected)
def run_suite(prs: Path, endpoint: str) -> None:
for fixture in prs.glob("*.json"):
data = json.loads(fixture.read_text())
comments = run_review(data["diff"], endpoint)
hits, total = score_comments(comments, data["expected_flags"])
print(f"{data['id']}: {hits}/{total} seeded bugs caught")
The code above is a simplified prototype. The prompt format and response schema will change depending on how you expose MonkeyCode in your environment. The point is the metric loop: change prompt, rerun, compare.
The rubric
I used a simple binary check: does the review comment contain any of the seeded flags? This is crude but effective.
| Seed | Caught | Missed |
|---|---|---|
| assignment in condition | ✅ | ❌ |
| hardcoded secret | ✅ | ❌ |
| missing boundary check | ❌ | ✅ |
You can extend the rubric with synonyms, severity parsing, or a second LLM call that judges whether the comment targets the same line. That turns the script from a toy into a real prompt-tuning instrument.
What got cut
- No web UI. CLI output was enough.
- No multi-turn conversation. This harness sends one diff and reads one response.
- No model comparison. I only used the free model.
- No human reviewer baseline. That would require a separate controlled study.
Every one of those cuts kept the project inside a single weekend.
What the harness teaches you
The first run of any fixture will likely surprise you. The free model will sometimes catch a bug you thought was obscure. Then it will miss a simpler bug right next to it. That variance is normal.
Run each fixture three times and take the median score. The harness makes that easy because the whole suite takes a few minutes on MonkeyCode's free server.
The biggest win is prompt iteration. For example, adding a line to the prompt that says "report line numbers and severity" usually changes recall more than changing the temperature. Without a scoring harness, you would judge that change by vibes. With it, you get a number.
Limitations
This is not a benchmark. The sample size is small, and the seeded issues are written by one person. The keyword matcher does not verify whether a comment is actually correct; it only checks whether the expected phrase appears.
AI reviewers are nondeterministic. Different runs produce different comments. Free tiers often have rate limits, and the exact token allowance may change. Always check the project documentation before running a large suite.
Who should not use this
Do not turn this harness into a formal review gate. It is a personal quality-checking tool, not an auditable process.
If your organization requires deterministic, compliant, or legally defensible code review, this approach is too loose. Payment code, medical software, and safety-critical systems need human review plus static analysis. A keyword-based score is not a substitute for real review.
Final notes
The harness lives in one folder with ten JSON fixtures and a single script. Adapting it to any AI review endpoint takes less than thirty minutes.
If you have been shipping AI review comments without measuring them, start with a fixture-based harness. With MonkeyCode's free tier, the cost is zero, and the insight is worth the weekend.
Top comments (0)