A Free-Resource Kit for Evaluating AI Code Reviewers: Prompt, Rubric, and Failure Modes

Evaluating an AI code reviewer doesn't require a paid enterprise plan or a production-scale dataset. A single take-home task, built around a PR with deliberately injected defects, produces enough signal to separate a useful reviewer from a talkative one. This article provides that complete kit, along with a concrete way to run the evaluation on a free server and free model access.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a take-home task beats a demo

A live demo usually shows only the happy path: the model catches the obvious off-by-one, everyone nods, and no one tests for false positives or hidden regressions. A take-home task, in contrast, gives the evaluator a fixed input and a scoring rubric, so every candidate faces identical conditions. The benchmark becomes reproducible across models, prompts, and infrastructure choices.

The strongest take-home tasks include a prompt that asks the reviewer to list concrete line-level issues, a rubric that penalizes both missed and invented defects, and a reference solution that defines what a correct answer looks like. Without those three pieces, the evaluation collapses into subjective impressions. The following section lays out a task that takes about two hours of an evaluator's time and can be repeated as often as needed.

The take-home task definition

Here is the prompt that should be given to any AI reviewer candidate, exactly as written:

You are reviewing a pull request in a Python microservice. Analyze the diff below and report only defects that are real, deterministic, and fixable. For each finding, provide the line number, a short explanation, and the exact replacement code. Do not comment on style, naming, or missing tests unless the missing test directly allows a listed defect to reach production. If you are not certain a finding is real, leave it out.

# service.py (before)
def calculate_discount(price, multiplier=1.0):
    if price < 0:
        raise ValueError("price cannot be negative")
    return price * multiplier

# service.py (after)
def calculate_discount(price, multiplier=1.0):
    if price < 0:
        raise ValueError("price cannot be negative")
    if multiplier < 0:
        multiplier = 0
    return price * multiplier - 1

In the same PR, config.yaml changed timeout: 5 to timeout: 15 because the author believed a longer timeout would fix a race condition. The code that uses the timeout does not read it from config.yaml; it is hardcoded. List this as a finding only if it qualifies under the rules above.

The prompt forces the candidate to distinguish between a real bug and a configuration change with no code impact. It also removes the temptation to pad the answer with style suggestions. The missing null check on multiplier is a valid finding, while the - 1 operation silently changes the output for every valid input, so it should be flagged as a regression. The config.yaml change is a red herring because the field is never read.

The scoring rubric

Each candidate answer is scored against the reference solution below. The rubric has four dimensions, each weighted equally:

• Detection rate (25%): The fraction of the three real defects that were correctly identified with lines. Missing any of them drops this score significantly.
• Precision (25%): The fraction of reported findings that are actually defects. Reporting the config change as a bug counts as a false positive.
• Actionability (25%): The clarity of the suggested replacement code. Vague suggestions like "consider a bounds check" earn partial credit, while exact patches earn full credit.
• Conciseness (25%): The answer is stripped of generic caveats and irrelevant commentary. Overlong responses lose points even when technically correct.

A candidate that scores below 60% should not be trusted with production reviews, regardless of how confident it sounds. This threshold is deliberately strict because a chatty reviewer that invents false defects is worse than one that stays silent.

Reference solution

The reference answer contains exactly three findings:

1. Line 4 (if multiplier < 0: multiplier = 0): Silently clamping negative multipliers changes the business rule. Reality is undefined; a negative multiplier should either be rejected or documented. Replacement: if multiplier < 0: raise ValueError("multiplier cannot be negative").
2. Line 5 (return price * multiplier - 1): Subtracting one from every result is an unrequested behavior change that breaks any existing price calculation. Replacement: return price * multiplier.
3. Missing null check on price: If price is None, the original comparison price < 0 raises TypeError. The function should validate the type or coerce it explicitly. Replacement: add if not isinstance(price, (int, float)): raise TypeError("price must be numeric") before the negative check.

The config.yaml change is not a finding because the timeout value is hardcoded, so the new value has zero effect. The reference solution deliberately omits this to test precision.

Common failure modes

In practice, AI reviewers fail in predictable ways. Knowing these failure modes makes the rubric easier to apply:

• Surface-level anchoring: The model focuses on the visibly added - 1 and ignores the subtle type bug on the preceding line.
• False-positive inflation: The model invents issues like "hardcoded timeout" or "missing docstring" to look thorough, which destroys precision.
• Config hallucination: The model assumes a changed value is used because the diff shows it appearing, without checking the surrounding code.
• Vague remedies: The model says "add input validation" without giving a patch, which is useless for a developer who needs a review, not a pep talk.

A good evaluation should log these patterns and compare them across multiple candidate models. The whole process can be automated with a small test harness that calls each model once, stores the answer, and computes the rubric scores from a parsed JSON output.

Running the evaluation on a free server

To run this task without setting up local GPU infrastructure, use MonkeyCode's free server option together with its free model access. The workflow is simple: create a project in the MonkeyCode web UI or CLI, paste the prompt and diff, and collect the model's response. The free tier is enough for several candidate evaluations because each request consumes a few thousand tokens, and the token allocation resets periodically, so a single take-home task comfortably fits inside it.

The exact quota and model list change over time, so read the MonkeyCode repository before relying on them for a large batch. The evaluation described here, however, is intentionally small and does not demand any special hardware. A candidate that cannot pass this test on free resources will not suddenly become reliable on paid infrastructure.

Limitations and who should skip this approach

This task measures only the model's ability to reason about a small, self-contained diff. It does not test long-context understanding, repository-wide search, or the ability to navigate a large monorepo with thousands of files. Teams that already have an established AI-review workflow and a labeled dataset of historical regressions will benefit more from a custom benchmark than from this generic task.

The take-home task also assumes the evaluator knows the ground truth. If the PR is taken from real code, a human reviewer must first verify the reference solution, which is easier said than done. For a first cut, start with the synthetic example above, then move to a real PR with a known bug that was caught in production.

Running the task yourself

Copy the prompt, the diff, and the rubric into a text file, then evaluate at least two different AI reviewers side by side. Track the four scores, and keep the results in a table for your team's records. If you want to try this with MonkeyCode's free tier, the setup takes about twenty minutes and needs no credit card. The most valuable outcome is not a cheap evaluation; it's a process that turns reviewer quality from a guess into a number.