The Take-Home Test I Give Every AI Code Reviewer

Somewhere on DEV this week, someone said what a lot of us were thinking: AI promoted every developer to reviewer, and nobody tested the reviewer. We benchmark models on leaderboards. We rarely ask whether the review bot actually catches bugs that matter to our repo.

So I've started treating AI reviewers like job candidates. Same process: a take-home task, a rubric, a sample solution, and the failure modes I keep seeing in eval setups. Here's the whole thing, so you can run it on your own reviewer this week.

The prompt

The take-home I send candidates — and myself:

You have a budget of $0 and one disposable machine. Take the attached repo; it contains five planted bugs, each tied to a specific file and line. Build a harness that runs an AI code reviewer against the repo, collects the findings, and scores them against the ground truth. Deliver the harness, a one-page report with precision and recall, and the exact command to reproduce your run. You may use free model access and a free server tier. You may not use a paid API key, and you may not run the eval against a repo with real secrets.

The constraints are the point. A $0 budget forces the candidate to know which free tiers actually work. A disposable machine forces them to think about isolation. Ground truth forces them to define "good" before the model defines it for them.

The rubric

I score submissions on four things, in this order:

Criterion	Weight	What I look for
Recall	40%	Did it find the planted bugs? Missing one is a yellow flag. Missing three is a no.
Precision	30%	How much noise came with the signal? Twenty findings for five planted bugs means the reviewer cries wolf.
Evidence	20%	Each finding cites a file, a line, and a reason. "This looks wrong" is not evidence.
Reproducibility	10%	Pinned model version, pinned prompt, one command. If I can't rerun it, it didn't happen.

Notice what's missing: the model's name. I don't care which model won. I care whether the candidate can build a test where any model can lose.

A passing solution

Here's the skeleton of a passing submission. It's a template, not a copy-paste — the review command is a placeholder, because every tool exposes a different CLI. The shape is what matters.

# eval_reviewer.py — template; replace REVIEW_CMD with your tool's CLI
import json
import subprocess
import sys
from pathlib import Path

PLANTED = {
    "src/auth.py:42": "missing nonce check in token refresh",
    "src/db.py:17": "SQL query built with an f-string",
    "src/cache.py:88": "TTL comparison uses >= instead of >",
    "src/upload.py:31": "file size checked after the file is written",
    "src/config.py:9": "default secret is hardcoded",
}

REVIEW_CMD = ["your-review-cli", "review", "--format", "json"]

def run_review(repo: Path) -> list[dict]:
    result = subprocess.run(
        REVIEW_CMD + [str(repo)], capture_output=True, text=True, timeout=600
    )
    if result.returncode != 0:
        raise RuntimeError(result.stderr)
    return json.loads(result.stdout)

def score(findings: list[dict]) -> dict:
    hits = set()
    for f in findings:
        key = f"{f.get('file')}:{f.get('line')}"
        if key in PLANTED:
            hits.add(key)
    recall = len(hits) / len(PLANTED)
    precision = len(hits) / max(len(findings), 1)
    return {
        "recall": round(recall, 2),
        "precision": round(precision, 2),
        "hits": sorted(hits),
        "total_findings": len(findings),
    }

if __name__ == "__main__":
    print(json.dumps(score(run_review(Path(sys.argv[1]))), indent=2))

Run it from the disposable machine and you get a number instead of a vibe.

Where the free tier fits

This is where MonkeyCode's free tier earns its place in the workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project is open source, and its free model access — currently a 10-million-token allocation — is enough to run this eval dozens of times without a credit card. The free server option gives you the disposable machine the prompt demands: an isolated environment where the reviewer can touch files but not your production network.

I care about that second part more than the first. An eval is only trustworthy if it's repeatable, and repeatable means the same environment every time. A throwaway server makes that easy. Your laptop makes it tempting to skip.

The failure modes

This task fails four predictable ways.

The first is testing the model instead of the workflow. Paste a PR into a chat window, call it an eval, report a gut feeling. No harness, no ground truth, no command to rerun. A reviewer you can't reproduce is a reviewer you can't improve.

The second is using bugs the model already knows. Public CVEs from 2023 get flagged because they're in the training data, not because the reviewer read the code. The fix is to plant bugs that are repo-specific — a wrong comparison in a function only your repo has, a secret default in a config file the model has never seen.

The third is optimizing recall into noise. Lower the threshold until the bot flags every line. Recall hits 100%. Precision collapses. The report looks great until a human reads 47 findings and finds three that matter.

The fourth is skipping the sandbox. Run the eval on a laptop, against a repo with real credentials, with the network wide open. The reviewer has shell access, remember. Test it like you'd test anything with shell access: in a box you can throw away.

Who shouldn't use this

A five-bug repo measures one thing: can the reviewer find what you told it to find. It won't tell you whether the reviewer invents problems, whether it's consistent across languages, or whether it slows your team down. Those questions need a longer trial, with real PRs and real humans keeping score.

But if you own a review pipeline, this is one afternoon well spent. The take-home is just an eval with a rubric. Run it on your own reviewer, your own repo, your own budget of $0. The candidate is the bot. You're the interviewer.

If you want to run it without a credit card, MonkeyCode's free model access and free server option are a reasonable starting point — and since the project is open source, the setup is yours to inspect.