Your AI Code Reviewer Is Probably Lying to You. Here's How to Measure It for Free

You've watched it happen: your AI code reviewer comments “Looks good to me” on a function that clearly dereferences None. Then it blocks a pull request for a style nit that nobody cares about. You're not sure if the tool is useful, or just confident.

Confidence is not accuracy. Without measurement, you're guessing. This article shows you how to build a zero-cost accuracy harness for any AI reviewer. It runs on free models and a free server, so the only investment is twenty minutes of setup.

Why trust a reviewer you've never tested?

We add AI reviewers to speed up code review. But most teams never ask the simplest question: how often is the AI right?

If you don't measure, you'll optimize for the wrong thing — like how fast the review comes back, not how much real bugs it catches. A fast reviewer that misses every memory leak is worse than a slow human who catches them.

The fix is a binary evaluation set. You feed the model code snippets with known labels: has_bug or clean. The model must answer YES or NO. Then you compute precision, recall, and F1 score. That's it.

What you need

You need an OpenAI-compatible API endpoint. I'm using MonkeyCode's free models for this walkthrough. MonkeyCode is an open-source platform that also includes a free server option, so the whole evaluation can run unattended without a credit card.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here's the stack:

• Python 3.9+
• The openai Python package
• An API key from MonkeyCode (free tier)
• A cron job or systemd timer on the free server

Design the experiment

Start with a small set of code samples. You want roughly half with a realistic bug, half clean. Avoid obvious traps like x = 1/0; those make the model look smart. Use bugs developers actually make:

• Missing input validation
• Off-by-one errors in loops
• Using == instead of is for None
• Forgetting to close resources
• Catching exceptions too broadly

Your prompt should force a binary answer. No “maybe”, no markdown, no explanations until after the flag. That keeps parsing trivial and metrics honest.

Build the harness

Here is a complete script. Save it as reviewer_audit.py.

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("MONKEYCODE_BASE_URL", "https://api.monkeycode.ai/v1"),
    api_key=os.getenv("MONKEYCODE_API_KEY"),
)

CASES = [
    {
        "code": """def get_total(items):
    total = 0
    for i in range(len(items) + 1):  # off-by-one
        total += items[i]
    return total""",
        "bug": True,
        "note": "IndexError when i == len(items)",
    },
    {
        "code": """def get_total(items):
    return sum(items)""",
        "bug": False,
        "note": "Clean implementation",
    },
    {
        "code": """def save_user(user):
    with open("users.txt", "a") as f:
        f.write(user)
    # no flush, no context issue here - actually fine""",
        "bug": False,
        "note": "Fine",
    },
    {
        "code": """def save_user(user):
    conn = get_connection()
    cursor = conn.cursor()
    cursor.execute("INSERT INTO users VALUES (?)", (user,))
    # conn is never closed""",
        "bug": True,
        "note": "Connection leak",
    },
    {
        "code": """def apply_discount(price, is_member):
    if is_member:
        return price * 0.9
    return price""",
        "bug": False,
        "note": "Fine",
    },
    {
        "code": """def apply_discount(price, is_member):
    return price * 0.9 if is_member else price * 1.0""",
        "bug": False,
        "note": "Fine too",
    },
]

def ask(code: str) -> bool:
    resp = client.chat.completions.create(
        model=os.getenv("MONKEYCODE_MODEL", "default"),
        messages=[
            {"role": "system", "content": "You are a code reviewer. Reply with exactly YES if the code has a bug that would cause a failure in production. Otherwise reply NO."},
            {"role": "user", "content": code},
        ],
        temperature=0.0,
    )
    return resp.choices[0].message.content.strip().startswith("YES")

def majority(code: str, runs: int = 3) -> bool:
    votes = [ask(code) for _ in range(runs)]
    return sum(votes) > runs // 2

def evaluate():
    tp = fp = fn = tn = 0
    for case in CASES:
        predicted = majority(case["code"])
        actual = case["bug"]
        if predicted and actual:
            tp += 1
        elif predicted and not actual:
            fp += 1
        elif not predicted and actual:
            fn += 1
        else:
            tn += 1

    precision = tp / (tp + fp) if (tp + fp) else 0.0
    recall = tp / (tp + fn) if (tp + fn) else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0

    print(f"True positives: {tp}")
    print(f"False positives: {fp}")
    print(f"False negatives: {fn}")
    print(f"True negatives: {tn}")
    print(f"Precision: {precision:.2f}")
    print(f"Recall: {recall:.2f}")
    print(f"F1 score: {f1:.2f}")

if __name__ == "__main__":
    evaluate()

The majority() function runs each sample three times and takes the vote. This smooths out the model's nondeterminism. You can change runs to 5 for extra stability.

Run it locally, then ship it to the free server

First, install the client and run the script locally:

pip install openai
export MONKEYCODE_API_KEY="your-api-key"
export MONKEYCODE_MODEL="your-preferred-model"
python reviewer_audit.py

Once it works, move it to MonkeyCode's free server. Since the script is small and stateless, the free server can handle it easily. Set up a weekly cron job to re-run the evaluation:

crontab -e
# Add this line to run Monday 9am
0 9 * * 1 cd /opt/reviewer-audit && python reviewer_audit.py >> audit.log

This gives you a trail of accuracy over time. If you change prompts or swap models, the scores will tell you if you broke something.

Interpreting the numbers

There is no universal “good” F1 for a code reviewer. It depends on how many real bugs exist in your codebase and how the model is calibrated. Use this decision table to decide your next move:

F1 score	What it means	Action
> 0.85	The reviewer finds most bugs without crying wolf.	Use it as a primary gate, but keep a human in the loop for edge cases.
0.70 – 0.85	Decent, but one class is likely off.	Check whether precision or recall is lower. Tune prompts to reduce false positives or negatives.
< 0.70	The reviewer is not ready for production.	Revert to manual review. Re-evaluate after prompt redesign or model change.

Remember to look at recall and precision separately. A model that always says YES gets 100% recall but poor precision. A model that says NO all the time gets 100% precision but 0% recall. F1 keeps both in tension.

Grow your own evaluation set

The six handcrafted cases are just a smoke test. To really audit your reviewer, you need cases from your actual repositories. Collect past pull requests where a bug slipped through or a valid change was rejected. Label them, then add them to a JSON file:

{
  "cases": [
    {"code": "...", "bug": true, "note": "Exploit found in auth"},
    {"code": "...", "bug": false, "note": "Clean refactor"}
  ]
}

Load that file in the script instead of the hardcoded list. After every incident, add a new case. Your evaluation set becomes a living regression suite for the AI reviewer — exactly like unit tests for your own code.

What this doesn't catch

This harness measures binary bug detection, not review quality. A model might get a perfect F1 but still write useless comments like “consider refactoring” everywhere. It might miss cross-file security issues. It also only sees small snippets; it cannot evaluate a 2,000-line diff.

Do not use this as a formal certification. It is a smoke test and a regression detector. It will catch a model update that secretly destroys accuracy — which is already a huge win.

The habit is worth more than the score

The point isn't to hit a magic F1 number. It's to start collecting evidence about your AI tooling. Once you have measurements, you stop having vibes and start having data. If you're not measuring your AI reviewer, you're just trusting a black box.

If you don't have an API key yet, MonkeyCode's free models and free server are a reasonable starting point — the same setup I used here. Run the harness once, and you'll know more about your reviewer than most teams ever do.

Go run it. Your AI will thank you — or expose you.