Show the Reviewer's Test Score Before It Says LGTM

Last Tuesday, I approved a pull request because an AI reviewer said LGTM. The reviewer was wrong. A null check was missing, and the bug reached production at 4 PM.

Here's the uncomfortable part. I never tested that reviewer. I tested the code and the build, but never the tool that told me what to think.

The conversation this week keeps circling one idea. We all became reviewers overnight, and nobody tested the reviewer itself. That stung because it's true.

So let's test the reviewer. This is a from-zero-to-working tutorial. You'll build a reviewer test harness.

It runs a free model on known-buggy code. It scores the findings. It hides the review until the score passes.

You can run the whole thing on MonkeyCode's free server with its free model access. MonkeyCode is an open-source project built for exactly this kind of experiment.

The free tier includes a 10-million-token allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here's the flow:

bug corpus → review agent → findings → scorer → gate → human

Read the flow from left to right. The human is the last stop, not the first. Most teams put the human first and the model last. That's backwards.

Each stage has a verification step. If a stage fails, you stop. That's the entire point.

Stage 1: Build the bug corpus

I wrote three small functions with planted bugs. Boring bugs, on purpose: a missing null check, a mutable default argument, a swallowed exception.

Boring bugs ship. That's why we test with them.

CASES = [
    {
        "name": "missing_null_check",
        "code": """
def get_email(db, user_id):
    row = db.execute("SELECT email FROM users WHERE id = ?", user_id)
    return row["email"]
""",
        "bug": "user_id can be None. The query returns no row. row is None."
    },
    {
        "name": "mutable_default",
        "code": """
def add_tag(tags, tag, seen=[]):
    if tag not in seen:
        seen.append(tag)
        tags.append(tag)
    return tags
""",
        "bug": "seen is shared across calls. State leaks between requests."
    },
    {
        "name": "swallowed_exception",
        "code": """
def send_report(report):
    try:
        post(report)
    except Exception:
        pass
""",
        "bug": "The exception vanishes. The caller thinks the report was sent."
    },
]

Verify this stage. Run each snippet and watch it fail. If it doesn't fail, the case is useless.

A corpus of lies produces a confident liar.

Stage 2: Point the reviewer at a free model

The reviewer is a small script. It sends each snippet to a model. It asks one question: BUG or CLEAN?

import os
import requests

MODEL_URL = os.environ["MODEL_URL"]
API_KEY = os.environ["API_KEY"]

def review(code):
    prompt = (
        "Does this code have a bug? "
        "Reply BUG or CLEAN, then one sentence.\n\n" + code
    )
    r = requests.post(
        MODEL_URL,
        json={"prompt": prompt},
        timeout=30,
    )
    return r.json()["text"]  # adjust the response field to your provider

The exact request shape depends on your provider. MonkeyCode's free server gives you a URL and a key.

Point this script at them. That's the whole setup.

Verify this stage. Run the reviewer on one clean snippet and one buggy snippet. It should tell them apart.

If it can't, stop. Don't tune prompts until the model is honest. Tune until it's reliable.

Stage 3: Score and gate

The scorer is brutally simple.

def score(results):
    hits = sum(
        1 for r in results
        if r["expected"] == "BUG" and r["verdict"] == "BUG"
    )
    return hits / len(results)

THRESHOLD = 0.8

def gate(results):
    s = score(results)
    if s >= THRESHOLD:
        return "post_review"
    if s >= 0.5:
        return "post_low_confidence"
    return "block_and_escalate"

The gate is the part people skip. They run a demo and turn the reviewer loose.

A demo is not a test. The gate turns the test into a decision.

Score	What happens
>= 0.8	Post the review. A human still approves.
0.5–0.8	Post findings, marked low confidence.
< 0.5	Block the review. Escalate to a human.

Notice what the table does. It doesn't remove the human. It moves the human to the right moment.

The reviewer earns the right to comment. The human keeps the right to decide.

Now the failure case. What if the model times out? My first version treated a timeout as CLEAN.

That's a dangerous default. A silent reviewer is worse than a wrong one. Make timeouts fail loud.

def review(code):
    try:
        r = requests.post(MODEL_URL, json={"prompt": prompt}, timeout=30)
        return r.json()["text"]
    except requests.Timeout:
        return "NO_VERDICT"

Treat NO_VERDICT as a blocked review. No score, no comment. The human decides.

Wire it together and run it.

def main():
    results = []
    for case in CASES:
        verdict = review(case["code"])
        results.append({
            "name": case["name"],
            "expected": "BUG",
            "verdict": verdict.split()[0],
        })
    print("score:", score(results))
    print("action:", gate(results))

if __name__ == "__main__":
    main()

export MODEL_URL="https://your-endpoint"
export API_KEY="your-key"
python harness.py

Verify this stage. Run the full harness and print the score.

Then break one case and watch the gate block. If the gate doesn't block, the gate doesn't work.

What evidence supports this?

The corpus is the evidence. The score is the evidence. The gate is the enforcement.

If you can't show the score, you're asking people to trust a black box. That's how LGTM ships bugs.

This is the part I'd defend in a design review. The corpus is reproducible, and so is the score.

Anyone on the team can rerun the harness and see the same number. That's what evidence looks like in a probabilistic world.

Would you approve a PR from a reviewer who never passed a test? I did once. Once was enough.

Who should not use this

Don't use this for security reviews. Don't use it for code that touches money or medical data.

A three-case corpus is a smoke test, not a guarantee. The model can pass the corpus and still miss your bug.

And if your team has no human reviewer, this harness won't save you. It only helps when a person reads the findings.

The gate doesn't replace the human. It makes the human's job visible.

MonkeyCode's free tier — 10 million tokens and a free server — is enough to run this whole harness today. Build your own corpus. Show the score before you say LGTM.