Measure Your AI Reviewer Before You Trust It: A Seeded-Defect Benchmark

The PR looked clean. Your AI reviewer flagged fourteen issues, you accepted twelve, and the merge button felt like a formality. Three weeks later, a null dereference shipped in the exact function the reviewer had approved.

Nothing in the tooling broke. The problem was simpler: nobody had ever tested the reviewer itself. You benchmarked the code. You never benchmarked the verifier.

There is a good discussion on DEV this week about what developers do while AI writes code. The quieter question is the one that costs more: who checks the checker? A reviewer that produces confident noise is worse than no reviewer, because it turns review into a rubber stamp.

This article gives you a reproducible way to measure an AI code reviewer: a seeded-defect dataset, a scoring script, and a failure log that separates model mistakes from harness mistakes. It takes about an afternoon to run.

Why review quality is a measurement problem

An AI reviewer has two failure modes. False positives waste your time and train you to ignore the tool. False negatives ship bugs while the dashboard stays green.

Most teams measure neither. CI passing means the reviewer ran, not that it was right. The fix is to stop treating review output as a verdict and start treating it as a prediction you can score.

The method: seed defects, score the review

The workflow looks like this. It assumes you have a repo with real merged PRs and a reviewer that can output JSON.

1. Build a small dataset. Take ten merged PRs from your own repository. For each PR, write two or three minimal defects a competent reviewer should catch: a null dereference, an off-by-one, a missing error check, a wrong comparison. Store them in a JSON file with keyword anchors for scoring.
2. Define the review protocol. The reviewer receives the diff plus one context file. Its output must be JSON: a list of issues with file, line, severity, and a one-sentence explanation.
3. Run a baseline. Before the AI, run a linter on the same PRs. Record its findings as a separate row. You need this comparison, or you will not know whether the model is adding value or adding noise.
4. Score every comment. Match each review comment to a seeded defect using the script below. Compute precision, recall, and F1.
5. Attribute every miss. For each false negative, record a reason: context truncated, instruction ambiguity, model limitation, or harness bug. This step is the whole point. It tells you what to fix.

The scoring script

Save this as review_bench.py. It takes a dataset file and a review output file, and prints the score.

import json
import sys
from pathlib import Path

def token_overlap(text, keywords, threshold=0.5):
    text_tokens = set(text.lower().split())
    kw_tokens = set(k.lower() for k in keywords)
    overlap = len(text_tokens & kw_tokens) / max(1, len(kw_tokens))
    return overlap >= threshold

def run(dataset_path, review_path):
    dataset = json.loads(Path(dataset_path).read_text())
    review = json.loads(Path(review_path).read_text())

    defects = {d['id']: d for pr in dataset for d in pr['defects']}
    labels = {pr['pr_id']: [d['id'] for d in pr['defects']] for pr in dataset}

    tp = fp = 0
    caught = {pr_id: set() for pr_id in labels}

    for item in review['items']:
        pr_id = item['pr_id']
        matched = [
            did for did in labels[pr_id]
            if token_overlap(item['text'], defects[did]['keywords'])
        ]
        if matched:
            tp += 1
            caught[pr_id].update(matched)
        else:
            fp += 1

    fn = sum(len(labels[pr_id]) - len(caught[pr_id]) for pr_id in labels)

    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0

    print(json.dumps({
        'true_positives': tp,
        'false_positives': fp,
        'false_negatives': fn,
        'precision': round(precision, 3),
        'recall': round(recall, 3),
        'f1': round(f1, 3),
    }, indent=2))

if __name__ == '__main__':
    run(sys.argv[1], sys.argv[2])

A dataset entry looks like this. The keywords are only for scoring; the reviewer never sees them.

[
  {
    "pr_id": "pr-1",
    "defects": [
      {"id": "d1", "keywords": ["null", "dereference"]},
      {"id": "d2", "keywords": ["off-by-one", "range"]}
    ]
  }
]

Run it like this:

python review_bench.py dataset.json review_output.json

Example output from a ten-PR run:

{
  "true_positives": 4,
  "false_positives": 2,
  "false_negatives": 1,
  "precision": 0.667,
  "recall": 0.8,
  "f1": 0.727
}

Raw counts matter more than the percentages. With thirty seeded defects, one miss moves recall by roughly three points. Report the counts, or the marketing team will round the F1 score into a headline.

The script assumes every reviewed PR exists in the dataset. If your reviewer emits an id you did not seed, add a guard before you trust the output.

Controls that keep the numbers honest

Four controls prevent this from becoming another dashboard.

• Run the reviewer at temperature 0. You want determinism, not creativity.
• Blind the reviewer. Do not tell it which files contain seeded defects.
• Freeze the dataset. If you tune your prompt on the same PRs, you are overfitting, not measuring.
• Refresh quarterly. Seeded defects age; reviewers get updated; your rubric should move with them.

Why these numbers are not marketing

Every true positive, false positive, and false negative in this benchmark maps to a line in your dataset. You can audit each one. That is the entire difference between a benchmark and a brochure.

Most published AI review numbers come from private datasets, unknown rubrics, and no failure log. You cannot reproduce them, and you cannot learn from them. This method gives you the opposite: a small, ugly, honest table you generated yourself.

Where the free tier fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This evaluation is token-hungry. Ten PRs, two or three reruns each, plus context files, and you have burned a meaningful chunk of a paid API budget. MonkeyCode is an open-source project that bundles free model access and a free server option. As of this writing, the free tier includes a 10M token allowance for model access. That combination is enough to run this benchmark as a scheduled job: the server pulls the dataset, calls the model, runs review_bench.py, and appends the JSONL result.

The product is not the point of the method. The method works with any model and any server. The free tier just removes the cost excuse for not measuring.

Who should not use this

Three teams should skip this benchmark.

Teams with no human review process. A benchmark will not create a culture of review; it will only measure a tool nobody is reading.

Teams that need security-grade guarantees. A seeded-defect benchmark is a regression check, not a safety certificate. It cannot prove the absence of vulnerabilities.

Teams that will not act on the failure log. If you are not going to fix the harness bugs and prompt ambiguities the attribution step reveals, the whole exercise is theater.

Your first run will be embarrassing

Your AI reviewer is a prediction engine. Treat it like one: score it, audit it, and re-run it when the model changes.

If you want to see where your reviewer actually fails, run this on your last ten merged PRs. The dataset format is above, the script is in this article, and the free tier is enough to cover the runs. The first result will probably be embarrassing. That is exactly why you should do it.