Seed a Bug, Score the Review: A Contract Test for AI Code Reviewers

AI reviews your pull request. Who reviews the review? Most teams trust the output. That trust is unearned. A reviewer is a function. Functions need tests. This tutorial builds a contract test for an AI code reviewer. It measures precision and recall. It runs in under an hour. It needs only a free model endpoint and a small Python script.

This workflow uses MonkeyCode's free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The method works with any OpenAI-compatible endpoint.

Why a contract test

A code review has a contract. Given a diff with a bug, the reviewer must report it. Given a clean diff, the reviewer must stay quiet. That contract is testable. You need three things: a labeled corpus, a harness, and a score. This tutorial builds all three.

Step 1: Build a labeled corpus

Start small. Five cases are enough. Four diffs contain one seeded bug each. One diff is clean. Each case needs an id, a diff, and a signal. The signal is a phrase the review must contain to count as a hit.

Create review_cases.json:

[
  {
    "id": "off-by-one",
    "diff": "-    return items[:n]\n+    return items[:n+1]",
    "signal": "off-by-one"
  },
  {
    "id": "null-check",
    "diff": "-    if user and user.name:\n+    if user.name:",
    "signal": "null"
  },
  {
    "id": "clean",
    "diff": "-    return total\n+    return total * 1.0",
    "signal": ""
  }
]

Use real diffs from your repo. Use bugs your team actually ships. The signal should be a word the reviewer is likely to use. off-by-one is better than index error if you want a stable match.

Verify: python -m json.tool review_cases.json prints valid JSON.

Step 2: Write the review harness

The harness sends each diff to the model endpoint. It stores the raw review text. This script is minimal. It assumes an OpenAI-compatible chat endpoint. Install requests first: pip install requests.

Create review_harness.py:

import json
import os
import requests

CASES = 'review_cases.json'
OUT = 'reviews.json'
ENDPOINT = os.environ['MODEL_ENDPOINT']
KEY = os.environ['MODEL_KEY']

def call_review(diff):
    resp = requests.post(
        ENDPOINT,
        headers={'Authorization': f'Bearer {KEY}'},
        json={
            'model': os.environ.get('MODEL_NAME', 'default'),
            'messages': [
                {'role': 'system', 'content': 'You review code. Report bugs only. Be specific. If no bugs, reply NO_BUGS.'},
                {'role': 'user', 'content': f'Review this diff:\n{diff}'}
            ],
            'temperature': 0
        },
        timeout=60
    )
    resp.raise_for_status()
    return resp.json()['choices'][0]['message']['content']

def main():
    cases = json.load(open(CASES))
    results = []
    for case in cases:
        text = call_review(case['diff'])
        results.append({'id': case['id'], 'review': text})
    json.dump(results, open(OUT, 'w'), indent=2)

if __name__ == '__main__':
    main()

Set the endpoint and key in your environment. Do not commit the key.

Verify: python review_harness.py creates reviews.json with one entry per case.

Step 3: Score the reviews

Now compare each review against the contract. A hit means the signal appears in the review text. A clean case counts as a false positive if the review does not say NO_BUGS.

Create score_reviews.py:

import json

cases = {c['id']: c for c in json.load(open('review_cases.json'))}
reviews = json.load(open('reviews.json'))

tp = fp = fn = 0
for item in reviews:
    case = cases[item['id']]
    text = item['review'].lower()
    if case['signal']:
        if case['signal'] in text:
            tp += 1
        else:
            fn += 1
    elif 'no_bugs' not in text:
        fp += 1

precision = tp / (tp + fp) if tp + fp else 1.0
recall = tp / (tp + fn) if tp + fn else 0.0
print(f'precision={precision:.2f} recall={recall:.2f}')

This is a naive matcher. It is enough for a regression gate. You can upgrade to fuzzy matching later.

Verify: python score_reviews.py prints two numbers. If recall is low, your prompt is missing bugs. If precision is low, your reviewer cries wolf.

Step 4: Turn the score into a gate

A score without a threshold is a report. A score with a threshold is a gate. Add this to score_reviews.py:

if recall < 0.8 or precision < 0.9:
    raise SystemExit('review contract failed')

Run it in CI. Run it on every prompt change. Run it before you trust a new model endpoint. The gate fails fast. The gate tells you which side of the contract broke.

Verify: break the prompt on purpose. The gate exits non-zero. Fix the prompt. The gate passes again.

What this catches

This test catches three failure modes. First, prompt drift: a new system message makes the reviewer vague. Second, endpoint drift: a free model update changes output style. Third, pipeline bugs: your code sends the wrong diff or truncates the response. All three are silent without a contract.

Limitations

This is not a benchmark. Five cases do not measure review quality. String matching misses semantic hits. A reviewer can name the bug without using your signal. A reviewer can also invent bugs that are not in the diff. Precision and recall are proxies, not truth.

The corpus needs maintenance. Bugs change. Signals change. If your team stops shipping null-check bugs, the null case becomes noise.

Free endpoints are not stable. They can change models, rate limits, or output formats. The gate will catch some changes. It will not catch all of them.

Who should not use this

Do not use this if you have no bug taxonomy. Without labels, the score is meaningless. Do not use this to compare models. The corpus is too small. Do not use this to replace human review. The gate tests a contract, not judgment.

Use it when you want a cheap warning system. Use it when you change your review prompt. Use it when a free endpoint looks too good to trust. Then read the reviews yourself. The gate is a tripwire, not a verdict.