Workshop: Test Your AI Reviewer With a 15-Case Golden Set in 60 Minutes

Every team that adds an AI code reviewer gains a new reviewer, but almost nobody adds a test for that reviewer. A golden set of fifteen real review cases, scored against a small rubric, is enough to catch the regressions that quietly break review quality. This workshop builds that harness in sixty minutes using free model tokens and a free server. You leave with a gate you can rerun on every model change.

Recent DEV discussions have made the same point from a different direction: AI promoted every developer to a reviewer, and nobody tested the reviewer. The fix is not a bigger benchmark; it is a small reproducible evaluation that runs on demand. A public benchmark measures a model against generic data, but your review workload lives in your diffs, your style, and your bug history. A project-specific golden set therefore beats a leaderboard for this decision.

What you need

• Python 3.11 or newer, using only the standard library
• An API key for the chat model you want to evaluate
• A free server for scheduled reruns; MonkeyCode's free tier includes both model tokens and a server, with a 10M token allowance at the time of writing. Limits change, so verify the current numbers in the docs before you start.

MonkeyCode is open source, and its free tier covers the two resources this lab needs: tokens for the runs and a server for the scheduled reruns. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below does not use any vendor SDK; it calls an OpenAI-compatible chat endpoint. You can switch providers by changing three environment variables.

Exercise 1: write the golden set (10 minutes)

A golden set is a small file of real inputs plus the criteria that separate a good answer from a bad one. You do not need model-written expected outputs; you need the judgment rules your team already applies in review. Save this shape as golden_set.json, then grow it to fifteen cases from your last month of real comments.

{
  "task": "code_review_comment",
  "criteria": [
    "catches the real bug, not a style nit",
    "names the exact file and line",
    "explains why the bug matters",
    "proposes a concrete fix",
    "stays under 120 words"
  ],
  "examples": [
    {"id": "case-01", "input": "checkout.py: total = sum(item.price for item in cart)", "context": "The sum ignores item.qty, so multi-item orders are undercharged.", "expected": "Flag that total ignores item.qty before apply_promo runs."}
  ]
}

Include five genuinely tricky cases, because a reviewer that passes easy ones and fails hard ones is exactly the failure mode you want to catch early.

Exercise 2: build the reviewer and the judge (15 minutes)

The harness has three parts. First, call the candidate model with a review prompt built from the case input. Second, grade the produced comment with a short rubric prompt that returns JSON only. Third, write every raw output to disk, because a run you cannot inspect is a run you cannot debug.

import json, os, sys, time, urllib.request

BASE_URL = os.environ.get('LLM_BASE_URL')
MODEL = os.environ.get('LLM_MODEL')
API_KEY = os.environ.get('LLM_API_KEY')
DATA = json.load(open('golden_set.json'))

def call_model(prompt, temperature=0.2):
    body = json.dumps({
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': temperature,
    }).encode()
    req = urllib.request.Request(
        BASE_URL + '/chat/completions', data=body,
        headers={'Authorization': 'Bearer ' + API_KEY,
                 'Content-Type': 'application/json'})
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)['choices'][0]['message']['content']

def judge(comment, example):
    rubric = '\n'.join(f'{i+1}. {c}' for i, c in enumerate(DATA['criteria']))
    prompt = f'''Score this review against the criteria.
1 point if fully met, 0.5 if partially met, 0 if not met.
Reply with JSON only, keys scores and total.

Criteria:
{rubric}

Review:
{comment}

Target:
{example['expected']}'''
    try:
        return float(json.loads(call_model(prompt, 0.0))['total'])
    except Exception:
        return 0.0

The judge is a second call to the same model in this lab, which is a known bias. You accept that for a short workshop, but you should switch the judge to a stronger model when the budget exists. The try/except converts a malformed judge reply into a zero, which keeps scoring strict instead of silently optimistic.

Exercise 3: run the full set and read the scorecard (15 minutes)

Run every case, collect per-case scores, and print the mean. The scorecard should show not only the total but also the weakest criteria. A review assistant that always misses the "explains why" criterion needs a prompt fix, not a model swap.

def build_prompt(example):
    src = example['input']
    ctx = example['context']
    return 'Review this diff in ' + src + '. Context: ' + ctx + '. Reply in under 120 words.'

results = []
for ex in DATA['examples']:
    comment = call_model(build_prompt(ex))
    results.append({'id': ex['id'], 'score': judge(comment, ex), 'comment': comment})

n = len(DATA['criteria'])
mean = sum(r['score'] for r in results) / len(results)
json.dump({'ts': time.time(), 'model': MODEL, 'mean': mean, 'results': results},
          open('run.json', 'w'), indent=2)
print(f'model={MODEL} mean={mean:.2f} max={n}')

Typical honest results land between 2.5 and 4.0 on a five-point scale. Do not be surprised by low scores on the tricky cases; the value of the lab is knowing which criteria fail before you ship the reviewer to your team.

Exercise 4: turn the mean into a decision gate (10 minutes)

The gate is three lines of logic: read the saved mean, compare it with a threshold, and exit non-zero on failure. In a repository, run it as a CI step that blocks a model upgrade when quality drops. On the free server, schedule the same script with cron if background jobs are allowed.

threshold = float(os.environ.get('SCORE_THRESHOLD', '4.0'))
print('PASS' if mean >= threshold else 'FAIL')
sys.exit(0 if mean >= threshold else 1)

A nightly run is the minimum cadence that makes sense, because model endpoints change behavior without changing version numbers. When the mean drops, the saved run.json tells you whether the model changed or the judge changed.

When this approach is the wrong tool

Decision you need	Golden-set lab	Full benchmark
Can this model review my diffs?	Yes	Insufficient
Is this checkpoint better than the last one?	Yes	Overkill
What is the p95 latency?	No	Yes
Can we claim SOTA accuracy?	No	Yes

Golden-set tests complement benchmarks; they do not replace them. Use a full benchmark to choose the model family, then use this lab to decide whether that model works on your actual diffs. The free server's limits around CPU and background jobs are documented on the provider site, and the lab itself needs only a few seconds of compute per run.

Run it again with another model

The whole exercise costs a few thousand tokens per run, which is why a free allowance matters. You can rerun the lab dozens of times per month without thinking about spend, and each rerun produces a comparable scorecard. The harness is provider-agnostic, so the same golden set can rank two models side by side. If you want to try it against the current free tier and server, the MonkeyCode docs are the fastest way to verify today's token limit and sign-up steps.