Budget Calibration: How to Test Whether Your AI Reviewer Still Works Before You Trust It

An LLM-based code reviewer is a moving target. The model behind it changes quarterly, the prompt drifts, and nobody reruns the test suite for the reviewer itself. This article shows a cheap, repeatable way to sanity-check your AI patch gate on free model access and a disposable server, before it costs you a merge.

I have been burned by this exact problem twice. The first time, a grading prompt rejected a patch that introduced a dangling std::string_view, so my team celebrated and moved on. Three weeks later, the same prompt approved a nearly identical patch because the upstream model had been swapped for a cheaper one. The second time, the reviewer started rejecting every refactor that touched a header file, and the merge queue backed up so badly that someone disabled the gate entirely.

The pattern is always the same: the code does not change, but the model does. And while developers diligently regression-test the code the AI writes, they almost never regression-test the AI that reviews the code.

Why reviewer drift is invisible until it hurts

Most CI pipelines test the generator, not the judge. You feed a model a buggy patch, check whether it produces a fix, and call it a day. That tells you nothing about whether the reviewer that approves or rejects that fix is still behaving itself.

Model providers ship new versions on aggressive release cycles, and teams switch endpoints to cut costs. Both actions silently change reviewer behavior. A prompt that said reject for a use-after-free in January can say approve in June without anyone touching the pipeline configuration.

A related DEV discussion last week asked a sharp question about human reviewers who were never tested after being promoted by AI (link). The machine counterpart deserves the same scrutiny. You would not trust a human reviewer who never took a certification exam, so why trust a model that has never seen a calibration set?

The calibration harness: a labeled set, a script, a baseline

A calibration harness does to your reviewer what golden tests do to your generator: it feeds it known inputs and checks that it still produces known verdicts. The whole thing fits in roughly seventy lines of Python and runs against any OpenAI-compatible /chat/completions endpoint.

Step 1: Build a small but honest labeled set

Ten to twelve patch snippets are enough for a smoke test. Include the bug classes your team actually cares about — use-after-free, null dereference, signed overflow, missing error handling — plus a few clean patches that must not be rejected. Each entry gets a gold label: approve or reject.

# calibrate_reviewer.py — a minimal LLM reviewer drift harness
import json, os, requests

CALIBRATION = [
    {
        "id": "dangling-sv",
        "gold": "reject",
        "patch": "void log_event(const Event& e) {\n    std::string_view v = e.name();\n    if (e.type() == EventType::USER) {\n        std::cout << v << \"\\n\";\n    }\n}",
    },
    {
        "id": "null-check-clean",
        "gold": "approve",
        "patch": "auto* cfg = get_config();\nif (!cfg) { return Error::MISSING_CONFIG; }\nreturn cfg->apply();",
    },
    # add 8-10 more: UB cases, clean refactors, partial fixes
]

Step 2: Define a grader that answers in JSON

The prompt must force a deterministic shape. Ask for a JSON object with a verdict field, parse only that field, and ignore the reasoning text. Set temperature to 0 and run each case three times, because even a temperature-zero model can flip on tokenizer details.

REVIEW_PROMPT = """You are a C++ code reviewer. Inspect the patch.\n\nIf the patch introduces a real bug (memory safety, UB, logic error), reply with JSON:\n{"verdict": "reject", "reason": "..."}\n\nOtherwise reply with JSON:\n{"verdict": "approve", "reason": "..."}\n\nPatch:\n{patch}"""

def grade(patch: str, base_url: str, model: str, key: str = "none") -> str:
    resp = requests.post(
        f"{base_url}/chat/completions",
        headers={"Authorization": f"Bearer {key}"},
        json={
            "model": model,
            "messages": [{"role": "user", "content": REVIEW_PROMPT.format(patch=patch)}],
            "temperature": 0,
        },
        timeout=120,
    )
    payload = json.loads(resp.text)
    return json.loads(payload["choices"][0]["message"]["content"])["verdict"]

Step 3: Run the comparison and save the baseline

A small driver loops over the calibration set, takes a majority verdict for each patch, and prints a table. The first successful run becomes your baseline report. Commit that report to the repository.

def run_config(base_url: str, model: str, runs: int = 3) -> dict:
    results = {}
    for case in CALIBRATION:
        votes = [grade(case["patch"], base_url, model) for _ in range(runs)]
        verdict = max(set(votes), key=votes.count)
        results[case["id"]] = {
            "gold": case["gold"],
            "model": verdict,
            "match": verdict == case["gold"],
        }
    return results

def report(results: dict) -> None:
    total = len(results)
    passed = sum(1 for r in results.values() if r["match"])
    print(f"agreement: {passed}/{total}")
    for pid, r in results.items():
        flag = "OK  " if r["match"] else "FAIL"
        print(f"[{flag}] {pid}: gold={r['gold']} model={r['model']}")

if __name__ == "__main__":
    candidate = {
        "base_url": os.getenv("CANDIDATE_BASE_URL"),
        "model": os.getenv("CANDIDATE_MODEL"),
    }
    report(run_config(**candidate))

Step 4: Decide with three hard rules

Do not rely on a single percentage. Use these gates in order:

1. Total agreement must not drop below the saved baseline.
2. Every security-labeled case must still return reject. If the dangling string_view case slips through, the reviewer is broken regardless of the overall score.
3. Clean patches must stay approved. A reviewer that starts rejecting refactors costs more team time than it saves.

If all three pass, the new configuration is safe to promote. If any fail, roll back the endpoint or the prompt and investigate before blaming the CI.

Where free models and a free server fit the workflow

Calibration runs are bursty and low-volume, which makes them a perfect candidate for free-tier resources. You spin up a disposable instance, point the harness at a free model endpoint, run a few dozen requests, and tear everything down. Paying for that experiment out of your production budget feels wasteful — and it is.

This is where an open-source project's hosted option can help. MonkeyCode is an open-source tool whose hosted server currently offers free model access and a free server option for trying the whole stack without pulling your own GPU out of the closet. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The exact quotas and model names change over time, so the project README is the source of truth, not this article. The important part is the shape of the workflow: you can run a meaningful calibration experiment for the cost of a few minutes of setup. Two environment variables — CANDIDATE_BASE_URL and CANDIDATE_MODEL — are all the harness needs to point at any compatible endpoint, including a hosted one.

Who should not use this approach

A calibration harness is not a magic shield. Use this decision table to draw honest boundaries:

Workload	Free hosted server + free model access?	Reason
Calibration on public or synthetic C++ snippets	Yes	Low volume, bursty, no private data in the prompt
Nightly CI gate on proprietary source	No	Sending proprietary code to a third-party API may violate compliance; check first
Security-critical merge verdicts	Partial	Treat the LLM verdict as triage, back it with ASan/TSan receipts
Sustained fuzz or sweep workloads	No	Free tier is for trying, not continuous load; current terms apply

If your team cannot send source code to third-party APIs at all, self-host the model or strip patches down to minimal repros before grading. If your team needs deterministic merge policy, do not use LLM reviewers in the first place — calibration only measures drift, it does not turn a stochastic system into a certified one.

Limitations of this harness

A ten-case set is a smoke test, not a benchmark. It catches gross drift, not subtle style changes on large patches. Ground truth also rots: a patch labeled approve last quarter can become buggy after a dependency update, so you must review the labels themselves periodically. Majority voting reduces noise but does not eliminate it. And the set only covers the bug classes it contains — if you never include a deadlock case, the harness will stay blind to deadlock drift.

The harness also assumes the endpoint speaks the OpenAI chat format. If your gateway wraps it differently, you will need a small adapter. And the JSON-only prompt is a convention, not a guarantee; some models occasionally return extra text, and you will need a fallback parser for production use.

The five-minute habit that pays off

Run the calibration once when you first set up an AI reviewer. Save the baseline report. Then rerun it every time you change a model, a prompt, or an endpoint — including when you explore free grants. That ritual has saved me from merging at least two genuinely broken configurations.

So go ahead: try the harness against the free endpoints, commit the baseline, and see whether your reviewer still remembers what a dangling string_view looks like. If it does not, you just found out for the cost of a coffee run — not a post-incident review.