Rubric Graders Need Canaries: Testing the Test That Grades C++ Patches

A pipeline starts scoring C++ patches with a model-backed rubric. Diffs that change printed precision, drop a noexcept guarantee, or leak a file descriptor now receive a 1-10 score before a human opens them. For a few weeks the dashboard looks reasonable, reviewers trust the number, and nobody asks whether the grader still separates good patches from bad ones. Then a prompt edit lands, the upstream model is swapped, and every patch scores 8. Nothing crashes. The grader simply stops discriminating, and the pipeline keeps reporting green.

That failure mode is independent of which model you call. A rubric grader is a program with an untested dependency, and it deserves the treatment the code under it already gets: golden cases, a canary set, a hard gate, and a ledger. The harness below is standard-library Python, a few JSON canary files, and one cron line. It is deliberately boring, because the interesting part is that the grader fails loudly instead of silently.

Layer 0 thinks, layer 1 judges

Split grading into two layers before writing any code. The deterministic layer answers questions with one correct answer: does the patched tree compile, does the golden case exit zero, does stdout match after a fixed locale and precision. The rubric layer answers questions with a range of defensible answers: is the naming consistent with the header it implements, does the commit message describe the actual change, does the new overload respect the documented contract.

Most silent regressions live on the boundary. A grader that is only deterministic cannot see prose contracts. A grader that is only a rubric cannot see a changed std::setprecision. Layer 0 gates the build, layer 1 explains the patch, and each layer needs its own canaries.

A canary here means a labeled example with a fixed expected answer that is committed to the repository. It is not a test of the patch. It is a test of the test.

Step 1 - Freeze golden cases for the deterministic layer

Start with one golden case per documented output contract. Keep the case small enough that a failure points at one behavior, and pin everything that could drift: locale, precision, and the toolchain described in your own build docs.

// golden/report_precision.cpp - contract: ratio printed with 3 decimals
#include <iostream>
#include <iomanip>

int main() {
    const double ratio = 2.0 / 3.0;
    std::cout << std::setprecision(3) << std::fixed << ratio << std::endl;
    return 0;
}

Run it through the same command the eval batch uses, so the grader and the patch pipeline cannot disagree about flags:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target report_precision
./build/report_precision > out.txt
diff -u golden/report_precision.expected out.txt

The diff is the grader. If it passes, the patch is not yet good, but it is at least not obviously broken.

Step 2 - Commit canaries in both directions

A canary set that only contains bad patches rewards a paranoid grader. A canary set that only contains good patches rewards a lenient one. Include both, and store each patch as its own file so the JSON stays readable and diffs stay reviewable.

eval/
  canaries/
    precision-3-to-2.json
    honest-refactor.json
    tie-case-review.json
    diffs/
      precision-3-to-2.patch
      honest-refactor.patch
      tie-case-review.patch

Each canary declares what it expects and why. The note field is the part humans actually maintain.

{
  "id": "precision-3-to-2",
  "kind": "rubric",
  "direction": "must-fail",
  "patch_path": "diffs/precision-3-to-2.patch",
  "expected": 3,
  "note": "Breaks the output contract in docs/report.md. Any score above 5 means the rubric is not reading the contract."
}

{
  "id": "honest-refactor",
  "kind": "rubric",
  "direction": "should-pass",
  "patch_path": "diffs/honest-refactor.patch",
  "expected": 8,
  "note": "Pure rename plus a comment fix, output byte-identical. Below 6 means the rubric punishes chore diffs."
}

{
  "id": "precision-3-to-2-build",
  "kind": "build",
  "patch_path": "diffs/precision-3-to-2.patch",
  "golden_cmd": "./build/report_precision | diff -u golden/report_precision.expected -",
  "note": "Must exit non-zero. If this passes, the deterministic layer is not running."
}

Keep the count honest. Twenty labeled canaries drawn from real review comments are worth more than two hundred generated ones, because generated canaries encode the same assumptions as the rubric prompt.

Step 3 - Measure agreement before trusting scores

Agreement is the only cheap health signal a rubric grader has. Exact match is strict; within-one match tolerates the ordinary noise of a 1-10 scale. Both numbers belong in the gate.

#!/usr/bin/env python3
# grade_gate.py - run canaries before trusting rubric scores.
import json, os, pathlib, subprocess, sys

CANARY_DIR = pathlib.Path(os.environ.get('CANARY_DIR', 'eval/canaries'))
LEDGER = pathlib.Path(os.environ.get('GRADER_LEDGER', 'eval/grader_ledger.jsonl'))
RUBRIC_ID = os.environ.get('GRADER_VERSION', 'unversioned')


def load_canaries():
    for path in sorted(CANARY_DIR.glob('*.json')):
        canary = json.loads(path.read_text())
        patch = (CANARY_DIR / canary['patch_path']).resolve()
        canary['diff_text'] = patch.read_text()
        yield canary


def call_model(prompt):
    # Transport layer. Plug in whichever client your runtime provides.
    # This function is untrusted by design; that is why canaries exist.
    raise NotImplementedError('wire your client here')


def rubric_score(diff_text, rubric):
    reply = call_model(rubric + '\n\nDIFF:\n' + diff_text)
    return json.loads(reply)

Now the comparison itself, plus the gate. The thresholds are policy defaults, not measured results: a canary set of forty concrete review-derived cases can usually carry a higher bar than one of twelve fuzzy ones. Tune them once, write the number in the repo, and treat any change to it as a reviewable commit.

def check_rubric_canaries(canaries, rubric, min_exact=0.80, min_within_one=0.95):
    exact = within_one = 0
    misses = []
    for canary in canaries:
        if canary.get('kind') != 'rubric':
            continue
        expected = canary['expected']
        got = rubric_score(canary['diff_text'], rubric)['score']
        if got == expected:
            exact += 1
        else:
            if abs(got - expected) == 1:
                within_one += 1
            misses.append({'id': canary['id'], 'expected': expected,
                           'got': got, 'delta': got - expected})
    total = sum(1 for c in canaries if c.get('kind') == 'rubric') or 1
    rates = {'exact': exact / total, 'within_one': (exact + within_one) / total}
    ok = rates['exact'] >= min_exact and rates['within_one'] >= min_within_one
    return ok, rates, misses

Build canaries run the golden command instead of a model, and check the opposite thing: they must fail.

def check_build_canaries(canaries, workdir):
    results = []
    for canary in canaries:
        if canary.get('kind') != 'build':
            continue
        subprocess.run(['git', 'apply', canary['patch_path']], cwd=workdir, check=True)
        proc = subprocess.run(canary['golden_cmd'], cwd=workdir, shell=True)
        subprocess.run(['git', 'restore', '.'], cwd=workdir, check=True)
        results.append({'id': canary['id'], 'exit': proc.returncode,
                        'ok': proc.returncode != 0})
    return results

Step 4 - Keep a grader ledger next to the token ledger

A score is only meaningful with the grader revision and the model identity attached. Write one JSON line per gate run, including the token count, so drift and cost are visible in the same file.

def log_run(rates, misses, model_id, tokens):
    record = {'grader': RUBRIC_ID, 'model': model_id, 'rates': rates,
              'misses': misses, 'tokens': tokens}
    with LEDGER.open('a') as fh:
        print(json.dumps(record), file=fh)

if __name__ == '__main__':
    rubric = pathlib.Path(sys.argv[1]).read_text()
    canaries = list(load_canaries())
    ok, rates, misses = check_rubric_canaries(canaries, rubric)
    build = check_build_canaries(canaries, '.')
    log_run(rates, misses, os.environ.get('MODEL_ID', 'unknown'),
            int(os.environ.get('TOKENS_USED', '0')))
    healthy = ok and all(r['ok'] for r in build)
    print(json.dumps({'healthy': healthy, 'rates': rates, 'misses': misses,
                      'build': build}, indent=2))
    sys.exit(0 if healthy else 1)

The exit code is the whole interface. A healthy gate returns 0 and the eval batch proceeds; an unhealthy gate returns 1 and the batch stops before it spends anything on scoring.

Step 5 - Run the gate where nobody has to remember it

A gate that depends on a human typing a command is a gate that will be skipped during the week it matters most. Two things make the scheduled version practical: a machine that stays on, and model access that does not require a per-run decision.

MonkeyCode, described by its maintainers as an open-source project, offers free model access and a free server option, with the free tier described as including 10M tokens. That combination is enough to host a weekly canary run plus the rubric calls it makes, without reserving a dedicated box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# cron: every Monday 06:17, on the free server option
17 6 * * 1 cd /srv/eval && python3 grade_gate.py rubrics/patch_v3.md >> logs/canary.log 2>&1

Read the ledger rather than the estimate. The claim worth trusting is not a predicted token count; it is the tokens field on last week's line, compared with the line before it.

Which layer owns which signal

Signal to grade	Layer 0 (deterministic)	Layer 1 (rubric)
Compiles, links, exits zero	Yes	No
stdout under fixed locale and precision	Yes	No
Wall-clock or allocation budget	Yes	No
Naming consistent with the header	Partly, via lint rules	Yes
Prose output contract in docs	No	Yes
Commit message matches the change	No	Yes
Whether the change is worth merging	No	Advisory only, never binding

If a row can be pushed into layer 0, push it. Every signal moved out of the rubric layer removes a source of drift, and the rubric layer keeps only the judgments a compiler cannot express.

Limitations and who should skip this

Canaries are a sample, and agreement is not correctness. If the labels are wrong, the gate certifies a wrong grader with a confident number. Re-label a handful of canaries every few months, especially the ones that never fail.

Rubric scores drift when the model or the prompt changes, and only a scheduled run of the ledger detects that. Single-run scores stay noisy even with a fixed prompt, so treat a one-point swing as a signal to check the ledger, not as a verdict on the patch.

The quota and server claims above are operator descriptions of the offer, not measurements taken for this article. Verify current terms before wiring a paid-critical pipeline to a free tier, and keep the gate able to run against a different client.

Skip this approach if you have fewer than roughly twenty real diffs to label, because reading the patches is then cheaper than grading them. Skip it for safety-critical, licensing, or compliance judgments, which need deterministic evidence and a named reviewer. Skip it if diff text cannot leave your network, unless every layer runs locally. And skip the rubric layer entirely if your grading budget exceeds the cost of the human review it is meant to save.

The harness is small on purpose. Golden cases, canaries in both directions, one agreement gate, one ledger. Start with a single canary that should fail and confirm the gate returns non-zero; if it returns zero, the grader was never watching.