Nobody ships a payment system without tests, but teams ship LLM judges into production on vibes every day. A grader, a triage classifier, an eval pipeline's scoring model — if an LLM's judgment gates something users care about, its behavior is a production dependency. And unlike code, it changes when you didn't change it: a model-version bump, a quiet provider update, a prompt tweak that fixed one case and broke five others you never noticed.
The fix is boring and it works: regression tests. This is post 3 of the assessment-first series, and it ships rubric-bench v0.1 — a small TypeScript library and CLI that treats the grading engine from post 2 like any other code under test.
Three concepts, one loop
A golden set is a versioned JSON file of cases: a question (prompt, model answer, weighted rubric), one student answer, and the verdict a trustworthy grader must return: correct, partial, or incorrect. A run scores a grader against every case and reports accuracy overall and per tag. A diff compares two runs of the same set, case by case; a regression is a case that passed before and fails now.
rubric-bench run --set golden.json --out current.json --html report.html
rubric-bench diff baseline.json current.json # exit 1 on regressions
The exit codes are the point. Wire diff into CI, and the build that changes your grading prompt fails when it breaks a case that used to pass. Grader behavior stops being a thing you discover from user complaints and becomes a thing your pipeline tells you before merge.
What the first real bench run found
The repo ships a 72-case golden set for introductory statistics: 60 genuine answers spread across correct, partial, and incorrect, written in actual undergraduate register (hedges, typos, half-remembered rules), plus 12 adversarial cases we'll get to next post. First full run of the post-2 grader on claude-sonnet-5: 69/72, 95.8%.
The three misses are the interesting part. All three were expected-partial, graded-incorrect — answers like "standard deviation is just easier to communicate than variance, so it became the convention." A human TA gives that half credit for knowing that SD is preferred while missing why (the units). The model judged the why-criterion unmet and the what-criterion unmet too, harshly but defensibly. None of the misses were expected-correct answers getting rejected, and none were wrong answers getting credit. The failure mode is concentrated exactly where human graders also disagree: the partial-credit gray zone. That's a distribution you can ship a pilot on, and precisely the region the tone-calibration work in post 5 targets.
Two engineering notes from the first run, because reality bit immediately. One API response came back with no text block and killed the whole 72-case run; the runner now captures per-case errors as failed cases with an error field, and the reference grader retries once. And back-to-back identical runs differ by a case or two — LLM-as-judge non-determinism is well documented — so a single flipped case in a diff is noise, and a pattern of flips (three partials regressing after a prompt edit) is signal. The tool reports both; reading them is on you.
How to build a golden set for your own judge
The template transfers to any LLM-judgment task — support-ticket triage, content moderation, resume screening:
-
Discrete verdicts, not scores. You can regression-test
correct/partial/incorrect. You can't meaningfully diff 7.2 vs 6.9. - Real inputs first. Pull actual answers, tickets, or submissions before writing synthetic ones. Synthetic cases written by the same model family you're testing are systematically too easy.
- Make the partials genuinely ambiguous. If every case is obvious, 100% accuracy tells you nothing. Your gray-zone cases are the ones that catch drift.
- Tag everything. Per-tag pass rates locate a regression in seconds — "the outlier questions broke" beats "accuracy dropped 4%."
- Version the set and freeze cited runs. Once a number appears in a report or a blog post, that run file never gets regenerated.
Sixty cases took one focused authoring session. That's the entire capital cost of knowing, forever, whether your judge still behaves.
The business case is a sentence long
A grader that silently drifts mid-semester is a support catastrophe with a student body as witnesses; a triage model that drifts after a provider update mis-routes customers for weeks before anyone correlates the timing. The Dartmouth pilot this series builds on ran its whole term on one pinned model for a reason. Pin the version, bench the swap, gate the prompt edits. It's the same discipline you already apply to schema migrations, applied to judgment.
Where this breaks
rubric-bench is regression testing, not psychometrics: it tells you the grader changed, not that it agrees with human raters at a publishable kappa; a real deployment eventually wants an inter-rater study against instructor grades, which this golden set can't substitute for. Seventy-two cases is a floor, not a benchmark; per-tag cells of 6 cases can't distinguish a real regression from noise at single-case granularity. And a golden set authored by one person inherits one person's judgment calls about partial credit — the honest fix is a second rater pass over the labels, which is on the roadmap.
Next post: the 12 cases I skipped. Students will put prompt injections in their homework, and what the bench found when I attacked my own grader surprised me twice.
Top comments (0)