Originally published on Loop & Retry — field notes on building LLM agents that survive production.
The last eval post argued that an LLM-as-judge is a biased instrument you're reading as a ruler. The natural follow-up: how do you check the ruler? If a model grades your agent and you ship on its scores, then the judge is a load-bearing part of your release process — and an unvalidated judge is a measurement device you've never once compared against a known standard. This post is about meta-evaluation: evaluating the thing that evaluates. It builds on both what to actually measure when your agent works (which covers the multi-layer evaluation harness) and how LLM judges are biased (which covers the specific failure modes).
The stakes are specific. A noisy judge just widens your error bars; annoying, survivable. A biased judge is the real problem, because bias doesn't cancel — it pushes your headline metric in one direction consistently, and every release looks better (or worse) than it is. You can average away noise. You cannot average away a systematic tilt you never measured.
Step 1: you need a gold set, and it has to be small enough to be real
Meta-evaluation requires ground truth the judge never sees during grading: a set of outputs with human labels. The instinct is to make it big. Resist. A gold set of 80 carefully-adjudicated examples beats 800 hastily-labeled ones, because the whole point is that these labels are more trustworthy than the judge — and label quality collapses when you rush volume. Include the hard cases on purpose: near-misses, outputs that are fluent but wrong, the adversarial shapes where you suspect the judge is weak. A gold set of only easy cases certifies a judge that will fail you exactly where it matters.
Adjudicate disagreements between human labelers explicitly and keep the ones that were genuinely ambiguous flagged — you'll want to know later whether the judge is "wrong" on a case that two humans also split on.
Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article
Top comments (0)