"LLM-as-judge" has quietly become infrastructure. We use one model to grade another's answers, to decide if a test really passed, to score RAG relevance, to moderate, to rank. It's cheap, fast, and scales to volumes no human can touch.
It's also a measurement instrument — and most teams deploy it without ever checking whether the instrument is accurate. Here's a practical checklist before you let an LLM grade anything that matters. (Context: I built a small harness that runs a local LLM as a test judge and scores it against ground truth; some numbers below come from that.)
1. Measure it against ground truth at least once
You cannot manage what you haven't measured. Assemble a set of cases where you know the right verdict (labelled correct/wrong) and run the judge over them. You're looking for:
- False-pass rate — of everything it approved, how much was actually wrong? (The dangerous direction.)
- Miss rate — of the genuinely wrong items, how many did it wave through?
- False-alarm rate — how often does it reject good work? (The annoying direction.)
In my run, a capable open model missed 26% of wrong outputs — a quarter of real failures, passed. You want your number, on your task, before you rely on it.
2. Check calibration, not just accuracy
Ask the judge for a confidence, then check whether that confidence means anything. A well-calibrated judge that says "90% sure" should be right about 90% of the time. Measure Expected Calibration Error (ECE).
The common failure: models are massively overconfident. Mine answered with ~100% confidence on nearly every call — including the wrong ones (ECE ≈ 0.31). If the confidence is always "very sure," it carries no signal and you can't use it to route hard cases to a human.
3. Pin everything and version it
An LLM judge whose model, prompt, or temperature drifts is a ruler that changes length. Pin:
- the exact model (and version),
- the prompt template,
- temperature (use 0 for grading — you want determinism, not creativity).
Open-weight models help here: you can pin a local model and know it won't change under you. A hosted endpoint can be swapped silently.
4. Watch for the known biases
LLM judges have documented, reproducible biases:
- Position bias — prefers the first (or last) option in a pairwise comparison. Mitigate by running both orders and averaging.
- Verbosity bias — rates longer answers higher regardless of correctness.
- Self-preference — a model tends to favour outputs that look like its own.
- Leniency — many judges default to "PASS" when unsure.
If your eval drives a leaderboard or a release decision, these biases become your biases.
5. Never let it be the only gate
The safe pattern isn't "judge decides, ship." It's:
- Use the judge as a cheap first-pass filter, not the final word.
- Keep a ground-truth backstop on a sample — spot-check the judge continuously, not once.
- Route low-confidence or high-stakes cases to a human or a stronger check.
- Treat a PASS as "probably fine," not "verified."
The one-line version
An LLM judge is a fast, cheap, biased, overconfident grader that is right often enough to be useful and wrong often enough to be dangerous. Measure it, pin it, debias it, and keep a human backstop — then it's a great tool. Trust it blindly and it will quietly approve your worst bug with total confidence.
Are you using LLM-as-judge in production? What did you find when you checked it against ground truth — or have you checked yet?
I'm Krishnasai Rollamadugu — a test automation architect researching trust and measurement in AI-driven software testing. Related: an open experiment measuring a local LLM judge — trust-judge.
Top comments (5)
Two things that change how to read the 26% miss rate in item 1.
First, it only has meaning with a count behind it. If the labelled set had 50 wrong outputs, 13 were waved through and the 95% interval is roughly 15% to 40%; with 200 it narrows to about 20% to 32%. Worth printing the interval next to the rate, because "26% vs 20%" between two judge prompts is often inside it.
Second, the miss rate is fixed by the judge but the false-pass rate you listed first is not: it depends on how often wrong outputs occur. With a 26% miss rate and, say, a 5% false-alarm rate, a set that is 50% wrong gives about 22% of approved items being wrong, while a set that is 10% wrong gives about 3%. So the same judge looks very different in a stress set built from known failures than on production traffic. Reporting miss rate and false-alarm rate on the labelled set, then computing the false-pass rate at your real failure rate, avoids quoting the wrong one.
How many wrong cases were in your ground-truth set?
@arhancanli Thanks — both points are right, and the second is a genuine framing mistake on my part.
On the denominator: agreed, the rate means nothing without N and an interval. My ground-truth set had [N] wrong cases, which puts the 95% band at roughly [X–Y]% — wide enough that "26% vs 20%" between two prompts is mostly noise until I grow the set. I should have printed the interval next to every rate, and I'll fix that in the post.
On miss rate vs false-pass rate: you're correct that I conflated them. Miss rate — P(approve | wrong) — is a property of the judge. The false-pass rate I led with isn't; it depends on how often wrong outputs actually occur, so it's a property of the dataset. My numbers came from a stress set built largely from known failures (a high wrong-rate), so that figure is near a worst case, not production traffic. The honest way to state it is what you described: report miss rate + false-alarm rate on the labelled set, then compute the deployed false-pass rate at the real failure rate via m·w / [m·w + (1−a)(1−w)]. I'll rework that section to do exactly that.
Ground truth: [N] wrong out of total, false-alarm rate [a]%. [If N is small, add: so the intervals are wide and I'll quote them rather than point rates.]
This is exactly the kind of correction the post was asking for — appreciate it.
Before you edit: the comment above still has template placeholders in it ([N], [X–Y], [w], [a]) and one of them rendered as a broken link, so the actual counts never made it in. Once you have the real number of wrong cases, the interval is quick to compute (Wilson is better than the normal approximation at small N), and the deployed false-pass formula is then just those two measured rates plus your production failure rate.
When you rework the section, I'd print the assumed failure rate next to the false-pass figure, so readers can swap in their own.
Ha, that's entirely on me. I pasted my own working notes with the blanks still in, so you got the template instead of the data. Thanks for catching it before it calcified.
Agreed on all of it. Wilson over the normal approximation, since my N is small enough that the difference is real rather than cosmetic, and I'll print the assumed failure rate right next to the false-pass figure so readers can swap in their own base rate instead of inheriting my stress set's. That stress set is the whole reason the single number misled in the first place, so making the base rate an explicit, swappable input is the right fix.
Reworking the comment and that section now with the actual counts and the interval spelled out — numbers this time, not placeholders.
Calibration matters more than a confident PASS. If the judge reports near 100% confidence even when wrong, human escalation rules become useless without continuous ground-truth sampling.