Your LLM Judge Agrees With Itself. That Is Not an Eval.

Series: What AI Gets Wrong

A judge model can give the same verdicts run after run and still be wrong. In a June 2026 preprint on judge reliability [1], Qwen 3 8B reproduced its own scores almost perfectly on MT-Bench, with a test-retest reliability of 0.992, while its verdicts leaned toward one answer slot by about 19 percentage points, whichever answer sat in it.

Even agreement with the truth can hide a lot. Suppose your system fails 5% of the time, and your judge answers "pass" for every single output. Measured against reality, that judge agrees with the truth 95% of the time. It also catches none of the failures [2].

Nobody sets out to build a judge that favours an answer slot or passes everything. But picture a team that wires in a model to score outputs: the score comes back stable, the dashboard goes green, and the number starts to carry weight in release decisions, because it looks like measurement.

My claim is narrow. Which judge model you pick is a small decision. Where human ground truth enters the evaluation loop is the larger one, and I group the published guidance into four entry points. Each has a cost, and a team can skip any of them and still get a clean-looking number.

Consistent is not valid

The preprint behind the opening [1] ran 21 judge models through roughly 541,000 judgments. Test-retest reliability is how closely a judge's scores match across repeated runs, and Qwen 3 8B's 0.992 is that statistic (Krippendorff's alpha). Its position bias was 0.192; in the paper's terms that is the distance of one answer slot's win rate from 50%, which is where the 19 points come from. The authors call this the consistency-bias paradox: a judge can reproduce its own answers almost perfectly while still letting answer order sway the verdict. (It is a preprint, so treat the specific figures as provisional until it has been through review.)

On a dashboard, consistency is the property you can see, because rerunning a judge is cheap. Validity needs something outside the judge to compare it against.

The second piece of evidence is about the metric itself. A 2025 ACL workshop paper titled "Judging the Judges" [3] compared thirteen judges (eleven LLMs and two string-match baselines) on a question-answering task, with humans as the reference. Two humans labelling the same answers agreed at a chance-corrected level of 0.96 (Scott's pi). The 70B Llama-3 judge reached 0.88 and GPT-4 Turbo 0.87. The small Llama-3 8B judge scored 0.59 on that measure, even though its raw percent agreement with humans was well above 80%. The paper's conclusion is that judges are hard to tell apart by percent agreement, and that the chance-corrected score is the more informative signal. The paper also estimates a leniency for each judge, how likely it is to say "correct" when it is not really applying the criteria. For most judges that estimate sat well above 50%, from roughly 0.74 to 0.90, and for Llama2-70B it was 0.99.

The reliability preprint [1] reports a similar gap at scale, with chance-corrected agreement trailing exact-match agreement by 33 to 41 points on MT-Bench, and judge rankings shifting by as many as 14 places depending on the benchmark.

The older, better-known work documented the biases underneath. The paper that introduced MT-Bench and Chatbot Arena [4] found that GPT-4 agreed with human experts about 85% of the time when ties were excluded, against 81% for two humans agreeing with each other. That is the result people quote to justify using a judge. Hold on to one detail: the 85% figure exists because 58 expert labellers cast roughly 3,000 votes. The evidence that a judge can match humans was produced by humans.

The same paper also documents why judges fail. Verdicts changed when the answer order was swapped; GPT-4 stayed consistent on 65% of pairs, GPT-3.5 on 46%, and Claude-v1 on 24%. An attack that put a rephrased version of an answer's own list in front of it fooled Claude-v1 and GPT-3.5 in over 90% of cases and GPT-4 in under 10%, though it was tried on only 23 answers. And the judges tended to favour their own outputs, although the authors were careful to say their data could not settle whether that was self-preference. Later work treats it as a real effect: a NeurIPS 2024 paper [5] found that the more reliably a model recognises its own writing, the more it favours it, and a NeurIPS 2024 workshop paper [6] links the bias to how familiar the text is to the judge.

So the metric to report is agreement with human labels, corrected for chance, on a set that represents your task. Agreement with another model, or with the same model on a second run, does not measure that.

Four places the human has to be

If you accept that, the design question changes. It stops being "which judge" and becomes "where does a person's judgment enter, how much of it, and what makes us do it again". Reading across the published guidance, I group it into four entry points. The grouping is mine; no single source proposes it.

1. Criteria: who decides what "good" means

The first entry point is the least obvious, because it happens before any judge exists. You need a rubric to grade outputs, and the intuitive plan is to write the rubric first. A 2024 UIST paper studying how people build LLM evaluators [7] found it only partly works like that. The authors call it criteria drift: some of what people care about only shows up once they grade real outputs, and the act of grading changes the rubric.

The practical consequence is that a person has to look at real outputs first. The practitioner guide suggests starting with around 30 examples [2], and a practitioner FAQ on evals, co-written by the guide's author [8], sets a minimum of 100 traces for this error analysis, to be continued until new traces stop revealing new failure modes.

This costs expert time up front and cannot be handed to a model, since the model is what the criteria are meant to constrain. A generic rubric (helpfulness, relevance, tone) with a judge attached is the tempting way around it; the score it returns will be steady, but nobody on the team will have defined the quality it measures.

2. Judge validation: a held-out human-labelled set

Once criteria exist, the judge has to be checked against people on examples it was not tuned on. Two decisions live here: the metric and the size of the set.

On the metric, the evidence above points one way. Use a chance-corrected statistic such as Cohen's kappa, or treat the judge as a binary classifier and report its true positive and true negative rates separately. The guide's version of this [2] is the most useful for a decision-maker, because it forces you to ask the question the 95% example hides: of the real failures, how many did the judge catch?

On size, the public sources count differently. A Microsoft Learn training module [9] suggests 50 to 100 expert-labelled interactions and treats a kappa above 0.7 as good agreement; that threshold is a rule of thumb from a training unit. The guide points to about 100 examples per failure mode [2], and the FAQ says 100 to 200 for each failure mode [8]. Note the units: Microsoft's figure is the whole calibration set, while the guide's and the FAQ's are per failure mode, so the numbers cannot be compared directly. The guide also cautions that with fewer than 60 examples the statistical uncertainty is often too large to conclude much [2]. Under the per-failure-mode guidance, my inference is that the cost grows with the number of distinct failure modes you care about.

Who labels matters as much as how many. The FAQ's authors recommend a single domain expert for most small to medium-sized companies, rather than a committee [8], and the guide is blunt that a developer standing in for the expert is a common way to go wrong [2]. A practitioner write-up on LLM evaluators makes the related point that agreement numbers can be inflated when the annotators are not experts [10].

3. Estimation: correcting the number you report

A validated judge is still imperfect, and its errors can shift the pass rate it reports. A small human-labelled sample can be used to correct that estimate, and a line of statistical work shows how. Prediction-powered inference [11] combines a small labelled set with a large set of model predictions to produce confidence intervals that stay valid. A follow-up called AutoEval Done Right [12] reports an effective human sample size up to 50% larger than the labels alone provide, while remaining unbiased; that is the best case in its experiments, not a typical gain.

An ICLR 2025 paper [13] proves that when the judge is no more accurate than the model being judged, no debiasing method can cut the number of human labels you need by more than half. In practice its authors found the savings were smaller still. The condition matters: it is about the judge's accuracy on the task compared with the model under test, so it bites hardest at the frontier, where the model being judged is about as capable as the judge. A judge that is clearly more accurate than the model it scores is outside the bound. For the hard cases, the ones where an eval earns its keep, the label budget stays at half or more of what you would need with no judge at all.

So the choice is whether to keep paying for a human sample every time you report a number. Without one, the pass rate you report is the judge's raw output.

4. Re-validation: what makes you check again

A judge is a model, and models change underneath fixed names. A 2023 study [14] compared GPT-4 across a few months and found, for one task (identifying prime numbers), accuracy falling from 84% in March to 51% in June, under the same model name. The authors concluded that these systems need continuous monitoring.

The guide's advice is to repeat validation after a model update [2], and the FAQ [8] suggests rerunning error analysis on production traces every few weeks, which is a natural moment to recheck the judge as well. Swapping or silently upgrading the judge changes the instrument: your historical scores came from one judge and the new ones from another. My view is that a judge upgrade should be gated like a schema migration, with a human re-check before the new scores are accepted. Without that check, a shift in the scores after a judge change cannot be read as a change in the system.

A cheap form of this is routing: cases where the judge and the humans disagree on the sampled set become the next review queue. The Microsoft Learn module [9] uses a related idea, running several judges and sending cases where they disagree with each other to a human. Nothing in this loop forces anyone to own the human-labelled set after launch, so the labels can go stale while the system and the judge both move.

Entry point	The decision	What published guidance says	The cost
Criteria	Who defines "good", and from what examples	Look at real outputs first; criteria shift as you grade	Expert time up front
Judge validation	Held-out labels and a chance-corrected metric	Whole set of 50 to 100 (Microsoft [9]); about 100 per failure mode (guide [2]); 100 to 200 per failure mode (FAQ [8])	Likely grows with the number of failure modes
Estimation	Correct the reported rate with human labels	Statistical correction helps, but at the frontier (judge no stronger than the model under test) label savings are bounded	A running label budget
Re-validation	What triggers a re-check	After model updates; periodic error-analysis cycles	Ownership of the golden set

What judging costs and where the effort goes

It is tempting to skip the steps above on cost grounds, so it helps to separate which cost is real.

Judging itself can be cheap to run. One paper [15] replaced one large judge with a panel of smaller models from different families and reported it came out over seven times cheaper than a GPT-4 judge, with less bias from any single model family.

From here I am inferring, not quoting. Expert time for criteria, labels and error analysis does not follow that price curve. The FAQ's authors report spending 60 to 80% of their development time on error analysis and evaluation [8]; that describes their own projects and covers more than labelling, but it hints at where the effort goes. Given that, a modest, well-labelled set and a cheap panel look to me like a better use of money than the largest single judge available.

The choice of judge model will probably be reopened every few months, while the questions of who labels, how many, how often and who owns the set stay with the system.

When a judge alone is enough

The "Judging the Judges" paper [3] points out that a judge with a steady bias can still order models much as a better judge would, even when its scores are off. AlpacaEval [16] fits that picture: even before its length correction, its Spearman rank correlation with human votes on Chatbot Arena was 0.94, and the length-controlled version raised it to 0.98. So if the question is whether version B beats version A, a judge with known biases may be enough, though AlpacaEval is itself anchored to human votes.

Some criteria do not need a judge at all. Whether an output contains a required field can be settled by a string or schema check. Anthropic's evaluation guide [17] lists exact match and string match under code-based grading, and a schema check is my extension of the same idea. Moving such checks into code should shrink the part of the problem that needs human labels.

A judge that has already been validated is the case vendor guidance has in mind when it says to scale up. OpenAI's guidance [18] puts that point where the judge has become quicker and cheaper than people while still matching their annotations reliably, and it names skipping that calibration as an anti-pattern. (Reading that as validation on your own data is my gloss.)

On using the same model as generator and judge, the sources disagree. The FAQ [8] says it is usually fine, provided you check alignment on a held-out human set. Comments in Anthropic's grader examples [17] recommend, as a general practice, using a different model from the one that wrote the output.

Humans are noisy too. In the MT-Bench data [4], two humans agreed only 63% of the time on first-turn questions once ties were counted (67% on second turns), so here "ground truth" is a labelled judgment by a qualified person. Anthropic's guide [17] rates human grading as high quality but slow and expensive, advises avoiding it where possible, and in its design principles favours many automatically graded questions over a few hand-graded ones. That pulls toward automation, and the same page says to test an automated grader's reliability first. I would put humans on the calibration set and let the judge handle the traffic.

The takeaway

Choose the judge last. First decide who defines the criteria, who labels the validation set, who pays for the correction sample, and what triggers a re-check when the model changes.

I would expect re-validation to be the entry point skipped most often, because it is the only one that has to keep happening after the first release.

The previous post on guardrails covered the runtime side of this question: Your Guardrail Can Detect an Attack. It Cannot Authorize a Refund.

Enjoyed this breakdown? I write about decisions like this one in my newsletter, AI + Architecture, where I cover the architectural choices behind AI systems, including how to decide what to measure and who does the measuring.

Subscribe here →

Views are my own and not those of my employer.

---

References

Numbers match the bracketed citations in the text. All fetched 2026-10-28. Papers marked preprint have not been confirmed as peer reviewed.

1. Norman, Rivera, Hughes, "Reliability without Validity" (preprint, June 2026): https://arxiv.org/abs/2606.19544
2. Hamel Husain, "Using LLM-as-a-Judge For Evaluation: A Complete Guide": https://hamel.dev/blog/posts/llm-judge/
3. Thakur et al., "Judging the Judges" (GEM2 workshop at ACL 2025, v6): https://arxiv.org/html/2406.12624v6
4. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023 Datasets and Benchmarks): https://arxiv.org/abs/2306.05685
5. Panickssery, Bowman, Feng, "LLM Evaluators Recognize and Favor Their Own Generations" (NeurIPS 2024): https://arxiv.org/abs/2404.13076
6. Wataoka, Takahashi, Ri, "Self-Preference Bias in LLM-as-a-Judge" (NeurIPS 2024 Safe GenAI Workshop): https://arxiv.org/abs/2410.21819
7. Shankar et al., "Who Validates the Validators?" (UIST 2024): https://arxiv.org/abs/2404.12272
8. Hamel Husain and Shreya Shankar, Evals FAQ: https://hamel.dev/blog/posts/evals-faq/
9. Microsoft Learn, "Implement LLM-as-judge evaluation for multi-agent systems": https://learn.microsoft.com/en-us/training/modules/aaai-design-evaluation-frameworks-multi-agent-azure/3-implement-model-judge-multi-agent-evaluation
10. Eugene Yan, "Evaluating the Effectiveness of LLM-Evaluators": https://eugeneyan.com/writing/llm-evaluators/
11. Angelopoulos et al., "Prediction-Powered Inference": https://arxiv.org/abs/2301.09633
12. Boyeau et al., "AutoEval Done Right": https://arxiv.org/abs/2403.07008
13. Dorner, Nastl, Hardt, "Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data" (ICLR 2025): https://arxiv.org/abs/2410.13341
14. Chen, Zaharia, Zou, "How is ChatGPT's behavior changing over time?": https://arxiv.org/abs/2307.09009
15. Verga et al., "Replacing Judges with Juries" (PoLL): https://arxiv.org/abs/2404.18796
16. Dubois et al., length-controlled AlpacaEval: https://arxiv.org/abs/2404.04475
17. Anthropic, "Create strong empirical evaluations": https://platform.claude.com/docs/en/docs/test-and-evaluate/develop-tests
18. OpenAI, Evaluation best practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices

---