LLM-as-judge uses a capable model to score or compare agent outputs against a rubric — filling the gap where quality is open-ended and human judgment doesn't scale.
Done well it approximates human judgment cheaply; done carelessly it produces confident nonsense. The keys: a specific rubric, pairwise over absolute scoring, and validating the judge against human labels.
Why use a model as a judge
Many of the qualities that matter most — helpfulness, faithfulness, reasoning quality — have no formula, and human judgment does not scale to thousands of cases on every change. Using a capable model as a judge fills that gap: you ask a model to score or compare outputs against a rubric. Done well it approximates human judgment at a fraction of the cost and effort. Done carelessly it produces confident, systematic nonsense — which is why the details below matter.
Why a model can judge at all
It can seem circular to use a model to evaluate a model — if the judge could reliably tell good from bad, why not use it as the agent? The resolution is that judging is easier than doing. Recognizing whether an answer is faithful to a source is narrower and more constrained than producing the faithful answer, the way it's easier to check a proof than to find one. The judge is handed the input, the output, and a rubric, and asked only to assess against that rubric.
The rubric is everything
The quality of the judgment depends almost entirely on the rubric. A vague instruction to "rate this 1-10" yields noise; a specific rubric that defines each level and what to look for yields something usably consistent. The judge's reliability comes from the structure the rubric provides.
judgment = judge_model(
task=original_input,
response=agent_output,
rubric="""Score faithfulness 1-5.
5 = every claim supported by the sources.
3 = mostly supported, minor unsupported detail.
1 = key claims not supported / contradicted.""",
)
Free Agent Evaluation QuickStart — the whole loop (define, measure, test, trust) on a few pages. Download it free.
Pairwise beats absolute scoring
Models are more reliable at comparing than at scoring in the abstract. Asking "which of these two responses is better?" tends to be far more consistent than "rate this 1-10," because absolute scores drift and cluster while comparisons are anchored. Whenever you can frame evaluation as a comparison — against a reference, or between two versions of the agent — you get more reliable signal.
Ask which is better, not how good.
The biases to defend against
- Position bias — judges can favor whichever answer comes first; swap the order and average to cancel it.
- Verbosity bias — judges often prefer longer answers regardless of quality; call it out in the rubric.
- Self-preference — a judge may favor outputs from its own model family; be aware when judge and agent share a model.
- Validate the judge — check it against human labels on a sample. An unvalidated judge is an opinion, not a measurement.
Going deeper? AI Agent Evaluation & Testing: The Complete Guide is the full reference — 40 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.
FAQ
What is LLM-as-judge?
Using a capable model to score or compare agent outputs against a rubric, in place of human judgment at scale. It's the workhorse for evaluating open-ended qualities like helpfulness and faithfulness that have no formula.
Is LLM-as-judge reliable?
It can be, with care: a specific rubric, pairwise comparison over absolute scoring, and validation against human labels. Without those it produces confident but systematic errors. An unvalidated judge is an opinion, not a measurement.
Why is pairwise scoring better than 1-to-10?
Models are more consistent comparing two outputs than scoring one in isolation. Absolute scores drift and cluster; comparisons are anchored. Framing evaluation as 'which is better, A or B?' yields more reliable signal.
What biases affect LLM judges?
Position bias (favoring the first answer), verbosity bias (favoring longer answers), and self-preference (favoring the judge's own model family). Counter them by swapping order and averaging, calling out length in the rubric, and validating against humans.
How do I know if my judge is accurate?
Validate it against human labels on a sample of cases. If the judge's scores track human judgment, you can trust it at scale; if not, fix the rubric. An unvalidated judge may be systematically wrong in ways you can't see.
Top comments (0)