DEV Community

M.T.Ramkrushna
M.T.Ramkrushna

Posted on

Your LLM Judge Is Biased: Three Fixes Before You Trust Its Scores

#ai #testing #python #llm

You set up an LLM-as-a-judge to grade your AI feature. The dashboard says quality went up 12%. You ship.

Then users complain the new version is worse.

What happened? Very often, nothing was wrong with your app's model. The judge was measuring the wrong thing. LLM judges are useful, but they have predictable biases, and if you don't design around them, your eval quietly rewards the wrong answers.

This Monday's AI insight: treat your LLM judge like any other model in production. Test it before you trust it.

The three biases that show up most

The MT-Bench / Chatbot Arena paper (Zheng et al., 2023) found that strong judges like GPT-4 can reach over 80% agreement with human preferences, about the same level humans reach with each other. The same paper also documents the failure modes:

  1. Position bias. In pairwise comparisons, the judge tends to favor an answer because of where it appears (often the first one). Wang et al. showed you can flip rankings just by swapping the order of the candidates.
  2. Verbosity bias. Longer, more detailed-looking answers get rated higher even when they are not more correct.
  3. Self-preference. A model tends to rate its own outputs (or outputs that look like its own style) higher. Panickssery et al. found that LLM evaluators can recognize their own generations and favor them.

None of these mean "don't use LLM judges." They mean the default setup ("here are two answers, which is better?") is not an eval yet.

Fix 1: Swap the order and judge twice

For pairwise comparisons, run the judge twice with the candidates swapped. Only count a win when both orders agree. Treat disagreement as a tie (or send it to a human).

from openai import OpenAI

client = OpenAI()

JUDGE_PROMPT = """You are grading two answers to the same question.
Question: {question}
Answer 1: {a}
Answer 2: {b}
Which answer is more correct and more helpful? Ignore length.
Reply with exactly one token: 1, 2, or TIE."""

def judge_once(question, a, b, model="gpt-4o-mini"):
    r = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(question=question, a=a, b=b)}],
    )
    return r.choices[0].message.content.strip()

def judge_pair(question, old, new):
    first = judge_once(question, old, new)   # old in slot 1
    second = judge_once(question, new, old)  # new in slot 1
    if first == "2" and second == "1":
        return "new"
    if first == "1" and second == "2":
        return "old"
    return "tie"  # the judge changed its mind when the order changed
Enter fullscreen mode Exit fullscreen mode

Track how often you get tie from disagreement. If it's high, the judge is mostly reacting to position, and its scores were never telling you much.

Fix 2: Pass/fail against a written rubric, with the reason first

1-to-10 scores feel precise, but judges (and people) are inconsistent about what a 6 vs. a 7 means. Hamel Husain's guide on LLM judges recommends binary pass/fail decisions with a written critique, defined with a domain expert.

A rubric also fights verbosity bias, because the judge checks specific criteria instead of a general impression:

RUBRIC = """Grade the answer to the customer's question.
PASS only if ALL are true:
- It states the correct refund window from the policy below.
- It does not promise anything the policy does not allow.
- It answers in under 120 words.
Policy: {policy}
Question: {question}
Answer: {answer}

First write one sentence of reasoning. Then on a new line write PASS or FAIL."""
Enter fullscreen mode Exit fullscreen mode

Asking for a short reason before the verdict also gives you something to read when you debug the judge.

Fix 3: Measure the judge against human labels

This is the step most teams skip. Before the judge gates a release:

  1. Take 50 to 100 real examples.
  2. Have a person who knows the domain label each one PASS or FAIL.
  3. Run the judge on the same examples.
  4. Compare. 
from sklearn.metrics import cohen_kappa_score, confusion_matrix

human = ["PASS", "FAIL", "PASS", ...]   # expert labels
judge = ["PASS", "PASS", "PASS", ...]   # judge verdicts

print(confusion_matrix(human, judge, labels=["PASS", "FAIL"]))
print("kappa:", cohen_kappa_score(human, judge))
Enter fullscreen mode Exit fullscreen mode

Look at the confusion matrix, not just one number. A judge that passes everything can look "90% accurate" if 90% of your examples are good. What matters is whether it catches the failures. Read every disagreement, tighten the rubric, and re-run until you trust it on the cases that matter.

Eugene Yan's survey of LLM-evaluator research is a good read here: judges do well on some tasks and poorly on others, so the only way to know for your task is to check.

A few more habits that help

  • Use a different model family as the judge than the one generating answers, when you can, to reduce self-preference.
  • Pin the judge's model version and temperature. A judge that silently changes is a moving ruler.
  • Re-check agreement whenever you change the judge prompt or judge model, the same way you'd re-run tests after a refactor.
  • Keep a few known-bad examples in every eval run. If the judge passes them, the run is invalid.

The takeaway

An LLM judge is a model making predictions about quality. Give it a clear rubric, remove the easy biases (order, length, self-preference), and measure it against people before it decides what ships.

What does your team use today: pairwise comparisons, 1-10 scores, or pass/fail?

References

Top comments (0)