LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the "we're shipping v2" message typed.

Then I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%.

The judge, the answers, the rubric and temperature 0 were all the same. The only change was which answer sat in the slot labeled "A". That is LLM-as-judge position bias, and if your eval script calls judge(new, old) in a fixed order, your headline number is probably measuring it too.

TL;DR

• LLM-as-judge position bias means a pairwise judge prefers whichever answer sits in a particular slot (usually the first) regardless of quality.
• In my run, swapping A and B flipped 87 of 300 verdicts (29%), and 78 of those flips went to whichever answer was in slot A.
• The bias shows up in close calls. When two answers are near-equal, slot position acts as the tiebreaker.
• Fix: judge every pair in both orders. Count a win only when both runs agree, and treat disagreement as a tie.
• Before trusting a judge, run the mirror test: put the identical answer in both slots. A fair judge has no reason to prefer either one.

What is LLM-as-judge position bias?

LLM-as-judge position bias is a judge model's tendency to pick an answer because of where it appears in the prompt, not because of what it says. It's one of the first biases documented for LLM judges (the MT-Bench / Chatbot Arena paper from Zheng et al. called it out back in 2023), and it's still easy to walk straight into.

It hits pairwise evals hardest, the "here are Answer A and Answer B, which is better?" setup. That's the setup most of us use to compare prompts, models or RAG configs, because it's easier than defining an absolute 1-10 scale.

How I caught it

My eval script looked like every eval script:

for q in questions:
    verdict = ask_judge(q.text, new_answer[q.id], old_answer[q.id])
    wins += verdict == "A"

The new answer always went in slot A. I didn't pick that order for any reason. It was just the order of the function arguments.

A coworker asked "did you swap them?" and I had no good answer. So I ran it again with the old answer in slot A. Here is what the 300 pairs did across both runs:

Outcome across both orders	Pairs
New prompt won both times	120
Old prompt won both times	93
Slot A won both times (verdict flipped)	78
Slot B won both times (verdict flipped)	9

Count the slot picks: slot A won 369 of 600 judgments, or 61.5%. A judge with no position preference would land somewhere near 50%.

The 66% headline was really 120 consistent wins plus 78 pairs where the new prompt just happened to sit in the favored seat.

Why does an LLM judge prefer the first answer?

An LLM judge prefers the first answer mostly because of how it reads: one token at a time, left to right. Answer A gets evaluated with nothing to compare it to. By the time the model reaches Answer B, it has already formed a picture of what "good" looks like for this question, and that picture came from A.

A few things stack on top of that:

• Close calls get decided by the seat. I read the 87 flipped pairs myself. In most of them I couldn't pick a winner either: both answers were correct, worded differently, and about the same length. With no real quality signal to go on, the position prior decides.
• Temperature 0 doesn't help. It makes the bias reproducible, which is worse in a way, because rerunning the same order gives you the same wrong answer and feels like confirmation.
• The direction isn't universal. Some judge and prompt combinations favor slot B, especially with long answers where the most recent text is freshest. My judge favored A. Yours might favor B. Measure it before you assume a direction.
• Format priors. "A" is the first option in countless multiple-choice examples the model has seen. Label tokens carry their own baggage before the model reads a word of the answers.

None of this means the judge is useless. On the 213 pairs where it agreed with itself, its picks matched mine on almost every one I spot-checked. The judge is fine at clear differences. On near-ties it isn't measuring quality at all.

How do you test an LLM judge for position bias?

Run the mirror test: build pairs where Answer A and Answer B are the exact same text, then ask the judge to pick. A fair judge has no basis to prefer either, so it should say TIE or split close to 50/50.

I ran 50 mirror pairs. My judge picked A 34 times, B 9 times and TIE 7 times. It was confidently picking a winner between two copies of the same text, and it usually picked the first one.

This takes ten minutes and tells you more about your judge than a week of reading verdicts. I now run it every time I change the judge model or the judge prompt.

How do you fix position bias in pairwise LLM evals?

Judge every pair twice, once in each order, and count a win only when both runs agree. If the judge contradicts itself, call it a tie. That's the whole fix:

JUDGE_PROMPT = """You are comparing two answers to the same question.

Question:
{question}

Answer A:
{a}

Answer B:
{b}

Briefly list the strengths and weaknesses of each answer.
Then end with exactly one line: VERDICT: A, VERDICT: B, or VERDICT: TIE"""


def ask_judge(question, a, b):
    out = call_llm(JUDGE_PROMPT.format(question=question, a=a, b=b), temperature=0)
    return parse_verdict(out)  # "A" | "B" | "TIE"


def judge_pair(question, new, old):
    run1 = ask_judge(question, new, old)  # new in slot A
    run2 = ask_judge(question, old, new)  # new in slot B

    if run1 == "A" and run2 == "B":
        return "new"
    if run1 == "B" and run2 == "A":
        return "old"
    # Self-contradiction, or a TIE in either run: no signal.
    return "tie"

Then report three numbers instead of one:

results = [judge_pair(q.text, new[q.id], old[q.id]) for q in questions]
n = len(results)

new_wins = results.count("new")
old_wins = results.count("old")
ties = results.count("tie")

print(f"new {new_wins/n:.0%}  old {old_wins/n:.0%}  tie {ties/n:.0%}")
print(f"new win rate on decisive pairs: {new_wins/(new_wins+old_wins):.0%}")

The tie rate is the important number here. It tells you how much of your eval set the judge can't actually separate. If it's 30%, a 5-point swing in your headline doesn't mean much.

Some other things that helped:

• Give the judge a TIE option. Without it, you're forcing a coin flip on every near-equal pair, and the position prior is what decides the flip.
• Ask for reasoning before the verdict. Put the verdict line last so the model writes its comparison before it commits to a letter. It doesn't remove the bias, but it cut my flip rate.
• Use logprobs if your API exposes them. Take P("A") in one order and P("B") in the swapped order, then average. You get a soft score that's symmetric by construction.

Isn't randomizing the A/B order enough?

Randomizing the order per pair is better than a fixed order, but it doesn't fix the problem. Random assignment spreads the position bias evenly across both systems, so your aggregate win rate stops being skewed in one direction. That's real progress.

What it hides is the noise. Those 87 coin-flip pairs are still in your data, now split randomly between "new won" and "old won." Your headline drifts toward 50% and looks like an honest close race, when actually a third of your eval set says nothing at all. Rerun it with a different random seed and the number moves.

Running both orders costs twice the judge calls. On 300 pairs, that's 300 extra API calls. Shipping a prompt that was never better costs more.

Which other LLM judge biases show up alongside position bias?

Position bias rarely travels alone. Watch for these two:

• Verbosity bias. Judges tend to favor the longer answer, even when the extra length is padding. If your new prompt makes answers 40% longer, some of its "wins" are word count. Check the win rate split by length difference.
• Self-preference. A judge can favor outputs that sound like its own writing. If you're comparing Model X against Model Y and Model X is also the judge, be suspicious of a big X win.

Swapping the order fixes neither of these. They need their own checks.

So was my new prompt actually better?

Yes, a little. On the 213 pairs where the judge agreed with itself, the new prompt won 120 to 93, which is 56%. That's a real but modest improvement. It isn't the 66% blowout I nearly announced. I shipped it with a much quieter message and a note that 29% of the eval set was a toss-up.

How does LLM-as-judge position bias distort pairwise evals?

LLM-as-judge position bias makes a pairwise judge favor whichever answer occupies a particular slot, usually the first. It mostly decides near-tie comparisons, which can quietly inflate or deflate a win rate by double digits. In my eval, swapping A and B flipped 87 of 300 verdicts and turned a 66% win into 43%. The fix is to judge every pair in both orders, count only verdicts that agree, treat contradictions as ties, and report the tie rate next to the win rate. Run the mirror test, with the identical answer in both slots, before you trust any judge.

---

Written by the developer behind Preterview, an interview prep platform.