Your judge says the new system prompt wins 68% of head-to-head comparisons. You swap which response is labeled A and which is B, rerun the identical harness at temperature 0, and the win rate drops to 41%. Same judge model, same rubric, same 200 pairs. The only thing that changed is presentation order.
That gap is LLM-as-judge position bias, and it is not a prompt-writing problem you can polish away. It is a structural property of asking an autoregressive model to compare two things laid out sequentially in one context. Worse, its severity scales inversely with the effect size you are trying to measure — which means it bites hardest exactly when you are doing the thing you built the eval for: distinguishing prompt v1 from prompt v2.
TL;DR
- LLM-as-judge position bias is a near-constant logit offset toward one slot (usually the first). Content signal competes with it; when the quality gap is small, the offset wins.
- Measure it in five minutes with a self-pair probe: feed the same response as both A and B. Every non-tie verdict is pure bias, no ground truth needed.
- Averaging both orders does not remove bias — it converts it into ties. The useful number is the order-consistency rate, not the averaged win rate.
- Count order-disagreements as ties. Dropping them inflates win rates and silently shrinks your sample.
- 1–10 Likert scoring makes it worse: judges cluster on 7/8/9, so score variance collapses and position noise dominates the ranking.
Why does swapping A and B flip the verdict?
Because the judge reads A before B, and its own critique of B is conditioned on its critique of A — never the reverse. The asymmetry is baked into causal attention, not into your wording.
Three mechanisms stack:
1. Conditioned reasoning. In a chain-of-thought judge template, the model discusses the first response first. Those tokens are now in context. When it evaluates the second response, it is doing so relative to an already-articulated frame ("A is thorough but verbose"). The second response gets graded as a delta against the first. The first response never gets that treatment.
2. A verdict-token prior. The final decision usually collapses to one token — A, B, 1, 2. That token has a pretraining prior. Across most templates, mass sits higher on the first-listed option. Treat it as a fixed bias term b added to the content-derived logit difference:
P(verdict = A) = σ( s(A) − s(B) + b )
where s(·) is the judge's latent quality estimate. If b is worth roughly the same as a small real quality gap, then any comparison where |s(A) − s(B)| ≲ |b| is decided by slot, not content.
3. Recency in long responses. For long outputs (2k+ tokens each), the bias can invert. The second response is closer to the verdict token, and attention over a long context flattens; the earlier response is compressed harder. I have seen the same judge model favor position 1 on short answers and position 2 on long ones. Never assume the direction — measure it per task.
The consequence worth internalizing: position bias severity is a function of your effect size. A judge comparing a competent model against a deliberately broken one will look nearly bias-free. The same judge comparing two variants of your production prompt will be close to a coin flip plus an offset. Vendor claims of "95% agreement with humans" are usually measured on the first regime and quoted into the second.
How do you measure LLM-as-judge position bias without human labels?
Feed the judge the same response twice, as both A and B. Any verdict other than "tie" is bias by construction. This self-pair probe needs no annotators, no gold labels, and no held-out set.
import anthropic, json, random
from collections import Counter
client = anthropic.Anthropic()
JUDGE = "claude-opus-4-5" # use one tier above the models under test
VERDICT_TOOL = {
"name": "record_verdict",
"description": "Record the comparison verdict.",
"input_schema": {
"type": "object",
"properties": {
"analysis": {"type": "string", "description": "Per-criterion comparison, both responses."},
"verdict": {"type": "string", "enum": ["A", "B", "tie"]},
},
"required": ["analysis", "verdict"], # analysis BEFORE verdict: reason, then decide
},
}
TEMPLATE = """Grade two candidate answers against the rubric.
<question>{q}</question>
<rubric>{rubric}</rubric>
<response_a>{a}</response_a>
<response_b>{b}</response_b>
Compare on each rubric criterion, addressing both responses under each
criterion (do not review A fully, then B fully). Prefer "tie" when the
difference is not decision-relevant. Ignore length and formatting unless
the rubric scores them."""
def judge(q, rubric, a, b):
r = client.messages.create(
model=JUDGE, max_tokens=1200, temperature=0,
tools=[VERDICT_TOOL], tool_choice={"type": "tool", "name": "record_verdict"},
messages=[{"role": "user", "content": TEMPLATE.format(q=q, rubric=rubric, a=a, b=b)}],
)
return next(c.input["verdict"] for c in r.content if c.type == "tool_use")
def self_pair_bias(items, rubric):
"""Same text in both slots. Any non-tie is pure position bias."""
c = Counter(judge(it["q"], rubric, it["resp"], it["resp"]) for it in items)
n = sum(c.values())
return {"tie_rate": c["tie"] / n, "slot1_pull": c["A"] / n, "slot2_pull": c["B"] / n}
Run that over 50–100 items before you trust a single win rate. A judge with a tie rate below ~0.9 on identical inputs cannot resolve small differences, full stop. If slot1_pull is 0.22 and slot2_pull is 0.03, you have a strong first-slot prior and every unswapped eval you have ever run overstated whatever you put in slot A.
Two things this probe reveals that a human-agreement number will not:
- Tie suppression. Many judges almost never emit "tie" because the template implies a winner must exist. Forcing a decision on identical inputs is how you discover it.
- Rubric leakage. If the tie rate is high on factual tasks and low on "helpfulness" tasks, your rubric's subjective criteria are where the noise lives. Split them out.
Does running both orders and averaging fix it?
No. Dual-order evaluation is necessary but it does not cancel the bias — it relocates it into the disagreement bucket. What you get is a cleaner accounting of how much of your result was ever real.
Run each pair twice, swapped, then classify:
def dual_order(item, rubric):
v1 = judge(item["q"], rubric, item["a"], item["b"]) # A=a, B=b
v2 = judge(item["q"], rubric, item["b"], item["a"]) # swapped
unswap = {"A": "b", "B": "a", "tie": "tie"}[v2] # map back to identity
first = {"A": "a", "B": "b", "tie": "tie"}[v1]
if first == unswap:
return first # order-consistent: a, b, or tie
return "tie" # order-disagreement -> tie, NOT discarded
The line that matters is the last one. The common bug is dropping disagreements and computing the win rate over the survivors. That does two bad things at once: it inflates the margin (you deleted exactly the ambiguous cases) and it hides that your effective sample is much smaller than your nominal one. Count them as ties.
Then report the order-consistency rate — the fraction of pairs where both orders agreed — alongside the win rate. It is your judge's reliability ceiling on this task. If consistency is 0.6, then 40% of your pairs carried no signal, and a "58% win rate" over 200 pairs is roughly 120 informative comparisons. Put a Wilson interval on it and most such results stop being significant:
def wilson(k, n, z=1.96):
if n == 0: return (0.0, 1.0)
p, d = k / n, 1 + z * z / n
c = p + z * z / (2 * n)
m = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5)
return ((c - m) / d, (c + m) / d)
Report wilson(wins_a, wins_a + wins_b). A win rate whose interval straddles 0.5 is not a result, no matter how many pairs you ran.
Why does 1–10 Likert scoring make position bias worse?
Because pointwise Likert judges compress everything into 7, 8, and 9. Score variance collapses to well under a point, so a positional or formatting nudge of a few tenths reorders your leaderboard.
Pointwise scoring does avoid position bias — there is no A/B slot to bias. It trades it for calibration drift: the same rubric yields different absolute scores across days, model versions, and even prompt cache states. Two systems both scoring "8.2" tells you nothing about which one your users prefer.
The practical middle ground:
- Binary per-criterion rubrics. Instead of "rate helpfulness 1–10," ask five yes/no questions ("Does it answer the literal question asked?", "Does every factual claim appear in the provided context?"). Binary items are far more reproducible, and summing them gives you a discrete score with real variance.
- Pairwise with explicit ties for preference-shaped questions, always dual-order.
- Bradley-Terry aggregation when comparing more than two systems. Fit latent strengths from all pairwise outcomes rather than computing per-pair win rates; it handles intransitivity and gives you comparable strengths without an all-pairs matrix at full sample size.
What else contaminates the judge alongside position?
Length bias. Judges reward longer, more structured answers even when the rubric says not to. Diagnostic: regress verdicts on response-length delta. If length explains a meaningful share of the verdicts, either truncate both responses to a comparable budget or add a length-matched control arm.
Self-preference. A judge tends to favor text that matches its own generation distribution. Never judge Claude outputs with the same Claude snapshot that produced them, and never judge GPT-5.x outputs with a GPT-5.x judge, when the arms come from different families. Use a third family as judge, or run both and check the verdicts agree.
Reasoning order. Templates that emit the verdict first and the justification after are cheaper and measurably more biased — the justification becomes post-hoc rationalization. Keep analysis before verdict in your tool schema, since the model fills the fields in order.
Logprob shortcuts. With providers that expose token logprobs (GPT-5.x), you can read P(A) vs P(B) on the verdict token and average probabilities across orders instead of hard labels — a smoother estimator. The Anthropic API does not expose logprobs, so on Claude judges use dual-order tie-counting instead of trying to recover a soft score.
Direct answer: why does swapping A and B flip your judge's wins?
Swapping A and B flips verdicts because an LLM judge adds a roughly constant positional offset to its content-based preference: the first response is reasoned about first and anchors the critique of the second, and the single-token verdict carries a pretraining prior toward one slot. When the two candidates genuinely differ a lot, content overwhelms that offset and the judge looks reliable. When they differ slightly — the regime of every real prompt-iteration eval — the offset decides the outcome, and reversing order reverses the winner. The fix is not a better rubric. It is measurement: run a self-pair probe to quantify the offset, run every comparison in both orders, count order-disagreements as ties rather than deleting them, report the order-consistency rate next to the win rate, and put a confidence interval on the result. Do that and most of the 5-point wins in your eval dashboard will turn out to be noise you were reading as signal.
Top comments (0)