If your pairwise judge sees answer A before answer B, it tends to prefer A. If you never swap the order, every win-rate you report is contaminated by which slot you happened to put each answer in.
The first time I actually measured this, I did not believe the number. I had a judge scoring 400 pairwise comparisons, model against baseline, and it reported a 61% win-rate for our new model. Then I ran the exact same 400 pairs with the two answers swapped in the prompt. The win-rate came back at 46%. Same answers, same judge, same rubric. The only thing I changed was which one I showed first, and the verdict moved by 15 points.
That is position bias, and it is not a quirk of my setup. Zheng et al. document it directly in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023). They call it position bias and show that strong judges, GPT-4 included, systematically favor one slot regardless of content, sometimes flipping their own verdict when you swap the responses. If you are running a pairwise eval and you present answers in a fixed order, you are baking that preference into your headline metric and calling it quality.
The thing you are actually measuring
A pairwise judge call takes a question, answer A, answer B, and returns a preference. The implicit claim is that the preference is a function of the two answers. It is not. It is a function of the two answers AND the order you presented them AND whatever slot the judge happens to lean toward. When those last two matter, your win-rate is measuring a mix of quality and seating arrangement, and you cannot tell how much of your 61% is real.
The failure is invisible if you only run each pair once. You get a clean-looking number, you ship, and the number was partly an artifact of prompt layout. The fix is not exotic. You run each pair in both orders and look at what the judge does with the swap.
Criterion one: run both orders and measure the disagreement rate
For every pair, call the judge twice. Once with the real answer as A and the candidate as B, once swapped. A judge with no position bias gives you consistent verdicts: if it preferred the candidate in one order, it prefers the candidate in the other. When the two orders disagree, that pair is telling you the order decided the call, not the content.
Two numbers fall out of this. The flip rate is the fraction of pairs where swapping the order changed the verdict. The position-preference rate is how often the judge picked the first slot regardless of what sat there. Here is the measurement, with a deterministic stub in place of a real judge so it runs as-is:
import random
random.seed(7)
def judge(question, answer_a, answer_b):
"""Return 'A' or 'B'. Replace this body with a real judge call.
This stub fakes a first-slot bias so the metrics below are non-trivial."""
# 30% of the time the judge just grabs the first slot (position bias);
# otherwise it 'reads' and prefers the answer tagged as stronger.
if random.random() < 0.30:
return "A"
return "A" if answer_a["quality"] >= answer_b["quality"] else "B"
def pairwise_both_orders(question, baseline, candidate):
"""Run one comparison in both orders. Return the two verdicts,
normalized to whether the CANDIDATE won."""
v1 = judge(question, baseline, candidate) # baseline in slot A
cand_won_order1 = (v1 == "B")
v2 = judge(question, candidate, baseline) # candidate in slot A
cand_won_order2 = (v2 == "A")
return cand_won_order1, cand_won_order2
pairs = [
{"q": f"q{i}",
"baseline": {"quality": random.random()},
"candidate": {"quality": random.random()}}
for i in range(400)
]
flips = 0
first_slot_picks = 0
total_calls = 0
candidate_wins_consistent = 0
consistent_pairs = 0
for p in pairs:
w1, w2 = pairwise_both_orders(p["q"], p["baseline"], p["candidate"])
# flip = the two orders disagree on who won
if w1 != w2:
flips += 1
else:
consistent_pairs += 1
if w1:
candidate_wins_consistent += 1
# count how often slot A won, across both calls, as a position signal
# order1: A holds baseline, so baseline winning == A won
first_slot_picks += (1 if not w1 else 0) + (1 if w2 else 0)
total_calls += 2
n = len(pairs)
print(f"flip rate: {flips / n:.3f}")
print(f"first-slot pick rate: {first_slot_picks / total_calls:.3f} (0.50 = unbiased)")
print(f"win-rate on consistent pairs only: "
f"{candidate_wins_consistent / consistent_pairs:.3f} "
f"(n={consistent_pairs})")
The first-slot pick rate is your diagnostic. If it sits near 0.50, the judge is not favoring a position and you can mostly relax. If it drifts to 0.60 or above, the judge is grabbing a slot, and any single-order win-rate you computed is off by roughly the amount that bias leaks into your particular matchup.
Criterion two: correct, do not just report
Once you have both orders, you have three honest options, in order of how much data they cost you.
Average over both orders. Count the candidate as winning a pair only if it wins in expectation across the two runs. A pair that splits (wins one order, loses the other) contributes half a win. This is the cheapest correction and it neutralizes a symmetric position preference, because the bias helps the candidate in one order and hurts it in the other, and the two roughly cancel.
Drop the pairs where the order flips the call. These are pairs the judge could not decide on the merits. Reporting a win-rate on the consistent pairs only, with the count shown, is more honest than folding coin-flips into the average. You lose sample size, so show the n.
Use the flip rate as a judge-quality gate. If more than, say, 15% of your pairs flip on swap, the problem is not your model, it is your judge. A judge that reverses itself on one in six comparisons is not a measuring instrument yet. Fix the rubric or the judge before you trust any win-rate it produces.
def scored_winrate(pairs, judge_pair):
"""Average-over-both-orders win-rate, plus the flip rate as a health check."""
credit = 0.0
flips = 0
for p in pairs:
w1, w2 = judge_pair(p)
credit += (w1 + w2) / 2.0 # half-credit on splits
flips += (w1 != w2)
return credit / len(pairs), flips / len(pairs)
Criterion three: know when a swap is not enough
Two orders catch left-versus-right bias. They do not catch a judge that always prefers the longer answer, or the one with more markdown, or the more confident tone. Those biases survive the swap because they travel with the content, not the slot. Position correction buys you one specific fix. It does not certify the judge overall. Zheng et al. are explicit that position bias is one of several, alongside verbosity and self-enhancement bias, so treat the swap as necessary, not sufficient.
FAQ
Do I have to double every judge call?
That doubles my eval cost. For the run you make a ship decision on, yes, run both orders. It is the only way to separate quality from seating. For continuous monitoring you can subsample: run both orders on a random 10-20% of pairs to track the flip rate over time, and only go full-swap when you are about to make a call. The cost is real but bounded, and it is cheaper than shipping on a contaminated number.
My judge barely flips (flip rate under 3%). Can I stop swapping?
If you have measured a low flip rate on a representative sample and the first-slot pick rate is near 0.50, single-order evals are defensible for that judge and that task. Re-check when you change the judge model, the rubric, or the answer format, because position bias is not a fixed property of the judge, it interacts with the prompt.
Does randomizing the order instead of swapping fix it?
Randomizing removes the systematic tilt from your aggregate win-rate, which is better than a fixed order. It does not tell you the flip rate, so you lose the diagnostic. Swapping every pair gives you both the correction and the measurement, which is why I prefer it for decision runs.
Open question
Averaging over both orders neutralizes a symmetric position bias. But nothing guarantees the bias is symmetric. A judge might favor slot A by 12 points and slot B by 4, in which case the two orders do not cancel and my averaged win-rate is still tilted. I can detect the asymmetry (the first-slot pick rate is not 0.50 even after swapping), but I do not have a clean estimator that corrects for an asymmetric position bias without assuming a model of how the bias combines with quality. Modeling it as an additive slot effect in a Bradley-Terry-style fit is the obvious next move, and I have not seen it done convincingly on real judge data. What does a principled asymmetric-position correction look like in practice?
Top comments (0)