If your pairwise judge sees answer A before answer B, it tends to prefer A. If you never swap the order, every win-rate you report is contaminated by which slot you happened to put each answer in.
The first time I actually measured this, I did not believe the number. I had a judge scoring 400 pairwise comparisons, model against baseline, and it reported a 61% win-rate for our new model. Then I ran the exact same 400 pairs with the two answers swapped in the prompt. The win-rate came back at 46%. Same answers, same judge, same rubric. The only thing I changed was which one I showed first, and the verdict moved by 15 points.
That is position bias, and it is not a quirk of my setup. Zheng et al. document it directly in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023). They call it position bias and show that strong judges, GPT-4 included, systematically favor one slot regardless of content, sometimes flipping their own verdict when you swap the responses. If you are running a pairwise eval and you present answers in a fixed order, you are baking that preference into your headline metric and calling it quality.
The thing you are actually measuring
A pairwise judge call takes a question, answer A, answer B, and returns a preference. The implicit claim is that the preference is a function of the two answers. It is not. It is a function of the two answers AND the order you presented them AND whatever slot the judge happens to lean toward. When those last two matter, your win-rate is measuring a mix of quality and seating arrangement, and you cannot tell how much of your 61% is real.
The failure is invisible if you only run each pair once. You get a clean-looking number, you ship, and the number was partly an artifact of prompt layout. The fix is not exotic. You run each pair in both orders and look at what the judge does with the swap.
Criterion one: run both orders and measure the disagreement rate
For every pair, call the judge twice. Once with the real answer as A and the candidate as B, once swapped. A judge with no position bias gives you consistent verdicts: if it preferred the candidate in one order, it prefers the candidate in the other. When the two orders disagree, that pair is telling you the order decided the call, not the content.
Two numbers fall out of this. The flip rate is the fraction of pairs where swapping the order changed the verdict. The position-preference rate is how often the judge picked the first slot regardless of what sat there. Here is the measurement, with a deterministic stub in place of a real judge so it runs as-is:
import random
random.seed(7)
def judge(question, answer_a, answer_b):
"""Return 'A' or 'B'. Replace this body with a real judge call.
This stub fakes a first-slot bias so the metrics below are non-trivial."""
# 30% of the time the judge just grabs the first slot (position bias);
# otherwise it 'reads' and prefers the answer tagged as stronger.
if random.random() < 0.30:
return "A"
return "A" if answer_a["quality"] >= answer_b["quality"] else "B"
def pairwise_both_orders(question, baseline, candidate):
"""Run one comparison in both orders. Return the two verdicts,
normalized to whether the CANDIDATE won."""
v1 = judge(question, baseline, candidate) # baseline in slot A
cand_won_order1 = (v1 == "B")
v2 = judge(question, candidate, baseline) # candidate in slot A
cand_won_order2 = (v2 == "A")
return cand_won_order1, cand_won_order2
pairs = [
{"q": f"q{i}",
"baseline": {"quality": random.random()},
"candidate": {"quality": random.random()}}
for i in range(400)
]
flips = 0
first_slot_picks = 0
total_calls = 0
candidate_wins_consistent = 0
consistent_pairs = 0
for p in pairs:
w1, w2 = pairwise_both_orders(p["q"], p["baseline"], p["candidate"])
# flip = the two orders disagree on who won
if w1 != w2:
flips += 1
else:
consistent_pairs += 1
if w1:
candidate_wins_consistent += 1
# count how often slot A won, across both calls, as a position signal
# order1: A holds baseline, so baseline winning == A won
first_slot_picks += (1 if not w1 else 0) + (1 if w2 else 0)
total_calls += 2
n = len(pairs)
print(f"flip rate: {flips / n:.3f}")
print(f"first-slot pick rate: {first_slot_picks / total_calls:.3f} (0.50 = unbiased)")
print(f"win-rate on consistent pairs only: "
f"{candidate_wins_consistent / consistent_pairs:.3f} "
f"(n={consistent_pairs})")
The first-slot pick rate is your diagnostic. If it sits near 0.50, the judge is not favoring a position and you can mostly relax. If it drifts to 0.60 or above, the judge is grabbing a slot, and any single-order win-rate you computed is off by roughly the amount that bias leaks into your particular matchup.
Criterion two: correct, do not just report
Once you have both orders, you have three honest options, in order of how much data they cost you.
Average over both orders. Count the candidate as winning a pair only if it wins in expectation across the two runs. A pair that splits (wins one order, loses the other) contributes half a win. This is the cheapest correction and it neutralizes a symmetric position preference, because the bias helps the candidate in one order and hurts it in the other, and the two roughly cancel.
Drop the pairs where the order flips the call. These are pairs the judge could not decide on the merits. Reporting a win-rate on the consistent pairs only, with the count shown, is more honest than folding coin-flips into the average. You lose sample size, so show the n.
Use the flip rate as a judge-quality gate. If more than, say, 15% of your pairs flip on swap, the problem is not your model, it is your judge. A judge that reverses itself on one in six comparisons is not a measuring instrument yet. Fix the rubric or the judge before you trust any win-rate it produces.
def scored_winrate(pairs, judge_pair):
"""Average-over-both-orders win-rate, plus the flip rate as a health check."""
credit = 0.0
flips = 0
for p in pairs:
w1, w2 = judge_pair(p)
credit += (w1 + w2) / 2.0 # half-credit on splits
flips += (w1 != w2)
return credit / len(pairs), flips / len(pairs)
Criterion three: know when a swap is not enough
Two orders catch left-versus-right bias. They do not catch a judge that always prefers the longer answer, or the one with more markdown, or the more confident tone. Those biases survive the swap because they travel with the content, not the slot. Position correction buys you one specific fix. It does not certify the judge overall. Zheng et al. are explicit that position bias is one of several, alongside verbosity and self-enhancement bias, so treat the swap as necessary, not sufficient.
FAQ
Do I have to double every judge call?
That doubles my eval cost. For the run you make a ship decision on, yes, run both orders. It is the only way to separate quality from seating. For continuous monitoring you can subsample: run both orders on a random 10-20% of pairs to track the flip rate over time, and only go full-swap when you are about to make a call. The cost is real but bounded, and it is cheaper than shipping on a contaminated number.
My judge barely flips (flip rate under 3%). Can I stop swapping?
If you have measured a low flip rate on a representative sample and the first-slot pick rate is near 0.50, single-order evals are defensible for that judge and that task. Re-check when you change the judge model, the rubric, or the answer format, because position bias is not a fixed property of the judge, it interacts with the prompt.
Does randomizing the order instead of swapping fix it?
Randomizing removes the systematic tilt from your aggregate win-rate, which is better than a fixed order. It does not tell you the flip rate, so you lose the diagnostic. Swapping every pair gives you both the correction and the measurement, which is why I prefer it for decision runs.
Open question
Averaging over both orders neutralizes a symmetric position bias. But nothing guarantees the bias is symmetric. A judge might favor slot A by 12 points and slot B by 4, in which case the two orders do not cancel and my averaged win-rate is still tilted. I can detect the asymmetry (the first-slot pick rate is not 0.50 even after swapping), but I do not have a clean estimator that corrects for an asymmetric position bias without assuming a model of how the bias combines with quality. Modeling it as an additive slot effect in a Bradley-Terry-style fit is the obvious next move, and I have not seen it done convincingly on real judge data. What does a principled asymmetric-position correction look like in practice?
Top comments (3)
I did not break flips down by rubric dimension, so I cannot give you the split you asked for with numbers, and I will not pretend otherwise. Qualitatively, ours matched your data: flips concentrated where both candidates were adequate, and the clear wins survived order swaps. Which is the reassuring reading, with one caveat: "the flips live in the ties" is only comforting if your downstream decision treats ties as ties. A win-rate that counts each flipped pair as a win for whoever drew the lucky order still moves with the ordering, even when every flip is a genuine tie. Both-orders-or-it-does-not-count, with flips scored as ties, is the cheapest fix I know. Your verbosity and self-preference compounders are real and order-swapping does nothing for either, agreed.
The 61% → 46% on a pure order swap is the kind of number that should terrify anyone shipping a win-rate. We hit the same wall running judge panels to verify multi-agent output, and the fix that stuck was exactly yours: every pair judged in both orders, and any pair that flips gets counted as a tie rather than pretending one order is truth. Your flip-rate and position-preference-rate split is a cleaner way to report it than what we were doing.
Two things compounded it for us. First, verbosity bias rode along with position bias — the longer answer won more often, and it wasn't always the better one. Second, self-preference: when the judge was the same model family as one of the generators, it quietly favored its own outputs, which no amount of order-swapping fixes.
Have you measured whether the flip rate is uniform across your rubric, or concentrated on the genuinely close pairs? In our data almost all the flips lived in the ties — oddly reassuring, since it meant the clear wins were real.
Interesting observation—position bias is a subtle but real issue in ranking models, especially when human preferences are learned through pairwise comparisons. In some of my work with model evaluation on VoltageGPU, we noticed similar artifacts when the input order wasn't randomized, leading to inflated confidence in early responses. It's a good reminder to always audit the training data distribution for such biases.