A Higher Pass Rate Can Mean a Worse Model. The Math Is Simpson's Paradox.

We shipped a model update last quarter that moved our aggregate pass rate from 81.2% to 83.6%. Everyone was happy. I was not, because I had seen this shape before. When I split the eval set by the four traffic slices we actually serve, the new model was worse on three of them. The aggregate went up anyway. This is not a paradox in the mystical sense. It is arithmetic, and it has a name.

What Simpson's paradox actually is

Simpson's paradox is when a trend that holds in every subgroup reverses once you pool the subgroups. The reversal comes from the subgroup sizes, not the subgroup rates. If your eval set has a different slice mix than it had last time, or a different mix than production, the pooled average is a weighted sum where the weights are doing the talking.

Here is the case that ruined my afternoon, with numbers close to the real ones.

import pandas as pd

data = {
    "slice":       ["short_factual", "multi_turn", "tool_use", "long_context"],
    "v1_pass":     [855, 560, 408, 10],
    "v1_total":    [900, 700, 600, 50],
    "v2_pass":     [792, 532, 390, 950],
    "v2_total":    [900, 700, 600, 1000],
}
df = pd.DataFrame(data)
for v in ["v1", "v2"]:
    df[f"{v}_rate"] = df[f"{v}_pass"] / df[f"{v}_total"]
print(df[["slice", "v1_rate", "v2_rate"]].round(3))

agg_v1 = df["v1_pass"].sum() / df["v1_total"].sum()
agg_v2 = df["v2_pass"].sum() / df["v2_total"].sum()
print(f"aggregate v1: {agg_v1:.3f}, aggregate v2: {agg_v2:.3f}")

Output:

            slice  v1_rate  v2_rate
0   short_factual    0.950    0.880
1      multi_turn    0.800    0.760
2        tool_use    0.680    0.650
3    long_context    0.200    0.950
aggregate v1: 0.815, aggregate v2: 0.833

v2 lost on three of four slices: short_factual (0.950 to 0.880), multi_turn (0.800 to 0.760), tool_use (0.680 to 0.650). It won only on long_context, and it won catastrophically large there (0.200 to 0.950, almost certainly because someone fixed a truncation bug in that eval fixture, not the model itself). The aggregate moved 1.8 points in v2's favor anyway. How? The v2 eval set has 1000 long_context rows against v1's 50. Twenty times the representation for the one slice where v2 wins. The aggregate is not measuring "is the model better". It is measuring "is the model better, weighted by whatever slice mix happened to be in this run".

Why the average is the wrong unit

The pooled pass rate answers a question almost nobody asks: what is the probability a uniformly random row from this particular eval set passes? Your users are not uniformly random rows from your eval set. They arrive in a fixed mix, and that mix is a property of your product, not your test harness. The fix is to stop comparing pooled numbers and compare slice by slice, then recombine using a single fixed weighting that you control. Standardize both models to the same slice weights (production traffic share is the honest choice).

w = {"short_factual": 0.45, "multi_turn": 0.30, "tool_use": 0.20, "long_context": 0.05}
df["w"] = df["slice"].map(w)
std_v1 = (df["v1_rate"] * df["w"]).sum()
std_v2 = (df["v2_rate"] * df["w"]).sum()
print(f"production-weighted v1: {std_v1:.3f}, v2: {std_v2:.3f}")

Output:

production-weighted v1: 0.814, v2: 0.802

Under the weighting that matches who we actually serve, v2 is 1.2 points worse. The naive aggregate said plus 1.8. The gap between those two numbers is 3.0 points, and it is invisible if you only log one scalar per run. This is the machinery epidemiologists call direct standardization, and the warning is old: Blyth's 1972 JASA paper "On Simpson's Paradox and the Sure-Thing Principle" laid out exactly how a treatment can look better marginally while being worse in every stratum. Nothing about LLMs changed the math.

What I do now

One number per eval run is a liability. The slice vector is the artifact. I store per-slice pass counts and totals every run, I fix a weighting derived from production and apply it to every model, and I never let two runs with different slice mixes be compared by their pooled rate. When the pooled number and the standardized number disagree, the standardized one is the quality signal and the pooled one is a sampling artifact.

FAQ

Is this just a sample-size problem I can fix with more eval data? No. More data shrinks variance, not bias. Simpson's reversal is a weighting effect, so a bigger eval set with the same skewed slice mix gives you a tighter estimate of the wrong number. Stratify, then standardize.

My slices have wildly different sizes. Won't the tiny ones be noisy under production weights? Yes, and that is information, not an objection. If long_context is 5% of traffic but swings your decision, you have too few long_context examples to decide. Put a confidence interval on each slice rate and widen your eval set where the weight times the uncertainty is largest.

Can I just always weight by production share and forget the pooled number? Production-weighted standardization is the right default for "which model serves users better", but keep the per-slice rates visible. A model that improves the average while regressing your highest-stakes slice (say, tool_use that triggers real actions) can still be the wrong ship.

Open question

Production-weighting assumes the slice definitions are fixed and the traffic mix is roughly stationary. Neither holds cleanly. Slices are a human carving of a continuous input space, and a model change can shift the traffic distribution itself (better long-context handling pulls in more long-context users). When the act of shipping moves the weights you standardized on, what is the honest denominator? I do not have a clean answer for evaluating under distribution shift that you yourself caused, and I have not seen one that survives contact with a fast-moving product.