Your Eval Suite Has a Flake Rate: Gating LLM Changes on Paired, Powered Tests

I have watched the same release go wrong three times. Someone tunes a prompt or swaps a model, the eval average moves from 0.81 to 0.83, the PR gets approved, and two weeks later a specific intent is failing in production. The regression was real. The improvement was not. Nobody did anything careless — the eval reported a number that looked like a measurement and was actually a coin flip.

The unglamorous part of LLMOps is not prompt management or tracing. It is that most eval suites cannot detect the changes teams ship on. Fixing that is a half-day of statistics and one CI job.

Three kinds of noise, only one of which is your change

Before you compare two configurations, know what is moving:

• Example sampling noise. Your golden set has 200 examples. Each one is worth 0.5 points. A "3-point gain" can be six examples flipping, several of which are borderline judgments.
• Judge noise. If an LLM grades the outputs and you left temperature above 0, the same output can pass on Tuesday and fail on Wednesday. Order matters too: ask a judge "which is better, A or B" and then "which is better, B or A" and you will find disagreement.
• Task noise. Sampling temperature, tool timeouts, and retrieval index drift all change outputs without any code change on your side.

You cannot remove all three. You can remove the first one almost entirely by comparing paired results: run baseline and candidate on the same examples and analyze the per-example difference. Example difficulty cancels out. A prompt that fixes one hard case and breaks another now shows up as one +1 and one -1, not as an aggregate you have to reason about.

Do the arithmetic before you build the gate

For a paired comparison of pass/fail outcomes, the standard error of the mean difference depends on the disagreement rate — how often the two configs disagree on the same example — not on the accuracy.

If 15% of examples flip between your two configurations and you have 200 paired examples:

• SE = sqrt(0.15 × 0.85 / 200) ≈ 2.5 percentage points
• Minimum detectable effect at 80% power, α=0.05 ≈ 2.8 × SE ≈ 7 points

So on that setup, a 2-point bump is invisible. To see a 5-point change under the same assumptions you need roughly 400 paired examples; to see 3 points, about 1,100. These are arithmetic, not benchmarks — plug in your own disagreement rate, which is usually higher than you expect for prompt changes.

Two consequences follow. First, stop gating on raw averages. Second, if your golden set is 200 examples and you cannot grow it, your gate should be "no regression beyond X" rather than "improvement required."

The gate itself

The whole thing is about thirty lines of stdlib Python. Bootstrap the paired differences, look at the interval, and ship only when the interval clears zero.

python
"""paired_gate.py — decide whether an LLM change is real or noise.
Run: python paired_gate.py    Deps: none (stdlib only)."""
import random

def paired_bootstrap(baseline, candidate, iters=5000, seed=7):
    """Scores aligned on the SAME example ids, 0/1 or 0..1.
    Returns (mean delta, 2.5th pct, 97.5th pct)."""
    diffs = [c - b for b, c in zip(baseline, candidate)]
    n = len(diffs)
    rng = random.Random(seed)  # fixed seed: the gate must be reproducible
    means = sorted(
        sum(diffs[rng.randrange(n)] for _ in range(n)) / n
        for _ in range(iters)
    )
    return sum(diffs) / n, means[int(0.025 * iters)], means[int(0.975 * iters)]

def required_n(disagreement_rate, mde, z_alpha=1.96, z_power=0.84):
    """Paired binary comparison: examples needed per side."""
    se = mde / (z_alpha + z_power)
    p = disagreement_rate
    return int((p * (1 - p)) / se**2) + 1

if __name__ == "__main__":
    # 200 paired examples; candidate looks "+3 points" better.
    baseline  = [1] * 160 + [0] * 40
    candidate = [1] * 166 + [0] * 34
    delta, lo, hi = paired_bootstrap(baseline, candidate)
    print(f"delta