My eval leaderboard published a negative retry count

I was looking at the EvalBench dashboard a few weeks ago and one cell stopped me:

Retries to valid
0.125
95% CI -0.035 – 0.285
n=40

A negative number of retries. You cannot retry a request negative-point-oh-three-five times. The mean was fine. The interval was nonsense, and it was nonsense in a way that pointed straight at how I was computing every number on the page.

EvalBench is a multi-provider LLM eval platform I built. Three suites run against OpenAI and Anthropic models across five domains: structured-output reliability, latency and cost with judge scoring, and RAG retrieval quality. Every suite emits the same MetricRecord shape, and the runner aggregates those records into leaderboard rows.

The interesting part was not the arithmetic. It was that a frontend formatting string had been quietly making a statistical decision for months.

Why I stopped publishing bare means

Here is the structured-output suite, 40 samples per model:

Model	Schema valid	First-attempt valid
anthropic/claude-sonnet-4-5	85%	80%
openai/gpt-4o	72.5%	72.5%

A 12.5 point gap. If I publish just those two numbers, every reader concludes Sonnet is meaningfully better at schema adherence.

Now the same table with intervals:

Model	Schema valid	95% CI
anthropic/claude-sonnet-4-5	85%	70.9% – 92.9%
openai/gpt-4o	72.5%	57.2% – 83.9%

The intervals overlap across roughly eleven points. At n=40 I cannot tell these models apart on this metric. The honest reading is "no detectable difference," not "Sonnet wins by 12.5."

That is the whole reason the interval math exists. A mean with no sample size and no interval is a claim disguised as a measurement.

Three metric types, three different intervals

The mistake I see in eval tooling is applying one interval formula to every column. Proportions, unbounded counts, and tail latencies have different distributions, and a single formula gets at least two of them wrong.

Proportions get Wilson, not normal approximation

Schema validity is a proportion. The textbook normal approximation, p ± 1.96 * sqrt(p(1-p)/n), breaks badly near 0 and 1. At p=1.0 it produces a zero-width interval, which claims perfect certainty from 40 samples.

Wilson does not do that:

def wilson_interval(values: Sequence[float]) -> Estimate:
    """Return the 95% Wilson interval for binary or fractional successes."""
    n = len(values)
    if n == 0:
        return _empty_estimate()

    z = 1.96
    mean = sum(values) / n
    denominator = 1 + z**2 / n
    center = (mean + z**2 / (2 * n)) / denominator
    half = (
        z
        * math.sqrt((mean * (1 - mean) + z**2 / (4 * n)) / n)
        / denominator
    )
    return Estimate(
        mean=mean,
        n=n,
        ci_low=max(0.0, center - half),
        ci_high=min(1.0, center + half),
    )

Four perfect scores gives you [0.510, 1.0], not [1.0, 1.0]. Four failures gives [0.0, 0.490]. The interval stays inside [0,1] and stays wide when n is small.

I pinned those exact values in a test rather than asserting loose properties:

@pytest.mark.parametrize(
    ("values", "mean", "ci_low", "ci_high"),
    [
        ([1.0, 1.0, 1.0, 1.0], 1.0, 0.5100999795960008, 1.0),
        ([0.0, 0.0, 0.0, 0.0], 0.0, 0.0, 0.48990002040399916),
        ([1.0, 0.0, 1.0, 0.0], 0.5, 0.15003570882017148, 0.8499642911798285),
        ([1.0, 0.5], 0.75, 0.19786250921045673, 0.9733234672343529),
    ],
)
def test_wilson_interval_matches_hand_computed_values(
    values: list[float], mean: float, ci_low: float, ci_high: float
) -> None:
    estimate = runner_module.wilson_interval(values)
    assert estimate.ci_low == pytest.approx(ci_low)

Hand-computed constants catch algebra typos that a property assertion like ci_low <= mean will happily let through.

p95 latency gets an order statistic

p95 latency is the one people get most wrong. A normal interval around a 95th percentile assumes the latency distribution is symmetric, and provider latency is not remotely symmetric. It has a long right tail from retries and queueing.

So p95 does not get a parametric interval at all. It gets a nearest-rank estimate with bounds pulled from the sorted sample:

def percentile_interval(values: Sequence[float], q: float) -> Estimate:
    """Return a nearest-rank percentile with a binomial order-statistic CI."""
    n = len(values)
    if n == 0:
        return _empty_estimate()

    ordered = sorted(values)
    estimate_index = min(n - 1, max(0, math.ceil(q * n) - 1))
    estimate = ordered[estimate_index]
    if n == 1:
        return Estimate(mean=estimate, n=n, ci_low=estimate, ci_high=estimate)

    lower_index = min(n - 1, max(0, _binomial_quantile(n, q, 0.025)))
    upper_index = min(n - 1, max(0, _binomial_quantile(n, q, 0.975)))
    return Estimate(
        mean=estimate,
        n=n,
        ci_low=ordered[lower_index],
        ci_high=ordered[upper_index],
    )

The count of observations below the 95th percentile follows a binomial distribution. Take the 2.5th and 97.5th quantiles of that binomial, use them as indices into the sorted latencies, and the bounds are actual observed latencies. No distribution assumption, and the interval cannot land somewhere the data never went.

_binomial_quantile builds the PMF by recurrence from the mode outward rather than calling math.comb, because binomial coefficients at n in the hundreds overflow into float garbage:

mode = min(n, max(0, math.floor((n + 1) * probability)))
masses = [0.0] * (n + 1)
masses[mode] = 1.0

failure_odds = (1.0 - probability) / probability
for successes in range(mode, 0, -1):
    masses[successes - 1] = (
        masses[successes] * successes / (n - successes + 1) * failure_odds
    )

Starting at the mode with mass 1.0 and walking outward keeps every ratio near unity. The masses get normalized by their own sum at the end, so the unnormalized start does not matter.

The bug

Two of those three formulas were right. The routing between them was not.

Here is what aggregate_records used to do:

proportion_metrics = {
    metadata.get("key")
    for metadata in suite.display_metrics
    if metadata.get("format") == "percent"
}

interval = (
    wilson_interval(values)
    if metric_key in proportion_metrics
    else normal_mean_interval(values)
)

Interval selection keyed off the display format string. And retries_to_valid was declared like this:

{
    "key": "retries_to_valid",
    "label": "Retries to valid",
    "format": "number",
    "higher_is_better": False,
}

Format is "number", so it missed proportion_metrics and fell through to normal_mean_interval, which has no clamps:

standard_error = statistics.stdev(values) / math.sqrt(n)
half = 1.96 * standard_error
return Estimate(mean=mean, n=n, ci_low=mean - half, ci_high=mean + half)

Mean 0.125, mostly zeros with a few ones, standard deviation large relative to the mean. mean - half goes below zero and nothing stops it.

normal_mean_interval is correct for what it is. The failure is that I made a statistical decision depend on a presentation field. "percent" is a formatting instruction for the frontend. I overloaded it into "this quantity is a bounded proportion," and the two meanings drifted apart the moment I added a metric that was a bounded count rather than a percentage.

retries_to_valid is bounded below at zero. It just is not a proportion, so neither branch fit it.

The fix

Metrics now declare their statistical support explicitly, separate from how they render:

{
    "key": "retries_to_valid",
    "label": "Retries to valid",
    "format": "number",         # presentation
    "support": "non_negative",  # statistics
    "higher_is_better": False,
}

Routing reads support:

def _interval_for_support(support: str, values: Sequence[float]) -> Estimate:
    if support == "proportion":
        return wilson_interval(values)
    if support == "non_negative":
        return non_negative_mean_interval(values)
    return normal_mean_interval(values)

non_negative_mean_interval clamps the lower bound at zero. I considered a bootstrap instead, which handles skewed low-count data better, and decided against it for now. The reasoning is in the docstring:

"""Return a 95% normal interval around the mean, clamped at zero.

Clamping (rather than bootstrapping) keeps this a small, deterministic
change to an existing formula instead of introducing a new resampling
dependency for what is fundamentally the same asymptotic-normal
approximation used elsewhere in this module. It is a known-conservative
fix: for genuinely skewed low-count data the true interval is narrower
than a clamped symmetric one, but the clamped bound is never wrong in
the way an unclamped negative bound is.
"""

A clamped interval is too wide near zero. That is a real cost and I would rather state it than pretend the clamp is free.

The other half of the fix is refusing to guess. Missing metadata raises instead of defaulting:

missing = [key for key in suite.metric_keys if key not in declared]
if missing:
    raise ValueError(
        f"suite {suite.name!r} is missing declared support for metrics: "
        f"{sorted(missing)}"
    )

A silent default is how this happened. A new suite cannot now reach the leaderboard without saying what kind of quantity each of its metrics is.

The test I should have written first

Every interval test I had pinned hand-computed values for one function. All of them passed while the dashboard published a negative retry count, because not one of them asked whether the right formula reached the right column.

That test is generic, cheap, and runs across every registered suite:

@pytest.mark.parametrize(
    "suite", registry_module.list_suites(), ids=lambda suite: suite.name
)
def test_aggregate_records_intervals_stay_within_declared_support(
    suite: Suite,
) -> None:
    ...
    for row in response.rows:
        for metric_key, estimate in row.metrics.items():
            support = support_by_metric[metric_key]
            if estimate.ci_low is None:
                continue
            if support == "proportion":
                assert 0.0 <= estimate.ci_low
                assert estimate.ci_high <= 1.0
            elif support == "non_negative":
                assert estimate.ci_low >= 0.0

It does not check any particular number. It checks that no published bound falls outside the range its quantity can physically occupy. Unit tests on formulas verify the math. This verifies the wiring, and the wiring is what broke.

There is also a regression test pinned to the exact reported case, 40 records with one outlier requiring 5 retries, mean 0.125, asserting ci_low >= 0. I confirmed it failed before the fix. A regression test you never watched fail is a test you are trusting for no reason.

The backend went from 262 tests to 267. Five tests, and one of them covers a class of failure that the other 262 could not see.

Takeaway

Publish the interval and the sample size next to every mean, and test that each interval respects the support of the thing it is measuring, because a leaderboard that cannot check its own bounds will cheerfully show you a negative retry count instead.