DEV Community: Maya Andersson

Run forty experiments against one eval set and you will find an improvement that is not there

Maya Andersson — Wed, 12 Aug 2026 14:07:15 +0000

You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an upward-biased estimate of the thing you care about. The size of that bias is computable, it is larger than most of the "wins" teams celebrate, and there is a decade-old literature on how to reuse a holdout without fooling yourself that I almost never see cited in LLM eval writeups.

The selection machine

The workflow looks responsible. A fixed eval set of 250 examples, a binary pass judgment per example, every candidate prompt measured on the same set. Best score ships. It looks like discipline.

The problem is the word "best." Each measurement is the true pass rate plus sampling noise. Pick the maximum of forty measurements and you have preferentially picked positive noise. The winner's measured score overstates its true score even when some candidates are genuinely better than others; the statistics community has called this the winner's curse for decades.

How big the bias is

For a pass rate near p on n examples, one measurement has standard error sqrt(p(1-p)/n). At p = 0.85 and n = 250 that is 2.26 points. Now suppose, worst case, all k candidates are actually identical: every measured difference is pure noise. The expected maximum of k standard normal draws is easy to compute numerically:

5 experiments: expected max 1.16 sigma, so +2.6 points at n = 250
10 experiments: 1.54 sigma, +3.5 points
40 experiments: 2.16 sigma, +4.9 points
100 experiments: 2.51 sigma, +5.7 points

Forty tries against a 250-example set buys you an expected phantom improvement of about five points. If your quarterly review says the prompt work moved the pass rate from 85 to 89, the null hypothesis says: that is what selection looks like when nothing improved at all.

Twelve lines to check my arithmetic:

import numpy as np

rng = np.random.default_rng(0)
n, p, k, sims = 250, 0.85, 40, 100_000

# k identical candidates, each measured once on n examples
runs = rng.binomial(n, p, size=(sims, k)) / n
best = runs.max(axis=1)

print(f"true rate: {p:.3f}")
print(f"mean best-of-{k}: {best.mean():.3f}")
print(f"inflation: {(best.mean() - p) * 100:.1f} points")

This prints about +4.7, just under the normal-approximation figure above for k = 40 (the binomial's discreteness and ties shave off a couple of tenths). Change k and re-run: the bias keeps growing, roughly like sqrt(2 ln k), so it slows with persistence but never stops.

The part the field already solved

Dwork, Feldman, Hardt, Pitassi, Reingold, and Roth worked out the general problem in 2015 under the name adaptive data analysis (the reusable holdout line of work; arXiv:1506.02629). Their observation: the moment your next experiment depends on the last measurement, the holdout stops being fresh, and classical guarantees quietly void themselves. Their mechanism, Thresholdout, answers holdout queries through a noise-adding gate so the set survives many more adaptive looks.

You do not need to implement Thresholdout to benefit from the diagnosis. Four practices capture most of the value:

Two sets, two roles. Iterate freely against a dev set. Confirm on a frozen set you touch only when shipping. The confirm set's job is to be boring: it answers a handful of times per quarter, not forty.
Selection-adjusted bars. If you compared k variants, the winner must clear the baseline by more than the expected max of k noise draws, not by more than zero. For k = 40 at n = 250, that bar is about five points, which is sobering and correct.
Count your touches. k is the one quantity in this whole analysis you control and can log exactly. Teams version their prompts and their datasets and keep no record of how many times the eval set has been queried. Make k a first-class number in the eval report.
Retire burned sets. After enough adaptive looks, refresh the eval set from new traffic and re-baseline. Not because the data drifted (that is a separate failure with its own essay) but because your process has memorized this particular sample's noise.

FAQ

Does a bigger eval set fix this? It helps: sigma scales as 1/sqrt(n), so quadrupling the set halves the inflation. It does not change the shape: the bias still grows with k. A bigger set lowers the noise floor. The selection effect sitting on top of it remains.

Is this just p-hacking with a new name? It is the same statistical family. P-hacking selects among analyses of one experiment; this selects among experiments measured on one reused set. The adaptive data analysis literature treats both as instances of the same leakage.

Has anyone measured whether this happens in practice, or is it theory? Measured, with a twist. Recht, Roelofs, Schmidt, and Shankar built a fresh CIFAR-10 test set in 2018 (arXiv:1806.00451) expecting to catch years of test-set reuse as overfitting. The drop they found pointed at distribution shift instead, and adaptivity-driven overfitting was smaller than theory allows. Two honest readings: benchmark-scale selection has correction mechanisms (many teams, public leaderboards, diverse models), and a single team hammering a private 250-example set with forty variants of one prompt is a much purer selection machine than ImageNet ever was. The math above is the worst case; your k and your n decide how close you sit to it.

Open question

Is there a principled touch budget for an eval set: a k beyond which refresh is mandatory, stated in advance the way significance levels are? I have a heuristic (refresh when the selection-adjusted bar exceeds the effect sizes you care about) but I have not seen a treatment that turns it into a rule teams actually adopt. If you have seen one in the wild, I want to read it.

I read the metric libraries of five widely-used eval tools. The metric was never the hard part.

Maya Andersson — Tue, 11 Aug 2026 18:53:06 +0000

Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. The pitch works because it is true, and because it quietly relocates the hard part of evaluation to somewhere you cannot see it.

I spent a week reading the actual metric catalogs of five widely-used eval libraries, not the landing pages. Arize Phoenix, DeepEval, Future AGI, Langfuse, and Ragas. I wanted to know two things. What do you get for free, and what does the free part let you stop thinking about. The answer to the second question is: almost nothing. The metric is the easy 20 percent. The two hard parts, choosing a metric that matches your failure mode and putting an error bar on the result, are still yours, and none of these tools does them for you.

Here is the survey, then the argument.

What five catalogs actually ship

I ordered these alphabetically on purpose. I am not ranking them, and I built a comparison table precisely so I would not be tempted to.

Tool	License	Built-in metrics (documented)	LLM-as-judge	Aggregation
Arize Phoenix	Elastic License 2.0 (source-available)	~16 pre-built evaluators: Faithfulness, Correctness, Q&A, Tool Selection, Hallucination, Toxicity	Yes, judge templates use function calling to extract structured verdicts	Datasets and Experiments: per-example scores, aggregate metrics, cross-experiment comparison
DeepEval	Apache-2.0	"50+" metrics: G-Eval, DAG, Answer Relevancy, Faithfulness, Tool Correctness, Hallucination, Bias	Yes, most metrics are LLM-judged (QAG, DAG, G-Eval)	Pytest-style pass/fail against thresholds; `-r` flag repeats a test case
Future AGI	Apache-2.0	72 local metrics: BLEU, ROUGE, embedding similarity, JSON-schema, plus judged faithfulness and function-call accuracy	Yes, local heuristics augmentable with a judge via `augment=True`	Aggregated metric type; datasets in the platform
Langfuse	MIT (core; `ee/` is commercial)	Managed evaluator catalog: Hallucination, Context-Relevance, Toxicity, Helpfulness (some maintained with Ragas)	Yes, numeric, categorical, or boolean judge scores	Dataset runs scored at aggregate level, side-by-side experiment comparison
Ragas	Apache-2.0	"30+" metrics: Faithfulness, Context Precision, Context Recall, Response Relevancy, Tool Call Accuracy	Yes, LLM-based metrics issue one or more judge calls	`evaluate()` returns mean scores across the dataset; `to_pandas()` for per-sample

A few honest notes before anyone quotes this table at me.

The metric counts are each project's own stated approximation, not a hand count. "50+", "30+", and "~16" are marketing-adjacent numbers from their docs, and "72" is a figure from Future AGI's SDK README. I did not recount anyone's catalog line by line, so treat the column as order-of-magnitude, not a leaderboard. Phoenix is worth a license asterisk: Elastic License 2.0 is source-available, not OSI-approved open source, which matters if you are redistributing, less so if you are just running it. Langfuse is MIT at the core with a separately licensed enterprise directory. The other three are Apache-2.0.

The overlap is the real story. Faithfulness shows up in four of the five. Tool-calling correctness shows up in four. Every one of them supports LLM-as-a-judge, and every one of them returns you a score. If you were choosing on metric coverage alone, you would have a hard time telling them apart, because for the common cases they are converging on the same list. That convergence is a signal. The catalog is becoming a commodity. Which means the catalog is not where your evaluation succeeds or fails.

The hard part number one: the metric that matches your failure

A built-in metric encodes a definition somebody else wrote. "Faithfulness" in a RAG library means the answer is grounded in the retrieved context. That is a real and useful thing to measure. It is also completely silent about whether your agent picked the right document to retrieve in the first place, whether it called the refund tool instead of the lookup tool, or whether it was confidently, groundedly wrong because the context itself was stale.

I have watched teams adopt a library, wire up five of its metrics, get a dashboard full of 0.9s, and ship a regression anyway, because none of the five metrics measured the thing that actually broke. The tool did its job. The scores were accurate. They were just answers to questions nobody's users were asking.

The move that works is unglamorous and the tools cannot do it for you. Write down your failure taxonomy first. Go read fifty real production traces, cluster the ways your system actually fails, and only then go shopping in the catalog for metrics that map onto those clusters. Most of the time you will find that two or three built-in metrics cover your top failure modes and the rest are noise you were about to pay judge-call money to compute. Occasionally you will find that your worst failure mode has no built-in metric at all, and you will write a custom judge. Every one of these five tools lets you write a custom metric, which tells you the authors already know the catalog does not cover you.

So the first hard part is selection, and selection is a function of your traces, not their docs.

The hard part number two: the error bar nobody ships

Here is the finding that sent me down this whole path. I read the aggregation behavior of all five tools carefully, because aggregation is where statistics is supposed to happen. What I found is that all five will happily give you a mean. Ragas returns {'faithfulness': 0.892}. Phoenix and Langfuse show aggregate scores across an experiment. DeepEval gives you a pass rate against a threshold. Future AGI has an aggregated-metric type.

Not one of them, as of July 2026, computes a confidence interval for you by default.

I want to be precise, because this is the kind of claim that gets misquoted into "tool X can't do statistics." That is not what I am saying. I am saying I went looking in the documented, out-of-the-box behavior of each library and did not find confidence-interval computation in any of them, Future AGI included. DeepEval's -r repeat flag gets you closest to the raw material, since you can run a case multiple times, but the docs stop at repeating, not at turning the repeats into an interval. The others aggregate to a mean and stop.

This matters more than the metric-count column everyone stares at. A faithfulness pass rate of 0.88 measured on 50 examples (44 of 50 above threshold) is not 0.88. It is 0.88 with a 95 percent Wilson interval running from roughly 0.76 to 0.94 (Wilson, 1927, the standard small-sample interval for a proportion). If your last release scored 0.82 (41 of 50) on the same 50 and you are about to write "faithfulness improved," you are reading noise as signal. The tool told you 0.82 and 0.88. It did not tell you those two numbers are statistically indistinguishable, because computing that was never its job, and it did not warn you that you were about to assume it was.

Here are the ten lines the catalog leaves to you.

from statsmodels.stats.proportion import proportion_confint

def pass_rate_with_ci(passes, n, alpha=0.05):
    rate = passes / n
    lo, hi = proportion_confint(passes, n, alpha=alpha, method="wilson")
    return rate, (lo, hi)

# 44 of 50 examples passed the metric's threshold
rate, (lo, hi) = pass_rate_with_ci(44, 50)
print(f"{rate:.2f}  95% CI [{lo:.2f}, {hi:.2f}]")
# 0.88  95% CI [0.76, 0.94]

That interval is the difference between "we improved" and "we cannot tell yet, run more examples." For continuous judge scores rather than pass/fail, bootstrap the mean instead of using Wilson, but the discipline is identical: never report a point estimate from an eval set without the interval around it, and never compare two releases whose intervals overlap as if one won.

So what is the tool for

This is not an argument against any of these libraries. I use them. The catalog genuinely saves you from reimplementing BLEU, ROUGE, embedding similarity, and a dozen judge prompts, and standardizing on one of them is a good decision. Phoenix rides OpenTelemetry if that is your stack. DeepEval's pytest ergonomics are the cleanest if you live in CI. Future AGI if you want eval, tracing, and guardrails in one place. Langfuse if you want tracing and evals in one product. Ragas is still the sharpest on RAG. They are all fine, and they are all converging.

The mistake is thinking the tool finished your evaluation. It started it. It computed the metric, which was the part a library can do, and it handed you back a number stripped of the two things that make the number mean anything: whether the metric matches your failure, and how much of the number is noise. Those two stay on your side of the API no matter which logo is on the dashboard.

FAQ

Which of these five should I pick?
Whichever fits your stack and license constraints, because on metric coverage they are close to interchangeable for common cases. Decide on ergonomics (pytest vs platform vs OTel), license (Apache-2.0 vs Elastic vs MIT-core), and whether you need the surrounding tracing and datasets, not on the metric count.

Is a bigger metric library better?
No. A bigger library raises the chance that a metric matching your failure mode exists, but it does nothing to raise the chance that you picked it. Ten well-chosen metrics beat seventy defaults every time.

Do I really need confidence intervals for eval?
If you are making ship/no-ship decisions on the difference between two scores, yes, always. The interval is what tells you whether the difference is real. Ten extra lines of code, and it is the highest-leverage code in your eval suite.

Does any tool do the interval for me?
Not by default, in any of the five I read, as of July 2026. If that changes I will happily update this. Until then, wrap your own.

Open question

If the metric catalog is commoditizing, the differentiation moves up a level, to selection and uncertainty. So why does no eval tool ship a "your last two runs are statistically indistinguishable" warning yet? It is a small feature. It would prevent a large fraction of the false "we improved" claims I see in eval writeups. My guess is that a tool that keeps telling you your result is inconclusive is a harder product to sell than one that always returns a confident number. I would love to be proven wrong by whichever project ships it first.

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

Maya Andersson — Tue, 11 Aug 2026 18:50:05 +0000

A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate drops below 88 percent. Their baseline is around 92. The monitor paged on four separate days last week. Four investigations, four shrugs, four "it recovered on its own."

Nobody chose a false-alarm rate for that monitor. But it has one, and it is not small. With roughly 150 judge scores an hour, a fresh window each check, and a true pass rate of 0.92, the chance that at least one hourly check dips below 0.88 at some point in a day is 53 percent. That makes the expected number of alert days in a week 3.7, and the single most likely weekly count exactly four. Their monitor was not detecting regressions. It was sampling noise on a schedule, and the on-call rotation was the readout.

An alert threshold on an eval score is a hypothesis test. Every tool that ships eval monitoring runs that test for you. What none of the configuration surfaces I read this month asks you for is the one number the test depends on: how many scores are in the window.

The test you are actually running

The setup is ordinary binomial arithmetic, which is what makes it checkable. Each check looks at a fresh window of n judge verdicts. The true pass rate is p. The monitor fires when the observed rate falls below a threshold t. Then:

import math
from scipy.stats import binom

def alpha_per_check(n, p0=0.92, t=0.88):
    """P(false alarm): windowed rate < t although nothing changed."""
    k = math.ceil(t * n) - 1          # largest count that fires
    return binom.cdf(k, n, p0)

def p_fire_today(n, checks=24, **kw):
    a = alpha_per_check(n, **kw)
    return 1 - (1 - a) ** checks      # tumbling windows, independent checks

def power_per_check(n, p1=0.85, t=0.88):
    k = math.ceil(t * n) - 1
    return binom.cdf(k, n, p1)        # P(one check catches a real drop)

Run that for a 0.88 threshold against a 0.92 baseline, across realistic window sizes, and add the probability that a single check catches a genuine regression to 0.85:

Scores per window	Fires per check on noise	Fires some time today	Expected alert days per week	Catches a real drop to 0.85, per check
25	13.5%	96.9%	6.8	52.9%
50	10.2%	92.4%	6.5	63.9%
100	5.6%	74.8%	5.2	75.3%
150	3.1%	53.3%	3.7	81.9%
200	1.8%	35.0%	2.5	86.3%
400	0.2%	4.9%	0.3	94.9%

Read the 25-per-window row twice. That monitor pages nearly every day on pure noise, and a single check still misses a real seven-point regression about half the time. It manages to be trigger-happy and insensitive simultaneously, which is what happens when the threshold is chosen as a round number four points under baseline instead of as a quantile of anything.

The window size is not an abstract parameter. It is your traffic times your sampling rate. Sample 10 percent of 500 traces an hour and n is 50. The sampling knob every tool gives you for cost control is also, silently, the monitor's sensitivity knob: cutting sampling from 100 percent to 10 percent widens the score's confidence interval by a factor of about 3.2, the square root of ten. None of this is exotic. It is the standard behaviour of binomial proportions, catalogued in detail in Brown, Cai and DasGupta's "Interval Estimation for a Binomial Proportion" (Statistical Science, 2001).

So the question I brought to the current tooling was not "who has alerting." Most do. It was three narrower questions.

Question one: which traces does the score come from?

This decides n, so it decides everything above.

Langfuse runs judge evaluators at ingest time on observations that match your filters, with a sampling percentage you set; matching observations enter an evaluation queue and are scored asynchronously. Filters cover observation type, trace name, tags, user, session and metadata, so the scored population is well defined, and the docs are explicit that sampling exists to manage evaluation cost.

LangSmith does the equivalent through automation rules: a filter selects runs, a sampling rate decides what fraction of the filtered runs the evaluator sees, and the scores attach back to the traces.

Opik's online evaluation rules score live production traces with LLM-as-judge metrics and write results back as feedback scores on each trace.

Future AGI's continuous eval tasks take the same two controls; a forward-only cursor means history is never backfilled.

Phoenix evaluates production traces when you run evals over them by SDK or in the app; for continuously scheduled scoring of live traffic its own docs point you to Arize AX, the commercial platform. DeepEval sits at the same boundary from the other side: the framework is offline-first, and production scoring is the companion Confident AI platform's job.

Question two: what turns a score into a page?

Langfuse has the most developed vocabulary here of the set I read. Monitors watch numeric, categorical or boolean scores; you set an operator, an alert threshold and an optional warning threshold over a lookback window, and route through Slack, webhooks or GitHub Actions. It is also the only config I found that makes you decide what an empty window means: treat missing data as zero, hold the previous severity, record NO_DATA silently, or page after sustained NO_DATA. How many monitors you can create depends on plan tier; the features themselves do not.

LangSmith alerts on five metric types, run count, cost, errors, latency and feedback score, that last one being where online eval results live, and it routes to Slack, PagerDuty, Dynatrace or any webhook.

Future AGI's monitors take a static or percentage-change threshold, or learn one from the historical mean, with separate warning and critical levels and a check frequency in minutes.

Opik writes rule scores onto traces and tracks them on dashboards; its online-rules documentation describes score computation and display, and no notification configuration appears on that page.

Phoenix routes threshold-based triggers on production traffic to Arize AX, per its own docs. I did not audit AX's alert configuration, nor Confident AI's, so the two commercial companions stay out of the comparison below.

Question three: where does sample size enter?

I read three alerting configuration surfaces end to end this month: Langfuse's monitor settings, LangSmith's alert settings and Future AGI's monitor model. In all three, the vocabulary is drawn from the same short list: a value, a direction, a window, sometimes a check frequency. None of the three has a field for the number of scores the window is expected to contain, and none derives the threshold from one. The window is specified in time, five minutes, an hour, a day, and how many scores fall inside it is whatever traffic and sampling happen to produce.

Two of the threshold types deserve a specific caution. A percentage-change threshold compares two noisy window estimates, and the difference of two independent windowed rates carries roughly twice the variance of either one, so at small n it is noisier than the static threshold it replaces. And an auto or anomaly threshold that learns the historical mean solves a real problem, baseline drift, but the alert it fires is still a point-estimate comparison; learning where the baseline sits is not the same as knowing how far a healthy window wanders from it.

To be precise about scope: this is what the configuration surfaces expose as of early August 2026, from each vendor's public docs and, where the code is public, its source tree. Any of them could ship a sample-size-aware policy tomorrow, and the gap is easy to work around today, which is the last section.

Setting the threshold from n instead

You do not need any vendor to fix this. Invert the binomial: decide the false-alarm rate you can staff, then compute the threshold your window size implies.

from scipy.stats import binom

def threshold_from_n(n, alpha, p0=0.92):
    """Largest rate threshold whose per-check false alarm stays <= alpha."""
    k = int(binom.ppf(alpha, n, p0))
    if binom.cdf(k, n, p0) > alpha:
        k -= 1
    return k / n

Scores per window	Page below (1% per check)	Warn below (10% per check)
25	0.720	0.800
50	0.800	0.840
100	0.840	0.870
150	0.860	0.887
200	0.870	0.890
400	0.885	0.900

Two things fall out. First, at 25 scores a window, an honest 1-percent page threshold is 0.72, a full sixteen points below the 0.88 the team "felt" was right; the small-n rows are exactly where intuition overshoots most. Second, the pairing maps cleanly onto the warning-plus-critical structure that Langfuse and Future AGI both expose and that LangSmith approximates with two rules: put the warning at the 10 percent quantile and the page at the 1 percent quantile, and both levels inherit a false-alarm ceiling you actually chose.

If your platform only takes time-based windows, fix the count instead: check every N scores rather than every hour. A count-based window makes n a constant, which makes the table above exact instead of approximate, and it stops quiet hours from paging you simply because n collapsed overnight.

FAQ

Is 0.92 a magic baseline? No. Every number above recomputes for your baseline and your window in the four lines of code shown; the shape of the conclusion, thresholds must move with n, survives any realistic parameter choice.

My monitor uses a rolling window checked every five minutes. Does the daily 53 percent still apply? Not directly, and the direction matters: the five-minute checks include the hourly windows among them, so the daily probability is at least the 53 percent, not less. Plugging 288 checks into the independence formula overstates it badly, though; overlapping windows are correlated, the per-check false alarm is unchanged, and correlated checks cluster their false alarms into the same bad hour.

Why not just require two consecutive breaches? That is a legitimate sample-size-aware policy, and on tumbling windows it squares the per-check false-alarm rate. You pay in detection delay, one extra check period, and in power against short-lived regressions. It is the cheapest fix on this page; the quantile threshold is the principled one.

Open question

Auto thresholds that learn the baseline mean already read the score history. The same history contains everything needed to learn the window-to-window variance and set the alert line at a chosen quantile of it, which would make the monitor's false-alarm rate a configured property instead of an accident of traffic. None of the docs I read says whether any of the auto modes does this today. If someone from one of these teams can point me at a sample-size-aware or variance-aware alert policy in their product, mine is exactly the kind of dashboard it would quiet, and I will happily run the comparison again.

Sources for the tool claims, all read in the first week of August 2026: Langfuse docs, "Monitors and Alerts" and the LLM-as-a-judge evaluator pages; LangSmith docs, "Online evaluations" and "Alerts"; Opik repository README and its online evaluation rules doc (comet-ml/opik); Phoenix repository docs, llm-evals page (Arize-ai/phoenix); DeepEval repository README (confident-ai/deepeval); Future AGI repository, tracer models eval_task.py and monitor.py (future-agi/future-agi). The binomial arithmetic is standard; see Brown, Cai and DasGupta, "Interval Estimation for a Binomial Proportion," Statistical Science 16(2), 2001.

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

Maya Andersson — Fri, 07 Aug 2026 19:01:00 +0000

The test you are actually running

import math
from scipy.stats import binom

def alpha_per_check(n, p0=0.92, t=0.88):
    """P(false alarm): windowed rate < t although nothing changed."""
    k = math.ceil(t * n) - 1          # largest count that fires
    return binom.cdf(k, n, p0)

def p_fire_today(n, checks=24, **kw):
    a = alpha_per_check(n, **kw)
    return 1 - (1 - a) ** checks      # tumbling windows, independent checks

def power_per_check(n, p1=0.85, t=0.88):
    k = math.ceil(t * n) - 1
    return binom.cdf(k, n, p1)        # P(one check catches a real drop)

Run that for a 0.88 threshold against a 0.92 baseline, across realistic window sizes, and add the probability that a single check catches a genuine regression to 0.85:

Scores per window	Fires per check on noise	Fires some time today	Expected alert days per week	Catches a real drop to 0.85, per check
25	13.5%	96.9%	6.8	52.9%
50	10.2%	92.4%	6.5	63.9%
100	5.6%	74.8%	5.2	75.3%
150	3.1%	53.3%	3.7	81.9%
200	1.8%	35.0%	2.5	86.3%
400	0.2%	4.9%	0.3	94.9%

So the question I brought to the current tooling was not "who has alerting." Most do. It was three narrower questions.

Question one: which traces does the score come from?

This decides n, so it decides everything above.

LangSmith does the equivalent through automation rules: a filter selects runs, a sampling rate decides what fraction of the filtered runs the evaluator sees, and the scores attach back to the traces.

Opik's online evaluation rules score live production traces with LLM-as-judge metrics and write results back as feedback scores on each trace.

Future AGI's continuous eval tasks take the same two controls; a forward-only cursor means history is never backfilled.

Question two: what turns a score into a page?

Future AGI's monitors take a static or percentage-change threshold, or learn one from the historical mean, with separate warning and critical levels and a check frequency in minutes.

Opik writes rule scores onto traces and tracks them on dashboards; its online-rules documentation describes score computation and display, and no notification configuration appears on that page.

Question three: where does sample size enter?

Setting the threshold from n instead

You do not need any vendor to fix this. Invert the binomial: decide the false-alarm rate you can staff, then compute the threshold your window size implies.

from scipy.stats import binom

def threshold_from_n(n, alpha, p0=0.92):
    """Largest rate threshold whose per-check false alarm stays <= alpha."""
    k = int(binom.ppf(alpha, n, p0))
    if binom.cdf(k, n, p0) > alpha:
        k -= 1
    return k / n

Scores per window	Page below (1% per check)	Warn below (10% per check)
25	0.720	0.800
50	0.800	0.840
100	0.840	0.870
150	0.860	0.887
200	0.870	0.890
400	0.885	0.900

FAQ

Open question

Upgrading the judge ends one score series and starts another

Maya Andersson — Thu, 06 Aug 2026 19:17:16 +0000

There is a mature literature on what happens when you swap one measuring instrument for another, and it is not in machine learning.

The standard treatment is Bland and Altman, "Statistical methods for assessing agreement between two methods of clinical measurement", Lancet 1986; i: 307-310. Their core move is the one worth borrowing: do not correlate the two instruments. Plot their difference against their average, and look at the shape.

Correlation is the wrong tool because it measures association rather than agreement, and it inflates with the spread of whatever sample you happened to use. Two instruments can correlate at 1.00 while one of them reads a fifth of the scale high everywhere it has room to. A constant offset is a perfect linear relationship, so correlation reports agreement that is not there.

An LLM-as-judge is an instrument. When we replaced ours with its successor and the weekly quality score moved from 0.78 to 0.83 in the same week, we had changed the system being measured and the thing measuring it at once, and the honest position was that we did not know which had moved. Our scores run 0 to 1.

The usual response is to estimate an offset and subtract it. That is a reasonable instinct and it is also a model, with assumptions, and the assumptions are checkable. Three checks, and the third one I got wrong the first time I ran it.

The apparatus

An anchor set: a fixed collection of (input, output) pairs, with the outputs frozen, scored under both judges. Paired, same items, same text, two raters.

Ours had 60 items. On it, sd(d) was 0.12, where d is the per-item difference, new judge minus old. Both judges had a score sd of 0.22 on the anchor set, so the ratio sd(new)/sd(old) came out at 1.0 to the precision I can report from two figures rounded to two decimals. That ratio matters more than it looks, and check two is where I explain why. The scores themselves are continuous, a weighted rubric rolled up to 0 to 1, not a grid of discrete levels; that turns out to matter for check three.

Check one: the offset, and how badly you know it

The offset is the mean of d, and the useful number is not the offset but its confidence interval:

half-width = 1.96 x sd(d) / sqrt(n)

Rearranged, that gives the anchor set size you need for a given precision:

±0.05 on the offset: 10 items at sd(d) = 0.08, 23 at 0.12, 50 at 0.18
±0.03 on the offset: 28 items at sd(d) = 0.08, 62 at 0.12, 139 at 0.18
±0.02 on the offset: 62 items at sd(d) = 0.08, 139 at 0.12, 312 at 0.18
±0.01 on the offset: 246 items at sd(d) = 0.08, 554 at 0.12, 1,245 at 0.18

That table uses z = 1.96, the large-sample form. Under the t distribution, which is what Bland and Altman specify because sd(d) is estimated rather than known, every cell rises by two or three items. Proportionally it only matters at the top: the 10 becomes 13 and the 23 becomes 25, while the 554 becomes 556.

At 60 items and sd(d) = 0.12, our offset is pinned to ±0.030.

Now put the two numbers next to each other. We were trying to detect a product improvement of about 0.05, using a correction whose own uncertainty was ±0.030. The error bar on the correction was about 60% of the thing being measured. Subtracting that offset relabels the confounding rather than removing it, and it does so in a way that looks precise because a single number has been subtracted.

Note which interval that is, because I conflated the two for longer than I would like. The ±0.030 is the precision of the mean offset, which is the right number for correcting an aggregate. Bland and Altman's headline statistic is the limits of agreement, mean ± 1.96 x sd(d), which for us is ±0.235, and at 60 items each of those limits is itself uncertain by roughly ±0.05. Any individual rescaled score carries the wide interval, not the narrow one. If you plan to correct a per-item score and act on it, you are working with something close to eight times the width you computed.

A word on why the paired design is worth the trouble, since the reason usually given is wrong. Scoring the same frozen outputs twice does not merely make sd(d) small. Against two disjoint samples of frozen outputs, one per judge, the paired design needs 2/(1 - rho) times fewer items and 1/(1 - rho) times fewer judge calls, so it wins on items even when the two judges are uncorrelated. What rho buys is the judge calls. Our sd(d) of 0.12 against a score sd of 0.22 implies rho of about 0.85, and at ±0.01 that is 3,719 items per arm unpaired, 7,438 in total, against 554 scored by both judges: thirteen times the items, or 6.7 times the judge calls.

(The identity sd(d)² = 2 x sd(score)² x (1 - rho) assumes both judges have the same score sd, which ours did. If yours do not, use rho = (sd_new² + sd_old² - sd_d²) / (2 x sd_new x sd_old), because the shortcut will quietly mis-state rho and everything downstream of it.)

Check two: did the spread change, or only the location?

Subtracting an offset assumes the new judge is the old judge shifted. In what I have seen, it is at least as common for a new judge to be compressed: more agreeable, more clustered in the middle, fewer confident lows. The mean moves and the standard deviation shrinks.

The diagnostic is the ratio sd(new) / sd(old) on the anchor set, plus the difference-against-average plot. If the difference d trends with the average score, or fans out at one end, the relationship is not a shift and no single constant will fix it.

Ours came back at 1.00, which is the boring answer and the one that let me use the simple rho identity above. When it does not come back near 1, the consequence is specific: a compressed judge attenuates every future comparison toward zero, your real improvements shrink, your real regressions shrink, and the series goes quiet. This is the same shape as the attenuation I wrote about last week, arriving by a different route. There the effect shrank because a binary judge's flips are bounded. Here it would shrink because the judge's scale is narrower.

I owe this check the same precision arithmetic I gave check one, and I have not done it. There is a proper paired-variance test (Pitman and Morgan's, which works by testing the correlation between the sum and the difference), and I have not worked out what interval 60 items buys on the ratio. So the point estimate is 1.00 and I am not going to pretend I know how tightly. I would rather say that than quote a number I have not derived.

Check three: did the ranking change, and what would "unchanged" have looked like?

The offset and the spread are both about the scale. The third question is about order: does the new judge rank items the way the old one did? Kendall's tau or Spearman's rho on the pairs.

Here is the mistake I made, and it is the reason this section exists.

I computed tau, got 0.65, and read that as substantial rank movement. It is not. Before reading tau you have to work out what tau you would get if nothing but noise had changed. For jointly normal scores, tau = (2/pi) x arcsin(rho), and our implied rho of 0.85 gives tau = 0.6483 on its own. So the gap I thought I was reading was 0.002, which at 60 items is not a gap at all. The arcsin value is the floor and I had mistaken it for a finding.

And then the uncomfortable corollary, which is the reason I now treat this check as the weakest of the three. Under joint normality, rho and tau carry the same information, so comparing observed tau against the rho-implied floor is not really a rank-change detector at all. It is a test of the normality and linearity assumption. It fires when the relationship between the judges is monotone but curved, or when a handful of extreme pairs are propping up the Pearson correlation.

One caveat on the floor itself, which I got wrong on the first attempt. The arcsin identity is a continuous-normal result. Our anchor scores are continuous, so it applies. If yours sit on a discrete rubric grid, ties break it in both directions at once: tau-a is dragged down by tied pairs, while tau-b's tie correction over-restores and lands above the continuous value. On a grid the arcsin number is simply the wrong null, and the floor has to be recomputed on your actual grid before any of this means anything. Say which variant you computed, too.

So the check is worth running, and what it earns you is a reason to look harder at the difference-against-average plot. It does not on its own earn you the conclusion that you have two series.

When you do conclude the ranks have moved, it decides whether splicing is possible. If ranks are preserved, some monotone transformation maps one scale to the other, and with enough anchor items you can estimate it. If they have moved, there is no item-level rescaling that makes a score from March comparable to a score from August. You can still equate the two score distributions and keep the aggregate series roughly comparable, at the cost of no individual rescaled score being trustworthy.

Anchor-set construction: freeze the outputs

The outputs must be frozen artefacts, not re-generated.

It is tempting to take last quarter's 200 eval inputs, run today's system on them, and score that with both judges. It feels like the same thing. It is not: the model, the prompt and the retrieval index have all moved, so the paired differences now contain both judge drift and system drift, and the whole point of the exercise was to separate those. Store the text, score the text, and accept that the anchor set stops being a test suite the moment you freeze it.

When the ranks have moved

I have found only one honest option, and it is expensive: run both judges in parallel on a sample of traffic for a transition window, publish both series side by side, and let the old series end rather than pretending it continues. Annotate the chart at the switch date. Anyone who reads across the boundary is then doing so knowingly.

The alternative, which I have watched happen, is that the offset gets subtracted, the chart stays smooth, and six weeks later a genuine regression is invisible because it is the same size as the correction nobody re-examined.

FAQ

How often does the judge actually change? More often than the calendar suggests. A version pin protects you from silent provider updates but not from your own team changing the rubric text, the temperature, the few-shot examples, or the parser. Every one of those is a new instrument by this argument, and the rubric edit is the most frequent and least announced.

Can I skip this if I only look at relative comparisons within a week? Partly. The judge's offset cancels out of a comparison where both arms were scored by the same judge in the same window. Its scale does not: a compressed judge attenuates the contrast by the same factor in both weeks, so the comparison stays valid in sign and direction but not in magnitude. That is still a good reason to structure reporting around within-week contrasts rather than a long-running level, because the level is where the trouble is worst, and levels are what end up on dashboards.

Is 554 anchor items realistic? For ±0.01, at our sd(d), that is what the arithmetic says. Most teams do not need ±0.01, and the table above lets you buy the precision you actually need.

Does a deterministic metric avoid all of this? For the metric itself, yes. Exact match does not drift. But most people are not upgrading exact match, they are upgrading the thing they use because exact match was inadequate, and that thing has a version.

Open question

I have been assuming the anchor set is representative of production traffic, and everything above inherits that assumption. But an anchor set is frozen by construction, and traffic is not. Six months in, the anchor set describes a distribution that no longer arrives, so the offset you measured is the offset on old traffic, which may not be the offset on current traffic.

Refreshing the anchor set restores representativeness and destroys comparability with everything measured before the refresh, which is the same problem one level up.

I do not have a clean answer. The best I have managed is overlapping anchor sets, retiring and adding a slice at a time so consecutive versions share most of their items, which is the common-item nonequivalent-groups design from the test-equating literature (Kolen and Brennan, Test Equating, Scaling, and Linking). Whether that is worth the machinery for an internal quality metric, I genuinely do not know.

A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.

Maya Andersson — Wed, 05 Aug 2026 19:28:06 +0000

Most of what I have written about LLM judges argues in one direction: your improvement is probably not real. Small eval sets, selection across many experiments, position bias, clustered examples. All of that pushes toward scepticism and I still believe it.

This post argues the other way, because there is a failure mode that runs opposite and I do not see it discussed. Judge noise biases your measured improvement toward zero. The interval gets wider, which everyone expects, and the point estimate also moves, systematically, in the same direction every time. A change that genuinely helped can read as flat, you will record a null result, and the noisier your judge the more confident that mistake becomes.

This is old statistics wearing new clothes. Spearman named it attenuation in 1904: an observed correlation is bounded above by the geometric mean of the reliabilities of the two instruments measuring it, so unreliable measurement makes real relationships look weaker than they are ("The proof and measurement of association between two things", American Journal of Psychology 15, 72-101, whose section II.2 is titled "'Attenuation' by errors"). An LLM judge is an instrument with reliability well under 1, and we report its output as though it were a ruler.

The arithmetic, which is embarrassingly simple

Take a binary judge: pass or fail. Suppose its flip probability is the same conditional on either true label, so it is as likely to turn a true pass into a reported fail as the reverse. Call that flip probability e.

If a model's true pass rate is p, the rate your judge reports is:

p_observed = p(1 - e) + (1 - p)e
           = e + p(1 - 2e)

Now measure two models, true rates p_A and p_B, with the same judge. Subtract:

p_observed(B) - p_observed(A) = (1 - 2e) * (p_B - p_A)

The e term cancels. What survives is a multiplier on your effect: every measured gap is the true gap times (1 - 2e).

That factor bites at values of e that are entirely ordinary:

Judge flip rate e	Disagreement with your labels	Attenuation factor	A true 5-point gap measures as
0.00	0%	1.00	5.0 points
0.10	10%	0.80	4.0 points
0.15	15%	0.70	3.5 points
0.20	20%	0.60	3.0 points
0.25	25%	0.50	2.5 points

I checked the algebra against simulation rather than trusting it: 6,000 simulated eval runs of 1,600 examples each, true rates 0.70 and 0.75, e = 0.15, mean observed gap 3.5 points against a predicted 3.5. Note what that does and does not establish. The bias is the same at every eval-set size, so it confirms the algebra and says nothing about how many examples you need. That question comes next, and it is worse than the table suggests.

It costs you twice, and the second cost is a noise term with two sources

Shrinking the effect is the visible cost. The second depends on which test you are running, and I had it wrong at first.

The intuitive story is that noise pushes both observed rates toward 0.5, since p_observed - 0.5 = (1 - 2e)(p - 0.5), and 0.5 is where per-example variance is greatest. That is true and it is the right picture to hold. But it is the noise term of an unpaired comparison, and unpaired is the wrong test here. Eval examples are not independent draws: some prompts are hard for every system, that shared difficulty correlates the two arms, and I have argued before that comparing two runs by their average pass rate is the wrong test for exactly this reason.

So what does judge noise do to a paired comparison? The arithmetic is prettier than I expected. Write the reported label as A' = A + F_A(1 - 2A), with F_A a Bernoulli flip at rate e independent of everything else. Expand E[A'B'] and E[A']E[B'] and the cross terms are identical, so they cancel:

Cov(A', B') = (1 - 2e)^2 * Cov(A, B)

The factor that attenuates your effect appears squared in the covariance between arms. That is an identity rather than a simulation result, and the ratio does not depend on how correlated your arms were to begin with. I checked it at three different difficulty spreads and got covariance ratios of 0.495, 0.490 and 0.493 against a predicted (1 - 2e)^2 = 0.490, with the correlation itself falling from 0.27 to 0.12 in the middle case.

It is worth saying up front what that identity is for, because it does more work later than it looks like it is doing here. Pairing helps you only through the covariance, so a judge flipping labels at rate e keeps exactly (1 - 2e)^2 of whatever pairing was buying you. That one fact is why every number in this post moves when your examples get more or less uniform in difficulty, and it is the last section of this piece.

Which is why there are two costs on the statistic and not three. McNemar's test looks only at the discordant pairs, so its noise term is the discordant-pair rate, and that rate is

p_discordant = p_A + p_B - 2 p_A p_B - 2 Cov(A, B)

so judge noise inflates it from both directions at once: the marginal rates slide toward 0.5, and the covariance is destroyed. I decomposed it one factor at a time, in both orders, and they come out close to even: of the total inflation, about 48 percent is the marginal shift and about 52 percent is the covariance loss. Which means the "toward 0.5" intuition is not superseded by the covariance identity, it is one of two roughly equal ingredients in the same noise term. On my model that rate went from 0.294 to 0.400 at e = 0.15, and the whole effect on the test statistic is:

z' / z = (1 - 2e) * sqrt(p_discordant / p_discordant')
       = 0.70 * sqrt(0.294 / 0.400)  =  0.60

Two factors on the statistic: the effect shrinks, and the discordance inflates. What I got wrong at first was adding a third, by counting the marginal-variance story separately from the discordance it is half responsible for.

So here is the power calculation for the design this actually is: a per-example difficulty term, marginal pass rates calibrated to 0.70 and 0.75 (0.699 and 0.749 after calibration), McNemar's exact test on the discordant pairs, 6,000 simulated runs per cell.

Eval set size	Power, perfect judge	Power, judge at e = 0.15
200	0.21	0.10
400	0.41	0.19
800	0.72	0.32
1,600	0.95	0.58
3,200	1.00	0.87

Those cells carry about a point of Monte Carlo error, so read them to two figures and no further. To reach 80 percent power on that effect I needed roughly 1,000 examples with a perfect judge and roughly 2,750 with the noisy one, about 2.75 times the eval set to buy back what a 15 percent flip rate cost.

At n = 800, a size that felt generous to me for years, that puts a real 5-point improvement below a one-in-three chance of surfacing. Most teams who try that change will shelve it on the evidence.

One caveat, because that table has a knob in it I nearly published without stating. Everything above assumes one level of between-arm correlation, which is to say one answer to how much your examples vary in difficulty. Change it and the whole table moves: at n = 800 a perfect judge's power runs from 0.59 at uncorrelated arms to 0.90 at a correlation of 0.57.

I expected the ratio between the two columns to be the stable thing. It is not, and it moves in the direction that makes this worse rather than better. Eval-set sizes for 80 percent power, across the same range:

Between-arm correlation	Perfect judge	Judge at e = 0.15	Ratio
0.00	1,254	2,888	2.3x
0.10	1,129	2,763	2.5x
0.27	918	2,551	2.8x
0.45	693	2,327	3.4x
0.57	544	2,178	4.0x

Those are closed-form rather than simulated, so the trend is exact rather than approximate. The first row is worth checking by hand, because at zero correlation pairing buys nothing and McNemar reduces to the unpaired test: the ratio there is just the variance inflation times the attenuation term, 1.13 * 2.04 = 2.3, which is the two factors from the previous section multiplied together.

So the more your examples differ in difficulty, the more a noisy judge costs you, and the covariance identity says why: pairing pays off precisely when the arms are correlated, and (1 - 2e)^2 is the fraction of that payoff you keep. At a correlation of 0.57 a perfect judge clears 80 percent power on about 540 examples and a judge flipping 15 percent of labels needs almost exactly four times as many. The 2.75x I quoted above is the middle of that range, not a constant. Measure your own arm correlation, which you can do directly from a paired run, and read the ratio off the row you land on.

What I do about it

Estimate e, and be honest about what you are estimating. You probably have a human-labelled set from validating the judge. What that gives you is the disagreement between two fallible instruments, which is a different quantity from e. If your humans flip at rate e_h and the judge at e_j, then for the observed disagreement d:

1 - 2d = (1 - 2*e_h) * (1 - 2*e_j)

I confirmed that on four million simulated labels and it holds to four decimal places. Which means dividing your gap by (1 - 2d) over-corrects by exactly 1/(1 - 2*e_h). Humans flipping 5 percent of labels inflates your effect by 11 percent; at 10 percent human error you are inflating by 25 percent. Two annotators agreeing does not make them right, which I have written about before, and it is precisely the assumption this correction smuggles in. So treat the corrected number as an upper bound unless you truly believe your labels are error-free.

Report the disattenuated effect only next to the raw one. Divide the observed gap by (1 - 2e): an observed 3.5 points at e = 0.15 implies a true gap near 5. The uncertainty is computable and belongs in the report. The standard error on (1 - 2e) from m human labels is 2*sqrt(e(1-e)/m), so at e = 0.15 with 100 labels the correction factor alone carries about 20 percent relative error, putting your corrected 3.5 points somewhere near 4.2 to 6.3 before any sampling error in the gap itself. At 500 labels that tightens to roughly 4.6 to 5.5.

Spearman's own worked example in the 1904 paper takes an observed 0.38 and corrects it to approximately 1, which he calls absolute and complete. The abuse is as old as the method.

Check whether you have noise or bias, and use the right null. Attenuation assumes the judge's errors do not depend on what you are measuring. If it prefers longer answers and model B is more verbose, that is a confound with a direction, and dividing by reliability amplifies it. I will not re-derive that argument here beyond the one thing people get wrong in code: the expected split of the disagreements is not even. Under symmetric noise E[flip_up] = (1-p)*e*n and E[flip_down] = p*e*n, so the expected ratio is (1-p) : p. At a pass rate of 0.70, perfectly symmetric noise produces a 30:70 split, and any test against 50/50 will call that lopsided and wrongly tell you not to correct. I ran exactly that mistake against simulated symmetric noise at pass rates from 0.5 to 0.9 and it produced a false "do not correct" verdict at every rate above 0.5.

import numpy as np
from scipy.stats import binomtest

def diagnose(human, judge, alpha=0.01):
    """human, judge: equal-length boolean arrays of pass/fail labels.
    Returns e_hat, its standard error, and whether the disagreement is
    consistent with symmetric noise (attenuation) rather than a directional bias."""
    human, judge = np.asarray(human, bool), np.asarray(judge, bool)
    m = human.size
    flip_up   = int((~human & judge).sum())      # human fail, judge pass
    flip_down = int((human & ~judge).sum())      # human pass, judge fail
    e_hat = (flip_up + flip_down) / m
    se_e  = np.sqrt(e_hat * (1 - e_hat) / m)

    # The null is the HUMAN FAIL RATE, not 0.5: symmetric noise flips
    # (1-p) of the failures up and p of the passes down.
    n_disagree = flip_up + flip_down
    pval = binomtest(flip_up, n_disagree, 1 - human.mean()).pvalue if n_disagree else 1.0
    return {"e_hat": e_hat, "se_e": se_e, "attenuation_ok": pval > alpha, "p_value": pval}

rng   = np.random.default_rng(0)
truth = rng.random(20_000) < 0.70                          # symmetric noise, e = 0.15
judge = np.where(rng.random(20_000) < 0.15, ~truth, truth)
print(diagnose(truth, judge))    # e_hat ~ 0.15, attenuation_ok True

Do not assume cutting e beats buying examples. I believed this and it is wrong. Going from e = 0.15 to e = 0.08 is worth about 1.7 times the eval set, and at a fixed budget doubling the examples beat cutting e at every size I tested: at n = 800 that is 0.57 power against 0.49. The honest reason to attack e is that tightening a rubric or splitting a compound criterion is usually cheaper than labelling another 800 cases, and it also recovers some of the pairing benefit noise destroyed. Both interventions are legitimate. And the sample-size requirement scales as the inverse square of (1 - 2e), not of e, which is worth getting right before quoting it at anyone.

Stop reading a flat result as evidence of no effect unless you know your power at the effect size you cared about. A null with unknown power carries no information, and attenuation biases even that absence toward the conclusion you are about to draw.

FAQ

Does this apply to continuous scores, not just pass or fail?
Not in the same way, and this is the part I had wrong until I simulated it. Under classical additive error, where the measured score is the true score plus independent noise, a raw mean difference is not attenuated at all, because the noise has mean zero. What attenuates is a correlation or a standardised effect size, and there the correction is division by the square root of the reliability when the other variable is measured without error, or by the geometric mean of both reliabilities when neither is. The binary case above is not a special case of that. It attenuates the raw gap precisely because a flip is bounded rather than additive, so the expected judge score is 0.64 where the true rate is 0.70. Identify your error process before reaching for a correction.

My judge disagrees with humans 5 percent of the time. Am I fine?
Mostly. The factor is 0.90, so you lose about a tenth of every effect. Worth knowing when you are arguing over a 1-point difference, not worth restructuring around. Confirm the 5 percent was measured on traffic-like cases rather than your cleanest examples.

Can I average several judge calls and skip the maths?
Averaging independent calls reduces the random component of e, so it is a real intervention, and it recovers some of the lost pairing correlation as well. The systematic component is untouched by averaging, and it was a bias rather than attenuation to begin with.

Does a paired test fix it?
No, and that is the finding that surprised me most. Pairing remains the right design and it still helps. But the (1 - 2e) multiplier is a property of the instrument rather than of the comparison, so pairing cannot reach it, and per-judgment noise actively erodes the between-arm correlation pairing depends on.

Were my past null results wrong?
Some, and you can work out which. Take the effect you were hoping for, your e, and your eval-set size, and compute the power you actually had. Any null where power was under about 0.5 tells you nothing in either direction. Three of mine qualified.

Open question

The awkward case is a judge whose error rate depends on what it is reading. Everything above assumes one e applied to both arms. If model B produces output that is easier to grade correctly then e_B < e_A, the arms attenuate by different factors, and the observed gap mixes a real quality difference with a difference in how legibly each model fails. Those two are not separable from the judge's output alone.

Estimating e per model on a small human-labelled slice was my first attempt, and at the sample sizes I could afford the two estimates were too noisy to propagate, which the standard error above explains: separating a 3-point difference in e needs a couple of thousand labels per arm, not the few hundred I assumed when I started. If you have a design that identifies model-dependent judge reliability without a full human pass over both arms, I would like to read it.

Your eval's confidence interval assumes independent examples. Yours are clustered.

Maya Andersson — Tue, 28 Jul 2026 21:24:36 +0000

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't.

I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong.

Why clustering shrinks your real sample size

Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half.

The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately:

Deff = 1 + (m̄ - 1) · ICC

where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is:

n_eff = n / Deff

That is the number of independent examples your clustered set is actually worth.

The number

Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher).

Deff  = 1 + (5 - 1) · 0.3 = 2.2
n_eff = 200 / 2.2 ≈ 91

Your 200-example eval is worth about 91 independent examples. The correct confidence interval is √2.2 ≈ 1.48 times wider than the naive one. So the interval you proudly reported as plus or minus 3.5 points is really plus or minus 5.2. The 4-point improvement you shipped last sprint, the one that "cleared the CI," may not clear the corrected interval at all.

And note the direction of the error. Clustering never makes your interval too wide. It always makes it too narrow. So the bias is always toward false confidence, toward shipping a change that didn't actually beat baseline.

The three clusterings I check for

Source clustering. Multiple examples generated from or grounded in the same document, table, or context. This is the big one for RAG evals, where synthetic questions are minted per-document. ICC here is often 0.2 to 0.4.
Conversation clustering. Multiple turns scored from the same multi-turn session. Turns within a session share the same user, same goal, same accumulated context, so they correlate hard. Scoring 10 turns from 20 conversations is not 200 independent points.
Template clustering. Paraphrases or perturbations of the same underlying case. If you augmented 50 seed cases into 150 by rewording, your n is closer to 50 for the purpose of the interval. The rewordings measure robustness to phrasing, not 150 independent capabilities.

The fix: resample clusters, not rows

The cleanest correction that does not require you to estimate ICC by hand is a cluster bootstrap. Instead of resampling individual examples (which assumes independence, reintroducing the exact bug), you resample whole clusters with replacement.

import numpy as np

def cluster_bootstrap_ci(scores, cluster_ids, B=5000, alpha=0.05, seed=0):
    rng = np.random.default_rng(seed)
    clusters = {}
    for s, c in zip(scores, cluster_ids):
        clusters.setdefault(c, []).append(s)
    keys = list(clusters)
    means = np.empty(B)
    for b in range(B):
        drawn = rng.choice(keys, size=len(keys), replace=True)   # resample CLUSTERS
        pooled = [s for k in drawn for s in clusters[k]]
        means[b] = np.mean(pooled)
    lo, hi = np.percentile(means, [100*alpha/2, 100*(1-alpha/2)])
    return float(lo), float(hi)

Resampling at the cluster level automatically bakes in the correlation structure. You do not have to estimate the ICC. The interval it returns is the honest one, and it will be wider than the binomial interval your CI currently prints. The wider interval is not an artifact of the bootstrap. The correlation in your data was always there, and this is just the first interval that stops ignoring it.

FAQ

What if my examples really are independent? Then the cluster bootstrap and the binomial interval will roughly agree, and you have lost nothing by checking. The check costs a few seconds of compute. Shipping a change that never actually beat baseline costs a lot more than that.

Do I need to know the ICC? No. The cluster bootstrap sidesteps it. You only need the design-effect formula if you want a back-of-envelope sense of how bad the problem is before you code anything.

Does this change my point estimate? No. The mean pass rate is unchanged. Only the interval around it widens. This is purely about how much you should trust the number, not the number itself.

Is this the same as just using a bigger eval set? No, and this is the trap. Adding 50 more questions from the same 8 documents barely helps, because you are adding within-cluster examples that carry little new information. To tighten a clustered interval you need more clusters (more documents, more conversations), not more examples per cluster.

Open question

I do not have a good rule of thumb for the minimum number of clusters before a cluster bootstrap itself becomes unstable. With very few clusters (say under 15) the bootstrap distribution gets lumpy and the interval is itself uncertain. I have been using a soft floor of 20 clusters and falling back to reporting the design-effect-adjusted interval below that, but I have not seen a principled threshold for eval-sized data. If you know the literature here better than I do, point me at it.

An LLM judge is a biased instrument, not a measurement

Maya Andersson — Wed, 22 Jul 2026 19:27:15 +0000

Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Variant B won. Same model, same judge, same test set. The only thing that changed was the order the two answers were pasted into the judge prompt.

That is not a flake. It is position bias, and it is one of three biases that a careful paper documented three years ago. Lianmin Zheng and coauthors measured them in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023, arXiv:2306.05685). Their headline number is genuinely encouraging: a strong judge like GPT-4 agreed with human preferences more than 80 percent of the time, about the same rate at which two humans agree with each other. Their second finding is the one people quote less often. The same judge shows position bias, verbosity bias, and self-enhancement bias.

If you report an eval score without accounting for those, you are reporting a property of the judge as if it were a property of your model.

Bias is not noise, so you cannot average it away

There is a habit, imported from good statistics, of assuming that if a measurement is noisy you take more readings and the noise shrinks. That works for random error. The standard error of a mean falls with the square root of the sample size, so a thousand judge calls do give you a tighter estimate than ten.

The catch is that position, verbosity, and self-preference are not random error. They are systematic. Run the judge a thousand times with the wordier answer always in slot B, and the wordier answer wins a little more often than it should, every time. The bias sits in the expected value, not the variance. More samples estimate the biased number more precisely, so you end up more sure of a number that is off center.

Three specific ways this shows up:

Position. Many judges prefer whichever answer they read first, or second, depending on the model. If your harness always puts the baseline first and the candidate second, every comparison inherits the same tilt.

Verbosity. Judges reward length and apparent thoroughness. A change that only made the output longer can win a bakeoff it did not deserve, and you will ship a prompt that pads.

Self-preference. A judge tends to score outputs from its own model family higher. This quietly contaminates any comparison where the judge is also, directly or by family, one of the contestants.

What the eval tools actually hand you

I use several of these, and the landscape is more alike than the marketing suggests. Almost all of them ship an LLM-as-judge, and most stop there.

DeepEval builds several of its metrics on a judge, G-Eval being a chain-of-thought scorer that asks a model to grade against a rubric. RAGAS scores faithfulness and answer relevancy with a judge as well. OpenAI Evals gives you model-graded templates alongside exact-match checks. Langfuse lets you attach judge evaluators to traced production data, which is useful precisely because it runs on real traffic rather than a frozen set.

A smaller group gives you cheap deterministic scorers you can run first, and treat the judge as the expensive second opinion. promptfoo mixes deterministic assertions (equality, regex, JSON-schema) with model-graded ones in the same test. Future AGI's eval SDK takes the same shape, running local metrics with no network call behind one evaluate() function and adding a judge only when you pass augment=True. Braintrust's autoevals ships both heuristic scorers and LLM-judge scorers. Langfuse and Braintrust, like Future AGI, also attach evaluation to tracing rather than being eval-only, so you can score what actually ran.

None of that is a knock on any of them, and it is not a ranking. Some of them go further than handing you the instrument. Langfuse's score analytics lines human annotations up against judge scores and reports Cohen's kappa, correlation, and percent agreement, with the interpretation bands printed next to the number. Braintrust puts human and automated scores side by side on the same traces so you can watch how often they agree and refine the scorer when they drift. That is real, and you should use it. It is also not the whole job. A tool can compute how often the judge agrees with your labels. It cannot produce the labels for your task, and it cannot decide how much agreement is enough for the call you are about to make. Those two stay with you no matter which tool you pick.

What to measure instead

Treat the judge like any other instrument you did not build. Calibrate it, then use it.

Calibrate against a human-labeled sample. Label a few hundred examples by hand, run the judge on the same set, and compute agreement. Cohen's kappa is the honest version because it discounts the agreement you would get by chance. The calculation is one function call, and as noted a couple of platforms will run it for you; what none of them can outsource is labeling your data and picking the threshold you will accept:

from sklearn.metrics import cohen_kappa_score

# human_labels and judge_labels are lists of the same labels, e.g. "pass"/"fail" or "A"/"B"
kappa = cohen_kappa_score(human_labels, judge_labels)
# < 0.41 is Fair-to-Slight on the Landis and Koch bands (the same ones Langfuse prints);
# there the judge is not yet a stand-in for your reviewers on this task

Swap positions and check consistency. If the winner flips when you swap the order, the score is order-dependent, not a property of the answers. This is a few lines around whatever judge call you already have:

def judge(question, answer_a, answer_b):
    """Return 'A' or 'B'. Wrap your own LLM-as-judge call here."""
    ...

def swap_consistency(question, a, b):
    first  = judge(question, a, b)   # candidate 'a' sits in slot A
    second = judge(question, b, a)   # candidate 'a' sits in slot B
    a_wins_first  = first  == "A"
    a_wins_second = second == "B"
    return a_wins_first == a_wins_second   # True if the verdict held after the swap

# Average this over your eval set. On my sets I start to distrust a judge below
# about 0.9 here: at that point it is grading position as much as content,
# so score both orders and average them.

Triage with something cheap. A local metric or a deterministic check can resolve the clear cases for free, which shrinks how many times you lean on the biased judge at all. Reserve the judge for the genuinely uncertain tail. This is the one place the tool choice helps, because the tools that ship local scorers make the triage a config flag rather than a rewrite.

Report agreement next to the score. "94 percent pass rate" means little on its own. "94 percent pass rate, judge-human kappa 0.71 on a 300-example calibration set, scored in both orders" is a number a reviewer can trust, and it is the same amount of work to print.

None of this makes the judge unbiased. It makes the bias visible and gives you a number for how large it is, which is the most you can honestly claim for an instrument you did not build.

FAQ

GPT-4 agrees with humans more than 80 percent of the time. Is that not good enough?
It depends entirely on what the disagreeing 20 percent correlates with. If those cases were random, you could average them away. They are not random. They line up with length and position, so the error has structure, and structure survives averaging. The aggregate agreement can look fine while every close call tilts the same direction.

Pairwise or pointwise scoring?
Pairwise (A versus B) is usually more stable than asking for an absolute 1-to-10 score, because relative judgments are easier for a model to make consistently. But pairwise is exactly where position bias lives, so if you go pairwise you have to score both orders and average. Pointwise avoids position bias and trades it for scale drift, where the judge's notion of "7 out of 10" wanders between runs.

Do I need human labels forever?
No, you need a calibration sample. Label a few hundred examples once, measure judge-human agreement, and recheck when something changes: a new judge model, a new prompt, a shift in the kind of inputs you see. The labels are a periodic audit, not a standing tax.

Will a bigger judge model fix this?
It raises raw agreement, which is why people reach for the strongest model they can afford. It can also make self-preference worse, because a stronger judge is often from the same family as the strong models you are testing. The cheap defense is to pick a judge from a different family than the models under test.

Open question

The thing I do not have a clean answer to is drift. You calibrate against human labels in July, the provider silently updates the hosted judge model in September, and your kappa is now describing a different instrument than the one you validated. Nobody I know version-pins their judge the way they pin a training dependency, and most hosted judges do not expose a stable version to pin to. So how often is often enough to recalibrate, and should a judge model be treated as a pinned dependency with its own changelog? I have not seen a convention settle, and I would take pointers from anyone who has one that works.

Your eval dashboard has 30 metrics. When one "moves," that is usually arithmetic, not a regression.

Maya Andersson — Tue, 21 Jul 2026 14:39:53 +0000

Here is the ritual. You ship a prompt change, rerun the eval suite, and open the dashboard. Thirty numbers sit there: faithfulness, answer relevance, context precision, toxicity, latency-adjusted quality, and two dozen more. Twenty-nine are flat. One dropped from 0.86 to 0.81 and the cell is red. Someone says "we regressed on groundedness," and the next hour goes to explaining a number that never needed explaining.

I want to make the boring case that most of these red cells are not findings. They are what you get when you run many comparisons and let each one fire on its own. This has a name, the multiple comparisons problem, and it has arithmetic you can do on a napkin.

Claim 1: the false alarm rate is per-metric, and you have a lot of metrics

Set a threshold for "this metric moved" that would fire 5% of the time by chance when nothing actually changed. That is what a two-sided test at alpha = 0.05 means: a 5% false positive rate per test, under the null.

One metric, one test, 5% chance of a false alarm. Fine. But you are not looking at one metric. Assume for a moment the metrics are independent (they are not, and that matters below). The probability that at least one of n metrics throws a false alarm is:

P(at least one false positive) = 1 - (1 - alpha)^n

n = 1:   1 - 0.95^1  = 0.05    (5%)
n = 20:  1 - 0.95^20 = 0.64    (64%)
n = 30:  1 - 0.95^30 = 0.79    (79%)

At 30 metrics, a run where nothing changed has a 79% chance of showing you at least one red cell. The expected count is just n times alpha: 30 x 0.05 = 1.5 false alarms per clean run. So the "one metric moved" you are staring at is, on average, exactly what a no-op change produces. You did not find a regression. You found the metric that lost this round of a lottery you run every deploy.

Claim 2: your metrics are correlated, which does not save you

The obvious objection: eval metrics are not independent, so the formula above is wrong. Correct, it is wrong. It is not wrong in your favor.

Correlation changes the shape of the false-alarm distribution but not its center. The expected number of false positives is n times alpha regardless of dependence, because expectation is linear and does not care whether the tests move together. Correlation mostly changes the variance. When metrics are highly correlated, false alarms clump: a clean run tends to show either zero red cells or several at once, because the correlated metrics fail together. That is worse for interpretation, not better, because a cluster of three red cells looks like a "real pattern" and is the single most convincing way to fool yourself. That cluster is three correlated metrics failing on the same noise, not three independent findings.

So you cannot correlation your way out of this. You have to correct for it.

Claim 3: the fix is a decision made before you look, not a harder stare after

There are two standard corrections, and they answer two different questions.

Bonferroni controls the family-wise error rate, the probability of even one false positive across the whole family. To hold that at 0.05 across 30 metrics, you test each at 0.05 / 30 = 0.00167. It is exact enough (it slightly over-corrects under dependence) and it is strict. Strict is the point when a single false regression would block a release.

Bonferroni is often too strict when you genuinely track many metrics and can tolerate a few false alarms among your flagged ones. That is what the false discovery rate is for. Benjamini and Hochberg's 1995 procedure (Controlling the false discovery rate, Journal of the Royal Statistical Society Series B) controls the expected proportion of your flagged metrics that are false, rather than the chance of any false flag at all. The procedure is short: sort your n p-values ascending, find the largest k where p(k) <= (k / n) x q for your target rate q, and flag everything up to k. It gives you far more power than Bonferroni while still bounding how much of your red is noise. For a dashboard where you expect a handful of real movements among thirty metrics, FDR is usually the right tool. For a release gate where one false block is expensive, family-wise control is.

Either way, the correction is a rule you commit to before opening the dashboard. The failure mode is picking the correction after you have already seen which cell is red, because by then you are choosing the test that lets you believe what you already decided.

Claim 4: separate the metric you are testing from the metrics you are watching

The cleaner fix is upstream of any correction. Most eval suites conflate two things:

Confirmatory metrics are the one or two you changed the prompt to improve. You have a directional hypothesis. You test those, at full alpha, and you are allowed to act on them.

Exploratory metrics are the other twenty-eight you keep on the board for monitoring. These do not get to trigger a "we regressed" conversation on a single run. They get watched over multiple runs, and a real regression in one shows up as a trend across several deploys, not a one-time red cell that is gone next Tuesday. A drop that reverts on the next run without any code change was regression to the mean, which is the second-most-common way a dashboard lies to you.

Decide which bucket each metric is in before the run. Two primary metrics tested at 0.05, twenty-eight secondary metrics under an FDR rule or simply held to a "two consecutive runs" bar. Now your false-alarm budget is one or two tests wide, not thirty.

The rule I actually use

Before the run, write down the one or two metrics the change is supposed to move, and the direction. Everything else is monitoring. For the monitoring set, do not react to a single red cell: apply an FDR correction across the set, or require the movement to survive a second run. When you must gate a release on a broad panel, use family-wise control and accept that you will miss small real effects, because at the gate a false block costs more than a missed 0.01.

None of this makes your eval more sensitive. It makes the eval account for how many comparisons you ran before a cell turned red.

FAQ

Does this mean my dashboard is useless?
No. A dashboard is a fine monitoring instrument. It is a bad hypothesis test. Use it to watch trends across runs, not to adjudicate a single deploy. The moment a single-run red cell triggers a decision, you have turned a monitor into a test without paying for the test.

If I only ever look at one metric, do I need any of this?
No. One pre-specified metric, one test, no correction needed. The problem is entirely created by the number of comparisons you let fire. Two or three metrics tied to a hypothesis tell you more than thirty scanned for whatever happened to move.

Bonferroni or Benjamini-Hochberg?
Bonferroni when a single false alarm is expensive, such as a release gate: it controls the chance of any false positive. Benjamini-Hochberg when you track many metrics and can tolerate a known fraction of false flags in exchange for catching more real ones. They optimize different things on purpose.

My eval scores are not p-values, they are just averages. Does this still apply?
Yes, and it is easier to abuse. A raw score delta with no notion of sampling variability is a comparison with an implicit, undocumented threshold. The multiplicity is still there; you have just hidden the alpha. Put a confidence interval on each delta and the correction problem becomes visible again.

Will correcting make me miss real regressions?
It will reduce power, yes. That is the trade. The answer is not to skip the correction, it is to keep the confirmatory set small so each test keeps full power, and let the large monitoring set be governed by trend, not by a single run.

Open question

Family-wise and FDR corrections both assume the family is well defined: you know how many tests you are running. On a live eval dashboard the family is open-ended. You add metrics over months, you rerun after every deploy, and you look at the board on days when nothing shipped. The true number of comparisons is not 30, it is 30 times every run times every glance. I do not have a clean way to define "the family" for a metric board that is queried continuously by a whole team. Sequential testing methods (alpha-spending, always-valid p-values) are the honest direction, but I have not seen them adapted well to eval suites where the metric set itself keeps growing. If you have a working formulation of the family for a continuously-watched dashboard, I would like to see it.

Your eval pass rate is 98 percent. Your confidence interval is probably wrong.

Maya Andersson — Thu, 16 Jul 2026 15:23:29 +0000

TL;DR. Almost every eval harness reports a pass rate with an error bar, and almost every one of those error bars comes from the normal approximation: p̂ plus or minus 1.96 times the square root of p̂(1 - p̂)/n. That formula is taught first, implemented everywhere, and reasonable near a pass rate of 50 percent. It falls apart at the extremes, which is precisely where any model worth shipping lives. At 49 of 50 passing it produces an upper bound of 1.0188, a probability above one. At 50 of 50 it produces the interval [1.0, 1.0], a claim of perfect certainty from fifty observations. Worse than either artifact: when the true pass rate is 98 percent and n is 50, its actual coverage is 63.5 percent. The reframing is that this is a solved problem, and has been since 1927. Invert the score test instead of approximating around the estimate, and you get the Wilson interval, which is one argument change in the library you already have installed.

The regression I shipped because I misread an interval

Three years ago I owned the eval suite for a document extraction pipeline. Fifty held-out documents, hand-labeled, each either extracted correctly or not. We were shipping a prompt change and the numbers looked clean: 49 of 50 passing, and the harness printed a 95 percent confidence interval of [0.941, 1.000].

I read the lower bound and did what most people do with a lower bound. I treated it as the pessimistic case. I told the stakeholder that the worst plausible outcome was about 94 percent, that we would be fine at 94 percent, and we shipped on a Thursday.

Production accuracy landed near 91 percent. That was outside the interval I had quoted, and I spent a week looking for the distribution shift that explained it. There was a small one. It was not the story. The story was that my interval had no business excluding 91 percent in the first place. Running the same 49 of 50 through the Wilson interval returns [0.895, 0.996]. The lower bound is 89.5 percent, not 94.1 percent. Wilson's interval contained the truth. Mine did not, and the gap between the two was not a rounding difference. It was 4.6 percentage points of false confidence, manufactured by a formula that should not have been applied to that data.

The distribution shift got the postmortem. The interval got nothing, because nobody in the room, including me, thought of the error bar as a thing that could itself be wrong. That is the failure mode I want to argue against here, criterion by criterion.

What the Wald interval is, and where it comes from

The estimator is uncontroversial. With k passes out of n cases, p̂ = k/n.

The interval most harnesses report is the Wald interval:

p̂ ± z · sqrt( p̂(1 - p̂) / n )

The derivation is a normal approximation to the binomial, with one extra move that does the damage: the standard error is evaluated at p̂, the number you happened to observe, rather than at p, the parameter you are trying to bracket. That substitution is harmless when p̂ is near 0.5 and n is large. Near the boundaries it is not harmless at all, because the standard error sqrt(p(1-p)/n) is itself a function of p that collapses to zero as p approaches 1. Plug in p̂ = 1.0 and the formula obediently reports that it has no uncertainty.

So the question is not whether the Wald interval is defensible in general. It is whether it is defensible in the regime where eval results actually land. Below are five criteria any interval estimator should satisfy, and how Wald does against each.

Criterion 1: the interval must stay inside the parameter space

A pass rate is a proportion. It lives in [0, 1]. An interval that includes values above 1 is not reporting uncertainty about a proportion, it is reporting that the model has been arithmetically mangled.

At 49 of 50, the Wald interval computes to [0.9412, 1.0188]. The upper bound is a probability of 1.0188.

Here is the part that keeps this from being obvious, and the reason I did not catch it for years. Your library hides it. In statsmodels, the proportion_confint function ends with an explicit clip for exactly two methods:

if method in ["normal", "agresti_coull"]:
    ci_low = np.clip(ci_low, 0, 1)
    ci_upp = np.clip(ci_upp, 0, 1)

So proportion_confint(49, 50, method="normal") returns [0.9412, 1.0000], and the 1.0188 never reaches your logs. The docstring says so plainly, and nobody reads it, because who reads the docstring for a confidence interval.

The clip is cosmetic, and I can show that precisely: clipping the upper bound from 1.0188 down to 1.0 only removes values in the range (1.0, 1.0188] from the interval, and no true proportion can live there. The set of true values the interval covers is identical before and after the clip. Coverage does not move by a single decimal place. What the clip accomplishes is that a number which would have announced the method's failure now looks like an ordinary, tidy upper bound of 1.0.

Criterion 2: the interval must not vanish when the data is unanimous

Run a perfect eval. 50 of 50. The Wald interval is [1.0, 1.0], width zero.

Read that as an epistemic claim and it says: having observed fifty successes, I am now certain, to the exclusion of all alternatives, that this system never fails. Not "very likely above 95 percent". Certain. A zero-width interval assigns probability zero to a true rate of 0.999.

Fifty of fifty is a perfectly ordinary eval outcome. It is also the outcome where the Wald interval fails hardest, and the failure has a sign: it always errs toward overconfidence, never toward caution.

Wilson at 50 of 50 returns [0.9287, 1.0]. Clopper-Pearson returns [0.9289, 1.0]. Both say the sensible thing: fifty consecutive passes is real evidence, it is consistent with a true rate around 93 percent, and you cannot rule out roughly a 1-in-14 failure rate on this evidence.

This connects to a heuristic some readers will know, the "rule of three": with zero failures in n trials, the upper bound on the failure rate is about 3/n. At n = 50 that gives 6 percent, implying a lower bound near 94 percent, which is close to Wilson's 92.9 percent but not equal to it. The reason is that the rule of three is the one-sided 95 percent bound. I checked: 1 - 0.05^(1/50) = 0.0582, and 3/50 = 0.06. The two-sided version needs 0.025 in each tail, which gives 1 - 0.025^(1/50) = 0.0711, closer to 3.7/n. Useful heuristic, commonly misquoted by half a tail.

Criterion 3: nominal coverage must resemble actual coverage

This is the criterion that matters, and the one nobody checks.

A 95 percent confidence interval makes a frequentist promise: across repeated experiments, the interval contains the true parameter 95 percent of the time. That promise is testable. The binomial has finitely many outcomes, so you do not even need simulation. Enumerate every k from 0 to n, ask whether the interval built from that k contains the true p, and weight by the binomial probability of observing that k. The answer is exact.

I ran that enumeration. At n = 50, nominal 95 percent:

true p	Wald	Wilson	Clopper-Pearson
0.50	0.935	0.935	0.967
0.80	0.938	0.951	0.967
0.90	0.879	0.970	0.970
0.95	0.920	0.962	0.988
0.98	0.635	0.922	0.982
0.99	0.395	0.911	0.986

At a true pass rate of 98 percent, an interval labeled "95 percent confidence" contains the truth 63.5 percent of the time. At 99 percent, it manages 39.5 percent. Those are not slightly optimistic numbers. An interval that misses the parameter more than a third of the time is not delivering what its label promises, and at 99 percent it misses more often than it hits.

Note the shape of the failure. Near p = 0.5 the Wald interval is fine (0.935, close enough to nominal). The degradation is monotone in how good your model is. The better the system under test, the more the interval lies, and it lies in the direction of telling you the system is more reliable than it is. Any team whose models improve over time is walking into this, and the error bar gets quieter about it every quarter.

None of this is news to statisticians. Brown, Cai and DasGupta laid it out in 2001 in Statistical Science, in a paper whose abstract describes the Wald interval's coverage as "erratic" and "chaotic," and states that "common textbook prescriptions regarding its safety are misleading and defective" (Brown, Cai and DasGupta, 2001, Statistical Science 16(2), 101 to 133). They recommend Wilson or the equal-tailed Jeffreys interval for small n, and Agresti-Coull for larger n. The paper is 25 years old. The formula it warns about is still the default in most eval code I read.

Criterion 4: more data must not make the interval worse

The standard defense of Wald is that it is asymptotic, so just use enough data. There are rules of thumb attached: n ≥ 30, or np ≥ 5, or np ≥ 10.

Test the rule. At p = 0.98, the binding constraint is the failure count, n(1 - p), because that is the small one. The np ≥ 5 rule, applied to failures, demands n ≥ 250.

At n = 250 and p = 0.98, Wald coverage is 0.873.

The rule is satisfied, and the interval is still nowhere near 95 percent. So the rule of thumb does not work, which is what "misleading and defective" meant.

It gets less comfortable. Coverage is not even monotone in n. Holding p = 0.98 fixed and computing exact coverage as n grows:

n	Wald coverage
125	0.916
142	0.941
150	0.800
225	0.935
250	0.873

Adding eight test cases, from 142 to 150, drops coverage from 0.941 to 0.800. More data, worse interval. I verified each of these two ways, by exact enumeration over the binomial and by a one-million-replication Monte Carlo, and they agree to three decimals (the Monte Carlo returns 0.9408, 0.8003 and 0.8732 for n = 142, 150 and 250).

The mechanism is not mysterious once you see it. k is discrete. As n changes, the lattice of achievable k values slides relative to the true p, and whether p sits inside the interval built from the most probable k flips on and off. That produces a sawtooth, which is why Brown, Cai and DasGupta call the behavior chaotic rather than merely biased. There is no threshold n above which you are safe, because the function you would be thresholding oscillates.

Which brings me to the honest caveat, and I would rather state it than have it found. Wilson oscillates too. Every interval for a discrete parameter does. Sweeping n from 50 to 600 at p = 0.98, Wilson's coverage ranges from 0.918 to 0.980, while Wald's ranges from 0.635 to 0.951. The difference is that Wilson's oscillation is centered near the nominal level, so its errors go in both directions and stay small. Wald's ceiling across that entire sweep is 0.951. It essentially never over-delivers, and its floor is 0.635. Wilson is not exact. It is well-behaved, which is a different and more achievable property.

Criterion 5: the interval should be asymmetric when the estimate is near a boundary

Wald intervals are symmetric by construction, because the formula is "estimate plus or minus a fixed quantity." Near a boundary, symmetry is the wrong shape.

At 49 of 50, the truth cannot be far above 0.98, since only 1.0 is available up there. It can quite easily be below, because 0.95 and 0.93 and 0.91 all produce 49 of 50 with unremarkable probability. The interval should be lopsided: short on the top, long on the bottom.

Wilson does this automatically. At 49 of 50 it returns [0.8950, 0.9965], which extends 0.085 below the point estimate and 0.0165 above it, roughly five times more room downward than upward. That asymmetry is not a patch bolted onto the formula. It falls out of the derivation, because Wilson inverts the score test: rather than approximating the standard error at p̂, it asks which values of p would fail to be rejected, evaluating the standard error at each candidate p. Solving that quadratic in p gives

center = (p̂ + z²/2n) / (1 + z²/n)

half-width = (z / (1 + z²/n)) · sqrt( p̂(1 - p̂)/n + z²/4n² )

Notice the center is not p̂. It is p̂ pulled toward 0.5, which is why the interval never escapes [0, 1] and never collapses at p̂ = 1: the z²/4n² term inside the square root keeps the width positive even when p̂(1 - p̂) is exactly zero. Wilson published this in 1927 (Wilson, 1927, Journal of the American Statistical Association 22(158), 209 to 212). It predates the eval harness by about ninety years.

The code

Pasteable, and the output below is what it actually prints:

# pip install statsmodels scipy
import numpy as np
from scipy.stats import binom, norm
from statsmodels.stats.proportion import proportion_confint

Z = norm.ppf(0.975)  # 1.959963...


def wald_by_hand(k, n):
    """The textbook normal-approximation interval, unclipped."""
    p = k / n
    se = np.sqrt(p * (1 - p) / n)
    return p - Z * se, p + Z * se


def show(k, n):
    print(f"\n{k}/{n} passing  (point estimate = {k/n:.3f})")
    lo, hi = wald_by_hand(k, n)
    print(f"  Wald, by hand      : [{lo:.4f}, {hi:.4f}]   width={hi-lo:.4f}")
    lo, hi = proportion_confint(k, n, alpha=0.05, method="normal")
    print(f"  Wald, statsmodels  : [{lo:.4f}, {hi:.4f}]   width={hi-lo:.4f}")
    lo, hi = proportion_confint(k, n, alpha=0.05, method="wilson")
    print(f"  Wilson             : [{lo:.4f}, {hi:.4f}]   width={hi-lo:.4f}")
    lo, hi = proportion_confint(k, n, alpha=0.05, method="beta")
    print(f"  Clopper-Pearson    : [{lo:.4f}, {hi:.4f}]   width={hi-lo:.4f}")


show(49, 50)
show(50, 50)


def exact_coverage(n, true_p, method):
    """Actual coverage, computed by enumerating every possible k.
    No simulation: the binomial has finitely many outcomes."""
    total = 0.0
    for k in range(n + 1):
        lo, hi = (wald_by_hand(k, n) if method == "wald"
                  else proportion_confint(k, n, alpha=0.05, method=method))
        if lo <= true_p <= hi:
            total += binom.pmf(k, n, true_p)
    return total


print("\n\nActual coverage of a nominal 95% interval, n=50")
print(f"{'true p':>8} {'Wald':>8} {'Wilson':>8} {'Clopper':>9}")
for p in [0.50, 0.80, 0.90, 0.95, 0.98, 0.99]:
    print(f"{p:>8.2f} {exact_coverage(50, p, 'wald'):>8.3f} "
          f"{exact_coverage(50, p, 'wilson'):>8.3f} "
          f"{exact_coverage(50, p, 'beta'):>9.3f}")

Output:

49/50 passing  (point estimate = 0.980)
  Wald, by hand      : [0.9412, 1.0188]   width=0.0776
  Wald, statsmodels  : [0.9412, 1.0000]   width=0.0588
  Wilson             : [0.8950, 0.9965]   width=0.1014
  Clopper-Pearson    : [0.8935, 0.9995]   width=0.1060

50/50 passing  (point estimate = 1.000)
  Wald, by hand      : [1.0000, 1.0000]   width=0.0000
  Wald, statsmodels  : [1.0000, 1.0000]   width=0.0000
  Wilson             : [0.9287, 1.0000]   width=0.0713
  Clopper-Pearson    : [0.9289, 1.0000]   width=0.0711


Actual coverage of a nominal 95% interval, n=50
  true p     Wald   Wilson   Clopper
    0.50    0.935    0.935     0.967
    0.80    0.938    0.951     0.967
    0.90    0.879    0.970     0.970
    0.95    0.920    0.962     0.988
    0.98    0.635    0.922     0.982
    0.99    0.395    0.911     0.986

Two library notes. In statsmodels, method="beta" is Clopper-Pearson (it is named for the Beta distribution used to compute it, which is a naming choice that has cost me time). In scipy, the same intervals are available as binomtest(k, n).proportion_ci(method="wilson") and method="exact" for Clopper-Pearson. I checked both libraries against each other and against the hand-rolled Wilson formula above; all three agree to machine precision.

What I do now

Wilson is the default. One keyword argument, no new dependency, correct behavior at the boundary, coverage that stays near nominal across the range where eval results actually land.

Clopper-Pearson when someone external is going to rely on the number, or when the cost of overstating reliability is asymmetric (safety filters, anything with a compliance story attached). It is called "exact" because it inverts the binomial test directly rather than approximating, but note from the table that its coverage runs to 0.982 and 0.986 where nominal is 0.95. It is conservative, and its intervals are wider than they strictly need to be. That is a deliberate trade, not a free upgrade. Agresti and Coull made this argument in 1998, in a paper titled "Approximate Is Better than 'Exact' for Interval Estimation of Binomial Proportions" (Agresti and Coull, 1998, The American Statistician 52(2), 119 to 126), and their point stands: guaranteed-conservative is not the same as accurate, and if you want coverage near 95 percent rather than above it, the approximate methods do better.

Report the interval, not the point estimate. "98 percent" and "98 percent, 95 percent CI [89.5, 99.7]" are the same measurement, and only one of them communicates that fifty test cases is fifty test cases. If the interval you get is too wide to support the decision you are making, the answer is more test cases, and the width tells you roughly how many. I worked that arithmetic in a separate piece on eval-set size, so I will not repeat it here.

And stop reading the lower bound as the pessimistic case. It is a bound on a plausible range, not a floor. I have made that mistake in production and it cost me a week of looking for a distribution shift that explained three of the nine points I was missing.

FAQ

Does this matter if my pass rate is around 70 percent?

Much less. At n = 50 and p = 0.80, exact Wald coverage is 0.938 against a nominal 0.95, which is a real but tolerable error. At p = 0.50 it is 0.935. The Wald interval was designed for this regime and behaves acceptably in it. The problem is that a 70 percent pass rate is usually a system you are still fixing, not one you are reporting on, and by the time you are writing the number in a document it has moved to 95 percent or higher. The method degrades precisely as your project succeeds, which is an unfortunate property for a measurement tool.

Is Wilson always better than Wald?

Not always, and I would rather not oversell it. At p = 0.50 and n = 50 they return identical coverage of 0.935, because near the center the two derivations nearly coincide. Wilson's advantage appears at the extremes and at small n, and it grows as you move toward the boundary. There is no regime I am aware of where Wilson is meaningfully worse, so "always use Wilson" is a defensible default even though "Wilson is always better" is too strong a claim. The honest version: Wilson is never worse in any way that matters, and is dramatically better exactly where you need it.

Why not just clip the Wald interval to [0, 1] and move on?

Because clipping changes nothing that matters. I showed this above: truncating an upper bound of 1.0188 to 1.0 only excludes values above 1.0 from the interval, and no true proportion lives there. The coverage is bit-for-bit identical before and after. What clipping does is remove the visible symptom, the absurd number that would have prompted you to check the method. statsmodels clips by default for normal and agresti_coull, which means the most common way to compute a Wald interval in Python is also the way that hides its most obvious failure.

What about the 50 of 50 case specifically? Is any interval sensible there?

Wilson and Clopper-Pearson both handle it, returning [0.9287, 1.0] and [0.9289, 1.0] respectively at n = 50. Both are saying the same reasonable thing: fifty consecutive passes is genuine evidence of a high rate, and it is also consistent with a true rate near 93 percent. Wald returns [1.0, 1.0], claiming certainty. If your eval dashboard has ever shown a perfect score with no error bar, or an error bar of zero width, this is why, and it is worth grepping your harness for.

Is Clopper-Pearson the safe choice since it is "exact"?

"Exact" describes the construction, not the coverage. Clopper-Pearson inverts the binomial test directly instead of approximating, which guarantees coverage of at least 95 percent. It overshoots: 0.982 at p = 0.98 and 0.986 at p = 0.99 in the table above, against a nominal 0.95. You are paying for that guarantee with width, and wide intervals have their own cost, since an interval too wide to distinguish two candidate models is not helping you decide anything. Use it when understating reliability is cheaper than overstating it. Otherwise Wilson.

Does the same problem hit the standard error I report on a per-metric basis?

Yes, for anything that is fundamentally a count of successes over trials. Pass rate, exact-match accuracy, tool-call validity, refusal rate, any binary judge verdict aggregated across a test set. If the reported number is k/n and the error bar is p̂ ± z·sqrt(p̂(1-p̂)/n), the analysis here applies unchanged. It does not apply to metrics that are means of continuous scores, like a 1-to-5 rating averaged across cases, which have a different and generally better-behaved sampling distribution.

What should I actually change in my code tomorrow?

Search for sqrt(p * (1 - p) / n) and for proportion_confint( without an explicit method argument, since statsmodels defaults to method="normal". Both are the Wald interval. Change the default to method="wilson". That is the whole migration. If you have historical eval numbers with published intervals, the point estimates are unaffected and only the intervals move, so a backfill is cheap and mostly widens old lower bounds.

Open question

The thing I have not resolved is what to do when the eval set is not a random sample, which is most of the time.

Everything above assumes n independent Bernoulli draws from a fixed distribution. Real eval sets are curated. We add cases because they broke something, we keep cases because they are hard, we group cases by document or by conversation so that the units are correlated rather than independent. Under curation the binomial model is the wrong likelihood, and an interval derived from it, Wilson included, is answering a question about a population that does not exist. Wilson gives you a correct interval for the wrong model. That is an improvement over Wald, which gives you an incorrect interval for the wrong model, but I notice it is a smaller improvement than this whole essay implies.

I have seen three responses. Treat the eval set as the entire population and report no interval at all, which is honest but gives up on generalization. Cluster-bootstrap at the document or conversation level, which handles the correlation but not the curation. Maintain a separate uncurated random sample purely for estimation, which is correct and which I have never seen anyone actually staff.

I do not have a defensible fourth answer. If you have shipped one, I would like to read it.

Comparing Two Eval Runs by Their Average Pass Rate Is the Wrong Test

Maya Andersson — Tue, 14 Jul 2026 16:54:09 +0000

TL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B passes 74.0 percent, and you conclude B is better. That reasoning throws away the one fact that matters most: both systems answered the same questions, so their per-item outcomes are correlated, not independent. Reading two separate averages (or eyeballing whether their confidence intervals overlap) is the wrong test for a same-items design. The fix is to pair the outcomes per item and test the difference directly. For pass/fail, that is McNemar's test on the items where the two runs disagree. For graded scores, it is a bootstrap over the per-item deltas. Report an effect size and a confidence interval on the delta, not two averages sitting next to each other.

I have shipped a regression because two dashboard numbers looked close enough. The new prompt scored 68.9 percent, the old one 69.7 percent, and I called it noise and moved on. It was not noise. The new prompt was quietly worse on a specific slice, and pairing the runs would have shown me a tight interval that sat entirely below zero. This has practical stakes, not just cosmetic ones: the unpaired reading can hide a regression that the paired reading would surface.

The rest of this post is one claim, argued criterion by criterion: when two eval runs share the same items, you owe them a paired analysis, and the paired analysis is not hard.

The setup, and why it looks reasonable

Here is the pattern I see in almost every eval report. There is a table with two rows. Row one is the baseline, row two is the candidate, and each row has a single number: mean pass rate, or mean rubric score, or mean judge score. Sometimes there is a 95 percent confidence interval on each row, drawn as a little error bar. The reader compares the two numbers, glances at whether the error bars overlap, and makes a call.

The instinct is not stupid. A mean is a legitimate summary, and an error bar is more honest than a bare point. The problem is narrower and more specific: the two error bars in that picture are each computed as if that run stood alone, and the comparison you actually care about (is B better than A) is a comparison the picture does not draw. The uncertainty you need is the uncertainty of the difference, and the difference has its own variance that depends on how the two runs move together across items.

That last phrase is the whole argument, so here it is in concrete terms.

Criterion 1: Same items means paired data, not two independent samples

Some prompts in your eval set are simply harder than others. A gnarly multi-hop question, an ambiguous instruction, a long context that buries the answer. Both A and B face that same hard prompt. When A fails it, B often fails it too, because the difficulty is a property of the item, shared by both systems.

Statistically, that shared difficulty induces a positive correlation between A's per-item outcome and B's per-item outcome. And the variance of a difference is not the sum of the variances when the two things are correlated:

Var(A - B) = Var(A) + Var(B) - 2 Cov(A, B).

When Cov(A, B) is positive (and with a shared test set it usually is), the variance of the delta is smaller than what you would get by treating the runs as independent. Two consequences follow, and they point in opposite directions depending on which mistake you make.

If you run an unpaired two-proportion test (the kind that assumes independence), you plug in a standard error that ignores that covariance term. You typically overestimate the uncertainty of the difference, which makes you underpowered: you fail to detect improvements that are actually there. If instead you eyeball two marginal confidence intervals and check for overlap, you hit a separate and well-documented trap. Two intervals can overlap while the paired difference is comfortably significant, because the overlap of marginal intervals is not the same question as whether the interval on the difference excludes zero.

This is not a niche observation. Dror, Baumer, Shlomov, and Reichart lay it out for language work in "The Hitchhiker's Guide to Testing Statistical Significance in NLP" (ACL 2018): the structure of your data determines which test is valid, and a shared test set produces dependent measurements that a naive test mishandles. Dietterich made the same point a generation earlier for classifiers in "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms" (Neural Computation, 1998), where he recommends tests that respect the paired structure of predictions on a single held-out set and warns against procedures that quietly assume independence.

So the first criterion is a modeling decision rather than a choice of statistic: same items, therefore paired. Everything below follows from taking that seriously.

Criterion 2: For pass/fail, McNemar's test looks only at the disagreements

When each item is a binary pass or fail, pairing the two runs sorts every item into one of four buckets:

both pass
both fail
A passes, B fails (call it a_only)
B passes, A fails (call it b_only)

The items where both systems agree carry no information about which system is better. They cancel. All the signal lives in the discordant pairs, the items where exactly one of the two got it right. McNemar's test (McNemar, "Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages," Psychometrika, 1947) asks a single question: among the discordant items, is the split between a_only and b_only further from 50/50 than chance would produce?

The exact version is a binomial test on the discordant count. The large-sample version is a chi-square statistic with a continuity correction, (|b - c| - 1)^2 / (b + c), on one degree of freedom. For the sample sizes typical of an eval set, I reach for the exact binomial and stop thinking about it.

import numpy as np
from scipy.stats import binomtest

rng = np.random.default_rng(0)
n = 500

# Some prompts are just harder for every system. That shared difficulty is
# exactly why the two runs are correlated, and exactly why you must pair.
difficulty = rng.beta(2, 2, size=n)
p_a = np.clip(0.45 + 0.40 * (1 - difficulty), 0, 1)   # system A
p_b = np.clip(0.50 + 0.40 * (1 - difficulty), 0, 1)   # system B, a bit stronger
pass_a = rng.random(n) < p_a
pass_b = rng.random(n) < p_b

a_only = int(np.sum(pass_a & ~pass_b))   # A right, B wrong
b_only = int(np.sum(pass_b & ~pass_a))   # B right, A wrong
n_disc = a_only + b_only

print(f"A={pass_a.mean():.3f}  B={pass_b.mean():.3f}  discordant: {a_only} vs {b_only}")
if n_disc == 0:
    print("no disagreements: McNemar is undefined, there is nothing to weigh")
else:
    p = binomtest(b_only, n_disc, 0.5, alternative="two-sided").pvalue
    print(f"exact McNemar p = {p:.4f}")

Notice the guard on n_disc == 0. It is not decoration. If your two runs are very similar (a small prompt tweak, a temperature change), you can land with almost no discordant items, and both the chi-square and the exact test degrade or become undefined. The honest reading in that case is not "p is large, no difference" but "I do not have enough disagreements to say anything," which is a different sentence and a cue to collect more items.

To make the reframe concrete, take an illustrative outcome from a run like the one above. Suppose across 500 items you see 40 items where only A passed and 53 where only B passed, with 407 concordant. The marginal gap is (53 - 40) / 500 = 2.6 points, which matches a 71.4 versus 74.0 headline. But McNemar is weighing 53 against 40 out of 93 discordant items, and the exact binomial p for that split is about 0.21 (illustrative). That 2.6-point gap is inside the range you would see from shuffling which system got the coin flip on the contested items. The averages made it look decided. The paired test says wait.

Criterion 3: For graded scores, bootstrap the per-item deltas

Plenty of evals do not produce a clean pass or fail. You get a rubric score in [0, 1], a 1-to-5 judge rating, a similarity score, a latency. The same logic holds, and the mechanical move is the same: form the per-item difference first, then reason about the distribution of those differences.

The paired bootstrap does this without assuming the deltas are normal. You compute diff_i = score_B(i) - score_A(i) for each item, then resample those paired differences with replacement many times, recomputing the mean each time, and read a confidence interval off the resampled means. The word "paired" is doing real work: you resample the differences, not the two score columns independently. Resampling the columns separately would break the pairing and reintroduce the very independence assumption you are trying to avoid.

import numpy as np

def paired_bootstrap_delta(scores_a, scores_b, n_boot=10_000, alpha=0.05, seed=1):
    rng = np.random.default_rng(seed)
    diffs = np.asarray(scores_b) - np.asarray(scores_a)   # pair FIRST
    n = len(diffs)
    if n == 0:
        raise ValueError("no items to compare")
    idx = rng.integers(0, n, size=(n_boot, n))            # resample the PAIRS
    boot = diffs[idx].mean(axis=1)
    lo, hi = np.quantile(boot, [alpha / 2, 1 - alpha / 2])
    return float(diffs.mean()), (float(lo), float(hi))

# Illustrative graded scores in [0, 1] on the same 400 items.
rng = np.random.default_rng(7)
difficulty = rng.beta(2, 2, 400)
scores_a = np.clip(rng.normal(0.70 - 0.20 * difficulty, 0.10), 0, 1)
scores_b = np.clip(scores_a + rng.normal(0.02, 0.08, 400), 0, 1)   # correlated with A

delta, (lo, hi) = paired_bootstrap_delta(scores_a, scores_b)
print(f"mean delta (B - A) = {delta:.3f}, 95% CI = [{lo:.3f}, {hi:.3f}]")

Bootstrap resampling of a single test set is not something I am inventing here. Koehn introduced it to machine translation evaluation in "Statistical Significance Tests for Machine Translation Evaluation" (EMNLP 2004), precisely to attach a confidence interval to a metric computed on one fixed test set, and the general method traces to Efron and Tibshirani's "An Introduction to the Bootstrap" (1993). The one discipline to keep is the pairing. If your items come in natural groups (several questions drawn from the same source document, or several turns from the same conversation), even the paired item bootstrap can understate uncertainty, and you want to resample at the level of the group. More on that in the open question.

Criterion 4: Report an effect size and a CI on the delta, not a bare p-value

Suppose the paired test comes back at p = 0.03. You still do not know whether B is better by half a point or by twelve. A p-value answers "could this be zero," not "how big is it and does the size matter to anyone." On a 5,000-item eval, a 0.3-point improvement can clear significance while being operationally meaningless. On a 120-item eval, a genuinely useful 4-point gain can miss the 0.05 line entirely.

So report the delta and its interval as the headline, and treat the p-value as secondary. For the binary case, the difference in paired proportions is (b - c) / n, and a standard error that respects the pairing is

SE = (1 / n) * sqrt(b + c - (b - c)^2 / n),

which for the illustrative 40-versus-53 example gives a 2.6-point delta with a roughly [-1.2, +6.4]-point 95 percent interval. That interval straddles zero, which is the same verdict McNemar gave, but now stated in units a product owner can act on. For the continuous case, the bootstrap interval on the mean delta is already in the right form.

This emphasis is exactly what the American Statistical Association argued for in its 2016 statement on p-values (Wasserstein and Lazar, "The ASA Statement on p-Values: Context, Process, and Purpose," The American Statistician): a p-value does not measure the size of an effect or the importance of a result, and good practice reports estimates with their uncertainty. Dror and colleagues make the same recommendation for language experiments. An effect size with an interval tells you direction, magnitude, and precision in one line. A lone p-value tells you one bit and hides the rest.

Criterion 5: If you compare many metrics at once, correct for it

Modern eval runs are wide. You are not testing one number, you are testing pass rate and faithfulness and a toxicity check and format-adherence and latency and a half dozen rubric dimensions, all at once, all with their own paired test. Each test at the 0.05 level has a 1-in-20 chance of a false alarm under the null. Run twelve of them and the probability of at least one false positive, if nothing truly changed, is 1 - 0.95^12, which is about 0.46. Almost a coin flip that you will "discover" a difference that is not there and chase it for a day.

There are two standard responses. Bonferroni divides your alpha by the number of tests (0.05 / 12 = 0.0042 each), which controls the chance of any false positive but is conservative and will hide real effects. Benjamini and Hochberg's procedure ("Controlling the False Discovery Rate," Journal of the Royal Statistical Society Series B, 1995) instead controls the expected fraction of your flagged results that are false, which keeps more power when several metrics really did move. For eval dashboards, where you would rather not miss a real regression, I default to Benjamini-Hochberg and reserve Bonferroni for the small number of metrics I would gate a release on. Either way, the point stands: the more comparisons you draw from one run, the more of them will look significant by luck, and a wide dashboard without any correction will regularly flag differences that are not real.

When the simple average is fine

I promised a measured claim, so here is the boundary. Comparing two averages is not always wrong, and there are cases where I do exactly that and sleep fine.

If the two runs used different, independently sampled item sets, they are not paired, and an unpaired analysis is the correct one. If the gap is so large that no reasonable interval could touch zero (A at 40 percent, B at 88 percent, on a few hundred items), the paired test will agree with your eyes and you can skip the ceremony for a quick read, then do it properly before you write it down. If you are not making a decision, just watching a number drift over time on a monitoring chart, a smoothed average is a fine early-warning signal and nobody needs a p-value to notice a cliff. And if your eval set is tiny (say under 30 items), no test rescues you; the honest move is to report the raw counts, resist a verdict, and go collect more data.

The through line is the same. The average is a fine description. It becomes the wrong test the moment you use two of them, side by side, on the same items, to decide which system won. That specific move, two averages side by side on the same items to decide which system won, is the one that needs a paired test.

I will also admit what pairing does not fix. It does not repair a biased judge, a leaky eval set, or items that do not represent production. Get a real improvement wrong and no statistic saves you. Pairing only makes sure that, given honest measurements, you read the comparison the measurements actually support.

FAQ

Is a two-proportion z-test fine if my eval set is large enough?
No, and size does not rescue it. The two-proportion z-test assumes the two samples are independent. With a shared test set they are not, because item difficulty is common to both runs. A larger n makes the wrong standard error more precisely wrong, not correct. What large n does buy you is stability for the paired test: McNemar and the paired bootstrap both behave well with more items, and their intervals tighten. So keep growing the eval set, but feed the counts into a paired procedure. The independence assumption is a property of the design, not of the sample size, and you cannot buy your way out of it with more rows.

My two confidence intervals overlap. Doesn't that mean no significant difference?
This is the most common trap in the whole topic. Overlapping marginal intervals do not imply the paired difference is non-significant. The interval you drew on A and the interval you drew on B each describe one run in isolation. The question you care about lives in a third interval, the one on the delta, which depends on how A and B covary across items. Because a shared test set makes them positively correlated, the interval on the difference is often much tighter than the overlap picture suggests. Draw the interval on the delta and check whether it excludes zero. Ignore the overlap of the two marginal bars.

What if I only have the two average pass rates, not the per-item results?
Then you cannot run the correct test, and you should go get the per-item results. This is the practical reason to log outcomes at the item level for every run: the paired analysis needs to know, item by item, whether each system passed. Two summary numbers have already discarded the pairing, and no post-hoc formula reconstructs it. If retrieving the raw outcomes is genuinely impossible, be honest that you can only make an unpaired, underpowered comparison, and treat any close call as undecided rather than shipping on it. Store the per-item pass/fail and scores by default so this never becomes the blocker.

When should I use McNemar's chi-square versus the exact binomial version?
Use the exact binomial test on the discordant count when that count is small, which for eval sets is most of the time. The chi-square form, with the continuity correction (|b - c| - 1)^2 / (b + c), is a large-sample approximation and is fine when the number of discordant pairs is comfortably into the dozens (a common rule of thumb is b + c of at least 25). Below that, the approximation drifts and the exact test is both safer and, in Python, no harder to call. When discordant pairs are near zero, neither version is meaningful; you simply do not have enough disagreements, and the answer is more data, not a smaller p.

How many items do I need for the paired test to detect a real difference?
It depends on the discordance rate, not the total item count, which is the counterintuitive part. Power comes from the items where the two systems disagree, so two nearly identical runs need a large set to accumulate enough discordant pairs, while two clearly different runs reach significance on far fewer items. As a rough planning move, estimate the fraction of items where you expect the runs to differ, multiply by your set size to get expected discordant pairs, and aim for that to be at least in the dozens. If you cannot get there, you are testing a difference too small to resolve at your current scale, which is itself a useful finding.

I track 15 metrics per run. Do I really need a multiple-comparisons correction?
Yes, if you are making decisions on whichever metric lights up. Fifteen independent tests at 0.05 give roughly a 1 - 0.95^15, about 0.54, chance of at least one false positive under the null. That is worse than a coin flip. You do not need to correct metrics you are only monitoring, but any metric that can trigger a decision (block a release, revert a prompt) belongs in a corrected family. I use Benjamini-Hochberg to control the false discovery rate across the dashboard, because it keeps more power than Bonferroni when several metrics genuinely moved, and I reserve the stricter Bonferroni cut for the two or three release-gating metrics where a single false alarm is expensive.

Open question

Pairing solves the correlation between two systems on the same item. It does not, by itself, solve the correlation between items that are not independent of each other.

Real eval sets are full of hidden clusters. Ten questions generated from the same source document. Eight turns sampled from the same conversation. A batch of items authored by the same annotator with the same blind spots. When items cluster like this, the effective number of independent observations is smaller than your row count, and both McNemar and the item-level paired bootstrap will hand you an interval that is too tight, because they treat correlated items as if they were independent draws. You end up overconfident in the opposite direction from where we started.

The candidate fixes are known in name: a cluster bootstrap that resamples whole groups instead of individual items, or a mixed-effects model with a random intercept per cluster. What I do not have is a clean, agreed-on recipe for the messy case where the clustering is partial and unlabeled, where some items share a document and others do not, and where nobody logged the grouping at eval-authoring time. How much does ignoring soft clustering actually inflate false positives on a typical agent eval, and is the cluster bootstrap worth the complexity for sets in the low thousands?

I have opinions and no proof. If you have run this comparison on your own evals, with the grouping tracked and the intervals computed both ways, I would genuinely like to see the numbers. That is the next thing I want to measure, and I would rather learn it from your data than guess.

The regression your eval set is too small to catch

Maya Andersson — Fri, 10 Jul 2026 16:53:27 +0000

TL;DR. To catch a drop from a 0.90 pass rate to 0.85 at 80% power (one-sided, alpha 0.05), you need about 253 examples. A 50-example set has roughly 35% power, so it misses that regression about two times in three. The move that matters is not "collect more data" as a slogan. Size the set to the effect you actually care about, report a confidence interval on each run instead of a bare point delta, and prefer per-criterion binary labels over vague graded scores.

The two-point win that wasn't

A few months ago I sat in a review where a team was pleased with itself. A new prompt had moved their eval pass rate from 87.5 percent to 90 percent, and someone had already written "prompt v3: +2pp" in the changelog. The eval set had forty examples.

Here is what those points were made of. On forty examples every result is a multiple of 2.5 points, so the pass rate can only land on 35 out of 40, or 36, or 37, with nothing in between. The move from 87.5 to 90 was 35 correct becoming 36 correct. One example. A single test case that used to fail now passed, and it could flip back next week when the decoding lands differently.

I asked the obvious question. If we reran the old prompt three more times, would it always score 35 out of 40? Nobody knew, because nobody had rerun it. The honest summary of that meeting is that we had watched one example change its mind, and we had written it into the changelog as progress.

This is not a story about one careless team. It is the default failure mode of eval-driven development. We compare two numbers, see a gap, and our brains supply a cause. The possibility we skip is that the gap sits entirely inside the noise.

Power is the number nobody computes

The quantity that decides whether your eval can see a regression is statistical power: the probability that your test declares a difference when a real difference of a given size exists. Power depends on three things. The effect size you want to catch, the baseline rate, and the number of examples. Most teams pick the effect size implicitly ("a five-point drop would matter"), never write down the baseline variance, and let the sample size be whatever happened to be sitting in the folder.

Jacob Cohen spent a career arguing that this is backwards, most durably in "Statistical Power Analysis for the Behavioral Sciences" (2nd ed., 1988). The discipline he asked for is boring and effective: decide the smallest effect that would change a decision, then size the study so you can actually see it. An eval set is a study. The same arithmetic applies, whether the outcome is a clinical result or a pass or fail from a grader.

For a pass rate the math is the two-proportion normal approximation. You have a baseline rate p0, a rate you would not want to miss p1, and you solve for the n that gives you, say, 80 percent power. For the case that comes up constantly, catching a slide from 0.90 to 0.85, the answer is about 253 examples. Not forty. Not fifty.

Run the numbers yourself

Here is the whole calculation in standard-library Python. No dependencies, so you can paste it into a scratch file and swap in your own baseline and the drop you care about.

import math

def _phi(x):  # standard normal CDF
    return 0.5 * (1.0 + math.erf(x / math.sqrt(2.0)))

def _z(p):  # inverse normal CDF (Acklam)
    a = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
         1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00]
    b = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
         6.680131188771972e+01, -1.328068155288572e+01]
    c = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
         -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00]
    d = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
         3.754408661907416e+00]
    plow, phigh = 0.02425, 1 - 0.02425
    if p < plow:
        q = math.sqrt(-2 * math.log(p))
        return (((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) / ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1)
    if p <= phigh:
        q = p - 0.5; r = q*q
        return (((((a[0]*r+a[1])*r+a[2])*r+a[3])*r+a[4])*r+a[5])*q / (((((b[0]*r+b[1])*r+b[2])*r+b[3])*r+b[4])*r+1)
    q = math.sqrt(-2 * math.log(1 - p))
    return -(((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) / ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1)

def power_one_proportion(n, p0, p1, alpha=0.05):  # one-sided
    z_a = _z(1 - alpha)
    return _phi((abs(p0 - p1) * math.sqrt(n) - z_a * math.sqrt(p0*(1-p0))) / math.sqrt(p1*(1-p1)))

def n_for_power(p0, p1, power=0.80, alpha=0.05):
    z_a, z_b = _z(1 - alpha), _z(power)
    return math.ceil(((z_a*math.sqrt(p0*(1-p0)) + z_b*math.sqrt(p1*(1-p1))) / abs(p0-p1))**2)

p0, p1 = 0.90, 0.85  # detect a 5-point drop in pass rate
for n in (50, 100, 250):
    print(f"n={n:4}  power to catch {p0}->{p1} = {power_one_proportion(n, p0, p1):.2f}")
print("n needed for 80% power:", n_for_power(p0, p1))

n=  50  power to catch 0.9->0.85 = 0.35
n= 100  power to catch 0.9->0.85 = 0.51
n= 250  power to catch 0.9->0.85 = 0.80
n needed for 80% power: 253

Read it slowly. A fifty-example set has about 35 percent power against a five-point drop, which means it misses that regression roughly two times in three. At a hundred examples you reach 51 percent, a coin flip. You do not clear 80 percent until about 250, and the closed-form requirement is 253.

Two caveats carry weight here. This is a normal approximation and a one-sided test, appropriate when you only care about catching a drop rather than an improvement. And it describes a single run measured against a fixed baseline. If you evaluate the same items before and after (a paired design), the right test is McNemar on the items that flipped, and it needs fewer examples because it cancels the item-to-item difficulty that otherwise inflates your variance. If instead you compare two independent runs, that is a two-sample problem, and you need roughly twice as many per run. The forty-example set was never in the conversation.

When more labels are not an option

"Collect more data" is easy to say and expensive to do, because the labels are the cost, not the inputs. Three things help when you are genuinely stuck at a small n.

Report an interval, not a point. Every pass rate is an estimate with a standard error, and near the boundaries where eval scores live (roughly 0.85 to 0.97) the ordinary Wald interval behaves badly. Use the Wilson score interval instead, from Wilson, E. B. (1927), "Probable inference, the law of succession, and statistical inference," JASA. For 36 out of 40, the Wilson 95 percent interval runs from about 0.77 to 0.96. Print that next to the number. An interval that wide makes the two-point win argue against itself, and nobody has to say a thing.

Prefer per-criterion binary labels over graded scores. A vague one-to-five rubric hides two problems: raters disagree about whether an answer is a three or a four, and that disagreement is pure measurement noise that widens every interval you compute. Splitting the judgment into specific binary criteria (did it cite a source, did it follow the format, did it answer the question) is easier to label reliably, and each example then yields several outcomes instead of one. The honest caveat is that criteria inside one example are correlated, so k criteria are not k independent examples. You still come out ahead, because the reliability gain is real and the correlated-labels problem is smaller than the rater-noise problem it replaces.

Fix the effect size before the run, not after. Decide in advance the smallest regression you care about, size the set to it, and write the decision rule down. Five points at 80 percent power means 253 examples. If you can only get 80, then say plainly that you are powered to catch about a ten-point drop and nothing smaller, and stop reading two-point moves as signal. A known blind spot is a manageable risk. An unknown one ships regressions to users.

The lesson I keep relearning is that eval sets fail quietly. A set that is too small does not error out. It returns a number with two decimal places and a confident sign, and that number is noise that reads exactly like a real measurement.

FAQ

Does a bigger or better model change the sample size?
Only through its pass rate. The arithmetic depends on the effect size and the baseline rate, not on which model produced the answers. A stronger model with a higher baseline actually needs fewer examples to catch the same absolute drop, because the variance p(1-p) shrinks as the rate approaches 1. Catching a five-point slide from 0.97 costs far fewer labels than the same slide from 0.90.

What if my eval is paired, the same items scored under both versions?
Then use McNemar's test on the discordant pairs, the items that passed under one version and failed under the other. It ignores the items that agree, and it needs fewer examples than the independent-samples formula because it removes item difficulty from the variance. The catch is that if almost nothing flips, you have almost no information no matter how many items you scored. Paired designs concentrate all the signal in the disagreements.

One-sided or two-sided?
One-sided is right when you will only act on a drop, which is the regression-detection case. If you will act on a move in either direction, use two-sided and expect to need somewhat more, since the critical value rises from 1.645 to 1.96. Do not reach for one-sided just to shrink the sample size on paper.

Open question

The clean part of this is the single-metric, binary-outcome case. The messy part is the multi-criterion eval, where each example carries five or ten correlated binary judgments and I want one honest power calculation for the whole thing. Treating the criteria as independent overcounts the evidence, and treating each example as one outcome throws information away. The right answer sits somewhere in between, governed by the intra-example correlation, and the design-effect corrections from cluster sampling are the closest tool I know of. The trouble is that they need a correlation estimate you usually do not have until you have already run the eval several times. I do not have a tidy recipe for sizing a correlated multi-criterion eval up front, and non-binary rubric scores make it harder still. If you have solved this in a way that survives contact with real data, I would like to read it.