Originally published on hexisteme notes.
In a public ML benchmark competition, we ran a submission policy: don't spend a submission — a scarce, rate-limited resource — unless a candidate change beats the current best by at least 2 percentage points on a fixed, 40-task local holdout set. It read like ordinary scientific discipline: don't act on noise, require a minimum effect size before you spend something scarce. Over a 125-day eligibility window, at one submission per day, exactly one submission went out — under an explicit calibration exemption written into the rule itself, not because anything cleared the +2pp bar. In 125 days, the threshold itself was cleared zero times.
Not because nothing we tried in that window ever worked. Because the ruler we'd built to decide "did this work" carried more noise than the thing it was supposed to be protecting us from misusing.
The Rule Nobody Could Pass
The policy had three conditions. Two were fine: reproduce across multiple random augmentation seeds, and a full-length dry run finishing without timing out. This piece is about the third — beat the local holdout by at least 2 points — because in 125 days nothing ever actually cleared it. The one submission that did go out was explicitly exempted from it by the rule's own calibration clause, not passed by it.
At first the near-silence looked like it might just be the truth: maybe nothing attempted in that window genuinely worked. That's possible — and it's also exactly what you'd see if the gate itself could not tell a real improvement apart from sampling noise. Those two very different realities produce an identical pass log. The only way to tell them apart is to stop staring at the log and start measuring the instrument that produced it.
Put the Two Numbers Side by Side
Before arguing about whether 2 percentage points was the right bar, we measured something nobody had measured yet: how noisy is the ruler itself, compared to the thing we actually cared about?
| Instrument | Standard deviation | Basis | Role |
|---|---|---|---|
| 40-task local holdout | ±4.79pp | Simulated, from a measured task-churn rate of 11/120 | The gate — decided whether you could submit |
| Larger 120-task local eval set | ±2.77pp | Simulated, from the same measured task-churn rate | Available, but never used as the gate |
| Actual scored result (the real target) | ≈0.2pp | Observed directly — 4 submissions of the same configuration, made by different teams | What you were actually trying to predict — those 4 submissions landed inside a 0.42-point band |
Worth flagging that "Basis" column: the two gate-side numbers aren't repeated measurements, they're a simulation seeded by one observed churn rate; only the target-side number is a direct, repeated observation. An essay arguing you should measure your own instrument's noise before trusting it doesn't get to skip that disclosure about its own numbers.
The gate was carrying close to 24 times more noise than the thing it existed to protect access to. And it wasn't just imprecise in some abstract sense — a companion analysis measured the false-positive rate of the "beat the holdout by 2 points" bar directly and got 39.4%. More than a third of the time, pure sampling noise, with zero real improvement behind it, would clear that bar on its own. The gate wasn't a strict filter guarding a scarce resource. It was closer to a coin flip wearing a lab coat.
This is a completely general shape, not a quirk of one competition: a performance budget enforced on noisy, shared CI runners gating a deploy whose real production regression rate is tiny and well characterized, or a flaky pre-merge suite blocking merges into a service whose actual incident rate barely moves release over release. In every case one diagnostic question settles the argument before you even reach threshold values: whose standard deviation is bigger, the gate's or the target's?
A Zero-Pass Log Is a Symptom, Not a Safety Certificate
It's tempting to read "the threshold was cleared zero times in 125 days" as proof the gate worked — screening out everything that wasn't good enough, letting nothing through that hadn't earned it. That reading doesn't survive contact with the numbers above. When a gate's own noise floor is several times larger than the effect size it's supposed to detect, a zero-pass log is exactly what you'd see whether the true rate of real improvements underneath was high or low. The gate cannot distinguish those two worlds, so its output doesn't carry that information either, no matter how strict it looks from the outside.
The general version: a team that points to "our gate rejects 95% of what comes through it" as evidence of rigor is citing a statistic about the gate, not about what it screened — unless it has separately confirmed the gate's noise floor sits below the effect size that matters. Otherwise a zero-pass (or near-zero-pass) log and a well-calibrated strict gate look identical from the outside, and only one of them is doing anything useful.
The Instinctive Fix Is Usually the Wrong One
Once a gate looks untrustworthy, the natural move is to make the gate itself more precise: a bigger holdout, more seeds, a tighter statistical test. That instinct is only correct if the resource being protected is the noisier side of the comparison. Here it was the reverse — the thing behind the gate, the actual scored result, was already dramatically more precise than the gate itself. Spending effort shrinking the local holdout's noise further would have meant polishing the wrong ruler, like recalibrating a bathroom scale to match a lab balance you already have standing right next to it, instead of just stepping on the lab balance.
The general rule: only invest in tightening a gate when the resource on the other side is itself the noisier one. If the target signal is already more precise than your proxy, refining the proxy changes nothing; you still have a cheap, noisy instrument standing in front of an expensive, precise one. The real question isn't how to make the proxy better — it's when you're allowed to consult the precise instrument directly.
Delete the Threshold, Not the Discipline
The other tempting move runs the opposite direction: since the gate clearly doesn't work, delete it — submit whenever you want. That's wrong too, for a reason that has nothing to do with noise. The gate existed to block a specific, real failure mode: spending a scarce resource by trying many variants and only reporting the one that happened to score best. That failure mode — call it fishing, or informal multiple testing — is just as real with a perfectly precise proxy as with a noisy one. Delete the gate and it comes straight back.
The old rule had quietly conflated two different jobs under one number: a statistical bar on a noisy proxy metric, and a discipline requirement against fishing. Only the first one was broken. So we deleted the first and kept, then sharpened, the second.
In place of the deleted statistical bar, we added a process requirement: state, in one sentence, the specific hypothesis this attempt tests, and why your cheap local measurement is structurally unable to resolve it — not "noisy this time," but incapable by design. Then write down, for every range the result could plausibly land in, what you'll conclude and do next, before you see it. If any range's answer is "we'll decide when we see it," the attempt doesn't qualify.
The anti-fishing rule stayed completely untouched: no resubmitting an unchanged configuration hoping for a better roll; one configuration, one measurement, remeasurement only after a real change. That's what separates selecting on noise — illegitimate, non-transferable — from measuring a hypothesis your cheap instrument structurally cannot answer — legitimate, transferable — while still blocking the failure mode the gate was built for.
Translated to ordinary software practice: you don't scrap "only one canary deployment in flight at a time." You might well scrap "must beat this noisy synthetic load test by X%" and replace it with "state which regression this canary is checking for, and write down what you'll do for every value the canary metric could show, before you ship it."
Commit the Reasoning Before You See the Result
Everything above — the rewritten rule, and a full table mapping every possible real-world outcome to a conclusion — was written down before the actual result existed. That ordering is the entire point. Rewrite a rule after you've already seen the number, and you are not correcting a bad gate. You're picking whichever justification would have let you do what you already wanted to do.
The pre-registered outcome table looked roughly like this (scores on this competition's normal 0–100-ish scale, baseline at 28.47):
| Real score lands in... | Conclusion |
|---|---|
| ≥ 30.5 | Local read had the direction wrong; every recent local-only verdict this week needs re-checking |
| 28.9 – 30.4 | Real effect exists but is small; local got sign or size wrong |
| 28.05 – 28.89 | Effect is zero within noise; local verdict stands |
| 27.5 – 28.04 | Effect is real and negative, matching local, just smaller |
| < 27.5 | Can't separate a real negative effect from an unrelated execution failure; inconclusive |
The candidate under test had searched a wider space per task — about 31% more candidate answers generated, roughly 22% more compute spent — and the real result came back at 28.19. That falls inside the 28.05–28.89 band. The conclusion followed mechanically, exactly as written down in advance: the extra search bought an effect indistinguishable from zero at this instrument's noise level. No re-litigating, no "well, technically." The table had already decided.
There was a second, unplanned payoff. An earlier calibration point had shown the local proxy's absolute level predicted the real score reasonably well — the real result had landed at about 97.6% of the local number. But one point can only validate level, never ranking; checking whether a noisy proxy preserves order takes at least two independent readings moving in a known direction. This supplied the second point: the ratio-adjusted local prediction was 27.93, the actual result 28.19, a gap of 0.26 — about the size of the real instrument's own noise floor. A gate can be too noisy to threshold on directly and still be trustworthy for ranking candidates against each other — a separate property from absolute-level accuracy, and one that takes a second measurement to check.
Where This Reasoning Breaks Down
Three honest limits, stated on the record rather than left implicit:
- The ≈0.2pp figure for the real target came from four submissions of identical code by different teams, not from our own repeated attempts — the anti-fishing rule forbids measuring that directly, since it would mean resubmitting unchanged code ourselves. If our own run-to-run variance differs from that borrowed estimate, the 24x figure moves. That's a known, accepted gap, not a hidden one.
- The whole argument assumes the resource stays scarce. A process gate built for one submission a day, over a fixed 125-day window, is calibrated to that scarcity — if the budget loosens later, its weight should be revisited.
- Every number here describes one evaluation environment. Public/private scoring environments can diverge; conclusions are scoped to what was measured, not guaranteed wherever the stakes end up highest.
The general habit worth keeping: before arguing where to set a gate's threshold, measure the gate instrument's own noise and put it next to the noise of the thing it protects. If the gate is the noisier side, the fix isn't a bigger or smaller number — it's moving what you're gating on entirely, from a statistical bar you can't trust to a discipline you can. Whichever fix you pick, write down why, and what every outcome will mean, before the result exists to tempt you.
More notes at hexisteme.github.io/notes.
Top comments (0)