Calibrate the Noise Floor of Your Eval Harness

Before you add another golden case to your prompt eval suite, measure how small a difference that suite can actually detect. Most teams treat a pass rate as a precise instrument and then chase two-point moves that live entirely inside sampling noise. The fix is not more cases but a calibration step that turns the suite into a measuring device with a known resolution, and it takes about an hour to build. This article covers a three-part harness: an A/A study for the noise floor, a grader calibration against human labels, and an alert gate that only fires above both thresholds.

Why a pass rate is not yet a measurement

A pass rate mixes three independent sources of variance, and each one moves the number you are staring at. Model sampling contributes variance whenever temperature is above zero, because the same case can pass on one run and fail on the next. Grader noise contributes variance whenever a judge or a heuristic makes different calls on equivalent outputs, especially near the boundary between "mostly right" and "wrong". Case heterogeneity contributes the largest term of all, because resampling a different mix of easy and hard cases moves the aggregate more than any real model change does.

A useful analogy is a kitchen scale with hundred-gram resolution. It will faithfully report that two bowls differ, yet that reading tells you nothing about a twenty-gram difference. Your suite behaves the same way, and the only honest way to learn its resolution is to weigh the same thing twice. That is the A/A study, and it is the cheapest experiment in the whole pipeline.

Run the suite against itself

The harness below is deliberately model-agnostic. You pass in a generate callable and a grade callable, so the same code runs against a hosted API, a local model, or a recorded fixture when you want deterministic tests.

# noisefloor.py
from __future__ import annotations
import random
from statistics import mean

Scored = dict[str, list[float]]  # case id -> per-sample grade in {0.0, 1.0}


def run_suite(cases, generate, grade, samples=5):
    """generate(case) -> str ; grade(case, output) -> float in [0, 1]."""
    return {c: [grade(c, generate(c)) for _ in range(samples)] for c in cases}


def pass_rate(scored: Scored) -> float:
    return mean(g for grades in scored.values() for g in grades)


def bootstrap_delta(a: Scored, b: Scored, iters=2000, seed=7):
    """Resample cases with replacement; samples stay paired inside a case."""
    rng = random.Random(seed)
    cases = sorted(set(a) & set(b))
    deltas = []
    for _ in range(iters):
        pick = [rng.choice(cases) for _ in cases]
        pa = mean(g for c in pick for g in a[c])
        pb = mean(g for c in pick for g in b[c])
        deltas.append(pa - pb)
    deltas.sort()
    return mean(deltas), deltas[int(0.025 * iters)], deltas[int(0.975 * iters) - 1]

Then run the identical commit twice under two arm labels and read the interval back. Nothing changed between the arms, so the interval is pure instrument noise, and its half-width becomes your minimum detectable effect.

arm_a = run_suite(cases, generate, grade, samples=5)
arm_b = run_suite(cases, generate, grade, samples=5)
delta, lo, hi = bootstrap_delta(arm_a, arm_b)
print(f"A/A delta {delta:+.2f} points, 95% CI [{lo:+.2f}, {hi:+.2f}]")
mde = max(abs(lo), abs(hi))

Work through the arithmetic on a hypothetical forty-case suite with five samples per case. That is two hundred graded observations per arm, and bootstrap intervals three to four points wide are entirely ordinary at that size. A two-point drop is then indistinguishable from the coin flips inside your own generator, no matter how confident the dashboard looks when it renders red.

This is exactly where calibration cost usually bites, and where free hosted capacity changes the economics. The A/A study doubles suite cost for one run, and any suite with a noisy grader wants five to ten samples per case rather than one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode, which the operator describes as an open-source project, advertises free model access with a stated ten-million-token allowance and a free server option, which is enough to run a first A/A calibration without requesting a budget line. Free tiers change, so verify current terms on the project page, and keep the harness provider-agnostic so calibration runs can use a free endpoint while release checks keep your production one.

Calibrate the grader before you trust the score

A grader is a component, not an oracle, so it deserves the same treatment as the model it scores. Collect forty to sixty outputs, label them by hand once, and compute agreement against your automated grader using Cohen's kappa rather than raw accuracy, because raw accuracy flatters any grader that leans toward one class.

def cohen_kappa(human: list[int], grader: list[int]) -> float:
    n = len(human)
    agree = sum(h == g for h, g in zip(human, grader)) / n
    p_h, p_g = sum(human) / n, sum(grader) / n
    chance = p_h * p_g + (1 - p_h) * (1 - p_g)
    return (agree - chance) / (1 - chance) if chance < 1 else 1.0

Kappa matters because a lenient grader compresses your whole suite into a plateau near the top, where every change looks like a rounding error. The named convention from Landis and Koch treats 0.61 to 0.80 as substantial agreement, and anything materially below that will inject more variance than the model changes you are trying to detect. Pair the label set with a handful of adversarial probes, because three probes catch the majority of broken heuristic graders.

PROBES = [
    ("", 0.0),                                  # empty output must never pass
    ("I cannot help with that request.", 0.0),  # refusal is not success
]


def grader_defects(grade, case, known_good):
    probes = PROBES + [(known_good, 1.0)]       # your own reference answer
    return [text for text, expected in probes if grade(case, text) != expected]

An empty completion scored as a pass is the classic defect, and it usually comes from a substring rule like "no error keyword present". Run the probes against every grader revision, because a grader refactor is a change to your measurement instrument and therefore a change to every historical number you have reported.

Gate alerts on the noise floor, not on the delta

The decision rule that follows from the calibration is short enough to memorize, and it removes most false pages.

Observed A/B delta	95% CI excludes zero	Action
Above MDE	Yes	Block the merge and notify the owner
Above MDE	No	Rerun with more samples before concluding anything
Below MDE	Either way	Log it; do not page a human

def should_alert(delta, lo, hi, mde):
    statistically_real = hi < 0 if delta < 0 else lo > 0
    practically_real = abs(delta) > mde
    return statistically_real and practically_real

Silent regressions do not disappear just because your gate is now honest about resolution, and that is the uncomfortable part. A real five-point behavior change on ten cases will hide under a four-point noise floor forever, which is why aggregate gating should be paired with per-case inspection rather than trusted alone. The gate exists to protect human attention, while the per-case view exists to explain what actually moved.

What this does not fix, and who should skip it

Deterministic graders on exact-match tasks do not need kappa, though they still benefit from an A/A run to catch flaky infrastructure and stale fixtures. Suites below roughly twenty cases produce bootstrap intervals so wide that the method becomes theater, and a manual read of ten outputs will serve you better. The MDE is also not portable: recalibrate whenever the suite, the grader, or the sampling configuration changes, because a number earned on one suite will mislead you on another.

The bootstrap assumes your cases are exchangeable, which breaks down when cases form clusters such as languages, tenants, or difficulty tiers. Stratified resampling inside each cluster fixes that, at the cost of a slightly longer script. And if your team never reports eval deltas to anyone, the calibration is a hobby, not infrastructure; skip it until a decision depends on the number.

The practical order is A/A study first, grader probes second, alert gate third, and golden-case expansion last. Each step is cheap, each one constrains the next, and together they turn a noisy dashboard into an instrument you can defend in review. If you want to try the A/A study on your own suite, the advertised free tier is enough for a first calibration pass.