Compute a Noise Floor Before You Call an Eval Regression

A single failing golden case tells you almost nothing, and a single passing run tells you even less. If your grader is a string match, a judge model, or a schema check on free-form output, then the same prompt can pass six times and fail twice without anything in your system changing. The practical fix is not a heavier eval framework; it is a per-case noise floor measured from repeated baseline runs, plus a rule that refuses to escalate a verdict the data cannot support.

This article builds that gate in roughly eighty lines of standard-library Python. It assumes you already keep a stamp of model, sampling parameters, and grader version next to every score, because a score without its stamp is not comparable to anything. The artifact here is the layer above that: repetition, a one-sided Wilson bound, and a quarantine list for cases that are too unstable to judge at all.

Why one run cannot separate noise from a regression

Think of each graded case as a coin whose bias you are trying to estimate from a handful of flips. A prompt that passes five of eight baseline repeats and three of eight candidate repeats looks alarming on a dashboard, but with n=8 the confidence interval is simply too wide to justify that alarm. Working the numbers makes the point concrete: six passes out of eight gives a one-sided 95% Wilson lower bound near 0.46, while three out of eight gives roughly 0.16. The candidate is below the baseline floor, yet it is not zero, so the honest verdict is "inconclusive, sample more" rather than "regression".

That distinction is where most eval suites quietly lose their credibility. Teams wire a threshold to a single run, the threshold fires on sampling jitter, and within a month everyone has learned to ignore the report. A noise floor turns the report back into a signal, because it encodes how much variation the harness itself produces before any prompt changes.

The harness: repetition, a floor, and a stamp guard

The module below is runnable as written; you only supply the load_project() adapter that returns your cases plus generate and grade callables.

#!/usr/bin/env python3
"""eval_noise_floor.py - separate grader noise from a real prompt regression.

Stdlib only, Python 3.10+. You supply load_project().
"""
from __future__ import annotations

import argparse, hashlib, json, math, pathlib, sys
from dataclasses import dataclass, asdict

Z = 1.645          # one-sided 95%
FLAKE_BAND = 0.25  # baseline-vs-baseline gap that marks a case unstable
SLACK = 0.05       # jitter you tolerate without escalating


def wilson_lower(passes: int, n: int, z: float = Z) -> float:
    """Lower bound of the Wilson score interval for a binomial proportion."""
    if n == 0:
        return 0.0
    p = passes / n
    denom = 1.0 + z * z / n
    centre = p + z * z / (2 * n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return max(0.0, (centre - margin) / denom)


@dataclass
class CaseResult:
    case_id: str
    passes: int
    n: int

    @property
    def rate(self) -> float:
        return self.passes / self.n if self.n else 0.0


def stamp(model_id, temperature, top_p, grader_version, case_ids) -> str:
    payload = json.dumps(
        {"model": model_id, "temperature": temperature, "top_p": top_p,
         "grader": grader_version, "cases": sorted(case_ids)},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode()).hexdigest()[:12]


def run(cases, generate, grade, repeats: int) -> list[CaseResult]:
    out = []
    for case in cases:
        passes = 0
        for i in range(repeats):
            completion = generate(case["prompt"], i)
            passes += int(bool(grade(case, completion)))
        out.append(CaseResult(case["id"], passes, repeats))
    return out


def verdict(base_a, base_b, cand) -> str:
    if abs(base_a.rate - base_b.rate) > FLAKE_BAND:
        return "quarantine"
    floor = max(0.0, wilson_lower(base_a.passes, base_a.n) - SLACK)
    if cand.rate >= floor:
        return "within_noise"
    if cand.passes == 0:
        return "regressed"
    return "inconclusive"


def main(argv=None) -> int:
    ap = argparse.ArgumentParser()
    sub = ap.add_subparsers(dest="cmd", required=True)
    c = sub.add_parser("collect")
    c.add_argument("--out", required=True)
    c.add_argument("--repeats", type=int, default=8)
    c.add_argument("--model", required=True)
    c.add_argument("--temperature", type=float, default=0.7)
    c.add_argument("--grader-version", required=True)
    p = sub.add_parser("compare")
    p.add_argument("baseline_a")
    p.add_argument("baseline_b")
    p.add_argument("candidate")
    args = ap.parse_args(argv)

    if args.cmd == "collect":
        cases, generate, grade = load_project()
        results = run(cases, generate, grade, args.repeats)
        manifest = {
            "stamp": stamp(args.model, args.temperature, 1.0,
                           args.grader_version, [c["id"] for c in cases]),
            "model": args.model,
            "grader_version": args.grader_version,
            "results": [asdict(r) | {"rate": round(r.rate, 3)} for r in results],
        }
        pathlib.Path(args.out).write_text(json.dumps(manifest, indent=2))
        return 0

    a, b, cand = (json.loads(pathlib.Path(f).read_text())
                  for f in (args.baseline_a, args.baseline_b, args.candidate))
    if len({r["stamp"] for r in (a, b, cand)}) != 1:
        print("refusing to compare: stamps differ", file=sys.stderr)
        return 2
    idx = [{r["case_id"]: CaseResult(r["case_id"], r["passes"], r["n"])
            for r in run_["results"]} for run_ in (a, b, cand)]
    table = {cid: verdict(idx[0][cid], idx[1][cid], idx[2][cid]) for cid in sorted(idx[0])}
    print(json.dumps(table, indent=2))
    return 1 if "regressed" in table.values() else 0


if __name__ == "__main__":
    raise SystemExit(main())

The two baseline collections are the part people skip, and they are the part that pays for itself. Both are produced by the same stamp against unchanged prompts, so any gap between them is pure harness variance, which is exactly the quantity you want to know before blaming a prompt. The exit code stays non-zero only when a case fails at zero passes or after escalation, so a noisy suite stops blocking deploys it should never have blocked.

python eval_noise_floor.py collect --out runs/base_a.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py collect --out runs/base_b.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py collect --out runs/cand.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py compare runs/base_a.json runs/base_b.json runs/cand.json

Repetition is cheap to design and expensive to run

Repeats multiply spend linearly, and the arithmetic gets uncomfortable fast. Forty cases at k=8 for two baselines plus a candidate is 960 generations, and escalating every inconclusive case to k=16 pushes the total past two thousand calls before you have tuned anything. That cost profile is the main reason teams keep shipping one-shot evals: the honest version is roughly ten times the traffic of the dishonest one.

This is the one place where MonkeyCode's operator-supplied availability claims become relevant to the method. The project advertises free model access for prototyping, currently described as a ten-million-token allowance, and a free server option for running long repeat jobs off your laptop. Both figures and terms are promotional and time-sensitive, so confirm the current numbers on the project's own page before you size a repeat budget around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the harness behind your own adapter regardless of which endpoint fills the generate callable. A regression gate that only runs on one provider's promotional tier is not a gate; it is a dependency, and it will fail the week the terms change.

What each verdict should trigger

Verdict	Evidence pattern	Action
within_noise	Candidate rate at or above the baseline floor	Ship, and log the rate so the floor drifts with reality
inconclusive	Below floor, but at least one pass	Re-run that case only, at k=16, before touching the prompt
regressed	Zero passes across repeats, or still below floor at k=16	Block the change and bisect the prompt or the model stamp
quarantine	Two identical baseline runs disagree by more than 0.25	Fix the grader first; the case cannot gate anything

Quarantined cases deserve a sentence of their own, because they are usually misread as model problems. A judge model that flips on the same completion twice, or a regex that depends on whitespace, will look exactly like an unstable prompt until you measure it twice. Pull those cases out of the gate, fix the grader, then reintroduce them with their own floor.

Limitations and who should not use this

Deterministic graders on structured output rarely need k=8, since exact key matching is effectively noiseless and a single sanity repeat is enough. Stateful agents that mutate a database, a queue, or a sandbox between calls break the independence assumption outright, so reruns must start from a reset fixture or the floor means nothing. Anyone unable to pin temperature, top-p, and grader version should stop here, because repetition without a stable stamp just produces a more expensive kind of guessing.

Regulated environments with data-retention obligations should also read the terms of any free tier before sending golden cases through it, and this harness gives you no protection there. Finally, teams chasing latency or cost budgets need a separate harness, because a pass-rate floor says nothing about the tail behaviour you actually care about.

If the repeat-heavy version sounds worth trying and the token bill is the blocker, the free model access and free server option in MonkeyCode are a reasonable place to start. Keep the adapter boundary intact, verify the terms yourself, and the gate stays yours even if the promotion does not.