Morgan Xu

Posted on Aug 29

Postmortem: The 93% Green Eval Was Statistical Noise

#ai #testing #postmortem #llm

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The team tested the model. Nobody tested the test. The eval went 93% green. The merge broke production. This postmortem explains how the number lied. It also ships a durable fix.

The timeline below is a composite. It models a failure pattern, not one real incident.

Timeline

09:00 — Merge window opens. A refactor patch enters review.
09:12 — The eval runs once. It passes 14 of 15 cases. The gate turns green.
09:40 — The patch merges. Staging gets the build.
11:05 — Production error rate climbs. The team rolls back at 11:20.
13:30 — Investigation starts. The team re-runs the exact same eval.
13:31 — It scores 13 of 15.
13:35 — It scores 12 of 15.
13:40 — It scores 15 of 15.

One patch. Four different pass rates. The single green run was luck, not evidence.

Contributing factors

The eval ran once per patch. One draw from a random distribution became a fact.
The gate compared a point estimate. Nobody measured variance.
The free model endpoint had variable latency. Timeouts truncated eval contexts.
The timeout handler counted a truncated run as a pass.
Nobody tested the eval itself. The reviewer was never reviewed.

Factor five is the expensive one. Evals look like code. They behave like experiments.

Why free tiers amplify this

Free model access bends incentives quietly. A team on a metered endpoint runs the eval twice. A team on a free tier runs it once, to save tokens. Single-run evals hide nondeterminism completely.

MonkeyCode is an open-source project with free model access and a free server option. Its free tier currently includes 10 million tokens, per the project's terms. The team used it to reproduce this failure at scale. Repetition finally cost zero dollars. The free server option kept the long repeat-N runs off metered CI workers. Quotas move. Check the repository README before you depend on the numbers.

The durable fix: a repeat-N gate

A point estimate cannot judge a nondeterministic system. A confidence interval can. The new gate checks two properties: rate and stability.

# repeat_n_gate.py — fail the gate on instability, not just on low pass rate
import argparse
import math

def wilson(passes: int, runs: int, z: float = 1.96):
    if runs == 0:
        return 0.0, 0.0
    p = passes / runs
    denom = 1 + (z * z) / runs
    center = (p + (z * z) / (2 * runs)) / denom
    margin = z * math.sqrt(
        (p * (1 - p)) / runs + (z * z) / (4 * runs * runs)
    ) / denom
    return max(0.0, center - margin), min(1.0, center + margin)

def run_eval_once() -> bool:
    # Replace with your real eval invocation.
    # Never count a timeout or truncated run as a pass.
    raise NotImplementedError

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--runs", type=int, default=20)
    ap.add_argument("--min-rate", type=float, default=0.90)
    ap.add_argument("--max-width", type=float, default=0.20)
    args = ap.parse_args()

    results = [run_eval_once() for _ in range(args.runs)]
    passes = sum(results)
    rate = passes / args.runs
    lo, hi = wilson(passes, args.runs)
    width = hi - lo

    print(f"passes={passes}/{args.runs} rate={rate:.2f} CI=[{lo:.2f}, {hi:.2f}] width={width:.2f}")

    if rate < args.min_rate:
        print("FAIL: pass rate below threshold")
        return 1
    if width > args.max_width:
        print("FAIL: confidence interval too wide — rerun or fix the eval")
        return 2
    print("PASS: rate and stability are acceptable")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The gate fails in two ways:

Exit 1 — pass rate below threshold.
Exit 2 — confidence interval too wide.

Width is the new signal. A wide interval means the eval needs more runs. It does not mean the model needs more prompts.

What the numbers look like

Runs	Passes	Rate	95% Wilson CI
1	1	1.00	[0.21, 1.00]
5	4	0.80	[0.38, 0.96]
20	18	0.90	[0.70, 0.97]
100	90	0.90	[0.83, 0.95]

One green run carries a huge error band. Twenty runs narrow the call. One hundred runs make it defensible. Pick N from the width you can tolerate. Do not pick N from your token budget alone.

Choosing N

The width formula is simple. For a pass rate near 0.90, the half-width is roughly 1.96 * sqrt(0.9 * 0.1 / N). A target half-width of 0.10 needs about 35 runs. A half-width of 0.05 needs about 140 runs.

Start with 20 runs. Measure the actual width. Raise N only when the gate tells you to.

Wiring it into CI

Run the gate on merge candidates. Not on every keystroke.
Store the last N results in a JSON file. Compare width over time.
Treat every timeout as a failed run. Never count truncation as a pass.
Lock temperature to 0 where the API allows it. Where it does not, expect noise.
Watch the width trend. Rising width means the endpoint changed, not your code.

Limitations

This gate tests stability. It does not test correctness. A stable wrong answer passes the repeat-N gate. Keep a held-out set that never touches tuning.

Repeat-N multiplies token use. Free tiers carry quotas. A 20-run gate over a 1,000-case suite will exhaust a small budget fast. Teams with deterministic endpoints can skip this entirely. Teams with tiny evals should spend budget on better cases first.

The composite failure above has one root cause. Variance was treated as truth. The remedy is boring — measure the spread before trusting the average. To reproduce this pattern cheaply, MonkeyCode's free tier or free server is a good start. Read their docs first. Quotas move fast.

DEV Community