Pair the Runs, Then Report the Interval

A pass rate printed to two decimal places is a point estimate with no error bar, and a delta between two such numbers is usually a claim the sample cannot carry. Agent suites stay small because tasks are expensive, so a handful of unlucky trials can move a leaderboard by more than the improvement being advertised. The correction is unglamorous and cheap: run every task as a matched pair, compute the difference inside each pair, and publish the interval around the mean difference. A number becomes defensible once the interval, the frozen task hash, and the cost per trial travel with it.

A point estimate cannot support a delta

Twenty tasks with one run each will show a difference between two identical configurations about half the time, because a single trial per task hides both task-difficulty spread and run-to-run noise. Reporting the best of three runs makes the distortion worse, since selection inflates whichever configuration was given more retries or warmer caches. The interval is the only part of the report that tells a skeptical reader whether the observed gap is larger than the noise floor of the suite itself.

Size the trial with a variance pilot

Variance is a property of the task set, not of the vendor, and it can be estimated before anyone approves a real budget. A pilot runs a small number of matched pairs and discards the resulting score entirely; the only output worth keeping is the paired standard deviation, which then decides how many pairs the publishable run needs. Free model access is a reasonable place to host that pilot, because the work is repetitive, the score is disposable, and the sample does not need production capacity. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same applies to the free server option, which is a convenient home for a long-running collector rather than a latency reference point.

import math

def required_pairs(sd, half_width, z=1.96):
    """Pairs needed so the 95% interval half-width is at most half_width."""
    return math.ceil((z * sd / half_width) ** 2)

# Illustrative arithmetic, not a measured claim about any model.
print(required_pairs(sd=0.18, half_width=0.05))  # 50 pairs
print(required_pairs(sd=0.31, half_width=0.05))  # 148 pairs

The first number matters more than it looks. A suite whose paired spread is small needs fifty pairs to resolve a five-point change, while a noisier suite needs roughly three times that, which is exactly the kind of cost decision a pilot should inform before the run starts.

The artifact: paired log and bootstrap interval

The log format is deliberately boring, because the analysis should never depend on a dashboard. Each row records one trial: the task identifier, the seed, the configuration name, a score in the closed unit interval, a cost figure, and a timestamp. Pairs are matched on (task_id, seed) so the comparison is within-task rather than between two differently sampled populations.

{"task_id": "repo-qa-014", "seed": 3, "config": "candidate", "score": 1.0, "cost_usd": 0.0041, "ts": "2026-10-15T09:02:11Z"}
{"task_id": "repo-qa-014", "seed": 3, "config": "baseline", "score": 0.0, "cost_usd": 0.0038, "ts": "2026-10-15T09:02:19Z"}

The analysis script is pure standard library, so it runs anywhere the log can be copied. It pairs the rows, computes per-task deltas, and resamples the deltas with a fixed seed so two analysts reading the same file get the same interval.

#!/usr/bin/env python3
"""paired_interval.py - paired bootstrap interval for two agent configs."""
import argparse, json, random, statistics
from collections import defaultdict


def load(path):
    pairs = defaultdict(dict)
    with open(path) as fh:
        for line in fh:
            if not line.strip():
                continue
            rec = json.loads(line)
            pairs[(rec["task_id"], rec["seed"])][rec["config"]] = float(rec["score"])
    return pairs


def paired_deltas(pairs, a, b):
    out = []
    for key in sorted(pairs):
        runs = pairs[key]
        if a in runs and b in runs:
            out.append(runs[a] - runs[b])
    return out


def bootstrap_ci(deltas, resamples, alpha, seed):
    rng = random.Random(seed)
    n = len(deltas)
    means = []
    for _ in range(resamples):
        total = 0.0
        for _ in range(n):
            total += deltas[rng.randrange(n)]
        means.append(total / n)
    means.sort()
    lo = means[int((alpha / 2) * resamples)]
    hi = means[int((1 - alpha / 2) * resamples) - 1]
    return lo, hi


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("log")
    ap.add_argument("--a", default="candidate")
    ap.add_argument("--b", default="baseline")
    ap.add_argument("--resamples", type=int, default=10000)
    ap.add_argument("--alpha", type=float, default=0.05)
    ap.add_argument("--seed", type=int, default=7)
    args = ap.parse_args()

    pairs = load(args.log)
    deltas = paired_deltas(pairs, args.a, args.b)
    if len(deltas) < 8:
        print(f"warning: only {len(deltas)} complete pairs, interval is uninformative")
    mean = statistics.fmean(deltas) if deltas else 0.0
    sd = statistics.stdev(deltas) if len(deltas) > 1 else 0.0
    lo, hi = bootstrap_ci(deltas, args.resamples, args.alpha, args.seed)
    print(json.dumps({
        "pairs": len(deltas),
        "mean_delta": round(mean, 4),
        "paired_sd": round(sd, 4),
        "ci_low": round(lo, 4),
        "ci_high": round(hi, 4),
        "resamples": args.resamples,
        "seed": args.seed,
        "crosses_zero": lo <= 0.0 <= hi,
    }, indent=2))


if __name__ == "__main__":
    main()

python paired_interval.py trials.jsonl --a candidate --b baseline --resamples 20000 --seed 7

Controls that keep the interval honest

A paired interval inherits every flaw in the pairing, so the pairing rules deserve as much review as the statistics. Each pair should share the task, the seed, the tool timeout, and the retry policy, and each configuration should run in its own process so a warm cache from one side cannot leak into the other. The task list must be frozen and hashed before the run, because adding easy tasks midway shifts both the mean and the spread in ways the interval cannot detect. Cost per trial belongs in the log as well, so a narrow interval that required four times the tokens is visible rather than implied.

Cluster effects are the quiet failure mode. Ten tasks drawn from one repository behave more like one or two independent observations than like ten, and a naive bootstrap will understate the interval accordingly. Resampling whole task families instead of individual pairs is the standard remedy, and it is worth doing whenever the suite spans a small number of repositories or workflows.

An append-only collector on the free server

A trial log split across laptops is not evidence, and the difference between a log and evidence is usually just whether writes are timestamped and sequential. A small collector keeps that property, accepts one JSON object per request, validates the required fields, and appends a canonical line to a single file. The server option is a reasonable host for this because the process is idle most of the day and only needs to accept occasional posts from a runner.

#!/usr/bin/env python3
"""collector.py - append-only trial log. Add auth before exposing this."""
import json, os
from http.server import BaseHTTPRequestHandler, HTTPServer

LOG = os.environ.get("TRIAL_LOG", "trials.jsonl")
REQUIRED = {"task_id", "seed", "config", "score"}


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        try:
            rec = json.loads(self.rfile.read(length))
            if not REQUIRED <= set(rec):
                raise ValueError("missing fields: " + ", ".join(sorted(REQUIRED - set(rec))))
        except Exception as exc:
            self.send_response(400)
            self.end_headers()
            self.wfile.write(str(exc).encode())
            return
        with open(LOG, "a") as fh:
            fh.write(json.dumps(rec, sort_keys=True) + "\n")
        self.send_response(202)
        self.end_headers()

    def log_message(self, *args):
        pass


HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

python collector.py &
curl -s -X POST localhost:8080 -H 'Content-Type: application/json' \
  -d '{"task_id":"repo-qa-014","seed":3,"config":"candidate","score":1.0,"cost_usd":0.0041}'
wc -l trials.jsonl

Read the interval before publishing anything

The decision rule is mechanical once the interval exists, and it removes most of the pressure to round a result upward.

Pairs	Paired sd	Mean delta	95% interval	Action
20	0.31	+0.06	[-0.08, +0.20]	Collect more pairs; do not publish a delta
120	0.19	+0.05	[+0.02, +0.08]	Publish with the interval and the task hash
120	0.19	+0.02	[-0.01, +0.05]	Report as inconclusive, not as a tie
120	0.19	+0.05	Wide after clustering	Report per task family, not pooled

The rows are illustrative arithmetic from the formulas above rather than measured results, and the point they make is the one a marketing page usually omits: samples that small settle very few questions. An interval that crosses zero is a real finding, and calling it a tie quietly replaces a measurement with a preference.

Limits and who should skip this

Free access and a free server change the budget, not the physics of the measurement. Free tiers can queue requests or apply rate limits, so wall-clock latency measured there must never be compared against paid runs, and the pilot is useful only for the paired spread. A single unauthenticated collector should not be exposed to the public internet, since anyone could append fabricated rows to the evidence file. Bootstrap intervals also assume the resampled units are close to independent, which fails whenever a few task families dominate the suite.

Teams certifying safety-critical agent behavior should not rely on this workflow, because a bootstrap interval says nothing about tail risk and nothing about the failure modes that never appear in the task set. The same caution applies to anyone who needs sub-percent resolution, or who has not frozen a task list yet; the workflow measures a fixed suite and cannot repair a suite chosen after the results were seen. Anyone who only needs a rough directional read should skip the collector entirely and keep the paired comparison, since the analysis is the part that changes decisions.

A useful next step is to run the variance pilot, write the resulting spread into the run plan, and record the date and task hash beside every number that later leaves the repository. The free model access and free server are the cheapest place to reproduce that pilot, provided the interval and not the point estimate is what gets published.