DEV Community: Casey Zhang

Calibrate the Task Before the Model: A Spec-Only Floor for Agent Benchmarks

Casey Zhang — Sun, 08 Nov 2026 19:16:07 +0000

You publish a table. Model A sits at 71% on your patch suite, model B at 64%. Someone asks a simple question — how many of those tasks are solvable from the issue text alone? — and the thread goes quiet.

That silence is the bug. Seven points of separation means nothing if four of your tasks never needed the repository in the first place. Before you compare endpoints, you have to know how much of your dataset a prompt alone can already solve.

This post is a working method for that: a spec-only floor, computed per task, cheap enough to run on a free tier, and boring enough to publish alongside your scores.

The problem is task-side, not model-side

Most agent benchmarks are built by collecting real issues and real patches. That is good for realism and terrible for calibration. A task titled "rename getUserData to fetchUser" is a find-and-replace. A task titled "fix flaky integration test" is a research project. Both land in the same average.

Worse, the easy end of the set is not random noise — it correlates with how much context leaked into the issue body. Stack traces, file paths, and pasted diffs all shrink the search space. So your pass rate partly measures how well you wrote your tickets.

The fix is not a harder dataset. It is a measurement of the dataset itself, taken with the same harness you use for models.

Step 1 — Define two passes over the same task

Every task gets run twice, with only one variable changed: repository access.

Spec pass — the agent receives the issue text, the test command, and the language/runtime version. No checkout, no file listing, no grep.
Full pass — identical prompt, identical budget, plus a read-only checkout of the repo at the recorded parent commit.

Everything else is pinned: same model endpoint, same temperature, same max steps, same wall-clock cap, same judge.

If the spec pass succeeds, the task did not require retrieval. That is a dataset fact, not a model opinion.

Step 2 — Record the floor in the dataset, not in a spreadsheet

Put the field in the task record so it travels with the data. A task line looks like this:

{"id":"t-0142","repo":"svc-billing","base_commit":"9f3c1ab","issue":"...","test_cmd":"pytest -q tests/test_invoice.py","floor_pass":0,"floor_n":3}

Then the scoring script reads it instead of recomputing it. Here is the harness sketch I use for the report. It is unexecuted here — wire run_task() to your own runner and keep the comparison logic untouched.

# floor_report.py — SKETCH, not run as written
import json, statistics
from collections import defaultdict

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

def run_task(task, mode, model, seed):
    """mode='spec' -> issue only. mode='full' -> issue + checkout."""
    # delegate to your existing sandboxed runner; return True/False
    raise NotImplementedError

def measure(tasks, mode, model, repeats=3):
    results = defaultdict(list)
    for t in tasks:
        for seed in range(repeats):
            results[t["id"]].append(run_task(t, mode, model, seed))
    return {k: sum(v) / len(v) for k, v in results.items()}

def report(tasks, model, repeats=3):
    spec = measure(tasks, "spec", model, repeats)
    full = measure(tasks, "full", model, repeats)
    rows = []
    for t in tasks:
        f, p = spec[t["id"]], full[t["id"]]
        rows.append((t["id"], f, p, p - f))
    keep = [r for r in rows if r[1] == 0]
    print(f"tasks={len(rows)} floor_clean={len(keep)}")
    if keep:
        print(f"mean_delta_over_floor={statistics.mean(r[3] for r in keep):.3f}")
    return rows

if __name__ == "__main__":
    report(load("tasks.jsonl"), model="<your-endpoint>")

The output you care about is floor_clean and the delta. Raw pass rate is not the headline.

Step 3 — Run the floor pass first, because it is the cheap one

Order matters. Spec passes use a fraction of the tokens and none of the sandbox disk, so they finish before your full runs start. That is exactly the profile where a free tier is useful rather than embarrassing.

MonkeyCode is the project I have been using for this pass: the operator advertises free model access and a free server option, and the allowance quoted to me for the free tier was 10 million tokens. Treat that number as a snapshot, not a contract — quotas on free tiers move, so check the page yourself before you size a run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I run the floor with concurrency 2 and a hard 90-second cap per task:

python floor_report.py --tasks tasks.jsonl --concurrency 2 --max-seconds 90

Ten tasks is enough to find your first contaminated item. You do not need the whole suite to learn that your bugs are well documented.

Step 4 — Score the delta, and say what each metric means

Report three numbers, not one. Collapsing them is where tables start to resemble marketing.

Metric	Definition	What it answers
`F`	mean spec-pass rate over `n` repeats	how much of the task is prompt-solvable
`P`	mean full-pass rate over `n` repeats	the number you used to publish alone
`D = P − F`	per-task delta, averaged over floor-clean tasks	value added by repository access

D is the honest comparison axis. Report F next to it, and report the repeat spread — the min and max across seeds — because a delta with a four-point spread across three runs is a coin arranging itself.

Step 5 — Pin the controls, or the floor drifts too

Five controls, all boring, all necessary:

Freeze the checkout at base_commit; no git pull between runs.
Same runtime image for both passes. The spec pass must not get a fresher toolchain.
Same judge and the same pinned rubric version for both passes, including the spec pass. A lenient judge inflates F and deflates D.
Same step and time budget. Giving the full pass more steps is a second variable you did not mean to add.
Log the prompt hash. If a task prompt changes mid-run, the floor is stale.

Step 6 — Act on the floor with a decision table

Do not delete tasks on instinct. Classify them.

Spec pass over 3 tries	Reading	Action
0/3	needs the repo	keep in the headline set
1–2/3	prompt hints are half the answer	keep, but report in a separate stratum
3/3	solved without retrieval	move to a smoke suite, or drop

The middle row is the one people delete by accident. Those tasks are not broken — they are just easier than the average they were hiding inside.

Limitations, and who should skip this

The spec-pass floor conflates two things: genuine prompt leakage and genuinely easy tasks. It also says nothing about whether a task is important, only whether it is self-contained. With n=3 repeats the estimate is coarse, and a task sitting at one success out of three is a coin flip you should not over-read.

Skip this method if you need absolute leaderboard numbers comparable to someone else's suite, if your tasks depend on private infrastructure you cannot sandbox twice, or if you already have enough budget to run a full factorial design — in that case the floor is a redundancy, not a discovery.

If you do try it, start with ten tasks from your existing suite and run only the spec pass. The free tier on MonkeyCode covers that comfortably, and the list of tasks your prompt alone can solve is usually more useful than the scoreboard it was hiding in.

Your Benchmark Forgot the Box: Attribute the Wobble Before You Blame the Model

Casey Zhang — Sat, 07 Nov 2026 17:58:44 +0000

You ran the same twelve tasks against two coding agents last month. One won by four points. You put the table in a doc and moved on.

Now ask the harder question. How much of that four-point gap belonged to the runtime rather than the model? A different machine. A cold dependency cache. A workspace that still had yesterday's build artifacts in it. None of that appears in your score, and all of it moves it.

Here is a cheap way to find out: hold the model fixed, flip only the box, and count how many tasks change verdict. If the box flips more tasks than your two candidates differ by, you do not have a model comparison yet. You have an environment comparison wearing a model's name.

The bug: you measured spread, not cause

Most of us now accept that agent scores wobble between runs. Good. That was the first lesson, and plenty of teams still have not learned it.

But knowing the wobble exists is not knowing what produced it. A re-run spread of six points could come from sampling temperature, from the grader being fuzzy, or from the machine underneath. Those three have completely different fixes, and averaging them together hides all three.

Attribution needs a controlled flip. Change one thing, hold everything else, and count verdicts that move. That is it. No statistics degree required, no leaderboard, no thirty-run sweep.

The experiment: flip the box, hold the model

You need three things: a task set small enough to run twice in an afternoon, a pass test that returns exit code 0 or not, and two environments that differ in one dimension you can name out loud.

Start with twelve tasks from your own repo. Twelve is not statistically impressive and it does not need to be. It is enough to detect a loud environment, which is the only thing this pass is for.

Step 1 — Write the pass test before you look at any model

The pass test is the whole contract. If it needs a human to read the diff and nod, it is not a pass test, it is a vibe.

bench/tasks/t01_parse_offsets/
├── prompt.md            # what the agent is asked to do
└── repo/
    ├── run_tests.sh     # exit 0 = pass, anything else = fail
    └── ...              # the checkout the agent patches

Two rules keep this honest. First, the test script must not reach the network. Second, it must clean up after itself, or your second run is really your first run plus leftovers. If your tasks cannot satisfy both rules today, fix that before reading any further.

Step 2 — Define two environments that differ in one dimension

A tab-separated file is enough, and it removes the temptation to hide a second variable in a config file nobody reads.

# bench/envs.tsv
name    base_url    workspace   reset   timeout_s
local-container http://127.0.0.1:8080   /tmp/bench/ws   wipe    120
remote-free https://YOUR-SECOND-ENV-HOST    /home/bench/ws  wipe    120

Name the dimension you are flipping in your notes. "Container vs remote host" is a dimension. "Container vs remote host, also different timeout, also different lockfile" is three dimensions and one useless experiment.

Step 3 — Run the probe

The harness below wipes the workspace per run, executes only the pass test, and appends one JSON line per (task, env, attempt). Label: not executed against your repo as written, and it will not be until you wire your own run_tests.sh.

#!/usr/bin/env python3
# bench/env_probe.py - flip the runtime, hold the model, record the verdict.
import argparse, csv, hashlib, json, pathlib, shutil, subprocess, time, uuid

def load_envs(path):
    with open(path, newline='') as f:
        return list(csv.DictReader(f, delimiter='\t'))

def env_sha(env, task):
    blob = json.dumps([env['name'], env['base_url'], task.name,
                       (task / 'repo' / 'run_tests.sh').stat().st_size],
                      sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def fresh_ws(env, task):
    ws = pathlib.Path(env['workspace']) / task.name
    if ws.exists():
        shutil.rmtree(ws)
    shutil.copytree(task / 'repo', ws)
    return ws

def run_one(env, task, attempt, run_id):
    ws = fresh_ws(env, task)
    t0 = time.monotonic()
    proc = subprocess.run(['bash', 'run_tests.sh'], cwd=ws,
                          capture_output=True, timeout=int(env['timeout_s']))
    wall_ms = int((time.monotonic() - t0) * 1000)
    return {'run_id': run_id, 'task': task.name, 'env': env['name'],
            'env_sha': env_sha(env, task), 'attempt': attempt,
            'exit_code': proc.returncode, 'passed': proc.returncode == 0,
            'wall_ms': wall_ms}

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--tasks', required=True)
    ap.add_argument('--envs', required=True)
    ap.add_argument('--out', required=True)
    ap.add_argument('--repeat', type=int, default=2)
    args = ap.parse_args()

    run_id = uuid.uuid4().hex[:12]
    tasks = sorted(p for p in pathlib.Path(args.tasks).iterdir() if p.is_dir())
    out = pathlib.Path(args.out)
    out.parent.mkdir(parents=True, exist_ok=True)
    with out.open('a') as fh:
        for attempt in range(args.repeat):
            for env in load_envs(args.envs):
                for task in tasks:
                    rec = run_one(env, task, attempt, run_id)
                    fh.write(json.dumps(rec) + chr(10))
    print('run_id=%s rows=%d' % (run_id, len(tasks) * args.repeat * 2))

if __name__ == '__main__':
    main()

Run it twice, not once. --repeat 2 gives you a within-environment signal for free: if attempt 0 and attempt 1 disagree on the same box, the box is already noisy and no cross-environment comparison will save you.

python3 bench/env_probe.py --tasks bench/tasks --envs bench/envs.tsv \
  --out runs/2026-11-08.jsonl --repeat 2

Where a free second environment honestly fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The expensive part of this experiment is not the analysis, it is standing up the second box. MonkeyCode advertises free model access and a free server option, which is exactly the two pieces you need here: a model you can hold constant while you flip the environment, and a second environment you can reach without a procurement ticket. The free token allowance was reported to me as 10 million tokens at the time of writing — treat any specific number as a snapshot with an expiry date and re-read the current terms the day you run, because quotas move.

Be precise about what that buys you, though. A shared free server is a second environment, not a reference environment. Its load is not yours to control, its disk state may persist in ways you did not choose, and nothing about its wall-clock is a latency baseline. Use it to detect verdict flips. Do not use it to publish p95 numbers, and do not treat it as the fast side of a "free is faster/slower" claim.

Step 4 — Build a flip table instead of a mean

Comparing means across environments throws away the only interesting information: which tasks moved.

#!/usr/bin/env python3
# bench/flip_table.py - paired verdicts in, flip table out.
import collections, json, sys

def main(path, env_a, env_b, attempt=0, gate=0.10):
    rows = [json.loads(l) for l in open(path) if l.strip()]
    pair = collections.defaultdict(dict)
    for r in rows:
        if r['attempt'] == attempt:
            pair[r['task']][r['env']] = r['passed']

    flips, n = 0, 0
    for task in sorted(pair):
        a, b = pair[task].get(env_a), pair[task].get(env_b)
        if a is None or b is None:
            continue
        n += 1
        flipped = a != b
        flips += flipped
        print('%-28s %-5s %-5s %s' % (task, a, b, '<- flip' if flipped else ''))

    rate = flips / n if n else 0.0
    print('tasks=%d flips=%d flip_rate=%.3f gate=%.2f' % (n, flips, rate, gate))
    sys.exit(0 if rate <= gate else 1)

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2], sys.argv[3])

python3 bench/flip_table.py runs/2026-11-08.jsonl local-container remote-free

The output shape looks like this. These numbers are placeholders to show the format, not a measurement of anything:

t02_retry_budget              True  False <- flip
t05_null_guard                True  True
t09_pagination_cursor         True  False <- flip
tasks=12 flips=2 flip_rate=0.167 gate=0.10

Two flips out of twelve is a loud box. Now compare that to your model gap: if agent A beats agent B on two tasks and the environment flips two tasks on its own, you cannot tell those apart, and neither can your reader.

Step 5 — Gate on the table

Pick the rule before you see the data, or you will rationalize whatever comes out.

Flip count (12 tasks)	Reading	Next action
0	Box is quiet enough for this task set	Proceed to the model comparison
1–2	Box noise is the same order as your effect	Widen the task set or pin harder before claiming a winner
3+	Box dominates	Fix the environment; publish nothing about models yet
Any	Same-env attempts disagree	Your pass test is flaky — fix the test first

The script exits non-zero above the gate so CI can hold the line for you. A flipped verdict that nobody reruns is just a bug with good manners.

Why these numbers are not marketing

The difference between a benchmark and a slide is what you ship alongside it. Ship the raw JSONL, the task list, the envs.tsv, and the env_sha on every row. A reader who has those can regenerate your table without asking you anything.

Also ship what is missing. Your twelve tasks came from one repo. Your environments differed in ways you may not have fully enumerated. Your pass test is binary, so it cannot see a patch that passes while breaking something else. Say all of that in the article, in the same section as the table, not in a footnote.

And state the direction of your incentive. If you sell the thing being measured, publish the harness so someone with the opposite incentive can run it. A number without a harness is an advertisement.

Limitations and who should not use this

Latency claims. The flip table deliberately ignores timing except as a smoke signal. If you need p95 wall-clock, you need dedicated capacity and a load generator, not this.
Security-sensitive code. Do not push a proprietary repo to a shared environment you have not read the data terms for. Use your own second box.
Stateful workloads. Wipe-per-run environments cannot tell you anything about migrations, caches, or long-lived sessions. Different experiment.
Teams without a binary pass test. If your tasks still need human review, spend the week writing the test. The flip table is worthless on top of a fuzzy oracle.

The method is also deliberately underpowered. Twelve tasks will catch a loud environment and will miss a quiet one. That is the correct trade for a pre-flight check you run in an afternoon, and the wrong trade for a published ranking.

What to do on Monday

Pick twelve tasks, write run_tests.sh, stand up a second box, run the probe twice. You will have a flip table before lunch and a much better question than "which agent is better": which of my numbers survive a change of machine?

If you run this, the artifact worth sharing is the flip table, not the mean.

Stop Peeking at Run Five: A Sequential Stopping Rule for Free-Tier Agent Benches

Casey Zhang — Fri, 06 Nov 2026 16:33:58 +0000

You know the scene. Someone drops a free endpoint URL into the team channel, a forty-task slice gets run by hand, and by Thursday night there are five green checks and a message reading "this one's good, ship it."

The five green checks are not the problem. The stopping is.

Sample five tasks, stop the moment the fourth one passes, and you have an 80% endpoint. Repeat that on ten endpoints and the one you adopt is the one that got lucky early — not the one that patches your code. This post replaces that instinct with a rule you pre-register, run in a loop, and defend in a review. The whole artifact is about forty lines of Python.

What this rule decides, and what it refuses to decide

It answers exactly one question: on this frozen task pool, is this endpoint worth adopting? It returns adopt, drop, or a budget-cap fallback.

It does not estimate a pass rate with a confidence interval. It does not rank two endpoints that differ by three points, and it will not tell you which of two acceptable endpoints is "better." If you need effect sizes, you need a fixed-N design with far more tasks than you probably have.

That refusal is the feature. A rule that returns a decision can be written down before you see any results. A rule that returns a number invites you to keep sampling until the number looks right.

You are not benchmarking models in general here. You are grading fitness-for-purpose against one slice, and the slice is the argument.

Step 1 — Freeze the pool before you touch the endpoint

Write the task records down first. If you can edit the pool after seeing a failure, you do not have a bench, you have a curation exercise.

field	example	rule
`task_id`	`repo-1042`	stable, never reused for a different task
`repo_commit`	pinned SHA	identical for every endpoint
`issue_text`	verbatim	no rewording between endpoints
`hidden_test`	path inside the image	never shown to the model
`order_seed`	`20261107`	fixes the sequence before run one

Build a pool of 60 held-out tasks, not 20. The stopping rule decides how many of the 60 you actually run, which means the pool has to be bigger than any likely verdict point — otherwise your cap is doing all the work and the sequential part is decoration.

Then shuffle once, with a seed you publish, and commit the resulting order. This kills the quiet cheat of skipping the task you suspect will fail.

Step 2 — Pin the runner so the environment is not a variable

Most "the endpoint got worse" stories are toolchain drift with a model-shaped alibi. Pin the image digest, the Python version, the test runner version, the per-task wall-clock limit, and whether the test phase gets network access.

The free server option is relevant for a boring reason: you want the harness on a machine whose image you control end to end, not on whatever a hosted sandbox hands you this week. Record the pin next to every ledger row.

# append the environment pin to the manifest, once per session
printf 'image=%s\n' "$(docker inspect -f '{{.Image}}' bench-runner)" >> manifest.txt
printf 'python=%s\n' "$(python3 -V 2>&1)" >> manifest.txt
printf 'order_seed=%s\n' 20261107 >> manifest.txt
printf 'wall_clock_s=%s\n' 300 >> manifest.txt

If the digest changes mid-plan, stop. The plan is void, not "roughly comparable."

Step 3 — Keep invalid runs out of the numerator and the denominator

Free-tier endpoints get interrupted. Rate limits fire, connections drop, a run returns a patch that never applied cleanly.

You have three options when that happens and only one is honest: mark the attempt invalid, re-run the same task in the pre-registered order, and continue. Silently retrying until you get a completion leaks selection into your sample — you have just conditioned on the endpoint eventually cooperating.

Report invalid rate as its own number, never folded into pass rate. In practice it is often the most decision-relevant metric you produce all week: an endpoint with a strong pass rate and a 30% invalid rate is not usable for the thing you are doing, no matter how good the numerator looks.

Step 4 — Apply the rule

The rule is Wald's sequential probability ratio test on pass/fail, with a hard run cap so an undecided case cannot run forever. Save this as sprt.py.

"""sprt.py — pre-registered sequential stopping rule for a pass/fail bench."""
import math
from dataclasses import dataclass


@dataclass(frozen=True)
class Plan:
    p_adopt: float = 0.60   # H0: the pass rate you would ship with
    p_drop: float = 0.40    # H1: the pass rate you would walk away from
    alpha: float = 0.10     # tolerated false-adopt rate
    beta: float = 0.10      # tolerated false-drop rate
    n_cap: int = 60         # hard ceiling = pool size

    def llr(self, passes: int, fails: int) -> float:
        return (
            passes * math.log(self.p_adopt / self.p_drop)
            + fails * math.log((1 - self.p_adopt) / (1 - self.p_drop))
        )

    def decide(self, passes: int, fails: int) -> str:
        n = passes + fails
        llr = self.llr(passes, fails)
        upper = math.log((1 - self.beta) / self.alpha)
        lower = math.log(self.beta / (1 - self.alpha))
        if llr >= upper:
            return "adopt"
        if llr <= lower:
            return "drop"
        if n >= self.n_cap:
            # out of budget and still undecided: fall back to the point estimate
            return "cap-adopt" if passes / n >= self.p_adopt else "cap-drop"
        return "continue"


def run_plan(results, plan=None):
    """results: iterable of 0/1 in the pre-registered order. Yields ledger rows."""
    plan = plan or Plan()
    passes = fails = 0
    for n, ok in enumerate(results, start=1):
        passes += int(ok)
        fails += 1 - int(ok)
        verdict = plan.decide(passes, fails)
        yield {
            "n": n, "passes": passes, "fails": fails,
            "llr": round(plan.llr(passes, fails), 3), "verdict": verdict,
        }
        if verdict != "continue":
            return

Here is the part that makes review meetings easier. With p_adopt=0.60 and p_drop=0.40 the two log-odds ratios are equal and opposite, so the LLR collapses to 0.4055 * (passes - fails) and both boundaries sit at ±2.197. You can verify every ledger row in a spreadsheet without running any code, and so can the reviewer who does not trust your script.

Step 5 — Calibrate the rule before you point it at anything

A stopping rule you have not checked is just a lookup table with confidence. Simulate it first. Save as calibrate.py.

import random
from sprt import Plan


def simulate(p_true, plan=None, trials=20_000, seed=7):
    plan = plan or Plan()
    rng = random.Random(seed)
    tally = {"adopt": 0, "drop": 0, "cap-adopt": 0, "cap-drop": 0}
    runs_used = 0
    for _ in range(trials):
        passes = fails = 0
        while True:
            if rng.random() < p_true:
                passes += 1
            else:
                fails += 1
            verdict = plan.decide(passes, fails)
            if verdict != "continue":
                tally[verdict] += 1
                runs_used += passes + fails
                break
    return tally, runs_used / trials


if __name__ == "__main__":
    for p in (0.30, 0.50, 0.70):
        tally, avg_runs = simulate(p)
        print(f"p_true={p:.2f} avg_runs={avg_runs:5.1f} {tally}")

python calibrate.py

Check three regimes, and check them in this order.

p=0.70 — above the adopt boundary. Most trials should stop with adopt, and reasonably early.
p=0.30 — below the drop boundary. Most trials should stop with drop.
p=0.50 — sitting between the boundaries, where the random walk has no drift. Most trials should exhaust the cap rather than invent a verdict.

If any of those three fails, your alpha, beta, or n_cap is wrong. Fix the rule before it touches a model, not after it tells you something inconvenient.

And do not treat my prose as evidence about the output. The script is short precisely so the numbers you trust are the ones your machine printed with your plan's settings.

The decision table

Run the pool in order and read the ledger against this.

ledger row	count	LLR ≈ 0.4055 × (p − f)	action
6 passes, 0 fails	n=6	+2.43 ≥ +2.197	`adopt`, stop
5 passes, 4 fails	n=9	+0.41	`continue`
2 passes, 7 fails	n=9	−2.03	`continue`
1 pass, 8 fails	n=9	−2.84 ≤ −2.197	`drop`, stop
35 passes, 25 fails	n=60	+4.06, still no verdict only if boundaries moved	`cap-adopt` if 35/60 ≥ `p_adopt`, else `cap-drop`

Note what the table does not contain: a column for "how good does it feel." Every row is arithmetic you can redo by hand, which is the whole difference between a bench and a leaderboard screenshot.

Where free access actually changes the math

Cost pressure biases small benches in a predictable direction. When each run costs money you run fewer, when you run fewer you stop earlier, and stopping earlier is exactly the bias the rule above exists to remove. Free model access removes that pressure, which is the only reason it belongs in a methodology post at all.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option covers the second requirement — a runner whose image and toolchain you pin yourself, so environment drift stops masquerading as model quality. Stated plainly as a limitation: availability windows on any free tier are not something you can schedule around, so the invalid-run path from Step 3 has to be a first-class branch in your harness, not an exception handler you write at 6pm.

Who should not use this

Fewer than ~40 tasks in the pool. Run ten tasks, call it a smoke run, and never quote a percentage from it. The rule needs a pool bigger than the verdict point to mean anything.
Anyone distinguishing 71% from 74%. Sequential tests decide; they do not resolve small differences. You need hundreds of tasks and a different instrument.
Anyone asking "does this endpoint emit patches at all?" One task, two minutes, no ledger. Match the rigor to the question.
Teams that cannot pin the runner. Without a pinned environment this is a vibe check with extra paperwork, and the paperwork will make it look more trustworthy than it is.

Closing the loop

Put the plan, the order seed, the environment pin, and the ledger path in the pull request that switches the endpoint. Six weeks later, "why did we pick this one" has an answer that is not a chat screenshot.

If you want to run a pre-registered 60-task plan without paying per call, MonkeyCode's free model access and free server option cover the two inputs this workflow needs — the stopping rule, the pinned image, and the ledger are the forty lines above, and they are yours to keep either way.

Grade the Grader First: A κ-Gate for Coding-Agent Benchmarks

Casey Zhang — Thu, 05 Nov 2026 15:12:27 +0000

You have forty patches. Two reviewers score them pass or fail against the same rubric. They agree on twenty-nine. So you compute the win rate, publish the ranking, and move on.

A week later someone reruns the sweep with a different reviewer and the ranking inverts. The agents did not change. The instrument did.

A patch label is a measurement, not a fact. If your two labelers cannot reproduce each other, no amount of endpoint comparison will save the result. This post is a workflow for checking that before you publish anything, and for refusing to publish when the check fails.

Why agreement comes before accuracy

Accuracy needs ground truth. For a coding patch you usually have something better and cheaper: a test command that either exits zero or does not. Start there.

But the interesting patches are the ones the tests do not settle. Refactors that keep tests green. Fixes that pass tests for the wrong reason. Partial fixes that satisfy two of three assertions. Those go to a human label, and that human label is where comparability dies quietly.

Step 1: Build a small, stratified, perturbed dataset

Forty tasks is enough for a calibration pass and small enough that two people can label it in an afternoon. Do not sample forty tasks from one difficulty band. Stratify deliberately.

bench/
  tasks/
    t-001/
      prompt.md          # the task as the agent sees it
      run.sh             # the check: exits 0 on pass
      mutate.py          # perturbation probe, optional
  labels/
    grader-a.csv
    grader-b.csv
  manifest.json

Three strata, roughly equal size:

Mechanical — lint, type error, import cycle. A correct patch is unambiguous.
Behavioral — one named test flips from red to green. Still mostly objective.
Judgment — refactor-only, or a fix where the test suite is a weak proxy.

If stratum 3 disagrees heavily and stratum 1 does not, you have learned something specific: your rubric is fine for mechanical work and unreliable for the work you actually care about. That is a useful finding, and it is invisible in a single blended score.

Add a perturbation probe while you are here. Rename the identifiers, shuffle the imports, reorder the tests. A patch that only passes on the original spelling of a function is a memorization signal, not a capability signal. It is a crude probe, but it costs ten minutes.

Step 2: Collect two independent labels

Independent means the second grader does not see the first grader's labels. Not the score, not the comments, not the diff annotations. Two humans reading the same patch cold.

Use a binary label plus an abstain option: pass, fail, unclear. Abstain is not a cop-out. It is a signal that the rubric is underspecified at that point, and it tells you exactly which clause to rewrite.

Keep a third labeler in reserve. Disagreements get adjudicated by that person and become your reportable subset. You publish the adjudicated subset plus the raw agreement, not one blended number.

Step 3: Compute κ before you compute anything else

Agreement that looks high can be pure chance. If 90% of patches pass, two graders who always say pass agree 90% of the time while measuring nothing. Cohen's κ corrects for that.

Reference implementation, no third-party dependencies:

# kappa.py
from collections import Counter

def cohen_kappa(a, b):
    assert len(a) == len(b) and a, 'need equal, non-empty label vectors'
    n = len(a)
    labels = sorted(set(a) | set(b))
    po = sum(1 for x, y in zip(a, b) if x == y) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum((ca[l] / n) * (cb[l] / n) for l in labels)
    if pe == 1.0:
        raise ValueError('both graders collapsed to one label; kappa undefined')
    return (po - pe) / (1 - pe), po

A point estimate on forty items is not enough. Bootstrap it:

import random

def bootstrap_kappa(a, b, iters=2000, seed=7):
    random.seed(seed)
    n = len(a)
    vals = []
    for _ in range(iters):
        idx = [random.randrange(n) for _ in range(n)]
        try:
            k, _ = cohen_kappa([a[i] for i in idx], [b[i] for i in idx])
        except (ValueError, AssertionError):
            continue
        vals.append(k)
    vals.sort()
    return vals[int(0.025 * len(vals))], vals[int(0.975 * len(vals))]

Then turn it into a gate, so the decision is mechanical and not a matter of mood on release day:

python kappa.py --grader-a labels/grader-a.csv \
                --grader-b labels/grader-b.csv \
                --gate 0.60
# exit 0 -> proceed to ranking
# exit 2 -> rubric revision required; no ranking published

Step 4: Read the result against a decision table

Thresholds are conventions, not physics. The bands below follow the well-known descriptive scale from Landis and Koch (Biometrics, 1977), which is widely cited for exactly this purpose. Cite it and move on; do not relitigate it in your results section.

κ (95% CI)	Reading	Action
< 0.40	Disagreement dominates	Do not publish a ranking. Rewrite the rubric to test-anchored binary criteria.
0.40 – 0.59	Marginal	Publish per-stratum agreement and the adjudicated subset only.
0.60 – 0.79	Substantial	Ranking is publishable with κ, CI, and n attached.
≥ 0.80	Strong	Publishable. Still report sample size and stratum breakdown.

Always report po (raw agreement) and the label marginals next to κ. A κ of 0.65 with 80% raw agreement means something different from a κ of 0.65 with 55% raw agreement.

Step 5: Run the calibration pass on a disposable box

This is where infrastructure matters more than people expect. A calibration sweep installs language toolchains, mutates repositories, and leaves test caches everywhere. You do not want any of that on your laptop, because next month you will want to rerun the same manifest and your local environment will have drifted.

A throwaway runner is the right shape: provision it, run the sweep, export labels/, manifest.json, and the harness logs, then destroy it. Same input, same box image, same output.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here for two operator-supplied availability claims that fit this workflow: free model access for the endpoint under test, and a free server option for the runner. The brief I was given also states a free allowance of 10M tokens, which is a reasonable size for a forty-task calibration pass — but free-tier terms and quotas change, so check the project page for the current numbers before you size a run and write those numbers into your manifest.

The reason this matters for methodology, not just cost: a benchmark that a stranger can rerun on free infrastructure with free endpoints is a benchmark that can be audited. Rankings that only reproduce on someone's paid cluster are closer to marketing than to measurement.

What this workflow does not fix

Be honest about the edges.

κ measures agreement, not correctness. Two graders can agree confidently and both be wrong. Anchor to run.sh exit codes wherever possible, and treat human labels as the exception path.
κ is prevalence-sensitive. Extreme pass rates inflate chance agreement and produce unstable κ values. Report marginals; if one label exceeds ~90% of the set, add harder tasks before trusting the number.
Forty items gives a wide interval. That is fine for a gate. It is not fine for claiming a two-point difference between endpoints.
Two graders only. For three or more, use Fleiss' κ or Krippendorff's α instead of averaging pairwise κ.
The perturbation probe is crude. It catches blatant memorization, not subtle contamination. It is a smoke test, not a proof.

Who should skip this

If you already run continuous evaluation where every task has an executable check and labels are generated, not judged, you do not need a κ-gate — your instrument is already objective. If you cannot get two independent human labelers for two hours, this workflow will not run, and you should say so in your write-up rather than implying comparability you did not measure.

And if you only need a smoke test to see whether an agent can edit a file without breaking imports, skip all of this. A κ-gate is for the moment you are about to publish a ranking.

Run the gate first. If κ clears 0.60 with an interval you are willing to defend, the ranking that follows is worth reading. If it does not, you just saved yourself from publishing a number that would flip on the next run — and that is the cheapest result in the whole benchmark.

Kill the Runner Mid-Patch: A Fault-Injection Bench for Agent Sessions

Casey Zhang — Tue, 27 Oct 2026 12:39:19 +0000

Here is a shape of failure worth ten minutes of your attention. An agent drafts a patch, the machine goes away mid-task, the retry starts from the top, and the migration lands twice. The pass rate stayed at 100% — because the interrupted run never reached a verdict at all.

That is not a model problem. It is a measurement problem, and a fault-injection bench fixes it.

A pass rate only describes the runs that finished

Every benchmark has a denominator, and most of them are accidental. Drop interrupted runs and you are scoring the survivors. Run that on flaky, shared, or reclaimable compute and your "best" endpoint is simply the one that got lucky with uptime.

So measure three things separately. Capability: did the patch pass? Efficiency: what did it cost in quota and wall time? Durability: does the result survive an interruption? Your demo shows the first, your billing shows the second, and nothing shows the third until it bites you in production.

Step 1 — Freeze the environment in a manifest

Before the first run, write down what you are actually running on. Anything you cannot name in a manifest is a variable you cannot control later. This is the same discipline as freezing a price envelope, applied to time and failure instead of cost.

# faultbench/manifest.yaml  (template, not a vendor spec)
run_id: 2026-10-27-faultbench-a
taskset_sha: "<git sha of your task files>"
runner_sha: "<git sha of the harness>"
env:
  python: "3.12"
  os: "linux"
  container_digest: "sha256:<digest>"
  cpus: 2
  mem_gb: 4
  disk: "ephemeral"
endpoint:
  provider_notes: "free model access via MonkeyCode"
  model_id: "<exact id the endpoint reports>"
  allocation_today: "<copy today's number from the project page>"
faults:
  plan:
    - "patch_applied:hard_exit"
    - "tests_run:hard_exit"
  repeats: 3

Two rules. First, copy today's free allocation into allocation_today instead of trusting memory — free-tier numbers move, and a note that ages badly is worse than no note. Second, never compare two endpoints whose manifests disagree on container digest, CPU count, or task set; you would be measuring the envelope, not the model.

If you want a place to run this without a cloud bill, MonkeyCode offers free model access and a free server option that are enough to host the harness itself; check the project's own pages for the current allocation and machine shape before you fill in the manifest.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 2 — Make the journal the source of truth, not the output

A resume is only safe if the runner can tell what already happened. Keep an append-only journal and let the artifact directory be disposable. The constraint below is the whole trick: one row per (task_id, checkpoint, attempt), so a double-apply becomes a row-level violation instead of a subtle diff.

# faultbench/journal.py  (template — adapt the schema, keep the constraint)
from __future__ import annotations

import hashlib
import sqlite3
import time
from pathlib import Path

SCHEMA = """
CREATE TABLE IF NOT EXISTS journal (
  task_id         TEXT NOT NULL,
  attempt         INTEGER NOT NULL,
  checkpoint      TEXT NOT NULL,
  payload_digest  TEXT NOT NULL,
  wall_ms         INTEGER NOT NULL,
  ts              REAL NOT NULL,
  PRIMARY KEY (task_id, checkpoint, attempt)
);
"""

CHECKPOINTS = ("context_loaded", "patch_drafted", "patch_applied", "tests_run")


def open_journal(path: Path) -> sqlite3.Connection:
    con = sqlite3.connect(path, isolation_level=None)  # autocommit
    con.execute("PRAGMA journal_mode=WAL")
    con.execute(SCHEMA)
    return con


def record(con: sqlite3.Connection, task_id: str, attempt: int,
           checkpoint: str, payload: bytes) -> None:
    if checkpoint not in CHECKPOINTS:
        raise ValueError(f"undeclared checkpoint: {checkpoint}")
    con.execute(
        "INSERT INTO journal VALUES (?,?,?,?,?,?)",
        (task_id, attempt, checkpoint,
         hashlib.sha256(payload).hexdigest(), 0, time.time()),
    )

On resume, read the last durable checkpoint for (task_id, attempt) and continue from the next one. If the row is missing, treat the state as unknown and fail loudly. Guessing is how you get the double migration.

Step 3 — Inject faults at named checkpoints only

Random kills produce unreproducible numbers. Kill at declared checkpoints, from an environment variable, so every interruption is a rerun of a known state.

# faultbench/cli.sh  (template — not executed for this article)
export FAULTBENCH_DB=runs/a-treat/journal.db

# positive control: no faults at all
uv run faultbench run --manifest manifest.yaml --faults off --out runs/a

# negative control: die before anything is written
uv run faultbench run --manifest manifest.yaml \
  --faults context_loaded:hard_exit --out runs/a-neg

# treatment: die immediately after the patch is applied
uv run faultbench run --manifest manifest.yaml \
  --faults patch_applied:hard_exit --repeat 3 --out runs/a-treat

uv run faultbench score --journal runs/a-treat/journal.db \
  --verdicts runs/a-treat/verdicts.json

Three repeats is not statistics. It is the minimum that tells you whether your own harness is deterministic, which you need before you say anything about a model.

Step 4 — Score survival, duplication, and replay cost

Three metrics, computed from the journal plus verdicts, nothing else:

RSR — resume survival rate. Interrupted tasks that end in a correct pass after resume, divided by all interrupted tasks. Manual edits count as failure.
Double-apply tasks. Tasks with two different payload digests at the same checkpoint. This is an idempotency defect, not a slow run.
Replay cost. Extra tokens and wall time spent re-deriving work that the journal already knew about.

# faultbench/score.py  (template)
import json
import sqlite3


def summarize(journal_db: str, verdicts_json: str) -> dict:
    con = sqlite3.connect(journal_db)
    verdicts = json.load(open(verdicts_json))

    per_task: dict[str, dict] = {}
    for task_id, attempt, checkpoint, digest in con.execute(
        "select task_id, attempt, checkpoint, payload_digest from journal"
    ):
        t = per_task.setdefault(task_id, {"attempts": set(), "writes": {}})
        t["attempts"].add(attempt)
        t["writes"].setdefault(checkpoint, set()).add(digest)

    interrupted = {k for k, v in verdicts.items() if v.get("interrupted")}
    survived = sum(
        1 for k in interrupted if verdicts[k]["status"] == "pass"
    )
    duplicated = sum(
        1 for t in per_task.values()
        if any(len(digests) > 1 for digests in t["writes"].values())
    )
    return {
        "rsr": survived / max(1, len(interrupted)),
        "double_apply_tasks": duplicated,
        "attempts_total": sum(len(t["attempts"]) for t in per_task.values()),
    }

Report the raw journal alongside the numbers. A rate without its ledger is a screenshot with extra steps.

Controls that keep the run honest

Control	Setup	Expected result	What failure means
Positive	Faults off	RSR = 1.0	Your harness, not the model, is broken
Negative	Kill at `context_loaded`	Zero writes recorded	Something wrote before the first checkpoint
Replay	Resume twice from one checkpoint	Second resume is a no-op	Journal is not authoritative
Silent	Kill between checkpoints	Loud `unknown_state` error	The runner is guessing

Why these numbers are not marketing

A marketing number is computed in someone else's envelope and reported without its dependencies. These three are computed in yours, from a manifest you can publish, and they degrade when your own code degrades. That is the point: RSR is partly a property of the harness, and pretending otherwise is how benchmark claims turn into aesthetics.

Where this bench is wrong

Checkpoint granularity dominates the result. A bench with four checkpoints will show better recovery than one with forty, on identical infrastructure. If your tasks are single-shot and uninterruptible, durability is undefinable and this whole harness is noise.

Free compute is also noisy by design. Reclaim events, cold starts, and quota resets are exactly the variance you are trying to measure, so never compare runs across days without re-reading the current allocation. And three repeats per fault point give you a sanity check, not a confidence interval.

Who should not use this

Skip it if you need a public leaderboard, if you cannot rerun a task three times, or if your task set is not deterministic enough that a "pass" means the same thing on Monday and Friday. Skip it too if your real failure mode is a bad review, not a dead process — you would be measuring the wrong interruption.

Run it on something small first

Take five tasks, two fault points, one afternoon. If the positive control is not exactly 1.0, stop and fix the harness before you look at any model. If it is, you now have a durability number that survives someone else trying to reproduce it.

Bring the journal, not the slogan.

Your Benchmark Has No Cost Axis: Score Free Endpoints by Quota per Accepted Patch

Casey Zhang — Mon, 26 Oct 2026 11:19:16 +0000

A team I traded notes with ran the same 40-task patch suite against two free endpoints. Endpoint A passed 71% of tasks, endpoint B passed 64%. They shipped A into their CI helper. Twenty-two days later A's monthly allowance was gone and the helper started failing open — while B, the "worse" endpoint, would have finished the month with quota to spare.

The pass rate was not wrong. It was just answering a different question than the one that mattered. On a free tier, the budget is finite, so the objective is not patches per run — it is patches per unit of quota.

Below is the accounting layer I add to agent benchmarks now: a spend ledger, three derived metrics, and a set of controls that keep the numbers from turning into marketing.

Why a pass rate without a cost axis misleads

A single number collapses two very different failure modes. An endpoint that fails a task on the first attempt and then fails it again on retries has burned budget for nothing. An endpoint that fails once and succeeds on attempt two has burned some budget for something.

Pass rate sees those as identical. Your quota does not.

There is a second distortion: retry policy. If your harness allows three attempts, an endpoint that flails is charged three times, and the flailing is silently hidden inside a "pass". Change the attempt budget and the ranking can flip. That is a harness artifact, not a capability difference.

The three metrics worth reporting

Fix the task set, fix the attempt budget, then report all three of these together. None of them is meaningful alone.

Acceptance rate — accepted tasks ÷ total tasks. The familiar number. Keep it for continuity.
QAP (quota per accepted patch) — total tokens consumed ÷ accepted tasks. This is the cost axis. Lower is better.
Retry tax — tokens spent on attempts that did not produce the accepted patch ÷ total tokens. A high retry tax means your pass rate is being bought with silent spend.

Add a fourth flag, censoring: did the run hit your token cap before finishing the task set? A censored run's acceptance rate is not comparable to an uncensored one, because the hardest tasks never got their full attempt budget.

Step 1: Freeze the task set and the attempt budget

Write both to a file and hash it. If the hash changes, the run is a new benchmark and the old numbers do not carry over.

sha256sum tasks/frozen.jsonl harness_config.json | tee runs/2026-10-26.manifest.sha256

harness_config.json should pin max_attempts, temperature (0 for comparability), cap_tokens, and the prompt template version. Nothing that can drift silently.

Step 2: Instrument the call, not the result

The ledger records one row per attempt, not one row per task. That is the whole trick — retries are where the cost hides.

# spend_ledger.py — template, not a tested library. Adapt the fields to your harness.
from dataclasses import dataclass
import json, time

@dataclass
class Attempt:
    run_id: str
    task_id: str
    arm: str              # endpoint label, e.g. "free-arm-a"
    attempt: int          # 1-based retry index
    prompt_tokens: int
    completion_tokens: int
    wall_ms: int
    accepted: bool        # set by YOUR checker, never by the model
    checker_error: str | None = None

    @property
    def tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

def run_task(call, checker, task, arm, run_id, max_attempts=3):
    rows = []
    for i in range(1, max_attempts + 1):
        t0 = time.monotonic()
        resp = call(task.prompt, temperature=0)
        wall = int((time.monotonic() - t0) * 1000)
        verdict = checker(task, resp.text)      # runs the test suite, returns pass/fail
        rows.append(Attempt(
            run_id=run_id, task_id=task.id, arm=arm, attempt=i,
            prompt_tokens=resp.usage.prompt_tokens,
            completion_tokens=resp.usage.completion_tokens,
            wall_ms=wall, accepted=verdict.ok,
            checker_error=None if verdict.ok else verdict.reason[:200],
        ))
        if verdict.ok:
            break
    return rows

If your provider does not return usage, log the request and response text and estimate tokens from a tokenizer you keep in the repo. Estimating is fine — but state that you estimated, and use the same estimator for every arm.

Step 3: Aggregate with censoring made explicit

# summarize.py — same caveat: template code, run it against your own data first.
def summarize(rows, cap_tokens):
    spend = sum(r.tokens for r in rows)
    accepted_ids = {r.task_id for r in rows if r.accepted}
    accepted = len(accepted_ids)
    first_try = sum(1 for r in rows if r.accepted and r.attempt == 1)
    wasted = sum(r.tokens for r in rows if not r.accepted)

    return {
        "tasks": len({r.task_id for r in rows}),
        "accepted": accepted,
        "acceptance_rate": round(accepted / max(1, len({r.task_id for r in rows})), 3),
        "tokens": spend,
        "qap": round(spend / accepted, 1) if accepted else None,
        "retry_tax": round(wasted / spend, 3) if spend else None,
        "first_try_share": round(first_try / accepted, 3) if accepted else None,
        "censored": spend >= cap_tokens,
    }

Run each arm with the same cap and the same order. Order matters more than people expect: a per-minute rate limit turns task ordering into a latency confounder, and latency is the quiet reason harnesses crash mid-run.

python -m bench.run --arm free-arm-a --tasks tasks/frozen.jsonl \
  --max-attempts 3 --cap-tokens 250000 --out runs/2026-10-26-a.jsonl

Step 4: Add the controls that make the table honest

Four controls, all cheap:

Negative control. One task whose correct patch you already know fails the checker. If an arm "passes" it, your checker is broken and every number above is void.
Floor task. A one-line fix any competent agent should pass. It measures harness overhead and token accounting, not capability.
Paired arms. Run both endpoints on the same task order in the same window. Sequential runs on different days pick up provider drift and file it as endpoint quality.
Retry budget parity. Same max_attempts everywhere. Three for one arm and one for another is not a comparison, it is two benchmarks sharing a folder.

Step 5: Read the numbers as a decision table, not a podium

Arm	Acceptance	QAP (tokens/patch)	Retry tax	Censored	Practical read
A	0.71	6,400	0.44	no	Wins on quality, eats quota on retries
B	0.64	3,100	0.12	no	Loses 7 points, survives the month
C	0.68	—	—	yes	Unusable: cap hit before the suite finished

A is the right pick for a burst workload with a deadline. B is the right pick for a background job that must keep running unattended. C should be dropped from the table entirely — a censored run is not a low score, it is a missing measurement.

Where a free tier genuinely fits this method

This accounting layer exists because finite allowances are the normal case, not an edge case. MonkeyCode is relevant here for two operator-stated reasons: free model access, and a free server option to host the harness itself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes the free model allowance as 10 million tokens; confirm the current figure and its reset behavior in-product before you put it in a table, because free-tier terms move and a stale quota number invalidates every QAP figure derived from it. The server option matters for a narrower reason: running the harness next to the endpoint removes your laptop's thermal throttling and home upload speed from the latency column, so wall_ms measures something closer to the provider.

Neither of those makes the numbers better. They just make the measurement less contaminated — which is the only thing a benchmark host should do.

Limitations, stated up front

Token accounting is provider-shaped. Cached prompt tokens, tool-call overhead, and system-prompt reuse are billed differently across providers. QAP compares arms only when measured by the same accounting method.
Free-tier availability is not a guarantee. Rate limits, model availability, and allowances change without notice. Re-run the manifest before you compare a new month's numbers to an old table.
QAP is not a price. It is quota consumption per accepted patch. If you later map it to money, keep the mapping in a separate column.
Small suites produce wide intervals. With 40 tasks, a 7-point gap is inside the noise of a single re-run. Report the re-run spread beside the point estimate.

Who should not use this

Skip this method if you need a latency SLA — an unattended free server and a free model allowance are the wrong substrate for p99 guarantees. Skip it if your checker is still a human eyeballing diffs; QAP without an automated checker just measures how long a person stared at the screen. And skip it if you cannot reserve quota for the benchmark separately from production, because a benchmark that starves your real workload will be quietly disabled by the first person it inconveniences.

If you already have a checker and a frozen suite, the ledger is about thirty lines. Start by adding attempt and the two token fields to whatever you currently log — that alone usually explains why last month's "best" endpoint stopped answering.

Two Runs, Six Points Apart: Measure the Re-Run Spread Before You Compare Endpoints

Casey Zhang — Tue, 20 Oct 2026 09:22:48 +0000

You run the same 60-task suite twice against the same endpoint. First pass: 71%. Second pass: 77%. Nothing changed — not the model, not the prompts, not the tasks.

Now someone drops a screenshot of a different endpoint at 74% and asks which one wins. You cannot answer that with single runs. Before you compare anything, you need to know how much your number moves when nothing changes. That spread is your noise floor, and any delta smaller than it is a coin flip wearing a percentage sign.

This post is a repeatable method for measuring that spread, computing the smallest difference your suite can actually resolve, and deciding when a comparison is worth reporting. The reference implementation is deliberately boring. Boring is the point.

Why a single run cannot answer a comparison question

A pass rate is a summary of Bernoulli trials. Each task either passes or fails, and each repeat is a fresh draw. With 60 tasks and a true pass rate near 0.75, the run-to-run standard deviation alone is roughly 6 percentage points — so a 71% and a 77% are entirely compatible with an unchanged system.

Add harness drift on top: a timeout that clipped one task, a retry that got counted as a pass, a container that came up with a cold cache on the first run. Those do not show up in the headline number. They show up as movement you cannot explain.

The fix has two halves. First, repeat each task K times so you have a distribution instead of a point. Second, compute the minimum detectable difference (MDD) for your suite, so you know which deltas are even worth discussing.

Step 1: Freeze the task set, the harness, and the scoring rule

Write down the task IDs, the harness commit, the max-turn or max-timeout setting, and the exact command that produces a pass/fail verdict. Hash the task directory and store the hash next to your results.

# Freeze the inputs once, per experiment
find tasks/ -type f -name '*.json' | sort | xargs sha256sum | sha256sum > tasks.sha256
git -C harness rev-parse HEAD > harness.rev

Any rerun that does not reproduce tasks.sha256 is a different experiment. Do not merge its numbers into the same table — file it separately and label it. This one habit kills most "the score went up and we don't know why" tickets.

Step 2: Run K repeats per task, not K repeats per suite

This is the step people skip. If you run the whole suite five times, you get five suite-level numbers and a standard deviation computed on 5 samples — too few to say anything, and blind to which tasks are unstable.

Run each task K times instead. K=5 is a reasonable start for a 40–80 task suite; K=3 tells you almost nothing about per-task instability but is still better than 1. Write results as JSON Lines so an interrupted run can resume instead of restarting from zero.

#!/usr/bin/env bash
# run_spread.sh — illustrative driver; adapt the run command to your harness
set -euo pipefail
K=${K:-5}
OUT="runs/$(date -u +%Y%m%dT%H%M%SZ).jsonl"
mkdir -p runs

for task in tasks/*.json; do
  id=$(basename "$task" .json)
  for k in $(seq 1 "$K"); do
    start=$(date +%s)
    if timeout 900 ./harness/run --task "$task" --json > /tmp/out.json 2>/tmp/err.log; then
      status=pass
    else
      status=fail
    fi
    elapsed=$(( $(date +%s) - start ))
    printf '{"task":"%s","repeat":%d,"status":"%s","seconds":%d}\n' \
      "$id" "$k" "$status" "$elapsed" >> "$OUT"
  done
done
echo "wrote $OUT"

Two details matter more than they look. First, timeout must be a hard ceiling so a hung task cannot silently skew your wall-clock budget. Second, write the log line after the verdict, never before — a crash mid-run should leave a missing record, not a fake pass.

Step 3: Score per task, with an interval

Aggregate to a per-task pass rate p_i = passes / K, then report the suite mean with a Wilson interval. Wilson behaves sanely near 0 and 1, where the normal approximation falls apart — and agent suites always have a few tasks sitting at 0/5.

# spread.py — reference implementation (Python 3.10+), unexecuted here
import json, math, statistics
from collections import defaultdict

def load(path):
    per = defaultdict(list)
    with open(path) as f:
        for line in f:
            r = json.loads(line)
            per[r["task"]].append(r["status"] == "pass")
    return {t: sum(v) / len(v) for t, v in per.items()}

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - m) / d, (c + m) / d)

def suite_interval(per_task, k):
    # average of per-task Wilson bounds, a conservative suite-level band
    los, his = zip(*(wilson(round(p * k), k) for p in per_task.values()))
    return statistics.mean(per_task.values()), statistics.mean(los), statistics.mean(his)

Report the band, not the point. "74.2% (95% band 69.8–78.1)" is a claim you can defend. "74.2%" invites a fight you will lose.

Step 4: Compute the minimum detectable difference

Now the number that actually governs your comparisons. Take two endpoints, A and B, both run with K repeats over the same N tasks. For each task, compute d_i = p_A_i - p_B_i. The MDD is roughly 2.8 standard errors of the mean difference — that is the two-sided alpha=0.05, 80% power threshold.

def mdd(per_a, per_b):
    tasks = sorted(set(per_a) & set(per_b))
    d = [per_a[t] - per_b[t] for t in tasks]
    n = len(d)
    se = statistics.stdev(d) / math.sqrt(n) if n > 1 else float("nan")
    return statistics.mean(d), 2.8 * se, n

A worked illustration, using made-up numbers: if mean(d) = +0.03 and MDD = 0.09, you have measured a 3-point edge on a suite that cannot resolve anything under 9 points. The honest conclusion is "no detectable difference at this sample size," not "B is slightly better."

Three levers shrink the MDD: more tasks, more repeats per task, and tasks that are less flaky. Only the first two are fully under your control.

Step 5: Use a paired rule before you call a winner

Paired designs are far more sensitive than comparing two independent suite means, because they cancel task difficulty. Count the tasks where A beats B and the tasks where B beats A; ties drop out. Then test those discordant counts.

def sign_test(wins_a, wins_b):
    n = wins_a + wins_b
    if n == 0:
        return 1.0
    k = min(wins_a, wins_b)
    tail = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)

A decision table keeps the rule out of your gut:

Observation	Report
\|mean(d)\| < MDD	"No detectable difference at N tasks, K repeats"
\|mean(d)\| ≥ MDD and sign test p < 0.05	"A ahead by X points, in the same direction on W of W+L discordant tasks"
\|mean(d)\| ≥ MDD but sign test p ≥ 0.05	"Led by a few heavy tasks; not a suite-wide effect"
Per-task rates cluster at 0/5 and 5/5	Suite is easy or saturated; widen difficulty before comparing

That last row is the one people ignore. A suite where most tasks are 0/5 or 5/5 has almost no resolving power, no matter how many repeats you run.

Step 6: Publish the spread, not just the mean

Ship a results table with columns for suite pass rate, Wilson band, MDD, discordant pair counts, and the input hashes from Step 1. Anyone can recompute your conclusion from that table. Nobody can recompute it from a screenshot.

Keep the raw JSONL. It is the only thing that lets you answer "was that run before or after the timeout change?" three weeks later.

Controls that quietly break repeatability

Four knobs account for most unexplained movement:

Retry accounting. Decide once whether a retried task counts as one attempt or two, and apply it to every endpoint.
Timeouts. A 900s ceiling that clips 8% of runs is a scoring parameter, not an implementation detail.
Temperature and sampling. Whatever you set, log it in the result file, because it is not visible in the aggregate.
Cold starts. The first run against any fresh runner includes setup cost the later ones do not.

Where a free runner fits

The whole method is cheap in thinking and expensive in wall-clock: N tasks × K repeats × 2 endpoints. A 60-task suite at K=5 across two endpoints is 600 runs, and most of that time is the harness waiting on a model.

That is the honest case for using free model access and a free server option to run the loop, rather than building runner infrastructure first. MonkeyCode exposes both — free model access and a free server option — as of this writing; quota, capacity, and terms are the operator's to state and change, so verify them yourself before you plan a 600-run experiment around them. I used the free server as the place to run a spread check like the one above, which keeps a noisy experiment off my laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two caveats if you go that route. Shared capacity means wall-clock variance is higher than on a box you control, so log seconds per run and treat outliers as a harness signal. And free-tier terms are a moving target: pin the terms you relied on in your run manifest alongside the task hash.

Limitations and who should skip this

K=5 over 60 tasks is not a certification. It resolves differences in the high single digits at best, and it says nothing about tasks your suite does not contain. If you need a procurement-grade claim, this method is a screening tool, not evidence.

Skip it if your test suite is itself nondeterministic in ways you cannot pin (live third-party APIs in assertions, wall-clock-dependent tests, network flakiness you have not isolated). Repeats will measure your test noise and you will confidently report it as model noise. Fix the suite first.

Also skip it if a 10-point difference is obvious to the naked eye and you only need a direction. Methodology has a cost; spend it on the close calls, where the argument actually lives.

Copy-paste checklist

[ ] Hash the task set and record the harness commit
[ ] Run K repeats per task, JSONL output, hard timeout
[ ] Compute per-task pass rates and a Wilson band
[ ] Compute MDD from paired per-task differences
[ ] Apply the sign test before naming a winner
[ ] Publish spread, MDD, discordant counts, and input hashes

If you run this once, you will stop arguing about three-point gaps. If you run it twice, you will start asking why your suite cannot resolve the differences you care about — which is the more useful question anyway.

Free Model Endpoints Need a Contract Bench, Not a Leaderboard Screenshot

Casey Zhang — Mon, 19 Oct 2026 08:09:12 +0000

You wire a small internal tool to a free model endpoint. The first call returns clean JSON. The second wraps the same object in a Markdown fence. The third times out at 30 seconds. The fourth reports 900 completion tokens for a 40-token answer.

The demo still looks fine. Your CI job does not.

That gap is why a free endpoint needs a contract bench. You are not asking whether the model is the smartest one available. You are asking whether the endpoint returns the shape your code needs, repeatedly, inside the time and token budget you can afford. Those are different questions. Only the second one protects a build.

MonkeyCode is an open-source project. Its current outreach advertises free model access, a free server option, and 10 million free tokens. Treat all three as time-bound commercial terms, not technical guarantees; verify the current offer before you depend on it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For this workflow, the free access is useful as a probe target, not as proof of quality. The bench below stays useful if you swap in another endpoint.

1. Write the contract before the prompt

A leaderboard asks which model wins. A contract bench asks which promises survive repetition. Start with the promises your code already makes.

Contract	Probe	Pass evidence	Failure class
JSON object	Ask for one object with two keys	Whole response parses; no fence stripping	`format_drift`
Schema	Fixed key set and value	Keys match exactly	`schema_miss`
Truncation	Small `max_tokens` with a long request	`finish_reason` is present and plausible	`truncation_ambiguity`
Token accounting	Known short prompt	`0 < completion_tokens <= max_tokens`	`accounting_anomaly`
Timeout	Client timeout enforced	Request ends within budget	`timeout`
Error contract	Malformed request	Non-2xx with a structured body	`error_contract`
Streaming, if used	Streamed response	Final event arrives	`stream_hang`

The table is deliberately narrow. It does not test reasoning, factuality, or code quality. Those need a different dataset. This bench tests whether the endpoint is a stable dependency for one client.

2. Build a small, frozen probe set

Start with four cases. Expand to at least 20 before you make a team decision. The shape matters more than the count.

Use four groups:

Strict JSON extraction, with and without a code-fence instruction.
Truncation pressure, where the answer cannot fit in the token budget.
Unicode and whitespace, because transport layers normalize text in surprising ways.
A malformed request control, sent outside the main run.

Store the cases in JSONL with an id, group, prompt, and expect field. Hash the file. Put the hash in your notes. If the fixture changes, the old numbers are a different experiment.

Do not put secrets, customer text, or production prompts in the fixture. A free service may log requests. Treat every probe as public.

3. Run it with controls

Controls are what separate a bench from a screenshot.

Set temperature to 0 when the endpoint accepts it. If it ignores the value, record that fact.
Fix max_tokens, client timeout, and client version for the whole run.
Run three attempts per case. Do not retry inside the measurement; record the retry as a new attempt.
Repeat in two windows at least two hours apart. Free capacity is not stationary.
Shuffle case order with a fixed seed. Do not let one slow request poison every later sample.
Log status, latency, usage, finish reason, and a hash of the raw body. Do not log the API key.
Send one malformed request. A 200 response is a contract failure, not a quirky success.

With 12 attempts, a p95 latency is a range check, not an SLA estimate. That is fine. You are looking for failure classes and gross instability, not a precise capacity number.

4. A harness you can adapt

The template below uses only the Python standard library. It is an unexecuted template: it assumes a chat-completions-shaped payload and a BASE_URL. Replace call_endpoint for a different API. Keep the record shape and scoring logic.

import hashlib
import json
import os
import random
import time
import urllib.error
import urllib.request
from pathlib import Path

BASE_URL = os.environ['BASE_URL'].rstrip('/')
API_KEY = os.environ['API_KEY']
MODEL = os.environ.get('MODEL', 'default')
TIMEOUT_S = 25.0
MAX_TOKENS = 64

CASES = [
    {
        'id': 'json-01',
        'group': 'strict_json',
        'prompt': 'Return only a JSON object with keys ok and count. count must be 2.',
        'kind': 'json',
    },
    {
        'id': 'json-02',
        'group': 'strict_json',
        'prompt': 'Return only a JSON object with keys ok and count. count must be 2. Do not use a code fence.',
        'kind': 'json',
    },
    {
        'id': 'long-01',
        'group': 'truncation',
        'prompt': 'Write a 1000-word explanation of HTTP caching.',
        'kind': 'text',
    },
    {
        'id': 'unicode-01',
        'group': 'unicode',
        'prompt': 'Return the exact text: café / 第二行 / end',
        'kind': 'text',
    },
]

def call_endpoint(prompt: str) -> dict:
    payload = {
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
        'max_tokens': MAX_TOKENS,
    }
    req = urllib.request.Request(
        f'{BASE_URL}/chat/completions',
        data=json.dumps(payload).encode('utf-8'),
        headers={
            'Authorization': f'Bearer {API_KEY}',
            'Content-Type': 'application/json',
        },
        method='POST',
    )
    started = time.perf_counter()
    status = None
    raw = ''
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT_S) as resp:
            status = resp.status
            raw = resp.read().decode('utf-8')
    except urllib.error.HTTPError as exc:
        status = exc.code
        raw = exc.read().decode('utf-8')
    except Exception as exc:
        return {
            'status': None,
            'text': '',
            'usage': {},
            'finish_reason': None,
            'elapsed_ms': round((time.perf_counter() - started) * 1000, 1),
            'raw_sha256': None,
            'transport_error': type(exc).__name__,
        }

    body = {}
    try:
        parsed = json.loads(raw)
        if isinstance(parsed, dict):
            body = parsed
    except json.JSONDecodeError:
        pass

    choice = (body.get('choices') or [{}])[0]
    message = choice.get('message') or {}
    return {
        'status': status,
        'text': message.get('content') or '',
        'usage': body.get('usage') or {},
        'finish_reason': choice.get('finish_reason'),
        'elapsed_ms': round((time.perf_counter() - started) * 1000, 1),
        'raw_sha256': hashlib.sha256(raw.encode('utf-8')).hexdigest(),
        'transport_error': None,
    }

def score(case: dict, rec: dict) -> str:
    if rec['transport_error']:
        return 'transport_error'
    if rec['status'] != 200:
        return 'http_error'
    if rec['elapsed_ms'] > TIMEOUT_S * 1000:
        return 'timeout'
    if case['kind'] == 'json':
        try:
            value = json.loads(rec['text'])
        except json.JSONDecodeError:
            return 'format_drift'
        if not isinstance(value, dict):
            return 'schema_miss'
        if set(value) != {'ok', 'count'}:
            return 'schema_miss'
        if value['count'] != 2:
            return 'content_miss'
    completion = rec['usage'].get('completion_tokens')
    if isinstance(completion, int) and completion > MAX_TOKENS:
        return 'accounting_anomaly'
    return 'pass'

def main() -> None:
    random.Random(7).shuffle(CASES)
    rows = []
    for repetition in range(3):
        for case in CASES:
            rec = call_endpoint(case['prompt'])
            rec.update({
                'case_id': case['id'],
                'group': case['group'],
                'repetition': repetition,
                'result': score(case, rec),
            })
            rows.append(rec)

    Path('results.jsonl').write_text(
        chr(10).join(json.dumps(row, ensure_ascii=False) for row in rows),
        encoding='utf-8',
    )

    counts = {}
    for row in rows:
        key = '{}:{}'.format(row['group'], row['result'])
        counts[key] = counts.get(key, 0) + 1

    latencies = sorted(row['elapsed_ms'] for row in rows)
    p95 = latencies[min(len(latencies) - 1, int(0.95 * (len(latencies) - 1)))]
    print(json.dumps({'counts': counts, 'p95_ms': p95}, indent=2))

if __name__ == '__main__':
    main()

Save it as bench.py and run it with BASE_URL='https://your-endpoint.example' API_KEY='...' python bench.py. Run the malformed-request control separately. Change the payload to invalid JSON and assert that the status is 4xx and the body is parseable. Do not fold that result into the pass count.

5. Read counts, not a composite score

A single score hides the failure you need to fix. Report a small set of separate numbers.

Metric	How to report it	Why it matters
Strict JSON	`passed / attempts` by group	Shows format drift directly
Timeout	`count / attempts`	Exposes client and capacity failures
Token anomaly	`count`	Catches impossible usage accounting
Latency	p50 and p95, plus max	Shows budget risk without hiding outliers
Error contract	`valid 4xx / malformed requests`	Shows whether failures are machine-readable
Raw evidence	fixture hash and body hashes	Lets another person reproduce the run

Use counts when the denominator is small. 9/12 strict-JSON passes, 3 format_drift is useful. 75% quality is not. The second sentence invites a comparison the bench cannot support.

6. Why these numbers are not marketing

Marketing numbers compress time, sample size, and failure detail. They report one score, one date, one model label. That is easy to read and easy to misuse.

A contract bench reports a fixture hash, a client version, a denominator, a failure histogram, and the time windows. Those details are boring. They are also what lets another developer reproduce your result or falsify it.

Do not turn the bench into a ranking. If you test two endpoints, keep the fixture and client fixed, report each failure class separately, and say how many attempts you made. If the free offer changes, your old number is a historical observation, not a current capability claim.

That is the difference between evidence and a screenshot. A screenshot shows a good moment. A bench shows the conditions under which the moment happened.

7. Decision table

Workload	Minimum evidence	Do not use the free endpoint when
Throwaway prototype	12 attempts, malformed control, classified failures	You cannot send the data under the current terms
CI job	30+ attempts across two windows, p95 under job timeout, fallback path	A timeout blocks every merge
Internal tool	Contract pass counts plus error-budget monitoring	You need a hard latency promise
Production dependency	Contract bench plus fallback, monitoring, and data policy	You need an SLA the free offer does not provide

The table is a gate, not a guarantee. A pass means the endpoint is worth another test. It does not mean the model is correct.

8. Limitations and who should skip this

This bench does not measure reasoning quality, factuality, or code correctness. It will not tell you which model writes better patches. Use a separate evaluation for that.

The sample is small. Twelve or thirty attempts can reveal gross instability, but it cannot prove steady-state capacity. The free offer is also time-bound. The 10 million token figure and the free server option are commercial terms, not protocol guarantees; they can change without notice.

Do not send secrets, regulated data, or customer content to a free endpoint unless the current terms clearly allow it. Do not use this workflow as your only production control. Teams that need deterministic latency, strict data residency, or a contractual SLA should not treat a free endpoint as a production dependency.

9. Where the free server fits

Use the free server option as a disposable probe runner. Keep the fixture, the results JSONL, and the summary in storage you control. If the server resets, you should lose a run, not the evidence.

Inject the API key at runtime. Do not put it in the fixture, the repository, or the result log. Treat the runner as cattle, not a database.

The useful output is not a slogan about a model. It is a sentence like this: under fixture hash abc123, with client version 0.4.1, 7 of 12 strict-JSON attempts passed and 5 failed as format drift across two windows. That sentence survives a demo, a handoff, and a skeptical reviewer.

If you want a low-stakes place to start, the project's current free access and free server offer are enough for one run. Verify the current terms, keep your raw counts, and let the bench decide what happens next.

Your Coding-Agent Benchmark Needs a Run Manifest, Not Another Leaderboard Row

Casey Zhang — Sun, 18 Oct 2026 06:54:28 +0000

A colleague pasted a screenshot into our channel last month: "agent X solves 78% of our internal suite." Someone asked which suite. Nobody knew. The screenshot had a bar, a percentage, and a logo, and it took three people an hour to establish that it was not comparable to anything we had ever run.

That is the normal failure mode of agent benchmarking. Not fraud. Not even sloppiness. Just a number published without the envelope that produced it.

You cannot fix that with a better leaderboard. You fix it by making the number's preconditions a machine-readable artifact that a reviewer can diff. Below is the contract I now require before I read anyone's percentage, including my own.

The number is the last line of the report, not the first

A pass rate is a summary of a run. A run is a bundle of decisions: which tasks, which snapshot, which retry policy, which token budget, which scoring script. Change any of them and the percentage means something else.

So write the bundle down first, hash it, and only then compute the number. If two runs disagree on any envelope field, the correct output is not a delta. It is a refusal.

Step 1: Freeze the slice before you run anything

Write your task IDs to a file, sort them, and hash the file. Not the directory. Not "the SWE tasks we usually use." A single file with stable, sorted content.

# tasks.txt — one stable task id per line, no comments
printf '%s\n' task-0011 task-0023 task-0047 > tasks.txt
sort -o tasks.txt tasks.txt
sha256sum tasks.txt | cut -d' ' -f1
# -> 3f9a...  paste this into the manifest

This one habit kills most benchmark disputes. If a task is added, the hash changes, and the hash change is visible in review. Nobody can quietly improve their score by dropping the tasks that fail.

Pick the slice for a stated reason and write that reason down too. A slice chosen because it was convenient is still fine — as long as the manifest says so and the conclusion stays inside that boundary.

Step 2: Declare the envelope in a manifest

Here is the file I use. It is deliberately boring, because boring files survive contact with reviewers.

{
  "run_id": "2026-10-18-acme-v3-r1",
  "started_at_utc": "2026-10-18T09:12:00Z",
  "operator": "casey",
  "task_slice_sha256": "3f9a...",
  "agent_id": "acme-cli",
  "harness_commit": "9c1b7de",
  "model_id": "provider/model-family",
  "model_snapshot": "<the pinned version string the provider returns>",
  "temperature": 0.0,
  "seed": 7,
  "max_output_tokens": 4096,
  "budget_tokens_per_task": 60000,
  "tool_timeout_s": 120,
  "retry_policy": "no retries on test failure; 2 retries on HTTP 5xx",
  "env_image_digest": "sha256:...",
  "scoring_script_sha256": "...",
  "runner": "free-server",
  "provider_region": "unspecified"
}

Two details earn their keep. model_snapshot is whatever version string the provider exposes that day — a family name alone is not an envelope field, because families move under you. retry_policy matters more than temperature in practice; an agent that gets two extra attempts at a failing test suite can look materially better than the same agent with one.

Step 3: Make the checker refuse bad comparisons

This is the whole artifact. Everything else is supporting cast.

#!/usr/bin/env python3
"""check_comparable.py — refuse to diff two runs with different envelopes."""
import json, sys
from pathlib import Path

ENVELOPE = {
    "task_slice_sha256", "agent_id", "harness_commit", "model_id",
    "model_snapshot", "temperature", "seed", "max_output_tokens",
    "budget_tokens_per_task", "tool_timeout_s", "retry_policy",
    "env_image_digest", "scoring_script_sha256",
}
PROVENANCE = {"run_id", "started_at_utc", "operator", "runner", "provider_region"}

def load(p):
    return json.loads(Path(p).read_text())

def main(a, b):
    ma, mb = load(a), load(b)
    blocking = sorted(k for k in ENVELOPE if ma.get(k) != mb.get(k))
    drift = sorted(k for k in PROVENANCE if ma.get(k) != mb.get(k))
    for k in blocking:
        print(f"BLOCK {k}: {ma.get(k)!r} != {mb.get(k)!r}")
    for k in drift:
        print(f"note  {k}: {ma.get(k)!r} != {mb.get(k)!r}")
    if blocking:
        print(f"\nnot comparable: {len(blocking)} envelope field(s) differ")
        return 3
    print("\ncomparable envelope: diff the outcomes")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Exit code 3 is the important line. In CI it fails the job. In review it is the comment that says "these two numbers are different experiments."

Field that differs	Verdict
`task_slice_sha256`	Not comparable — different question
`model_snapshot`	Not comparable — different subject
`temperature`, `seed`, `budget_tokens_per_task`	Not comparable — different conditions
`env_image_digest`, `scoring_script_sha256`	Not comparable — different referee
`runner`, `provider_region`, `started_at_utc`	Comparable, note it in the write-up

The last row is where people over-correct. Recording that run A used a laptop and run B used a free server is useful context, not a disqualification — unless your conclusion depends on latency or hardware.

Step 4: Report discordant pairs, not just averages

Two runs over the same slice produce paired outcomes. Averages throw that structure away.

#!/usr/bin/env python3
"""paired.py — per-task deltas with a seeded bootstrap interval."""
import json, random, sys

def load(p):
    return {r["task_id"]: r for r in json.load(open(p))}

def main(pa, pb, iters=10000, seed=7):
    a, b = load(pa), load(pb)
    shared = sorted(set(a) & set(b))
    if set(a) != set(b):
        print(f"WARN task sets differ; using {len(shared)} shared tasks")
    deltas = [int(b[t]["passed"]) - int(a[t]["passed"]) for t in shared]
    a_pass_b_fail = sum(1 for t in shared if a[t]["passed"] and not b[t]["passed"])
    b_pass_a_fail = sum(1 for t in shared if b[t]["passed"] and not a[t]["passed"])
    n, rng = len(deltas), random.Random(seed)
    boots = sorted(sum(deltas[rng.randrange(n)] for _ in range(n)) / n
                   for _ in range(iters))
    lo, hi = boots[int(0.025 * iters)], boots[int(0.975 * iters)]
    print(f"n={n} delta={sum(deltas)/n:+.3f} 95%CI=[{lo:+.3f},{hi:+.3f}]")
    print(f"A passed / B failed: {a_pass_b_fail}")
    print(f"B passed / A failed: {b_pass_a_fail}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

With 20 tasks and a +0.150 delta, expect something like 95%CI=[-0.050,+0.350] and a discordant split around 2 / 5. That interval straddles zero. The honest sentence is "directionally positive, not established" — and the discordant counts tell you which tasks to read first.

The numbers above are an expected output shape for the script, not measurements from a real run. Run it on your own outcomes before quoting anything.

Step 5: Ship the manifest next to the number

A pass rate without a manifest is a rumor with a percent sign. Publish run.manifest.json, outcomes.json, and the exact command line. A reviewer should be able to check comparability in under a minute without asking you a question.

One practical shortcut: you need a second environment to test whether your envelope survives a change of runner. That is a cheap check with a free-tier setup — run the same slice on your laptop and on rented-or-free infrastructure, then feed both manifests to the checker. MonkeyCode's free model access and free server option are one way to get that second environment without a budget conversation; the operator tells me the free tier is described as a 10M-token allowance plus a free server, which you should re-verify on the current pricing page before you size a run around it, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this does not prove

A matching envelope proves comparability, not quality. Your 20 tasks are still 20 tasks, and the bootstrap interval is indicative rather than inferential at that size.

Slice selection bias survives every hash. If you picked tasks your agent happens to be good at, the manifest will faithfully document a flattering and useless number. Negative controls and pre-registration address that; the manifest only stops you from lying by accident.

Free tiers and hosted runners also add variance you cannot fully pin: quota throttling, shared capacity, and provider-side version drift between two runs on different days. Record the timestamp and rerun the manifest check before you combine them.

Who should skip this

Skip the manifest if you are doing a one-off smoke test to decide whether an agent is worth an afternoon. Skip it for a demo. Skip it if your conclusion is "this looks promising, let's try it on real work."

Adopt it the moment a number leaves your terminal. The first time someone quotes your percentage in a planning meeting, the manifest is the difference between a measurement and a slogan.

Free Tiers Are a Moving Target: Benchmark Them With a Frozen Envelope, Not a Screenshot

Casey Zhang — Sat, 17 Oct 2026 05:53:48 +0000

You ship a fix on Monday. By Thursday a teammate posts a number in the channel: "free tier, 87% pass rate." Nobody can reproduce it. The tasks are gone, the endpoint changed, and the run that produced it quietly dropped 40% of its requests on rate limits. The percentage survives in slides long after the conditions that produced it have evaporated.

Free model access makes this worse, not better. Free capacity is the most volatile thing you will ever benchmark: it moves with load, with quotas, with whatever the provider changed this morning. So the number is only worth something if the conditions travel with it.

This is a methodology piece. Below is a small harness that refuses to print an aggregate until its run envelope is complete, two controls that catch a lying grader, and a decision table for deciding whether a number is usable at all.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding-agent project that offers free model access and a free server option; those two availability claims are what make the workflow below runnable without provisioning anything. I deliberately do not restate quotas, model names, or hardware here — check the project's current pages for today's terms, because that is exactly the kind of data this article tells you not to trust second-hand.

1. Decide what the number is allowed to claim

Before the loop, write the sentence the result will support. If you cannot write it, you are not ready to run.

Bad: "Model A is better than model B."

Good: "On 40 frozen tasks, with a 4k output budget per task and 3 scheduled attempts each, at 2026-10-17T09:00Z, model A produced 27 verifiable passes and 5 invalid runs; model B produced 24 and 2. This supports a claim about this envelope only."

The second sentence is boring. It is also the only kind that survives a re-run a week later.

Two rules make it work. First, tie the claim to a timestamp and an endpoint, not to a brand. Second, treat invalid runs as a first-class result, because on free capacity they are often the dominant signal.

2. Write the envelope before you write the loop

The envelope is a JSON manifest. Seven fields do most of the work, and one of them is the hash of your task file so the dataset cannot drift out from under the number.

{
  "endpoint": "https://<host-from-provider-docs>/v1",
  "model_alias": "<exact alias as listed today>",
  "token_budget_per_task": 4096,
  "max_attempts": 3,
  "temperature": 0.0,
  "server_class": "free-tier-server, 1 vCPU, 2 GB RAM (as provisioned)",
  "concurrency": 1,
  "run_date_utc": "2026-10-17T09:00:00Z",
  "retry_policy": "retry on 429/5xx/timeout with 2s, 8s backoff; never retry on 4xx",
  "task_set_sha256": "<filled in by the harness>",
  "invalid_rate_ceiling": 0.10
}

Notice what is absent: no "best", no "fastest", no score. An envelope is a description of a measurement, not a verdict.

The invalid_rate_ceiling is the line you draw in advance. When invalid runs exceed it, the aggregate is not a weak result — it is no result.

3. Build the harness that refuses to aggregate

The core discipline is a gate: if the envelope is incomplete, or too many calls failed, the script exits non-zero and prints why. Never let a denominator shrink silently.

# harness.py — example for a chat-completions style endpoint.
# Run: python harness.py --manifest envelope.json --tasks tasks.jsonl
# Not executed in this article; adapt error handling to your provider's docs.
import argparse, hashlib, json, os, sys, time, urllib.request, urllib.error

REQUIRED = ["endpoint", "model_alias", "token_budget_per_task", "max_attempts",
            "temperature", "server_class", "run_date_utc", "retry_policy",
            "invalid_rate_ceiling"]

def sha256_file(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def call(m, prompt, key):
    body = json.dumps({"model": m["model_alias"],
                       "messages": [{"role": "user", "content": prompt}],
                       "temperature": m["temperature"],
                       "max_tokens": m["token_budget_per_task"]}).encode()
    req = urllib.request.Request(m["endpoint"].rstrip("/") + "/chat/completions",
        data=body, headers={"Content-Type": "application/json",
                            "Authorization": f"Bearer {key}"})
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=180) as r:
            payload = json.loads(r.read())
        return {"status": 200, "latency_s": time.time() - t0, "payload": payload}
    except urllib.error.HTTPError as e:
        return {"status": e.code, "latency_s": time.time() - t0, "payload": None}
    except Exception as e:
        return {"status": 0, "latency_s": time.time() - t0, "payload": None,
                "error": repr(e)}

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--manifest", required=True)
    ap.add_argument("--tasks", required=True)
    ap.add_argument("--out", default="runs.jsonl")
    a = ap.parse_args()

    m = json.load(open(a.manifest))
    missing = [k for k in REQUIRED if m.get(k) in (None, "")]
    if missing:
        sys.exit(f"REFUSING TO RUN — envelope incomplete: {missing}")

    m["task_set_sha256"] = sha256_file(a.tasks)
    json.dump(m, open("envelope.frozen.json", "w"), indent=2, sort_keys=True)

    key = os.environ["PROVIDER_API_KEY"]
    tasks = [json.loads(l) for l in open(a.tasks) if l.strip()]
    scheduled = len(tasks) * m["max_attempts"]  # denominator, fixed up front

    rows, invalid = [], 0
    with open(a.out, "w") as out:
        for t in tasks:
            for attempt in range(1, m["max_attempts"] + 1):
                r = call(m, t["prompt"], key)
                ok = r["status"] == 200 and r["payload"] is not None
                if not ok:
                    invalid += 1
                out.write(json.dumps({"task_id": t["id"], "attempt": attempt,
                                      "status": r["status"], "latency_s": r["latency_s"],
                                      "usage": (r["payload"] or {}).get("usage")}) + "\n")
                rows.append({"task_id": t["id"], "ok": ok})
                if ok:
                    break           # keep attempts cheap, but keep the denominator fixed
                time.sleep(2 if attempt == 1 else 8)

    rate = invalid / scheduled
    print(f"scheduled={scheduled} invalid={invalid} invalid_rate={rate:.3f}")
    if rate > m["invalid_rate_ceiling"]:
        sys.exit("NO AGGREGATE — invalid rate above ceiling; the run describes the tier, not the model")

if __name__ == "__main__":
    main()

The scheduled = len(tasks) * max_attempts line is the whole point. Your pass rate is passes divided by tasks scheduled, and you report it next to the invalid count. If you divide by successful calls instead, a flaky free tier will look better the more it fails.

4. Add two controls before you trust the grader

A grader that always says "pass" produces a beautiful benchmark. Two cheap controls catch it.

Negative control. Add one task whose correct outcome is failure — for example, "return the SHA-256 of an empty string as 64 zero characters." Nothing should pass it. If your report shows a pass on the negative control, stop: your acceptance check is broken, and every other row is noise.

Replay control. Re-run ten task IDs from the same dataset 24 hours later, same envelope, and compare per-task verdicts. Record the flip rate. On free capacity, a flip rate above roughly 20% means your point estimate should be published as a range, or not at all.

Neither control is statistical sophistication. They are plumbing checks, and most published free-tier numbers skip both.

5. Report five numbers, not one

With runs written to JSONL, aggregate offline so you can re-cut without re-spending quota:

Pass rate over the frozen denominator, with the scheduled count printed beside it.
Invalid rate, split by class: 429 / 5xx / timeout / malformed body.
Tokens per solved task — median over passed tasks only, with the count, because it answers "what does a solution cost me" better than an average over everything.
p95 latency on valid calls, since the tail is what breaks interactive loops.
Replay flip rate, from step 4, as a stability annotation on the pass rate.

Publish the task set hash and the frozen envelope.frozen.json alongside these. That is what turns a screenshot into a measurement.

6. A decision table for the result

Observation	What the number means	Action
Negative control passes	Grader is broken	Discard run; fix acceptance checks
Invalid rate > ceiling	You measured the tier's reliability, not the model	Report invalid rate only; do not publish a pass rate
Replay flips > 20%	Envelope is not stable	Publish as interval; increase attempts before comparing
Valid runs, stable replay, n ≤ 40	Directional only	Do not claim differences under ~10 points
Valid runs, stable replay, n ≥ 200, two dates	Comparable within this envelope	Safe to cite with hash and date attached

Read the table before you read the percentage. Most disputes about "who is better" are really disputes about which row of this table the number sits in.

Limitations, and who should skip this

This harness measures behaviour inside one envelope. It says nothing about a model's intrinsic capability, and it will not survive a change to the endpoint, the prompt template, or the token budget. Forty tasks cannot resolve a two-point difference; if that is your question, you need a different instrument, not a bigger table.

Skip this approach if you need SLA-backed numbers for procurement, if you cannot re-run in a week, or if nobody will maintain the task set. A stale harness produces confident garbage.

Also skip it if you only want a demo. Demos are fine — just label them as demos and keep them out of the slide that says "measured".

If you want to try the pipeline without provisioning machines first, MonkeyCode's free model access and free server option are enough to produce a first frozen envelope end to end; read their current terms for what is included today, then write your manifest before you write your loop.

Balance the Outcome Ledger: A Publication Gate for Coding-Agent Benchmarks

Casey Zhang — Fri, 16 Oct 2026 04:51:54 +0000

Last week a teammate pasted a number into a planning doc: 78% pass rate on a 40-task agent suite. Nobody argued. The number was clean, the slide was clean, and the meeting moved on.

Then you opened the run logs. Three tasks never produced an outcome at all. The harness timed out twice, and one grader crashed on a parse error. Those three were dropped from the export, so the denominator quietly became 37. The suite was still called "40 tasks" in the title.

That is not a lie. It is worse than a lie. It is bookkeeping so sloppy that nobody can tell whether the number is true.

The front page this week was full of AI takes, including a good thread on plain AI fatigue. My honest read is that fatigue is not cured by fewer numbers. It is cured by numbers that are cheap to check. So here is a ledger rule and a small gate you can run before any agent score leaves your machine.

The rule: every task that enters must exit exactly once

A benchmark claim is a financial statement. It has entries, exits, and a reconciliation. If the ledger does not balance, the headline is not a measurement — it is a story with digits attached.

Every task attempt must land in exactly one bucket:

pass — the grader's success condition was satisfied.
fail — the task was attempted and the condition was not satisfied.
unscoreable — the harness itself could not produce a verdict, with a named reason class.

The conservation law is one line: pass + fail + unscoreable == tasks_started. And tasks_started must equal the declared dataset size, or you must list the tasks that never started by ID.

Step 1 — Write the manifest (JSON, no dependencies)

Keep the manifest in JSON rather than YAML so the validator runs on a stock Python install with zero packages.

{
  "claim_id": "agent-x-mini-2026-10-16",
  "dataset": {
    "name": "local-40",
    "task_count": 40,
    "task_ids_sha256": "replace-with-digest-from-gate",
    "strata": { "repo_edit": 24, "test_write": 10, "refactor": 6 }
  },
  "metrics": [
    { "name": "pass_rate", "numerator": "outcome == 'pass'", "denominator": "dataset.task_count", "unit": "tasks" },
    { "name": "unscoreable_rate", "numerator": "outcome == 'unscoreable'", "denominator": "dataset.task_count", "unit": "tasks" }
  ],
  "controls": [
    { "task_id": "neg-01", "expected": "fail", "role": "negative" },
    { "task_id": "neg-02", "expected": "fail", "role": "negative" },
    { "task_id": "pos-01", "expected": "pass", "role": "positive" }
  ],
  "envelope": {
    "runner": "monkeycode-free-server",
    "network": "off",
    "wall_clock_s_per_task": 900,
    "max_steps": 40,
    "toolchain_sha": "replace-with-toolchain-digest"
  },
  "rerun_recipe": "python gate.py --manifest bench.manifest.json --runs runs.jsonl"
}

Note two deliberate choices. Every metric is written as an explicit fraction, not as a word like score. And the negative controls carry an expected outcome, so the gate can fail when the harness is broken rather than when the model is.

Step 2 — Emit one JSONL record per attempt

One line per attempt. No aggregation at write time, no filtering at export time. Filtering is where denominators die.

{"run_id":"r1","task_id":"neg-01","attempt":1,"outcome":"fail","reason":null,"wall_clock_s":41}
{"run_id":"r1","task_id":"t-013","attempt":1,"outcome":"unscoreable","reason":"harness_timeout","wall_clock_s":900}
{"run_id":"r1","task_id":"t-027","attempt":1,"outcome":"unscoreable","reason":"grader_error","wall_clock_s":12}
{"run_id":"r1","task_id":"pos-01","attempt":1,"outcome":"pass","reason":null,"wall_clock_s":88}

Keep the allowed reason classes short and fixed: harness_timeout, grader_error, env_flake, missing_artifact. A free-text reason field will be filled with "weird" and "idk" within a week, and then you cannot count anything.

Step 3 — The gate itself

This is the whole validator. Standard library only.

#!/usr/bin/env python3
"""Publication gate for coding-agent benchmark claims. Stdlib only."""
import hashlib, json, math, pathlib, sys

BUCKETS = ("pass", "fail", "unscoreable")
REASONS = {"harness_timeout", "grader_error", "env_flake", "missing_artifact"}


def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = (z / d) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, center - half), min(1.0, center + half))


def read_jsonl(path):
    text = pathlib.Path(path).read_text()
    return [json.loads(line) for line in text.splitlines() if line.strip()]


def main(manifest_path, runs_path):
    manifest = json.loads(pathlib.Path(manifest_path).read_text())
    runs = read_jsonl(runs_path)
    errors, declared = [], manifest["dataset"]["task_count"]

    ids = sorted({r["task_id"] for r in runs})
    digest = hashlib.sha256("\n".join(ids).encode()).hexdigest()
    if digest != manifest["dataset"]["task_ids_sha256"]:
        errors.append("dataset_digest_mismatch: you ran a different suite than you published")
    if len(ids) != declared:
        errors.append(f"dataset_size_mismatch: declared {declared}, observed {len(ids)}")

    by_task = {}
    for r in runs:
        by_task.setdefault(r["task_id"], []).append(r)

    for tid, attempts in sorted(by_task.items()):
        if len(attempts) != 1:
            errors.append(f"multiple_attempts:{tid}={len(attempts)} (retries are attempts, not replacements)")
        last = attempts[-1]
        if last.get("outcome") not in BUCKETS:
            errors.append(f"unknown_outcome:{tid}={last.get('outcome')}")
        if last.get("outcome") == "unscoreable" and last.get("reason") not in REASONS:
            errors.append(f"uncoded_unscoreable:{tid}")

    counts = {b: sum(1 for r in runs if r.get("outcome") == b) for b in BUCKETS}
    if sum(counts.values()) != declared:
        errors.append(f"ledger_does_not_balance: {sum(counts.values())} exits vs {declared} entries")

    for c in manifest.get("controls", []):
        row = by_task.get(c["task_id"], [None])[-1]
        if row is None:
            errors.append(f"control_missing:{c['task_id']}")
        elif row.get("outcome") == "unscoreable":
            errors.append(f"control_inconclusive:{c['task_id']} ({row.get('reason')})")
        elif row.get("outcome") != c["expected"]:
            errors.append(f"control_violated:{c['task_id']} expected {c['expected']} got {row['outcome']}")

    k, lo, hi = counts["pass"], *wilson(counts["pass"], declared)
    scoreable = declared - counts["unscoreable"]
    k2 = counts["pass"]
    lo2, hi2 = wilson(k2, scoreable) if scoreable else (0.0, 0.0)

    print(f"claim       : {manifest['claim_id']}")
    print(f"dataset     : {manifest['dataset']['name']} sha256 {digest[:12]}")
    print(f"entries     : {declared}")
    print(f"exits       : {sum(counts.values())}")
    for b in BUCKETS:
        extra = ""
        if b == "unscoreable":
            detail = sorted((r["reason"] for r in runs if r.get("outcome") == b))
            extra = f" ({', '.join(detail)})"
        print(f"  {b:<12}: {counts[b]}{extra}")
    print(f"pass_rate   : {k}/{declared} = {k/declared:.3f}  95% CI [{lo:.3f}, {hi:.3f}]  <- headline")
    print(f"            : {k2}/{scoreable} = {k2/scoreable:.3f}  95% CI [{lo2:.3f}, {hi2:.3f}]  <- labeled secondary")
    print(f"verdict     : {'BLOCKED' if errors else 'PUBLISHABLE'}")
    for e in errors:
        print(f"  ! {e}")
    return 1 if errors else 0


if __name__ == "__main__":
    a = sys.argv
    sys.exit(main(a[a.index("--manifest") + 1], a[a.index("--runs") + 1]))

Step 4 — Run it and read the report card

python gate.py --manifest bench.manifest.json --runs runs.jsonl

claim       : agent-x-mini-2026-10-16
dataset     : local-40 sha256 3f9c1a7b0e44
entries     : 40
exits       : 40
  pass        : 29
  fail        : 8
  unscoreable : 3 (grader_error, harness_timeout, harness_timeout)
pass_rate   : 29/40 = 0.725  95% CI [0.570, 0.842]  <- headline
            : 29/37 = 0.784  95% CI [0.624, 0.889]  <- labeled secondary
verdict     : PUBLISHABLE

Both lines are real. The headline keeps the full denominator, because an unscoreable task is an outcome of your harness, not an absence from the suite. The secondary line is the number people actually want, and it is fine to show it as long as the label travels with it.

Notice how wide the interval is. On 40 tasks, 29/40 and 30/40 are not different results. If your release notes say "improved from 72% to 75%," the gate just told you that you are reading noise.

Why this makes the number a measurement and not marketing

Five properties separate the two, and the gate enforces all five.

A named numerator and denominator. Not "score," not "performance index." Two counts and a division.
A stated interval. A point estimate without uncertainty is a claim about your confidence, not about the system.
A first-class unscoreable bucket. Attrition is printed next to the result, with reason classes, never dropped at export.
Controls with expected outcomes in the manifest. If a control violates its expectation, the gate blocks the publication, because the instrument is broken.
A re-run recipe a stranger can execute. If someone else cannot reproduce the ledger from your manifest and logs, you published a story.

That last point is the one that costs you an afternoon and saves you a quarter.

Where free compute fits in this loop

The expensive part of a gate like this is not the arithmetic. It is the control slice and the re-run smoke test — the runs that exist purely to prove the instrument works, which feel wasteful when you are paying per call.

This is where a free tier changes the economics of doing it honestly. MonkeyCode provides free model access and a free server option, so the control slice and the smoke test can run without a spend decision blocking them. Check the current terms for the applicable allowance and availability before you depend on it, since free-tier limits and hardware change over time. Incidentally, this article was prepared as part of MonkeyCode's product outreach.

One caution that matters more than the free part. A different runner is a different envelope. Record the runner, network policy, wall clock, and toolchain digest in the manifest, and never compare a pass rate from one envelope to a pass rate from another. A ledger balances inside a single envelope. Across envelopes, you are comparing two different instruments.

If you want to try this on your own suite, run the control slice on the free server and get a PUBLISHABLE verdict from gate.py before you quote anything.

Limitations, and who should skip this

The gate validates bookkeeping, not task quality. Forty weak tasks with a balanced ledger is still a weak benchmark, and a balanced ledger will not fix a suite that leaked into training data. Wilson intervals also assume independent, identically distributed tasks, which agent benchmarks only approximate — treat the interval as a floor on your uncertainty, not a precise bound. And the reason classes above are a starting taxonomy; yours should reflect the failures your harness actually produces.

Skip this if you are doing a vibe check on a personal project — the overhead is real for ten tasks. Skip it too if you need a leaderboard number: a publication gate is necessary and nowhere near sufficient, because it says nothing about a hidden split, seed sensitivity, or task contamination. Fix those separately.

Use it if you are about to put a percentage in a doc that other people will make decisions from. Then do the boring thing first: count the entries, count the exits, and make them match.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the Denominator Frozen: A Checker for Agent Pass Rates

Casey Zhang — Sun, 11 Oct 2026 20:34:19 +0000

You closed the laptop on a green terminal. Four tasks, two passes, a free server still spinning. In the team channel the number had already become "50% solved."

Nobody attached the task list. Nobody said what "solved" ignored.

This is a procedure for freezing that meaning before the run, then refusing to print a rate when the logs drift. The artifact is a small checker you can run locally. It does not rank vendors.

The failure you are actually looking at

A percentage looks precise. It is only as stable as the denominator.

Change one task ID after you peek, and the rate moves. Drop a timeout, and it moves again. Call an empty patch a pass because the agent wrote a confident summary, and you have left measurement entirely.

You want the opposite habit. Decide the slice, the metrics, the control, and the access path. Then generate the number, or generate an error.

Step 1: Commit the contract while `run_id` is still null

Put scoring_contract.json in the eval repo. JSON keeps the checker free of extra dependencies. YAML is fine later if you already pin a parser.

{
  "contract_version": 1,
  "run_id": null,
  "locked_at": null,
  "task_ids": [
    "t_03_rename_helper",
    "t_11_fix_off_by_one",
    "t_18_add_missing_test",
    "t_22_reject_network_call"
  ],
  "metrics": {
    "primary": "task_pass",
    "task_pass": "tests_green AND patch_applies_clean AND no_forbidden_path",
    "abstain": "exit_code_0 AND patch_empty",
    "timeout": "wall_seconds > 180",
    "post_lock_exclusion": "forbidden"
  },
  "controls": {
    "narration_probe": "summary only; a non-empty patch fails the probe"
  },
  "access_class": {
    "host": "unspecified",
    "model_access": "unspecified",
    "quota_assumed": false,
    "permanence_assumed": false
  }
}

This file is a proposal you fill in, not a published score. Four tasks are a smoke slice. They can expose a sloppy denominator. They cannot support a model comparison.

Field notes, in the order you should fill them:

task_ids is the denominator. A folder glob is not allowed at score time.
task_pass is a conjunction. Any false clause fails the task, including a dirty apply or a forbidden path.
narration_probe is a control condition. It is not a second leaderboard and it does not enter the numerator.
access_class records where the run happened. Treat it as a condition, not as a trophy.

Step 2: Record the host you actually used

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You can point access_class.host at MonkeyCode's free server option and model_access at its free model access. Those two availability claims are operator-supplied for this draft. They are the reason a locked four-task slice is cheap to execute.

Do not write a token quota, a hardware name, or a duration into the contract unless a current primary page states them and you store the URL plus the date you read it. This article does not. Free paths can be queued or withdrawn. Set quota_assumed and permanence_assumed to false and leave them there.

If a teammate later reruns the same IDs on another host, mint a new run_id. Averaging across access classes hides the variable you forgot to name.

The product is a place to run the protocol. It is not the conclusion of the protocol. Delete the name and the checker still tells you when a rate is unfit to quote.

Step 3: Lock, then refuse an unlocked rate

Save the checker as refuse_unlocked_rate.py. The sample below is unexecuted teaching code. Run it on your own fixture before you trust the exit codes.

#!/usr/bin/env python3
"""Emit a pass rate only for a locked scoring contract."""

import json
import sys
from datetime import datetime, timezone
from pathlib import Path

REQUIRED = ("task_ids", "metrics", "controls", "access_class")

def load(path):
    return json.loads(Path(path).read_text(encoding="utf-8"))

def dump(path, obj):
    Path(path).write_text(json.dumps(obj, indent=2) + "\n", encoding="utf-8")

def lock(path, run_id):
    contract = load(path)
    if contract.get("locked_at"):
        sys.exit("already locked; start a new contract file")
    missing = [k for k in REQUIRED if not contract.get(k)]
    if missing:
        sys.exit(f"refusing to lock incomplete contract: {missing}")
    access = contract["access_class"]
    if access.get("quota_assumed") or access.get("permanence_assumed"):
        sys.exit("refusing to lock assumed quota or permanence")
    if access.get("host") in (None, "", "unspecified"):
        sys.exit("set access_class.host before lock")
    contract["run_id"] = run_id
    contract["locked_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    dump(path, contract)
    print(f"locked {run_id} at {contract['locked_at']}")

def score(contract_path, events_path):
    contract = load(contract_path)
    events = load(events_path)
    if not contract.get("locked_at") or not contract.get("run_id"):
        sys.exit("unlocked contract; no rate emitted")
    access = contract["access_class"]
    if access.get("quota_assumed") or access.get("permanence_assumed"):
        sys.exit("assumed quota or permanence; no rate emitted")
    allowed = list(contract["task_ids"])
    if len(allowed) != len(set(allowed)):
        sys.exit("duplicate task_ids")
    by_id = {}
    for ev in events:
        tid = ev["task_id"]
        if tid not in allowed:
            sys.exit(f"event outside locked slice: {tid}")
        if ev.get("excluded_after_lock"):
            sys.exit(f"post-lock exclusion starts a new study: {tid}")
        if "task_pass" not in ev:
            sys.exit(f"missing task_pass: {tid}")
        by_id[tid] = ev
    missing = [tid for tid in allowed if tid not in by_id]
    if missing:
        sys.exit(f"denominator drift, missing {missing}")
    passed = 0
    for tid in allowed:
        ev = by_id[tid]
        if ev.get("abstain") and ev.get("task_pass"):
            sys.exit(f"abstain cannot pass: {tid}")
        if ev.get("task_pass") is True:
            passed += 1
    print(json.dumps({
        "run_id": contract["run_id"],
        "locked_at": contract["locked_at"],
        "n": len(allowed),
        "task_pass_count": passed,
        "task_pass_rate": passed / len(allowed),
        "access_class": access,
        "note": "valid only for this slice, metric, and access class"
    }, indent=2))

if __name__ == "__main__":
    cmd = sys.argv[1]
    if cmd == "lock":
        lock(sys.argv[2], sys.argv[3])
    elif cmd == "score":
        score(sys.argv[2], sys.argv[3])
    else:
        sys.exit("usage: lock|score")

Fill the host with a label you can defend, then lock. Scoring before that step should fail.

python3 - <<'PY'
import json
p = "scoring_contract.json"
with open(p, encoding="utf-8") as f:
    c = json.load(f)
c["access_class"]["host"] = "operator-supplied free server option"
c["access_class"]["model_access"] = "operator-supplied free model access"
with open(p, "w", encoding="utf-8") as f:
    json.dump(c, f, indent=2)
    f.write("\n")
PY
python3 refuse_unlocked_rate.py lock scoring_contract.json run_2026_10_12_a
python3 refuse_unlocked_rate.py score scoring_contract.json events.json

The script does not fetch a quota. It only checks that you did not pretend to have one.

events.json must contain every locked ID, including failures:

[
  {"task_id": "t_03_rename_helper", "task_pass": true, "abstain": false, "wall_seconds": 40},
  {"task_id": "t_11_fix_off_by_one", "task_pass": false, "abstain": false, "wall_seconds": 55},
  {"task_id": "t_18_add_missing_test", "task_pass": false, "abstain": false, "wall_seconds": 200},
  {"task_id": "t_22_reject_network_call", "task_pass": true, "abstain": false, "wall_seconds": 30}
]

On this fixture the checker would print task_pass_count 2 and n 4. That printout is an example of the schema. It is not a measurement of any agent. Replace the booleans with values your harness computed from tests.

Step 4: Make exclusions and probes impossible to clean up

Run the narration probe on the same IDs before you trust task_pass. The prompt asks for a summary and forbids a patch. A non-empty diff fails the probe.

Keep that result beside the rate. Do not fold it into the numerator.

Timeouts remain in n. An abstain remains in n and cannot set task_pass. A test you discover is flaky after locked_at does not get an excluded_after_lock flag on this run. Copy the contract, explain the flake, and score a new run_id.

If your harness has no hidden tests, write that inside metrics. Silence is how a public sample becomes a private-sounding claim.

Why the printed rate is still not marketing

A marketing number is allowed to move after the audience reacts. A measurement is not.

You publish the contract hash with the rate. You publish the misses. You publish the access class as a constraint. You do not publish a comparative adjective.

"Better" needs a paired design you did not run: same IDs, same budget policy, same tests, a second access class, and a rule for ties written in advance. Until that second contract exists, the honest sentence is local. On this locked slice, under this access class, this many tasks met task_pass.

That sentence will not travel as a billboard. Good.

Evidence sentence	Slogan it must not become
"2 of 4 `task_pass` on this locked `run_id`."	"50% accuracy."
"Narration probe emitted no patch on this slice."	"The harness is fair."
"Access class recorded; terms read that day."	"Free hosting means the score will hold."
"Flake deferred to a new `run_id`."	"We removed the bad task and improved."

The left column is dull. Dull is the feature. A marketing line needs the right column, which this checker is built to withhold.

Step 5: Store a bundle a stranger can rerun

A rate without a bundle is a rumor. After the checker exits 0, write a directory that contains only boring files.

runs/run_2026_10_12_a/
  scoring_contract.json
  events.json
  narration_probe.json
  commands.txt
  README.md

commands.txt lists the test invocation, not a paraphrase.

sha256sum scoring_contract.json events.json > runs/run_2026_10_12_a/SHA256SUMS
python3 refuse_unlocked_rate.py score scoring_contract.json events.json | tee runs/run_2026_10_12_a/rate.json

The README gets five lines: slice purpose, access class, date you read the host terms, probe result, and the sentence "not a vendor ranking." If you cannot write that last line without hedging, you are not done.

Who should not use the approach

Do not use a four-task lock to choose a vendor, set a price, or announce a winner. The sample is too small, and this draft supplies no confidence interval you can hide behind.

Do not use it if you cannot keep the patch, the test command, and the contract file. A chat scroll cannot be re-scored.

Do not use it when task text is copied from a popular public repo and you have no note about possible overlap with model training data. Say "contamination unknown" next to the rate, or do not publish the rate.

Skip the free-server path if your question depends on a reserved machine, a fixed region, or a paid throughput guarantee. Those properties are outside the operator-supplied claims used here.

Limits of the checker

The script trusts task_pass. You still have to fill that boolean from a test command, not from model prose. A lying harness makes a locked lie.

Clock skew, queue delay, and partial outages on a free host belong in the event log as timeouts or as a failed run, not as deleted rows. This article states no latency budget and no uptime figure.

The sample events are unexecuted illustrations. Do not cite them. Do not add a model name to the JSON to make a screenshot look finished.

Ship the smaller sentence

Lock four tasks you can explain without a diagram. Run the probe. Run one agent pass. Paste the checker JSON, or paste the error.

If that log is something your team will rerun, the operator-supplied free model access and free server option are a reasonable place to do the next slice. Read the live terms on the day you lock the contract, store the date in your notes, and keep the claim as narrow as the run_id.