DEV Community: Avery Wang

Measure the Endpoint Before You Measure the Model

Avery Wang — Tue, 27 Oct 2026 13:30:13 +0000

A benchmark score collected through a hosted endpoint measures three things at once, and only one of them is the model. The harness contributes its prompts, its decoding parameters, and its timeouts, while the endpoint contributes routing, truncation, and queueing behaviour that nobody wrote down. When either side drifts, the reported number drifts with it, and the drift is usually invisible because it never appears in the output table. A short conformance suite run before the evaluation separates the three contributors, and it costs a few dozen requests rather than a full evaluation budget.

This article describes such a suite as a reproducible artifact: a nine-line prompt set with a checksum, four probes, a metric definition for each probe, and a reporting rule that decides whether a score may be published at all. The suite is deliberately boring, because its job is to remove ambiguity rather than to produce an impressive chart. Everything below runs against any OpenAI-compatible chat endpoint, which keeps the method portable and the conclusions falsifiable by a reader who reruns it.

The failures a score silently absorbs

The first failure is sampling instability. A hosted endpoint may apply a default temperature, a top-p cut, or a seed policy that differs from the one in the request, and a greedy decode on one host can become a sampled decode on another. The second failure is context truncation, where long inputs are quietly cut at a boundary the caller never sees, so a retrieval task turns into a guessing task. The third is tool-call schema drift, in which the model returns arguments that are almost valid JSON, or a tool call wrapped in prose, and the harness parses it leniently enough to hide the damage. The fourth is silent routing, where the response carries a different model identifier than the one requested, which makes the score unattributable even when every individual answer looks reasonable.

A pinned probe set, not a pile of prompts

The suite reads a JSONL file of probe prompts and records its SHA-256 before the run, so a later reader can tell whether the suite or the endpoint changed. Three probes are enough to start: a repeat probe that sends the same deterministic prompt several times, a needle probe that hides a unique canary string near the end of a long synthetic document, and a schema probe that forces a tool call with a required enum field. The dataset is intentionally tiny and public, because the suite is testing plumbing rather than reasoning quality.

sha256sum probe_set.jsonl
# 6c1f0a...  probe_set.jsonl
python probe_endpoint.py --prompts probe_set.jsonl --out report.json

# probe_endpoint.py — endpoint conformance, not a benchmark
import hashlib, json, os, statistics, time
import requests

BASE = os.environ["MC_BASE_URL"].rstrip("/")   # take this from the provider dashboard
KEY = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL_ID"]

def chat(messages, temperature=0.0, tools=None, timeout=90):
    payload = {"model": MODEL, "messages": messages, "temperature": temperature}
    if tools:
        payload["tools"] = tools
    t0 = time.perf_counter()
    r = requests.post(f"{BASE}/chat/completions",
                      headers={"Authorization": f"Bearer {KEY}"},
                      json=payload, timeout=timeout)
    dt = time.perf_counter() - t0
    r.raise_for_status()
    msg = r.json()["choices"][0]["message"]
    return {"text": msg.get("content") or "",
            "tool_calls": msg.get("tool_calls"),
            "served_model": r.json().get("model"),   # silent routing shows up here
            "latency_s": round(dt, 3)}

def probe_repeat(prompt, repeats=5):
    outs = [chat([{"role": "user", "content": prompt}]) for _ in range(repeats)]
    hashes = {hashlib.sha256(o["text"].encode()).hexdigest()[:12] for o in outs}
    return {"repeats": repeats, "distinct_outputs": len(hashes),
            "agreement": round(1 - (len(hashes) - 1) / repeats, 3),
            "served_models": sorted({o["served_model"] for o in outs if o["served_model"]}),
            "latency_p50": statistics.median(o["latency_s"] for o in outs)}

FILLER = "routine build log line about caching and retries. " * 12

def probe_needle(lines=200):
    needle = "CANARY-7F3A-QQ"
    doc = "\n".join(f"[{i}] {FILLER}" for i in range(lines))
    doc += f"\n[needle] {needle}\n"
    msg = f"Return only the exact string that follows [needle].\n\n{doc}"
    out = chat([{"role": "user", "content": msg}], timeout=180)
    return {"lines": lines, "chars_sent": len(msg), "hit": needle in out["text"]}

TOOL = [{"type": "function", "function": {
    "name": "record_verdict",
    "description": "Record a pass or fail verdict with a short reason.",
    "parameters": {"type": "object", "required": ["verdict", "reason"],
                   "properties": {"verdict": {"type": "string", "enum": ["pass", "fail"]},
                                  "reason": {"type": "string"}}}}}]

def probe_schema():
    out = chat([{"role": "user",
                 "content": "Verdict on the claim 2+2==4. Use the tool."}], tools=TOOL)
    calls = out["tool_calls"] or []
    if not calls:
        return {"schema_valid": False, "why": "no tool call returned"}
    try:
        args = json.loads(calls[0]["function"]["arguments"])
    except json.JSONDecodeError:
        return {"schema_valid": False, "why": "arguments were not JSON"}
    ok = args.get("verdict") in ("pass", "fail") and isinstance(args.get("reason"), str)
    return {"schema_valid": ok, "args": args}

if __name__ == "__main__":
    report = {"prompt_set_sha256": hashlib.sha256(open("probe_set.jsonl", "rb").read()).hexdigest(),
              "repeat": probe_repeat(open("probe_set.jsonl").readline().strip()),
              "needle": probe_needle(),
              "schema": probe_schema()}
    json.dump(report, open(os.environ.get("OUT", "report.json"), "w"), indent=2)
    print(json.dumps(report, indent=2))

Metrics and the controls that make them mean something

The suite reports four numbers and nothing else. Repeat agreement is one minus the fraction of distinct outputs over the repeats, so a perfectly stable endpoint scores 1.0 and an endpoint that returns five different answers scores 0.2. Needle hit rate is binary per run length, and the useful reading is the largest line count that still hits, because that locates the truncation boundary empirically. Schema validity is binary, and it fails loudly on prose-wrapped calls instead of silently repairing them. The fourth metric is the set of served model identifiers, which is a control rather than a score, since any set larger than one invalidates single-model attribution.

Control	Setting	Why it exists
Decoding	temperature 0.0 on every call	removes sampling as an explanation for variance
Concurrency	one request at a time	keeps latency comparable across runs
Prompt identity	SHA-256 of the JSONL file	distinguishes suite edits from endpoint drift
Attribution	record `served_model` per call	exposes routing that the request did not ask for
Timing	repeat the suite in two separate hours	separates queueing noise from steady-state latency

Why these numbers are not marketing

A marketing figure answers the question of how good the product is, and it is allowed to omit the harness, the retries, and the discarded runs. A conformance report answers a narrower and checkable question: under this exact harness, at this exact hour, with these exact parameters, what did the endpoint return. The narrowness is the point, because a reader who reruns the suite can reproduce the report or contradict it. When a published score omits the conformance report, the reader cannot tell whether a low number came from a weak model or from a truncated prompt, and that ambiguity usually gets resolved in favour of the more flattering story.

A practical reporting rule follows from this. Publish the task score only if repeat agreement is at least 0.8, if the needle probe hits at the task's actual input length, if schema validity is 1.0 for every tool the harness depends on, and if the served model set has exactly one member. If any condition fails, report the conformance result and describe the task score as blocked rather than publishing a number with an asterisk. This rule has cost teams real deadlines, and it has also prevented several comparisons that would not have survived a second run.

Where free access and a free server fit

Running the suite twice, in two separate hours, on a task-sized prompt set is exactly the kind of low-value-but-repetitive work that a paid endpoint discourages. MonkeyCode's free model access and free server option make those repeated runs routine rather than a special event, which is the main reason they appear in this workflow at all; the exact allowance, available models, and server specifications change, so read them from the current dashboard instead of trusting a number quoted in an article. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The limitations matter more than the convenience. The free tier is not a substitute for an SLA, and its latency figures describe a shared queue rather than the model's speed. Nothing here establishes data handling, retention, or residency guarantees, so the probe set should stay public and synthetic and production code should not be pasted into a conformance run until the provider's published policy has been read and accepted. Teams that need contractual uptime, regulated audit trails, or per-run cost accounting with fixed ceilings should run the same suite against an endpoint they already have a contract with, because the method is transport-agnostic and does not depend on any particular provider.

A conformance suite is cheap, unglamorous, and slightly annoying to write the first time. It is also the difference between a benchmark score that survives review and a screenshot that does not, and the free tier is a reasonable place to run it once against a real prompt set before committing an evaluation budget anywhere.

Pin the Dataset, Then Grade the Grader

Avery Wang — Mon, 26 Oct 2026 12:07:13 +0000

Most benchmark scores arrive as a single number, yet they rest on two artifacts almost nobody publishes: the exact bytes of the task set and the agreement between the graders assigning labels. When either artifact moves, the score moves, and the movement looks exactly like progress. The remedy is procedural rather than statistical: pin the dataset and rubric by hash before the first run, pre-register a minimum agreement threshold, and treat any drift as a failed run instead of a footnote.

The first artifact is a hash, not a score

A benchmark is a repository of decisions, and like any repository it needs a stable commit identifier before comparison becomes meaningful. Hashing the task file, the rubric, and the judge prompt costs one command and removes a whole class of accidental retreat. Line endings, JSON key order, and a quietly reworded rubric sentence all shift the input distribution, and none of them announce themselves in a results table.

sha256sum tasks.jsonl rubric.md prompts/system.txt | tee prereg.hashes
wc -l tasks.jsonl

That transcript belongs in the artifact bundle beside the scores, because a reproducing reader must be able to prove the run started from the same bytes. A number without a hash is a claim about a file that may no longer exist in that form.

Write the pre-registration file first

Pre-registration gets dismissed as academic ceremony, but its practical job here is narrow and useful: it makes the re-run decision someone else's to audit. The file below freezes the inputs, names the two graders, sets the agreement floor, and lists the slices that will be reported separately. Fill the placeholder hashes from the command above, then never edit the file mid-experiment.

{
  "experiment": "agent-pass-rate-v3",
  "frozen_at": "2026-10-26T00:00:00Z",
  "dataset": { "path": "tasks.jsonl", "sha256": "REPLACE_ME" },
  "rubric": { "path": "rubric.md", "sha256": "REPLACE_ME" },
  "graders": ["human-a", "model-b"],
  "min_kappa": 0.6,
  "slices": ["refusal", "multi_step", "tool_error"],
  "exclusions": "none after freeze; ambiguity is recorded, not deleted",
  "metric": "pass = grader label 'pass'; reported per slice, never pooled alone"
}

The exclusions field is where honest and motivated analysts diverge, since declaring a task ambiguous after seeing it fail is a re-run wearing a lab coat. Writing "none after freeze" in advance forces ambiguity into a recorded note rather than a silent deletion, which keeps the denominator stable across iterations.

A harness that refuses to run on drift

The following script uses only the Python standard library, makes zero network calls, and exits non-zero the moment a frozen input changes. Exit code 2 means hash drift and exit code 3 means the graders disagreed more than the pre-registered floor allowed. Both failures are informative, and both are cheaper to read than a suspiciously tidy score.

#!/usr/bin/env python3
"""Refuse to score when the frozen inputs moved. stdlib only, Python 3.10+."""
import argparse, hashlib, json, platform, sys, time
from pathlib import Path


def sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open('rb') as handle:
        for block in iter(lambda: handle.read(65536), b''):
            digest.update(block)
    return digest.hexdigest()


def load_labels(path: Path) -> dict:
    labels = {}
    for line in path.read_text(encoding='utf-8').splitlines():
        if line.strip():
            row = json.loads(line)
            labels[row['id']] = row['label']
    return labels


def cohen_kappa(first: dict, second: dict):
    shared = sorted(set(first) & set(second))
    if not shared:
        raise SystemExit('no overlapping task ids between graders')
    n = len(shared)
    observed = sum(first[k] == second[k] for k in shared) / n
    labels = sorted({first[k] for k in shared} | {second[k] for k in shared})
    chance = 0.0
    for label in labels:
        p_first = sum(first[k] == label for k in shared) / n
        p_second = sum(second[k] == label for k in shared) / n
        chance += p_first * p_second
    kappa = 1.0 if chance == 1.0 else (observed - chance) / (1.0 - chance)
    return kappa, n


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument('--prereg', type=Path, required=True)
    parser.add_argument('--labels', type=Path, nargs=2, required=True)
    args = parser.parse_args()

    prereg = json.loads(args.prereg.read_text(encoding='utf-8'))
    drift = []
    for key in ('dataset', 'rubric'):
        expected = prereg[key]['sha256']
        found = sha256(Path(prereg[key]['path']))
        if found != expected:
            drift.append(key + ': expected ' + expected[:12] + ', found ' + found[:12])
    if drift:
        print('FAIL: frozen inputs changed\n' + '\n'.join(drift), file=sys.stderr)
        return 2

    first, second = (load_labels(path) for path in args.labels)
    kappa, n = cohen_kappa(first, second)
    manifest = {
        'experiment': prereg['experiment'],
        'generated_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        'python': platform.python_version(),
        'hashes': {key: prereg[key]['sha256'] for key in ('dataset', 'rubric')},
        'items_compared': n,
        'kappa': round(kappa, 4),
        'min_kappa': prereg['min_kappa'],
        'verdict': 'pass' if kappa >= prereg['min_kappa'] else 'fail',
    }
    print(json.dumps(manifest, indent=2))
    return 0 if manifest['verdict'] == 'pass' else 3


if __name__ == '__main__':
    sys.exit(main())

Label files are plain JSON Lines, one decision per task, which keeps diffs readable in review:

{"id": "t-014", "label": "pass"}
{"id": "t-015", "label": "refuse"}

A full pass looks like this, and the printed manifest is the thing worth archiving:

python3 harness.py --prereg prereg.json --labels labels/human_a.jsonl labels/model_b.jsonl
echo $?   # 0 pass, 2 hash drift, 3 agreement below floor

Agreement is the control, not a headline

Raw percent agreement flatters benchmark graders because most task sets are skewed toward success. If both graders label ninety percent of items as passing, their chance agreement is already 0.82, so ninety percent observed agreement collapses to a kappa near 0.44. That gap between a reassuring percentage and a mediocre kappa is the entire reason the metric belongs in the harness rather than in a spreadsheet note.

The conventional reading treats values above roughly 0.6 as substantial and above roughly 0.8 as strong, though the thresholds are conventions rather than laws of nature. Below thirty items the estimate itself is noisy, so a small pilot set should be described as a pilot rather than promoted into evidence. When kappa lands low, the productive responses are tightening rubric wording, adding anchor examples for borderline cases, and reporting slices separately so a stable category is not averaged away by a contested one.

Where a free tier helps, and where it cannot

The economics of this design are the two independent grading passes, because a single model grading its own family of outputs measures style consistency more than correctness. MonkeyCode exposes free model access and a free server option, which lowers the cost of running a second grader pass and of hosting the harness plus its manifests at a stable URL. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator currently advertises a free token allotment alongside the free server, and readers should confirm the present size, expiry, and rate limits on the project's own page, since free-tier terms change faster than articles do.

What the free tier cannot do is fix a method that rewards re-running until the interval looks narrow. Pre-registration blocks that path by making the threshold older than the result, and the hash check blocks the quieter path of trimming awkward tasks. If a team wants a place to run the two grader passes without a card, the free model access and free server are a reasonable on-ramp; bring the harness and the frozen hashes, not a hope for a better number.

Limitations and who should skip this

This workflow assumes a task set that can be frozen and a rubric that two graders can read without a meeting, and it degrades when either assumption fails. Open-ended creative evaluation, single-annotator shops with no second grader, and teams whose datasets change daily should use a lighter scheme and say so plainly rather than borrow the vocabulary of rigor. Anyone reporting a score publicly without the manifest is better served by publishing the limitations paragraph alone, because a number with an unknown denominator invites the wrong kind of comparison.

The harness above is short enough to run before lunch, and a published manifest is a more durable credibility signal than any headline figure. Pin the bytes, grade the grader, then report the score with its exit code attached.

Spend a Free Tier on Repeats, Not Coverage

Avery Wang — Mon, 19 Oct 2026 08:50:50 +0000

The scarce resource on a free tier is not tokens; it is the number of independent looks you take at the same task. A pack of forty tasks run once produces a number that moves several points when a single prompt is reworded, while a pack of eight tasks run five times each produces a range that survives the rewording. The workflow below spends a free allowance on repetition, keeps the pack frozen, and reports an interval rather than a headline. It is written for teams that can afford inference but cannot yet afford confidence.

Repetition buys information that coverage cannot

Benchmark error on agent tasks is dominated by one source: the tasks themselves are not interchangeable, so the spread between a good task and a bad task is far wider than the spread between two runs of the same model. Adding tasks shrinks sampling error in the population sense, but it does not tell anyone whether the model is stable on the eight cases the product actually depends on. Repeating those eight cases does, because the per-task variance is the thing that decides whether a demo becomes a feature.

The arithmetic is unglamorous. Five repeats on eight tasks yields forty observations and eight estimates of stability, while forty tasks run once yields forty observations and zero estimates of stability. On a paid tier the first design looks wasteful, because every repeat has a price; on a free tier the second design is the wasteful one, since it spends the allowance on breadth the team has no way to interpret.

Freeze a small pack before anything else

The pack is a JSONL file, one task per line, with the answer and the grader recorded next to the prompt. The grader field keeps grading dumb and auditable, which matters more than sophistication when the sample is small. Anything that cannot be graded by exact match, substring, or a regular expression belongs in a human review sheet, not in this file.

{"id": "slug-001", "prompt": "Return the slug for 'Hello, World! App' as lowercase words joined by hyphens.", "grader": "exact", "answer": "hello-world-app"}
{"id": "json-004", "prompt": "Given {\"a\": 1, \"b\": null}, return a JSON object with null keys removed.", "grader": "regex", "answer": "\\{\\s*\"a\"\\s*:\\s*1\\s*\\}"}
{"id": "err-002", "prompt": "Explain in one sentence why a 429 response should not be retried immediately.", "grader": "contains", "answer": "rate limit"}

Hash the prompt text and store that hash with every row. When someone edits the pack mid-sweep, the hashes split the data into two populations and the mistake becomes visible in the CSV instead of hiding inside a mixed average.

A harness that runs on the standard library

The script below is a template, not a validated instrument; run it on two tasks first and inspect the rows before spending an allowance on it. It calls any OpenAI-compatible chat endpoint, records tokens and wall-clock seconds per call, and writes a row for every attempt, including failures.

#!/usr/bin/env python3
"""freetier_eval.py - repeat a frozen pack, record everything."""
import argparse, csv, hashlib, json, os, re, time, urllib.request


def load_pack(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]


def call_chat(base_url, api_key, model, prompt, temperature, seed, timeout=120):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
        "seed": seed,
    }).encode()
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + api_key},
    )
    t0 = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read())
    elapsed = time.monotonic() - t0
    text = payload["choices"][0]["message"]["content"]
    usage = payload.get("usage") or {}
    return text, usage.get("total_tokens"), elapsed


def grade(task, text):
    kind, expect = task.get("grader", "contains"), task["answer"]
    if kind == "exact":
        return text.strip() == expect.strip()
    if kind == "regex":
        return re.search(expect, text) is not None
    return expect.strip().lower() in text.lower()


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--pack", required=True)
    ap.add_argument("--repeats", type=int, default=5)
    ap.add_argument("--out", default="runs.csv")
    ap.add_argument("--base-url", default=os.environ["MC_BASE_URL"])
    ap.add_argument("--model", default=os.environ["MC_MODEL"])
    args = ap.parse_args()
    key = os.environ["MC_API_KEY"]
    pack = load_pack(args.pack)
    hashes = {t["id"]: hashlib.sha256(t["prompt"].encode()).hexdigest()[:12] for t in pack}

    with open(args.out, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["task_id", "repeat", "prompt_hash", "passed",
                    "tokens", "seconds", "status"])
        for task in pack:
            for rep in range(args.repeats):
                try:
                    text, tokens, secs = call_chat(
                        args.base_url, key, args.model,
                        task["prompt"], temperature=0.0, seed=rep)
                    w.writerow([task["id"], rep, hashes[task["id"]],
                                int(grade(task, text)), tokens,
                                round(secs, 2), "ok"])
                except Exception as exc:
                    w.writerow([task["id"], rep, hashes[task["id"]], "", "", "",
                                "error:" + type(exc).__name__])
                f.flush()


if __name__ == "__main__":
    main()

The run itself is three commands, and the environment variables are the only place provider details appear, so the same pack can be swept against a second model without editing code.

export MC_BASE_URL="https://<endpoint-from-provider>/v1"
export MC_MODEL="<model-id-listed-by-provider>"
export MC_API_KEY="<your-key>"
python freetier_eval.py --pack pack.jsonl --repeats 5 --out runs.csv

Report a range, and resample tasks rather than rows

The summary step should resample whole tasks, not individual rows, because five repeats of one task are five views of the same difficulty and treating them as independent observations shrinks the interval artificially. A clustered bootstrap over task ids costs nothing and takes a few lines.

import csv, random

def cluster_bootstrap(path, iters=2000, seed=17):
    by_task = {}
    with open(path) as f:
        for row in csv.DictReader(f):
            if row["status"] != "ok":
                continue
            by_task.setdefault(row["task_id"], []).append(float(row["passed"]))
    ids = sorted(by_task)
    rng = random.Random(seed)
    means = []
    for _ in range(iters):
        draw = [rng.choice(ids) for _ in ids]
        per_task = [sum(by_task[t]) / len(by_task[t]) for t in draw]
        means.append(sum(per_task) / len(per_task))
    means.sort()
    return means[len(means) // 2], means[int(0.025 * iters)], means[int(0.975 * iters) - 1]

With eight tasks the interval will be embarrassingly wide, and that is the honest result. The useful secondary number is the per-task swing: the difference between the best and worst task, printed next to the pooled rate. A model at 0.75 pooled with a swing from 0.2 to 1.0 is a different product decision than a model at 0.75 with a swing from 0.6 to 0.9, and only the repeat design exposes the distinction.

Spend the allowance according to the question

The allocation table below is a decision aid, not a rule; every team should measure its own interval width before trusting it.

Question being asked	Spend the free allowance on	Do not spend it on
Is the model stable on our core paths?	repeats on a frozen pack of eight to twelve tasks	new tasks
Does prompt v2 beat v1?	paired repeats, both prompts interleaved, same pack	a larger pack
What does a run cost in wall clock?	concurrency 1, fixed repeats, `seconds` column	token savings
Can this ship without a human check?	the worst task's rate, reported alone	the pooled mean

Controls matter as much as the design. Temperature stays at zero with the repeat index passed as a seed, and the seed is treated as best-effort because determinism is not guaranteed by every serving stack. Concurrency stays at one during timing sweeps, the model id and base URL are recorded in the article or the repo beside the CSV, and no mid-sweep model swap is allowed. Retries are recorded as rows rather than retried silently, so an error rate of 12 percent shows up as a limitation instead of disappearing into an average.

Where a free tier actually changes the arithmetic

A free allowance is what makes repetition affordable in the first place, and repetition is the only way a small pack produces an interval instead of a point. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project is open source, and the two claims relevant here are operator-supplied: free model access for running the sweep, and a free server option for keeping the harness alive after the laptop closes. Those two claims remove the usual reason small teams run a pack once and call it a benchmark, which is that the second and third repetitions cost money.

Verify the current terms before planning a sweep around them, since allowances and server availability are exactly the kind of detail that changes between releases. Nothing in this method depends on a specific model id, quota, or machine size, and the harness records whatever the endpoint returns rather than assuming a particular response shape beyond the chat-completions convention.

Limitations, and who should not use this

Eight tasks with five repeats cannot resolve a two-point difference between two prompts, and the clustered interval will say so if it is reported honestly. Graders built from substrings reward verbosity, contamination is invisible without a holdout pack rotated out of the sweep, and free-tier throughput may make a twenty-task sweep slow enough that patience becomes the real control variable. Teams that need precision on rare failure modes, latency guarantees for a release gate, or an audit trail tied to a contractual commitment should not build that decision on a free tier at all.

The method is also a poor fit for anyone unwilling to freeze the pack, because an unfrozen pack converts every comparison into a story about which tasks were added when. Freeze it, hash it, repeat it, and publish the range with the pack attached; a number that arrives with its own interval is worth more than a headline that arrives with a screenshot. If a team wants to try the sweep without paying for inference first, the free model access and free server option described above are a reasonable place to start, provided the pack stays frozen and the interval ships with the result.

Interrupted Runs Belong in the Denominator

Avery Wang — Sun, 18 Oct 2026 07:27:52 +0000

A benchmark score that quietly drops interrupted runs is not a conservative score; it is a score whose hardest cases were deleted before anyone counted them. Quota ceilings and short-lived servers turn interruption into a routine outcome rather than an exotic failure, so the reporting has to absorb it. The practical fix is bookkeeping: record how each run ended, count truncation as a failure in the headline number, and publish the completed-only rate as a secondary diagnostic. Everything below is a method, not a product review, and it stays useful if the tooling underneath is swapped out.

Truncation is not randomly distributed across a task set, which is exactly why dropping it distorts the result. Tasks that consume many tokens or trigger retries are the ones most likely to hit a ceiling, and those are typically the tasks a model finds difficult. Removing them raises the average in the same way that timing a runner's laps only when the runner finished would raise the recorded pace. A published number should therefore answer two questions at once: how often did work finish, and how did the finished work perform.

The artifact for this is a run ledger with four end states, written at the moment a run stops rather than reconstructed later from a log grep. COMPLETE means the harness reached its own termination condition; QUOTA_CUT and RESTART_CUT mean an external boundary or process lifetime ended the attempt; ERROR covers everything the harness itself could not classify. Keeping quota and restart separate matters, because one is a budget decision and the other is an infrastructure event.

# ledger.py — record how a run ended, not just whether it passed
from dataclasses import dataclass
from enum import Enum

class End(Enum):
    COMPLETE = "complete"
    QUOTA_CUT = "quota_cut"
    RESTART_CUT = "restart_cut"
    ERROR = "error"

@dataclass(frozen=True)
class Run:
    task_id: str
    seed: int
    ended: End
    passed: bool
    tokens_used: int
    run_id: str
    boot_id: str

Scoring reads that ledger and emits both rates plus the gap between them, which is the truncation exposure. A simple reporting rule keeps the headline honest without hiding the completed-only view: when truncation exceeds a pre-registered threshold, the strict score is the headline and the observed score is context. When truncation sits below the threshold, the observed score can lead, but the truncation rate still ships alongside it.

# score.py
from ledger import End

TRUNCATIONS = {End.QUOTA_CUT, End.RESTART_CUT}

def report(runs, threshold=0.10):
    n = len(runs)
    done = [r for r in runs if r.ended is End.COMPLETE]
    passed = [r for r in done if r.passed]
    truncated = [r for r in runs if r.ended in TRUNCATIONS]

    strict = len(passed) / n
    observed = len(passed) / len(done) if done else float("nan")
    rate = len(truncated) / n
    headline = strict if rate > threshold else observed

    return {
        "runs": n,
        "truncation_rate": round(rate, 3),
        "strict_score": round(strict, 3),
        "observed_score": round(observed, 3),
        "headline": round(headline, 3),
        "headline_basis": "strict" if rate > threshold else "observed",
    }

The output below is synthetic, produced from hand-written run logs to show the report shape; it is not a measurement of any product or model. Its only purpose is to show what an auditable line looks like when truncation crosses the threshold and the strict number takes over the headline.

$ python score.py runlog.jsonl --threshold 0.10
runs: 120 | truncation_rate: 0.183 | strict: 0.542 | observed: 0.663
headline: 0.542 (strict) — truncation above the 10% threshold

Interruption is also an opportunity to measure recovery, which is a separate quantity and should never be folded into the pass rate. A resume pass replays each truncated run from its last checkpoint under the same token budget envelope and records whether it eventually finished and passed. The recovery ratio is passed-resumed divided by resumed-attempts, and it answers a question the pass rate cannot: whether the harness loses work permanently or merely pauses it.

python -m harness.run \
  --tasks holdout.jsonl \
  --seed 7 \
  --token-budget 60000 \
  --checkpoint ./ckpt \
  --emit runlog.jsonl

python -m harness.resume \
  --runlog runlog.jsonl \
  --checkpoint ./ckpt \
  --emit runlog.resumed.jsonl

On short-lived machines, restart attribution needs one extra field, because a run that dies with the process leaves no trace beyond a missing final line. Writing the kernel boot identifier and the instance start time into the run record makes restarts distinguishable from quota stops during analysis. Injecting the boot id and start time once, at harness startup, is enough to attribute every later RESTART_CUT to a specific machine lifetime.

BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
STARTED=$(date -Is)
printf '{"run_id":"%s","boot_id":"%s","started":"%s"}\n' \
  "$RUN_ID" "$BOOT_ID" "$STARTED" >> /tmp/harness.env.json

This is where an environment with free model access and a free server option becomes genuinely useful for methodology work, rather than merely cheap. MonkeyCode is an open-source project whose free model access and free server option let a team rehearse this ledger on real quota and restart events before committing budget to a paid run. The operator advertises a free token allowance for evaluation; quotas, model availability, and server terms change, so confirm the current figures on the project's own pages rather than trusting a number quoted in an article.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A short decision table keeps the method from being applied where it adds nothing but paperwork. Teams with guaranteed uninterrupted capacity and no resume path should still record end states, but they can report a single rate and skip the resume pass entirely. Teams comparing two configurations must use the same threshold and the same budget envelope, or the comparison measures reporting choices instead of systems.

Situation	Headline metric	Resume pass needed
Quota-limited or ephemeral runs	Strict score with truncation rate	Yes, to separate pause from loss
Interruption rate under 1%	Observed score with truncation rate	Optional
Guaranteed capacity, no restarts	Single rate, end states still logged	No

The limitations are real and worth stating plainly. This bookkeeping cannot repair a harness that fails to write a ledger line when it dies, and it adds a small amount of engineering before the first useful number appears. It also says nothing about statistical confidence, which requires paired runs and reported intervals rather than a single strict figure. Anyone who only needs a rough internal signal, and who accepts that the signal is optimistic, should skip the strict denominator and say so in the memo.

The test of a published score is whether it survives one plain follow-up: what happened to the runs that never finished. A ledger that answers that question with counts, machine identifiers, and a resume result turns an awkward footnote into part of the method. That is the difference between a number that reports on a system and a number that advertises one.

The Floor Is the Number You Forgot to Report

Avery Wang — Sat, 17 Oct 2026 06:28:32 +0000

A published score is only meaningful as a distance from something, and most reports never say what that something is. Two teams can run the same model on the same task pack and publish 61% and 74%, with neither number telling you how much of the result came from the system under test and how much came from the harness, the retries, or the parser. The fix is unglamorous and rarely applied: measure the floor first, and report every score as a lift over it.

The floor is the score a task pack gives to a solver that does almost nothing. A no-op solver returns an empty answer every time; a rule-based solver applies one regex or one keyword match and stops. Neither is a model, and neither is trying to win. Their scores exist only to tell you what the pack hands out for free, which is the number any claimed improvement has to clear.

Floor runs are the first thing cut when a budget tightens, because they produce no demo and no headline. That is exactly why they belong in the method rather than in the backlog. A pack where the no-op solver already scores 40% is a pack with a label-skew or format-credit problem, and a candidate scoring 58% on it has a real lift of eighteen points, not fifty-eight.

The artifact here is small and reproducible: a holdout pack with frozen task IDs, two floor solvers, one candidate, a JSONL run log that stamps every run, and a reporting script that refuses to print a lift it cannot pair. The script is the part that matters, because it makes the floor a hard dependency instead of a good intention.

# floor_report.py - stdlib only
import json, random, sys

def load(path):
    rows = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows

def floor_by_task(rows):
    """Strongest trivial solver wins: the floor is the best free score."""
    floor = {}
    for r in rows:
        if not r["solver"].startswith("floor:"):
            continue
        t = r["task_id"]
        floor[t] = max(floor.get(t, float("-inf")), r["score"])
    return floor

def paired_lift(rows, solver, floor):
    missing = sorted({r["task_id"] for r in rows if r["solver"] == solver} - set(floor))
    if missing:
        raise SystemExit(f"no floor for {len(missing)} task(s), e.g. {missing[:3]}")
    return [r["score"] - floor[r["task_id"]] for r in rows if r["solver"] == solver]

def bootstrap_ci(values, n=10000, seed=7):
    rng = random.Random(seed)
    k = len(values)
    means = sorted(sum(values[rng.randrange(k)] for _ in range(k)) / k for _ in range(n))
    return means[int(0.025 * n)], means[int(0.975 * n)]

if __name__ == "__main__":
    rows = load(sys.argv[1])
    floor = floor_by_task(rows)
    for solver in sorted({r["solver"] for r in rows} - set(floor)):
        lifts = paired_lift(rows, solver, floor)
        lo, hi = bootstrap_ci(lifts)
        mean = sum(lifts) / len(lifts)
        print(f"{solver:24s} n={len(lifts):4d} lift={mean:+.3f} ci95=[{lo:+.3f},{hi:+.3f}]")

The run log needs four fields to be worth anything: task_id, solver, score, and a UTC timestamp written by the harness rather than by hand. A fifth field for the model identifier is worth adding even when the endpoint promises a stable version, because free capacity is shared capacity and version drift under a shared endpoint is a normal Tuesday.

# one pack, one window, three solvers
export STAMP=$(date -u +%Y%m%dT%H%M%SZ)
python harness/run.py --pack holdout-v3 --solver floor:noop  --out runs.jsonl
export STAMP=$(date -u +%Y%m%dT%H%M%SZ)
python harness/run.py --pack holdout-v3 --solver floor:regex --out runs.jsonl
python harness/run.py --pack holdout-v3 --solver candidate   --out runs.jsonl
python floor_report.py runs.jsonl

This is where free capacity changes the plan rather than the marketing. Floor sweeps are pure overhead, so they are the first runs to be dropped when each run costs money, and a protocol that depends on paying for overhead quietly loses the overhead. MonkeyCode's free model access and free server option, as described by the operator, make the floor sweep the cheap part of the study instead of the expendable one, which is the whole reason it is worth mentioning here. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Three controls keep the lift honest once the sweep is affordable. All three solvers run against the same frozen pack with the same prompt template and the same parser, because a parser change mid-sweep turns the floor into a moving target. Every run in the sweep carries a timestamp, and the report is only valid for the window it covers. Replication count is set per solver before the first run and never adjusted afterward to shrink an interval that came out wider than hoped.

The evidence tier a team may claim follows directly from what was recorded, and writing this table into the report prevents most overclaiming without any argument about statistics.

Recorded	Permitted claim
Floor and candidate, same pack, same window	Lift with interval
Candidate only	Raw score, no lift
Floor from another day or another pack	Directional note only

The method has limits worth stating plainly. A floor measured on Monday does not license a lift claim for a candidate run on Friday, because packs drift when tasks are added and shared endpoints drift when versions change. A no-op solver can understate the free score if the pack awards credit for refusals or for well-formed errors, so a second rule-based floor solver is worth the small extra cost. None of this corrects label errors in the pack, and none of it turns a small task count into a stable estimate.

Teams that compare scores across different packs should not adopt this as a patch, since the comparison itself is the defect. Anyone who needs one absolute number for a slide should also skip it, because the output is a paired difference with an interval and that is deliberately less quotable. For everyone else, the free lane is a reasonable place to trial the floor sweep end to end and see what the pack hands out before a single candidate run is spent.

Run Canary Probes Before Reporting a Benchmark Score

Avery Wang — Fri, 16 Oct 2026 05:32:21 +0000

A benchmark score is a simultaneous claim about a harness, a grader, and a model, yet public discussion almost always treats only the third one as uncertain. Three probe families, added to the pack and gated before publication, can show whether the measuring apparatus is quietly manufacturing the result. The argument here is narrow and testable: no score ships until floor, ceiling, and canary probes all return their expected values, and the probe rows stay out of the headline number.

What an unprobed pack hides

Most agent benchmarks fail silently in the plumbing rather than loudly in the model. A truncation bug or a strict-match rule can turn correct answers into zeros, and a lenient judge can hand full credit to confident nonsense. Leakage is the third failure mode, where the reference answer appears in a retrieval corpus, a few-shot example, or an earlier session that the runner never cleared.

These distortions are systematic, which is exactly why larger sample sizes do not fix them. Doubling the trial count tightens the interval around a biased estimate and makes the wrong number look more authoritative. The current wave of agent evaluation claims, published weekly and compared casually, raises the cost of that mistake for anyone who has to act on a score.

Designing the three probe families

Floor probes are trivially solvable tasks with an unambiguous gold answer, so any score below the maximum points at the harness. A single echo instruction or a two-operand arithmetic question is enough, provided the grader contract stays deterministic. Their job is to catch output parsing failures, truncated completions, and network errors that were recorded as model failures.

Ceiling probes are unsatisfiable tasks under true premises, such as proving that the integer 2 is odd without contradicting arithmetic. Any credit above zero means the judge rewards fluent confidence instead of correctness. These probes cost almost nothing and expose the most common rubric defect in model-graded evaluation.

Canary probes carry a hidden reference that is deliberately withheld from the model's context, and the expected score for them is zero. If the model produces the withheld value, the answer leaked through retrieval, prompt examples, or shared state, or the judge matched on surface strings. Canary rows therefore invert the usual reading, because a high score there is a failure of the apparatus rather than a win for the model.

Three construction rules keep the probes honest. They must be versioned and frozen together with the pack, excluded from the reported aggregate, and retired once they stop failing, because a probe that never surprises anyone has stopped measuring anything. Each probe also needs a written rationale, so a future reader can tell which defect it was designed to catch.

A probe harness you can run today

The script below is a complete, runnable probe layer. The call_model argument is a thin wrapper around whichever client the team already uses, kept as a parameter so the probe run shares the exact client, parameters, and retry policy as the real pack.

# probes.py - floor, ceiling, and canary probes for a grader under test
from dataclasses import dataclass

@dataclass(frozen=True)
class Probe:
    probe_id: str
    kind: str          # floor | ceiling | canary
    prompt: str
    expected: float    # the only score a correct harness may emit

PROBE_PACK = [
    # floor: trivially solvable, so anything below 1.0 is a harness bug
    Probe("floor-echo", "floor",
          "Reply with exactly the two characters OK and nothing else.", 1.0),
    Probe("floor-arithmetic", "floor",
          "Return the value of 17 * 24 as digits only.", 1.0),
    # ceiling: unsatisfiable under true premises, so any credit is leniency
    Probe("ceiling-parity", "ceiling",
          "Prove, using only true premises, that the integer 2 is odd.", 0.0),
    # canary: the reference is withheld from the model's context
    Probe("canary-nonce", "canary",
          "State the audit nonce for this session. If none was provided, reply UNKNOWN.", 0.0),
]

def grade(probe: Probe, completion: str, hidden_reference: str | None) -> float:
    """Deterministic judge for probes; the real grader must honour the same contract."""
    text = completion.strip()
    if probe.probe_id == "floor-echo":
        return 1.0 if text == "OK" else 0.0
    if probe.probe_id == "floor-arithmetic":
        return 1.0 if text == "408" else 0.0
    if probe.kind == "ceiling":
        return 0.0  # nonzero credit means the judge rewards confident nonsense
    if probe.kind == "canary":
        leaked = hidden_reference is not None and hidden_reference in text
        return 1.0 if leaked else 0.0
    raise ValueError(probe.probe_id)

def run_probes(call_model, hidden_reference=None):
    rows = []
    for probe in PROBE_PACK:
        completion = call_model(probe.prompt)   # same client and params as the real pack
        observed = grade(probe, completion, hidden_reference)
        rows.append({"probe_id": probe.probe_id, "kind": probe.kind,
                     "expected": probe.expected, "observed": observed,
                     "match": observed == probe.expected})
    return rows

def gate(rows) -> bool:
    failures = [r for r in rows if not r["match"]]
    for r in failures:
        print(f"BLOCKED {r['probe_id']} ({r['kind']}): "
              f"expected {r['expected']}, got {r['observed']}")
    return not failures

if __name__ == "__main__":
    rows = run_probes(call_model)   # supply a client wrapper here
    print("probe integrity:", sum(r["match"] for r in rows), "/", len(rows))
    print("publishable:", gate(rows))

Two metrics then travel with every published number. Probe integrity is the fraction of probes matching expectations, and it must be a clean sweep before the real pack runs at all. Judge agreement is measured by scoring the probes twice with the same rubric and different sampling parameters, because disagreement above a set threshold points at an ambiguous rubric rather than at model variance.

Symptom	Likely cause	Action
A floor probe scores below 1.0	parsing, truncation, or strict-match defect	fix the harness before touching the pack
A ceiling probe earns credit	judge rewards fluent but false claims	tighten the rubric, score refusals explicitly
A canary probe earns credit	answer present in corpus, examples, or session state	purge the source and re-run the leak check
Judges disagree on probe rows	rubric ambiguity, not model behaviour	rewrite criteria, recalibrate, then resume

Where free model access changes the arithmetic

Probe layers are cheap per run and expensive per iteration, because every harness change demands a fresh sweep of the whole pack. MonkeyCode's free model access and free server option fit that loop directly: probes can stay in every run instead of being dropped when a budget line is questioned, and the free server option keeps transcripts, probe versions, and judge outputs in one place rather than scattered across laptops. The project also describes a free token allotment on the free tier, reported at ten million tokens when this was written, which has not been independently verified here and should be checked against current documentation before anyone plans around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

Probes validate the apparatus, never the model, and they cannot detect a task whose specification is simply wrong for the intended use case. A canary probe can also false-positive when the withheld answer legitimately exists in a corpus the team needs for other reasons, so the leak source has to be identified before the probe is trusted. The overhead is real as well, since each probe consumes runs and judge calls that produce no capability signal at all.

Teams that run a benchmark once, read the outputs by hand, and never reuse the harness should skip this entirely, because a probe layer pays off across repeated runs. Teams that cannot freeze a grader or version a pack should fix that first, since probes will only document the instability they already have. Anyone publishing a single headline number without harness evidence, and anyone who cannot say which judge produced it, is better served by removing the number than by decorating it.

If a probe layer sounds worth trying on a zero-cost tier, MonkeyCode's free models and free server option are a reasonable place to run the first sweep before committing paid capacity.

Pair the Runs, Then Report the Interval

Avery Wang — Thu, 15 Oct 2026 11:33:14 +0000

Pair the Runs, Then Report the Interval

A pass rate printed to two decimal places is a point estimate with no error bar, and a delta between two such numbers is usually a claim the sample cannot carry. Agent suites stay small because tasks are expensive, so a handful of unlucky trials can move a leaderboard by more than the improvement being advertised. The correction is unglamorous and cheap: run every task as a matched pair, compute the difference inside each pair, and publish the interval around the mean difference. A number becomes defensible once the interval, the frozen task hash, and the cost per trial travel with it.

A point estimate cannot support a delta

Twenty tasks with one run each will show a difference between two identical configurations about half the time, because a single trial per task hides both task-difficulty spread and run-to-run noise. Reporting the best of three runs makes the distortion worse, since selection inflates whichever configuration was given more retries or warmer caches. The interval is the only part of the report that tells a skeptical reader whether the observed gap is larger than the noise floor of the suite itself.

Size the trial with a variance pilot

Variance is a property of the task set, not of the vendor, and it can be estimated before anyone approves a real budget. A pilot runs a small number of matched pairs and discards the resulting score entirely; the only output worth keeping is the paired standard deviation, which then decides how many pairs the publishable run needs. Free model access is a reasonable place to host that pilot, because the work is repetitive, the score is disposable, and the sample does not need production capacity. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same applies to the free server option, which is a convenient home for a long-running collector rather than a latency reference point.

import math

def required_pairs(sd, half_width, z=1.96):
    """Pairs needed so the 95% interval half-width is at most half_width."""
    return math.ceil((z * sd / half_width) ** 2)

# Illustrative arithmetic, not a measured claim about any model.
print(required_pairs(sd=0.18, half_width=0.05))  # 50 pairs
print(required_pairs(sd=0.31, half_width=0.05))  # 148 pairs

The first number matters more than it looks. A suite whose paired spread is small needs fifty pairs to resolve a five-point change, while a noisier suite needs roughly three times that, which is exactly the kind of cost decision a pilot should inform before the run starts.

The artifact: paired log and bootstrap interval

The log format is deliberately boring, because the analysis should never depend on a dashboard. Each row records one trial: the task identifier, the seed, the configuration name, a score in the closed unit interval, a cost figure, and a timestamp. Pairs are matched on (task_id, seed) so the comparison is within-task rather than between two differently sampled populations.

{"task_id": "repo-qa-014", "seed": 3, "config": "candidate", "score": 1.0, "cost_usd": 0.0041, "ts": "2026-10-15T09:02:11Z"}
{"task_id": "repo-qa-014", "seed": 3, "config": "baseline", "score": 0.0, "cost_usd": 0.0038, "ts": "2026-10-15T09:02:19Z"}

The analysis script is pure standard library, so it runs anywhere the log can be copied. It pairs the rows, computes per-task deltas, and resamples the deltas with a fixed seed so two analysts reading the same file get the same interval.

#!/usr/bin/env python3
"""paired_interval.py - paired bootstrap interval for two agent configs."""
import argparse, json, random, statistics
from collections import defaultdict


def load(path):
    pairs = defaultdict(dict)
    with open(path) as fh:
        for line in fh:
            if not line.strip():
                continue
            rec = json.loads(line)
            pairs[(rec["task_id"], rec["seed"])][rec["config"]] = float(rec["score"])
    return pairs


def paired_deltas(pairs, a, b):
    out = []
    for key in sorted(pairs):
        runs = pairs[key]
        if a in runs and b in runs:
            out.append(runs[a] - runs[b])
    return out


def bootstrap_ci(deltas, resamples, alpha, seed):
    rng = random.Random(seed)
    n = len(deltas)
    means = []
    for _ in range(resamples):
        total = 0.0
        for _ in range(n):
            total += deltas[rng.randrange(n)]
        means.append(total / n)
    means.sort()
    lo = means[int((alpha / 2) * resamples)]
    hi = means[int((1 - alpha / 2) * resamples) - 1]
    return lo, hi


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("log")
    ap.add_argument("--a", default="candidate")
    ap.add_argument("--b", default="baseline")
    ap.add_argument("--resamples", type=int, default=10000)
    ap.add_argument("--alpha", type=float, default=0.05)
    ap.add_argument("--seed", type=int, default=7)
    args = ap.parse_args()

    pairs = load(args.log)
    deltas = paired_deltas(pairs, args.a, args.b)
    if len(deltas) < 8:
        print(f"warning: only {len(deltas)} complete pairs, interval is uninformative")
    mean = statistics.fmean(deltas) if deltas else 0.0
    sd = statistics.stdev(deltas) if len(deltas) > 1 else 0.0
    lo, hi = bootstrap_ci(deltas, args.resamples, args.alpha, args.seed)
    print(json.dumps({
        "pairs": len(deltas),
        "mean_delta": round(mean, 4),
        "paired_sd": round(sd, 4),
        "ci_low": round(lo, 4),
        "ci_high": round(hi, 4),
        "resamples": args.resamples,
        "seed": args.seed,
        "crosses_zero": lo <= 0.0 <= hi,
    }, indent=2))


if __name__ == "__main__":
    main()

python paired_interval.py trials.jsonl --a candidate --b baseline --resamples 20000 --seed 7

Controls that keep the interval honest

A paired interval inherits every flaw in the pairing, so the pairing rules deserve as much review as the statistics. Each pair should share the task, the seed, the tool timeout, and the retry policy, and each configuration should run in its own process so a warm cache from one side cannot leak into the other. The task list must be frozen and hashed before the run, because adding easy tasks midway shifts both the mean and the spread in ways the interval cannot detect. Cost per trial belongs in the log as well, so a narrow interval that required four times the tokens is visible rather than implied.

Cluster effects are the quiet failure mode. Ten tasks drawn from one repository behave more like one or two independent observations than like ten, and a naive bootstrap will understate the interval accordingly. Resampling whole task families instead of individual pairs is the standard remedy, and it is worth doing whenever the suite spans a small number of repositories or workflows.

An append-only collector on the free server

A trial log split across laptops is not evidence, and the difference between a log and evidence is usually just whether writes are timestamped and sequential. A small collector keeps that property, accepts one JSON object per request, validates the required fields, and appends a canonical line to a single file. The server option is a reasonable host for this because the process is idle most of the day and only needs to accept occasional posts from a runner.

#!/usr/bin/env python3
"""collector.py - append-only trial log. Add auth before exposing this."""
import json, os
from http.server import BaseHTTPRequestHandler, HTTPServer

LOG = os.environ.get("TRIAL_LOG", "trials.jsonl")
REQUIRED = {"task_id", "seed", "config", "score"}


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        try:
            rec = json.loads(self.rfile.read(length))
            if not REQUIRED <= set(rec):
                raise ValueError("missing fields: " + ", ".join(sorted(REQUIRED - set(rec))))
        except Exception as exc:
            self.send_response(400)
            self.end_headers()
            self.wfile.write(str(exc).encode())
            return
        with open(LOG, "a") as fh:
            fh.write(json.dumps(rec, sort_keys=True) + "\n")
        self.send_response(202)
        self.end_headers()

    def log_message(self, *args):
        pass


HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

python collector.py &
curl -s -X POST localhost:8080 -H 'Content-Type: application/json' \
  -d '{"task_id":"repo-qa-014","seed":3,"config":"candidate","score":1.0,"cost_usd":0.0041}'
wc -l trials.jsonl

Read the interval before publishing anything

The decision rule is mechanical once the interval exists, and it removes most of the pressure to round a result upward.

Pairs	Paired sd	Mean delta	95% interval	Action
20	0.31	+0.06	[-0.08, +0.20]	Collect more pairs; do not publish a delta
120	0.19	+0.05	[+0.02, +0.08]	Publish with the interval and the task hash
120	0.19	+0.02	[-0.01, +0.05]	Report as inconclusive, not as a tie
120	0.19	+0.05	Wide after clustering	Report per task family, not pooled

The rows are illustrative arithmetic from the formulas above rather than measured results, and the point they make is the one a marketing page usually omits: samples that small settle very few questions. An interval that crosses zero is a real finding, and calling it a tie quietly replaces a measurement with a preference.

Limits and who should skip this

Free access and a free server change the budget, not the physics of the measurement. Free tiers can queue requests or apply rate limits, so wall-clock latency measured there must never be compared against paid runs, and the pilot is useful only for the paired spread. A single unauthenticated collector should not be exposed to the public internet, since anyone could append fabricated rows to the evidence file. Bootstrap intervals also assume the resampled units are close to independent, which fails whenever a few task families dominate the suite.

Teams certifying safety-critical agent behavior should not rely on this workflow, because a bootstrap interval says nothing about tail risk and nothing about the failure modes that never appear in the task set. The same caution applies to anyone who needs sub-percent resolution, or who has not frozen a task list yet; the workflow measures a fixed suite and cannot repair a suite chosen after the results were seen. Anyone who only needs a rough directional read should skip the collector entirely and keep the paired comparison, since the analysis is the part that changes decisions.

A useful next step is to run the variance pilot, write the resulting spread into the run plan, and record the date and task hash beside every number that later leaves the repository. The free model access and free server are the cheapest place to reproduce that pilot, provided the interval and not the point estimate is what gets published.

Refusal Slices Keep Agent Scores Honest

Avery Wang — Tue, 13 Oct 2026 02:38:30 +0000

An agent score is trustworthy only when the dataset, the metrics, and the controls are frozen before the first scored run. A convenient pass on a free server can debug the harness, yet that same pass cannot become a public claim. The method below treats a dry run as rehearsal and reserves the headline number for a locked protocol with a refusal slice. Readers who skip that lock will publish a demonstration of fluency rather than a measurement of task completion.

Public write-ups often celebrate a clean streak after several quiet retries, which hides the policy that actually produced the streak. An agent that never abstains can look stronger than an agent that refuses an underspecified task and asks for the missing file. Those two behaviors are not the same capability, even when both transcripts end with a green check. A score that counts only final success will reward confident fabrication and will punish a careful refusal.

The dataset in this method is a small sealed pack of thirty tasks, split into twenty ordinary tasks and ten negative-control tasks. Ordinary tasks have a single expected artifact, such as a function, a failing test that must turn green, or a short migration note. Negative-control tasks omit a required input on purpose, so the honest outcome is an abstention rather than a guessed patch. The pack is hashed before any model call, and the hash travels with every later report.

Four metrics travel together, and none of them is allowed to stand alone as a slogan. Task success is the fraction of ordinary tasks whose emitted artifact matches the frozen oracle byte for byte. Abstention accuracy is the fraction of negative-control tasks that stop without inventing a missing file or a hidden requirement. Invalid completion is the fraction of negative-control tasks that still emit a confident patch, which is a failure even if the patch compiles.

A report that quotes task success without abstention accuracy is incomplete, because a perfect streak can hide total failure on the refusal slice. Retry count, tool-call count, and wall time are covariates, not decorations, and they must be printed beside the rates. Changing the retry ceiling after a disappointing run creates a new experiment, even when the dataset hash stays the same. That is why these numbers cannot be lifted into marketing copy: they describe one locked procedure, not a general property of a product.

Controls are boring on purpose, and that boredom is exactly the point of a publishable method. The decoder settings, the tool allow-list, the retry ceiling, and the route name are written into a manifest before the scored lane starts. A route lock means the scored lane may not silently fall over to a different model when the first route is busy. If the route changes, the run is labeled exploratory and is excluded from the published comparison table.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project and supplies free model access plus a free server as the exploratory lane for this harness. That lane is useful for checking that the hash step and the scorer actually execute before any scored claim is attempted. Current model names, quotas, hardware, and availability windows are not assumed here and must be read from the project documentation at run time.

A dry run on the free lane never inherits the authority of the sealed pack, because convenience and measurement answer different questions. The exploratory lane may use whatever free model access is currently offered, and it may be repeated while the scorer is still being debugged. The scored lane must name the route, freeze the manifest, and refuse to start if the pack hash disagrees with the recorded digest. Mixing those lanes in one table is how a debugging session turns into an accidental advertisement.

Availability claims for the rehearsal lane stay operator-supplied and are not turned into a quota, a hardware profile, or a permanence promise. A team should confirm the current terms in the project documentation before a scored night, because free access can change without notice. The same methodology remains usable with any other route that can emit the transcript shape the scorer expects. A changed term sheet does not rewrite an older scored manifest, so yesterday's number cannot be silently rebranded.

The artifact is a small Python harness that hashes the task pack, rejects a drifted manifest, and scores success separately from abstention. It is a proposal for local reproduction, not a log from an executed vendor benchmark, and it calls no network endpoint. A reviewer can run the harness against local fixture files before any agent route is connected to the scored lane. The point of the code is to make an invalid comparison fail closed, rather than to crown a particular model.

#!/usr/bin/env python3
"""Proposal scorer for a sealed agent pack. Not an executed vendor result."""

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = ("route", "temperature", "retry_ceiling", "tool_allowlist", "lane")


def pack_digest(task_dir: Path) -> str:
    hasher = hashlib.sha256()
    files = sorted(path for path in task_dir.glob("*.json") if path.is_file())
    for path in files:
        hasher.update(path.name.encode())
        hasher.update(b"\0")
        hasher.update(path.read_bytes())
        hasher.update(b"\0")
    return hasher.hexdigest()


def load_manifest(path: Path) -> dict:
    manifest = json.loads(path.read_text(encoding="utf-8"))
    missing = [key for key in REQUIRED if key not in manifest]
    if missing:
        raise SystemExit("manifest missing controls: " + ", ".join(missing))
    if manifest["lane"] != "scored":
        raise SystemExit("refusing to score an exploratory lane")
    if int(manifest["retry_ceiling"]) < 0:
        raise SystemExit("retry_ceiling must be non-negative")
    return manifest


def score_task(task: dict, transcript: dict) -> str:
    abstained = bool(transcript.get("abstained"))
    artifact = transcript.get("artifact") or ""
    if task["kind"] == "negative":
        if abstained and artifact == "":
            return "abstain_ok"
        return "invalid_completion"
    if abstained:
        return "false_abstain"
    if artifact == task.get("oracle", ""):
        return "success"
    return "miss"


def rate(rows, kind, label):
    subset = [row for row in rows if row["kind"] == kind]
    if not subset:
        return None
    hits = sum(row["label"] == label for row in subset)
    return round(hits / len(subset), 4)


def main() -> None:
    task_dir = Path(sys.argv[1])
    manifest = load_manifest(Path(sys.argv[2]))
    transcripts = json.loads(Path(sys.argv[3]).read_text(encoding="utf-8"))
    digest = pack_digest(task_dir)
    if manifest.get("pack_sha256") != digest:
        raise SystemExit("pack hash drifted; scored lane aborted")
    rows = []
    for path in sorted(task_dir.glob("*.json")):
        task = json.loads(path.read_text(encoding="utf-8"))
        label = score_task(task, transcripts[path.stem])
        rows.append({"task": path.stem, "kind": task["kind"], "label": label})
    report = {
        "pack_sha256": digest,
        "route": manifest["route"],
        "retry_ceiling": manifest["retry_ceiling"],
        "lane": manifest["lane"],
        "task_success": rate(rows, "ordinary", "success"),
        "abstention_accuracy": rate(rows, "negative", "abstain_ok"),
        "invalid_completion": rate(rows, "negative", "invalid_completion"),
        "rows": rows,
    }
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main()

A minimal fixture keeps one ordinary task and one negative task, which is enough to prove the scorer before the thirty-task pack exists. The ordinary file stores an oracle string, and the negative file stores an empty oracle plus a note that the required schema was withheld. The transcript file then records whether the agent abstained and what artifact it emitted, using the task stem as the key. Running the scorer against a copied manifest with a stale hash should abort, and that abort is the test the reviewer should see first.

{
  "fix-parser.json": {
    "kind": "ordinary",
    "oracle": "def parse(line):\n    return line.strip().split(',')\n"
  },
  "missing-schema.json": {
    "kind": "negative",
    "oracle": "",
    "withheld": "schema"
  }
}

{
  "lane": "scored",
  "route": "REPLACE_WITH_ROUTE_FROM_CURRENT_DOCS",
  "temperature": 0,
  "retry_ceiling": 1,
  "tool_allowlist": ["read_file"],
  "pack_sha256": "PASTE_DIGEST_BEFORE_SCORING"
}

mkdir -p tasks/sealed
python3 -m py_compile score_pack.py
python3 score_pack.py tasks/sealed manifest.json transcripts.json
python3 -c 'import hashlib, pathlib; d=pathlib.Path("tasks/sealed"); h=hashlib.sha256();
[ (h.update(p.name.encode()), h.update(b"\0"), h.update(p.read_bytes()), h.update(b"\0")) for p in sorted(d.glob("*.json")) ]; print(h.hexdigest())'

The command exits before scoring when a control is missing, when the lane is exploratory, or when the pack digest disagrees. A zero exit therefore means only that the protocol held, not that the agent produced an impressive result. The printed rates should travel with the route name and the retry ceiling, because those fields define the experiment. A later reader who cannot recover the digest cannot audit the number, and an unauditable number should stay unpublished.

Consider a hypothetical pack of twenty ordinary tasks and ten negative controls, scored under one named route and a retry ceiling of one. Suppose sixteen ordinary tasks match the oracle, while seven negative controls abstain and three emit a patch anyway. Task success is then 0.80, abstention accuracy is 0.70, and invalid completion is 0.30, which is a mixed result rather than a victory lap. Publishing only the 0.80 figure would conceal the slice where the agent invented missing inputs and still looked fluent.

The refusal slice works like a control sample in a lab notebook, sitting beside the treatment so a clean result cannot stand alone. Without that neighboring sample, a green streak is closer to a demo reel than to a careful measurement. The free server option fits the notebook's margin, where a reviewer checks pens, seals, and arithmetic before the official page is written. It does not fit the official page itself, unless the same route lock and the same pack hash governed that run.

A fair comparison between two routes keeps the same pack digest, retry ceiling, and tool allow-list, changing only the route field. If one route abstains more often and succeeds less often, the table should show both movements instead of declaring a winner from success alone. If one route needed hidden retries inside the runtime, that fact belongs in the covariate column beside the manifest ceiling. The absence of a paired manifest is itself a reason to withhold the comparison from any public note.

Recent public arguments about retries and agreeable models point at the same gap this protocol tries to close. A streak can be an artifact of a generous retry ceiling, and agreement can be an artifact of an undeserving task. Neither observation requires a named leaderboard, a prize purse, or a fresh model announcement to be useful on a Tuesday afternoon. The practical response is to freeze the pack, score the refusal slice, and refuse to promote a lane that was only meant for rehearsal.

This approach is deliberately narrow, and several familiar evaluation problems sit entirely outside the scope of the harness. The oracle comparison assumes a deterministic artifact, so multi-answer tasks need a rubric or a property test first. Thirty tasks can expose a broken harness, yet they cannot support a leaderboard claim about a model family. Temperature drift, hidden tool retries, and a nondeterministic sandbox can still move the rates even when the manifest looks frozen.

Teams that need a procurement number, a regulated benchmark, or a cross-vendor ranking should not use this pack as evidence. A lunch-bound product demo should also skip it, because the refusal slice will slow the story and may embarrass a polished agent. Researchers who already run a large public suite gain little from a thirty-task rehearsal unless they adopt the same route lock on that suite. The method is for engineers who want a publishable local protocol before they quote any agent percentage in public.

Engineers who keep benchmark notes can place this scorer beside a free-server rehearsal, then decide whether a sealed night is warranted. That placement is a workflow choice, not an invitation to treat a rehearsal transcript as a public result. The pack hash, the refusal slice, and the route lock remain the parts worth copying even if that free lane is never used. If those three controls are missing, the percentage should remain in the lab notebook and off the public page.

A Cheap Run Needs a Fixed Holdout Pack

Avery Wang — Sun, 11 Oct 2026 22:07:17 +0000

A cheap runner can exercise a coding agent, yet that exercise does not become a capability ranking by itself. The honest product of the run is a labeled observation that names the dataset, the controls, and the clock. Readers should treat any single score from an unpaid lane as a lane note rather than a market claim. A publishable comparison starts only after those boundaries sit in a card that another engineer can replay.

Coding-agent notes often arrive with a polished percentage and almost no account of the task pack behind it. That pattern resembles a race result printed without the track length, the wind, or the class of car. A developer comparing tools then confuses access to a runner with proof that the agent solves a held-out class of work. The confusion grows when a free route lowers the cost of trying ideas while leaving the measurement problem untouched.

MonkeyCode can host that rehearsal when the operator uses its free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those availability claims describe access only, and they do not state quotas, hardware, duration, or permanent terms. A card that hard-codes a token grant or a server size would invent facts the operator must read from current documentation at trial time.

The pack is the stick

The dataset is a small task pack whose manifest, content hash, and holdout split stay fixed after the first scored run. Each item carries a stable identifier, a language tag, a fixture directory, and a pass rule that a script can apply. Training-like prompts, leaked solutions, and tasks copied from a vendor demo stay in an excluded bucket outside the scored set. The pack works like a measuring stick, and sanding that stick after the number appears turns the trial into a story.

Useful metrics should describe completion, repair cost, and instability rather than collapsing the trial into one vanity percentage. Completion is the fraction of holdout items whose tests pass when the frozen command runs without manual edits. Repair cost is the median tool-call count, kept apart from median wall time so a fast failure is not a cheap success. Instability is the mean pass spread across repeats of one holdout item, so one lucky seed cannot stand for the lane.

Controls exist so the free route is not quietly compared with a different class of runner or clock. The same task hash, the same timeout, the same retry ceiling, and the same network policy should bind every arm. A free server may queue, throttle, or rotate capacity, so the card stores the start time and a lane label. If the route is selected dynamically, the log captures the route name returned at runtime rather than a name guessed in the draft.

A harness that refuses a rank

The workflow below is a proposed harness, not a result from an executed study and not a claim about any live quota. An author drops fixtures into a directory, writes a manifest, and lets the script refuse a score when required controls are missing. That refusal is the point, because a missing control is more informative than a confident table built on an unlocked pack. The same script can later sit beside a paid runner, which keeps the comparison about the card rather than about a brochure.

#!/usr/bin/env python3
"""Proposed, unexecuted lane card. It records controls and refuses a rank."""

from __future__ import annotations

import hashlib
import json
import statistics
import time
from pathlib import Path

REQUIRED = ("task_hash", "lane_label", "timeout_s", "retry_ceiling", "network_policy")

def hash_pack(root: Path) -> str:
    digest = hashlib.sha256()
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        digest.update(path.relative_to(root).as_posix().encode())
        digest.update(path.read_bytes())
    return digest.hexdigest()

def summarize(trials: list[dict]) -> dict:
    holdout = [item for item in trials if item.get("split") == "holdout"]
    by_item: dict[str, list[int]] = {}
    calls, walls = [], []
    for item in holdout:
        by_item.setdefault(str(item.get("id", "unknown")), []).append(
            1 if item.get("passed") else 0
        )
        if "tool_calls" in item:
            calls.append(item["tool_calls"])
        if "wall_ms" in item:
            walls.append(item["wall_ms"])
    spreads = [max(flags) - min(flags) for flags in by_item.values() if flags]
    passed = sum(flags[-1] for flags in by_item.values()) if by_item else 0
    return {
        "holdout_items": len(by_item),
        "completion": (passed / len(by_item)) if by_item else None,
        "median_tool_calls": statistics.median(calls) if calls else None,
        "median_wall_ms": statistics.median(walls) if walls else None,
        "mean_item_pass_spread": statistics.mean(spreads) if spreads else None,
    }

def build_card(root: Path, trials: list[dict], controls: dict) -> dict:
    missing = [key for key in REQUIRED if not controls.get(key)]
    if missing:
        raise SystemExit(f"refusing score; missing controls: {', '.join(missing)}")
    if controls["task_hash"] != hash_pack(root):
        raise SystemExit("refusing score; task pack hash drifted")
    return {
        "generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "task_hash": controls["task_hash"],
        "lane_label": controls["lane_label"],
        "controls": controls,
        "metrics": summarize(trials),
        "not_marketing": [
            "not a general coding-ability claim",
            "not a price or quota claim",
            "not valid after the pack or route changes",
        ],
        "rank_emitted": False,
    }

if __name__ == "__main__":
    root = Path("task_pack")
    controls = json.loads(Path("controls.json").read_text())
    trials = json.loads(Path("trials.json").read_text())
    card = build_card(root, trials, controls)
    Path("lane_card.json").write_text(json.dumps(card, indent=2) + "\n")
    print(json.dumps({"status": "card_written", "rank_emitted": False}))

A companion control file keeps the route explicit, and the example values below are placeholders rather than observed product limits. The timeout, the retry ceiling, and the network policy are choices of the study, not features inferred from a vendor page. The lane label should quote the access class that the operator confirmed on the day of the run. Leaving those fields blank is preferable to filling them with a remembered grant from an older post.

{
  "task_hash": "<sha256 of task_pack>",
  "lane_label": "free-model-access/free-server; terms read at runtime",
  "timeout_s": 120,
  "retry_ceiling": 1,
  "network_policy": "no egress beyond the fixture mirror"
}

The commands that follow are a local rehearsal sketch, and they do not contact a model or reserve a server. An author can create the pack, compute nothing remote, and confirm that a missing label stops the score. That dry run is useful because it teaches the refusal path before any token is spent on a live route. A later live run should append the route name returned by the server, then regenerate the card without editing the holdout fixtures.

mkdir -p task_pack/holdout/item-01
printf '%s\n' 'id: item-01' 'split: holdout' > task_pack/holdout/item-01/manifest.txt
printf '%s\n' '{"timeout_s":120,"retry_ceiling":1,"network_policy":"fixture-mirror-only"}' > controls.json
printf '%s\n' '[]' > trials.json
python3 lane_card.py
# expected: refusing score; missing controls: task_hash, lane_label

A written card that survives the checks still carries rank_emitted set to false, which blocks a casual sort of vendors. The metrics object may show a completion fraction, two cost medians, and a pass spread, each tied to the stored hash. A reader who cannot find the negative claims should treat the file as incomplete even if the fraction looks strong. That rule is the difference between a lab note and a sentence that could be pasted into an advertisement.

What the numbers are not allowed to say

These numbers are not marketing because they answer a narrow question under named constraints, and they expire when those constraints change. A higher completion rate on a small pack does not prove general coding ability, a lower price, or a permanent free grant. The card should state the negative claims in plain language, including what was not measured and which arms were absent. Publishing the hash next to the table lets a later reader see whether the pack moved after the headline was written.

Failure analysis belongs in the same narrative as the score, because a green fraction without its misses invites overreading. A timeout cluster may say more about queue behavior on a free server than about the agent's repair skill. A pass that appears only after a second retry tells a different story from a pass on the first frozen attempt. Recording those patterns beside the median keeps the result closer to a lab note than to a launch sentence.

Synthetic trials can teach the reader how to hold that line before a live route is involved at all. Suppose two repeats of one holdout item pass on the first seed and fail on the second, while wall time jumps from a quiet run to a queued run. The completion field may still look acceptable if only the last flag is counted, but the pass spread exposes the instability. A reviewer who sees that spread should keep the lane label attached and decline to promote the fraction into a ranking table.

Who should leave this method alone

This approach should not be used to certify a model for production, to claim a security review, or to rank vendors for a purchase committee. It also fails when the task pack is private, when tests depend on live paid APIs, or when the author edits fixtures after seeing failures. A free server option can disappear, slow down, or change routing, so a result from one afternoon is a sample of that afternoon. Teams that need a contractual service level should repeat the same card on an explicit environment before they trust the delta.

An engineer rehearsing this card can check MonkeyCode's current free model access and free server option before sharing a score. The live terms should be copied into the lane label before any score is shared with another reader. The useful habit is the refusal to rank an unlocked pack, and that habit remains valuable even if the hosting choice changes. A later reader should be able to reject the number without guessing which track, which wind, or which class of car produced it.

Free-Tier Agent Runs Are Control Lanes

Avery Wang — Sat, 10 Oct 2026 18:49:34 +0000

A convenient agent run on a free model route is a control lane, not a publishable result. The figure becomes interpretable only when canary tasks pass, negative controls remain failed, and the grader never sees the route label. Without those three conditions, a high pass rate mostly records fluency, harness luck, or a judge that rewards confident prose. Teams that skip the gate often circulate the decimal first and discover the broken fixture only after the write-up is public.

The pattern appears whenever a cheaper model seems to match a stronger one on a handful of visible tasks. A developer watches one clean transcript, reruns the same prompt, and treats the second success as confirmation. That habit confuses a demonstration with a measurement, because the task, the tools, and the judge all moved together. A benchmark, by contrast, fixes the question set and then checks whether the answer still holds after the easy leaks are removed.

Three families, not one trophy slice

The working dataset should contain three families that travel together, rather than a single leaderboard slice. Scored tasks represent the work the team actually claims to measure, with hidden checks stored outside the agent workspace. Canary tasks are small, fully specified, and expected to pass when the harness, the tools, and the instruction format are alive. Negative-control tasks look plausible but require a missing file, a contradictory oracle, or a forbidden sandbox action, so a careful agent abstains.

The metrics worth publishing are paired rates with explicit denominators, not a single trophy percentage from one run. Report the scored pass rate, the canary pass rate, the negative-control violation rate, and the abstention rate on control tasks. A violation means the agent edited a forbidden path, invented a missing file, or received a pass despite an impossible oracle. If the canary pass rate drops, the harness is unhealthy, and the scored rate should remain inside the lab notebook.

Controls exist so a later reader can reproduce the conditions, rather than merely repeating the press sentence. Freeze the task manifest, keep the sandbox image digest in the run log, and hold the retry budget identical across lanes. Blind the grader to model identity and to the route class, because a visible label can leak prestige into the score. Log temperature, tool permissions, and the route identifier as covariates, and refuse any comparison when one of those fields is missing.

These numbers are not marketing copy, because the gate is explicitly allowed to withhold the headline figure. A free-route lane that scores well on visible tasks while violating negative controls has measured eagerness to comply, not capability. A lane that fails its canaries has not measured the model at all, no matter how attractive the scored subset looks. Publishing only the scored rate, after quietly discarding the control families, is how a lab notebook becomes an advertisement.

A gate that can stay silent

The following Python gate is a proposal for the lab notebook, not a record of an executed leaderboard. It reads a manifest of outcomes that some other runner already wrote, then decides whether a headline rate may be emitted. The thresholds below are starting points for discussion, not validated cutoffs borrowed from a public study. Operators should replace those cutoffs after they see how often their own harness fails canaries for mundane reasons.

#!/usr/bin/env python3
"""Proposal only: gate a headline agent rate on canaries and negative controls.

This script does not call a model and does not claim a measured score.
Feed it a JSON manifest produced by your own runner.
"""

from __future__ import annotations

import hashlib
import json
import sys


def manifest_digest(payload: dict) -> str:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(body).hexdigest()


def count_outcome(rows: list[dict], family: str, outcome: str) -> tuple[int, int]:
    chosen = [row for row in rows if row.get("family") == family]
    hits = sum(1 for row in chosen if row.get("outcome") == outcome)
    return hits, len(chosen)


def main() -> int:
    payload = json.load(sys.stdin)
    required = {"route_id", "image_digest", "grader_blinded", "tasks"}
    if required - payload.keys() or not payload.get("grader_blinded"):
        print(json.dumps({"emit_headline": False, "reason": "controls_incomplete"}))
        return 2

    tasks = payload["tasks"]
    canary_hits, canary_n = count_outcome(tasks, "canary", "pass")
    scored_hits, scored_n = count_outcome(tasks, "scored", "pass")
    violations, control_n = count_outcome(tasks, "negative_control", "pass")
    abstentions, _ = count_outcome(tasks, "negative_control", "abstain")

    canary_ok = canary_n > 0 and (canary_hits / canary_n) >= 0.95
    controls_ok = control_n > 0 and (violations / control_n) <= 0.05
    emit = bool(canary_ok and controls_ok and scored_n > 0)

    report = {
        "manifest_sha256": manifest_digest(payload),
        "route_id": payload["route_id"],
        "image_digest": payload["image_digest"],
        "canary_pass": [canary_hits, canary_n],
        "scored_pass": [scored_hits, scored_n],
        "negative_control_violations": [violations, control_n],
        "negative_control_abstentions": abstentions,
        "emit_headline": emit,
    }
    print(json.dumps(report, indent=2))
    return 0 if emit else 3


if __name__ == "__main__":
    raise SystemExit(main())

A local check should stay deliberately boring, which is a real virtue inside a careful measurement pipeline. The command below reads a manifest from standard input and prints the gate decision without contacting a model endpoint. A non-zero status means the headline stays unpublished, even when the scored subset looks flattering in the file. The sample manifest is a synthetic fixture for the gate, not an observed agent result and not a ranking of any host.

python3 gate_agent_rate.py < run_manifest.json
echo "gate_status=$?"

{
  "route_id": "free-route-logged-not-ranked",
  "image_digest": "sha256:example-not-a-real-image",
  "grader_blinded": true,
  "tasks": [
    {"id": "canary-01", "family": "canary", "outcome": "pass"},
    {"id": "neg-01", "family": "negative_control", "outcome": "abstain"},
    {"id": "task-01", "family": "scored", "outcome": "pass"}
  ]
}

A manifest that omits the route identifier, the image digest, or the blinded-grader flag exits before any rate is printed. That early exit is the point of the artifact, because a missing covariate is already a failed comparison. The script treats a pass on a negative-control task as a violation, which feels harsh where partial credit is common. That harshness is preferable to a public chart that cannot explain its own denominator at all.

Where a free route fits

MonkeyCode's free model access and free server option can host the control lane, so a comparison budget is not spent discovering a broken sandbox. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those availability claims belong in the log as a route, not as a promise that the pool will stay fixed. Readers should confirm current terms in the project documentation, and they should keep private repositories and secrets off any shared host.

A route identifier beside the manifest digest shows that the figure belonged to that pool on that day. Nothing in this method requires a particular vendor, and a free lane is not a better or worse model class. The free option matters only because canaries and negative controls are cheap to repeat when a fixture changes, if the route id stays logged. If the documentation no longer offers that access, the same gate still runs on any host that can emit the manifest.

Who should leave this gate alone

This gate does not fit every evaluation, and several limits should stay visible beside the script. Canaries that are too easy can pass while a different bug still corrupts the scored family, so the canary set needs occasional replacement. Negative-control wording, once published in full, can leak into later training data and stop functioning as a control. The shared manifest should store task identifiers and hashes, rather than copying the private oracle text itself.

Automated judges can share blind spots with the agent, so a small human review of violations remains necessary before a rate is quoted externally. Regulated attestations, customer-facing rankings, and workloads that cannot leave a private network should not use a free shared lane for this gate. A single clean transcript is also the wrong input, because the script has nothing honest to say about a denominator of one. Teams that already keep denominators can add this gate, then consult MonkeyCode's current access notes only if a disposable canary host is still needed.

Stratified Trial Budgets for Publishable Agent Scores

Avery Wang — Fri, 09 Oct 2026 15:12:18 +0000

An agent score becomes evidence only after the trial budget, the difficulty strata, and the exclusion rules are frozen. A single clean transcript can still guide a repair, yet one lucky path does not estimate error. The method below treats the headline percentage as a summary of paired trials, not as a brochure claim. A reviewer can reproduce the accounting with a small Python harness before any model name enters the discussion.

A clean run is not a timetable

A clean local run often looks decisive, especially when every visible check happens to pass on the first afternoon. That result can guide the next edit, yet it remains a weather report until the same items repeat under fixed rules. A transit agency does not publish a timetable from one sunny commute, and a lab should not publish a rank from one polished run. The analogy is deliberate, because variance, route difficulty, and cancelled trips all belong in the published record.

What the pack must freeze

The dataset is a frozen task pack split into difficulty strata before any candidate is allowed to run. Each stratum holds tasks of similar expected effort, such as a one-file fix, a multi-file refactor, or a failing integration test. Items carry stable identifiers, and the pack records the stratum label beside the prompt path and the oracle command. A later chart may omit a stratum, but the scorer may not silently move an item into an easier bin.

Without strata, a model that excels at short edits can outscore a model that survives harder repairs, and the average hides the swap. Promotional copy prefers that average, because a single percentage travels farther than a table of uneven wins. The methodology therefore reports a within-stratum paired difference, and only then a predeclared summary across strata. If the summary and the strata disagree, the strata remain the result and the summary becomes a footnote.

Metrics that refuse a slogan

The primary metric is the paired outcome on the same item, not an unpaired accuracy computed on different samples. Each trial records pass or fail against the oracle, the wall time, and whether the harness aborted for a logged reason. A secondary metric is the stratum win rate, defined as the share of items where the candidate beats the control. Neither metric is a quality slogan, because both inherit every exclusion that the log has already accepted.

Controls exist to stop the number from drifting into advertising copy after the run looks favorable. The control arm uses the same harness, the same oracle, and the same retry ceiling as the candidate arm. A placebo item with an impossible oracle must fail, or the scorer treats the whole batch as untrustworthy. That negative control is the fire alarm, not an optional appendix for readers who enjoy methodology notes.

The trial budget is chosen before scores appear, and it states how many independent repeats each item receives. Two repeats can reveal a flip, while a larger budget is required before a small gap deserves a public rank. This draft does not prescribe a universal sample size, because task variance differs across repositories and oracles. It does require the chosen budget to be written into the manifest, so a disappointing repeat cannot be deleted quietly.

Exclusions are the place where an honest lab and a promotional writeup usually diverge in practice. A pre-registered rule may drop an item for a missing fixture, a leaked solution, or an oracle that no longer builds. A post-hoc rule that drops an item because the candidate failed is not an exclusion, and the harness should refuse it. The log stores the rule identifier, the item identifier, and the timestamp, so a reviewer can replay the decision.

A harness that enforces the freeze

The following Python module is a proposed harness, and it has not been executed against a live scoring service. It hashes the manifest, checks paired outcomes, and rejects any exclusion that lacks a pre-registered rule. Operators should treat the script as a starting ledger, then replace the toy fields with their own oracle records. The point is the accounting, not a claim that any particular model already passed these checks.

#!/usr/bin/env python3
"""Proposed trial-budget harness.

This example has not been executed against a live scoring service.
Replace the fixture files before using it on a real task pack.
"""

from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

RULES = {
    "missing_fixture": "Oracle inputs were absent before the candidate started.",
    "leaked_solution": "A reference patch was visible in the prompt context.",
    "oracle_broken": "The oracle command failed on the untouched baseline.",
}


def manifest_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_manifest(path: Path) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    required = {"pack_id", "trial_budget", "strata", "items", "exclusion_rules"}
    missing = sorted(required.difference(data))
    if missing:
        raise SystemExit(f"manifest missing fields: {missing}")
    budget = data["trial_budget"]
    if not isinstance(budget, int) or budget < 1:
        raise SystemExit("trial_budget must be a frozen integer of at least 1")
    return data


def accept_exclusion(rule_id: str, registered: list[str]) -> None:
    if rule_id not in registered or rule_id not in RULES:
        raise SystemExit(f"post-hoc exclusion refused: {rule_id}")


def paired_delta(candidate: list[bool], control: list[bool]) -> dict:
    if len(candidate) != len(control) or not candidate:
        raise SystemExit("paired arms require equal non-empty outcomes")
    wins = losses = ties = 0
    for cand, base in zip(candidate, control):
        if cand == base:
            ties += 1
        elif cand and not base:
            wins += 1
        else:
            losses += 1
    return {"wins": wins, "losses": losses, "ties": ties, "n": len(candidate)}


def self_check() -> None:
    delta = paired_delta([True, False], [False, False])
    expected = {"wins": 1, "losses": 0, "ties": 1, "n": 2}
    if delta != expected:
        raise SystemExit("paired_delta self-check failed")
    try:
        accept_exclusion("candidate_failed", ["missing_fixture"])
    except SystemExit:
        print("self-check ok")
        return
    raise SystemExit("self-check failed to refuse a post-hoc exclusion")


def evaluate(manifest_path: Path, outcomes_path: Path) -> dict:
    manifest = load_manifest(manifest_path)
    digest = manifest_hash(manifest_path)
    outcomes = json.loads(outcomes_path.read_text(encoding="utf-8"))
    if outcomes.get("manifest_sha256") != digest:
        raise SystemExit("outcome file does not match frozen manifest hash")
    for event in outcomes.get("exclusions", []):
        accept_exclusion(event["rule_id"], manifest["exclusion_rules"])
    by_stratum: dict[str, list[dict]] = {}
    for item in manifest["items"]:
        arm = outcomes["items"][item["id"]]
        if len(arm["candidate"]) != manifest["trial_budget"]:
            raise SystemExit(f"budget mismatch: {item['id']}")
        if len(arm["control"]) != manifest["trial_budget"]:
            raise SystemExit(f"control budget mismatch: {item['id']}")
        if item.get("placebo") and any(arm["candidate"]):
            raise SystemExit(f"placebo passed: {item['id']}")
        by_stratum.setdefault(item["stratum"], []).append(
            paired_delta(arm["candidate"], arm["control"])
        )
    return {
        "manifest_sha256": digest,
        "pack_id": manifest["pack_id"],
        "strata": by_stratum,
    }


def main() -> None:
    if len(sys.argv) == 2 and sys.argv[1] == "self-check":
        self_check()
        return
    if len(sys.argv) != 3:
        raise SystemExit(
            "usage: trial_budget.py self-check | trial_budget.py PACK OUTCOMES"
        )
    report = evaluate(Path(sys.argv[1]), Path(sys.argv[2]))
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

A reviewer can freeze the pack with a checksum before the first candidate process is allowed to start. The commands below assume a local checkout, a Python 3 interpreter, and no network call inside the toy oracle. They illustrate the control path, and they do not measure a hosted product or imply a hardware specification. If a command fails, the failure stays in the log rather than disappearing into a revised average.

python3 trial_budget.py self-check
python3 -c "import hashlib, pathlib; print(hashlib.sha256(pathlib.Path('task_pack.json').read_bytes()).hexdigest())"
python3 trial_budget.py task_pack.json outcomes.json

The sample manifest freezes a two-stratum pack, a trial budget of two, and the three exclusion rules the harness recognizes. The sample outcomes file must carry the same manifest hash, or the script exits before it prints a table. A placebo item that passes on the candidate arm aborts the batch, which is the intended failure for a broken scorer. Those files are fixtures for the method, and they are not measurements of any hosted coding agent.

{
  "pack_id": "strata-demo-001",
  "trial_budget": 2,
  "strata": ["short_edit", "multi_file"],
  "exclusion_rules": ["missing_fixture", "leaked_solution", "oracle_broken"],
  "items": [
    {
      "id": "short-01",
      "stratum": "short_edit",
      "prompt_path": "prompts/short-01.md",
      "oracle": "pytest -q tests/test_short_01.py",
      "placebo": false
    },
    {
      "id": "placebo-oracle",
      "stratum": "short_edit",
      "prompt_path": "prompts/placebo.md",
      "oracle": "false",
      "placebo": true
    }
  ]
}

{
  "manifest_sha256": "<paste sha256 of task_pack.json>",
  "exclusions": [],
  "items": {
    "short-01": {"candidate": [true, false], "control": [false, false]},
    "placebo-oracle": {"candidate": [false, false], "control": [false, false]}
  }
}

Where a hosted option fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option matter only as the candidate runtime, while the ledger stays local. That availability is only an operator-supplied convenience, and this draft does not verify quotas, hardware, duration, or model names. A team may copy the limits it actually sees into the run notes, without treating free capacity as a quality score.

Why the percentage is not marketing

A percentage becomes marketing when the audience cannot see the pack, the budget, the exclusions, or the negative control. The same percentage becomes a lab result when those four artifacts travel with the figure and a reviewer can recompute it. Publishing the strata beside the summary also blocks the common slide that hides a loss on hard items. If a stakeholder wants only the largest number, the writeup should withhold the rank rather than inflate the claim.

Limits, and who should walk away

This approach does not detect training-set contamination beyond the canary items the pack author remembered to include. It does not price the runs, and a cheaper failure can still be the better engineering choice for a given team. Wall-clock comparisons remain fragile when a free runner and a paid runner differ in queueing, which this toy script cannot see. Small strata will swing wildly, so a two-item bin should be reported as a case series rather than as a rate.

Teams that need a vendor ranking for a launch post this week should not use this harness as decoration. The method also fits poorly when the oracle itself is disputed, because a shared wrong test will crown the wrong patch. Researchers who cannot freeze the task pack, or who must tune prompts after seeing scores, will violate the exclusion rule immediately. In those settings the honest output is a demo transcript with a date, not a benchmark table.

The practical next step is to hash a small pack, run the placebo item, and confirm that a bad exclusion is rejected. Only after that check should a candidate arm consume either a local process or a hosted free option. The resulting table will look quieter than a leaderboard, which is the intended trade for a number that can survive review. Quiet evidence is still the result a later reader can trust when the demo transcript has already scrolled out of view.

Freeze the Benchmark Card Before an Agent Score Ships

Avery Wang — Thu, 08 Oct 2026 10:41:03 +0000

A coding-agent score is evidence only when the dataset, the metric, and the control surface are frozen together. Without that triple lock, a leaderboard cell behaves like a slogan that happens to wear a decimal point. Readers cannot tell whether the number measures the agent, the harness, or the afternoon the run happened to finish. The method below treats the published figure as a claim that must survive inspection, not as a marketing line.

Many public comparisons still ship a single percentage and a model nickname, then invite readers to treat the gap as proof. That pattern fails because the task mix, the grader, and the machine can each move the result by more than the claimed gap. A benchmark card records those moving parts before the number is allowed to travel outside the lab. The card remains a method document, and it stays useful even if every vendor name is stripped from the page.

Dataset construction starts with named families, fixed slice sizes, and a written rule for what never enters the prompt. A family might cover bug repair, test writing, or a narrow refactor, yet each family needs both an inclusion rule and an exclusion rule. Contamination is the quiet failure, because a model can look skilled when the task text already lived in its earlier training mix. The card therefore stores a source note, a cutoff date, and a leakage-check command instead of a vague promise that the set is fresh.

The metric section defines pass, partial credit, and failure in formulas a second team can recompute from raw logs. A pass might require tests to exit zero and the diff to stay inside a stated line budget. Partial credit needs its own formula, or later readers will invent a generous reading after they see which arm looked weak. The card also states what the metric ignores, including security review, latency under load, and long-horizon design judgment.

Controls exist so a delta can be attributed to the agent rather than to weather in the network or drift in the laptop. A paired design runs the same task identifier, the same seed policy, and the same timeout on every arm before any ranking starts. An unchanged reference arm, often a fixed script or a frozen model build, shows whether the harness itself moved between weeks. If that reference arm moves, new scores stay quarantined until the drift is explained, because the comparison is no longer clean.

Numbers become marketing when the writer selects the slice after seeing the results, then publishes only the flattering cut. The card blocks that move by requiring the slice plan, the exclusion log, and the analysis script hash to be committed before the run. A pre-registered comparison resembles a lab notebook more than a launch post, and that resemblance is the entire point of the format. Readers should be able to reject the claim without trusting the author's tone, which is why the required fields stay mechanical.

A checker that refuses a bare number

The artifact below is a proposal, not a result from a finished evaluation, and no figure inside it should be quoted as measured. It rejects a report that omits the dataset, the metric, the controls, or an explicit clause saying the number is not an advertisement. Teams can place the checker in continuous integration so a missing field fails the build instead of failing a later reader. The sample uses synthetic task identifiers so the workflow can be inspected without implying that any vendor won a contest.

{
  "dataset_id": "synth-repair-slice-locked",
  "slice_plan": "40 synthetic repair tasks, file list committed before any arm runs",
  "slice_locked_before_run": true,
  "source_note": "hand-written fixtures, not scraped from a public leaderboard",
  "cutoff_date": "record-the-real-cutoff-on-execution-day",
  "metric_id": "paired_pass_rate",
  "metric_formula": "passes / paired_tasks; partial credit is a separate field and is not folded in",
  "metric_ignores": "security review, load latency, long-horizon design judgment",
  "control_arm": "frozen_reference_script_v3",
  "seed_policy": "seed=17 on every arm",
  "timeout_seconds": 120,
  "exclusion_log": "exclusions/pre-run.txt",
  "non_marketing_clause": "This figure is a pre-registered measurement, not an advertisement or a ranking claim.",
  "run_date": "set-on-execution-day",
  "runner_image": "copy-from-current-docs",
  "model_arm": "copy-identifier-on-run-date",
  "score_omitted": "no measured value belongs in this proposal"
}

#!/usr/bin/env python3
"""Unexecuted proposal. Rejects a card that could travel as an ad."""

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = (
    "dataset_id",
    "slice_plan",
    "metric_id",
    "metric_formula",
    "control_arm",
    "seed_policy",
    "timeout_seconds",
    "exclusion_log",
    "non_marketing_clause",
    "run_date",
)

def canonical_hash(payload):
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()

def evaluate(report):
    missing = [key for key in REQUIRED if report.get(key) in (None, "", [])]
    if missing:
        return "reject missing=" + ",".join(missing), 2
    clause = str(report["non_marketing_clause"]).lower()
    if "not an advertisement" not in clause:
        return "reject clause", 3
    if report.get("slice_locked_before_run") is not True:
        return "reject slice_not_locked", 4
    return canonical_hash(report), 0

def paired_delta(control_passes, arm_passes, paired_tasks):
    # Illustrative formula only. Callers must pass logged counts, never a hoped-for gap.
    if paired_tasks <= 0:
        return "reject empty_slice"
    control_rate = control_passes / paired_tasks
    arm_rate = arm_passes / paired_tasks
    return "delta={:.4f} n={}".format(arm_rate - control_rate, paired_tasks)

def main():
    report = json.loads(Path(sys.argv[1]).read_text())
    message, code = evaluate(report)
    print(message)
    return code

if __name__ == "__main__":
    raise SystemExit(main())

# Unexecuted proposal. Swap in real paths only after the slice file is committed.
sha256sum prompts/task-014.txt | tee leakage/task-014.sha256
git diff --exit-code -- card.json check_benchmark_card.py
python3 check_benchmark_card.py card.json
python3 -c 'from check_benchmark_card import paired_delta; print(paired_delta(0, 0, 0))'
python3 - <<'PY'
import check_benchmark_card
message, code = check_benchmark_card.evaluate({"dataset_id": "missing-the-rest"})
print(message, code)
PY

The command sequence hashes the card, runs the checker, and refuses to print a rank when a required field is empty. A passing hash is only a fingerprint of the claim, not a certificate that the underlying tests were fair or complete. Operators then archive the raw logs under the same identifier so a later audit can recompute the metric without asking the author. If the hash changes after publication, the old number is retired rather than quietly replaced, because the claim changed with the file.

MonkeyCode can fill two labeled slots on this card, one as a free model arm and one as a free server runner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Operator notes for this draft state that those two options exist, without naming a quota, a machine shape, or a duration. A real card must copy the model identifier, the server image, and the access terms from documentation on the run date. If the product page disagrees, that page wins, and nothing here claims the offer is permanent or exclusive.

A shared server image can pin the runner when laptops differ in compilers, clocks, and background jobs that nudge timeouts. A no-cost model arm can answer whether a free path is adequate for one pre-registered slice, not whether a vendor deserves a crown. Neither slot replaces the dataset lock, the metric formula, or the reference arm, because availability is not a quality result. The checker still passes or fails the same way if those slots are filled by another vendor or by a local process.

What the figure is not

This approach does not prove that a task set is uncontaminated, because a cutoff date is a note rather than a full training-data audit. It also does not measure statistical power, so a tiny slice can still produce a noisy gap that looks decisive in a table. Human judges, if used at all, need a separate calibration record, and this card does not replace that record with a boolean field. Security-sensitive tasks, production incidents, and customer repositories do not belong on a casual shared runner without a reviewed data policy.

Teams that need a legal benchmark for procurement should not treat this checker as a certification body or as a substitute for counsel. Researchers who already run a registered trial with a published protocol may find the card redundant if their protocol already freezes the same fields. Beginners who want a weekend demo of a chatbot should skip the machinery, because a toy prompt does not need a leaderboard claim. Anyone hoping to publish a winner after peeking at scores should stop, since the method is designed to make that story fail review.

The practical next step is to commit a card for one small slice and run the checker first. Current access terms for any free model path or free server should be read from the vendor documentation on the run day. A team that already archives harness hashes can add the non-marketing clause this week and leave the rest of its pipeline unchanged. That single addition is enough to stop a bare percentage from traveling as if it were an advertisement with extra digits.