DEV Community: Avery Wang

A Cheap Run Needs a Fixed Holdout Pack

Avery Wang — Sun, 11 Oct 2026 22:07:17 +0000

A cheap runner can exercise a coding agent, yet that exercise does not become a capability ranking by itself. The honest product of the run is a labeled observation that names the dataset, the controls, and the clock. Readers should treat any single score from an unpaid lane as a lane note rather than a market claim. A publishable comparison starts only after those boundaries sit in a card that another engineer can replay.

Coding-agent notes often arrive with a polished percentage and almost no account of the task pack behind it. That pattern resembles a race result printed without the track length, the wind, or the class of car. A developer comparing tools then confuses access to a runner with proof that the agent solves a held-out class of work. The confusion grows when a free route lowers the cost of trying ideas while leaving the measurement problem untouched.

MonkeyCode can host that rehearsal when the operator uses its free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those availability claims describe access only, and they do not state quotas, hardware, duration, or permanent terms. A card that hard-codes a token grant or a server size would invent facts the operator must read from current documentation at trial time.

The pack is the stick

The dataset is a small task pack whose manifest, content hash, and holdout split stay fixed after the first scored run. Each item carries a stable identifier, a language tag, a fixture directory, and a pass rule that a script can apply. Training-like prompts, leaked solutions, and tasks copied from a vendor demo stay in an excluded bucket outside the scored set. The pack works like a measuring stick, and sanding that stick after the number appears turns the trial into a story.

Useful metrics should describe completion, repair cost, and instability rather than collapsing the trial into one vanity percentage. Completion is the fraction of holdout items whose tests pass when the frozen command runs without manual edits. Repair cost is the median tool-call count, kept apart from median wall time so a fast failure is not a cheap success. Instability is the mean pass spread across repeats of one holdout item, so one lucky seed cannot stand for the lane.

Controls exist so the free route is not quietly compared with a different class of runner or clock. The same task hash, the same timeout, the same retry ceiling, and the same network policy should bind every arm. A free server may queue, throttle, or rotate capacity, so the card stores the start time and a lane label. If the route is selected dynamically, the log captures the route name returned at runtime rather than a name guessed in the draft.

A harness that refuses a rank

The workflow below is a proposed harness, not a result from an executed study and not a claim about any live quota. An author drops fixtures into a directory, writes a manifest, and lets the script refuse a score when required controls are missing. That refusal is the point, because a missing control is more informative than a confident table built on an unlocked pack. The same script can later sit beside a paid runner, which keeps the comparison about the card rather than about a brochure.

#!/usr/bin/env python3
"""Proposed, unexecuted lane card. It records controls and refuses a rank."""

from __future__ import annotations

import hashlib
import json
import statistics
import time
from pathlib import Path

REQUIRED = ("task_hash", "lane_label", "timeout_s", "retry_ceiling", "network_policy")

def hash_pack(root: Path) -> str:
    digest = hashlib.sha256()
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        digest.update(path.relative_to(root).as_posix().encode())
        digest.update(path.read_bytes())
    return digest.hexdigest()

def summarize(trials: list[dict]) -> dict:
    holdout = [item for item in trials if item.get("split") == "holdout"]
    by_item: dict[str, list[int]] = {}
    calls, walls = [], []
    for item in holdout:
        by_item.setdefault(str(item.get("id", "unknown")), []).append(
            1 if item.get("passed") else 0
        )
        if "tool_calls" in item:
            calls.append(item["tool_calls"])
        if "wall_ms" in item:
            walls.append(item["wall_ms"])
    spreads = [max(flags) - min(flags) for flags in by_item.values() if flags]
    passed = sum(flags[-1] for flags in by_item.values()) if by_item else 0
    return {
        "holdout_items": len(by_item),
        "completion": (passed / len(by_item)) if by_item else None,
        "median_tool_calls": statistics.median(calls) if calls else None,
        "median_wall_ms": statistics.median(walls) if walls else None,
        "mean_item_pass_spread": statistics.mean(spreads) if spreads else None,
    }

def build_card(root: Path, trials: list[dict], controls: dict) -> dict:
    missing = [key for key in REQUIRED if not controls.get(key)]
    if missing:
        raise SystemExit(f"refusing score; missing controls: {', '.join(missing)}")
    if controls["task_hash"] != hash_pack(root):
        raise SystemExit("refusing score; task pack hash drifted")
    return {
        "generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "task_hash": controls["task_hash"],
        "lane_label": controls["lane_label"],
        "controls": controls,
        "metrics": summarize(trials),
        "not_marketing": [
            "not a general coding-ability claim",
            "not a price or quota claim",
            "not valid after the pack or route changes",
        ],
        "rank_emitted": False,
    }

if __name__ == "__main__":
    root = Path("task_pack")
    controls = json.loads(Path("controls.json").read_text())
    trials = json.loads(Path("trials.json").read_text())
    card = build_card(root, trials, controls)
    Path("lane_card.json").write_text(json.dumps(card, indent=2) + "\n")
    print(json.dumps({"status": "card_written", "rank_emitted": False}))

A companion control file keeps the route explicit, and the example values below are placeholders rather than observed product limits. The timeout, the retry ceiling, and the network policy are choices of the study, not features inferred from a vendor page. The lane label should quote the access class that the operator confirmed on the day of the run. Leaving those fields blank is preferable to filling them with a remembered grant from an older post.

{
  "task_hash": "<sha256 of task_pack>",
  "lane_label": "free-model-access/free-server; terms read at runtime",
  "timeout_s": 120,
  "retry_ceiling": 1,
  "network_policy": "no egress beyond the fixture mirror"
}

The commands that follow are a local rehearsal sketch, and they do not contact a model or reserve a server. An author can create the pack, compute nothing remote, and confirm that a missing label stops the score. That dry run is useful because it teaches the refusal path before any token is spent on a live route. A later live run should append the route name returned by the server, then regenerate the card without editing the holdout fixtures.

mkdir -p task_pack/holdout/item-01
printf '%s\n' 'id: item-01' 'split: holdout' > task_pack/holdout/item-01/manifest.txt
printf '%s\n' '{"timeout_s":120,"retry_ceiling":1,"network_policy":"fixture-mirror-only"}' > controls.json
printf '%s\n' '[]' > trials.json
python3 lane_card.py
# expected: refusing score; missing controls: task_hash, lane_label

A written card that survives the checks still carries rank_emitted set to false, which blocks a casual sort of vendors. The metrics object may show a completion fraction, two cost medians, and a pass spread, each tied to the stored hash. A reader who cannot find the negative claims should treat the file as incomplete even if the fraction looks strong. That rule is the difference between a lab note and a sentence that could be pasted into an advertisement.

What the numbers are not allowed to say

These numbers are not marketing because they answer a narrow question under named constraints, and they expire when those constraints change. A higher completion rate on a small pack does not prove general coding ability, a lower price, or a permanent free grant. The card should state the negative claims in plain language, including what was not measured and which arms were absent. Publishing the hash next to the table lets a later reader see whether the pack moved after the headline was written.

Failure analysis belongs in the same narrative as the score, because a green fraction without its misses invites overreading. A timeout cluster may say more about queue behavior on a free server than about the agent's repair skill. A pass that appears only after a second retry tells a different story from a pass on the first frozen attempt. Recording those patterns beside the median keeps the result closer to a lab note than to a launch sentence.

Synthetic trials can teach the reader how to hold that line before a live route is involved at all. Suppose two repeats of one holdout item pass on the first seed and fail on the second, while wall time jumps from a quiet run to a queued run. The completion field may still look acceptable if only the last flag is counted, but the pass spread exposes the instability. A reviewer who sees that spread should keep the lane label attached and decline to promote the fraction into a ranking table.

Who should leave this method alone

This approach should not be used to certify a model for production, to claim a security review, or to rank vendors for a purchase committee. It also fails when the task pack is private, when tests depend on live paid APIs, or when the author edits fixtures after seeing failures. A free server option can disappear, slow down, or change routing, so a result from one afternoon is a sample of that afternoon. Teams that need a contractual service level should repeat the same card on an explicit environment before they trust the delta.

An engineer rehearsing this card can check MonkeyCode's current free model access and free server option before sharing a score. The live terms should be copied into the lane label before any score is shared with another reader. The useful habit is the refusal to rank an unlocked pack, and that habit remains valuable even if the hosting choice changes. A later reader should be able to reject the number without guessing which track, which wind, or which class of car produced it.

Free-Tier Agent Runs Are Control Lanes

Avery Wang — Sat, 10 Oct 2026 18:49:34 +0000

A convenient agent run on a free model route is a control lane, not a publishable result. The figure becomes interpretable only when canary tasks pass, negative controls remain failed, and the grader never sees the route label. Without those three conditions, a high pass rate mostly records fluency, harness luck, or a judge that rewards confident prose. Teams that skip the gate often circulate the decimal first and discover the broken fixture only after the write-up is public.

The pattern appears whenever a cheaper model seems to match a stronger one on a handful of visible tasks. A developer watches one clean transcript, reruns the same prompt, and treats the second success as confirmation. That habit confuses a demonstration with a measurement, because the task, the tools, and the judge all moved together. A benchmark, by contrast, fixes the question set and then checks whether the answer still holds after the easy leaks are removed.

Three families, not one trophy slice

The working dataset should contain three families that travel together, rather than a single leaderboard slice. Scored tasks represent the work the team actually claims to measure, with hidden checks stored outside the agent workspace. Canary tasks are small, fully specified, and expected to pass when the harness, the tools, and the instruction format are alive. Negative-control tasks look plausible but require a missing file, a contradictory oracle, or a forbidden sandbox action, so a careful agent abstains.

The metrics worth publishing are paired rates with explicit denominators, not a single trophy percentage from one run. Report the scored pass rate, the canary pass rate, the negative-control violation rate, and the abstention rate on control tasks. A violation means the agent edited a forbidden path, invented a missing file, or received a pass despite an impossible oracle. If the canary pass rate drops, the harness is unhealthy, and the scored rate should remain inside the lab notebook.

Controls exist so a later reader can reproduce the conditions, rather than merely repeating the press sentence. Freeze the task manifest, keep the sandbox image digest in the run log, and hold the retry budget identical across lanes. Blind the grader to model identity and to the route class, because a visible label can leak prestige into the score. Log temperature, tool permissions, and the route identifier as covariates, and refuse any comparison when one of those fields is missing.

These numbers are not marketing copy, because the gate is explicitly allowed to withhold the headline figure. A free-route lane that scores well on visible tasks while violating negative controls has measured eagerness to comply, not capability. A lane that fails its canaries has not measured the model at all, no matter how attractive the scored subset looks. Publishing only the scored rate, after quietly discarding the control families, is how a lab notebook becomes an advertisement.

A gate that can stay silent

The following Python gate is a proposal for the lab notebook, not a record of an executed leaderboard. It reads a manifest of outcomes that some other runner already wrote, then decides whether a headline rate may be emitted. The thresholds below are starting points for discussion, not validated cutoffs borrowed from a public study. Operators should replace those cutoffs after they see how often their own harness fails canaries for mundane reasons.

#!/usr/bin/env python3
"""Proposal only: gate a headline agent rate on canaries and negative controls.

This script does not call a model and does not claim a measured score.
Feed it a JSON manifest produced by your own runner.
"""

from __future__ import annotations

import hashlib
import json
import sys


def manifest_digest(payload: dict) -> str:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(body).hexdigest()


def count_outcome(rows: list[dict], family: str, outcome: str) -> tuple[int, int]:
    chosen = [row for row in rows if row.get("family") == family]
    hits = sum(1 for row in chosen if row.get("outcome") == outcome)
    return hits, len(chosen)


def main() -> int:
    payload = json.load(sys.stdin)
    required = {"route_id", "image_digest", "grader_blinded", "tasks"}
    if required - payload.keys() or not payload.get("grader_blinded"):
        print(json.dumps({"emit_headline": False, "reason": "controls_incomplete"}))
        return 2

    tasks = payload["tasks"]
    canary_hits, canary_n = count_outcome(tasks, "canary", "pass")
    scored_hits, scored_n = count_outcome(tasks, "scored", "pass")
    violations, control_n = count_outcome(tasks, "negative_control", "pass")
    abstentions, _ = count_outcome(tasks, "negative_control", "abstain")

    canary_ok = canary_n > 0 and (canary_hits / canary_n) >= 0.95
    controls_ok = control_n > 0 and (violations / control_n) <= 0.05
    emit = bool(canary_ok and controls_ok and scored_n > 0)

    report = {
        "manifest_sha256": manifest_digest(payload),
        "route_id": payload["route_id"],
        "image_digest": payload["image_digest"],
        "canary_pass": [canary_hits, canary_n],
        "scored_pass": [scored_hits, scored_n],
        "negative_control_violations": [violations, control_n],
        "negative_control_abstentions": abstentions,
        "emit_headline": emit,
    }
    print(json.dumps(report, indent=2))
    return 0 if emit else 3


if __name__ == "__main__":
    raise SystemExit(main())

A local check should stay deliberately boring, which is a real virtue inside a careful measurement pipeline. The command below reads a manifest from standard input and prints the gate decision without contacting a model endpoint. A non-zero status means the headline stays unpublished, even when the scored subset looks flattering in the file. The sample manifest is a synthetic fixture for the gate, not an observed agent result and not a ranking of any host.

python3 gate_agent_rate.py < run_manifest.json
echo "gate_status=$?"

{
  "route_id": "free-route-logged-not-ranked",
  "image_digest": "sha256:example-not-a-real-image",
  "grader_blinded": true,
  "tasks": [
    {"id": "canary-01", "family": "canary", "outcome": "pass"},
    {"id": "neg-01", "family": "negative_control", "outcome": "abstain"},
    {"id": "task-01", "family": "scored", "outcome": "pass"}
  ]
}

A manifest that omits the route identifier, the image digest, or the blinded-grader flag exits before any rate is printed. That early exit is the point of the artifact, because a missing covariate is already a failed comparison. The script treats a pass on a negative-control task as a violation, which feels harsh where partial credit is common. That harshness is preferable to a public chart that cannot explain its own denominator at all.

Where a free route fits

MonkeyCode's free model access and free server option can host the control lane, so a comparison budget is not spent discovering a broken sandbox. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those availability claims belong in the log as a route, not as a promise that the pool will stay fixed. Readers should confirm current terms in the project documentation, and they should keep private repositories and secrets off any shared host.

A route identifier beside the manifest digest shows that the figure belonged to that pool on that day. Nothing in this method requires a particular vendor, and a free lane is not a better or worse model class. The free option matters only because canaries and negative controls are cheap to repeat when a fixture changes, if the route id stays logged. If the documentation no longer offers that access, the same gate still runs on any host that can emit the manifest.

Who should leave this gate alone

This gate does not fit every evaluation, and several limits should stay visible beside the script. Canaries that are too easy can pass while a different bug still corrupts the scored family, so the canary set needs occasional replacement. Negative-control wording, once published in full, can leak into later training data and stop functioning as a control. The shared manifest should store task identifiers and hashes, rather than copying the private oracle text itself.

Automated judges can share blind spots with the agent, so a small human review of violations remains necessary before a rate is quoted externally. Regulated attestations, customer-facing rankings, and workloads that cannot leave a private network should not use a free shared lane for this gate. A single clean transcript is also the wrong input, because the script has nothing honest to say about a denominator of one. Teams that already keep denominators can add this gate, then consult MonkeyCode's current access notes only if a disposable canary host is still needed.

Stratified Trial Budgets for Publishable Agent Scores

Avery Wang — Fri, 09 Oct 2026 15:12:18 +0000

An agent score becomes evidence only after the trial budget, the difficulty strata, and the exclusion rules are frozen. A single clean transcript can still guide a repair, yet one lucky path does not estimate error. The method below treats the headline percentage as a summary of paired trials, not as a brochure claim. A reviewer can reproduce the accounting with a small Python harness before any model name enters the discussion.

A clean run is not a timetable

A clean local run often looks decisive, especially when every visible check happens to pass on the first afternoon. That result can guide the next edit, yet it remains a weather report until the same items repeat under fixed rules. A transit agency does not publish a timetable from one sunny commute, and a lab should not publish a rank from one polished run. The analogy is deliberate, because variance, route difficulty, and cancelled trips all belong in the published record.

What the pack must freeze

The dataset is a frozen task pack split into difficulty strata before any candidate is allowed to run. Each stratum holds tasks of similar expected effort, such as a one-file fix, a multi-file refactor, or a failing integration test. Items carry stable identifiers, and the pack records the stratum label beside the prompt path and the oracle command. A later chart may omit a stratum, but the scorer may not silently move an item into an easier bin.

Without strata, a model that excels at short edits can outscore a model that survives harder repairs, and the average hides the swap. Promotional copy prefers that average, because a single percentage travels farther than a table of uneven wins. The methodology therefore reports a within-stratum paired difference, and only then a predeclared summary across strata. If the summary and the strata disagree, the strata remain the result and the summary becomes a footnote.

Metrics that refuse a slogan

The primary metric is the paired outcome on the same item, not an unpaired accuracy computed on different samples. Each trial records pass or fail against the oracle, the wall time, and whether the harness aborted for a logged reason. A secondary metric is the stratum win rate, defined as the share of items where the candidate beats the control. Neither metric is a quality slogan, because both inherit every exclusion that the log has already accepted.

Controls exist to stop the number from drifting into advertising copy after the run looks favorable. The control arm uses the same harness, the same oracle, and the same retry ceiling as the candidate arm. A placebo item with an impossible oracle must fail, or the scorer treats the whole batch as untrustworthy. That negative control is the fire alarm, not an optional appendix for readers who enjoy methodology notes.

The trial budget is chosen before scores appear, and it states how many independent repeats each item receives. Two repeats can reveal a flip, while a larger budget is required before a small gap deserves a public rank. This draft does not prescribe a universal sample size, because task variance differs across repositories and oracles. It does require the chosen budget to be written into the manifest, so a disappointing repeat cannot be deleted quietly.

Exclusions are the place where an honest lab and a promotional writeup usually diverge in practice. A pre-registered rule may drop an item for a missing fixture, a leaked solution, or an oracle that no longer builds. A post-hoc rule that drops an item because the candidate failed is not an exclusion, and the harness should refuse it. The log stores the rule identifier, the item identifier, and the timestamp, so a reviewer can replay the decision.

A harness that enforces the freeze

The following Python module is a proposed harness, and it has not been executed against a live scoring service. It hashes the manifest, checks paired outcomes, and rejects any exclusion that lacks a pre-registered rule. Operators should treat the script as a starting ledger, then replace the toy fields with their own oracle records. The point is the accounting, not a claim that any particular model already passed these checks.

#!/usr/bin/env python3
"""Proposed trial-budget harness.

This example has not been executed against a live scoring service.
Replace the fixture files before using it on a real task pack.
"""

from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

RULES = {
    "missing_fixture": "Oracle inputs were absent before the candidate started.",
    "leaked_solution": "A reference patch was visible in the prompt context.",
    "oracle_broken": "The oracle command failed on the untouched baseline.",
}


def manifest_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_manifest(path: Path) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    required = {"pack_id", "trial_budget", "strata", "items", "exclusion_rules"}
    missing = sorted(required.difference(data))
    if missing:
        raise SystemExit(f"manifest missing fields: {missing}")
    budget = data["trial_budget"]
    if not isinstance(budget, int) or budget < 1:
        raise SystemExit("trial_budget must be a frozen integer of at least 1")
    return data


def accept_exclusion(rule_id: str, registered: list[str]) -> None:
    if rule_id not in registered or rule_id not in RULES:
        raise SystemExit(f"post-hoc exclusion refused: {rule_id}")


def paired_delta(candidate: list[bool], control: list[bool]) -> dict:
    if len(candidate) != len(control) or not candidate:
        raise SystemExit("paired arms require equal non-empty outcomes")
    wins = losses = ties = 0
    for cand, base in zip(candidate, control):
        if cand == base:
            ties += 1
        elif cand and not base:
            wins += 1
        else:
            losses += 1
    return {"wins": wins, "losses": losses, "ties": ties, "n": len(candidate)}


def self_check() -> None:
    delta = paired_delta([True, False], [False, False])
    expected = {"wins": 1, "losses": 0, "ties": 1, "n": 2}
    if delta != expected:
        raise SystemExit("paired_delta self-check failed")
    try:
        accept_exclusion("candidate_failed", ["missing_fixture"])
    except SystemExit:
        print("self-check ok")
        return
    raise SystemExit("self-check failed to refuse a post-hoc exclusion")


def evaluate(manifest_path: Path, outcomes_path: Path) -> dict:
    manifest = load_manifest(manifest_path)
    digest = manifest_hash(manifest_path)
    outcomes = json.loads(outcomes_path.read_text(encoding="utf-8"))
    if outcomes.get("manifest_sha256") != digest:
        raise SystemExit("outcome file does not match frozen manifest hash")
    for event in outcomes.get("exclusions", []):
        accept_exclusion(event["rule_id"], manifest["exclusion_rules"])
    by_stratum: dict[str, list[dict]] = {}
    for item in manifest["items"]:
        arm = outcomes["items"][item["id"]]
        if len(arm["candidate"]) != manifest["trial_budget"]:
            raise SystemExit(f"budget mismatch: {item['id']}")
        if len(arm["control"]) != manifest["trial_budget"]:
            raise SystemExit(f"control budget mismatch: {item['id']}")
        if item.get("placebo") and any(arm["candidate"]):
            raise SystemExit(f"placebo passed: {item['id']}")
        by_stratum.setdefault(item["stratum"], []).append(
            paired_delta(arm["candidate"], arm["control"])
        )
    return {
        "manifest_sha256": digest,
        "pack_id": manifest["pack_id"],
        "strata": by_stratum,
    }


def main() -> None:
    if len(sys.argv) == 2 and sys.argv[1] == "self-check":
        self_check()
        return
    if len(sys.argv) != 3:
        raise SystemExit(
            "usage: trial_budget.py self-check | trial_budget.py PACK OUTCOMES"
        )
    report = evaluate(Path(sys.argv[1]), Path(sys.argv[2]))
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

A reviewer can freeze the pack with a checksum before the first candidate process is allowed to start. The commands below assume a local checkout, a Python 3 interpreter, and no network call inside the toy oracle. They illustrate the control path, and they do not measure a hosted product or imply a hardware specification. If a command fails, the failure stays in the log rather than disappearing into a revised average.

python3 trial_budget.py self-check
python3 -c "import hashlib, pathlib; print(hashlib.sha256(pathlib.Path('task_pack.json').read_bytes()).hexdigest())"
python3 trial_budget.py task_pack.json outcomes.json

The sample manifest freezes a two-stratum pack, a trial budget of two, and the three exclusion rules the harness recognizes. The sample outcomes file must carry the same manifest hash, or the script exits before it prints a table. A placebo item that passes on the candidate arm aborts the batch, which is the intended failure for a broken scorer. Those files are fixtures for the method, and they are not measurements of any hosted coding agent.

{
  "pack_id": "strata-demo-001",
  "trial_budget": 2,
  "strata": ["short_edit", "multi_file"],
  "exclusion_rules": ["missing_fixture", "leaked_solution", "oracle_broken"],
  "items": [
    {
      "id": "short-01",
      "stratum": "short_edit",
      "prompt_path": "prompts/short-01.md",
      "oracle": "pytest -q tests/test_short_01.py",
      "placebo": false
    },
    {
      "id": "placebo-oracle",
      "stratum": "short_edit",
      "prompt_path": "prompts/placebo.md",
      "oracle": "false",
      "placebo": true
    }
  ]
}

{
  "manifest_sha256": "<paste sha256 of task_pack.json>",
  "exclusions": [],
  "items": {
    "short-01": {"candidate": [true, false], "control": [false, false]},
    "placebo-oracle": {"candidate": [false, false], "control": [false, false]}
  }
}

Where a hosted option fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option matter only as the candidate runtime, while the ledger stays local. That availability is only an operator-supplied convenience, and this draft does not verify quotas, hardware, duration, or model names. A team may copy the limits it actually sees into the run notes, without treating free capacity as a quality score.

Why the percentage is not marketing

A percentage becomes marketing when the audience cannot see the pack, the budget, the exclusions, or the negative control. The same percentage becomes a lab result when those four artifacts travel with the figure and a reviewer can recompute it. Publishing the strata beside the summary also blocks the common slide that hides a loss on hard items. If a stakeholder wants only the largest number, the writeup should withhold the rank rather than inflate the claim.

Limits, and who should walk away

This approach does not detect training-set contamination beyond the canary items the pack author remembered to include. It does not price the runs, and a cheaper failure can still be the better engineering choice for a given team. Wall-clock comparisons remain fragile when a free runner and a paid runner differ in queueing, which this toy script cannot see. Small strata will swing wildly, so a two-item bin should be reported as a case series rather than as a rate.

Teams that need a vendor ranking for a launch post this week should not use this harness as decoration. The method also fits poorly when the oracle itself is disputed, because a shared wrong test will crown the wrong patch. Researchers who cannot freeze the task pack, or who must tune prompts after seeing scores, will violate the exclusion rule immediately. In those settings the honest output is a demo transcript with a date, not a benchmark table.

The practical next step is to hash a small pack, run the placebo item, and confirm that a bad exclusion is rejected. Only after that check should a candidate arm consume either a local process or a hosted free option. The resulting table will look quieter than a leaderboard, which is the intended trade for a number that can survive review. Quiet evidence is still the result a later reader can trust when the demo transcript has already scrolled out of view.

Freeze the Benchmark Card Before an Agent Score Ships

Avery Wang — Thu, 08 Oct 2026 10:41:03 +0000

A coding-agent score is evidence only when the dataset, the metric, and the control surface are frozen together. Without that triple lock, a leaderboard cell behaves like a slogan that happens to wear a decimal point. Readers cannot tell whether the number measures the agent, the harness, or the afternoon the run happened to finish. The method below treats the published figure as a claim that must survive inspection, not as a marketing line.

Many public comparisons still ship a single percentage and a model nickname, then invite readers to treat the gap as proof. That pattern fails because the task mix, the grader, and the machine can each move the result by more than the claimed gap. A benchmark card records those moving parts before the number is allowed to travel outside the lab. The card remains a method document, and it stays useful even if every vendor name is stripped from the page.

Dataset construction starts with named families, fixed slice sizes, and a written rule for what never enters the prompt. A family might cover bug repair, test writing, or a narrow refactor, yet each family needs both an inclusion rule and an exclusion rule. Contamination is the quiet failure, because a model can look skilled when the task text already lived in its earlier training mix. The card therefore stores a source note, a cutoff date, and a leakage-check command instead of a vague promise that the set is fresh.

The metric section defines pass, partial credit, and failure in formulas a second team can recompute from raw logs. A pass might require tests to exit zero and the diff to stay inside a stated line budget. Partial credit needs its own formula, or later readers will invent a generous reading after they see which arm looked weak. The card also states what the metric ignores, including security review, latency under load, and long-horizon design judgment.

Controls exist so a delta can be attributed to the agent rather than to weather in the network or drift in the laptop. A paired design runs the same task identifier, the same seed policy, and the same timeout on every arm before any ranking starts. An unchanged reference arm, often a fixed script or a frozen model build, shows whether the harness itself moved between weeks. If that reference arm moves, new scores stay quarantined until the drift is explained, because the comparison is no longer clean.

Numbers become marketing when the writer selects the slice after seeing the results, then publishes only the flattering cut. The card blocks that move by requiring the slice plan, the exclusion log, and the analysis script hash to be committed before the run. A pre-registered comparison resembles a lab notebook more than a launch post, and that resemblance is the entire point of the format. Readers should be able to reject the claim without trusting the author's tone, which is why the required fields stay mechanical.

A checker that refuses a bare number

The artifact below is a proposal, not a result from a finished evaluation, and no figure inside it should be quoted as measured. It rejects a report that omits the dataset, the metric, the controls, or an explicit clause saying the number is not an advertisement. Teams can place the checker in continuous integration so a missing field fails the build instead of failing a later reader. The sample uses synthetic task identifiers so the workflow can be inspected without implying that any vendor won a contest.

{
  "dataset_id": "synth-repair-slice-locked",
  "slice_plan": "40 synthetic repair tasks, file list committed before any arm runs",
  "slice_locked_before_run": true,
  "source_note": "hand-written fixtures, not scraped from a public leaderboard",
  "cutoff_date": "record-the-real-cutoff-on-execution-day",
  "metric_id": "paired_pass_rate",
  "metric_formula": "passes / paired_tasks; partial credit is a separate field and is not folded in",
  "metric_ignores": "security review, load latency, long-horizon design judgment",
  "control_arm": "frozen_reference_script_v3",
  "seed_policy": "seed=17 on every arm",
  "timeout_seconds": 120,
  "exclusion_log": "exclusions/pre-run.txt",
  "non_marketing_clause": "This figure is a pre-registered measurement, not an advertisement or a ranking claim.",
  "run_date": "set-on-execution-day",
  "runner_image": "copy-from-current-docs",
  "model_arm": "copy-identifier-on-run-date",
  "score_omitted": "no measured value belongs in this proposal"
}

#!/usr/bin/env python3
"""Unexecuted proposal. Rejects a card that could travel as an ad."""

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = (
    "dataset_id",
    "slice_plan",
    "metric_id",
    "metric_formula",
    "control_arm",
    "seed_policy",
    "timeout_seconds",
    "exclusion_log",
    "non_marketing_clause",
    "run_date",
)

def canonical_hash(payload):
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()

def evaluate(report):
    missing = [key for key in REQUIRED if report.get(key) in (None, "", [])]
    if missing:
        return "reject missing=" + ",".join(missing), 2
    clause = str(report["non_marketing_clause"]).lower()
    if "not an advertisement" not in clause:
        return "reject clause", 3
    if report.get("slice_locked_before_run") is not True:
        return "reject slice_not_locked", 4
    return canonical_hash(report), 0

def paired_delta(control_passes, arm_passes, paired_tasks):
    # Illustrative formula only. Callers must pass logged counts, never a hoped-for gap.
    if paired_tasks <= 0:
        return "reject empty_slice"
    control_rate = control_passes / paired_tasks
    arm_rate = arm_passes / paired_tasks
    return "delta={:.4f} n={}".format(arm_rate - control_rate, paired_tasks)

def main():
    report = json.loads(Path(sys.argv[1]).read_text())
    message, code = evaluate(report)
    print(message)
    return code

if __name__ == "__main__":
    raise SystemExit(main())

# Unexecuted proposal. Swap in real paths only after the slice file is committed.
sha256sum prompts/task-014.txt | tee leakage/task-014.sha256
git diff --exit-code -- card.json check_benchmark_card.py
python3 check_benchmark_card.py card.json
python3 -c 'from check_benchmark_card import paired_delta; print(paired_delta(0, 0, 0))'
python3 - <<'PY'
import check_benchmark_card
message, code = check_benchmark_card.evaluate({"dataset_id": "missing-the-rest"})
print(message, code)
PY

The command sequence hashes the card, runs the checker, and refuses to print a rank when a required field is empty. A passing hash is only a fingerprint of the claim, not a certificate that the underlying tests were fair or complete. Operators then archive the raw logs under the same identifier so a later audit can recompute the metric without asking the author. If the hash changes after publication, the old number is retired rather than quietly replaced, because the claim changed with the file.

MonkeyCode can fill two labeled slots on this card, one as a free model arm and one as a free server runner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Operator notes for this draft state that those two options exist, without naming a quota, a machine shape, or a duration. A real card must copy the model identifier, the server image, and the access terms from documentation on the run date. If the product page disagrees, that page wins, and nothing here claims the offer is permanent or exclusive.

A shared server image can pin the runner when laptops differ in compilers, clocks, and background jobs that nudge timeouts. A no-cost model arm can answer whether a free path is adequate for one pre-registered slice, not whether a vendor deserves a crown. Neither slot replaces the dataset lock, the metric formula, or the reference arm, because availability is not a quality result. The checker still passes or fails the same way if those slots are filled by another vendor or by a local process.

What the figure is not

This approach does not prove that a task set is uncontaminated, because a cutoff date is a note rather than a full training-data audit. It also does not measure statistical power, so a tiny slice can still produce a noisy gap that looks decisive in a table. Human judges, if used at all, need a separate calibration record, and this card does not replace that record with a boolean field. Security-sensitive tasks, production incidents, and customer repositories do not belong on a casual shared runner without a reviewed data policy.

Teams that need a legal benchmark for procurement should not treat this checker as a certification body or as a substitute for counsel. Researchers who already run a registered trial with a published protocol may find the card redundant if their protocol already freezes the same fields. Beginners who want a weekend demo of a chatbot should skip the machinery, because a toy prompt does not need a leaderboard claim. Anyone hoping to publish a winner after peeking at scores should stop, since the method is designed to make that story fail review.

The practical next step is to commit a card for one small slice and run the checker first. Current access terms for any free model path or free server should be read from the vendor documentation on the run day. A team that already archives harness hashes can add the non-marketing clause this week and leave the rest of its pipeline unchanged. That single addition is enough to stop a bare percentage from traveling as if it were an advertisement with extra digits.

A Benchmark Card Makes an Agent Score Auditable

Avery Wang — Thu, 08 Oct 2026 08:27:25 +0000

A published coding-agent score describes one frozen task slice, and it does not describe an entire product. That slice needs a named dataset, an explicit metric contract, and controls a later reader can rerun. Without those three artifacts beside the number, the percentage behaves like a poster rather than evidence. Careful readers then import a marketing reading that the measurement itself never earned and cannot support.

Recent public notes on agent rankings often print one resolved rate and then stop the explanation entirely. The missing piece is not a longer adjective list, but the identity of the tasks behind that rate. A rate computed on easy stubs will flatter a system that fails on multi-file repairs with hidden tests. That gap is why a benchmark card belongs in the same commit as the score, rather than in a later aside.

What the card records

The dataset section names the source repository, the pinned commit, and the inclusion rule used during sampling. Tasks enter the slice only when they carry a failing test, a bounded edit surface, and a license that permits prompt redistribution. Stratification by language and by estimated edit span keeps a Python-heavy week from masquerading as a general coding result. Each retained task also stores the commit that produced the failing test, so a later rerun can detect an upstream rewrite.

The metric contract defines the numerator, the denominator, and the attempt budget before any model is called. Resolved rate counts a task only when hidden tests pass within that budget, or when a passing patch arrives earlier. Pass-at-k stays in a separate column, because a lucky later attempt is not the same event as a first repair. Duration and token totals can sit beside quality, yet they do not replace the contract that says what resolved means.

Controls exist so the number can fail, which is the property that separates a benchmark from a brochure. A no-edit control submits the untouched repository and must stay near zero on tasks that truly require a patch. A known-fix control applies a recorded one-line repair and must pass, proving the harness actually executes the hidden tests. If either control misbehaves, the run is discarded rather than averaged into a leaderboard that nobody can later audit.

Marketing copy selects the friendliest slice after the fact and then narrates the winner as if that slice had been fixed. A benchmark card reverses the order by recording the slice first and by refusing publication when the recorded slice no longer matches. The match check does not prove that a model is strong, and it only proves the claim still points at the same tasks. Readers can then dispute the metric without guessing which repositories were quietly dropped from the denominator.

Freeze the slice locally

The module below is a proposal for that card, and it has not been executed against a live agent fleet. It writes a JSON manifest, checks stratum floors, and exits non-zero when the slice no longer matches the recorded digest. Operators may rename fields, but they should not delete the refusal path that blocks a drifted publish. The example uses a local JSON file so the method can be reviewed without a network call or a vendor SDK.

#!/usr/bin/env python3
"""Proposal: freeze a benchmark card. Not executed against a live agent fleet."""

from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = (
    "task_id",
    "repo",
    "commit",
    "language",
    "edit_span",
    "license",
    "has_failing_test",
)
ALLOWED_LICENSES = {"mit", "apache-2.0", "bsd-3-clause"}
MIN_PER_STRATUM = 5
ATTEMPT_BUDGET = 1


def load_tasks(path: Path) -> list[dict]:
    tasks = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(tasks, list) or not tasks:
        raise SystemExit("task file must be a non-empty list")
    for task in tasks:
        missing = [field for field in REQUIRED if field not in task]
        if missing:
            raise SystemExit(f"{task.get('task_id', '?')} missing {missing}")
        if task["has_failing_test"] is not True:
            raise SystemExit(f"{task['task_id']} lacks a failing test")
        if task["license"] not in ALLOWED_LICENSES:
            raise SystemExit(f"{task['task_id']} license is outside the inclusion rule")
        if not isinstance(task["edit_span"], int) or task["edit_span"] < 1:
            raise SystemExit(f"{task['task_id']} has a non-positive edit span")
    return tasks


def stratum_key(task: dict) -> str:
    bucket = "narrow" if task["edit_span"] <= 30 else "wide"
    return f"{task['language']}:{bucket}"


def stratum_counts(tasks: list[dict]) -> dict[str, int]:
    counts: dict[str, int] = {}
    for task in tasks:
        key = stratum_key(task)
        counts[key] = counts.get(key, 0) + 1
    short = {key: count for key, count in counts.items() if count < MIN_PER_STRATUM}
    if short:
        raise SystemExit(f"stratum floor missed: {short}")
    return counts


def digest_of(tasks: list[dict]) -> str:
    blob = json.dumps(tasks, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()


def build_card(tasks: list[dict]) -> dict:
    return {
        "dataset_digest": digest_of(tasks),
        "task_count": len(tasks),
        "strata": stratum_counts(tasks),
        "metric": {
            "name": "resolved_rate",
            "numerator": "hidden tests pass within the attempt budget",
            "denominator": "tasks in this frozen slice",
            "attempt_budget": ATTEMPT_BUDGET,
            "pass_at_k_reported_separately": True,
        },
        "controls": ["no_edit_must_fail", "known_fix_must_pass"],
        "not_a_claim_about": [
            "unseen repositories",
            "security behavior",
            "runner permanence",
        ],
    }


def freeze(tasks: list[dict], dest: Path) -> None:
    card = build_card(tasks)
    dest.write_text(json.dumps(card, indent=2) + "\n", encoding="utf-8")
    print(f"froze {card['dataset_digest'][:12]} tasks={card['task_count']}")


def verify(tasks: list[dict], card_path: Path) -> None:
    recorded = json.loads(card_path.read_text(encoding="utf-8"))
    actual = digest_of(tasks)
    if recorded.get("dataset_digest") != actual:
        raise SystemExit("slice drifted; refuse to publish the score")
    stratum_counts(tasks)
    print(f"slice ok {actual[:12]} tasks={len(tasks)}")


def main() -> None:
    if len(sys.argv) != 4:
        raise SystemExit("usage: benchmark_card.py freeze|verify tasks.json card.json")
    command, task_path, card_path = sys.argv[1], Path(sys.argv[2]), Path(sys.argv[3])
    tasks = load_tasks(task_path)
    if command == "freeze":
        freeze(tasks, card_path)
    elif command == "verify":
        verify(tasks, card_path)
    else:
        raise SystemExit("use freeze or verify")


if __name__ == "__main__":
    main()

A minimal task file can live next to the module, with one object per repository repair the harness is allowed to score. Each object records language, edit span, license, and whether a failing test already exists before the agent runs. The stratum floor of five and the attempt budget of one are example thresholds in this proposal, not universal standards. A one-task file should fail freeze, and that failure demonstrates the floor instead of a broken script.

[
  {
    "task_id": "demo-1",
    "repo": "example/widgets",
    "commit": "abc123",
    "language": "python",
    "edit_span": 12,
    "license": "mit",
    "has_failing_test": true
  }
]

python3 benchmark_card.py freeze tasks.min.json card.json
python3 benchmark_card.py verify tasks.min.json card.json

The second script is a separate publish gate, and it is labeled unexecuted because no live outcomes were supplied here. It reads a local report, demands both harness controls, and only then computes a resolved rate. The printed rate stays tied to the attempt budget in that report, so a later plot cannot silently change the denominator. Operators should treat a missing control flag as an incomplete run, not as a zero that can be averaged away.

#!/usr/bin/env python3
"""Unexecuted proposal: refuse a score unless both harness controls passed."""

import json
import sys
from pathlib import Path


def resolved_rate(outcomes: list[dict], budget: int) -> float:
    eligible = [row for row in outcomes if row["attempts"] <= budget]
    if not eligible:
        raise SystemExit("no eligible attempts")
    passed = sum(1 for row in eligible if row["hidden_tests_passed"])
    return passed / len(eligible)


def main() -> None:
    report = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    if report.get("no_edit_must_fail") is not True:
        raise SystemExit("no-edit control failed; do not publish")
    if report.get("known_fix_must_pass") is not True:
        raise SystemExit("known-fix control failed; do not publish")
    rate = resolved_rate(report["outcomes"], report["attempt_budget"])
    print(f"resolved_rate={rate:.3f} n={len(report['outcomes'])}")
    print("not a product verdict; read not_a_claim_about in the card")


if __name__ == "__main__":
    main()

python3 publish_gate.py report.json

Availability does not create the claim

MonkeyCode's free model access and free server option can host the harness while a team is still arguing about the metric contract. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free access matters only as a way to repeat the same frozen slice while the card is still under review. A free server option can keep the harness and the manifest together so the digest check is not a laptop-only ritual.

This draft does not name models, quotas, hardware, duration, or permanence, because those details were not verified here. Current product documentation should be read before a run is scheduled, and observed limits should be copied into the card. Neither availability option selects the dataset, defines the resolved rate, or repairs a broken no-edit control by itself. A score produced on a convenient runner still needs the card, since convenience is not a substitute for a frozen claim.

This protocol does not measure security behavior, long-horizon planning, or skill on private monorepos absent from the slice. Hidden tests can leak through verbose logs, so the harness should redact failure output before any retry prompt is built. Small slices carry wide uncertainty, and a three-point gap on forty tasks is often noise rather than a stable ranking. Human review remains necessary when tests pass by deleting assertions or by weakening the oracle that was supposed to judge the patch.

Teams that need a procurement number by Friday should not wear this card as a costume for a decision already made. Researchers who cannot publish the task list, or a durable pointer to it, cannot invite others to audit the rate. Agents that only chat, and never emit a patch against a repository, sit outside a resolved-rate contract entirely. Operators without a failing-test oracle should choose another metric instead of forcing this harness to invent one.

A useful reading of the finished card sounds like a lab note rather than a product launch sentence. It says which languages were thick enough to interpret, which strata missed the floor and were therefore withheld, and which control failed. It also says what the number is not, including unseen repositories, security work, and any claim that a runner remains available. That negative space is what keeps a later slide from promoting a slice result into a universal ranking.

The practical next step is to freeze at least a few dozen tasks, run both controls, and only then attach a model score. Readers who need a shared runner while they rehearse that order can try MonkeyCode's free model access and free server option. The manifest should still land in version control beside the score, because a convenient runner does not audit itself. Until the card and the controls travel with the percentage, the number remains a headline instead of evidence.

Session Envelopes Decide Whether an Agent Delta Ships

Avery Wang — Thu, 24 Sep 2026 21:59:18 +0000

An agent leaderboard becomes evidence only when the dataset slice, the metric definition, and the server session travel together. A lone pass rate can rise because the queue shortened, the fallback model changed, or an easy stratum dominated the sample. The method below treats those shifts as disqualifying events rather than as footnotes added after publication. Readers can delete every product name and still apply the same gate to ordinary local logs.

Free model access and a free server do not remove variance; they relocate it into admission control, shared queues, and silent routing. A kitchen scale on a moving truck still prints grams, yet the truck's motion is inside every reading. Benchmark authors who ignore that motion will publish a difference that belongs to the platform, not to the agent. The honest response is a pre-registered envelope for session health, not a larger headline font.

Evaluation engineers who rank coding agents need a bill of materials, much as a release engineer needs a lockfile. Without that bill, two runs that share a product name can still be different experiments in disguise. The audience here is the person who must defend a number in review, not the person who needs a demo screenshot. A demo may stay informal, while a published score cannot borrow that same looseness of setup.

The dataset in this contract is a frozen slice of tasks, not a wandering folder of prompts. Each task carries a stable identifier, a stratum label such as repair, migration, or explanation, and a content digest computed before the first run. Strata exist so that a cluster of short edits cannot outvote a handful of long migrations in the mean. The slice file is committed beside the harness, and any edit after the first scored run forces a new slice identifier.

The primary metric is a paired completion delta against a fixed baseline on the same task identifiers, not an unpaired mean across different slices. A secondary metric records timeout rate, and a third records whether the session stayed inside the health envelope. Variance is summarized with a blocked bootstrap over strata, and the publish rule demands a pre-registered minimum detectable effect. If the interval crosses zero, the harness emits no winner, because a noisy tie is not a ranking.

The control set is written into the manifest before any agent process is invoked on the slice. The baseline is a named, version-pinned runner, and the candidate sees the same task bytes, time budget, and tool permissions. A session stamp records server identity, queue delay, retry count, and a quota-state token read from the provider's live response. Runs with a missing stamp, a mid-run route change, or a quota-state outside the envelope are excluded before aggregation.

Contamination often arrives as a courtesy from the platform rather than as an obvious crash in the log. A provider may retry, shift region, or swap a route while still returning a completion that looks successful. The stamp treats that courtesy as a changed instrument, the way a chemist rejects a reading after an unlogged pipette change. Exclusion is therefore a validity rule for the instrument, not a punishment aimed at the candidate agent.

Numbers stop being marketing when a stranger can rerun the gate and obtain the same publish decision from the same logs. The harness below is a proposal with synthetic rows, and those rows are not measurements of any product. They exist so a reviewer can inspect the exclusion logic without waiting on a live network call. A real campaign replaces the fixture with captured stamps and does not reuse the toy pass flags as evidence.

#!/usr/bin/env python3
"""Session-stamp publish gate. Proposal only; rows are synthetic."""

from __future__ import annotations

import hashlib
import json
import statistics
from dataclasses import dataclass


@dataclass(frozen=True)
class TaskRow:
    task_id: str
    stratum: str
    digest: str
    baseline_pass: int
    candidate_pass: int
    queue_ms: int
    retries: int
    quota_state: str
    server_id: str
    route_changed: bool


ENVELOPE = {
    "allowed_quota_states": {"ok", "within_budget"},
    "max_queue_ms": 4000,
    "max_retries": 1,
    "required_server_id": "srv-lab-a",
    "min_detectable_effect": 0.05,
    "min_kept_tasks": 8,
}

# Synthetic fixture. Do not cite these flags as a product result.
FIXTURE = [
    TaskRow("t1", "repair", "a1", 1, 1, 800, 0, "ok", "srv-lab-a", False),
    TaskRow("t2", "repair", "a2", 0, 1, 900, 0, "ok", "srv-lab-a", False),
    TaskRow("t3", "repair", "a3", 1, 1, 1200, 0, "ok", "srv-lab-a", False),
    TaskRow("t4", "migration", "b1", 0, 0, 1500, 1, "ok", "srv-lab-a", False),
    TaskRow("t5", "migration", "b2", 0, 1, 9000, 0, "ok", "srv-lab-a", False),
    TaskRow("t6", "migration", "b3", 1, 1, 700, 0, "exhausted", "srv-lab-a", False),
    TaskRow("t7", "explain", "c1", 1, 1, 600, 0, "ok", "srv-lab-a", False),
    TaskRow("t8", "explain", "c2", 0, 1, 650, 0, "ok", "srv-other", False),
    TaskRow("t9", "explain", "c3", 1, 0, 700, 0, "ok", "srv-lab-a", True),
    TaskRow("t10", "repair", "a4", 0, 1, 500, 0, "within_budget", "srv-lab-a", False),
]


def slice_digest(rows: list[TaskRow]) -> str:
    payload = "|".join(f"{r.task_id}:{r.digest}:{r.stratum}" for r in rows)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]


def keep(row: TaskRow) -> bool:
    return (
        row.quota_state in ENVELOPE["allowed_quota_states"]
        and row.queue_ms <= ENVELOPE["max_queue_ms"]
        and row.retries <= ENVELOPE["max_retries"]
        and row.server_id == ENVELOPE["required_server_id"]
        and not row.route_changed
    )


def paired_delta(rows: list[TaskRow]) -> float:
    return statistics.fmean(r.candidate_pass - r.baseline_pass for r in rows)


def blocked_means(rows: list[TaskRow]) -> dict[str, float]:
    strata: dict[str, list[TaskRow]] = {}
    for row in rows:
        strata.setdefault(row.stratum, []).append(row)
    return {name: paired_delta(group) for name, group in sorted(strata.items())}


def publish_decision(rows: list[TaskRow]) -> dict:
    kept = [row for row in rows if keep(row)]
    decision = {
        "slice_digest": slice_digest(rows),
        "n_seen": len(rows),
        "n_kept": len(kept),
        "dropped": [row.task_id for row in rows if not keep(row)],
        "stratum_deltas": blocked_means(kept) if kept else {},
        "headline": None,
        "reason": "",
    }
    if len(kept) < ENVELOPE["min_kept_tasks"]:
        decision["reason"] = "too_few_clean_tasks"
        return decision
    effect = paired_delta(kept)
    decision["paired_delta"] = round(effect, 4)
    if abs(effect) < ENVELOPE["min_detectable_effect"]:
        decision["reason"] = "below_minimum_detectable_effect"
        return decision
    spreads = list(blocked_means(kept).values())
    width = max(spreads) - min(spreads) if len(spreads) > 1 else 1.0
    if effect - width <= 0 <= effect + width:
        decision["reason"] = "stratum_spread_crosses_zero"
        return decision
    decision["headline"] = round(effect, 4)
    decision["reason"] = "publish_blocked_delta_only"
    return decision


if __name__ == "__main__":
    print(json.dumps(publish_decision(FIXTURE), indent=2))

Running the proposal on the bundled fixture should refuse a headline, and that refusal is the expected lesson. Four synthetic rows fall outside the envelope through queue delay, an exhausted quota state, a foreign server, or a route change. Six clean rows remain, which sits below the pre-registered floor of eight, so the reason field records too_few_clean_tasks. Lowering that floor after seeing the gap would convert a written control into a marketing dial.

A reviewer reproduces the decision with one local command and then stores the JSON beside the slice file. The digest printed by the script must match the digest recorded when the slice was frozen, or the study has changed. No network call belongs in this check, because a methodology test that needs today's API cannot prove yesterday's log. Teams can wrap the command in a review bot they already trust, without adding a new leaderboard service.

python3 session_gate.py > publish_decision.json
python3 -c 'import json; d=json.load(open("publish_decision.json")); assert d["headline"] is None; print(d["reason"], d["dropped"])'
sha256sum slice.json envelope.json >> run_bill.txt

Capturing a usable session stamp is a logging problem long before it becomes a model-quality problem. Each attempt should append one JSON object with task identifier, server identifier, queue delay, retries, quota state, and route change. Missing keys are failures, and backfilling them from memory is prohibited because memory is how stale quotas enter papers. The envelope file is reviewed like a schema, and an edit to it demands the same new identifier that a slice edit demands.

Even after the gate opens, the published figure is a blocked delta on a named slice, not a universal intelligence score. Stratum means stay visible so a gain on short repairs cannot hide a loss on long migrations. Timeout rate remains beside the delta, since a faster failure is not a completion under this contract. Readers who want a single crown should look elsewhere, because this contract is built to withhold crowns.

Operators who already hold MonkeyCode free model access and a free server option can treat that environment as one candidate session. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate does not assume a quota size, a hardware profile, a duration, or a permanent offer, because those fields go stale. A reviewer should copy the live access terms into the session stamp and ignore any remembered allowance printed elsewhere. One practical next step is to point this same script at a private log export and withhold any ranking until exclusions are empty.

This gate is the wrong tool for exploratory chats, one-off demos, and any study that cannot freeze the task bytes. It also fails teams who need a causal claim about model quality, because a paired delta on a tiny slice is only a publish filter. Shared free servers can still correlate errors across runs, so exclusion removes some contamination without creating independence. Authors who cannot record queue delay or quota state should not publish a numeric ranking from those logs at all.

A finished report returns to that contract rather than to a product scoreboard or a remembered allowance. A dataset slice, a paired metric, and a session envelope either agree with one another, or the report stays silent. Silence is the feature that separates a lab note from a campaign graphic with extra decimals. Until the stamp, the strata, and the baseline version can be reloaded together, the printed decimals remain decoration.

Hash the Task Pack Before Ranking Coding Agents

Avery Wang — Wed, 23 Sep 2026 17:21:57 +0000

A coding-agent ranking is trustworthy only after the task pack, the metric, and the run controls are hashed together. Unpinned models, live network calls, and unpublished graders turn yesterday's score into unverifiable advertising. This method treats agent evaluation like an API load test: freeze the workload, name the clocks, and publish the bundle. Strangers should be able to replay the same pack and obtain the same pass or fail vector.

Realistic API tests already refuse to measure a service against the public internet at noon. Agent benches that pull fresh GitHub issues, live docs, or today's model alias repeat that mistake in a prettier dashboard. The workload must be a file and the grader a command, or the published number is only a weather report. The rest of this article builds that closed loop with a small Python harness and a control sheet that travels with every score.

The task pack is a JSONL file of coding jobs, each with a frozen prompt, a fixture tree, and a shell grader. Tasks that need the open web are rewritten against local fixtures so the agent cannot harvest a newer answer from a search tool. Hidden tests stay off the prompt disk and run only after the agent writes an artifact, which keeps leakage from inflating pass rates. A SHA-256 of the sorted pack file becomes the dataset identity that every later score must cite.

Pass rate without weights is a trap when ten toy edits sit beside one multi-file refactor. Each task carries an integer weight for review cost, and the headline metric is weighted pass rate under a wall-clock budget. Median latency and recorded token spend travel as secondary axes so a cheap fast fail cannot hide behind a single percentage. Rankings that omit the budget or the weight vector are incomplete even when every test is green.

The control sheet pins the endpoint identifier, decoding temperature, sampler seed, tool-schema hash, and maximum wall seconds per task. Changing any field creates a new experiment rather than a better score on the old leaderboard. That rule is the same one load testers use when they refuse to mix mild and harsh arrival rates in one percentile chart. If a vendor renames a model alias, the sheet records the old pin as stale and the comparison stops.

A compact pack file looks ordinary on disk, which is the point: the dataset is boring, versioned, and diffable. Each line names one job, points at fixtures, and declares the only command that may grade the agent's output. Reviewers should read the grader before they read the leaderboard, because the command is the metric in executable form. The snippet below is a proposal for a three-task starter pack, not a claim about any public contest.

{"id":"fix-off-by-one","weight":1,"prompt_file":"prompts/fix-off-by-one.md","fixture_dir":"fixtures/fix-off-by-one","grade":["python","graders/fix-off-by-one.py"],"budget_s":90}
{"id":"parse-csv-dates","weight":2,"prompt_file":"prompts/parse-csv-dates.md","fixture_dir":"fixtures/parse-csv-dates","grade":["python","graders/parse-csv-dates.py"],"budget_s":120}
{"id":"split-god-module","weight":5,"prompt_file":"prompts/split-god-module.md","fixture_dir":"fixtures/split-god-module","grade":["python","graders/split-god-module.py"],"budget_s":240}

The harness never talks to the open web during a scored run, and it refuses to start when the pack hash drifts. It writes a bundle that contains the pack digest, the control sheet, per-task pass bits, wall times, and an optional spend field supplied by the adapter. Teams can swap the adapter for a local script, a vendor SDK, or a scratch endpoint without changing the grader. The listing below is labeled as a proposed runner; it is meant to be executed against fixtures the operator already owns.

#!/usr/bin/env python3
"""Proposed agent-benchmark runner. Freeze pack + controls, then emit a bundle."""
from __future__ import annotations

import hashlib, json, os, subprocess, sys, time
from pathlib import Path

PACK = Path("pack.jsonl")
BUNDLE = Path("score_bundle.json")

CONTROLS = {
    "endpoint_id": os.environ.get("AGENT_ENDPOINT", "local-adapter"),
    "temperature": 0.0,
    "seed": 7,
    "tool_schema_sha256": os.environ.get("TOOL_SCHEMA_SHA256", ""),
    "network": "blocked",
}

def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def load_pack(path: Path) -> tuple[str, list[dict]]:
    raw = path.read_bytes()
    digest = sha256_bytes(raw)
    tasks = [json.loads(line) for line in raw.splitlines() if line.strip()]
    if not tasks:
        raise SystemExit("empty pack")
    ids = [t["id"] for t in tasks]
    if len(ids) != len(set(ids)):
        raise SystemExit("duplicate task ids")
    return digest, tasks

def grade_task(task: dict, workdir: Path) -> dict:
    started = time.monotonic()
    budget = int(task["budget_s"])
    try:
        proc = subprocess.run(
            task["grade"],
            cwd=workdir,
            capture_output=True,
            text=True,
            timeout=budget,
            env={**os.environ, "NETWORK": "blocked"},
        )
        elapsed = time.monotonic() - started
        passed = proc.returncode == 0
        return {
            "id": task["id"],
            "weight": int(task["weight"]),
            "passed": passed,
            "returncode": proc.returncode,
            "elapsed_s": round(elapsed, 3),
            "timeout": False,
        }
    except subprocess.TimeoutExpired:
        return {
            "id": task["id"],
            "weight": int(task["weight"]),
            "passed": False,
            "returncode": None,
            "elapsed_s": float(budget),
            "timeout": True,
        }

def weighted_pass(rows: list[dict]) -> float:
    num = sum(r["weight"] for r in rows if r["passed"])
    den = sum(r["weight"] for r in rows)
    return round(num / den, 4) if den else 0.0

def main() -> None:
    pack_sha, tasks = load_pack(PACK)
    expected = os.environ.get("EXPECTED_PACK_SHA256")
    if expected and expected != pack_sha:
        raise SystemExit(f"pack hash mismatch: {pack_sha}")
    rows = []
    for task in tasks:
        workdir = Path("runs") / task["id"]
        workdir.mkdir(parents=True, exist_ok=True)
        # Adapter contract: write artifact into workdir before the grader runs.
        adapter = os.environ.get("AGENT_ADAPTER", "./adapters/local.sh")
        subprocess.run(
            [adapter, task["id"], str(workdir), task["prompt_file"]],
            check=False,
            timeout=int(task["budget_s"]),
        )
        rows.append(grade_task(task, workdir))
    elapsed_values = sorted(r["elapsed_s"] for r in rows)
    median = elapsed_values[len(elapsed_values) // 2]
    bundle = {
        "pack_sha256": pack_sha,
        "controls": CONTROLS,
        "headline_metric": "weighted_pass_under_budget",
        "weighted_pass": weighted_pass(rows),
        "median_elapsed_s": median,
        "tasks": rows,
    }
    BUNDLE.write_text(json.dumps(bundle, indent=2, sort_keys=True))
    print(json.dumps({"pack_sha256": pack_sha, "weighted_pass": bundle["weighted_pass"]}))

if __name__ == "__main__":
    main()

A matching adapter can be a thin shell wrapper that sends the frozen prompt to whatever endpoint the lab is measuring that day. The wrapper must not mutate fixtures, must not fetch extra context, and must exit when the task budget elapses. Spend, if the endpoint reports it, is copied into the bundle as a secondary field rather than folded into the headline pass rate. Mixing spend into the pass fraction without a published formula is how cost disappears from a ranking and reappears as surprise invoices.

export EXPECTED_PACK_SHA256="$(python3 - <<'PY'
import hashlib, pathlib
print(hashlib.sha256(pathlib.Path('pack.jsonl').read_bytes()).hexdigest())
PY
)"
export AGENT_ADAPTER=./adapters/local.sh
export AGENT_ENDPOINT=local-adapter
python3 harness.py

Numbers leave the lab only after the bundle names the pack hash, the control sheet, and the weighted pass rate in one document. A later run that cannot reproduce the pass vector under the same hash is not a regression in the agent; it is a broken experiment. Publishing the percentage alone, without those three fields, is the agent-eval equivalent of posting p99 latency without the traffic shape. Readers should treat such a percentage as marketing copy even when the underlying model is strong.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A lab that needs a scratch box for the adapter and harness can use MonkeyCode's free model access and free server option, then keep the signed bundle and discard the machine. That pairing matters only as a place to execute the same frozen pack; it does not replace the hash, the weights, or the wall-clock bound. Claims about named models, quotas, hardware, or lasting performance are out of scope here because they are not part of the method.

The approach fails for agents whose job is live browsing, incident response on production traffic, or research over a corpus that cannot legally be snapshotted. It also fails for vendor bake-offs that refuse to pin an endpoint and still want a single winner by Friday. Teams without rights to ship fixtures and hidden tests should not pretend a public demo repository is a dataset. In those settings the honest artifact is a qualitative report, not a ranked table.

Limitations remain even when the pack is frozen. Graders can be wrong, weights can encode taste, and a zero temperature pin does not make a non-deterministic tool stack repeatable. The harness records timeouts as failures, which punishes slow correct answers and can hide partial progress that a human reviewer would salvage. Secondary axes such as median time and reported spend need their own units and missing-data rules, or they become decorative columns beside the headline.

The load-test analogy is the check against self-deception. Nobody ships an API percentile from a laptop on café wifi and calls it capacity planning, yet agent scores still travel that way in launch posts. Hash the pack, print the controls, weight the tasks, and bound the clock before the ranking is allowed to leave the repository. The number that survives that gate is smaller, duller, and finally comparable.

Replay Fixtures Separate Agent Scores From API Weather

Avery Wang — Tue, 22 Sep 2026 15:36:13 +0000

A published agent score remains marketing until the tool log is hashed and replayed under isolation controls. Live API calls inject vendor drift, rate-limit noise, and hidden retries that later captions cannot reconstruct. Treating the fixture pack as data, not scenery, is what turns a fluent demo into a measurement. Teams that skip this step compare weather reports from different cities and name the gap a ranking.

Developer demos often treat tool calling as a product feature rather than a measurement surface. That framing is understandable, because an agent invoking an API looks like competence in motion. Competence in motion is still not a dataset, because the world behind the API keeps moving. The same prompt can harvest a new JSON shape or a new auth error while the score prints one decimal.

Load testing a checkout service against the public internet during a sale produces a similar illusion. The latency histogram then describes the sale and the retry budget as much as the service itself. Agent runs that punch live search, tickets, or calendars import the same contamination into the mean. The proposed protocol isolates that weather by replaying a hashed fixture pack and publishing the controls beside the metrics.

Dataset: a fixture pack, not a prompt list

The dataset is a directory of tasks where every tool request already has a recorded response, including failures. Each task file binds a prompt, an allowed tool schema, request fingerprints, and the exact response bytes. Tasks that require a real side effect, such as opening a ticket, do not belong in this pack. Those tasks belong in an operational drill that should never be averaged into a public model score.

A proposed fixture looks like the following JSON document. Fingerprints and schema digests are truncated here so nobody treats them as a downloaded pack.

{
  "task_id": "billing.refund.v3",
  "prompt": "Refund order 1842 if the payment captured and the window is open.",
  "tools": [
    {
      "name": "payments_get",
      "schema_sha256": "sha256:task-local-schema-digest"
    }
  ],
  "calls": [
    {
      "request_fingerprint": "sha256:task-local-call-digest",
      "response_status": 200,
      "response_bytes": "{\"captured\":true,\"window_open\":true}",
      "latency_ms_recorded": 41
    }
  ],
  "expected": {
    "action": "refund",
    "order_id": "1842"
  }
}

The request fingerprint is a canonical hash of method, path, and a sorted JSON body. Cosmetic key order therefore cannot mint a cache miss or a silent live fetch during scoring. Response bytes are stored literally, because pretty-printed JSON and compact JSON are different fixtures under a hash. The pack still needs a manifest that records timeout, retry budget, and every task digest as first-class data.

# proposed: protocol_sheet.py — unexecuted example for a replay pack
import hashlib, json, pathlib, sys

def sha256_file(path: pathlib.Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()

def build_manifest(pack_dir: pathlib.Path) -> dict:
    tasks = sorted(
        path for path in pack_dir.glob("*.json") if path.name != "manifest.json"
    )
    return {
        "protocol": "tool-replay-v1",
        "timeout_s": 30,
        "retries": 0,
        "seed": 20260922,
        "tasks": {path.name: sha256_file(path) for path in tasks},
    }

if __name__ == "__main__":
    pack = pathlib.Path(sys.argv[1])
    manifest = build_manifest(pack)
    payload = json.dumps(manifest, sort_keys=True, indent=2) + "\n"
    out = pack / "manifest.json"
    out.write_text(payload, encoding="utf-8")
    print(sha256_file(out), out)

Running that script produces a manifest hash that later public reports must quote beside the mean.

python protocol_sheet.py ./packs/billing-v3
# prints the manifest digest and path; commit both before any scoring run

If a single fixture byte changes, the hash changes, and yesterday's leaderboard row becomes incomparable rather than quietly stale. Timeout and retry count sit in the same file because they change the metric even when the caption still says accuracy. A score collected with two retries is a different measurement from a score collected with zero retries.

Metrics: a vector with a variance band

The headline number people want is task success, yet success without a failure taxonomy is a labeled coin flip. The proposed script records exact action match, schema-valid calls, unauthorized attempts, and timeout under the frozen budget. Averaging those four fields into one trophy is allowed only after the report prints the whole vector. Early collapse hides agents that succeed by calling tools the published schema never offered to anyone.

Variance belongs in the same paragraph as the mean, because sampling agents are not fully deterministic under replay. The protocol repeats each task a fixed number of times with a documented seed and a bootstrap interval. A ranking that moves less than that interval is a tie, not a launch claim. Publishing the interval is how a group admits noise instead of decorating noise as progress.

# proposed: score_replay.py — unexecuted example, not a published result
import hashlib, json, random, statistics, time
from typing import Callable

class ReplayMismatch(RuntimeError):
    pass

def bootstrap_mean(values, n=1000, seed=20260922):
    rng = random.Random(seed)
    means = []
    for _ in range(n):
        sample = [values[rng.randrange(len(values))] for _ in values]
        means.append(sum(sample) / len(sample))
    means.sort()
    low = means[int(0.025 * n)]
    high = means[int(0.975 * n)]
    return statistics.mean(values), low, high

def fingerprint(method: str, path: str, body: dict) -> str:
    canonical = json.dumps(
        {"method": method, "path": path, "body": body},
        sort_keys=True,
        separators=(",", ":"),
    )
    return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest()

def replay_or_raise(task, method, path, body):
    expected = fingerprint(method, path, body)
    for call in task["calls"]:
        if call["request_fingerprint"] == expected:
            return call["response_status"], call["response_bytes"]
    raise ReplayMismatch(f"unrecorded call on {task['task_id']}: {expected}")

def score_task(task, agent_fn: Callable) -> dict:
    started = time.monotonic()
    try:
        result = agent_fn(
            task, lambda method, path, body: replay_or_raise(task, method, path, body)
        )
        elapsed = time.monotonic() - started
        success = int(
            result.get("action") == task["expected"]["action"]
            and result.get("order_id") == task["expected"]["order_id"]
        )
        return {
            "success": success,
            "schema_ok": 1,
            "unauthorized": 0,
            "timeout": int(elapsed > 30),
            "elapsed_s": elapsed,
        }
    except ReplayMismatch:
        return {
            "success": 0,
            "schema_ok": 0,
            "unauthorized": 1,
            "timeout": 0,
            "elapsed_s": time.monotonic() - started,
        }

def summarize(rows):
    successes = [row["success"] for row in rows]
    mean, low, high = bootstrap_mean(successes)
    n = len(rows)
    return {
        "n": n,
        "success_mean": round(mean, 4),
        "success_ci95": [round(low, 4), round(high, 4)],
        "unauthorized_rate": round(sum(row["unauthorized"] for row in rows) / n, 4),
        "timeout_rate": round(sum(row["timeout"] for row in rows) / n, 4),
    }

The ReplayMismatch path is the entire point of the harness, not an error to be softened later. An agent that invents a fourth query has left the dataset even when the extra query looks reasonable. The harness must not silently fetch live bytes in order to be helpful during a public run. Helpful harnesses are the usual way that careful benchmarks slowly decay into product tours with extra decimals.

Controls: isolation that would embarrass a flaky suite

Controls are the unglamorous twin of metrics, and they are what keep the printed numbers from being marketing. The protocol pins the working directory, the timeout clock, and a network allowlist that cannot reach the original APIs. Retry count stays at zero unless the metric name itself includes the retry policy in plain language. If the allowlist is missing, the fixture pack is only a suggestion, and suggestions do not generate rankings.

A second control is runner diversity, because a protocol that only passes on one laptop is a local script. A clean checkout of the same manifest hash on a second host is the minimum replication step worth citing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that second runner when the goal is proving the harness. The comparison that matters is whether two processes print the same digest and overlapping confidence intervals.

# proposed second-runner check after the pack and harness are committed
git clone <harness-repo> && cd <harness-repo>
git checkout <commit-with-manifest>
python protocol_sheet.py ./packs/billing-v3
python - <<'PY'
import hashlib, pathlib
path = pathlib.Path("packs/billing-v3/manifest.json")
print(hashlib.sha256(path.read_bytes()).hexdigest())
PY
# abort the scoring run if this digest does not match the published protocol sheet

Clock control deserves its own paragraph because traces love to print model latency as if it were task difficulty. Recorded fixture latencies may feed a secondary cost axis, but they must not mix into the success mean. Mixing them produces a ranking that rewards whichever replay host had the quieter neighbor that afternoon. That ranking is meteorology again, and meteorology does not belong on an agent leaderboard under any caption.

Why the numbers are not marketing

Marketing numbers omit the protocol sheet, while a measurement quotes the sheet in the same block as the mean. A non-marketing report lists the manifest hash, timeout, retry budget, seed, unauthorized rate, and confidence interval together. It also states the negative space: which live systems were not called and which ranking gaps are smaller than the interval. Tasks excluded for side effects belong in that same block, or readers will assume the pack is complete.

The following report block is the proposed publication unit. Anything shorter should be treated as an anecdote, not as a ranking input.

protocol: tool-replay-v1
manifest_sha256: <digest of billing-v3>
timeout_s: 30
retries: 0
seed: 20260922
repeats_per_task: 5
success_mean: <fill from summarize()>
success_ci95: [<low>, <high>]
unauthorized_rate: <fill from summarize()>
timeout_rate: <fill from summarize()>
live_network: denied
excluded: ticket.write, calendar.create (side-effect drills)

Those figures are illustrative placeholders for the shape of a report, not measurements of any hosted model. Replacing the placeholders without rerunning the harness would recreate the problem this article is trying to kill. The discipline is the sheet, not the decimal, and the sheet is what makes a later audit possible. A decimal without a sheet should be read as an anecdote, however many dashboards repeat it.

Limitations and who should not use this protocol

Replay fixtures measure policy following against a recorded world rather than competence against a living production surface. Fraud patterns and incident response will look too easy or too dated under a frozen tool log. Teams in those settings need operational drills with live canaries, and they should keep those drills off the public average. Mixing the two audiences is how a careful replay score becomes an unsafe claim about production readiness.

The protocol is a poor fit for agents whose value is exploration rather than tool-schema fidelity. A mismatch is scored as unauthorized even when the extra call would have been wise in an open world. Researchers studying open-world browsing should say so instead of forcing that work through a fingerprint table. Cheap extra capacity can also replicate a bad metric at larger scale, and that replication remains marketing.

The practical next step is to write the protocol sheet for one internal pack and refuse unmatched scores. A score that cannot point at a manifest hash should not travel into a comparison table. Readers who later want an extra machine for the replication check can inspect MonkeyCode after that sheet exists. The sheet is the product of the benchmark, and the spare server is only a witness to the hash.

Control Deltas Turn Agent Scores Into Evidence

Avery Wang — Mon, 21 Sep 2026 13:56:10 +0000

Control deltas turn raw agent scores into evidence because they hold the dataset, the metric, and the runtime constant. A pass rate quoted without a frozen control is closer to marketing copy than to a measurement. The same agent can look strong on a laptop and weak in CI once the tool timeout or the retriever index changes. Publishing the signed difference against a pinned baseline removes that theatrical gap between demos and labs.

Think of a track meet that reports only finish times and never names the wind, the track, or the starter pistol. Runners would still be ranked, yet nobody could replay the race or trust the order. Agent leaderboards often work that way, treating the hidden prompt, the tool sandbox, and the judge as scenery. A benchmark methodology has to freeze those props before anyone averages a score.

The evaluation contract in this article is a small YAML file plus a runner that refuses to print a score without a control. The contract pins a dataset hash, a metric module, a control agent identifier, and a runtime pin. Those four fields are the difference between a blog number and a result another team can replay. What follows is a practical protocol for dataset, metrics, and controls, with a Python harness that other machines can execute.

When the dataset is a living folder of prompts, every quiet edit silently rebases the history of every published chart. The honest unit is a snapshot: one JSONL file, one SHA-256 digest, and a sampling rule that does not change mid-report. Each line should carry a task id, a prompt, and an expected artifact so the metric can fail closed instead of guessing. A pack of twenty well-specified tasks is enough to prove the harness; a larger pack should wait until the protocol itself is boring.

Metric functions belong in versioned source, not in a spreadsheet cell that someone might round for a slide. Exact-match strings, schema-valid JSON, and a bounded wall-clock flag are dull on purpose, because dull functions replay next week. Model-as-judge scores can sit beside those checks, but they cannot be the only column if the judge itself is an unpinned model. The report should print each metric under its module path and content digest so reviewers can see what was actually computed.

The control agent is the quiet half of the experiment, like a placebo arm in a trial that a launch blog would rather omit. It may be a scripted stub that returns empty tool calls, a frozen checkpoint, or last week's production prompt under the same sandbox. What matters is that it consumes the same tasks, the same timeout, and the same network policy as the candidate. The number that leaves the lab is then candidate minus control, with a simple spread over task-level paired differences.

Marketing wants a single percentage that can travel without footnotes, screenshots, or a disappointed control. Evidence wants a delta, a protocol hash, and a sentence about who is missing from the sample. If the control already scores 0.81, a candidate at 0.84 is a small lift rather than a revolution. If the control scores 0.10 because the sandbox was misconfigured, the candidate's 0.90 is a hardware anecdote wearing a lab coat.

The worked example below is a proposed harness, not a claim about any production fleet. It pins the protocol, hashes the task pack, runs control and candidate on each row, and prints a delta report. Teams can swap the call_agent function for their own HTTP client, CLI wrapper, or in-process stub. The important behavior is that the script exits nonzero when the control arm is missing or the dataset digest does not match.

# eval_protocol.yaml — proposed contract, not a live leaderboard
protocol_id: agent-bench-2026-09-21
dataset:
  path: tasks.jsonl
  sha256: REPLACE_AFTER_HASHING
metrics:
  module: metrics.py
  functions: [exact_match, json_schema_ok, within_deadline]
control:
  name: frozen-stub-v1
  kind: scripted
runtime:
  python: "3.11"
  timeout_s: 30
  seed: 20260921
candidate:
  name: agent-under-test

{"id": "t001", "prompt": "Return {\"ok\": true, \"n\": 3}", "expect": {"ok": true, "n": 3}, "deadline_s": 5}
{"id": "t002", "prompt": "Return {\"ok\": true, \"n\": 0}", "expect": {"ok": true, "n": 0}, "deadline_s": 5}

Save those two objects as tasks.jsonl, one object per line, then hash the file before any candidate run is allowed to print.

# bench.py — proposed evaluation harness (unexecuted example)
from __future__ import annotations

import hashlib, json, time, sys
from pathlib import Path
from statistics import mean, pstdev

import yaml

def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()

def exact_match(got, expect) -> float:
    return 1.0 if got == expect else 0.0

def json_schema_ok(got, expect) -> float:
    if not isinstance(got, dict):
        return 0.0
    return 1.0 if set(expect).issubset(got) else 0.0

def within_deadline(elapsed_s: float, deadline_s: float) -> float:
    return 1.0 if elapsed_s <= deadline_s else 0.0

def call_agent(kind: str, prompt: str) -> dict:
    # Proposed stub: replace with a real client. Control stays deterministic.
    if kind == "scripted":
        return {"ok": True, "n": 0}
    if 'n": 3' in prompt:
        return {"ok": True, "n": 3}
    return {"ok": True, "n": 0}

def run_arm(kind: str, tasks: list[dict]) -> list[dict]:
    rows = []
    for task in tasks:
        started = time.perf_counter()
        got = call_agent(kind, task["prompt"])
        elapsed = time.perf_counter() - started
        rows.append({
            "id": task["id"],
            "exact_match": exact_match(got, task["expect"]),
            "json_schema_ok": json_schema_ok(got, task["expect"]),
            "within_deadline": within_deadline(elapsed, task["deadline_s"]),
        })
    return rows

def paired_delta(control_rows, candidate_rows, key: str):
    by_id = {row["id"]: row for row in control_rows}
    pairs = [row[key] - by_id[row["id"]][key] for row in candidate_rows]
    mu = mean(pairs)
    sd = pstdev(pairs) if len(pairs) > 1 else 0.0
    return mu, sd

def main(protocol_path: str) -> int:
    proto = yaml.safe_load(Path(protocol_path).read_text())
    data_path = Path(proto["dataset"]["path"])
    digest = sha256_file(data_path)
    pinned = proto["dataset"]["sha256"]
    if pinned != "REPLACE_AFTER_HASHING" and pinned != digest:
        print(f"dataset digest mismatch: {digest} != {pinned}", file=sys.stderr)
        return 2
    tasks = [json.loads(line) for line in data_path.read_text().splitlines() if line.strip()]
    control = run_arm(proto["control"]["kind"], tasks)
    candidate = run_arm("candidate", tasks)
    print(f"protocol_id={proto['protocol_id']}")
    print(f"dataset_sha256={digest}")
    print(f"n_tasks={len(tasks)}")
    print(f"runtime_python={proto['runtime']['python']} timeout_s={proto['runtime']['timeout_s']} seed={proto['runtime']['seed']}")
    for key in proto["metrics"]["functions"]:
        c_mean = mean(row[key] for row in control)
        k_mean = mean(row[key] for row in candidate)
        delta, sd = paired_delta(control, candidate, key)
        print(f"{key}: control={c_mean:.3f} candidate={k_mean:.3f} delta={delta:.3f} sd={sd:.3f}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1] if len(sys.argv) > 1 else "eval_protocol.yaml"))

Hash the pack, write the digest into the YAML file, then run the harness on the same tree the protocol names.

python - <<'PY'
from pathlib import Path
import hashlib
print(hashlib.sha256(Path("tasks.jsonl").read_bytes()).hexdigest())
PY
pip install pyyaml
python bench.py eval_protocol.yaml

The printed line that matters is not candidate=0.850 sitting alone on a slide. The printed line that matters is delta=0.150 next to a dataset digest, a protocol id, and a runtime pin. Anyone who cannot reproduce that delta on a second machine is looking at a demo, even if the first number was real on Friday afternoon. Variance across tasks is part of the result; hiding the spread is how a thin lift becomes a press sentence.

A control delta can still lie if the candidate was trained or prompted on the same twenty tasks. The protocol does not detect leakage; it only detects protocol drift between two runs that claim to be comparable. Teams that publish should say whether the pack was held out, and they should rotate the pack when it becomes a study guide. Until that sentence exists, the delta is a replayable demo rather than a generalization claim.

The runtime pin is the least glamorous field and the one that breaks first in the wild. Python minor versions, JSON library quirks, and DNS timeouts all move scores without moving anything that looks like intelligence. Recording the interpreter version, the timeout, and the seed next to the delta keeps those ghosts visible. If two labs cannot match those three values, they are not yet arguing about agents.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A pinned protocol is only useful if a second machine can run it, and MonkeyCode's free model access and free server option are one way to give that machine a shared place to call a model and execute the runner. That is an availability convenience rather than a quality ranking, and it does not replace a local pin when a team already has locked CI. Engineers who need a shared box can replay the YAML there and compare deltas against the harness output.

This approach is the wrong tool for several common jobs, and those jobs should stay elsewhere. Safety certification, red-team coverage, and open-ended writing quality still need human review that a three-metric stub cannot impersonate. Teams that cannot freeze a task pack, or that change tools between the control run and the candidate run, will manufacture deltas that describe process noise. Vendor bake-offs that allow each model a private prompt and a private timeout are not using this protocol, even when the chart colors match.

The honest publication format is therefore short and slightly inconvenient. Name the protocol id, the dataset digest, the metric module, the control identity, and the paired delta with its spread. Leave the raw candidate percentage in an appendix where it cannot be screenshotted alone. If the control arm cannot run, the write-up should not run either. That rule is the entire methodology, dressed as a script that another checkout can execute.

Sign the Metric Before You Publish an Agent Score

Avery Wang — Sun, 20 Sep 2026 11:50:55 +0000

A headline agent score is not a measurement until the metric function is frozen, hashed, and tested against golden fixtures. Teams still paste a single percentage into a README and treat that figure as evidence. A one-line change in the scorer can move the number more than any model swap. This article treats the metric as an instrument that must be signed before any ranking is quoted.

Measuring agents without a signed metric is like weighing samples on a scale whose firmware updates overnight. The reading looks precise and the units look familiar, yet the instrument is a different object every morning. Freezing task files helps, but it does not bind the function that turns traces into points. When that function drifts, yesterday’s ranking and today’s ranking are not comparable, even with identical task bytes.

A usable dataset for this protocol is a directory of task cards with stable identifiers and content hashes, not a living spreadsheet. Each card should name inputs, oracles, and a timeout that is recorded rather than silently extended after a slow run. The split between probe items and evaluation items belongs in the same digest, because a moved row is a new experiment. The commands below hash that tree so a published number can point at one object and no other.

# content-address the evaluation pack; do not edit files after this digest is quoted
find eval_pack -type f -print0 | sort -z | xargs -0 sha256sum > eval_pack.sums
sha256sum eval_pack.sums > eval_pack.tree.sha256
cat eval_pack.tree.sha256

Metrics need a contract: a versioned function, a fixture suite, and an interval estimate instead of a naked mean. The contract states which fields of a trace count and how missing patches, crashed shells, or truncated logs are scored. Golden fixtures are tiny traces with known scores, and they fail the build if a refactor of the scorer changes them. Bootstrap intervals then show whether a claimed gap is larger than sampling noise on the same frozen pack.

# metric_contract.py — labeled sketch for a frozen scorer, not a vendor result
from __future__ import annotations

import hashlib, json, random
from pathlib import Path
from typing import Callable

CONTRACT_ID = "agent-metric/v3-patch-apply"
PASS, FAIL, INVALID = 1.0, 0.0, float("nan")

def digest_source(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def score_trace(trace: dict) -> float:
    if trace.get("schema") != "trace.v1":
        return INVALID
    if trace.get("exit_code") not in (0, 1):
        return INVALID
    if not trace.get("patch"):
        return FAIL
    oracle = set(trace.get("oracle_files") or [])
    touched = set(trace.get("touched_files") or [])
    tests = trace.get("tests_passed")
    if oracle and not oracle.issubset(touched):
        return FAIL
    return PASS if tests is True else FAIL

FIXTURES = [
    ({"schema": "trace.v1", "exit_code": 0, "patch": "diff", "oracle_files": ["a.py"],
      "touched_files": ["a.py"], "tests_passed": True}, PASS),
    ({"schema": "trace.v1", "exit_code": 0, "patch": "", "oracle_files": ["a.py"],
      "touched_files": [], "tests_passed": False}, FAIL),
    ({"schema": "trace.v2", "exit_code": 0, "patch": "diff", "oracle_files": [],
      "touched_files": [], "tests_passed": True}, INVALID),
]

def assert_fixtures(fn: Callable[[dict], float]) -> None:
    for i, (trace, expected) in enumerate(FIXTURES):
        got = fn(trace)
        if got != expected and not (got != got and expected != expected):
            raise AssertionError(f"fixture {i}: expected {expected!r}, got {got!r}")

def valid_scores(rows: list[float]) -> list[float]:
    return [x for x in rows if x == x]

def bootstrap_mean_ci(rows: list[float], n: int = 2000, alpha: float = 0.05, seed: int = 7) -> dict:
    rng = random.Random(seed)
    xs = valid_scores(rows)
    if len(xs) < 8:
        return {"n_valid": len(xs), "mean": None, "ci95": None, "note": "too few valid runs"}
    means = []
    for _ in range(n):
        sample = [xs[rng.randrange(len(xs))] for _ in xs]
        means.append(sum(sample) / len(sample))
    means.sort()
    lo = means[int(alpha / 2 * n)]
    hi = means[int((1 - alpha / 2) * n)]
    return {
        "n_valid": len(xs),
        "n_invalid": len(rows) - len(xs),
        "mean": sum(xs) / len(xs),
        "ci95": [lo, hi],
        "seed": seed,
        "resamples": n,
    }

if __name__ == "__main__":
    assert_fixtures(score_trace)
    print(json.dumps({
        "contract_id": CONTRACT_ID,
        "metric_sha256": digest_source(Path(__file__)),
        "fixture_count": len(FIXTURES),
    }, indent=2))

Controls keep the instrument honest without turning the write-up into a brochure. A stub agent that returns empty patches should sit near the floor if the metric actually penalizes missing work. A shuffled-oracle pass should collapse toward chance if filenames or ordering leak the answer into the scorer. Seed locks and environment fingerprints sit beside the score so a later reader can see whether the runtime or the trace schema changed. None of these controls prove a product is better; they only show that the number was produced under a named procedure.

# controls.py — labeled sketch: stub agent and shuffled oracles as scorer sanity checks
import json, os, platform, random, subprocess, sys
from pathlib import Path
from metric_contract import score_trace, bootstrap_mean_ci, CONTRACT_ID, digest_source

def fingerprint() -> dict:
    py = subprocess.check_output([sys.executable, "-c", "import sys; print(sys.version)"], text=True).strip()
    return {
        "python": py,
        "platform": platform.platform(),
        "cwd": str(Path.cwd()),
        "metric_sha256": digest_source(Path("metric_contract.py")),
        "tree_sha256": Path("eval_pack.tree.sha256").read_text().split()[0] if Path("eval_pack.tree.sha256").exists() else None,
    }

def stub_agent(task: dict) -> dict:
    return {
        "schema": "trace.v1",
        "exit_code": 1,
        "patch": "",
        "oracle_files": task.get("oracle_files", []),
        "touched_files": [],
        "tests_passed": False,
        "agent": "stub.empty_patch",
    }

def shuffle_oracles(tasks: list[dict], seed: int = 13) -> list[dict]:
    rng = random.Random(seed)
    files = [t.get("oracle_files", []) for t in tasks]
    rng.shuffle(files)
    out = []
    for task, oracle in zip(tasks, files):
        item = dict(task)
        item["oracle_files"] = oracle
        out.append(item)
    return out

def run_pack(tasks: list[dict], agent) -> list[float]:
    return [score_trace(agent(t)) for t in tasks]

if __name__ == "__main__":
    tasks = json.loads(Path("eval_pack/tasks.json").read_text())
    stub_rows = run_pack(tasks, stub_agent)
    shuffled = shuffle_oracles(tasks)
    report = {
        "contract_id": CONTRACT_ID,
        "env": fingerprint(),
        "stub_empty_patch": bootstrap_mean_ci(stub_rows),
        "shuffled_oracles_on_stub": bootstrap_mean_ci(run_pack(shuffled, stub_agent)),
        "seed_lock": {"pythonhashseed": os.environ.get("PYTHONHASHSEED")},
    }
    Path("score_report.json").write_text(json.dumps(report, indent=2, default=str))
    print(json.dumps(report, indent=2, default=str))

export PYTHONHASHSEED=0
python metric_contract.py
python controls.py
# quote score_report.json as a whole; do not lift only the mean field into a title

The report is the artifact, and the mean is only one field inside it. A serious write-up quotes the contract identifier, the metric digest, the tree digest, the valid-run count, the invalid-run count, and the interval. If two agents overlap inside that interval on the same pack, the honest sentence is that the pack cannot separate them yet. Stretching a 0.04 gap into a crown graphic is how a measurement turns into marketing without any change in the underlying traces.

Reproducing that report is cheaper when the harness can run in a disposable workspace that already exposes models and a server, rather than on a laptop that silently differs from last week. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option can host the same frozen pack and the same signed scorer so a reviewer reruns metric_contract.py and controls.py without rebuilding the instrument by hand. The product is not the metric, and a convenience host does not enlarge a confidence interval or repair a leaking oracle.

Limitations follow from the statistics, not from taste. Bootstrap intervals treat tasks as roughly exchangeable, while agent packs are often clustered by repository, language, or bug class, which narrows the real uncertainty. Golden fixtures catch scorer regressions, but they cannot certify that partial credit matches a human maintainer’s judgment on messy diffs. Invalid runs must stay visible; dropping crashes until the mean looks tidy is another quiet rescoring. Wall-clock, token use, and retry counts belong in the report as context, yet they are not a substitute for the signed success function.

Teams that need regulatory evidence, customer acceptance, or safety certification should not treat this sketch as a qualification method. Groups that cannot freeze the pack, or that let prompt text and hidden tools change between runs, will hash a moving target and then argue about ghosts. Writers who only want a single percentage for a launch post will find the extra fields inconvenient, which is the point of putting them in the file. A ranking that cannot name its instrument is not a ranking that later engineers can refute.

The core conclusion stays small on purpose. Publish the contract, the digests, the fixtures, and the interval, then let the mean sit in the middle of that paragraph. Readers who rerun the stub and the shuffle already know whether the scorer still behaves like yesterday’s instrument. That habit is slower than a leaderboard screenshot, and it is the difference between a number that can be cited and a number that can only be advertised. If the signed report is reproduced in a shared workspace, the useful next step is to keep the pack and the scorer in lockstep rather than to chase a prettier headline.

Calibrate the Judge Before the Agent Score

Avery Wang — Sat, 19 Sep 2026 09:57:23 +0000

An agent percentage is not a measurement until the scoring harness itself can be replayed against frozen golden traces. Leaderboards still publish a single success rate as if the grader, the timeouts, and the parse rules were as stable as the task files. Those hidden pieces move whenever a prompt template, a JSON schema, or a retry policy changes underneath the same dataset name. This article treats harness calibration as the first published result, and the candidate agent as a later comparison that inherits that protocol.

Public conversation keeps returning to a familiar worry that evaluation suites age faster than the systems they claim to rank. That worry is reasonable, because a test that once discriminated among tools can become a puzzle that many large models memorize. Expanding the suite without locking the grader simply multiplies an unstable instrument across more task files. A more honest step is to freeze a small golden-trace pack and refuse ranking until the harness scores those traces identically twice.

A golden trace is not a unit test for the agent; it is a unit test for the judge that assigns credit. Each trace records the task identifier, the exact stdout, the exit status, and the wall time from the original run. The protocol card then states how those fields map onto pass, fail, skip, and parse_error, including the timeout ceiling. If two independent checkouts of the harness disagree on a golden trace, the candidate ranking is invalid rather than merely late.

The dataset side of the card is a digest, not a friendly folder name on someone's laptop. Task files, fixtures, and the golden traces themselves are hashed together so a renamed repository cannot quietly swap a prompt. The metric contract sits beside that digest and names the aggregation rule, because micro-averages and family macros are not interchangeable summaries. Controls in this design are golden replay, an order shuffle, and an idempotent rescore, which catch graders that depend on global state.

Protocol card

{
  "protocol_id": "harness.v3.golden",
  "dataset_glob": "tasks/*.json",
  "dataset_sha256": "replace-after-hashing",
  "golden_glob": "golden/*.json",
  "metrics": {
    "pass_rule": "exit_code==0 and schema_ok and wall_ms<=timeout_ms",
    "timeout_ms": 15000,
    "aggregate": "macro_by_task_family"
  },
  "controls": ["golden_replay", "idempotent_rescore", "order_shuffle"]
}

That object is intentionally boring, because a timeout and a hash travel better than a rounded marketing percentage. The card also refuses to mention a candidate agent, since the harness must stand on its own before any comparison is fair. Teams that skip this file usually discover the omission later, when a one-line JSON parsing change moves every historical score. A leaderboard percentage without a calibrated judge resembles a stopwatch that was never zeroed at the starting line.

Replay kit

The following Python module implements the three controls against a directory of traces and prints a calibration report. It is labeled as a local replay proposal, not as evidence from a production contest or a vendor bake-off. Readers should replace the digest placeholders after the first real run, rather than copying a fictional hash into a blog table.

# harness_calibrate.py
from __future__ import annotations

import hashlib
import json
import random
from pathlib import Path
from typing import Any, Callable, Dict, List, Tuple

Trace = Dict[str, Any]
Score = Dict[str, Any]


def sha256_tree(root: Path, pattern: str) -> str:
    digest = hashlib.sha256()
    paths = sorted(root.glob(pattern))
    if not paths:
        raise FileNotFoundError(f"no files for {pattern}")
    for path in paths:
        digest.update(path.name.encode("utf-8"))
        digest.update(path.read_bytes())
    return digest.hexdigest()


def load_json_dir(root: Path, pattern: str) -> List[Trace]:
    items: List[Trace] = []
    for path in sorted(root.glob(pattern)):
        payload = json.loads(path.read_text())
        payload["_source"] = path.name
        items.append(payload)
    return items


def score_trace(trace: Trace, timeout_ms: int) -> Score:
    schema_ok = isinstance(trace.get("stdout"), str)
    if schema_ok and trace.get("expect_json"):
        try:
            json.loads(trace["stdout"])
        except json.JSONDecodeError:
            schema_ok = False
    elif trace.get("expect_json"):
        schema_ok = False
    wall = int(trace.get("wall_ms", timeout_ms + 1))
    passed = (
        int(trace.get("exit_code", 1)) == 0
        and schema_ok
        and wall <= timeout_ms
    )
    return {
        "id": trace.get("id", trace.get("_source")),
        "family": trace.get("family", "ungrouped"),
        "pass": passed,
        "parse_error": not schema_ok,
        "timeout": wall > timeout_ms,
    }


def macro_by_family(scores: List[Score]) -> Dict[str, float]:
    buckets: Dict[str, List[int]] = {}
    for row in scores:
        buckets.setdefault(row["family"], []).append(int(row["pass"]))
    return {
        name: (sum(vals) / len(vals) if vals else 0.0)
        for name, vals in sorted(buckets.items())
    }


def idempotent_rescore(traces: List[Trace], scorer: Callable[[Trace], Score]) -> bool:
    first = [scorer(item) for item in traces]
    second = [scorer(item) for item in traces]
    return first == second


def order_shuffle_stable(
    traces: List[Trace],
    scorer: Callable[[Trace], Score],
    seeds: Tuple[int, ...] = (0, 1, 7, 13),
) -> bool:
    baseline = macro_by_family([scorer(item) for item in traces])
    for seed in seeds:
        shuffled = list(traces)
        random.Random(seed).shuffle(shuffled)
        again = macro_by_family([scorer(item) for item in shuffled])
        if again != baseline:
            return False
    return True


def calibrate(root: Path, timeout_ms: int = 15000) -> Dict[str, Any]:
    traces = load_json_dir(root, "golden/*.json")
    scorer = lambda item: score_trace(item, timeout_ms)
    scored = [scorer(item) for item in traces]
    expected = {item["id"]: bool(item["must_pass"]) for item in traces}
    mismatches = [row["id"] for row in scored if row["pass"] != expected.get(row["id"])]
    report = {
        "dataset_sha256": sha256_tree(root, "tasks/*.json"),
        "golden_sha256": sha256_tree(root, "golden/*.json"),
        "golden_n": len(traces),
        "mismatches": mismatches,
        "idempotent": idempotent_rescore(traces, scorer),
        "order_stable": order_shuffle_stable(traces, scorer),
        "macro_by_family": macro_by_family(scored),
    }
    report["ready_for_candidates"] = (
        not mismatches and report["idempotent"] and report["order_stable"]
    )
    return report


if __name__ == "__main__":
    print(json.dumps(calibrate(Path(".")), indent=2))

A matching golden file makes the contract visible to anyone who clones the repository and runs the module. The fixture below is synthetic on purpose, so it should not be cited as a result from a live agent evaluation. Known-good traces prove the happy path still parses; they do not prove that the candidate model understood the task.

{
  "id": "json.extract.001",
  "family": "extract",
  "exit_code": 0,
  "wall_ms": 820,
  "expect_json": true,
  "stdout": "{\"ok\": true, \"value\": 42}",
  "must_pass": true
}

A second fixture should fail on purpose, because a harness that only sees successes cannot detect a grader that always returns true. Known-bad traces act like a laboratory blank, proving the instrument can still reject malformed output. Without that blank, a broken parser that treats every blob as valid JSON will inflate later agent scores by one hidden constant. Publishing the blank is what separates a measurement from a brochure, even when the candidate percentage looks impressive.

{
  "id": "json.extract.002",
  "family": "extract",
  "exit_code": 0,
  "wall_ms": 640,
  "expect_json": true,
  "stdout": "almost json but not",
  "must_pass": false
}

Commands for a stranger replaying the kit stay small and should live in continuous integration beside the percentage job. The hash step belongs in that pipeline so a silent fixture edit cannot land next to an unchanged published score. A failing calibration job is the desired outcome when someone improves the parser without updating golden expectations.

python -m pip install pytest
python harness_calibrate.py
python - <<'PY'
from pathlib import Path
from harness_calibrate import calibrate
report = calibrate(Path("."))
assert report["ready_for_candidates"], report
print("harness calibrated:", report["dataset_sha256"][:12])
PY

A short pytest file turns those assertions into a gate that ranking jobs must pass before any candidate is scored. The test does not claim that any hosted model is strong; it claims that the local judge is boringly consistent. Timeout handling is part of the metric contract, so a slow but pretty answer is still a failure under this protocol.

# test_harness_calibrate.py
from pathlib import Path
from harness_calibrate import calibrate, score_trace


def test_golden_pack_locks_the_judge():
    report = calibrate(Path("."))
    assert report["mismatches"] == []
    assert report["idempotent"] is True
    assert report["order_stable"] is True
    assert report["ready_for_candidates"] is True


def test_timeout_is_part_of_the_metric():
    slow = {
        "id": "slow.001",
        "family": "extract",
        "exit_code": 0,
        "wall_ms": 20000,
        "stdout": "{}",
        "expect_json": True,
    }
    row = score_trace(slow, timeout_ms=15000)
    assert row["pass"] is False
    assert row["timeout"] is True

Only after that gate is green does a candidate agent deserve a percentage stored beside the protocol hashes. People reading a blog table rarely notice a missing digest, which is why marketing scores travel farther than protocol cards. A fair write-up lists the candidate score as a comparison that inherits the frozen harness, not as an isolated trophy. Family-level rates should remain visible, because a two-item family must not be summarized as if it contained a hundred tasks.

When a team wants a second machine to rerun the candidate arm after calibration, a free remote shell can be enough. Disclosure: This article was prepared as part of MonkeyCode's product outreach, and the product notes below are availability claims rather than benchmark results. MonkeyCode currently offers free model access and a free server option that can host the same replay commands after local calibration. Those notes are not a quota, a hardware spec, or a claim that any particular model won the synthetic tasks above.

What the numbers still cannot mean

The numbers stay non-marketing for a dull reason: every published figure names the digest it depends on. A reader who cannot obtain the golden traces cannot audit the judge, and a reader who cannot hash the tasks cannot detect silent edits. Golden traces still freeze yesterday's failures, so they will not detect a novel cheating strategy the original agents never tried. A purely syntactic JSON check will miss semantic wrongness, and an unpinned model judge will reintroduce the drift this protocol tried to remove.

The shuffle test only proves order independence of the scorer, not independence from the host clock or a polluted virtualenv. Teams that should skip this workflow include daily prompt shootouts and classrooms that want a live demo more than a ledger. Vendors that refuse to publish failing traces cannot use this kit without turning the blank into another marketing surface. Safety-critical evaluations still need human review on top of schema checks, because a well-formed object can still be a harmful action.

Stale tests will keep aging, and public models will keep saturating yesterday's puzzles without asking anyone's permission. Ranking can still be honest if the first published artifact is the harness report, with hashes, mismatches, and a ready_for_candidates bit. Candidate percentages then become a second paragraph instead of the only sentence a reader remembers from the table.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Agent Scores Without a Null Pack Are Marketing

Avery Wang — Fri, 18 Sep 2026 08:17:33 +0000

An agent percentage published without a versioned dataset, a locked metric function, and a passing null pack is marketing copy. Weekend evaluations often move because the judge prompt, the task mix, or the timeout changed, not because the agent improved. A measurement protocol therefore refuses to print a success rate until those three pieces are sealed and the negative controls behave. The small harness below treats that refusal as a hard gate rather than a polite style recommendation.

Public coding-agent numbers have started to resemble product landing pages, even when the underlying tasks are thin. Saturated unit tests and loosely written judges both inflate pass rates without showing whether the agent can miss on purpose. When a suite cannot produce a clean miss, the headline percentage is no longer a measurement; it is a selected screenshot. That failure mode matches a load test that never opens a connection to the real database.

A defensible agent benchmark is a sealed protocol, not a folder of prompts edited after the first chart appeared. The dataset hash, the metric implementation, the timeout budget, and the null pack travel together as one version. Changing any of those fields creates a new protocol identity, so older percentages cannot be compared as if they were the same exam. The useful analogy is pinning a compiler and a test runner before quoting a runtime score.

The null pack is the piece most leaderboards skip because it can kill a flattering number. It is a small set of tasks whose outcomes are known in advance: items that must pass, items that must fail, and items that must be refused rather than guessed. If a supposedly strong agent sails through poisoned files, or fails tasks a stub solver should finish, the run is invalid and no percentage is emitted. The pack is not extra credit; it is the instrument check that comes before the experiment.

The following protocol is a proposed, unexecuted example. It does not claim production results, vendor rankings, or model-specific scores. Teams can copy the files, seal them, and replace the solver command with whatever agent they already run locally.

# protocol.yaml — identity for one exam, not a leaderboard
schema: agent-bench-protocol/v1
protocol_id: coding-repair.nullpack.2026-09-18
dataset:
  path: ./tasks/repair_v3.jsonl
  sha256: "PENDING_SEAL"
metric:
  module: ./metric_exact_diff.py
  sha256: "PENDING_SEAL"
budget:
  timeout_sec: 90
  max_files_touched: 8
null_pack:
  path: ./tasks/null_pack_v1.jsonl
  sha256: "PENDING_SEAL"
  rules:
    must_pass: ["NP-STUB-01", "NP-STUB-02"]
    must_fail: ["NP-POISON-01", "NP-LEAK-02"]
    must_refuse: ["NP-SECRET-01"]
publish:
  min_n: 40
  require_ci: true
  forbidden_fields: ["single_percentage_only"]

Sealing replaces every PENDING_SEAL value with a digest and then refuses further edits without a new protocol_id. The command below is intentionally boring, because a benchmark that needs a clever seal is already drifting toward theater.

#!/usr/bin/env bash
# seal_protocol.sh — proposed example, run from the protocol directory
set -euo pipefail
sha256sum tasks/repair_v3.jsonl tasks/null_pack_v1.jsonl metric_exact_diff.py \
  | tee protocol.sums
python3 - <<'PY'
import hashlib, pathlib, re, sys
text = pathlib.Path("protocol.yaml").read_text()
sums = {}
for line in pathlib.Path("protocol.sums").read_text().splitlines():
    digest, name = line.split()
    sums[name] = digest
mapping = {
    "./tasks/repair_v3.jsonl": sums["tasks/repair_v3.jsonl"],
    "./metric_exact_diff.py": sums["metric_exact_diff.py"],
    "./tasks/null_pack_v1.jsonl": sums["tasks/null_pack_v1.jsonl"],
}
for path, digest in mapping.items():
    text = text.replace('sha256: "PENDING_SEAL"', f'sha256: "{digest}"', 1)
if "PENDING_SEAL" in text:
    sys.exit("seal incomplete")
pathlib.Path("protocol.sealed.yaml").write_text(text)
print("sealed protocol written")
PY

The metric module must be a function with its own digest, not a chat prompt that a reviewer rephrases between runs. Exact-diff scoring is deliberately unimpressive: it compares a normalized unified diff against an oracle patch and returns a boolean. Teams that want a language-model judge can still use one, but that judge becomes part of the sealed metric file, not a hidden sidebar comment. If the judge changes, the protocol identity changes, and yesterday's percentage is retired.

# metric_exact_diff.py — proposed example
from pathlib import Path

def normalize(diff: str) -> str:
    lines = []
    for line in diff.splitlines():
        if line.startswith(("index ", "diff --git", "---", "+++")):
            continue
        lines.append(line.rstrip())
    return "\n".join(lines)

def score(result_diff: str, oracle_path: str) -> dict:
    oracle = normalize(Path(oracle_path).read_text())
    got = normalize(result_diff)
    return {"pass": got == oracle, "metric": "exact_diff_v1"}

Null-pack records look like ordinary tasks, except each row carries an expected class instead of an open-ended success flag. must_pass rows are trivial repairs a stub can finish, which detects a broken runner. must_fail rows include poisoned instructions and leaked answers that a careful agent should reject. must_refuse rows mention secrets or out-of-scope files; a completion that writes those files fails the instrument check even if the rest of the suite looks green.

# run_protocol.py — proposed example; prints no headline until gates pass
import json, random, statistics, subprocess, sys, time, yaml
from pathlib import Path
from metric_exact_diff import score

def load_jsonl(path):
    rows = []
    with open(path) as handle:
        for line in handle:
            rows.append(json.loads(line))
    return rows

def run_solver(task, timeout):
    started = time.time()
    proc = subprocess.run(
        ["bash", "./solve.sh", task["id"]],
        capture_output=True, text=True, timeout=timeout,
    )
    return proc.stdout, time.time() - started, proc.returncode

def bootstrap_ci(flags, rounds=1000):
    means = []
    n = len(flags)
    for _ in range(rounds):
        sample = [flags[random.randrange(n)] for _ in range(n)]
        means.append(sum(sample) / n)
    means.sort()
    lo = means[int(0.025 * rounds)]
    hi = means[int(0.975 * rounds)]
    return lo, hi, statistics.mean(flags)

def main(protocol_path):
    proto = yaml.safe_load(Path(protocol_path).read_text())
    if "PENDING_SEAL" in Path(protocol_path).read_text():
        sys.exit("refuse: protocol is not sealed")
    null_rows = load_jsonl(proto["null_pack"]["path"])
    for row in null_rows:
        diff, _, _ = run_solver(row, proto["budget"]["timeout_sec"])
        passed = score(diff, row["oracle"])["pass"]
        expected = row["expected_class"]
        if expected == "must_pass" and not passed:
            sys.exit(f"null pack failed open: {row['id']}")
        if expected in {"must_fail", "must_refuse"} and passed:
            sys.exit(f"null pack failed closed: {row['id']}")
    tasks = load_jsonl(proto["dataset"]["path"])
    if len(tasks) < proto["publish"]["min_n"]:
        sys.exit("refuse: sample smaller than publish.min_n")
    flags = []
    for task in tasks:
        diff, elapsed, code = run_solver(task, proto["budget"]["timeout_sec"])
        flags.append(1.0 if score(diff, task["oracle"])["pass"] else 0.0)
    lo, hi, mean = bootstrap_ci(flags)
    report = {
        "protocol_id": proto["protocol_id"],
        "n": len(flags),
        "mean": round(mean, 4),
        "ci95": [round(lo, 4), round(hi, 4)],
        "null_pack": "pass",
    }
    Path("report.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report))

if __name__ == "__main__":
    main(sys.argv[1])

A team that wants a number for slides will dislike this runner, because the happy path is easy to block. That is the point. Marketing copy needs a single percentage with no denominator, no interval, and no story about the tasks that should have failed. A measurement needs the opposite: a protocol identity, a sample size, a confidence interval, and a null pack that actually bites. If the interval is wide, the honest publication is the interval, not a rounded headline.

Variance is not a footnote for later. Two agents can share a mean and still disagree on which files they break, which is why a protocol that only stores a percentage cannot be replayed. The report above keeps n and ci95 in the same object as protocol_id, so a later reader can see whether the exam was large enough to quote. Teams that need a ranking can still rank, but they rank sealed protocols against each other rather than mixing leftover prompts from different weeks.

Running the gate locally stays small on purpose. The solver is a shell entrypoint, the metric is a file, and the dataset is content-addressed. That shape fits a short-lived server as well as a laptop, which is where free model access can participate without becoming the subject of the exam. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host the sealed runner when a laptop is a poor place to keep timeouts honest; they do not replace the protocol, the null pack, or the interval. The measurement remains valid if those product pieces are removed.

Several limitations follow from the same design, and they are not cosmetic. Exact-diff metrics under-count valid alternate patches, so this harness is a conservative instrument, not a creativity contest. Bootstrap intervals assume the task rows are exchangeable; they are not a substitute for stratified sampling when the dataset is a pile of unrelated bugs. The null pack only detects the failures someone thought to encode, which means a clever leak outside those rows can still flatter a solver. Timeouts, network flakes, and non-deterministic tools can also fail the pack for reasons that have nothing to do with agent quality.

This approach is a poor fit for sales demos, for regulatory claims, and for any score that must be produced before the protocol can be sealed. It is also the wrong tool when the sample is smaller than the publish floor, when the oracle patches are themselves unreviewed, or when a team needs a qualitative read of design taste rather than a boolean metric. Those jobs want notebooks, design reviews, or hold-the-line tests of a different kind. They do not want a percentage that pretends to be an exam.

The core conclusion does not change after the code is copied. An agent score is a measurement only when the dataset, the metric, and the null pack share a version and the run can still fail closed. Numbers that cannot fail are advertisements with extra decimal places. Seal the protocol, watch the null pack, and publish the interval; the ranking, if it is still needed, can wait until those pieces exist.