Hash the Suite. Lock the Controls. Then You May Quote a Number.

You get the ping at 16:40. "We're at 81% now. Can marketing have it?"

You ask which tasks, which commit, which temperature, which timeout, which tools, and what a pass even means. They send a screenshot. One green tile. No files.

That number is a press release. A benchmark is a replay kit: a frozen dataset, a metric spec, a control block, and one command that rebuilds the table from those artifacts. If you cannot rerun it, you cannot defend it.

This is a methodology, not a leaderboard. The scripts below are a template. Comments that look like scores are illustrative. They are not results from a private suite I ran for you.

What "not marketing" actually means

A marketing number is a point estimate with no denominator, no variance, and no way to reproduce the run. A benchmark number is boring on purpose. It names the dataset version, the metric formula, the controls locked before the first call, and the cost envelope that produced it.

You ship four files, not a percentage:

1. tasks.jsonl — the suite, then a hash of that file.
2. metrics.py — the only functions allowed to score a run.
3. controls.yaml — seed, temperature, timeout, retries, tool allowlist, model endpoint.
4. replay.sh — one command that rebuilds report.md from the three files above.

If any of those four is missing, you do not publish a score. You publish a demo.

The job this kit is built for

You are comparing two coding agents on a private suite of repository tasks. Each task is a git tree plus a hidden test command. You need the harness to sit overnight without a credit-card surprise, and you need the same files to replay on Monday morning.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access and a free server option. You can point the control block at that endpoint and leave the replay kit running unattended. Strip the product out and the four files still work against any OpenAI-compatible base URL. The host is plumbing. The methodology is the point.

Step 1 — Freeze the dataset, then hash it

Do not score against a living folder named tasks/. Living folders drift. Someone tweaks a fixture. Someone "cleans up" a prompt. Your 81% becomes a different experiment with the same label.

Export one JSONL file. One object per task. Then hash the bytes you will actually feed the harness.

# example layout — adjust paths; do not treat these counts as a published n
python3 - <<'PY'
import hashlib, json, pathlib
src = pathlib.Path("tasks.jsonl")
raw = src.read_bytes()
digest = hashlib.sha256(raw).hexdigest()
print(f"tasks.jsonl sha256={digest} bytes={len(raw)} lines={raw.count(10)}")
PY

Keep the schema tiny so you cannot hide extra hints in random fields:

{"id": "repo-017", "repo": "fixtures/repo-017.tar", "hidden_test": "pytest -q tests/test_hidden.py", "timeout_s": 120, "size_bucket": "sm", "tags": ["api", "refactor"]}

Rules you write down before the first call:

1. Hidden tests never enter the prompt. If the agent can read tests/test_hidden.py, the task is contaminated.
2. size_bucket is assigned by lines of production code, not by how you feel about the ticket.
3. You do not add, drop, or rewrite a task after controls.yaml is locked. You cut a new suite version instead.

If you cannot explain how a task entered the suite, it does not enter the suite.

Step 2 — Write the metric spec as code, not as a slide

Pass rate is a ratio. Ratios hide the denominator, the variance, and the cost of being right. Your metric module should refuse to print a single percentage without those three.

# metrics.py — template, not a published result
from __future__ import annotations

import math
from dataclasses import dataclass

@dataclass(frozen=True)
class Trial:
    task_id: str
    passed: bool
    timeout: bool
    tool_violation: bool
    prompt_tokens: int
    completion_tokens: int
    wall_s: float
    retries_used: int

def wilson_interval(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n <= 0:
        raise ValueError("n must be > 0")
    p = k / n
    z2 = z * z
    denom = 1.0 + z2 / n
    center = (p + z2 / (2 * n)) / denom
    half = (z * math.sqrt((p * (1 - p) + z2 / (4 * n)) / n)) / denom
    return (max(0.0, center - half), min(1.0, center + half))

def summarize(trials: list[Trial]) -> dict:
    n = len(trials)
    k = sum(1 for t in trials if t.passed)
    lo, hi = wilson_interval(k, n)
    tokens = sum(t.prompt_tokens + t.completion_tokens for t in trials)
    walls = sum(t.wall_s for t in trials)
    return {
        "n": n,
        "passes": k,
        "pass_rate": k / n,
        "wilson95": [lo, hi],
        "timeout_rate": sum(t.timeout for t in trials) / n,
        "tool_violation_rate": sum(t.tool_violation for t in trials) / n,
        "tokens_per_pass": (tokens / k) if k else None,
        "wall_s_per_task": walls / n,
        "retries_mean": sum(t.retries_used for t in trials) / n,
    }

Notice what this refuses to do. It will not let you quote pass_rate without n and a Wilson interval. It will not let a timeout look like a wrong answer. It will not let a tool violation look like a model miss. Cost sits next to correctness. That is the whole point.

A number is marketing when it is a point estimate. A number is a measurement when it carries a sample size, an interval, and a cost.

Step 3 — Lock the control block before the first token

Controls are the difference between an eval and a vibe. You write them down. Then you stop touching them.

# controls.yaml
suite_file: tasks.jsonl
suite_sha256: REPLACE_WITH_HASH
model_base_url: http://127.0.0.1:8080/v1
model_name: FROM_ENDPOINT_NOT_FROM_A_BLOG
temperature: 0.0
seed: 7
max_retries: 0
per_task_timeout_s: 180
max_output_tokens: 2048
tool_allowlist:
  - read_file
  - apply_patch
  - run_hidden_test
forbidden_paths:
  - tests/test_hidden.py
  - .git/
stop_on_control_mismatch: true

Number the lock procedure. Do not skip a line.

1. Fill suite_sha256 from Step 1. If the file bytes change, the run aborts.
2. Set max_retries to 0 for the headline table. Retries are a different product. Mix them in and you are scoring a loop, not an agent.
3. Pin temperature and seed. Creative sampling is a demo setting.
4. Name every allowed tool. An unbounded shell is not a coding-agent benchmark. It is a sysadmin benchmark.
5. Record the endpoint URL, not a nickname. Nicknames drift when someone "upgrades" the proxy.

If a later run cannot load this file and reproduce the same process table, the later run is a new experiment. Give it a new name.

Step 4 — Make replay the only way to print a table

Humans edit dashboards. Scripts edit report.md. You want the second one.

#!/usr/bin/env bash
# replay.sh — rebuild report.md from frozen inputs only
set -euo pipefail
python3 - <<'PY'
import hashlib, json, pathlib, sys, yaml
ctrl = yaml.safe_load(pathlib.Path("controls.yaml").read_text())
raw = pathlib.Path(ctrl["suite_file"]).read_bytes()
got = hashlib.sha256(raw).hexdigest()
if got != ctrl["suite_sha256"]:
    sys.exit(f"suite hash mismatch: got {got}")
print("controls ok; suite hash ok")
PY
python3 harness.py --controls controls.yaml --out runs.jsonl
python3 report.py --runs runs.jsonl --controls controls.yaml --out report.md

harness.py is your agent loop. Keep it dull. For each task it must write one JSONL row with the Trial fields from metrics.py. No extra narrative. No "mostly correct" flag. Binary pass, timeout, tool violation, tokens, wall clock, retries used.

report.py should render a table you would actually paste into a review:

| field | value |
|---|---|
| suite_sha256 | abc… |
| n | 40 |
| passes | 17 |
| pass_rate | 0.425 |
| wilson95 | 0.285 – 0.578 |
| timeout_rate | 0.100 |
| tool_violation_rate | 0.050 |
| tokens_per_pass | 12840 |
| wall_s_per_task | 74.2 |

Those interior figures are format examples. They are not a claim about any vendor, model, or night of running. If you publish a table, it must come from replay.sh on your suite, with your hash in the first row.

Why this table is harder to use as marketing

Look at the interval, not the point. On n=40, a 0.425 pass rate has a wide Wilson band. Anyone quoting "43%" without that band is rounding a small sample into a slogan.

Look at cost per pass, not cost per call. Cheap failures are not efficiency. They are a model that gives up early.

Look at timeout rate and tool-violation rate as first-class rows. If your "smarter" agent climbs pass rate by blowing the timeout budget or by reading hidden tests, you did not get smarter. You changed the rules.

That is the control story in one sentence: you freeze the rules so the number cannot be rescued by a quiet rule change.

A decision table you can keep next to the PR

Use this before anyone pastes a percentage into a README.

Question	If the answer is no	What you publish instead
Can you hash tasks.jsonl and match controls.yaml?	The suite moved.	"demo, suite unpinned"
Is pass defined by a hidden test the agent cannot read?	You scored the prompt.	"prompt eval, not agent eval"
Are retries zero in the headline run?	You scored a loop.	a separate retry study
Do you report n and a 95% interval?	You scored a point.	no public number
Do you report tokens per pass and wall time?	You hid the bill.	correctness plus envelope
Can a stranger rerun replay.sh?	You scored a screenshot.	internal note only

If two rows fail, you do not negotiate. You withhold the number.

Limitations

This kit does not make a small suite large. Wilson intervals get honest, not narrow. Private tasks leak the moment an engineer pastes one into a chat window. Free servers preempt. Free model endpoints can change behavior without sending you a changelog. Pin what you can pin. Do not pretend a pin is a contract.

The harness also cannot see contamination you failed to encode. If hidden tests live in the tarball under another name, forbidden_paths will not save you. If the prompt already contains the patch, you are scoring a copy task.

Do not treat OpenAI-compatible as "same model." Base URL, model name, and decoding knobs are part of the control block. Change one, and you start a new run id.

Who should not use this

Do not use this kit as a procurement score if you cannot freeze tools and tests. Buyers who need a certified figure need a lab protocol and a contract, not a JSONL file on a free server.

Do not use it if your n is a handful of golden tickets you already memorized. You will get a tight-looking rate and a worthless interval.

Do not use it to compare agents that were allowed different tools, different retry budgets, or different hidden tests. That comparison is a category error. Split the studies.

If you only need a live demo for a hallway, skip the kit. Run the demo. Do not call it a benchmark.

Run it overnight, then read the files

Copy the four files onto a quiet machine. Fill the hash. Keep retries at zero. Start replay.sh. In the morning you read report.md, not a chat transcript.

If you want that quiet machine to be a free server with free model access, MonkeyCode can host the same scripts. The replay kit stays the source of truth either way.