Publish the Fixture Hash With the Score

A coding-model score is evidence only when the dataset, the grader, and the runtime share one frozen version. A single pass rate without those pins behaves like a press release, because later reruns cannot prove they measured the same task. Teams that want numbers an auditor can replay should version fixtures first and publish variance second. Marketing leaderboards collapse several unstable choices into one headline, which is why they age so badly.

Think of the evaluation set as a sealed envelope rather than a mood board of prompts. If the envelope gains examples or loses edge cases, the score becomes a moving target dressed as a constant. Public coding puzzles leak into training corpora the way popular jokes leak into every conversation, so yesterday's hard item becomes today's autocomplete. A serious protocol therefore treats the dataset as a contract with a hash, a date, and an explicit holdout that never appears in demos.

The proposed contract below is a working sketch for discussion, not a claimed production evaluation run. Each item names a language, a failing test, and a forbidden import list that blocks hidden helpers. The hash of that file belongs beside the metric so a reader knows which world produced the number. Changing a single expected string without bumping the version turns yesterday's failure into today's quiet success.

# eval/fixtures/v0.3.1.yaml — proposed contract, not a live leaderboard
schema: coding-eval/v1
dataset_id: internal-repair-holdout
dataset_version: 0.3.1
frozen_at: "2026-09-03"
holdout_policy: never-in-prompts-or-blog-posts
items:
  - id: repair-csv-header-01
    language: python
    prompt_file: prompts/repair-csv-header-01.md
    test_file: tests/test_repair_csv_header_01.py
    timeout_s: 8
    forbidden_imports: ["pandas", "requests"]
    expected_files: ["repair.py"]
  - id: fix-off-by-one-window-02
    language: python
    prompt_file: prompts/fix-off-by-one-window-02.md
    test_file: tests/test_fix_off_by_one_window_02.py
    timeout_s: 8
    forbidden_imports: ["numpy"]
    expected_files: ["window.py"]

A metric family refuses the fantasy of one winner by reporting several incomparable measurements together. Functional tests answer whether the patch behaves, while a static check answers whether the patch stayed inside the allowed surface. A repair-loop count answers how much extra conversation the task consumed, which remains a cost even when the final file is green. Publishing only the green rate hides expensive retries the way a restaurant review that skips the bill hides the evening.

# eval/score.py — proposed scorer; treat output as a draft protocol
from __future__ import annotations
import hashlib, random, statistics, subprocess, sys, time
from pathlib import Path

def file_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

def run_pytest(test_file: Path, timeout: int) -> dict:
    start = time.monotonic()
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", str(test_file)],
        capture_output=True, text=True, timeout=timeout,
    )
    return {
        "passed": proc.returncode == 0,
        "seconds": round(time.monotonic() - start, 3),
        "tail": (proc.stdout + proc.stderr)[-800:],
    }

def score_item(item: dict, seed: int) -> dict:
    random.seed(seed)
    result = run_pytest(Path(item["test_file"]), item["timeout_s"])
    result.update(id=item["id"], seed=seed)
    return result

def summarize(rows: list[dict]) -> dict:
    flags = [1.0 if r["passed"] else 0.0 for r in rows]
    lat = [r["seconds"] for r in rows]
    return {
        "n": len(rows),
        "pass_rate": round(sum(flags) / len(flags), 4),
        "pass_stdev": round(statistics.pstdev(flags), 4) if len(flags) > 1 else 0.0,
        "latency_p50": round(statistics.median(lat), 3),
        "latency_max": round(max(lat), 3),
    }

# tests/test_repair_csv_header_01.py — proposed holdout item, unexecuted example
from pathlib import Path
import importlib.util

def load_repair():
    path = Path("repair.py")
    spec = importlib.util.spec_from_file_location("repair", path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return mod

def test_header_is_normalized():
    mod = load_repair()
    text = "Name, AGE, city\nAda,36,London\n"
    out = mod.normalize_csv(text)
    assert out.splitlines()[0] == "name,age,city"

Controls matter as much as the tests because an unpinned decoding setting is a second hidden dataset. Temperature, seed list, token cap, network access, and working directory all change what a recorded success actually means. A fresh process per item prevents one solution from leaking into the next through files, environment variables, or leftover agent memory. The commands below pin those knobs for a local rerun so a reviewer can replay the same sealed envelope.

# proposed local control plane; adjust paths, do not treat as a vendor SLA
export EVAL_DATASET=eval/fixtures/v0.3.1.yaml
export EVAL_SEEDS="11,29,47"
export EVAL_TEMPERATURE=0
export EVAL_NETWORK=off
export EVAL_WORKDIR=$(mktemp -d /tmp/eval-XXXXXX)
python - <<'PY'
import hashlib, os, pathlib
p = pathlib.Path(os.environ["EVAL_DATASET"])
print("dataset_hash", hashlib.sha256(p.read_bytes()).hexdigest())
print("seeds", os.environ["EVAL_SEEDS"])
print("workdir", os.environ["EVAL_WORKDIR"])
PY
python eval/run_suite.py --seeds "$EVAL_SEEDS" --dataset "$EVAL_DATASET"

# eval/run_suite.py — proposed runner, unexecuted example
from __future__ import annotations
import argparse, json, yaml
from pathlib import Path
from score import file_hash, score_item, summarize

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--dataset", required=True)
    parser.add_argument("--seeds", required=True)
    args = parser.parse_args()
    payload = yaml.safe_load(Path(args.dataset).read_text())
    seeds = [int(s) for s in args.seeds.split(",")]
    rows = []
    for item in payload["items"]:
        for seed in seeds:
            rows.append(score_item(item, seed))
    report = {
        "dataset_id": payload["dataset_id"],
        "dataset_version": payload["dataset_version"],
        "dataset_hash": file_hash(Path(args.dataset)),
        "seeds": seeds,
        "summary": summarize(rows),
        "rows": rows,
    }
    Path("eval-report.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report["summary"], indent=2))

if __name__ == "__main__":
    main()

A publish rule should stay boring on purpose so enthusiasm cannot edit the protocol after a disappointing night. A team may print a score only when the fixture hash, the scorer commit, the seed list, and the environment digest share one report. If any of those four fields is missing, the number stays internal because an outsider cannot audit the measurement. The table below is a decision aid rather than a certification scheme, and it exists to stop a glowing screenshot from becoming official.

Condition	Publishable as evidence	Better treated as
Hash, scorer, seeds, and env digest present; n ≥ 3 seeds	Yes, with variance	—
Pass rate only, no fixture hash	No	Demo note
Holdout items used in a blog prompt	No	Contaminated run
Network left open on a task that forbids it	No	Uncontrolled run
Single seed, temperature unset	No	Exploratory probe

Variance is the part marketing almost always drops, because a wide spread makes the headline look unfinished. Three seeds will not satisfy a journal, yet they already expose the brittle case where one lucky decode inflates the story. If pass rate moves more than a small band across seeds, the honest report says the task is unstable under the current prompt. Latency belongs beside correctness for the same reason a train schedule belongs beside a destination name.

Free shared capacity can host this evaluation protocol for a spare rerun without quietly becoming the protocol itself. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that a team can use as one more spare runner for sealed reruns. The fixture files and the grader should remain in the team's own repository so a vendor outage cannot rewrite history.

The method fails closed for several audiences, and that refusal is part of the design rather than an afterthought. Certification labs, procurement teams, and safety reviewers need independent infrastructure and sample sizes this sketch does not provide. Launch teams chasing a screenshot should not wear the protocol as a costume, because freezing fixtures slows content more than it slows engineering. Students who are still learning to prompt can keep a looser notebook and should not present a weekend pass rate as a paper.

The scorer measures unit-test greenness rather than maintainability, security, or architectural fit, so a green file can still hide a destructive rewrite. Public problem statements remain contamination risks even with a private holdout, because similar tasks circulate under different names. Free servers and free model access can throttle or change without notice, so they should not freeze an environment digest for a formal claim. The YAML contract cannot stop a human from editing expected output after a bad night, so the hash must be published before the metric.

A useful picture is a grocery scale that prints its calibration date next to the weight. Nobody trusts a number from a scale that is recalibrated in the back room between customers, and coding evals deserve the same suspicion. Pin the envelope, print the hash, report the spread, and leave the cardboard crown in the drawer. The work is slower than a leaderboard screenshot, and that slowness is the signal that the number might still mean something next month.