Write the Provenance Ledger Before the Benchmark Number

A benchmark score is only as trustworthy as the record that produced it, and most published records omit nearly everything that matters. A percentage appears next to a terminal screenshot, the write-up ships, and six weeks later nobody can name the dataset revision or the decode settings behind it. The remedy is unglamorous and cheap to build: every run emits a provenance ledger, and no score gets quoted without one.

The argument below is methodological rather than promotional, and the artifact is small enough to adopt in an afternoon. Three separate drifts corrupt comparisons, and each of them can be pinned down with hashes and identifiers. Pinning is not the same as freezing forever, because a pin with a recorded hash can be bumped deliberately and the bump then becomes visible in code review.

Three drifts that quietly change the question

Dataset drift arrives first, usually as a quiet append to a JSONL file that nobody re-hashes before the next sweep. Harness drift follows when a default temperature, retry count, or timeout changes between releases of the runner itself. Sampling drift is the honest one, because the same model, prompt, and seed still returns different completions across repeats and provider-side revisions.

The first two drifts are accidental, and the third is unavoidable. A ledger records all three at write time rather than reconstructing them during a postmortem. Teams that skip the record tend to attribute the third drift to the second one, which produces confident and wrong conclusions about a harness upgrade.

The ledger as a build artifact

The recorder below uses only the Python standard library, so it drops into an existing harness without new dependencies. It is illustrative code that has not been executed against a live endpoint in this write-up, and field names should be adapted to whatever schema registry the team already owns.

# ledger.py - emit one provenance record per benchmark run
import hashlib, json, platform, subprocess, time
from pathlib import Path

def sha256_16(path):
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for chunk in iter(lambda: fh.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()[:16]

def git_rev():
    try:
        out = subprocess.check_output(
            ["git", "rev-parse", "--short", "HEAD"], text=True)
        return out.strip()
    except Exception:
        return "no-git"

def build_ledger(run_id, tasks_path, model_id, seed,
                 temperature, max_tokens, harness_version):
    tasks = Path(tasks_path)
    return {
        "run_id": run_id,
        "created_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "dataset": {
            "file": tasks.name,
            "sha256_16": sha256_16(tasks_path),
            "n_tasks": sum(1 for _ in tasks.open()),
        },
        "harness": {
            "version": harness_version,
            "git": git_rev(),
            "python": platform.python_version(),
        },
        "sampling": {
            "model_id": model_id,
            "seed": seed,
            "temperature": temperature,
            "max_tokens": max_tokens,
        },
        "host": {"platform": platform.platform()},
    }

if __name__ == "__main__":
    rec = build_ledger("r11", "tasks/pin.jsonl",
                       "endpoint-reported-id", 11, 0.0, 1024, "0.4.2")
    Path("runs/r11").mkdir(parents=True, exist_ok=True)
    Path("runs/r11/ledger.json").write_text(
        json.dumps(rec, indent=2, sort_keys=True))

Two fields carry disproportionate weight during an audit. The truncated dataset hash catches accidental edits to a task file, and the endpoint-reported model identifier catches silent provider-side swaps that a friendly alias would conceal. The ledger belongs in version control next to the results, not on the laptop of whoever ran the sweep.

Report the interval, not the point

A single mean over a few hundred tasks is a point estimate wearing a lab coat, and its variance is usually larger than the effect under discussion. The paired bootstrap below resamples per-task deltas, which keeps the comparison honest when the task set is small and the scoring is coarse. The arrays in the example are placeholders and were not produced by any live run.

# paired.py - paired bootstrap interval over per-task deltas
import random, statistics

def paired_delta_ci(baseline, candidate, iters=10_000, seed=7, alpha=0.05):
    assert len(baseline) == len(candidate)
    rng = random.Random(seed)
    deltas = [c - b for b, c in zip(baseline, candidate)]
    n = len(deltas)
    boots = sorted(
        statistics.fmean(rng.choices(deltas, k=n)) for _ in range(iters))
    mean = statistics.fmean(deltas)
    lo = boots[int(alpha / 2 * iters)]
    hi = boots[int((1 - alpha / 2) * iters) - 1]
    return {"n": n, "mean_delta": round(mean, 4),
            "lo": round(lo, 4), "hi": round(hi, 4)}

if __name__ == "__main__":
    base = [0, 1, 1, 0, 1, 1, 0, 0, 1, 1]
    cand = [1, 1, 1, 0, 1, 1, 1, 0, 1, 1]
    print(paired_delta_ci(base, cand))

The acceptance rule matters more than the code. An interval that contains zero is not evidence of improvement, no matter how flattering the mean delta looks in a slide. When the interval straddles zero, the correct response is to extend repeats or widen the task set, not to reinterpret the point estimate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's operator-supplied free model access and free server option matter here for one narrow reason, which is that repeats are the quantity this method actually consumes, and a free tier lowers the marginal cost of running them. Quota, hardware, and model availability should be checked on the project's own pages rather than assumed from this article.

# The harness owns its own flags; the endpoint only supplies identifiers.
export MC_MODEL_ID="<identifier the endpoint reports>"
export MC_BASE_URL="<base URL from the free server option>"
for s in 11 12 13; do
  python -m harness.run --tasks tasks/pin.jsonl \
    --model "$MC_MODEL_ID" --base-url "$MC_BASE_URL" \
    --temperature 0 --seed "$s" --out "runs/r$s"
done
sha256sum tasks/pin.jsonl runs/r*/ledger.json

The exact flag names depend on the harness in use, and the identifiers above are placeholders rather than documented values. What makes the triple run checkable is the hash list, which turns three directories into one auditable statement. If any hash differs between the runs, the comparison is invalid before the scores are even read.

Turning ledger state into a decision

The table below is the part that keeps benchmark work from drifting into marketing. Each row pairs an observed ledger condition with a concrete action, so the decision is made by the record rather than by whoever is most invested in the result.

Ledger condition	Interval	Action
Dataset hash and model id stable	excludes zero	Ship the claim with the ledger attached
Dataset hash changed mid-series	any	Discard the series and rerun from the pinned set
Model id changed with no release note	any	Treat as a new candidate and re-baseline
Everything stable	straddles zero	Add repeats or tasks; do not publish

A row that reads "discard the series" is the most valuable one in practice. Series assembled across a silent dataset edit are extremely persuasive and completely unreproducible, and the ledger catches them before they reach a changelog. The cost of that catch is a few lines of hashing code and one extra directory per run.

Limits and who should not adopt this

The ledger records conditions, not quality, and it will happily certify a well-pinned set of badly written tasks. Task quality needs a separate grading artifact, and coverage questions stay outside the scope of this method entirely. Shared free endpoints can also be contended, so latency readings from such a run should not be quoted as performance evidence.

Teams that need certified compliance evaluations, or fair absolute comparisons across vendors, should not use an internal ledger as a substitute for an accredited benchmark. The same warning applies to anyone who wants one number to settle an architecture argument, because the interval will usually refuse to do that. The approach fits internal regression tracking, harness upgrades, and pre-release sanity checks where reproducibility is the actual goal.

If a live endpoint is needed to exercise the paired loop, the free model access and free server option are enough to run the three-seed sequence above and generate a first ledger of your own. The scores can wait until the hashes agree.