The Benchmark Borrow Slip: A One-Page SOP So Nobody Quotes an Eval Number You Can't Reprove

Any week when the leaderboards move, a number from someone else's eval lands in your roadmap slide. The slide says your agent scores 0.71 on the internal task set, and nobody in the room can name the harness commit, the box, or the token budget behind it. Two sprints later the model changes, the score drops, and the conversation stops being about a regression and becomes about whether anyone trusts the metric at all.

That is the scene this SOP is built for. Instead of a spreadsheet row nobody owns, you issue a borrow slip for every score you intend to repeat out loud: a small JSON artifact that pins the harness, the box, the budget, and the raw result together. If a number cannot produce a slip, it is a rumor, and rumors are allowed in chat but not in a deck.

Why an eval number decays in about a week

Three kinds of drift eat a score, and only one of them is the model. Harness drift is a prompt template edit, a new answer parser, or a grader threshold change that silently moves the denominator. Box drift is a free container with a different CPU quota, which changes timeouts, retries, and therefore pass rates on anything slow. Budget drift is running the same suite with half the tokens and measuring truncated answers instead of model quality.

You will not prevent all three. Your job is to make each one visible on an artifact that travels with the number, so the next person can see the difference before they argue about it.

2. The artifact: bench_slip.py, standard library only

This is a reference implementation. Dry-run it against your own harness directory before you wire it into CI, and treat the field list as the contract rather than the script as gospel. It records a harness hash, a box fingerprint including cgroup limits, the model id and sampling settings, the token budget versus tokens used, and the score itself.

#!/usr/bin/env python3
"""bench_slip.py - stamp an eval run so its number can be reproved later."""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import platform
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

SUFFIXES = (".py", ".yaml", ".yml", ".json", ".jsonl", ".toml")


def _sh(cmd: str) -> str:
    try:
        out = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=5)
        return out.stdout.strip() or "unavailable"
    except Exception:
        return "unavailable"


def harness_hash(root: Path) -> str:
    digest = hashlib.sha256()
    for path in sorted(root.rglob("*")):
        if path.is_file() and path.suffix in SUFFIXES:
            digest.update(str(path.relative_to(root)).encode())
            digest.update(path.read_bytes())
    return digest.hexdigest()[:16]


def box_fingerprint() -> dict:
    mem_kb = "unavailable"
    meminfo = Path("/proc/meminfo")
    if meminfo.exists():
        mem_kb = meminfo.read_text().splitlines()[0].split()[1]
    return {
        "host": platform.node(),
        "kernel": platform.release(),
        "machine": platform.machine(),
        "python": platform.python_version(),
        "cpu_count": os.cpu_count(),
        "mem_kb": mem_kb,
        "cgroup_cpu_max": _sh("cat /sys/fs/cgroup/cpu.max 2>/dev/null"),
        "cgroup_mem_max": _sh("cat /sys/fs/cgroup/memory.max 2>/dev/null"),
        "git_sha": _sh("git rev-parse --short HEAD 2>/dev/null"),
    }


def new_slip(args) -> dict:
    ratio = args.tokens_used / args.token_budget if args.token_budget else 0.0
    slip = {
        "run_id": args.run_id,
        "created_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "harness": {
            "path": str(Path(args.harness).resolve()),
            "hash": harness_hash(Path(args.harness)),
        },
        "model": {
            "id": args.model,
            "temperature": args.temperature,
            "max_tokens": args.max_tokens,
        },
        "budget": {
            "tokens_used": args.tokens_used,
            "token_budget": args.token_budget,
            "ratio": round(ratio, 3),
            "over_budget": ratio > 1.0,
        },
        "box": box_fingerprint(),
        "result": {"metric": args.metric, "score": args.score, "n_tasks": args.n_tasks},
        "notes": args.notes or "",
    }
    if slip["budget"]["over_budget"]:
        print("WARN: run exceeded its token budget; truncated answers are likely.", file=sys.stderr)
    return slip


def check(old: dict, new: dict) -> int:
    drift = []
    for key in ("hash",):
        if old["harness"][key] != new["harness"][key]:
            drift.append(f"harness.{key}: {old['harness'][key]} -> {new['harness'][key]}")
    if old["model"] != new["model"]:
        drift.append(f"model changed: {old['model']} -> {new['model']}")
    for key in ("cgroup_cpu_max", "cgroup_mem_max", "python", "cpu_count"):
        if old["box"].get(key) != new["box"].get(key):
            drift.append(f"box.{key}: {old['box'].get(key)} -> {new['box'].get(key)}")
    if old["result"]["n_tasks"] != new["result"]["n_tasks"]:
        drift.append(f"task count: {old['result']['n_tasks']} -> {new['result']['n_tasks']}")
    delta = new["result"]["score"] - old["result"]["score"]
    print(f"score: {old['result']['score']} -> {new['result']['score']} ({delta:+.3f})")
    if drift:
        print("NOT COMPARABLE:")
        for line in drift:
            print(f"  - {line}")
        return 1
    print("COMPARABLE: harness, model and box match.")
    return 0


def as_markdown(slip: dict) -> str:
    b, r = slip["budget"], slip["result"]
    return (
        f"**Run {slip['run_id']}** - {slip['created_utc']}\n\n"
        f"- model `{slip['model']['id']}` temp {slip['model']['temperature']}\n"
        f"- harness `{slip['harness']['hash']}`\n"
        f"- box `{slip['box']['host']}` cpu_max `{slip['box']['cgroup_cpu_max']}`\n"
        f"- tokens {b['tokens_used']}/{b['token_budget']} (ratio {b['ratio']})\n"
        f"- {slip['result']['metric']} = **{r['score']}** over {r['n_tasks']} tasks\n"
    )


def main() -> int:
    p = argparse.ArgumentParser()
    sub = p.add_subparsers(dest="cmd", required=True)

    n = sub.add_parser("new")
    n.add_argument("--run-id", required=True)
    n.add_argument("--harness", required=True)
    n.add_argument("--model", required=True)
    n.add_argument("--metric", default="pass@1")
    n.add_argument("--score", type=float, required=True)
    n.add_argument("--n-tasks", type=int, required=True)
    n.add_argument("--tokens-used", type=int, required=True)
    n.add_argument("--token-budget", type=int, required=True)
    n.add_argument("--temperature", type=float, default=0.0)
    n.add_argument("--max-tokens", type=int, default=1024)
    n.add_argument("--notes", default="")
    n.add_argument("--out", default="slips")

    c = sub.add_parser("check")
    c.add_argument("old")
    c.add_argument("new")

    args = p.parse_args()
    if args.cmd == "new":
        slip = new_slip(args)
        out = Path(args.out)
        out.mkdir(parents=True, exist_ok=True)
        target = out / f"{slip['run_id']}.json"
        target.write_text(json.dumps(slip, indent=2) + "\n")
        print(as_markdown(slip))
        print(f"wrote {target}")
        return 0
    old = json.loads(Path(args.old).read_text())
    new = json.loads(Path(args.new).read_text())
    return check(old, new)


if __name__ == "__main__":
    raise SystemExit(main())

Two commands are all a teammate needs to learn, and the second one is the whole point of the exercise:

mkdir -p slips && git rev-parse --short HEAD

python bench_slip.py new \
  --run-id 2026-11-07-a \
  --harness ./eval \
  --model "<provider>/<model-id>" \
  --score 0.71 --n-tasks 120 \
  --tokens-used 187340 --token-budget 200000 \
  --out slips

python bench_slip.py check slips/2026-10-30-a.json slips/2026-11-07-a.json

The check command exits non-zero when the harness hash, model settings, or container limits moved. That makes "are these two numbers comparable?" a shell exit code instead of an argument between two people who each remember a different Tuesday.

3. Three roles, one ten-minute handoff

Keep the cast small, and rotate it so nobody becomes the only person who can read a slip.

Role	Owns	Must hand off
Run Owner	Executes the suite, writes the slip, attaches raw output	Slip path + raw log link to the Score Keeper
Score Keeper	Decides whether the number clears the quoting table	One-line verdict with the tier, posted with the slip
Quote Reviewer	Checks every slide, ticket, or post that repeats the number	Sign-off comment, or a rejection that names the missing field

The handoff is deliberately boring: Run Owner drops a slip path in the channel, Score Keeper replies with a tier from the table below, and Quote Reviewer either signs off or blocks. If a number appears anywhere without a verdict line, Quote Reviewer blocks it by default rather than chasing context, because missing context is exactly the failure this SOP exists to catch.

4. The 30-minute run, step by step

1. Freeze the harness. Tag or commit the eval directory, then record git rev-parse --short HEAD next to the slip. Never edit graders during a scoring run.
2. Restore the box profile. Recreate the same container limits you used last time; if you cannot, accept that this run starts a new lineage.
3. Warm up once. Run five tasks and discard them, so cold-start latency does not leak into timeouts.
4. Set the budget before the suite, not after. Pick --token-budget from last run's tokens_used plus a margin, and write it down.
5. Run the suite. Capture stdout and any retry log to a file; the slip references the path, not a pasted excerpt.
6. Publish the slip. bench_slip.py new writes JSON plus a Markdown block you can paste straight into the wiki.
7. Diff against the previous slip. Run check; if it exits 1, the verdict is "not comparable" and the discussion moves to which input changed.
8. Record the verdict. Score Keeper posts one line: tier, slip id, and the shortest honest caveat.

5. Can you quote this number?

This is the table the Score Keeper applies. It exists because most benchmark drama comes from numbers quoted at a higher confidence than their evidence supports.

Evidence on the slip	Tier	Who may see it	Required footnote
No slip, or check exits 1	Not quotable	Nobody outside the channel	none - do not quote
Slip exists, box differs from baseline	Internal only	Your team	"hardware profile changed"
Slip + matching box + budget under limit	External with caveats	Docs, blog posts	model id, harness hash, token budget
Two matching slips, same lineage	Quotable	Decks, customers	lineage id and both run ids
Contract, SLA, or pricing	Out of scope	-	use a pinned dedicated environment

Notice that the top row is the cheapest rule to enforce and the one that prevents the most embarrassment. A number with no slip is not a weak number; it is not a number yet.

6. Where the free path fits, and where it does not

Running this workflow on paid infrastructure for every iteration is wasteful, and that is the honest reason to start with a free tier instead. MonkeyCode, the open-source coding agent project, currently offers a free token grant of 10 million tokens and a free server option, per the team; terms, model availability, and quotas can change, so confirm the current details on the project page before you plan a sprint around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What that free path is genuinely good for is the cheap half of this SOP: warm-up runs, harness smoke tests, and the first slip of a new lineage, where you mainly want to prove the script produces a clean JSON before you spend anything real. Point --harness at your eval directory, run the five-task warm-up, and watch whether cgroup_cpu_max is populated at all on that machine, because an empty field is a fingerprint you cannot diff later. What it is not good for is the tier that says "quotable" in a customer deck, since a free box can be recycled or reshaped under you and the whole point of the slip is that the box is pinned.

If you want to try the harness end to end before committing budget, start on the free path and keep the slips even after you move to paid hardware, because the lineage is the valuable part.

Limitations and who should skip this

This SOP is deliberately small, and small tools fail in predictable ways. A directory hash will flag whitespace-only reformatting as drift, so expect some false alarms and treat them as cheap noise rather than reasons to loosen the hash. The box fingerprint reads Linux cgroup v2 files, so on macOS or Windows you get unavailable fields and should extend the script instead of pretending it worked. Single-run scores with 20 or 30 tasks are noisy enough that comparing them is close to reading tea leaves, which is why the table asks for two matching slips before anyone calls a number quotable.

Skip this entirely if your numbers feed a contract, an SLA, or a customer pricing decision, because those need a pinned dedicated environment and a real evaluation owner, not a wiki page and a shell script. Skip it too if you have no written eval harness yet, since a slip for an ad-hoc prompt is just bureaucracy with extra steps. For everyone else, one page, three roles, and thirty minutes per run is usually enough to stop the meeting from arguing about last week's hardware.

7. Paste this into the wiki

Benchmark Borrow Slip - one-pager. Every score that leaves this team carries a slip in slips/.
Roles: Run Owner (runs, writes slip), Score Keeper (assigns tier), Quote Reviewer (blocks unslipped numbers).
Run: freeze harness, restore box profile, warm up 5 tasks, set token budget, run suite, bench_slip.py new, bench_slip.py check against the previous slip, post one-line verdict.
Rule: no slip, no number. check exits 1, the numbers are not comparable, and the discussion moves to which field changed.
Out of scope: contracts, SLAs, pricing - those need a pinned dedicated environment.

MonkeyCode provides free models that can run this workflow.