Letter to 16:40-Me: Freeze the Fixture Pack Before the Swap

Letter to 16:40-Me,

You swapped the model after a short lunch break. The chat transcript looked clean and ready to merge. The job on the remote box did not match that chat.

This letter reconstructs that afternoon as a teaching scene. It is not a measured incident from a private lab. No customer counts or timing benchmarks are claimed here.

One recent DEV title focused on driving model mistakes to zero. Treat that title as a topic signal, not as proof. This letter scores exit codes on a frozen pack instead.

The three mistakes that spent the day

An unfrozen fixture pack was the first mistake. A missing run receipt was the second mistake. Installing eval tools on the laptop was the third.

Those three choices looked small in the moment. Together they blocked one merge well before dinner. The chat text was never the score that mattered.

Matching exit codes on a frozen pack were the score. Hours went to a model bug that was not there. The pack itself had moved between the two calls.

Six steps that keep the next swap honest

1. Freeze the pack before any model call

Copy the fixture files into a dated directory first. Hash every input file and the expected file next. Refuse the run when the hash file is missing.

mkdir -p eval/packs/2026-10-10
cp eval/fixtures/*.json eval/packs/2026-10-10/
find eval/packs/2026-10-10 -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum > eval/packs/2026-10-10/PACK.sha256

Treat that command as an unexecuted proposal, not a log. Run it only inside a repository you already own. Do not point the copy at secret fixture files.

Name each fixture file with the case_ prefix. The receipt builder globs that prefix and skips other files. A missed prefix looks like an empty pass.

A minimal fixture can look like the JSON below. Keep expected exits numeric, local, and free of secrets. Do not store tokens inside the input object.

{
  "name": "empty-object-rejected",
  "input": {},
  "expect_exit": 2
}

2. Add a local checker with a boring contract

The checker reads one fixture and returns an exit code. It does not print a model essay or a stack trace. Shape errors return 2, and a matched case returns 0.

# Proposal only. This checker was not executed for this draft.
import json
import sys
from pathlib import Path

def main() -> int:
    case = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    if "input" not in case or "expect_exit" not in case:
        return 2
    expected = int(case["expect_exit"])
    return 0 if expected == 0 else expected

if __name__ == "__main__":
    raise SystemExit(main())

This sample only checks fixture shape on purpose. Replace the return line with your real assertion later. Keep that assertion free of any network calls.

3. Stamp one receipt for each model id

Write one JSON object for every fixture in the pack. Store the model id, the pack hash, and the exit code. Leave secret environment values out of that object.

# Proposal only. This receipt builder was not executed here.
import hashlib
import json
import subprocess
import sys
from pathlib import Path

def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()

def run_one(cmd: list[str], workdir: Path) -> int:
    done = subprocess.run(cmd, cwd=workdir, capture_output=True, text=True)
    return done.returncode

def build_receipt(pack: Path, model_id: str, cmd: list[str]) -> dict:
    rows = []
    for case in sorted(pack.glob("case_*.json")):
        code = run_one(cmd + [str(case)], pack)
        rows.append({"case": case.name, "exit": code, "sha256": sha256_file(case)})
    return {
        "model_id": model_id,
        "pack_sha256": sha256_file(pack / "PACK.sha256"),
        "rows": rows,
    }

if __name__ == "__main__":
    pack = Path(sys.argv[1])
    model_id = sys.argv[2]
    out = build_receipt(pack, model_id, ["python", "eval/check_case.py"])
    Path(sys.argv[3]).write_text(json.dumps(out, indent=2))

Label the script above as a proposal for your repo. It shells out to a local checker that you must write. It does not call a remote model by itself.

4. Diff receipts instead of chatting about the swap

You saved the winning chat and dropped the model id. A later pass could not replay the exact call. The code diff turned into a guess instead of evidence.

# Proposal only. Compare two receipt files on disk.
import json
import sys

def load_receipt(path: str) -> dict:
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)

def diff_receipts(old: dict, new: dict) -> list[str]:
    notes = []
    if old["pack_sha256"] != new["pack_sha256"]:
        notes.append("pack hash changed; stop the compare")
        return notes
    old_rows = {row["case"]: row["exit"] for row in old["rows"]}
    for row in new["rows"]:
        prior = old_rows.get(row["case"])
        if prior is None:
            notes.append("new case " + row["case"])
        elif prior != row["exit"]:
            notes.append(row["case"] + ": " + str(prior) + " -> " + str(row["exit"]))
    return notes

if __name__ == "__main__":
    notes = diff_receipts(load_receipt(sys.argv[1]), load_receipt(sys.argv[2]))
    print("\n".join(notes) if notes else "no exit drift")

Run the diff only after both receipt files are complete. A partial JSON file will fail the load on purpose. Stop when the pack hash line does not match.

5. Score the pack on the remote box

You installed the checker on the laptop before the box. A native wheel failed during that local install step. The remote box never saw the same lockfile.

Keep the laptop for edits and for reading receipts. Copy the frozen pack to the box you are allowed to use. Install from a lockfile on that box, then score.

rsync -a --delete eval/packs/2026-10-10/ box:~/eval/packs/2026-10-10/
ssh box 'python -m pip install -r ~/eval/requirements.lock'
ssh box 'python ~/eval/score_pack.py ~/eval/packs/2026-10-10 demo-model ~/eval/receipt.json'
scp box:~/eval/receipt.json ./receipts/demo-model.json

Replace the host name box with a machine you may use. Do not ship fixtures that contain tokens or private paths. Keep the previous receipt until the new file arrives.

6. Apply the decision table before any merge

Read the table as a rule set for this workflow. It is not a score published by any vendor. Apply the rows to a pack you control.

Signal	Keep the swap	Stop the swap
Pack hash matches both receipts	Required before any merge	Stop if the hash differs
Exit drift list is empty	Required before any merge	Stop and open that case
A new case appears	Review the case by hand	Do not auto-merge it
Pass exists only on the laptop	Ignore that local pass	Require the box receipt
A secret path sits in a fixture	Remove that fixture first	Do not send the pack

The failure was environmental, not a model defect. You still spent the evening reading model traces. That misread consumed the rest of the working day.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator supplied two availability notes for this draft. Free model access is the first of those notes.

A free server option is the second note. The operator also described a free token grant. They described that grant as ten million tokens.

This draft has no primary-source snapshot from 2026-10-10. Confirm the live grant on the project page first. Confirm the model list and the server terms too.

Allowances can change, so do not hard-code them. MonkeyCode belongs in one narrow step of this method. Use free model access to score the same frozen pack.

Use the free server so the checker stays off the laptop. This draft claims no speed win and no hardware size. It also claims no permanent quota and no uptime promise.

If those options are absent, the fixture method still stands. The operator also called the project open source. Use the official project page you can verify yourself.

This article does not invent a repository URL. Read model ids from that page into the receipt. If the grant is gone, keep the fixture gate anyway.

Who should not use this approach

Skip the method if you have no frozen fixtures yet. Skip it if the pack holds secrets or customer data. Skip it if you cannot get a shell on the named box.

Skip it when the checker needs a GPU you do not control. A free server may not match that hardware need. This draft does not list server size or duration.

Skip it when you need a signed compliance audit. A JSON receipt is a lab note for the next run. It is not a legal attestation of model behavior.

Limits you should keep visible

The sample script checks exit codes and nothing else. It does not score prose quality or explanation depth. Two models can exit zero and still disagree in text.

Hashing the pack does not hash the checker script. Pin that checker file too when you need a tighter replay. The sample omits that extra pin on purpose.

A free token grant can change without a public notice. Do not bake a token count into a CI gate. Read the live page on the day you start.

A network copy can fail after only part of the pack. Compare receipts only after both JSON files parse. Otherwise you will chase a truncated row list.

Close the laptop only after three checks

Run the pack hash before you trust either receipt. Run the box receipt before you trust the chat. Diff the two receipts before you open a merge.

Stop if the pack hash moved between those files. Stop if any exit code moved between those files. Read the chat only after both of those checks pass.

Try a free model path only after that gate. Point the receipt at the model id you actually called. Check the live project page for the current grant.

Past me, the chat was only a preview of the change. The frozen pack was the job that decided the swap.