Build a Tiny Lab Receipt Before You Trust Model JSON

The two files sat on my desktop in Halifax like they were arguing. Same lab. Same split. Same wet Tuesday. One said "accuracy": 1.0. The other said "accuracy": 1. A third dump, the pretty one, added "notes": "held out val, looks solid".

If you diff those files as text, everything looks broken. If you paste them into a grade script, one of them may parse and one may not. So which one actually failed?

FAIL unknown keys: ['notes']

That is the expected output of this case study. Not a leaderboard. Not a new framework. A tiny Python receipt that treats model JSON as untrusted handwriting.

The learning question is narrow. When a helper model extracts numbers from a student lab, which differences are science, and which differences are punctuation wearing a lab coat?

I needed that question because I had started using a free model as a clerk. I would paste a training log, ask for JSON, and drop the result into a folder named results/. Two evenings later the folder looked like a filing cabinet. Lots of paper. No idea what was evidence.

You have probably done a version of this. The model is fast. The JSON looks official. Then your comparison script screams, and you spend twenty minutes debugging an accuracy number that never moved.

Background

This was Lab 3 in a CS course. Tiny classifier. CPU. I already had the real metrics in a handwritten note: run_id=lab3, split=val, accuracy=1, loss=0.25. Those numbers were boring on purpose. I wanted a fixture I could stare at, not a subplot from a contest notebook.

The goal was not “call an API.” The goal was a receipt. A receipt is a canonical string: required keys only, numbers normalized, keys sorted, no commentary. If two extracts produce the same receipt, the science did not change. If they do not, something real moved — or the clerk invented a field.

Think of it like comparing two photos of the same whiteboard. The handwriting can wiggle. The digits should not.

Prerequisites

You need Python 3.11 or newer and the standard library. No PyTorch in this writeup. No extra packages. I ran the commands below on Python 3.11.9. If python3 --version prints 3.10, the code still likely works, but I did not treat that as the fixture.

mkdir -p lab_receipt/fixtures
cd lab_receipt

The receipt

Here is lab_receipt.py. It is the whole project.

#!/usr/bin/env python3
"""Canonical lab receipt for untrusted model JSON."""
from __future__ import annotations

import json
import sys
from decimal import Decimal, InvalidOperation
from pathlib import Path

REQUIRED = ("run_id", "split", "accuracy", "loss")
NUMBER_KEYS = frozenset({"accuracy", "loss"})


def normalize_number(value: object) -> int | float:
    if isinstance(value, bool) or not isinstance(value, (int, float, str)):
        raise TypeError(f"not a number: {value!r}")
    try:
        amount = Decimal(str(value).strip())
    except (InvalidOperation, ValueError) as exc:
        raise TypeError(f"not a number: {value!r}") from exc
    integral = amount.to_integral_value()
    if amount == integral:
        return int(integral)
    return float(amount.quantize(Decimal("0.000001")))


def to_receipt(payload: dict) -> str:
    if not isinstance(payload, dict):
        raise TypeError("payload must be an object")
    missing = [key for key in REQUIRED if key not in payload]
    extra = sorted(set(payload) - set(REQUIRED))
    if missing:
        raise ValueError(f"missing keys: {missing}")
    if extra:
        raise ValueError(f"unknown keys: {extra}")
    clean: dict[str, object] = {}
    for key in REQUIRED:
        value = payload[key]
        if key in NUMBER_KEYS:
            clean[key] = normalize_number(value)
            continue
        if not isinstance(value, str) or not value.strip():
            raise ValueError(f"{key} must be a non-empty string")
        clean[key] = value.strip()
    return json.dumps(clean, sort_keys=True, separators=(",", ":"))


def load_json(path: Path) -> dict:
    with path.open(encoding="utf-8") as handle:
        data = json.load(handle)
    if not isinstance(data, dict):
        raise TypeError(f"{path} must contain a JSON object")
    return data


def main(argv: list[str]) -> int:
    if len(argv) != 3:
        print("usage: python lab_receipt.py <left.json> <right.json>", file=sys.stderr)
        return 2
    left_path = Path(argv[1])
    right_path = Path(argv[2])
    try:
        left = to_receipt(load_json(left_path))
        right = to_receipt(load_json(right_path))
    except (OSError, json.JSONDecodeError, TypeError, ValueError) as exc:
        print(f"FAIL {exc}")
        return 1
    if left == right:
        print("MATCH")
        print(left)
        return 0
    print("DRIFT")
    print(f"left:  {left}")
    print(f"right: {right}")
    return 1


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Read to_receipt once. Unknown keys are a hard error. I used to strip extras silently. That is how notes becomes a second source of truth you never agreed to measure.

Fixtures

Write three files. Do not pretty-print them in a way that hides the point.

fixtures/run_a.json:

{"run_id": "lab3", "split": "val", "accuracy": 1.0, "loss": 0.25}

fixtures/run_b.json:

{"loss": 0.250000, "accuracy": 1, "split": "val", "run_id": "lab3"}

fixtures/pretty_extra.json:

{
  "run_id": "lab3",
  "split": "val",
  "accuracy": 1.0,
  "loss": 0.25,
  "notes": "held out val, looks solid"
}

Same science in all three? Your eyes say yes. The receipt says no for the third.

What I ran

python lab_receipt.py fixtures/run_a.json fixtures/run_b.json

Expected output:

MATCH
{"accuracy":1,"loss":0.25,"run_id":"lab3","split":"val"}

Notice 1.0 became 1. Notice key order in the files did not matter. The receipt is the thing you store, not the model’s original string.

Now the failing fixture. Predict it before you run it. Which line explodes, and why?

python lab_receipt.py fixtures/run_a.json fixtures/pretty_extra.json

Expected output:

FAIL unknown keys: ['notes']

That notes field is the filing cabinet problem in miniature. It is not false. It is worse: it is unofficial true. Tomorrow you will compare two runs and your diff will light up because one clerk felt chatty.

One more error input, because strings love to wear number costumes.

fixtures/string_acc.json:

{"run_id": "lab3", "split": "val", "accuracy": "about 1", "loss": 0.25}

python lab_receipt.py fixtures/run_a.json fixtures/string_acc.json

Expected output:

FAIL not a number: 'about 1'

If your helper model hedges, the receipt must refuse. A hedge is not a metric.

Results

I pointed the same locker at a handful of extracts I had already saved. The string diffs were noisy. After canonicalization, two pairs MATCHED that I had marked as “accuracy changed.” They had not. One was 0.25 versus 0.250000. One was key order. I had been about to rerun a training job to chase a ghost.

The pretty file failed immediately. Good. I would rather a red FAIL than a notebook that grows a shadow schema every night.

I still needed somewhere to draft the clerk step and somewhere to park the comparison so I was not rerunning it from a library Wi-Fi session. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access to propose candidate JSON from logs, and the free server option to run lab_receipt.py against a folder of fixtures. The model never wrote the receipt. It only proposed a payload. If the payload grew a notes key, the job printed FAIL and I kept the handwritten lab numbers.

If you already have a cron box, you do not need that path. The script does not care where it runs.

What actually broke

The concept that moved in my head was not “JSON is picky.” I already knew that. The concept was contract versus prose. A model is a prose engine. A lab result is a contract. If you let prose leak into the contract, your future self will debug literature.

Common mistake: calling json.loads and then == on dicts. In Python, { "accuracy": 1 } and { "accuracy": 1.0 } are not equal. Try it.

python -c 'print({"accuracy": 1} == {"accuracy": 1.0})'

Expected output:

False

That single False is why students rerun jobs. Another mistake: stripping unknown keys to be “robust.” Robust to what? To the clerk inventing a grade column named notes? A third mistake: rounding everything to two decimals and then wondering why two genuine runs collapsed into a MATCH. The quantize step is a choice. I used six places because Lab 3 never needed more. If you are logging a loss that lives at 1e-7, this receipt will lie to you. Change the quantum, or do not canonicalize that field.

Python’s own json.dumps(..., sort_keys=True, separators=(",", ":")) is the boring primary tool here. I did not invent a new interchange format. I just stopped treating the model’s pretty printer as a lab notebook. If you want the heavier standard later, RFC 8785 describes JSON Canonicalization Scheme. This homework locker does not implement that RFC. It implements four keys and a bad attitude toward extras.

Limitations

Do not use this as a judge for generated code quality. It will not tell you whether the model cited a real paper. It will not tell you whether accuracy=1 is overfitting on eight training rows — and in a homework-sized set, it probably is. It will not survive a schema that needs nested objects unless you extend it. It is the wrong tool if you need cryptographic provenance, a production feature store, or a grader that accepts human comments in-band.

Also: I am a student documenting a lab habit, not a vendor benchmark. I am not publishing latency numbers, token ceilings, or hardware claims. Those go stale by lunch.

After this, you should be able to explain

You should be able to point at two JSON blobs and say, out loud, whether a diff is science or spelling. You should be able to name the required keys of your own lab before you invite a model to fill them. You should treat extra keys as failures, not as bonuses. And you should store the receipt string, because that is the only object your later script should hash.

Add a unit lock if you want the extension. Make accuracy require a sibling accuracy_unit whose only legal value is "ratio". Then feed a payload with "accuracy": 91 and no unit. If your receipt still MATCHES the 1 fixture, the locker is too polite.

If you try it, tell me which fixture failed first. I care less about the MATCH cases. The interesting night is the one where the pretty file looks kinder than the truth.