Detect Silent Model Drift With a Stdlib-Only Golden Set Harness

A hosted model can change between two of your deploys, and nothing in your lockfile moves. Your unit tests still pass, your container still builds, and your users still get slightly different JSON than they got last Tuesday. The practical fix is not a bigger test suite; it is a frozen golden set, a deterministic scorer, and an append-only ledger that fails the run when the pass rate drops below a floor.

This article builds exactly that, using only the Python standard library. It runs on a free-tier host, it costs one file of input data, and it answers a single narrow question: did the contract my product depends on stop holding?

Why ordinary tests miss model drift

Application tests assert on code you control. A model response is an input from outside your repository, and it arrives as free text that happens to be parseable.

When a provider adjusts decoding defaults, swaps capacity, or routes you to a different checkpoint, the shape of the output can shift while the API status code stays 200. Nothing throws. The failure surfaces later, as a support message about a field that used to be filled in.

A golden set inverts that. Instead of asserting on the current behavior, it asserts on a small number of hand-checked examples that were correct on a specific day, and it keeps the receipts.

Step 1: Freeze a small golden set as JSONL

The input file is plain JSONL, one contract per line. Keep it small on purpose: 20 to 40 cases is enough to catch a formatting or label flip, and it fits inside a free-tier request budget.

{"id":"invoice_date","input":"Invoice paid 2026-03-04, net 30 days.","expect":{"label":"invoice","fields":{"due_date":"2026-04-03"}}}
{"id":"refund_request","input":"I want my money back for order 8812, it never arrived.","expect":{"label":"refund","fields":{"order_id":"8812"}}}
{"id":"no_action","input":"Thanks, the docs answered my question.","expect":{"label":"none","fields":{}}}

Every line has three parts: an id you will grep for in the ledger, the raw input you send, and the expect object you compare against. Do not paste generated examples into this file. If you did not read the output and agree with it, it does not belong in the baseline.

Step 2: Write a transport adapter that assumes nothing

The runner below assumes an OpenAI-style /chat/completions request body. That assumption is isolated in one function so you can replace it when your endpoint differs.

#!/usr/bin/env python3
"""golden_run.py - regression run against a chat endpoint. Stdlib only."""
import json, os, sys, time, hashlib, pathlib, urllib.request

BASE = os.environ["MODEL_BASE_URL"].rstrip("/")
KEY = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_ID"]
GOLDEN = pathlib.Path(os.environ.get("GOLDEN", "golden.jsonl"))
LEDGER = pathlib.Path(os.environ.get("LEDGER", "ledger.jsonl"))

SYSTEM = (
    "You are a strict extractor. Return ONLY minified JSON shaped as "
    '{"label": string, "fields": object}. Omit absent fields. '
    "No prose, no markdown fences."
)

def call(text, timeout=30):
    body = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [{"role": "system", "content": SYSTEM},
                     {"role": "user", "content": text}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=timeout) as r:
        payload = json.loads(r.read())
    return payload["choices"][0]["message"]["content"]

Keep temperature at zero when the endpoint supports it. If the parameter is rejected or ignored, note that in your README rather than deleting it, because a later comparison depends on knowing what you actually sent.

Step 3: Score the contract, not the prose

Scoring is exact after light normalization: trim, lowercase, collapse whitespace. That is deliberately unforgiving, because the goal is to detect change, not to grade writing.

def norm(v):
    return " ".join(v.strip().lower().split()) if isinstance(v, str) else v

def score(case, raw):
    try:
        got = json.loads(raw)
    except json.JSONDecodeError:
        return {"case": case["id"], "ok": False, "why": "invalid_json"}
    want = case["expect"]
    if norm(got.get("label")) != norm(want["label"]):
        return {"case": case["id"], "ok": False, "why": "label",
                "got": got.get("label"), "want": want["label"]}
    fields, gotf = want.get("fields", {}), (got.get("fields") or {})
    for k, v in fields.items():
        if norm(gotf.get(k)) != norm(v):
            return {"case": case["id"], "ok": False, "why": f"field:{k}",
                    "got": gotf.get(k), "want": v}
    return {"case": case["id"], "ok": True, "why": "pass"}

Two failure reasons are distinguishable here and that distinction is the whole point: invalid_json is a format drift, label or field:* is a content drift.

Step 4: Append every run to a ledger and exit non-zero

The ledger is one JSON object per line, append-only, with a run_id derived from the system prompt, model id, and base URL. Change any of those and the id changes, which keeps old and new comparisons honest.

def run():
    run_id = hashlib.sha256(
        (SYSTEM + MODEL + BASE).encode()).hexdigest()[:12]
    rows = []
    for line in GOLDEN.read_text().splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        case = json.loads(line)
        raw, err = "", None
        for attempt in range(3):
            try:
                raw = call(case["input"])
                break
            except Exception as exc:      # timeout, 429, DNS, 5xx
                err, raw = repr(exc), ""
                time.sleep(2 ** attempt)
        if not raw:
            rows.append({"case": case["id"], "ok": False,
                         "why": "transport", "err": err})
            continue
        r = score(case, raw)
        r["raw"] = raw[:400]
        rows.append(r)
    total, passed = len(rows), sum(r["ok"] for r in rows)
    rate = round(passed / total, 3) if total else 0.0
    record = {"run_id": run_id, "ts": int(time.time()), "total": total,
              "passed": passed, "rate": rate, "rows": rows}
    with LEDGER.open("a") as fh:
        fh.write(json.dumps(record) + "\n")
    print(f"{passed}/{total} ({rate:.0%}) run_id={run_id}")
    floor = float(os.environ.get("PASS_FLOOR", "0.9"))
    return 0 if total and rate >= floor else 1

if __name__ == "__main__":
    sys.exit(run())

Exit codes matter more than dashboards when you are alone. Non-zero from this script can page you, block a deploy, or simply stop a scheduled job from marking the day green.

Step 5: Run it on a schedule without paying for a scheduler

On a free-tier host, cron is the cheapest scheduler you have. Secrets stay in a .env file that the job sources, so the crontab itself contains no keys.

# /srv/app/run-golden.sh
set -euo pipefail
cd /srv/app
set -a; . ./.env; set +a
python3 golden_run.py >> /var/log/golden.log 2>&1

# off-peak, once a day; adjust to the hour your host is quietest
17 4 * * * /srv/app/run-golden.sh

Compare the last two runs with one command. The jq filter prints only the cases that flipped.

tail -n 2 ledger.jsonl \
  | jq -c '{run_id, ts, rate, fails: [.rows[] | select(.ok == false) | .case]}'

Free-tier containers often sleep between requests. A 30-second timeout and a three-attempt backoff absorb most cold-start noise, and the transport reason keeps that noise out of your content-drift count.

Reading the ledger: a decision table

Signal in the ledger	Likely cause	First action
Many transport, few content failures	throttling, cold start, or a slow window	Move the job off-peak, lower case count, raise timeout
Sudden invalid_json on every case	response formatting default changed	Re-check the adapter and system prompt before blaming the model
One field fails on most cases	real content change	Pin a previous model id if one is offered, or add a normalization step
One case fails, everything else passes	a genuinely hard example	Re-read the expected value; the baseline may be wrong

A single bad run is not a verdict. Two consecutive runs with the same failure set is the threshold worth acting on.

Limitations and who should skip this

Exact-match scoring is brittle by design and wrong for open-ended generation. Do not point this harness at a summarizer, a chat assistant, or anything whose value is in the phrasing.

A 20-to-40 case set cannot detect rare failures, and it gives no statistical confidence. Teams that need eval coverage, graders, or per-release reports should use a real evaluation platform instead.

The adapter is single-turn. Agents that call tools across multiple round trips need a different harness, or at least a recorded-transcript replay layer in front of this one.

Finally, the runtime budget is real. Free-tier quotas, sleep windows, and timeouts change; the ledger tells you when they did, but it will not make them go away.

Where MonkeyCode fits in this loop

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness is endpoint-agnostic, which is the point. In my own loop, MonkeyCode is the endpoint behind MODEL_BASE_URL and the host that runs the cron job, because its free model access and free server option keep a nightly regression run off the invoice while the project is still pre-revenue. The operator describes the free tier as including a substantial token allowance — on the order of ten million tokens — plus a free server option; verify the current terms on the project page before you size a golden set against it, since quotas and limits move faster than articles do. The project is open source, so the transport adapter is the only part you may need to touch.

If the loop is useful, the smallest honest start is one line in golden.jsonl: take a real failure your users already reported, freeze the correct output, and let the ledger watch it for you.