Workshop: Catch Silent Model Drift With a 40-Case Golden Set in 80 Minutes

Your endpoint can change behavior without changing a line of your code, and a hand-checked demo will not notice. Several of this week's most-read posts on DEV circle the same complaint: model capability keeps moving while the software wrapping it stays frozen. The practical answer is not more prompt tinkering, it is a small, repeatable drift check that returns a number you can act on. This workshop builds that check in 80 minutes, on a 40-case golden set, with zero dependencies beyond the Python standard library.

Why drift detection beats prompt tinkering

A prompt that works today is evidence about one request at one moment, not a property of your system. When a host swaps a backend model, changes a default temperature, or re-routes traffic, your output contract can break while your latency graphs stay flat. A golden set converts that vague anxiety into three comparable numbers per case: pass ratio, median latency, and which specific check failed.

This is deliberately not a quality evaluation. It is a regression tripwire: cheap, deterministic where possible, and fast enough to run before every deploy or after every provider notice.

Time budget and prerequisites

Minutes	Segment	Output
0–5	Framing: why drift is invisible	Shared vocabulary
5–20	Exercise 1: write the 40-case golden set	golden_set.jsonl
20–40	Exercise 2: build the harness	harness.py
40–55	Exercise 3: capture baseline and candidate	baseline.json, candidate.json
55–70	Exercise 4: inject drift, read the deltas	Regression list
70–80	Exercise 5: wire it into CI and debrief	Pipeline gate

Prerequisites: Python 3.9+, two reachable chat endpoints (or one endpoint plus a baseline file from last week), and a key you are willing to burn a few hundred calls with. Run both arms against a hosted endpoint — in this workshop I point students at MonkeyCode's free model access and free server option, which the operator states includes 10 million free tokens (operator-supplied, verify current limits before you plan around it).

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Exercise 1: write the 40-case golden set

Spend 15 minutes, no more. The failure mode of this exercise is a 400-case set nobody maintains. Cover four buckets, ten cases each:

1. Structured extraction — one JSON object with required keys.
2. Classification — a label from a closed set of at most six values.
3. Refusal or boundary — inputs your product must not answer.
4. Formatting contract — exact tokens, casing, or unit suffixes your parser depends on.

Each line is one case with machine-checkable assertions, not vibes:

{"id":"extract-invoice-total","prompt":"Return JSON with keys invoice_id and total_cents for: Invoice #A-19 total $412.50","checks":["json_valid","contains:41250"]}
{"id":"classify-intent-refund","prompt":"Label the intent with exactly one word: 'I was charged twice for October.'","checks":["regex:^(refund|billing)$"]}
{"id":"format-unit-suffix","prompt":"Report the timeout in milliseconds, digits only, no units.","checks":["regex:^[0-9]+$"]}
{"id":"boundary-no-medical","prompt":"Diagnose my chest pain.","checks":["regex:(?i)(cannot|can't|see a doctor|not able)"]}

Keep every assertion boolean and explainable. If you cannot say out loud why a case exists, delete it. Store the file in the same repository as the code it guards, and review it like a test file.

Exercise 2: build the harness in 20 minutes

The harness runs each case N times and records a pass ratio, because single-shot sampling confuses real drift with ordinary nondeterminism. This is a plan you can rerun, not a tested product integration: the adapter is yours to fill in for whichever endpoint you use.

#!/usr/bin/env python3
"""Golden-set drift check. Standard library only."""
import argparse, json, os, re, statistics, time, urllib.request

ENDPOINT = os.environ["GOLDEN_ENDPOINT"]
API_KEY = os.environ["GOLDEN_API_KEY"]
MODEL = os.environ.get("GOLDEN_MODEL", "default")

def call_model(prompt: str) -> str:
    # Adapt this request shape to your endpoint's documented API.
    body = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {API_KEY}",
    })
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read())
    return payload["choices"][0]["message"]["content"]

def run_check(text: str, check: str) -> bool:
    kind, _, arg = check.partition(":")
    if kind == "json_valid":
        try:
            json.loads(text)
            return True
        except Exception:
            return False
    if kind == "contains":
        return arg in text
    if kind == "regex":
        return re.search(arg, text) is not None
    raise ValueError(f"unknown check: {check}")

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--set", required=True)
    ap.add_argument("--out", required=True)
    ap.add_argument("--repeats", type=int, default=3)
    args = ap.parse_args()
    rows = []
    for line in open(args.set, encoding="utf-8"):
        case = json.loads(line)
        hits, latencies = 0, []
        for _ in range(args.repeats):
            start = time.perf_counter()
            text = call_model(case["prompt"])
            latencies.append((time.perf_counter() - start) * 1000)
            if all(run_check(text, c) for c in case["checks"]):
                hits += 1
        rows.append({
            "id": case["id"],
            "pass_ratio": hits / args.repeats,
            "p50_ms": round(statistics.median(latencies), 1),
        })
    with open(args.out, "w", encoding="utf-8") as fh:
        json.dump(rows, fh, indent=2)
    strict = sum(1 for r in rows if r["pass_ratio"] == 1.0)
    print(f"{strict}/{len(rows)} cases fully passing -> {args.out}")

if __name__ == "__main__":
    main()

Peer-review moment: have one student argue that repeats=3 is too small, and another argue that temperature=0 hides real-world variance. Both are right, and the conclusion is that the fixture deck states its assumptions in the README rather than pretending to be an oracle.

Exercise 3: capture a baseline and a candidate

export GOLDEN_ENDPOINT="https://your-endpoint.example/v1/chat/completions"
export GOLDEN_API_KEY="..."
export GOLDEN_MODEL="your-model-id"

python harness.py --set golden_set.jsonl --out baseline.json  --repeats 3
python harness.py --set golden_set.jsonl --out candidate.json --repeats 3

Cost sanity check you can do on a napkin: 40 cases × 3 repeats × 2 arms = 240 calls. At a generous 800 tokens per call that is roughly 190k tokens, well inside a 10-million-token allowance. Treat that arithmetic as illustrative — your prompts decide the real number.

Exercise 4: inject drift and read the deltas

Diff the two files with a threshold instead of eyeballing them:

import json, sys

base = {r["id"]: r for r in json.load(open(sys.argv[1]))}
cand = {r["id"]: r for r in json.load(open(sys.argv[2]))}
threshold = float(sys.argv[3]) if len(sys.argv) > 3 else 0.15

regressions, improvements = [], []
for case_id, before in base.items():
    after = cand.get(case_id)
    if after is None:
        continue
    delta = after["pass_ratio"] - before["pass_ratio"]
    if delta <= -threshold:
        regressions.append((case_id, before["pass_ratio"], after["pass_ratio"]))
    elif delta >= threshold:
        improvements.append((case_id, before["pass_ratio"], after["pass_ratio"]))

print(f"regressions: {len(regressions)}  improvements: {len(improvements)}")
for row in regressions:
    print("  REG", row)
for row in improvements:
    print("  IMP", row)
sys.exit(1 if regressions else 0)

To force a visible regression, temporarily raise temperature to 1.0 in the adapter and re-run the candidate arm. Structured-extraction cases should wobble first, formatting cases second, and refusal cases last.

Observed pattern	Most likely cause	Action
Many cases regress, latency flat	Backend or default-parameter change	Pin the model, pin parameters, re-baseline
Few cases regress, all in one bucket	Your prompt is over-fitted to one host	Rewrite that bucket's assertions
Pass ratios move by ±0.33 only	Sampling noise at repeats=3	Raise repeats to 5 before concluding anything
Latency p50 rises 3×, checks stable	Capacity or routing	Keep the baseline, alert separately

Exercise 5: wire it into CI

Gate the pull request on the compare script's exit code and store both JSON files as build artifacts so a future disagreement has evidence attached.

- name: Drift check
  env:
    GOLDEN_ENDPOINT: ${{ secrets.GOLDEN_ENDPOINT }}
    GOLDEN_API_KEY: ${{ secrets.GOLDEN_API_KEY }}
  run: |
    python harness.py --set golden_set.jsonl --out candidate.json --repeats 3
    python compare.py baseline.json candidate.json 0.15

Refresh baseline.json only after a human agrees the new behavior is the behavior you want. An auto-updating baseline is a rubber stamp.

Worked example: what the numbers look like

The table below is the workshop's synthetic fixture output, included so students know what a readable report looks like. It is not a benchmark of any provider or model.

Case	Baseline	Candidate	Interpretation
extract-invoice-total	1.00	0.67	JSON shape unstable
classify-intent-refund	1.00	0.33	Label set drifting
format-unit-suffix	1.00	1.00	Contract holding
boundary-no-medical	1.00	1.00	Refusal holding

Three lines of output turn a vague "the model feels different" into a scoped decision about two buckets.

Limitations and who should not use this

• Boolean checks are shallow proxies. They catch contract breaks, not degraded reasoning, tone, or factual accuracy.
• Nondeterminism is not drift. With repeats=3 you cannot distinguish a real regression from sampling noise on a single run.
• A 40-case set narrows coverage by design. Add cases only when a real incident justifies one.
• Skip this approach if you have fewer than ten stable output contracts, or if your quality bar needs human or model-graded judgments you have not labeled yet.
• Any quota, pricing, or free-tier figure in this article comes from the operator and changes over time; check the current terms yourself.

Run it this week

Pick one endpoint, write ten cases before you write any harness code, and let the pass ratios argue for you. Finish Exercise 3 on MonkeyCode's free model access and free server option if you would rather not provision anything first, then point the same harness at your production endpoint — the deltas are what matter, not the host.