Replay Cassettes Separate Harness Drift from Model Drift

A prompt suite has three moving parts — prompt text, the grader, and the model snapshot — and a single pass-rate number moves whenever any one of them changes. Replay cassettes remove that ambiguity by freezing model output on disk and letting you re-run only the parts you actually edited. The rule that makes it work is narrow: the network appears in exactly one code path, the recording path, and every other path either reads a hash-addressed file or fails loudly. Everything below is a harness you can build in an afternoon, plus the cases where it will lie to you.

The cassette is a recording keyed by input, not a mock

Most teams reach for mocks first: hand-written fake responses that look plausible and drift away from reality within a week. A cassette is different because it is produced by a real call and addressed by a fingerprint of the exact input that produced it. Change one word of the prompt and the fingerprint changes, so the old tape cannot silently answer a new question. That single property is what turns a cache into a regression detector.

# cassette.py
import hashlib, json, os, time

DIR = os.environ.get("CASSETTE_DIR", "cassettes")

class CassetteMiss(Exception):
    """Raised in replay mode when no tape matches the input fingerprint."""

def fingerprint(prompt, params):
    blob = json.dumps({"prompt": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]

def get(prompt, params, produce, mode="record"):
    key = fingerprint(prompt, params)
    path = os.path.join(DIR, key + ".json")
    if os.path.exists(path):
        with open(path) as fh:
            return json.load(fh), True          # hit
    if mode == "replay":
        raise CassetteMiss(key)                 # never fall through to network
    tape = produce(prompt, params)
    tape.update({"key": key, "params": params,
                 "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
    os.makedirs(DIR, exist_ok=True)
    with open(path, "w") as fh:
        json.dump(tape, fh, indent=2, sort_keys=True)
    return tape, False                          # miss, now recorded

The params argument deserves a moment, because it is where people quietly break determinism. Temperature, seed, max tokens, system prompt, and any tool-selection flag all belong in the fingerprint, even when the provider claims they are no-ops. If a knob can change the text, it belongs in the key, and if it cannot change the text, storing it costs you nothing but a few bytes.

Recording is the only place the network lives

The runner below reads a JSONL case file, asks the cassette layer for a tape, and hands the recorded text to a named grader. Note that graders are looked up by name from a registry rather than stored in JSON, because functions do not serialize and a grader you cannot diff is a grader you cannot audit. Note also that a replay miss exits with code 2, which is a different signal from a failing assertion and should page a different person.

# harness/run.py
import json, os, sys
from cassette import get, CassetteMiss
from client import call_model          # the only module that opens a socket

MODE = os.environ.get("CASSETTE_MODE", "replay")
GRADERS = {
    "json_shape": lambda text, exp: _parses(text) and set(exp["keys"]) <= set(_parses(text)),
    "contains":   lambda text, exp: exp["needle"].lower() in text.lower(),
    "refusal":    lambda text, exp: text.strip().lower().startswith(exp["prefix"]),
}

def run(suite_path):
    hits = misses = 0
    for line in open(suite_path):
        if not line.strip():
            continue
        case = json.loads(line)
        try:
            tape, hit = get(case["prompt"], case.get("params", {}),
                            produce=call_model, mode=MODE)
        except CassetteMiss as exc:
            print("CASSETTE_MISS", exc, file=sys.stderr)
            return 2
        hits += hit
        misses += not hit
        verdict = GRADERS[case["grader"]](tape["text"], case.get("expect", {}))
        print(json.dumps({"id": case["id"], "hit": hit, "verdict": verdict,
                          "finish": tape.get("finish_reason")}))
    total = hits + misses
    print(f"hit_rate={hits/total:.2f}", file=sys.stderr)
    return 0 if hits + misses else 1

if __name__ == "__main__":
    sys.exit(run(sys.argv[1]))

Run it twice to see the whole point. The first invocation with CASSETTE_MODE=record pays for real calls and writes tapes; the second with CASSETTE_MODE=replay finishes in milliseconds and provably makes zero network requests. If your CI cannot run the replay pass with networking disabled and still pass, you have not actually separated harness changes from model changes, and the pass rate remains a single undifferentiated number.

$ CASSETTE_MODE=record python -m harness.run cases/canary.jsonl   # needs a model
$ CASSETTE_MODE=replay python -m harness.run cases/golden.jsonl   # offline, exit 2 on miss
$ git diff --stat cassettes/ | tail -1                            # which tapes moved

Spend paid or free capacity on recording, never on replays

Recording is the expensive half of this design: every new case and every prompt edit produces a cache miss that needs a real completion, while replay needs nothing at all. That asymmetry maps cleanly onto MonkeyCode's free model access and free server option, which is where this workflow touches the product. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The practical split is to run the recording job on the free server instance against the free model tier, and keep the replay job on the machine that runs your test suite.

Keep the calibration honest, because free tiers are promotional by nature and change without notice. Check the project page for the current model list, quota, and server specification instead of hardcoding assumptions from a blog post, and treat the cassette directory as the durable asset rather than the provider. If a free model disappears, your tapes still replay, and only the refresh job needs new wiring.

Canaries belong at the boundary, not in the middle

Uniform sampling wastes most of its budget on cases that cannot flip: a case scoring 0.9 against a 0.5 threshold will survive almost any prompt edit. The cases that move are the ones sitting near the decision boundary, so select canaries by margin rather than by category. This is a five-line function and it usually cuts the recording cost of a canary suite by an order of magnitude.

def boundary_band(rows, band=0.10):
    """rows: [{"id", "score", "threshold"}]; keep cases that can actually flip."""
    return [r for r in rows if abs(r["score"] - r["threshold"]) <= band]

Run the band selection against a fresh sample, record those cases, and then freeze them. When a prompt edit lands, replay the frozen band first; if nothing flips there, the cheap regression check is done and the full suite can wait for nightly. If something does flip, you now have a cassette from before and after, which is a two-file diff rather than an argument.

A decision table for when replay is the wrong tool

Question you are asking	Cassette replay	Fresh sampling
Did my prompt or grader edit break a known case?	answers it	too noisy
Is the new model snapshot better overall?	cannot answer	correct tool
Is latency or cost per call acceptable?	invalidated by design	correct tool
Can this run offline in CI on every push?	yes	no

What replay cannot tell you, and who should skip it

Replay proves that your harness is stable against a fixed set of outputs, and it proves nothing about the current model, which may have changed the moment you stopped recording. Latency, token accounting, and refusal behaviour under load all disappear the instant a tape replaces a socket, so never cite a replay run as a performance measurement. The grader remains the weakest link, and a cassette suite will happily certify a grader that has quietly stopped agreeing with human labels.

Skip this approach if you do not yet have a stable case set, because cassette infrastructure amplifies a bad suite rather than fixing it. Skip it if your only real question is whether a newer model is better, since that question needs fresh samples and paired comparisons rather than frozen text. And skip it in any setting where you must prove provenance of each individual output, because a replayed tape is a cached artifact and not a fresh decision.

If you want to try the recording half without provisioning anything, MonkeyCode's free model access and free server option are enough to record a first band of a few hundred boundary cases; commit the cassettes/ directory alongside the harness, and let the tapes accumulate as the cheapest regression evidence your team owns.

Comments

[Zhang Enquan]

You're drawing the right line, and I ran your experiment in reverse by accident last week.

I tokenized 10,000 real prompts, then changed the harness on purpose: swapping the tokenizer (cl100k to o200k) moved the median token count by -0.54% and the batch total by -0.97%. Changing nothing about the harness and switching content type instead - prose 4.58 chars per token vs structured docs 3.78 - moved it 18%. So one harness change was noise, and one "nothing important changed" swap was the largest single variable in the data. Nobody would guess that split from the pass rate, which is your whole argument: without the tape you can't tell which kind of change you're looking at.

One field to add to the fingerprint, from that same exercise: tokenizer revision alongside model. Same prompt, same temperature, same seed - and a silent vocabulary update changes where truncation lands and what a stop sequence costs, while the sha256 of the fingerprint input stays byte-identical, so the old cassette answers happily and the diff never gets attributed. Model + tokenizer revision in the key is one line, and right now it's the only drift your harness can't see.