48-Hour Field Notes: My Replay Cassette Kept Passing While the Live Call Failed

My test suite was green for eleven hours while the same code failed in production, and that contradiction is the whole story here. I had built a record-and-replay harness for LLM-backed functions so my tests would stop depending on a live call. What I did not anticipate was that the harness itself would become the most convincing liar in the repository. How do you debug a system whose fixtures were generated by the bug you are hunting?

This is the third entry in my running field-notes series on failures that only make sense after you look at the environment, not the source file. Previous entries covered exit codes, percentiles, and cache invalidation. This one is about determinism as a debugging tool, and about the specific way determinism can betray you when you record it too early.

What I was actually trying to fix

The problem was mundane: every push ran a handful of tests that called a model and asserted something about the structured output. Those tests were slow, occasionally flaky, and impossible to run on a plane. I wanted a cassette layer, in the spirit of HTTP recording libraries, where the first run captures a request and response pair and later runs read from disk.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used its free model access to record cassettes without watching a spend meter, and its free server option to run the replay suite somewhere other than my laptop.

That is the entire role the product played in this story: recording and running. Everything below is ordinary Python, and none of it depends on which provider answered the call.

Hours 0–8: the recorder

The first version keyed each cassette on a canonical hash of model tag, prompt, and parameters. I deliberately excluded wall-clock timestamps and request IDs from the key, because including them makes every key unique and turns your cache into a log file.

import hashlib, json
from pathlib import Path

CASSETTE_DIR = Path("cassettes")

def call_key(model_tag: str, prompt: str, params: dict) -> str:
    canonical = json.dumps(
        {"model_tag": model_tag, "prompt": prompt, "params": params},
        sort_keys=True, separators=(",", ":"),
    )
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16]


def record_or_replay(client, model_tag, prompt, params, mode="replay"):
    path = CASSETTE_DIR / f"{call_key(model_tag, prompt, params)}.json"
    if mode == "replay":
        if not path.exists():
            raise FileNotFoundError(f"missing cassette {path.name}; record it first")
        return json.loads(path.read_text())["response"]

    response = client.complete(model_tag=model_tag, prompt=prompt, params=params)
    CASSETTE_DIR.mkdir(exist_ok=True)
    path.write_text(json.dumps(
        {"request": {"model_tag": model_tag, "prompt": prompt, "params": params},
         "response": response},
        indent=2,
    ))
    return response

Two decisions in that function matter more than they look. First, sampling parameters belong in the key, because a cassette recorded at one temperature is not the same fixture at another. Second, any field you cannot reproduce must be normalized or dropped before hashing, or replay quietly becomes re-record.

The normalization trap

I initially hashed params straight from a dict that included a caller-supplied request_id. Every replay missed, every run recorded a fresh cassette, and the suite got slower instead of faster. The fix was a small normalize() step that strips transport-only fields, and a rule I now enforce in review: if it can change without changing the answer, it does not belong in the key.

Hours 8–22: validation, and where the nesting bug hid

While recording, I added a validator so I would not freeze garbage into fixtures. My first attempt was honest but shallow, and shallow is exactly the wrong shape here.

# the version that let the bug through
def validate_loose(payload: str) -> dict:
    doc = json.loads(payload)
    assert isinstance(doc.get("items"), list) and doc["items"]
    return doc

That check confirms the outer container exists and is non-empty, and nothing else. The model occasionally returned each item's tags as a comma-joined string instead of an array. Downstream, for tag in item["tags"] happily iterated characters, produced nonsense labels, and never raised. Top-level validation said yes; the field two levels down said no.

The corrected version pins the shape all the way down and refuses extra keys, so a new field is a deliberate schema change rather than silent drift.

import jsonschema

SCHEMA = {
    "type": "object",
    "required": ["items"],
    "additionalProperties": False,
    "properties": {
        "items": {
            "type": "array", "minItems": 1,
            "items": {
                "type": "object",
                "required": ["id", "tags"],
                "additionalProperties": False,
                "properties": {
                    "id": {"type": "string", "pattern": "^[a-z0-9_-]{1,40}$"},
                    "tags": {"type": "array", "minItems": 1,
                             "items": {"type": "string"}},
                },
            },
        },
    },
}

def validate(payload: str) -> dict:
    doc = json.loads(payload)
    jsonschema.validate(instance=doc, schema=SCHEMA)
    return doc

Hours 22–36: the cassette froze the bug

Here is the failure I want you to take away. I recorded the cassettes before tightening validation, so several fixtures contained stringified tags. The suite went green, because replay returned the exact broken payload that the broken parser expected. Green tests, wrong code, and a fixture that would defend the bug forever.

Would you have caught that in review? I did not, because the cassette diff looked like successful recordings rather than captured mistakes. Three changes fixed it, and none of them are clever:

1. Record only through the strict validator, so a bad payload can never become a fixture.
2. Store a schema_version inside every cassette and delete cassettes whose version no longer matches.
3. Treat cassette files as reviewable code, not as generated artifacts to ignore.

That third rule is the one people skip, and it is the one that would have saved me a night.

Hours 36–48: a freshness probe and a decision table

Replay proves your parser is consistent with your fixtures. It cannot prove your fixtures are consistent with the world. So I added a probe that re-calls a small sample of recorded cases on a schedule and compares the live result against the cassette structurally, not textually.

# weekly, not on every push
python -m harness.probe --sample 5 --strict --report probe.json

When the probe disagrees, the cassette is stale and I re-record deliberately. The table below is what I now use to decide, and it has removed most of the guesswork.

Situation	Replay	Re-record	Live call
Refactoring internal logic, same prompt	Yes	No	No
Prompt or parameter changed	No	Yes	No
Schema version bumped	No	Yes	No
Suspected provider-side drift	No	No	Probe first
Every push in CI	Yes	No	No
Pre-release smoke check	No	No	Yes, small sample

Limitations, and who should skip this

Replay cannot detect provider behaviour change, latency shifts, or token accounting unless you record those explicitly. Cassettes also capture whatever was in the prompt, so treat them as potentially sensitive data and scrub before committing. This approach scales to a small, curated suite; at a few thousand fixtures the maintenance cost exceeds the value.

Do not adopt this if your actual goal is statistical evaluation of output quality, since replay freezes one sample and says nothing about variance. Do not adopt it if data residency rules forbid storing prompts on disk. And do not adopt it if nobody will review cassette diffs, because unreviewed fixtures are just bugs with better storage.

What I would repeat

I would keep the canonical-key recorder, the strict nested schema, and the weekly probe, in that order of importance. I would record later than I did, and I would version the schema from day one instead of retrofitting it at hour thirty. Next time I will also add a single assertion that fails loudly when a cassette predates the current schema version.

If you are building something similar, start with the validator and record only what it accepts. The replay layer is easy; deciding what deserves to be frozen is the part that decides whether your green build means anything at all.

Comments

[Launch Gate]

The schema_version bump handles payload-shape drift, but the frozen-bug failure had a second version skew worth naming: those cassettes were stale relative to the parser, not just the schema. A cassette recorded before the tags fix was a fixture produced under the old contract, and nothing in the file said which contract that was. Cheap addition to the header: the git SHA of the recording commit, next to schema_version. Then CI can flag any cassette that predates the last commit touching the parser or validator - which is exactly the population that needs deliberate re-recording after a fix, because otherwise every parser change carries an invisible audit burden: which of these 40 files were recorded under the buggy expectation? The weekly probe does not cover this, since it compares the live world against the cassette; parser drift is the cassette disagreeing with your own code, and only provenance tells you that. It also makes the "review cassette diffs like code" rule much easier to keep: the header tells the reviewer which contract produced the file before they read a single line of payload.