Detecting When a Recorded Cassette No Longer Matches the Live API

#ai #debugging #llm #machinelearning

The whole suite is green, has been green for four months, and the feature is broken in production. The provider added a field, renamed one, changed a finish-reason value or started returning a different error envelope — and your cassettes, which are frozen copies of the API as it was on the day you recorded them, have been happily replaying the old world ever since.

The symptom

It shows up as one of a small set of complaints, all of which sound like a code bug and are not:

A field your parser reads is undefined in production and populated in tests.
A branch keyed on a finish reason or a stop reason never fires live, because the value it compares against is no longer one the provider emits.
Error handling that is thoroughly tested falls through to the generic handler in production, because the error body shape changed.
A newly added request parameter is silently absent from the replay, because the default cassette matcher does not look at the request body at all.

That last one is worth stating plainly, because it surprises people: VCR.py matches a request to a recorded interaction on the URI and the HTTP method by default. Two completely different chat completions to the same endpoint are the same request as far as playback is concerned. Your test can send a prompt you have never recorded and get a confident, stale answer back.

Why the suite cannot notice

A cassette is a fixture whose defining virtue is that it was real once. Every other kind of test data announces its staleness eventually — a hand-written fixture is obviously a guess, a type definition fails to compile when the SDK changes — but a cassette looks authentic forever. Nothing in the replay path consults the live API, which is the entire point of using one, and so nothing in the replay path can tell you the live API moved.

Type checking does not save you either, because the cassette is JSON on disk and is deserialised into whatever your code expects. The contract you are actually testing is “my parser handles the bytes I saved”, which was true when you saved them and is not the contract you care about.

The re-record-and-diff job

The fix is a scheduled job — nightly or weekly, on a schedule, never on the pull-request path — that re-records the same interactions against the live API into a scratch directory and compares the shapes. It costs a small number of real API calls and it is the only mechanism that closes the loop.

With vcrpy and the pytest-recording plugin, the pieces exist already. The plugin adds a rewrite record mode on top of VCR.py’s own once, new_episodes, none and all, documented as rewriting a cassette from scratch rather than extending it with new entries — which is exactly the semantics a drift check needs. The vcr_config fixture forwards keyword arguments to VCR.py, so filter_headers and match_on are configured in one place.

# conftest.py
import pytest

@pytest.fixture(scope="module")
def vcr_config():
    return {
        # Never let a key reach disk.
        "filter_headers": ["authorization", "x-api-key", "api-key"],
        # The default is (uri, method), which cannot tell two prompts apart.
        "match_on": ["method", "scheme", "host", "port", "path", "query", "body"],
    }

Run the suite normally in CI with recording off, so a test that reaches the network fails instead of silently calling a provider. The plugin’s --block-network option does this globally, with --allowed-hosts for anything legitimately local.
In the scheduled job, copy the committed cassette directory to a reference location, then run the same suite with --record-mode=rewrite and network allowed. Every cassette is re-cut from live responses.
Normalise both trees — committed and freshly recorded — and compare. Do not compare files.
On a difference, open one issue per drifted cassette carrying the normalised diff, and do not auto-commit the new recordings. A drift that is auto-accepted is a drift that was never reviewed.

Comparing shape, not bytes

A byte comparison of two recordings of the same request will always differ, and will therefore always be ignored. Response ids, request ids, created timestamps, latency headers and token counts are different on every call by design. What you want to know is whether the structure changed: which keys exist, at which paths, with which JSON types, and which values are drawn from a closed set.

# tools/cassette_shape.py
from typing import Any

def shape(node: Any, path: str = "") -> dict[str, str]:
    """Flatten a JSON body to {path: type}, collapsing arrays to their first element."""
    out: dict[str, str] = {}
    if isinstance(node, dict):
        for key, value in node.items():
            out.update(shape(value, f"{path}.{key}" if path else key))
    elif isinstance(node, list):
        out[path] = "array"
        if node:
            out.update(shape(node[0], f"{path}[]"))
    else:
        out[path] = type(node).__name__
    return out

# Values that are part of the contract even though they are strings.
ENUM_PATHS = {
    "choices[].finish_reason",
    "stop_reason",
    "error.type",
    "type",
}

Compare the two flattened maps and report three categories separately, because they mean different things. A removed path is urgent: code that reads it is already broken in production. A changed type at a path is equally urgent, and is the one that produces the confusing bugs, because a field that became a string containing JSON still passes a truthiness check. An added path is informational — it will not break you, but it is often the visible edge of a feature you now want, and it dates the cassette.

Then compare enum values separately: collect the observed values at each path in ENUM_PATHS across the whole recording tree, and flag any value present live that is absent from the committed set. A new finish_reason is precisely the kind of change that breaks a switch statement with no default branch and no test coverage.

The set of paths worth treating as enums is provider-specific and moves. Anthropic’s error envelope, for example, wraps a type and a message inside a top-level error object alongside a request_id, and its documentation states plainly that the values inside these objects may expand over time. Treat your committed list as a snapshot, not a specification — see Anthropic’s errors reference.

What to do with a reported drift

Resist the reflex to re-record and move on. The drift report is the only moment you get to ask whether your code handles the new shape, and re-recording answers that question by making the test agree with whatever the provider now does — including the part that is breaking you. Work in the other order: write a failing test against the new shape by hand, fix the code, and only then accept the new recordings.

Two hygiene items make the job much less noisy. Pin the API version header where the provider offers one, so a shape change is a decision you made rather than a morning surprise — see what a silent model update does to a frozen fixture. And keep a cassette.all_played assertion on the tests that care: VCR.py exposes all_played and play_count on the cassette object, and a cassette with an interaction nothing ever requests is usually a test that stopped calling what it thinks it calls.

If the drift report is chronically noisy for a particular fixture, that fixture is probably the wrong tool: some things are better asserted against a hand-written body whose whole purpose is legible on the page. That trade-off is the subject of when to record and when to hand-write.