Case Study: A Record/Replay Cassette That Makes an AI Agent Test Deterministic

The first failure in this build had nothing to do with model quality: the agent's test suite went red because a network call timed out, then green again ten minutes later with no code change. Record every model response once into a plain JSONL file, replay it forever in CI, and fail loudly whenever the prompt or request shape drifts. That single rule turned a flaky agent job into a deterministic one, and the replay side costs nothing to run.

Below is the full case study of a small classify-and-route agent: the constraints, the cassette module, the drift guard that saved the most debugging time, and the parts I would not recommend.

Background: an agent that could not be tested

The project was small on purpose. One job reads a failing test log, asks a model to pick the likely owner module, and posts a suggestion to a review thread. Three functions, one prompt file, and roughly forty lines of glue code — the kind of thing you write in an afternoon and then maintain for a year.

The problem showed up in week two. Every assertion compared model output against expected strings, so the suite depended on sampling, provider latency, and whatever the current model snapshot preferred that morning. A red build told you nothing, because you could not separate a regression from a slow Tuesday.

The usual fixes are worse than the disease. Mocking the client returns whatever you typed, so the test asserts your own fiction. Skipping the test in CI means nobody notices when the prompt file quietly loses a section.

Goal and constraints

• Deterministic replay. Same input, same output, with no network and no credentials needed.
• Loud drift. A changed prompt or request shape must fail with a readable diff instead of passing on stale data.
• Cheap recording. The one pass that touches a real model should not require a paid key or a dedicated box.
• Reviewable artifacts. A human should be able to read the recorded fixture inside a normal pull request.

Implementation

1. Hash a normalized request, not the raw payload

Providers attach ids, timestamps, and usage counters to responses, and tool-call blocks carry randomly generated identifiers. If you hash the raw payload, identical requests produce different keys, and your cassette grows one duplicate entry per run.

import hashlib, json

VOLATILE = {"id", "created", "system_fingerprint", "usage"}

def normalize(request: dict) -> dict:
    body = {k: v for k, v in request.items() if k not in VOLATILE}
    body["messages"] = [
        {"role": m.get("role"), "content": m.get("content")}
        for m in body.get("messages", [])
    ]
    return body

def request_key(request: dict) -> str:
    blob = json.dumps(
        normalize(request), sort_keys=True, separators=(",", ":"), ensure_ascii=False
    )
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()

Put a template_sha field into the request as well. Editing prompts/classify.md then changes the key automatically, which is exactly the behaviour you want.

2. Store entries as append-only JSONL

One line per call keeps diffs readable and lets you append without rewriting the file. Record mode is idempotent: if the key already exists, return the stored response instead of paying for a second call.

# cassette.py
import json, time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable

class CassetteMiss(RuntimeError):
    def __init__(self, key: str, mode: str):
        super().__init__(f"no recorded response for key {key[:12]} (mode={mode})")
        self.key = key

@dataclass
class Cassette:
    path: Path
    mode: str = "replay"              # "record" | "replay"
    call: Callable[[dict], dict] | None = None
    entries: dict = field(default_factory=dict)

    def __post_init__(self):
        self.path = Path(self.path)
        if self.path.exists():
            for line in self.path.read_text(encoding="utf-8").splitlines():
                if line.strip():
                    entry = json.loads(line)
                    self.entries[entry["key"]] = entry

    def send(self, request: dict) -> dict:
        key = request_key(request)
        if self.mode == "record":
            if key in self.entries:
                return self.entries[key]["response"]      # idempotent re-record
            if self.call is None:
                raise RuntimeError("record mode requires a call function")
            response = self.call(request)
            entry = {
                "key": key,
                "recorded_at": int(time.time()),
                "request": normalize(request),
                "response": response,
            }
            self.entries[key] = entry
            with self.path.open("a", encoding="utf-8") as fh:
                fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
            return response
        try:
            return self.entries[key]["response"]
        except KeyError:
            raise CassetteMiss(key, self.mode) from None

3. Turn a miss into a readable diff

A missing key is the most common failure, and a bare hash tells you nothing useful. Because you stored the normalized request next to the response, you can show the closest entry and print only the fields that changed.

def explain_miss(cassette, request):
    current = normalize(request)
    best, best_score = None, -1
    for entry in cassette.entries.values():
        score = sum(1 for k, v in current.items() if entry["request"].get(k) == v)
        if score > best_score:
            best, best_score = entry, score
    if best is None:
        return "cassette is empty; record a new entry"
    diffs = [
        f"{k}: recorded={best['request'].get(k)!r} current={current.get(k)!r}"
        for k in sorted(set(best["request"]) | set(current))
        if best["request"].get(k) != current.get(k)
    ]
    return "\n".join(diffs)

Nine times out of ten the diff names template_sha, and you immediately know a prompt edit caused the miss rather than a provider outage.

4. Wire the cassette into pytest

# tests/conftest.py
import os, pytest
from pathlib import Path
from cassette import Cassette

@pytest.fixture
def cassette(request):
    mode = os.environ.get("CASSETTE_MODE", "replay")
    path = Path(__file__).parent / "cassettes" / f"{request.node.name}.jsonl"
    return Cassette(path, mode=mode)

Run plain pytest and the suite replays. Run CASSETTE_MODE=record pytest -k classify only when you intentionally change behaviour, then review the cassette diff exactly like any other fixture file.

5. Guard freshness so green stays honest

Replay suites are dangerously easy to keep green forever. Add one test that compares the recorded template hash with the current prompt file, and make the failure message actionable.

# tests/test_freshness.py
import hashlib, json
from pathlib import Path

def test_prompt_matches_cassette():
    template = Path("prompts/classify.md").read_bytes()
    first = Path("tests/cassettes/test_classify_issue.jsonl").read_text(
        encoding="utf-8"
    ).splitlines()[0]
    recorded = json.loads(first)["request"]["template_sha"]
    assert recorded == hashlib.sha256(template).hexdigest(), (
        "prompt changed; re-record the cassette on purpose"
    )

Where free model access and a free server fitted in

The record pass is the only step that needs a live model, and it is also the step that runs least often. That is precisely where MonkeyCode's free model access and free server option were useful in this build: the recording run happened on the hosted free server rather than behind a paid key on someone's laptop, and the resulting JSONL was committed for everyone else.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two cautions matter more than the convenience. Treat any free tier as a moving target and check the project's current terms for which models are available and what limits apply, because availability and quotas change. Also remember that a free model is a reasonable recording source and a poor benchmarking source, so do not publish quality comparisons built on it.

Checks you can run without trusting me

• No network on replay. Install pytest-socket and run pytest --disable-socket; the suite should still pass with zero outbound calls.
• No credentials on replay. Unset your provider API key, run pytest, and confirm nothing fails with an auth error.
• Drift is loud. Change one sentence in prompts/classify.md and watch the freshness test fail with the message above.
• Diffs stay small. Adding a new test case adds one JSONL line, so review cost grows linearly instead of exploding.

Decision table

Situation	Record	Replay	Live call
Unit tests in CI	no	yes	no
Prompt iteration	yes	—	yes
Post-deploy smoke test	no	no	yes
Comparing two models	yes, per model label	yes	—

Limitations and who should not use this

• Open-ended generation. If your product ships freeform prose, exact replay tests the plumbing rather than the quality, so you still need a rubric or a human review step alongside it.
• Stale cassettes hide real regressions. The freshness test catches prompt edits, not upstream behaviour changes, so schedule a deliberate re-record on a fixed cadence.
• Secrets leak easily. Redact before writing anything to disk, and never commit a cassette that contains raw user data or tokens.
• Tiny projects. If you have one test and one provider call, a single hand-written mock is simpler and this machinery costs you an afternoon for no benefit.

Lessons from the build

1. Normalize before you hash, because provider ids will betray you otherwise.
2. Store the request beside the response, since debugging is mostly a diffing exercise.
3. Make drift a failing test rather than a note in a README nobody reads.
4. Keep recording separate from asserting, so nobody re-records just to turn CI green.

If you want to try the record pass before wiring a paid key, start with the project's free model access and free server option, commit the cassette, then delete your provider credentials and confirm the replay suite still passes.