Weekend Build Log: Record Your AI Calls Once, Replay Them Forever

It is Saturday morning. You change one line in a prompt template. Then you run the test suite.

Forty model calls later, three tests fail. You cannot tell why.

Did your prompt edit break the parser? Or did the model answer differently today? You re-run. Now four tests fail. Nothing is reproducible, and every debug cycle costs tokens and minutes.

That is where this build log starts. The fix is not a smarter model or a bigger context window. It is a cassette: record each call once, replay it forever.

The one rule for the weekend

One rule keeps the scope honest. If a feature does not shorten the record-then-replay loop, it waits until next weekend.

Built this weekend	Skipped on purpose
JSON cassettes on disk	Vector store, embeddings
SHA-256 key over model, prompt, temperature	Semantic or fuzzy prompt matching
Replay mode with loud misses	Auto re-record on failure
Pytest fixture plus one CI command	Metrics dashboard, web UI
Three-check triage list for misses	Multi-provider fan-out

That is the whole product. A folder of JSON files and a function that refuses to phone home.

Step 1 - Design the cassette key before writing code

The key decides everything else. Hash exactly the inputs that change model behavior, and nothing else.

The code below is a sketch I have not run against a live endpoint. Treat it as a shape to copy, not a validated benchmark.

# cassette.py
from __future__ import annotations
import hashlib, json, os, pathlib, time

CASSETTE_DIR = pathlib.Path(os.environ.get("CASSETTE_DIR", "cassettes"))
MODE = os.environ.get("CASSETTE_MODE", "replay")  # record | replay | off


class CassetteMiss(RuntimeError):
    def __init__(self, key: str, prompt: str) -> None:
        super().__init__(f"no cassette for {key}")
        self.key = key
        self.prompt = prompt


def key_for(model: str, prompt: str, temperature: float) -> str:
    payload = json.dumps(
        {"model": model, "prompt": prompt, "temperature": temperature},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]

Two details save you hours later. Normalize whitespace in the prompt before hashing. Pin the exact model string, because floating aliases will silently invalidate every key you own.

Step 2 - Record once, on purpose

Recording is the only step that spends tokens, so it should be a deliberate act. Never let it happen during a normal test run.

def call(model: str, prompt: str, temperature: float = 0.0) -> str:
    key = key_for(model, prompt, temperature)
    path = CASSETTE_DIR / f"{key}.json"

    if MODE == "replay":
        if not path.exists():
            raise CassetteMiss(key, prompt)
        return json.loads(path.read_text(encoding="utf-8"))["text"]

    text, usage = live_call(model, prompt, temperature)  # your existing client

    if MODE == "record":
        CASSETTE_DIR.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps({
            "key": key,
            "model": model,
            "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest(),
            "text": text,
            "usage": usage,
            "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        }, indent=2), encoding="utf-8")

    return text

Record at temperature 0. Store the usage block even if you ignore it. Next month you will want to know how many tokens your test suite really needs.

CASSETTE_MODE=record pytest tests/test_prompts.py -x
git add cassettes/ && git commit -m "record prompt cassettes"

Step 3 - Replay in CI with no network and no keys

Replay must fail loudly. Silent fallback to a live call destroys the whole point of the exercise.

# tests/conftest.py
import pytest

@pytest.fixture(autouse=True)
def force_replay(monkeypatch):
    monkeypatch.setenv("CASSETTE_MODE", "replay")


def test_parser_survives_a_short_answer():
    from cassette import call
    out = call("small-model", BUILD_PROMPT)
    assert parse_block(out)["steps"], "parser returned no steps"

CI command: CASSETTE_MODE=replay pytest -q. No API key in the runner, no rate limit at 2am, no flaky red builds caused by a provider hiccup.

Step 4 - Triage a cassette miss in three checks

A miss is information, not a bug. Work through these in order before you touch anything.

1. Diff the prompt string. A reordered bullet or a new example changes the key.
2. Diff the model string. An alias that moved breaks every cassette in the folder.
3. Diff the temperature and parameters. One changed default is enough to miss.

If all three match and you still miss, your key function is under-specified. Add the missing input to key_for and re-record.

Should you re-record?

Situation	Re-record?	Reason
Prompt wording changed on purpose	Yes	The old answer describes the old question
Parser bug fixed	No	Model output is unchanged
Model version retired	Yes	You are now testing a different system
Test asserts exact prose	No, fix the test	Brittle assertion, not a data problem
Output contract changed	Yes, then diff	You want the new shape reviewed

Review cassette diffs like code. A 200-line JSON diff in a pull request is a prompt change nobody noticed during standup.

Where a free tier actually fits

Recording is the expensive half, and you do it rarely. That makes it a decent candidate for an operator-supplied free tier rather than your production key.

MonkeyCode describes free model access and a free server option. I have not benchmarked either, and free terms change without notice, so verify the current ones yourself before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest uses in this workflow:

• Run the record pass before you set up billing, so the harness exists at all.
• Run the replay job on the free server, so CI does not depend on your laptop being awake.

Anything beyond that is speculation until you measure it. Do not commit a pipeline to quotas or hardware you have not confirmed in writing.

If you want to try the record pass with the free model access and free server described above, start with ten calls, not a hundred, and read the numbers yourself.

What this approach does not test

Be honest about the boundary. Record and replay tests your parsing, prompt assembly, and error handling against fixed evidence.

It does not detect provider drift. Keep a small live smoke test, five calls, running nightly, and let it fail loudly.

It also produces weak assertions on open-ended generation. Assert on structure, field presence, and length bounds. Do not assert on prose you cannot reproduce.

One more constraint: redact at the boundary before you write a cassette. A recording is a file in a repo, and repos get cloned.

Who should skip this build

• Teams whose main risk is provider-side regression. You need live checks, not tapes.
• Products with data rules that forbid third-party inference, even for a recording pass.
• Anyone who needs real latency numbers. A cassette returns in microseconds; production does not.
• A side project with three prompts. Run them live and go outside.

Sunday night state

You end the weekend with a cassettes/ folder, an offline test suite, and a triage list taped to the wall. You skipped embeddings, dashboards, and auto re-record. That was the right call.

If you have one spare hour next weekend, add a --strict flag that fails on any uncommitted cassette change. One flag, ten minutes, and prompt edits stop sneaking into main.