Replay the Captured Request Before You Swap the Model

The product manager clicked Generate weekly digest and save it, then watched a green toast appear. The network panel showed 201 Created, which should have meant the row existed for that tenant. Support still could not find the digest, because the write landed in a sandbox schema the app never reads. Was the model actually wrong here, or did we just celebrate the wrong layer again?

I do not think the current argument about smarter models is the production bottleneck most teams actually have. The first end-to-end user action already fails at a handoff the prompt never sees. Auth, provider, persistence, and audit each return a status, and we rarely replay all four. If you cannot replay yesterday's captured user request, swapping today's model is still theater.

That sounds harsh, and I mean it as an opinion rather than a gentle suggestion about process. A prototype that streams pretty tokens can hide a 401 on the save path until someone actually clicks persist. It can also hide a 409 when two workers finalize the same digest key after a retry. Why would swapping in a different completion model change any of those status contracts at all?

Here is the working path I want in the repo before anyone files a try another model ticket. Capture one real user request as a fixture, including tenant, idempotency key, and the intended side effect. Replay that fixture through a provider seam, not through a concrete vendor SDK call. If that replay is red, you are debugging the application rather than the prompt.

Think of the model like a guest chef walking into a restaurant you already operate every night. The guest can plate a beautiful special, but the ticket still has to survive the pass, the allergen check, and the kitchen printer. Swapping chefs does not fix a ticket printer that drops the table number on busy Fridays. Your AI feature is that same meal moving across named stations: authenticate, complete, persist, and audit.

I will keep this example small and label it as a proposed slice, not a customer case with invented timings. The user action is create a weekly digest for tenant T and persist it exactly once. The first failing layer in this design is persistence identity, because a 201 that writes the wrong schema is a lie. We should make that lie fail a test instead of failing in a support thread.

The fixture is the artifact, not the chat transcript, because transcripts do not pin tenant identity or idempotency. I want a JSON document that a test can load without talking to a network on the happy path. Proposed contents look like the block below, and they should live next to the test rather than in a prompt file.

{
  "action": "create_weekly_digest",
  "actor_id": "user_82",
  "tenant_id": "tenant_acme",
  "idempotency_key": "digest:tenant_acme:2026-W38",
  "prompt": "Summarize shipped work for the week and persist the digest.",
  "expect": {
    "auth": 204,
    "complete": 200,
    "persist": 201,
    "audit": 202
  }
}

Notice the expect map talks in status codes, not in model brand names or temperature folklore. The week identity sits inside the idempotency key so a retry cannot invent a second row. If your UI retries on a flaky connection, that key is the only thing standing between one digest and a folder of twins. Does your current demo even send that key, or does the browser just hope for luck?

The provider seam exists so a free completion path and a later billed path cannot leak into handlers. I do not want environment branches sprinkled across views like confetti after a noisy launch party. A tiny protocol keeps the handler honest, and the replay can inject a fake completer that never spends a token. Proposed code, not a benchmark of any hosted model, looks like the following sketch.

# proposed slice — illustrative, not measured in production
from typing import Protocol, TypedDict

class Completion(TypedDict):
    text: str
    finish: str

class Completer(Protocol):
    def complete(self, prompt: str, tenant_id: str) -> Completion: ...

class DigestRecord(TypedDict):
    tenant_id: str
    idempotency_key: str
    body: str

class DigestStore(Protocol):
    def create_once(self, record: DigestRecord) -> str: ...
    # returns "created" | "conflict" | "wrong_tenant"

The handler should be boring on purpose, because boring code is what a replay can actually accuse. Auth runs first and can return 401 or 403 before any completion is purchased from a provider. Persist maps store outcomes onto 201, 409, and 422, which is the layer that lied in the opening story. Audit is last and must not be able to change the persist result after the row exists.

def create_digest(req, authz, completer: Completer, store: DigestStore, audit) -> tuple[int, dict]:
    decision = authz.allow(req.actor_id, req.tenant_id, "digest:write")
    if decision == "unauthenticated":
        return 401, {"error": "unauthenticated"}
    if decision == "denied":
        return 403, {"error": "forbidden"}

    completion = completer.complete(req.prompt, req.tenant_id)
    if completion["finish"] != "stop":
        return 422, {"error": "incomplete_completion"}

    outcome = store.create_once({
        "tenant_id": req.tenant_id,
        "idempotency_key": req.idempotency_key,
        "body": completion["text"],
    })
    if outcome == "wrong_tenant":
        return 422, {"error": "tenant_mismatch"}
    if outcome == "conflict":
        return 409, {"error": "digest_exists", "key": req.idempotency_key}

    audit.emit("digest.created", req.tenant_id, req.idempotency_key)
    return 201, {"key": req.idempotency_key, "tenant_id": req.tenant_id}

Would I put streaming tokens into this handler on day one of the feature work? No, because streaming makes people stare at language while the persist path stays untested. Language is only the garnish on this plate, and the contract is the plate itself. Ship those statuses first, then add streaming only if the contract still holds under a retry.

The replay test is where the opinion becomes executable, and I want it to fail on the opening bug. A fake store that writes sandbox while the request says tenant_acme must not be allowed to return 201. A second call with the same key must return 409, not another green celebration toast. Proposed pytest below is the whole point of this article, not a decorative appendix.

# tests/test_digest_replay.py — proposed harness
import json
from pathlib import Path
from types import SimpleNamespace

FIXTURE = json.loads(Path("tests/fixtures/create_weekly_digest.json").read_text())

class FakeAuth:
    def __init__(self, decision): self.decision = decision
    def allow(self, actor_id, tenant_id, perm): return self.decision

class FakeCompleter:
    def complete(self, prompt, tenant_id):
        return {"text": f"digest for {tenant_id}", "finish": "stop"}

class FakeStore:
    def __init__(self, outcome): self.outcome = outcome
    def create_once(self, record):
        if record["tenant_id"] != "tenant_acme":
            return "wrong_tenant"
        return self.outcome

class FakeAudit:
    def __init__(self): self.events = []
    def emit(self, *args): self.events.append(args)

def replay(auth_decision, store_outcome):
    req = SimpleNamespace(**{k: FIXTURE[k] for k in ("actor_id", "tenant_id", "idempotency_key", "prompt")})
    return create_digest(req, FakeAuth(auth_decision), FakeCompleter(), FakeStore(store_outcome), FakeAudit())

def test_happy_path_matches_fixture_expect():
    status, body = replay("allow", "created")
    assert status == FIXTURE["expect"]["persist"]
    assert body["tenant_id"] == "tenant_acme"

def test_sandbox_write_is_not_a_201():
    status, body = replay("allow", "wrong_tenant")
    assert status == 422
    assert body["error"] == "tenant_mismatch"

def test_retry_does_not_duplicate_digest():
    status, _ = replay("allow", "conflict")
    assert status == 409

def test_save_is_blocked_when_session_is_cold():
    status, _ = replay("unauthenticated", "created")
    assert status == 401

Run those tests like a skeptic, not like a demo narrator looking for a round of applause.

pytest tests/test_digest_replay.py -v

When test_sandbox_write_is_not_a_201 fails, you have found the opening incident without paging support. When test_retry_does_not_duplicate_digest fails, your UI retry is about to double-file the week. Only after those assertions stay green would I discuss swapping the completer, because until then the model is the wrong suspect.

What usually fails along the way is not the fake completer, which is almost too easy to write quickly. The authz adapter silently uses a default tenant when the session cookie is stale, and every replay looks like a 201. The store uses INSERT without a unique constraint on the idempotency key, so the conflict path never exists in production. The audit call sits inside the persist transaction in staging and outside it in production, so a 201 can vanish after a crash.

Production caveats are mostly about honesty at the edges you cannot fake with a stub. A fixture cannot prove the real provider will finish cleanly under load, so keep a live canary against a non-prod tenant. Idempotency keys must be unique in the real table, not only in the in-memory stub, or the 409 test is fiction. Cross-region retries need the same key in the header, or you will mint twins the replay never saw.

Who should not bother with this approach yet, even if the demo looks convincing on a laptop? If you are hacking a local toy with no tenants, no persistence, and no user besides yourself, the harness is overhead. If your product cannot name the write permission yet, fix authorization before you capture fixtures, because a completer 200 is not a grant. If you need subsecond token streaming as the actual product, this slice will feel too strict, and that discomfort is the point.

Before I would merge the feature, I want a short proof I can reuse the following week without a demo.

• Captured request names tenant, actor, and idempotency key.
• Handler returns 401, 403, 409, 422, and 201 without a vendor SDK.
• Sandbox writes cannot return 201, and retries cannot create a second row.
• Non-prod canary replays the same expect map and records four status codes.

I keep seeing teams argue about model quality the way kitchens argue about salt while the ticket printer is on fire. Can you replay last week's user request today and get the same four status codes back from each layer? Tell me which layer handoff is least stable in your stack, and send a concrete failure state or response code. A model name is not an incident report, and I will not treat it as one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free models that can run this workflow. A free server option is enough to reproduce the setup.