Record, Replay, Assert: Testing LLM Agents in CI Without Paying for Every Run

Most agent codebases have one of two test suites.

The first one calls the real model. Every pull request spends real money, takes four minutes, and fails one time in ten because the model phrased a tool call differently. Developers learn to re-run the job until it goes green, which is the same as having no test suite.

The second one mocks the model with a function that returns "OK". It runs in 200 ms and has never caught a bug, because the bugs in an agent are in the shape of what the model returns — a tool call with a missing argument, two tool calls in one turn, a stop reason nobody handled — and "OK" has no shape.

There is a third option, and it is the same one the HTTP world settled on a decade ago: record the real interaction once, replay it deterministically, and assert on what your code did with it. This article is the agent-specific version of that idea — where to cut the seam, what to key the recordings on, which normalization mistakes silently break it, and how to split CI so the real model still gets exercised without being in the path of every merge. It is the test-infrastructure half of the discipline we teach in the LLM evaluation and testing course at Cursuri-AI.ro; the quality-measurement half is a different article.

What you are actually testing

Get this straight first, because it decides everything downstream.

An agent is two things bolted together: a model that proposes actions, and a harness that executes them — the loop, the tool dispatcher, the argument parser, the retry logic, the termination check, the state you carry between turns. Model quality is measured with evals: a dataset, a scorer, a number that moves. That is a separate discipline (I wrote it up on dev.to as "Stop Vibe-Checking Your LLM"), and it is inherently statistical.

The harness is ordinary software. It has branches, and the branches have bugs, and those bugs are deterministic — given the same model output, the harness does the same wrong thing every time. Which means the harness can be tested the way any other software is tested: fixed inputs, exact assertions, sub-second runs.

The only obstacle is that the "fixed inputs" are model outputs, and model outputs are expensive and non-deterministic to produce. Recording removes that obstacle. Once the model's responses are pinned to a file, the entire agent run is a pure function of the recording, and you can assert on it like a unit test.

So the goal is not "test that the agent answers correctly". The goal is: given this sequence of model responses, does the harness dispatch the right tools with the right arguments, in the right order, handle the errors, and stop when it should?

Layer 1: one seam

Every call to the model goes through exactly one function. Not "mostly one". One.

# llm.py
from dataclasses import dataclass
import anthropic

client = anthropic.Anthropic()

@dataclass(frozen=True)
class Request:
    model: str
    system: str
    tools: list[dict]
    messages: list[dict]
    max_tokens: int = 16000

def complete(req: Request) -> anthropic.types.Message:
    return client.messages.create(
        model=req.model,
        system=req.system,
        tools=req.tools,
        messages=req.messages,
        max_tokens=req.max_tokens,
    )

The agent loop imports complete and nothing else from the SDK. Tool execution, message assembly, termination — all of it lives above the seam and never touches the network. If your loop currently builds the request inline in three places, this refactor is the actual work; the recorder is twenty lines.

The seam sits at the SDK call, not at the HTTP layer. You could go lower — vcrpy (and pytest-recording, the pytest plugin around it) record raw HTTP and would work — but then your cassettes contain serialized SSE frames, retry traffic, and header noise, and a streaming refactor invalidates every one of them. Recording the parsed response object is smaller, readable in a diff, and survives transport changes.

Layer 2: the cassette

A cassette is a JSON file: a list of (request fingerprint, response) pairs, in the order they happened. Replay looks up the fingerprint and returns the stored response. Record calls the model and appends.

# cassette.py
import hashlib, json, os
from pathlib import Path

MODE = os.environ.get("LLM_RECORD", "replay")   # "replay" | "record"

def fingerprint(req) -> str:
    canonical = json.dumps(
        {
            "model": req.model,
            "system": req.system,
            "tools": sorted(req.tools, key=lambda t: t["name"]),
            "messages": req.messages,
        },
        sort_keys=True, ensure_ascii=False, separators=(",", ":"),
    )
    return hashlib.sha256(canonical.encode()).hexdigest()[:16]

class Cassette:
    def __init__(self, path: Path):
        self.path = path
        self.entries = json.loads(path.read_text()) if path.exists() else []
        self.cursor = {}          # fingerprint -> how many times served

    def __call__(self, req, live):
        fp = fingerprint(req)
        n = self.cursor.get(fp, 0)
        hits = [e for e in self.entries if e["fp"] == fp]
        if n < len(hits):
            self.cursor[fp] = n + 1
            return hits[n]["response"]
        if MODE != "record":
            raise AssertionError(
                f"No recording for request {fp} (occurrence {n}). "
                f"Prompt or history changed? Re-record with LLM_RECORD=record."
            )
        resp = live(req)
        self.entries.append({"fp": fp, "response": resp.model_dump(mode="json")})
        self.path.write_text(json.dumps(self.entries, indent=2, ensure_ascii=False))
        self.cursor[fp] = n + 1
        return resp.model_dump(mode="json")

Two details in there do real work.

The occurrence counter. In an agent loop, consecutive requests differ (the history grows), so the fingerprint alone is usually unique. But retries re-send an identical request, and a model asked the same question twice may legitimately answer differently. Keying on (fingerprint, nth occurrence) keeps both cases replayable in order without collapsing them.

The hard failure on a miss. A cache miss in replay mode is a test failure, not a fallback to the live model. If your PR changed the system prompt, every downstream fingerprint changes, and the right outcome is a loud red build that says "your prompt changed; re-record deliberately" — not a silent live call that costs money and passes by accident.

Wire it in with a pytest fixture that swaps complete for the cassette:

# conftest.py
import pytest
from pathlib import Path
import llm
from cassette import Cassette

@pytest.fixture
def recorded(request, monkeypatch):
    path = Path("tests/cassettes") / f"{request.node.name}.json"
    cassette = Cassette(path)
    live = llm.complete
    monkeypatch.setattr(llm, "complete", lambda req: cassette(req, live))
    return cassette

One test, one cassette, named after the test. Commit the cassettes. They are fixtures, and they belong in review like any other fixture.

The normalization bugs that break replay silently

The fingerprint is a hash of the request. Anything that makes the request differ between the recording run and the replay run makes every test miss. In practice the culprits are always the same four, and — not coincidentally — they are the same four that silently break prompt caching, because a prompt cache is also a hash of a prefix.

1. A timestamp in the system prompt. f"Today is {date.today()}" is the most common one. Inject the date as a parameter and freeze it in tests.
2. Unsorted tool schemas. Tools built from a dict or a set come out in arbitrary order. Sort by name before hashing (the code above does) and before sending — the model sees the order too.
3. Request-scoped IDs in the messages. A session ID or trace ID pasted into the first user turn. Move it to metadata or strip it in fingerprint().
4. Non-deterministic serialization. json.dumps without sort_keys=True, floats that render differently, a dict that got mutated between the record and the assert.

Rule of thumb: if fingerprint(req) is not stable across two runs of the same test in the same commit, fix that before you record a single cassette. A stable fingerprint is also the cheapest possible cache-hit-rate test you will ever write.

Layer 3: assert on the trajectory, not the prose

With the model pinned, what do you actually check? Not the final text — you recorded it, so asserting on it proves nothing. Assert on what the harness did.

The most useful single artifact is the trajectory: the ordered list of tool calls the harness dispatched, with their parsed arguments and the result it returned to the model. Have your loop emit it as a plain list:

@dataclass
class Step:
    tool: str
    args: dict
    result: str
    is_error: bool

class Trajectory(list[Step]):
    def tools(self) -> list[str]:
        return [s.tool for s in self]

Then the tests read like specifications:

def test_refund_flow_looks_up_before_refunding(recorded):
    traj, final = run_agent("Refund order 4471, it arrived damaged")

    assert traj.tools() == ["get_order", "issue_refund"]
    assert traj[1].args == {"order_id": "4471", "reason": "damaged"}
    assert not any(s.is_error for s in traj)
    assert len(traj) <= 4, "loop should terminate within budget"

def test_unknown_order_stops_without_refunding(recorded):
    traj, final = run_agent("Refund order 9999")

    assert "issue_refund" not in traj.tools()
    assert traj[0].is_error                     # get_order returned not-found
    assert "9999" in final                      # and the model told the user

Notice what these catch. The first would fail if a refactor started issuing refunds before the lookup, if argument parsing dropped reason, if the loop ran away. The second would fail if error results stopped being passed back as is_error, or if the harness swallowed the error and let the model proceed. None of those are model-quality questions. All of them have shipped to production in real systems.

Three more assertions worth keeping in every agent suite:

• Every tool_result has a matching tool_use_id. Hand-rolled loops get this wrong under parallel tool calls, and the API rejects the next request.
• Parallel calls come back in one message. If the model issued two tool_use blocks, the harness must return both results in a single user turn. Splitting them works today and quietly degrades the model's willingness to parallelize tomorrow.
• Termination is explicit. end_turn ends, tool_use continues, max_tokens is handled (usually: retry with more room or surface an error), and anything else fails loudly. Recording is the only cheap way to get a max_tokens stop into a test — see the next section.

Synthetic cassettes: the edge cases you cannot record

Some of the most important harness paths are ones a well-behaved model rarely produces on demand: a tool call naming a tool that does not exist, an argument that fails schema validation, a response truncated at max_tokens, an empty content array, a refusal.

You do not need to coax the model into these. A cassette is JSON. Write it by hand.

[
  {
    "fp": "any",
    "response": {
      "id": "msg_synthetic_01",
      "type": "message",
      "role": "assistant",
      "model": "claude-opus-5",
      "stop_reason": "tool_use",
      "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "issue_refnud", "input": {"order_id": "4471"}}
      ],
      "usage": {"input_tokens": 1200, "output_tokens": 40}
    }
  }
]

The misspelled tool name is deliberate — that is the branch under test. Point the fixture at that file (with fingerprint matching relaxed to "next entry, whatever the request") and assert that the harness returns a tool_result with is_error: true and a message the model can act on — rather than raising KeyError three layers up and taking the worker down. Keep the synthetic cassettes in their own directory and label them; the value of recorded ones is that they are real, and mixing the two erodes that.

This is where recording earns its keep over live testing. You cannot reliably test the unknown-tool branch against a live model. You can test it in 30 ms against a file, forever.

Streaming, thinking blocks, and other things that leak through the seam

Streaming. Record the final assembled message, not the event stream. Your stream-to-message assembler is a pure function of events and gets its own unit test with a hand-written event list. Everything above the seam only ever sees a complete message, which is exactly the property that makes the design work.

Thinking blocks. Current models return thinking blocks you are expected to pass back on the next turn. They are part of the recorded response and part of the next request's fingerprint, and that is fine in replay — nothing goes to the network. Two cautions, though. The signatures inside those blocks are bound to the exact history that produced them, and on the newest models editing earlier turns before a live replay gets the request rejected outright — so if you ever build a "replay this cassette against the real model" tool, feed the history back unmodified and in order. And do not assert on thinking content. It is not stable across model versions, and you do not want a test that breaks because the model reasoned differently on the way to the same tool call.

Usage and cost. usage is in the recording. It is a decent place to add a cheap regression guard: "this flow used to cost 6k input tokens; fail if a cassette re-record pushes it above 10k". You will catch a runaway prompt weeks before the invoice does.

The model ID. It is in the fingerprint, which means bumping the model re-records everything. That is correct behavior — a new model is a new set of trajectories, and re-recording is the moment you review them.

Re-recording is a code review, not a chore

When a prompt change invalidates cassettes, the workflow is:

1. Run the suite with LLM_RECORD=record for the affected tests only.
2. Look at the diff of the cassettes, specifically the trajectories. Did the tool order change? Did an argument change? Did a flow that used to finish in three calls now take five?
3. Commit the new cassettes in the same PR as the prompt change.

Step 2 is the part that gets skipped and is the part that matters. A cassette diff is a before/after of your agent's behavior on the exact scenarios you decided were important. Treat it the way you would treat a snapshot-test diff in a UI codebase: mostly noise, occasionally the thing that saves you. This review discipline is also where model upgrades stop being scary — the AI agents architecture course builds a full loop this way, precisely so that swapping the model underneath is a re-record and a diff, not a leap of faith.

Two rules keep re-recording honest. Never re-record on CI; only from a developer machine, deliberately, with the diff in front of a human. And never let LLM_RECORD=record be the default anywhere — the environment variable's absence should mean replay, and a missing cassette should mean failure.

The CI split

Now the part that makes the whole thing sustainable. You run two jobs, with different triggers, budgets, and credentials.

On every PR — replay. No API key in the environment at all. Cassettes only. Runs in seconds, costs nothing, and fails deterministically. This is the job that gates merges.

Nightly (or on a run-live label) — record against the real model. A small curated set — the ten or twenty scenarios you would be embarrassed to break — plus the eval suite. This job has the API key, a hard spend cap, and its own failure semantics: a trajectory that differs from the committed cassette is a report, not a block, because the model is allowed to vary. What it is not allowed to do is fail the eval gate.

# .github/workflows/agent.yml (sketch)
jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e .[test]
      - run: LLM_RECORD=replay pytest tests/agent -q
      # no ANTHROPIC_API_KEY here — a live call would fail loudly

  live:
    if: github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'run-live')
    runs-on: ubuntu-latest
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e .[test]
      - run: LLM_RECORD=record pytest tests/agent -q -m curated
      - run: python -m evals.run --gate 0.90

The separation of credentials is not a detail. A PR job with no API key cannot accidentally hit the model, cannot leak the key to a fork, and cannot cost money. All three failure modes have bitten teams that put one job in charge of both.

What this does not give you

Recording is a harness test. It will not tell you:

• whether the model's answer was good — that is evals;
• whether a new model version behaves differently on inputs you did not record — that is evals plus the nightly live run;
• whether your tool implementations are correct — those get their own unit tests, with the model nowhere near them;
• whether the system holds up under concurrency, rate limits, or provider outages — that is a resilience concern, and it needs a different kind of test.

What it does give you is the thing agent teams most often lack: a test suite that engineers actually run before pushing, because it is fast, free, and fails for reasons that are their fault.

TL;DR

• Agents are a model plus a harness. The harness is deterministic software; test it like software.
• Put every model call behind one function. Record its responses to a JSON cassette; replay them in tests.
• Fingerprint requests on (model, system, sorted tools, messages) and key entries on (fingerprint, occurrence). A miss in replay mode is a failure, never a live call.
• Fix the four fingerprint killers — timestamps, unsorted tools, request IDs, unstable serialization. They break your prompt cache too.
• Assert on the trajectory (tool order, parsed args, error handling, termination), never on the recorded prose.
• Hand-write synthetic cassettes for the paths you cannot record: unknown tool, bad args, max_tokens, refusal.
• Record the final message, not the stream. Never assert on thinking content.
• Two CI jobs: replay on every PR with no API key; live record plus evals nightly with a spend cap.

If you are building the rest of the production picture around this — caching, retries, structured outputs, the deployment shape — the advanced LLM integration course covers the layers this article deliberately left out, and Build and Ship a Production AI SaaS walks the whole thing from an empty repo to paying users, test suite included.

---

Code in this article targets the Anthropic Python SDK's Messages API shape (tool_use / tool_result blocks, stop_reason values) as documented in September 2026; the recording pattern itself is provider-agnostic. vcrpy and pytest-recording are referenced as HTTP-level alternatives — the SDK-level seam described here is deliberately simpler. Verify SDK field names against current documentation before copying the snippets into production.