Fixture Design for Recorded Tool-Calling Conversations

#ai #llm #machinelearning #programming

Fixtures are abandoned for one of two reasons: nobody can read the diff when one changes, or re-recording rewrites the whole file and nobody can tell what moved. Both are properties of the format, decided before the first fixture is written, and both are cheap to get right and expensive to change later.

What the format has to survive

Review. A reviewer looking at a changed fixture must be able to see, in the diff, what the model started doing differently. If the diff is one line of minified JSON, the review is “looks fine” and the fixture has stopped doing its job.
Re-recording. Running the recorder again after a change must produce a file that differs only where behaviour differed. Random ids, timestamps and token counts all violate this, and a fixture that changes entirely on every recording gets re-recorded without being read.
A provider change. A fixture recorded against one API and stored in that API’s wire format is unusable the day you add a second. Storing a neutral shape costs an adapter and saves re-recording everything.
Being committed. Real conversations contain real data. A format with a defined place for redaction gets redacted; one without gets a customer’s address in the repository.

The shape

One file per scenario, named for the scenario. A small header, then a flat list of turns where each turn says who produced it and what it contained.

{
  "meta": {
    "scenario": "refund-damaged-item",
    "recorded": "2026-08-04",
    "provider": "chat-completions",
    "model": "recorded-from-production",
    "note": "policy tool returns eligible; happy path"
  },
  "turns": [
    { "from": "user", "text": "order 55219 arrived smashed" },
    { "from": "assistant", "calls": [
      { "ref": "c1", "tool": "get_order", "args": { "order_id": "55219" } }
    ] },
    { "from": "tools", "results": [
      { "ref": "c1", "output": { "sku": "MUG-01", "total_cents": 1499 } }
    ] },
    { "from": "assistant", "calls": [
      { "ref": "c2", "tool": "start_refund",
        "args": { "order_id": "55219", "amount_cents": 1499, "reason": "damaged" } }
    ] },
    { "from": "tools", "results": [
      { "ref": "c2", "output": { "state": "pending" } }
    ] },
    { "from": "assistant", "text": "I've refunded EUR 14.99." }
  ]
}

Four decisions are doing the work. args is a parsed object even though Chat Completions transmits a string, because a JSON string inside JSON is unreadable in a diff and the adapter can re-encode it in one line. ref is c1, not call_9xKq2LmR, so ids are stable across recordings. Tool results are their own turn rather than being attached to the call, so the two sides of the exchange are separately editable. And meta.note is prose stating what this scenario is for — the field that stops a fixture being deleted during a clear-out because nobody knew what it covered.

Pretty-print with two spaces and keep object keys in a stable order. Whether you sort keys or preserve insertion order matters less than that the recorder always does the same thing, because an unstable ordering makes every re-record a whole-file diff.

What gets stripped, and why each one

Provider call ids — random per run, so they turn every recording into a full-file change. Replace with c1, c2 in order of appearance.
Timestamps and request ids — same reason. If a turn genuinely depends on time, that is a value the test should inject, not a value the fixture should carry.
Token counts and latency — they move with model versions and belong in a cost record, not a behavioural fixture. A fixture that fails because a count changed by three tokens teaches everyone to ignore fixture failures.
Personal data — replaced with stable pseudonyms, not deleted, because a field that disappears changes the shape the loop is tested against. Run the redaction in the recorder so it cannot be forgotten, and assert in a test that no fixture matches your obvious patterns.
Keys and tokens — never present in a tool argument in the first place, and a recorder that refuses to write a file containing one is a cheap second line of defence.

What is kept: whitespace and casing exactly as the model produced them. Normalising those loses the thing you would want to see if a prompt change started producing differently formatted arguments.

One fixture, several tests

The reason to separate model turns from tool results is that most of the value comes from recombining them. One recorded conversation supports a happy-path test that uses both sides; a failure test that keeps the assistant turns and substitutes an error for one result; a truncation test that keeps everything and shortens one output past your cap; a redaction test that adds a secret to one output and asserts it never reaches the prompt.

import fixture from "./fixtures/refund-damaged-item.json";

// Same conversation, one result replaced.
const withPolicyOutage = {
  ...fixture,
  turns: fixture.turns.map((t) =>
    t.from === "tools"
      ? { ...t, results: t.results.map((r) =>
          r.ref === "c2"
            ? { ...r, output: { error: "upstream_timeout" }, is_error: true }
            : r) }
      : t),
};

Derive variants in the test rather than committing four near-identical files. A committed variant drifts from its parent the first time somebody re-records only one of them, and then two fixtures claim to describe the same conversation and disagree.

Resist putting expectations in the fixture. It is tempting to keep the assertions next to the data — a list of tools that must have run, a final string that must appear — and it collapses the distinction the whole format rests on. A fixture describes what happened; a test describes what should happen. Merge them and re-recording quietly rewrites your expectations, which is how a suite becomes worthless without anyone noticing that it has.

Re-recording without losing the diff

Re-recording is the moment a fixture suite either keeps its value or quietly loses it. Make it a deliberate command with the scenario named — never a flag that rewrites everything that failed, which is how a real regression gets committed as an update.

Re-record one scenario at a time, named explicitly, into the same file.
Read the diff. If the only changes are ids or timestamps, your stripping is incomplete; fix the recorder rather than accepting the noise.
For every genuine change, decide whether it is an improvement or a regression before committing. This is the entire point of the format: the diff has to be small enough that the question is answerable.
Update meta.recorded and, if the behaviour changed, the note. A note describing behaviour the fixture no longer contains is worse than no note.

Keep the fixtures in the repository rather than in object storage. They are small, they are the thing a reviewer needs to see next to the code change, and a fixture nobody can read without credentials is a fixture nobody reads. The same argument applies at larger scale to replayed production traffic, where the volume forces a different answer and the review problem does not go away.