Record Once, Replay Forever: Cassette Tests for Flaky Agent Tool Calls

An agent fails a task on Tuesday morning. You rerun the identical task on Wednesday and it passes. No commit landed in between.

That is not a fix. It is a new experiment with different inputs. Retrieval changed, rate limits reset, and sampling took another path.

Cassette testing closes that gap. You record the tool boundary once, then replay it a hundred times. The idea comes from HTTP recording, and it transplants to agents with little friction.

Live reruns destroy the evidence you need

Four things change between two live runs of the same task:

• Retrieval returns a different top document.
• A rate limit turns a 200 into a 429.
• Sampling picks a different tool on step four.
• The prompt embeds today's date or a fresh run id.

Each one rewrites the trace. A failure you cannot reproduce is an anecdote, not a bug report.

Freeze the tool boundary, not the model

Model output is stochastic on purpose. Tool arguments and responses are plain data. Data can be frozen, diffed, and replayed.

So record the dispatcher. Keep the model live if you prefer, or stub it. The interesting failure usually sits downstream of a tool result.

Cassette format: one JSONL file per run

Each line is one tool call. Keep both the readable arguments and a digest.

{"seq":1,"tool":"search_docs","args":{"q":"retry budget","top_k":5},"args_digest":"sha256:9f2c41ab","response":{"hits":3,"top":"retry-budget.md"}}
{"seq":2,"tool":"read_file","args":{"path":"docs/retry-budget.md"},"args_digest":"sha256:41ab07de","response":{"bytes":812,"sha":"sha256:0d1e..."}}

seq keeps ordering honest. args_digest is the lookup key. args exists so humans can read the file during review.

A replay dispatcher with zero dependencies

Standard library only, so it runs on any small box.

# cassette.py — stdlib only, Python 3.11
import hashlib, json
from pathlib import Path

def digest(args):
    raw = json.dumps(args, sort_keys=True, separators=(',', ':')).encode()
    return 'sha256:' + hashlib.sha256(raw).hexdigest()[:16]

class Cassette:
    def __init__(self, path):
        self.path = Path(path)
        lines = self.path.read_text().splitlines()
        self.entries = [json.loads(l) for l in lines if l.strip()]
        self.index = {(e['tool'], e['args_digest']): e for e in self.entries}
        self.hits, self.misses = {}, []

    def call(self, tool, args):
        key = (tool, digest(args))
        if key not in self.index:
            self.misses.append({'tool': tool, 'args_digest': key[1]})
            raise KeyError(f'cassette miss: {tool} {key[1]}')
        self.hits[key] = self.hits.get(key, 0) + 1
        return self.index[key]['response']

    def report(self):
        repeats = {f'{t}:{d}': n for (t, d), n in self.hits.items() if n > 1}
        return {'entries': len(self.entries), 'used': len(self.hits),
                'repeats': repeats, 'misses': self.misses}

The record side is one function. Wire it into your tool dispatcher and never touch it again.

def record(sink, seq, tool, args, response):
    row = {'seq': seq, 'tool': tool, 'args': args,
           'args_digest': digest(args), 'response': response}
    sink.write(json.dumps(row, sort_keys=True) + '\n')

Misses raise on purpose. A silent fallback to live calls would hide exactly the drift you are hunting. Treat these snippets as a starting point and run them against your own dispatcher first.

Three mismatch signatures, three diagnoses

Symptom	Likely cause	First action
Miss on arguments	Timestamps, uuids, or absolute paths in args	Normalize args before digesting
Same key hit many times	Retry storm or agent loop	Compare counts against recorded seq
Replay passes, live fails	Recorded response is stale	Re-record, then diff both cassettes

The third row matters most. It tells you the world moved, not your code.

The debug loop, end to end

1. Record the failing run once, with network access.
2. Replay it twenty times with networking disabled.
3. Confirm the pass rate is zero or one, never fifty percent.
4. Diff the cassettes from a failing and a passing run.
5. Freeze the offending key as a fixture.
6. Re-record after the fix and keep both files in review.

# 1. record one failing run
python agent.py --task retry-budget --cassette runs/fail.jsonl

# 2. replay it 20 times, no network
for i in $(seq 20); do python agent.py --task retry-budget --replay runs/fail.jsonl; done

# 3. re-record after a fix and diff the two
diff <(jq -S 'del(.recorded_at)' runs/fail.jsonl) \
     <(jq -S 'del(.recorded_at)' runs/fix.jsonl)

Step three is the gate. If replay is still flaky, your non-tool layers are nondeterministic. Fix those before blaming retrieval.

Normalize before you digest

• Sort keys and strip transport-only fields.
• Drop timestamps, uuids, and request ids.
• Relativize file paths to the repo root.
• Keep semantic arguments intact, including filters and limits.

Over-normalize and unrelated calls collide. Under-normalize and every run misses. Start strict, then loosen one rule at a time.

What this method cannot do

• It does not capture model nondeterminism. Sampling stays live unless you stub it.
• It does not preserve timing or load. Failures that need concurrency will not appear.
• It can leak secrets. Redact responses before the cassette enters a repository.
• It struggles with side effects. Writes, payments, and deploys need a dry-run mode.

Skip this approach if your failures depend on live ranking quality or on wall-clock behavior. It also fits poorly when tools are non-idempotent and cannot be stubbed safely.

Run the loop on the cheapest box you have

Replay is CPU work. Only recording touches a model. Split the two and most debugging stops competing for a budget.

MonkeyCode, an open-source project per the operator, offers free model access and a free server option. That split matches this loop: record once on the hosted server, then replay the cassette anywhere, including CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Quota figures move, so treat any number as a snapshot. The operator cites a 10M-token allowance; confirm the current figure on the project page before planning a large record batch.

If you want to test the idea without provisioning anything, record one cassette on the free server and check whether twenty replays give you the same failure.

Ship checklist

• A cassette file per failing run, committed with the trace.
• A replay command that works with networking disabled.
• One fixture pinned to the historical bug.
• A redaction pass before every commit.
• A note in the PR that replay is deterministic, live is not.

The cassette is the artifact. The model stays a variable, and that is fine.