A Trace You Cannot Replay Is Only a Log

Most agent debugging loops are expensive in the worst possible way. You re-run the entire agent to see a bug a second time, paying model tokens, tool calls, and a fresh layer of nondeterminism for the privilege. A recorded trace tells you what happened once. It rarely tells you what will happen after your fix.

That gap is where releases stall. Teams keep hardening their tracing — span cardinality limits, causal ordering, closed-span gates — and still cannot answer the only question blocking the merge: did my change alter the agent's behavior, or did the world move underneath it? The moment a trace can be replayed offline, it stops being a log and becomes a test.

The loop has three parts. Record the boundaries that actually decide behavior, replay them without touching a model endpoint, and treat the first replay miss as your diff.

Three boundaries decide almost everything

Almost every agent bug I have chased came down to a model request, a tool call, or an environment read. Everything else — the span tree, the log lines, the token counters — is commentary on those three. Capture the inputs to those boundaries and their outputs, and you have captured the run.

The trick is keying. A cassette is a map from a canonical hash of the boundary input to the observed output. Hash the message list, the tool schemas, the decoding parameters, and the step index; that key identifies the decision. If the agent reaches the same call with a different prompt, or at a different position in the loop, the key changes and replay fails loudly instead of quietly passing.

Here is a reference implementation. It is compact on purpose, and it is a reference rather than a drop-in: I have not run it against your stack, so treat the shape as the artifact and adapt the edges.

# tape.py — record/replay harness for agent runs (reference implementation)
import hashlib, json, time

class ReplayMiss(RuntimeError):
    """The run asked for a decision that was never recorded."""

def _key(kind: str, payload: dict) -> str:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str)
    return f"{kind}:{hashlib.sha256(blob.encode()).hexdigest()[:16]}"

class Tape:
    def __init__(self, path: str, mode: str = "record"):
        assert mode in ("record", "replay")
        self.mode, self.path = mode, path
        self.queues: dict[str, list] = {}
        if mode == "replay":
            with open(path) as fh:
                for line in fh:
                    rec = json.loads(line)
                    self.queues.setdefault(rec["key"], []).append(rec)
        else:
            self.fh = open(path, "a")

    def call(self, kind: str, payload: dict, do):
        k = _key(kind, payload)
        if self.mode == "record":
            t0 = time.monotonic()
            out = do()
            self.fh.write(json.dumps({
                "key": k,
                "kind": kind,
                "ms": int((time.monotonic() - t0) * 1000),
                "out": out,
            }) + "\n")
            self.fh.flush()
            return out
        queue = self.queues.get(k)
        if not queue:
            raise ReplayMiss(f"{k} was never recorded — the run diverged here")
        return queue.pop(0)["out"]

The agent loop then routes its two expensive calls through the tape and stays otherwise unchanged.

def run(task, tape, client, tools, max_steps=12):
    msgs = [{"role": "user", "content": task}]
    for step in range(1, max_steps + 1):
        schemas = sorted(tools)
        decision = tape.call(
            "llm",
            {"messages": msgs, "tools": schemas, "step": step},
            lambda: client.chat(messages=msgs, tools=schemas),
        )
        if decision["tool"] is None:
            return decision["content"]
        name, args = decision["tool"], decision["args"]
        result = tape.call(
            "tool",
            {"name": name, "args": args, "step": step},
            lambda: tools[name](**args),
        )
        msgs += [
            {"role": "assistant", "content": decision["content"]},
            {"role": "tool", "name": name, "content": json.dumps(result)},
        ]
    raise RuntimeError("step budget exhausted")

Recording a failing run costs exactly one real pass:

$ python -m agent --task "$(cat tasks/issue-412.txt)" --tape runs/412.jsonl --mode record
$ wc -l runs/412.jsonl
17 runs/412.jsonl

Seventeen decisions. That is the whole run, and it now fits in version control next to the failing task. Replay is the same command with a different mode, and no network traffic at all:

$ python -m agent --task "$(cat tasks/issue-412.txt)" --tape runs/412.jsonl --mode replay
ReplayMiss: llm:9f2c41d0a7b3e815 was never recorded — the run diverged here

That exception is the debugging signal. The cassette length tells you how far the run agreed with the recording before it diverged, and the key tells you which input changed. A prompt edit that shifts the first tool call shows up as a miss at step 2. A tool whose schema changed bumps the llm key at step 1. Neither costs a model call to discover.

One consequence of keying on the step index is deliberate strictness: a refactor that produces the same decisions in a different order fails the replay. That is usually what you want before a release, and it is noisy if you are mid-refactor. Keep a separate cassette branch for exploring rather than loosening the key.

What belongs in the cassette, and what must never be

Recording everything is easy and useless. Recording the deciding inputs is a design decision, and it is worth writing down.

Boundary	Capture	Failure it exposes
Model request	message list, sorted tool schemas, decoding params, step index	prompt or schema drift
Tool call	name, args, idempotency key, wall-clock ms	changed arguments, silent retries
Environment	allowlisted keys, values hashed	config drift between machines
Harness	git SHA of the agent loop	cassette recorded under a different code path

Redaction creates a real tension here. Secrets and user data must not land in a cassette, but a prompt that has been scrubbed produces a different key, and a different key means the recording no longer matches the run. The workable compromise is to keep the deciding input intact as a hash field and store a redacted copy beside it for humans to read. Replay uses the hash; you read the copy.

Side effects need the same care. A cassette replays tool calls by returning recorded output, so anything with an irreversible effect — a payment, a deploy, a message send — must be stubbed at the tool layer, not merely recorded. If you cannot stub it, do not route it through the tape; wrap the read path and leave the write path alone.

Limits, and who should skip this

The cassette proves your harness logic, not the model's. Providers can return different text for the same request even with a seed set, so replay is a statement about your code path rather than about inference reproducibility. If the bug lives inside sampling behavior, this loop will not find it, and no amount of tracing will.

Cassettes also rot. Every prompt template edit, tool schema change, or loop restructuring invalidates the recordings that touch it, and the honest response is to re-record rather than to loosen the key. That cost is the price of the guarantee, and it is worth paying only when the failure recurs.

Skip this approach if your runs are short and cheap enough that re-running is free, if your failures are always environmental, or if the agent's whole purpose is long-horizon drift and pinning any single decision would be misleading. A replay harness is for agents with a repeatable entry point, a bounded step budget, and at least one bug that keeps coming back.

Where free capacity changes the arithmetic

The asymmetry in this loop is that recording is expensive and replay is free. One recorded run pays the full model bill once; every later iteration on prompts, tool schemas, and routing logic runs against the cassette with zero tokens and zero tool side effects.

That is where MonkeyCode's free model access fits, and it is a narrow fit rather than a headline. The operator provides free model access and a free server option. The free access lowers the cost of the one pass that has to hit a real endpoint, and it matters most when a cassette rots and needs re-recording. The free server option matters for an adjacent reason: a cassette has to be written on the machine where the tools actually run, because that is the environment holding the credentials, the network path, and the real data shapes. The operator also states a free token allowance, currently described as ten million tokens; quotas and terms are the operator's to define and can change, so confirm them before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the ordering straight. Free access makes the recording pass cheap, and replay is what makes a cheap recording worth having. If you already keep a trace suite for agents, record one cassette for your worst recurring failure before you touch another prompt, then diff the replay instead of the logs.