Replay or It Didn't Happen: Making Agent Runs Reproducible

An agent fails in production. You have the transcript: the model's reasoning, the tool names, the arguments, the outputs. You read it and the bug is obvious — the agent called search_orders with the wrong customer ID. You fix the prompt, redeploy, and the failure never reproduces. Six weeks later it happens again, with a different tool.

The transcript told you what happened. It did not let you make it happen again. Those are different artifacts, and most agent stacks only ship the first one.

The transcript is not the trace

The transcript is what the model said. The trace is everything the process observed: tool arguments before dispatch, raw responses before parsing, latency, retry counts, and the exact model/prompt/tool-schema versions in play. You need the second to reproduce, and you need it recorded at the boundary the model does not control.

That boundary is the tool function. Record there and you get three things at once: a deterministic stub for replay, an input for offline evals, and a diffable record of what your agent actually did versus what the prompt claims it does.

Record at the call site, not in the framework

Framework-level tracing (LangSmith, Langfuse, OpenTelemetry spans) is good for observability but usually records post-processed results. If your tool returns a dataclass and the recorder serializes it after your parser ran, you have lost the ability to test the parser.

Here is a dependency-free record/replay layer. It is short enough to read in one sitting and complete enough to run.

# replay.py — record/replay at the tool boundary. No dependencies.
import json, hashlib, functools, pathlib

TRACE = pathlib.Path("traces/run.jsonl")
MODE = "record"  # "record" | "replay" | "strict-replay"

def _key(tool_name: str, args: dict) -> str:
    # sort_keys: dict ordering must not change the hash.
    payload = json.dumps({"t": tool_name, "a": args}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

def _load() -> dict:
    if not TRACE.exists():
        return {}
    rows = (json.loads(line) for line in TRACE.read_text().splitlines())
    # Last write wins: a retried call records twice, replay uses the final result.
    return {r["key"]: r["result"] for r in rows}  # cache this per run in prod

def tool(fn):
    @functools.wraps(fn)
    def wrapper(**kwargs):
        key, recorded = _key(fn.__name__, kwargs), _load()

        if MODE == "record":
            result = fn(**kwargs)
            with TRACE.open("a") as f:
                f.write(json.dumps({"key": key, "tool": fn.__name__,
                                    "args": kwargs, "result": result}) + "\n")
            return result

        if key not in recorded:
            # Strict mode is the point: an unrecorded call means the agent
            # took a path your trace never covered. Fail loudly.
            if MODE == "strict-replay":
                raise RuntimeError(f"unrecorded call: {fn.__name__}({kwargs})")
            return fn(**kwargs)

        return recorded[key]
    return wrapper

@tool
def http_get(url: str) -> dict:
    import urllib.request
    with urllib.request.urlopen(url, timeout=10) as r:
        return json.loads(r.read())  # real I/O; only reached in record mode

Write a header line into run.jsonl before any tool calls: prompt hash, model ID and version, tool schema hash, and a timestamp. Without it you cannot tell whether a replay divergence is a bug or a prompt change, and you will spend a day finding out the hard way.

Three replay modes and what each one hides

• Record executes for real. Everything downstream is a side effect you now own — see the pitfalls.
• Strict replay fails on any unrecorded call. This is what you want in CI: it tells you the agent explored a new branch, which is either the fix you shipped or the regression you just introduced.
• Loose replay falls through to live I/O for misses. Useful against staging, dangerous against production, and it makes tests silently non-deterministic when a cache expires. I have seen a "passing" eval suite that was quietly hitting a live API for 30% of its calls.

Default to strict. Opt into loose per test, with the miss count asserted.

Normalizing keys without lying

The hash is only as good as the arguments. A few things break it:

• Timestamps and UUIDs in args. Agents love since=datetime.now(). Every replay is a miss. Inject a frozen clock instead of stripping the field — stripping it hides the ordering bug you were trying to catch.
• Unordered collections. json.dumps(sort_keys=True) sorts dict keys, not lists. Sort set-derived lists before they reach the tool.
• Floats and serialization drift. 0.1 + 0.2 versus 0.3 produces different hashes across languages and JSON encoders. Round at the tool signature, not in the recorder.

If you normalize by deletion, write down what you deleted in the trace header. Six months later that comment is the only thing standing between you and a "why does replay pass but prod fail" afternoon.

What replay cannot fix

Replay makes the tool layer deterministic. It does nothing for the model. Even with temperature=0, providers change weights, quantization, and routing behind a stable model name. Your recorded run is a snapshot of one sample from a distribution, and replaying it proves the code around the model still works — not that the model will behave the same tomorrow.

That gap is why you still need fresh eval runs against the real model on a schedule, seeded with the traces that failed. Replay is the regression harness; live eval is the drift detector. Teams that run only one of them are surprised by a different class of bug every quarter.

It also does not fix the world moving underneath you. A replayed fetch returns yesterday's page. That site may have since started blocking automated clients in its robots.txt or terms — a real and growing constraint — and your replay will happily show green. Keep a small, scheduled live-canary set for anything touching third-party endpoints, and treat a replayed response as evidence about your code, never about the remote service.

Pitfalls

• Recording side effects as if they were reads. A replay that re-runs send_email in loose mode is an incident, not a test. Split reads from writes and only replay reads; gate writes behind an explicit dry-run flag.
• Traces as a PII landfill. Raw tool args and responses contain customer data, tokens, and full documents. Redact at write time, set a retention window, and treat traces/ as production data with access control. A trace directory in a public repo is a breach.
• Stale stubs masking real breakage. A recorded 200 from an API you since migrated away from will keep passing. Version the trace alongside the tool schema hash and invalidate on change.
• Unbounded trace files. One JSONL per run, rotated, not one global append-only file that grows until _load() takes 400ms per call.
• Recording inside the sandbox without a policy. If you run fleets, do it in a runtime where egress and filesystem writes are already constrained (NVIDIA's OpenShell is one example of that shape); a recorder that can write anywhere is a second problem, not a solution.

Monday checklist

• [x] Trace written at the tool boundary, before parsing, not after.
• [x] Header line per run: prompt hash, model ID + version, tool schema hash.
• [x] strict-replay is the default in CI; loose mode is opt-in and asserted.
• [x] Frozen clock and seeded IDs injected, not stripped from args.
• [x] Writes are separated from reads; no side-effecting tool is replayable.
• [x] Redaction applied at write time; retention window set; traces/ access-controlled.
• [x] Traces invalidated when the tool schema hash changes.
• [x] A scheduled live-eval run against the current model, seeded from failed traces.
• [x] A small live canary for third-party endpoints that can change their terms.