Production failed on Friday. My final transcript looked clean. The agent answered, cited sources, and summarized. The raw trace told a different story. It called the same endpoint three times with stale arguments.
Re-running the agent wasted tokens and time. Replaying the trace took seconds. I built a diff-first replay harness. Logs became the source of truth for debugging.
This post shows how to replay agent runs from JSONL traces. It also shows where a free server and a free model allowance fit in the loop.
Why re-running is the wrong default
Re-running an agent is a roll of the dice. Temperature, tool latency, and cached state change every run. You pay tokens for each attempt. You also need live credentials and network access.
Replaying from logs removes all three costs. The run becomes a static file. You inspect diffs instead of rerunning fate. Deterministic. Offline. Fast.
Replay cannot fix missing logs. If you did not trace it, you cannot replay it. Start with logging. Everything else is downstream.
The trace schema I replay
Each line in my trace JSONL is one tool call. The schema is minimal. It stores run ID, step, tool name, arguments, an output hash, token count, and a timestamp. Enough to rebuild the call sequence and compare runs.
{"run_id": "run_0042", "step": 1, "tool": "search_issues", "args": {"q": "broken:true", "page": 1}, "output_sha": "a1b2c3", "tokens": 812, "ts": "2026-08-31T09:12:04Z"}
{"run_id": "run_0042", "step": 2, "tool": "search_issues", "args": {"q": "broken:true", "page": 1}, "output_sha": "a1b2c3", "tokens": 812, "ts": "2026-08-31T09:12:09Z"}
{"run_id": "run_0043", "step": 1, "tool": "search_issues", "args": {"q": "broken:true", "page": 1}, "output_sha": "d4e5f6", "tokens": 817, "ts": "2026-08-31T09:20:11Z"}
Run 0042 repeated the same call twice within five seconds. Run 0043 returned a different output hash for identical arguments. Both patterns are visible without re-executing anything.
The replay script
replay_diffs.py parses the trace and groups calls by argument signature. It flags identical repeats and cross-run drift. Then it replays each unique call against a mock executor.
import hashlib, json
from collections import defaultdict
def sig(call):
return hashlib.sha256(
json.dumps(call["args"], sort_keys=True).encode()
).hexdigest()
def load_trace(path):
with open(path) as fh:
return [json.loads(line) for line in fh if line.strip()]
def diff_runs(run_a, run_b):
a = load_trace(run_a)
b = load_trace(run_b)
by_sig = defaultdict(list)
for call in a + b:
by_sig[sig(call)].append(call)
for key, calls in by_sig.items():
hashes = {c["output_sha"] for c in calls}
if len(hashes) > 1:
print(f"drift: {calls[0]['tool']} -> {len(hashes)} distinct outputs")
Sample output:
$ python replay_diffs.py trace_run_0042.jsonl trace_run_0043.jsonl
repeat: run_0042 step 1 == step 2 (same args, same output)
drift: search_issues -> 2 distinct outputs
replay: 3 unique calls, 1 mock mismatch
The script costs one SHA-256 per call. No model inference. No network. No credentials. It runs in milliseconds on a laptop.
Where the free tier actually helps
Replay is cheap. Shipping traces and classifying diffs is where costs accumulate. A trace collector must run 24/7. A summarizer needs model tokens to label each diff.
MonkeyCode's free tier covers both stages in this setup. The project is open source. The current allowance includes 10M tokens, and the free server hosts the collector.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The collector is a 40-line FastAPI app plus SQLite. It stores every trace line. One endpoint returns the diff list between two runs. Check current terms before putting any free tier behind production traffic. Free allowances change.
Summarize diffs with one LLM call
Do not read hundreds of diff lines by hand. Send them to a summarizer with a strict output format. Each diff returns one of three severities: expected, suspicious, or fatal.
You receive a list of tool-call diffs between two agent runs.
Return one line per diff: pattern, severity, suggested fix.
Allowed severities: expected, suspicious, fatal.
This is the only token-consuming stage. In the sample above, summarization cost a few hundred tokens. It is the cheapest part of the loop.
Decision table
Use this table when replay flags a diff.
| Pattern | Evidence | Fix |
|---|---|---|
| Same args, same output, repeated | Retry loop or duplicate call | Cap tool calls per step |
| Same args, different output | External data changed | Add freshness checks to context |
| Different args, same tool | LLM exploring too much | Tighten tool budget or prompt |
| Tool count spikes per step | Agent browsing instead of deciding | Reduce available tools |
Apply fixes in severity order. fatal first. Re-trace after each fix. The comparison window stays small, so feedback arrives in minutes.
The debug loop
- Run the agent with JSONL trace logging enabled.
- Ship each line to the collector on the free server.
- Run
replay_diffs.pyagainst the two latest runs. - Ask the summarizer to classify all diffs.
- Fix the most severe diff. Re-trace. Compare again.
Expect one or two iterations per defect class. The loop treats symptoms as data, not as failures.
Limitations
Replay only works if the trace is complete. Missing tool calls become invisible bugs. Mock executors drift from real APIs. Keep the mock minimal and update it with real responses. The summarizer can misclassify. Review fatal labels before changing code.
Who should skip this approach? Teams with stateless single-shot scripts. If the agent never loops and never mutates state, replay adds ceremony. A plain log file is enough.
Who benefits? Anyone running agents with tool budgets, retries, or external lookups. That is most agent workloads in production today.
Start with one failed run. Add trace logging. Replay instead of re-running. The diff will show where the agent went wrong.
Top comments (2)
The tricky edge case with comparing argument SHAs across runs is auto-generated values like request UUIDs or timestamps slipping into the payload. The hash diverges even when the agent took the identical path.
The other pattern that bit me on trace replays was cascading mutations. If step two writes a file or modifies a record, step four's read output hash changes between runs. If the trace tags whether a tool is read-only or state-mutating up front, you can immediately tell whether a downstream diff is real external drift or just the expected echo of an earlier local write.
Hello Glad to see you, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.
This is exactly the direction agent observability needs. I especially like treating the trace as the source of truth instead of repeatedly invoking a nondeterministic agent.
I would push the architecture further by making each tool invocation an immutable event with a canonicalized argument hash, parent step ID, dependency metadata, latency, token accounting, and response fingerprint. Then replay can reconstruct the execution DAG rather than only comparing sequential calls.
For drift detection, I would distinguish deterministic tool drift from environmental drift. Store normalized response snapshots and schema versions, then compare semantic fields rather than relying exclusively on output hashes. You can also build invariant checks for duplicate calls, unexpected tool fanout, budget exhaustion, and state mutation.
The strongest extension would be automatic regression gates in CI. A known trace becomes a fixture, replay produces a diff, and deployment fails when critical behavioral invariants regress. That turns debugging infrastructure into an agent reliability system.
Excellent practical approach. I would genuinely enjoy collaborating on this kind of agent evaluation architecture.