The hardest part of building with LLM agents isn't writing the agent. It's the Tuesday afternoon when one misbehaves in production and you have to work out why. The run was non-deterministic, so you can't just re-run it. The logs are spread across your app, your model provider, and your vector DB. And the inputs that actually produced the bad output are already gone.
I hit this loop enough times that I built a tool for it. Retrace records an agent run, lets you replay it step by step like a video, and lets you fork from the step that broke to test a fix — re-running only the part that changed. This post is a walk through the workflow, the design decisions behind it, and where it's still rough. It's open source (MIT) and I built it solo, so the "still rough" section is honest.
Why are agents so hard to debug?
A stack trace tells you the line that threw. An agent failure is rarely that tidy. The real cause is usually one of:
- a tool call three steps back that returned junk,
- a prompt that slowly drifted off-task across turns,
- a retrieval that pulled the wrong context,
- or a model call that was fine in isolation but wrong given the state.
By the time the bad answer surfaces, the chain that produced it is buried. Print statements don't help either, because the next run takes a different path. You need the actual run preserved, not a summary of it.
How do you record a run without rewriting your agent?
You wrap your agent once. In Python it's a decorator; in TypeScript it's a wrapper. After that, every model call, tool call, and error is captured automatically as a span on a trace. Calls to OpenAI, Anthropic, and Gemini are instrumented for you, so there's no manual logging to maintain.
The capture path is built to not lose data and to not get in your way:
- an offline buffer queues events and flushes on reconnect,
- transport falls back WebSocket → HTTP → OpenTelemetry if the socket drops,
- PII (emails, keys, tokens) is redacted before anything is stored,
- a sample rate keeps high-volume agents affordable.
Once recorded, the run becomes an interactive timeline you can play, pause, and scrub. That alone changed how I debug: I stopped guessing from logs and started watching where the run actually went wrong.
How does forking actually work?
This is the part I use most. You pick the step that went wrong, change its input, and Retrace re-executes the agent from that point forward. Everything before the fork replays from the recording, so it costs almost nothing. Only the part that genuinely diverges makes real model calls.
Then you get a side-by-side diff of the original run versus the fork, down to the first step where they differ. So instead of "I think this prompt change helped," you get a concrete before/after on the exact run that failed.
Can you prove a fix worked before shipping?
Forking gets you a comparison; proving is the next step. You can re-run a proposed change against the failed run and get a verdict: improved, regressed, unchanged, or inconclusive. There's a bisect that binary-searches the run to find the exact step whose change flips the result, and a way to apply one fix across many failed runs at once to check it holds beyond a single example.
If you want this in CI, there's an eval gate: score a run with an LLM judge against your own criteria, and the gate exits non-zero below your threshold, so a regression fails the build instead of reaching users.
- name: Eval Gate
run: retrace eval gate --evaluation $EVAL_ID --trace $TRACE_ID --threshold 0.8
What's under the hood?
- Python and TypeScript SDKs, published to PyPI and npm.
- A Fastify + PostgreSQL backend, with pgvector for semantic search over past runs (find a span or a past failure by meaning, not just by id).
- Buffered ingestion with backpressure, so a burst of traffic degrades gracefully instead of overloading the datastore.
- OpenTelemetry ingest, so you can push spans from any language even without the SDK.
Where it's still rough (the honest part)
I built this solo with an AI-assisted workflow, and it's early. Replay re-calls models live, so some divergence reflects provider non-determinism rather than a real regression — you treat diffs as signals to inspect, not gospel. The docs have gaps. And I haven't stress-tested every framework people use.
That's exactly why I'm posting. If you run agents in production: what part of debugging a bad run wastes the most of your time today — reproducing it, finding the root cause, or proving the fix? I want to build against real workflows, not my assumptions. If you try it and it breaks, telling me where is the single most useful thing right now.
Retraceai-tech
/
retrace-sdk
Python & TypeScript SDKs for Retrace — the execution replay engine for AI agents.
Record every LLM call, tool invocation, and error your agent makes. Replay it step-by-step like a video. Fork from any point, change the input, and watch the whole agent re-execute down a new path. Share any run as an interactive, public link.
Python · TypeScript · How It Works · Docs
Python
pip install retrace-sdk
import retrace
retrace.configure(api_key="rt_...")
@retrace.record(name="research-agent", resumable=True)
def run_agent(prompt: str):
plan = call_planner(prompt) # captured automatically
results = call_tools(plan) # captured automatically
return summarize(results) # captured automatically
run_agent("What changed in vector databases this year?")
See python/README.md for the full reference.
TypeScript
npm install retrace-sdk
import { configure, trace } from "retrace-sdk";
configure({ apiKey: "rt_..." });
const…
Top comments (0)