Debugging AI agents by replaying the exact run, not the logs

Yash — Wed, 08 Jul 2026 20:10:17 +0000

The hardest part of building with LLM agents isn't writing the agent. It's the Tuesday afternoon when one misbehaves in production and you have to work out why. The run was non-deterministic, so you can't just re-run it. The logs are spread across your app, your model provider, and your vector DB. And the inputs that actually produced the bad output are already gone.

I hit this loop enough times that I built a tool for it. Retrace records an agent run, lets you replay it step by step like a video, and lets you fork from the step that broke to test a fix — re-running only the part that changed. This post is a walk through the workflow, the design decisions behind it, and where it's still rough. It's open source (MIT) and I built it solo, so the "still rough" section is honest.

Why are agents so hard to debug?

A stack trace tells you the line that threw. An agent failure is rarely that tidy. The real cause is usually one of:

a tool call three steps back that returned junk,
a prompt that slowly drifted off-task across turns,
a retrieval that pulled the wrong context,
or a model call that was fine in isolation but wrong given the state.

By the time the bad answer surfaces, the chain that produced it is buried. Print statements don't help either, because the next run takes a different path. You need the actual run preserved, not a summary of it.

How do you record a run without rewriting your agent?

You wrap your agent once. In Python it's a decorator; in TypeScript it's a wrapper. After that, every model call, tool call, and error is captured automatically as a span on a trace. Calls to OpenAI, Anthropic, and Gemini are instrumented for you, so there's no manual logging to maintain.

The capture path is built to not lose data and to not get in your way:

an offline buffer queues events and flushes on reconnect,
transport falls back WebSocket → HTTP → OpenTelemetry if the socket drops,
PII (emails, keys, tokens) is redacted before anything is stored,
a sample rate keeps high-volume agents affordable.

Once recorded, the run becomes an interactive timeline you can play, pause, and scrub. That alone changed how I debug: I stopped guessing from logs and started watching where the run actually went wrong.

How does forking actually work?

This is the part I use most. You pick the step that went wrong, change its input, and Retrace re-executes the agent from that point forward. Everything before the fork replays from the recording, so it costs almost nothing. Only the part that genuinely diverges makes real model calls.

Then you get a side-by-side diff of the original run versus the fork, down to the first step where they differ. So instead of "I think this prompt change helped," you get a concrete before/after on the exact run that failed.

Can you prove a fix worked before shipping?

Forking gets you a comparison; proving is the next step. You can re-run a proposed change against the failed run and get a verdict: improved, regressed, unchanged, or inconclusive. There's a bisect that binary-searches the run to find the exact step whose change flips the result, and a way to apply one fix across many failed runs at once to check it holds beyond a single example.

If you want this in CI, there's an eval gate: score a run with an LLM judge against your own criteria, and the gate exits non-zero below your threshold, so a regression fails the build instead of reaching users.

- name: Eval Gate
  run: retrace eval gate --evaluation $EVAL_ID --trace $TRACE_ID --threshold 0.8

What's under the hood?

Python and TypeScript SDKs, published to PyPI and npm.
A Fastify + PostgreSQL backend, with pgvector for semantic search over past runs (find a span or a past failure by meaning, not just by id).
Buffered ingestion with backpressure, so a burst of traffic degrades gracefully instead of overloading the datastore.
OpenTelemetry ingest, so you can push spans from any language even without the SDK.

Where it's still rough (the honest part)

I built this solo with an AI-assisted workflow, and it's early. Replay re-calls models live, so some divergence reflects provider non-determinism rather than a real regression — you treat diffs as signals to inspect, not gospel. The docs have gaps. And I haven't stress-tested every framework people use.

That's exactly why I'm posting. If you run agents in production: what part of debugging a bad run wastes the most of your time today — reproducing it, finding the root cause, or proving the fix? I want to build against real workflows, not my assumptions. If you try it and it breaks, telling me where is the single most useful thing right now.

Docs

Retraceai-tech / retrace-sdk

Python & TypeScript SDKs for Retrace — the execution replay engine for AI agents.

Record every LLM call, tool invocation, and error your agent makes. Replay it step-by-step like a video. Fork from any point, change the input, and watch the whole agent re-execute down a new path. Share any run as an interactive, public link.

Python · TypeScript · How It Works · Docs

Python

pip install retrace-sdk

import retrace

retrace.configure(api_key="rt_...")

@retrace.record(name="research-agent", resumable=True)
def run_agent(prompt: str):
    plan = call_planner(prompt)    # captured automatically
    results = call_tools(plan)     # captured automatically
    return summarize(results)      # captured automatically

run_agent("What changed in vector databases this year?")

See python/README.md for the full reference.

TypeScript

npm install retrace-sdk

import { configure, trace } from "retrace-sdk";
configure({ apiKey: "rt_..." });

const

…

View on GitHub

I built Retrace: replay and fork your AI agent runs like a video

Yash — Tue, 07 Jul 2026 21:22:54 +0000

If you've built anything with LLM agents, you know the annoying part isn't writing the agent — it's figuring out what went wrong when it misbehaves. The run is non-deterministic, the logs are scattered, and reproducing the exact failure is a pain.

I built Retrace to make that loop less miserable: record a run, replay it step by step, and fork from any step to test a fix.

How it works

You wrap your agent function once — a decorator in Python or a wrapper in TypeScript. After that, every LLM call, tool call, and error is captured automatically as a span inside a trace. Calls to OpenAI, Anthropic, and Gemini get picked up without extra instrumentation.

Then two things:

Replay — the run shows up as an interactive timeline you can play and scrub through, so you can see exactly where it broke instead of guessing from logs.
Fork — pick the step that went wrong, change the input, and Retrace re-executes the agent from that point forward. Everything before the fork replays from the recording, so you're not paying to re-run the whole thing, and you get a side-by-side diff of the original vs. the new path.

The stack

Python + TypeScript SDKs (published to PyPI and npm)
Fastify + PostgreSQL, with pgvector for semantic search over past runs
Buffered ingestion over WebSocket with HTTP and OpenTelemetry fallbacks, so a run isn't lost if the socket drops

Honest context

I built this solo, It's still early — I'd genuinely like feedback from people who run agents in production about what's useful and what's missing.

docs

If you try it, tell me where it breaks. That's the most useful thing right now.

DEV Community: Yash