DEV Community

Cover image for Time-Travel Debugging and Drift Measurement: How to Audit an AI Agent.
Mir Arshad Ali Talpur
Mir Arshad Ali Talpur

Posted on

Time-Travel Debugging and Drift Measurement: How to Audit an AI Agent.

Time-travel debugging reconstructs what an AI agent knew at a given moment, and drift measurement flags when its behavior departs from its baseline. Together they turn agent auditing from guesswork into a repeatable process. This guide covers both, with an implementation using the open-source ZizkaDB.

An auditor doesn't ask "is the agent working right now?" They ask "what did it know last Tuesday when it made this decision, and has it been behaving the same way since?" Those are two different questions, and most monitoring setups can only half-answer either of them. A dashboard shows current state. An error log shows what broke. Neither reconstructs a past moment, and neither tells you whether today's normal behavior is quietly becoming tomorrow's incident.

Time-travel debugging and drift measurement are the two capabilities that close that gap. This post covers what each one does, why agents need both, and how to implement them with the open-source ZizkaDB.

Two different questions, two different tools

Time-travel debugging answers: what did the system know, and what state was it in, at a specific point in the past? It's a reconstruction of a single moment, on demand, after the fact.

Drift measurement answers: has the system's behavior changed compared to how it used to behave? It's a comparison across many moments, watching for a trend rather than inspecting a point.

They're complementary rather than overlapping. Drift measurement tells you that something changed and roughly when. Time-travel debugging lets you go to that moment and see what changed. An audit that only drifts-detects tells you there's a problem without letting you inspect it. An audit that only replays moments requires you to already know which moment to look at. Together, they form a loop: drift detection flags the window, time-travel debugging examines it.

Why agents need both, and ordinary logging doesn't cover them

The state problem. An agent's decision depends on more than its final output: what it retrieved, what tools returned, what was in its context window, which model and prompt version was live. A line in a log shows you the output. It doesn't reconstruct the state the agent was reasoning over. Time-travel debugging is built specifically to answer "what did the agent know at time T," not just "what did it say."

The moving-target problem. Agents change without anyone touching their code. A model provider ships a silent update. A retrieval index gets new documents. Users discover new phrasing that pushes the agent into behavior nobody tested. None of these show up as a deployment event, so ordinary release-based monitoring misses them entirely. Drift measurement watches behavior itself, not deployments, which is the only way to catch this class of change.

The audit problem. An audit, whether internal, a customer's due diligence, or a regulator's request, usually starts with a specific incident or a specific date, not with "please show me everything." You need to answer for a particular moment, with confidence that the record wasn't reconstructed from memory or guesswork after the fact. That is exactly what time-travel debugging is for, and exactly what ordinary logs, which sample activity rather than reconstruct state, tend to be too thin to support.

What ZizkaDB provides

ZizkaDB is an open-source, self-hosted audit trail database for AI agents built around exactly this pair of problems. Its README lists both functions plainly:

Function What it does
db.at() Reconstruct what the agent knew at a timestamp
db.baseline() Detect when agent behavior drifts vs. past sessions

Both sit on top of the same underlying record: every agent step logged as an event, linked to the step that caused it via parent_id. That causal chain is what makes db.at() a real reconstruction rather than a snapshot, and what makes db.baseline() a comparison against real history rather than a synthetic test set.

Time-travel debugging with db.at()

The basic pattern is to log events as the agent runs, the same way you would for any audit trail:

python
import asyncio
from zizkadb import ZizkaDB

async def main():
async with ZizkaDB(host="http://localhost:8000") as db:
user = await db.log(
agent="support-bot", event="user_message",
data={"text": "Why was my order delayed?", "order_ref": "ORD-8842"},
)
lookup = await db.log(
agent="support-bot", event="tool_call",
data={"tool": "lookup_order", "order_id": "ORD-8842"},
parent_id=user.event_id,
)
answer = await db.log(
agent="support-bot", event="llm_response",
data={"model": "gpt-4o", "text": "Your order ships tomorrow."},
parent_id=lookup.event_id,
)

asyncio.run(main())

When a complaint comes in three weeks later, db.at() lets you go back to that moment and see what the agent had in front of it, rather than relying on someone's memory of what the tool probably returned:

python
state = await db.at(agent="support-bot", timestamp="2026-09-01T14:32:00Z")
state.print()

I haven't run this against a live instance, so treat the exact call signature as illustrative; check the current docs for the precise arguments db.at() accepts. The concept, however, is what matters for auditing: you're not trusting a summary written after the fact, you're replaying the actual recorded state.

Drift measurement with db.baseline()

Drift measurement needs a "normal" to compare against, which is what db.baseline() establishes from an agent's past sessions:

python
baseline = await db.baseline(agent="support-bot", window="30d")

Once a baseline exists, ongoing sessions can be checked against it, and a widening gap is your signal to look closer, using db.why() to trace a specific bad step or db.at() to inspect the state at the moment things started to shift. As with db.at(), I don't have a captured example run to show you, so verify the exact method signature and output format in the repo's docs before using this in production.

What counts as "drift" is worth being deliberate about. Useful signals include:

Tool-usage shape. An agent suddenly calling a tool it rarely used, or skipping one it always used, is often the first visible sign that something upstream changed.
Response length or structure. A support agent whose answers double in length, or whose refusal rate climbs, is behaving differently even if no single reply looks wrong.
Escalation and human-override rate. If human reviewers are stepping in more often, that's drift measured through the outcome that matters most.
Decision distribution. For an agent that classifies or recommends, watch whether the split between outcomes shifts over time, not just whether any single decision was correct.
A worked scenario: catching drift before a complaint

Say support-bot's baseline shows it resolves order-status questions using lookup_order in roughly 90% of sessions. Two weeks after a routine dependency update, db.baseline() shows that share has dropped to 60%, and human-review escalations have crept up.

That's the signal, not yet the explanation. The investigation looks like this:

Drift measurement flags the window. Something changed starting around the dependency update.
Time-travel debugging inspects it. Use db.at() on a handful of sessions from just after the update to see what state the agent was reasoning over.
Causal lineage explains the mechanism. Use db.why() on a specific bad answer to trace it back, the same technique covered in our causal lineage guide, which might show, for example, that a tool's response schema changed and the agent started silently falling back to a weaker answer path instead of calling lookup_order.
You fix the cause, not the symptom. Rather than patching individual bad replies, you fix the schema mismatch that's been quietly degrading every session since the update.

Without drift measurement, you'd likely learn about this from a spike in complaints, weeks after it started, with no easy way to find the first affected session. Without time-travel debugging, you'd know something changed but have no way to inspect the state that produced the change short of trying to reproduce it live, which frequently doesn't work against a non-deterministic system.

Why this matters for auditing specifically, not just debugging

Debugging and auditing overlap but aren't identical. Debugging is done by the team that built the agent, usually right after something breaks. Auditing is often done by someone else, on a schedule or in response to an incident, and it has to hold up as evidence, not just as an explanation.

That changes what you need from the tooling:

Reproducibility of the record, not the run. An auditor doesnn't need to re-run the agent. They need to trust that what db.at() shows them is what actually happened, not a best guess. That's why ZizkaDB pairs these functions with tamper-evident, checksum-backed event storage, so the record you're time-traveling into is verifiably the original one.
A defensible answer to "how long has this been happening?" This is precisely what db.baseline() is for. It's a materially different exercise from noticing one bad reply and hoping it was a one-off.
Compliance relevance. The EU AI Act's Article 12 asks providers of high-risk systems to keep logs that help identify situations involving risk and support post-market monitoring (Article 72). Drift measurement is close to a direct implementation of ongoing post-market monitoring. Time-travel debugging is close to a direct implementation of the record-keeping needed to investigate a flagged situation. For the full compliance picture, including human oversight and record-keeping design, see our guide to making AI agents EU AI Act compliant.
Human oversight evidence. If reviewers approve or override agent decisions, logging those as events (as covered in the causal lineage post) means db.at() can reconstruct not just what the agent did, but what a human did in response, at the same moment.
How this fits with observability

If you already run an observability tool for latency, cost, or error rates, you don't need to replace it. Span-tree observability tools are good at how the system performed. Time-travel debugging and drift measurement, as ZizkaDB frames it, are built to answer why a specific decision happened and whether behavior has changed, using an explicit causal record and Postgres-backed storage you control rather than a hosted trace pipeline. Many teams will want both: performance monitoring for operations, and an audit trail for the questions that come from outside the engineering team.

Getting started

Requires Docker; the first image pull can take 5 to 10 minutes.

bash
curl -fsSL https://raw.githubusercontent.com/Zizka-ai/ZizkaDB/main/scripts/quickstart-remote.sh | bash

Read the script before piping it into a shell, as you would with any installer. Or self-host from a clone:

bash
git clone https://github.com/Zizka-ai/ZizkaDB.git && cd ZizkaDB
bash scripts/setup-local.sh

That gives you the API at localhost:8000, the dashboard at localhost:3001, and Swagger docs at localhost:8000/swagger. From there:

Instrument one agent's sessions with db.log() and parent_id, as in the causal lineage guide.
Let it run long enough to establish a meaningful baseline, then call db.baseline().
Pick a past session and call db.at() on it, to confirm the reconstruction shows what you expect.
Build a habit of checking drift on a schedule, and treat a widening gap as a trigger to time-travel into the affected sessions, not just a number to note and move past.

ZizkaDB is open source under AGPL-3.0 (the MCP server is MIT), with native support for Python, TypeScript, LangChain, CrewAI, LiveKit, and MCP, plus an optional managed cloud at db.zizka.ai if you'd rather not run infrastructure. Code and docs are on GitHub.

Conclusion

An agent that can't be time-traveled into can't really be audited, only guessed about. An agent that isn't measured for drift can only be audited after someone notices something's wrong. Used together, the two turn "we think it's fine" into "here's what it knew, here's how its behavior has moved, and here's the moment things changed." That's the actual bar an audit has to clear, and it's the reason these two capabilities, more than dashboards or error rates, are what make an agent auditable rather than merely monitored.

Top comments (0)