Time-Travel Debugging for AI Agents with LangGraph.js Checkpoints

#typescript #langgraph #agents #debugging

Book: AI That Plans
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Normal debugging assumes you can reproduce the bug. Run it again, get the same
behaviour, narrow it down.

An agent does not cooperate. The run that emailed the wrong customer happened
once, at 3am, after eleven turns of accumulated state you cannot recreate by
re-running the same input. Re-running gives you a different run.

But the state at every step was written down. A checkpointer is not only crash
recovery — it is a recording, and you can restart from any frame of it.

Every step is addressable

const config = { configurable: { thread_id: "run-8812" } };

for await (const s of graph.getStateHistory(config)) {
  console.log({
    step: s.metadata?.step,
    next: s.next,
    ckpt: s.config.configurable?.checkpoint_id,
    keys: Object.keys(s.values),
  });
}

Each snapshot carries its own config with a checkpoint_id. That config is
the address of a moment in the run, and passing it back is what replays from
there.

const history = [];
for await (const s of graph.getStateHistory(config)) history.push(s);
history.reverse();                        // oldest first

const before = history.find((s) => s.next.includes("send_email"));
await graph.invoke(null, before.config);  // re-run from just before the email

invoke(null, checkpointConfig) continues from that checkpoint with the state
exactly as it was. No input, no restart — resumption at a chosen frame.

Change one thing and see what happens

Replay alone tells you what happened. The useful move is replaying with an
edit, which is how you test a hypothesis.

const forked = await graph.updateState(
  before.config,
  { customerEmail: "correct@acme.com" },   // the value you suspect was wrong
);

const result = await graph.invoke(null, forked);

updateState writes a new checkpoint descending from that one and returns its
config. The original history is untouched, so you can fork the same frame
repeatedly with different values.

That is the closest thing to a controlled experiment available on a system
whose core component is nondeterministic. Was it the wrong email in state, or
the model misreading a correct one? Fork, correct the state, replay. If the
email is right this time, the bug is upstream in whatever wrote that field.

Replaying is not free, and it is not pure

Two things to hold onto before doing this against anything real.

Replaying re-executes nodes. Every model call after the fork point runs
again and is billed again. Every tool call runs again — including the one that
sends the email you were investigating.

So fork into a context where side effects are inert:

const debugCtx = {
  ...ctx,
  tools: wrapAll(ctx.tools, {
    mode: "dry-run",
    onWrite: (name, args) => log.info("suppressed write", { name, args }),
  }),
};

A dry-run wrapper that lets reads through and records writes without
performing them. Without it, debugging a run that sent a wrong email sends
more wrong emails.

The model may behave differently. Same state, different sampling. If the
forked run does the right thing, that is weak evidence — it might have done
the right thing originally too, one time in five. Fork several times before
concluding.

Keeping the recording useful

Checkpoints are only a debugger if they are still there when you need them.

-- keep failures long, successes briefly
DELETE FROM checkpoints
WHERE thread_id IN (
  SELECT thread_id FROM runs
  WHERE outcome = 'complete' AND finished_at < now() - interval '3 days'
);

DELETE FROM checkpoints
WHERE thread_id IN (
  SELECT thread_id FROM runs
  WHERE outcome <> 'complete' AND finished_at < now() - interval '30 days'
);

Retention by outcome. Successful runs are rarely investigated; failed ones are
exactly what you want a month later when a pattern emerges.

Size is the constraint. Every checkpoint holds the message window, so a long
run writes a large row per step. Compacting the window before checkpointing
keeps this affordable, and it is worth checking the table size early rather
than discovering it as a disk alert.

Make a thread id findable

Time travel is useless if you cannot locate the run. The thread id has to
reach everything a person might start from:

logger.info("agent", { threadId, runId, userId, outcome });
res.setHeader("X-Run-Id", threadId);
await db.message.create({ data: { ...msg, threadId } });

Deriving it from a domain id rather than a random uuid pays off here —
refund-ord_8812 is greppable from a support ticket. A uuid means someone has
to join through three tables first.

What this replaces

The habit it displaces is adding log lines and waiting for the bug to recur.
On a system where a run costs money and recurrence is probabilistic, that loop
can take weeks.

With state history you can answer, from a single failed run:

what the state was when the wrong decision was made
which node wrote the bad value
whether correcting that value fixes the outcome
whether the same fix holds across several replays

That is most of a root-cause investigation, from a recording you were already
keeping for crash recovery.

If you wrote your own loop

None of this needs the framework — it needs the property the framework gives
you. Persist a checkpoint per step, keyed by run and sequence, never overwrite
in place, and store enough to resume: cursor, state, window.

await store.append({ runId, seq, cursor, state, window, at: new Date() });

append, not upsert. The moment you overwrite, you have crash recovery and
you no longer have a recording, and the difference only becomes obvious the
first time you want to know what the state looked like three steps ago.

If this was useful

AI That Plans covers checkpointing as
infrastructure — durable state, resume, forking a run to test a hypothesis,
and the retention that keeps a recording worth having.