Checkpointing Multi-Step Agents in Node So a Crash Doesn't Cost You

#typescript #agents #ai #node

Book: AI That Plans
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

A long agent run is a transaction with no rollback and no commit
point. It calls a model twenty times, hits an external API a dozen
times, and holds all of it in a variable.

Then the pod is evicted, the deploy rolls, or the function times out.
Everything is gone. The retry starts at step one and pays for
eighteen steps it already completed.

Checkpointing is the fix, and the interesting part is not writing
state to Redis. It is what happens when you replay a step whose side
effect already happened.

What a checkpoint is

Not the transcript. A checkpoint is the smallest thing from which
execution can continue.

export type Checkpoint = {
  runId: string;
  seq: number;
  cursor: string;          // which step comes next
  state: RunState;         // accumulated results
  window: MessageParam[];  // current model context
  costUsd: number;
  updatedAt: string;
};

export type RunState = {
  invoiceIds: string[];
  validated: Record<string, boolean>;
  notified: string[];
};

cursor is what makes resume possible. Without it you know what
happened and not what comes next, and you are back to replaying from
the start.

window is included because rebuilding the model's context from
state is lossy. What you send back has to be what it saw.

The full transcript stays out. It grows without bound, it is not
needed to continue, and it belongs in episodic storage with its own
retention.

Write after each step, atomically

export interface CheckpointStore {
  load(runId: string): Promise<Checkpoint | null>;
  save(cp: Checkpoint): Promise<void>;
}

export class PgCheckpoints implements CheckpointStore {
  constructor(private db: Db) {}

  async save(cp: Checkpoint) {
    await this.db.query(
      `INSERT INTO checkpoints (run_id, seq, data, updated_at)
       VALUES ($1, $2, $3, now())
       ON CONFLICT (run_id) DO UPDATE
         SET seq = EXCLUDED.seq,
             data = EXCLUDED.data,
             updated_at = now()
       WHERE checkpoints.seq < EXCLUDED.seq`,
      [cp.runId, cp.seq, JSON.stringify(cp)],
    );
  }

  async load(runId: string) {
    const { rows } = await this.db.query(
      "SELECT data FROM checkpoints WHERE run_id = $1", [runId],
    );
    return rows[0] ? (rows[0].data as Checkpoint) : null;
  }
}

The WHERE checkpoints.seq < EXCLUDED.seq is the detail worth
copying. If two workers both believe they own a run — which happens
after a network partition — the monotonic sequence means an older
checkpoint cannot overwrite a newer one. Without it, a zombie worker
rewinds a run that has moved on.

The loop with resume

export async function runDurable(runId: string, store: CheckpointStore) {
  let cp = await store.load(runId) ?? initial(runId);

  while (cp.cursor !== "done") {
    const step = STEPS[cp.cursor];

    const next = await step.execute(cp, { runId, seq: cp.seq });

    cp = {
      ...cp,
      seq: cp.seq + 1,
      cursor: next.cursor,
      state: { ...cp.state, ...next.state },
      window: next.window,
      costUsd: cp.costUsd + next.costUsd,
      updatedAt: new Date().toISOString(),
    };

    await store.save(cp);
  }

  return cp.state;
}

Resume is load ?? initial. That is the whole recovery path — start
the same function with the same runId and it continues where it
stopped.

Which means your queue's retry is now correct by default rather than
by accident: a job that crashes gets redelivered, calls the same
function, and resumes.

The replay trap

Here is the bug that makes checkpointing subtle rather than
mechanical.

The checkpoint is written after the step completes. So a crash
between "step's side effect happened" and "checkpoint saved" leaves
the run pointing at a step that already ran.

step 12: send_invoice_email  → email sent ✅
         → process killed    ✗ checkpoint not written
resume:  cursor is still 12  → email sent again

The customer gets two emails. Adding more checkpoints does not fix
it — there is always a gap between the effect and the record, because
they are two different systems.

The fix is the same one that makes any at-least-once system correct:
make the step idempotent, keyed on something derived from the run and
the step rather than from the attempt.

const sendInvoiceEmail: Step = {
  async execute(cp, ctx) {
    const key = `${ctx.runId}:${cp.cursor}:${cp.state.invoiceIds.join(",")}`;

    await mailer.send({
      to: cp.state.customerEmail,
      template: "invoice",
      idempotencyKey: key,
    });

    return { cursor: "await_payment", state: {}, /* ... */ };
  },
};

cp.cursor in the key rather than cp.seq is deliberate. seq
increments on every write, so a replay of the same logical step would
produce a different key and defeat the dedupe. The cursor identifies
which step, which is what you want to run once.

Where the downstream system has no idempotency support, do the
dedupe yourself before the effect:

if (await store.wasDone(key)) return skip(cp);
await effect();
await store.markDone(key);

That narrows the window rather than closing it. Closing it entirely
requires the effect and the record to share a transaction, which is
only available when the effect is in your own database.

Split steps at effect boundaries

Step design is what makes any of this tractable. A step that does
three unrelated side effects can only be replayed as a unit.

// hard to resume: three effects, one checkpoint
async function processInvoice(cp) {
  await charge(cp);
  await email(cp);
  await updateLedger(cp);
}

// resumable: one effect per step
const STEPS = {
  charge:  { next: "email" },
  email:   { next: "ledger" },
  ledger:  { next: "done" },
};

One externally visible effect per step. Then a replay repeats at most
one operation, and that one has an idempotency key.

Model calls are the easy case — they are effectively pure from your
side. Group as many as you like into a step. It is the writes that
need their own.

Operational details that bite later

Size. Checkpoints hold the model window, which grows. A run with
a long conversation writes a large row after every step. Compact the
window before checkpointing, not after.

Retention. Completed runs keep their checkpoints forever unless
you delete them. A deleted_at sweep on terminal runs, on a
schedule.

Schema drift. A checkpoint written by yesterday's deploy is read
by today's code. Version the payload and refuse to resume something
you cannot interpret:

if (cp.version !== CURRENT_VERSION) {
  throw new IncompatibleCheckpoint(cp.version, CURRENT_VERSION);
}

A loud failure beats resuming with fields silently missing.

Concurrency. Two workers on one run is a real scenario. The
monotonic-seq guard prevents rewinding; a short lease per run
prevents the duplicated work in the first place.

If you are on LangGraph.js

The framework does the mechanism for you — compile with a
checkpointer, invoke with a thread_id, and state is written after
each node.

const graph = workflow.compile({ checkpointer });
await graph.invoke(input, { configurable: { thread_id: runId } });

What it does not do is make your side effects idempotent. The replay
trap is identical inside a graph: a node that sent an email and did
not reach its checkpoint sends it again on resume. Node boundaries
are your step boundaries, and the same key discipline applies.