Resuming an AI Agent After a Deploy Killed It Mid-Task

#typescript #agents #node #langgraph

Book: AI That Plans
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Deploys are the most common way an agent run dies. Not crashes, not timeouts —
a routine rollout, mid-afternoon, while a fourteen-step run is at step nine.

The pod gets SIGTERM. Whatever was in memory is gone. If the job is retried it
starts from zero, pays for those nine steps again, and re-does any side
effects they caused.

Making that survivable is four separate things, and teams usually build one of
them and assume they are covered.

1. State that lives outside the process

If you are on LangGraph.js this is the checkpointer, and the important part is
which one:

// dies with the process — fine for tests, useless for this
.compile({ checkpointer: new MemorySaver() })

// survives
.compile({ checkpointer: new PostgresSaver(pool) })

If you wrote your own loop, it is a row you update after each step:

export type Checkpoint = {
  runId: string;
  seq: number;
  version: number;        // see §4
  cursor: string;         // which step comes next
  state: RunState;
  window: MessageParam[];
  costUsd: number;
};

await store.save({ ...cp, seq: cp.seq + 1 });

Either way, the property you need is that a different process can pick the
run up knowing only its id.

2. A shutdown that stops taking work and finishes what it has

The default Node shutdown drops everything. What you want is: stop accepting,
let the current step finish and checkpoint, then exit.

let draining = false;
const inFlight = new Set<Promise<unknown>>();

process.on("SIGTERM", async () => {
  draining = true;
  await worker.pause();                    // stop pulling new jobs

  const deadline = Date.now() + 90_000;
  while (inFlight.size && Date.now() < deadline) {
    await Promise.race([
      Promise.allSettled([...inFlight]),
      new Promise((r) => setTimeout(r, 500)),
    ]);
  }
  await pool.end();
  process.exit(0);
});

Then check the flag between steps, so a long run yields at a clean boundary
rather than being cut mid-step:

while (cp.cursor !== "done") {
  if (draining) {
    await store.save(cp);                  // already saved, but be certain
    logger.info("yielding for shutdown", { runId: cp.runId, cursor: cp.cursor });
    return { status: "paused" as const, runId: cp.runId };
  }
  cp = await runStep(cp);
  await store.save(cp);
}

The configuration that breaks this is invisible in code: your orchestrator's
termination grace period must exceed that 90-second deadline, or it sends
SIGKILL partway through and the handler was decorative. On Kubernetes that is
terminationGracePeriodSeconds; check it, because the default is 30.

3. Something that resumes it

A paused run does not restart itself. Either the queue redelivers, or a
sweeper finds it.

export async function sweepStalled() {
  const stalled = await db.$queryRaw<{ run_id: string }[]>`
    SELECT run_id FROM checkpoints
    WHERE status = 'running'
      AND updated_at < now() - interval '5 minutes'
    LIMIT 100`;

  for (const { run_id } of stalled) {
    await queue.add("resume-agent", { runId: run_id }, {
      jobId: `resume-${run_id}`,            // dedupe
    });
  }
}

jobId as a dedupe key matters — two sweeper instances will find the same
stalled run, and without it you get two workers resuming one agent, which is a
different and worse problem.

The resume path is the same function:

worker.process("resume-agent", async (job) => {
  const cp = await store.load(job.data.runId);
  if (!cp) return;
  await runDurable(cp);
});

4. The trap: the code that resumes is not the code that paused

This is the one that surprises people. A deploy is why the run paused, so by
definition the process resuming it is running new code.

If you changed the state shape in that deploy, the checkpoint is now a payload
your code does not understand. Fields it expects are missing; fields it does
not know about are present.

The failure is quiet: state.approvals is undefined, a .length throws
somewhere unrelated, or worse, a boolean defaults to false and the agent
re-does an approved step.

Version the payload and refuse what you cannot read:

export const STATE_VERSION = 4;

export function load(raw: unknown): Checkpoint {
  const cp = CheckpointSchema.parse(raw);
  if (cp.version > STATE_VERSION) {
    throw new NewerCheckpoint(cp.version, STATE_VERSION);   // rolled back
  }
  return migrate(cp);
}

function migrate(cp: Checkpoint): Checkpoint {
  let c = cp;
  if (c.version < 3) c = { ...c, state: { ...c.state, approvals: [] }, version: 3 };
  if (c.version < 4) c = { ...c, state: renameField(c.state), version: 4 };
  return c;
}

Forward migrations, and an explicit error on a newer checkpoint, which
happens during a rollback, when old code meets new state. Failing loudly there
is far better than interpreting fields by luck.

The operational rule that follows: treat agent state like a database
schema. Additive changes are safe; renames and removals need a migration and
a deprecation window where both shapes are readable.

What resume must not do

Replay a side effect. The checkpoint is written after a step completes, so a
crash between "email sent" and "checkpoint saved" leaves the cursor pointing
at a step that already ran.

const key = `${cp.runId}:${cp.cursor}`;
await mailer.send({ ...msg, idempotencyKey: key });

Keyed on the cursor, not on the sequence number — seq increments on every
save, so a replayed step would generate a different key and defeat the dedupe
entirely.

This is why step boundaries should sit at effect boundaries: one externally
visible effect per step, so a replay repeats at most one operation and that
one is protected.

Test it by actually killing the process

it("completes the task across a hard restart", async () => {
  const runId = crypto.randomUUID();
  const child = fork("./worker.js", [runId]);

  await waitFor(() => store.load(runId).then((c) => c?.seq >= 3));
  child.kill("SIGKILL");                    // no graceful shutdown at all

  await runDurable(await store.load(runId));

  const final = await store.load(runId);
  expect(final.cursor).toBe("done");
  expect(mailer.sent).toHaveLength(1);      // not two
});

SIGKILL rather than SIGTERM is deliberate: it tests the resume path without
the graceful handler, which is what an OOM kill or a node failure actually
looks like. sent).toHaveLength(1) is the assertion that catches the replay
bug.

The short list

Durable checkpoints in Postgres, not memory. A SIGTERM handler that drains,
with a grace period configured to match. A sweeper with a dedupe key.
Versioned state with forward migrations and a loud error on rollback.
Idempotency keyed on the cursor.

Miss any one and the failure is the same from outside: a run that quietly
restarts, costs twice, and occasionally emails someone twice.

If this was useful

AI That Plans covers durable agents —
checkpoint design, resume, step boundaries around side effects, and state that
survives the deploy that interrupted it.