If an agent process dies and its workflow restarts without saved progress, completed side effects can repeat: the same email, ticket or payment a second time. AI agent crash recovery needs two parts, a checkpoint after each step and an idempotency key on each action. In 1,000 modeled crashes of a 12-step agent, restarting redid 5.68 finished steps on average and repeated an action in 70% of runs; a checkpoint after each step cut the redone work to 0.04 steps, and keys that the receiving service deduplicated brought duplicates to 0%.
Why a restart is worse than lost time {#problem}
Definition
Checkpoint: a saved record of a run's progress (the next step to run and the outputs of finished steps), written to storage that survives the process. Idempotency key: a stable identifier sent with an action, so the receiving service can recognise a retry of the same action and return the first result instead of acting again.
An agent run is a list of steps. Most steps only cost tokens and time: read the request, search the docs, write a summary. A few steps change the outside world: send an email, open a ticket, charge a card. If the process dies at step 6 and the agent starts again from step 1, the cheap steps are wasted money, and the side-effect steps happen twice. The customer gets two emails and two charges.
This is the same problem as a retried network request, which we covered for counters in why a like count can be wrong but a balance never. An agent just has more steps, so there are more places to crash.
What the simulation measured {#test}
The script labs/ai-24/simulate-agent-crash-resume.mjs (Node 22, standard library only) runs a 12-step support task: steps 4, 7 and 10 send an email, create a ticket and charge a test card through a fake external service that records every action it really performs. It runs 1,000 times with seed 24. Each run is killed at a random step, then resumed, and the same crash plan is replayed against three strategies.
| Strategy | Finished steps redone (avg) | Wasted tokens (avg) | Runs with an action twice |
|---|---|---|---|
| A: restart from step 1 | 5.68 | 14,300 | 70% |
| B: checkpoint after each step | 0.04 | 1,188 | 3.9% |
| C: checkpoint + idempotency keys | 0.04 | 1,188 | 0% |
node labs/ai-24/simulate-agent-crash-resume.mjs.
How does AI agent crash recovery use checkpoints? {#checkpoint}
Strategy B saves { runId, nextStep, outputs } after every finished step. On restart it loads that record and starts at nextStep with the earlier outputs in hand, so step 7 can still use what step 2 found.
restart: redone = steps finished before the crash avg 5.68 of 12
checkpoint: redone = 0, except a crash in the save gap avg 0.04
In production the checkpoint must live somewhere that survives the restart, such as a database row, a file on disk or a key in a durable store, because a checkpoint in process memory dies with the process. Agent frameworks expose the same idea; LangGraph's persistence layer saves graph state as checkpoints per thread and offers storage-backed checkpointers for real deployments. The step that was half done when the crash hit still starts over from its beginning, and that lost partial work is not in the 0.04 figure, which counts finished steps only.
The simulation assigns 40 ms to each save: 12 × 40 ms = 0.48 s per completed run. Restarting adds 18.3 s on average under the invented step costs.
The gap checkpoints cannot close {#gap}
A side-effect step does two things in order: perform the action, then save the checkpoint. If the crash lands between them, the email went out but the saved state does not know it. For the email at step 4, the resumed run sees nextStep = 3 (the code counts from zero) and sends the email again.
The simulation assumes 15% of crashes on a side-effect step land in that gap. That gave 39 of 1,000 runs, which is where the 3.9% comes from. Swapping the order (save first, then act) does not help, because a crash after the save then skips the action entirely and the duplicates turn into missing actions.
Idempotency keys close the gap {#keys}
Strategy C sends each action with a key built from the run and the step, for example run-81:step-10:charge. Deriving the key from the persisted run identity and step keeps it stable across retries. The service stores the first result under the key, and when the same key arrives again it returns that result and does nothing new. In the simulation the service blocked exactly the 39 repeats, and duplicates fell to 0%.
This only works if the receiving service checks the key. Stripe's idempotent requests are a documented example: for API v1 it saves the status code and body of the first request with a key and returns them for retries, and errors if the parameters differ. Stripe may prune API v1 keys once they are at least 24 hours old. Reusing a key after it has been pruned creates a new request. So receiver-side deduplication prevents repeated actions while the key remains valid, and a resume days later needs its own check. Part 43 of the main series ("You pressed Pay twice") covers the general rule, idempotent operations. For failover one layer down, see what happens when the main server dies at 3 am.
On your own services, the receiver must claim the key atomically, for example with an insert under a unique constraint, so two concurrent requests with the same key cannot both pass the check. For changes inside one database, commit the key, request parameters, result and business change atomically. For external actions, persist an outbox entry in that transaction, then deliver it with a stable key that the receiver deduplicates. An outbox can deliver duplicates, as AWS's transactional outbox guide notes, so the receiver's key check has to stop the second email or charge.
What we saw when we ran it {#we-ran-it}
We ran node labs/ai-24/simulate-agent-crash-resume.mjs on our own Mac on 2026-11-13. It played a 12-step task 1000 times with seed 24, crashed each run at a random step and resumed it three ways. The terminal printed one table. In the restart row, 5.68 finished steps were redone on average and a side effect repeated in 70% of runs.
Our terminal after the run: three strategies, 1000 seeded runs, one summary table.
With strategy B, checkpoints cut redone steps to about 0.04 on average (39 repeated steps over 1000 runs, printed rounded), but duplicates did not go to zero: 39 of 1000 crashes landed after a side effect and before the checkpoint save, so those runs sent the email, the ticket or the card charge again, which is 3.9% of runs. Strategy C kept the same average of about 0.04 redone steps, and the fake service blocked all 39 repeats by key, so duplicates dropped to 0%.
Stripe's API reference page on idempotent requests, the receiver-side check we lean on.
That page says that for API v1, once endpoint execution has begun, Stripe saves the status code and body of the first request with a key and returns them for retries. API v2 handles keys differently. In your own agent, checkpoints save the time and the receiving service's key check prevents the second charge.
The core, as pseudocode {#code}
This is pseudocode for strategy C. store is durable storage scoped by run ID. step.run has no side effects: it only computes a step's output, or the request parameters for an action, from saved outputs, so a retry sends the same parameters.
// resume-with-keys (pseudocode): the shape of strategy C
function runAgent(runId, steps, store, service) {
const saved = store.load(runId); // durable, scoped to this run
if (saved && saved.runId !== runId) throw new Error("checkpoint is for another run");
const id = saved ? saved.runId : runId; // persisted identity drives the keys
const start = saved ? saved.nextStep : 0;
const outputs = saved ? { ...saved.outputs } : {};
for (let i = start; i < steps.length; i++) {
const step = steps[i];
let out = step.run(outputs); // no side effects
if (step.effect) {
const key = `${id}:step-${i + 1}:${step.effect}`; // same on every retry
out = service.perform(step.effect, key, out); // service returns the first receipt for a seen key
}
outputs[i] = out;
store.save(id, { runId: id, nextStep: i + 1, outputs });
}
return outputs;
}
If a model call builds the parameters, save them before the action so a retry cannot change them. To see a crash in the gap and the blocked repeat, run the full simulation: node labs/ai-24/simulate-agent-crash-resume.mjs.
What this test does not show {#limits}
- No real process was killed and no real disk was used; the crash is a thrown error and the checkpoint is a string in memory that is assumed to survive.
- The 15% gap share is an assumption. The 3.9% was observed in the seeded simulation under the assumed crash distribution.
- Token and time costs per step are made up, so the table shows the shape of the effect only.
- Steps here are deterministic. A real model may produce a different output when a step reruns, which is one more reason to save outputs in the checkpoint rather than recompute them.
A related agent failure, acting on a screen that changed since the last look, is covered in why a computer-use agent checks after every click.
Our take {#our-take}
Save progress and outputs after every step, to storage that outlives the process, and resume from there. Then give every side effect a stable key and make sure the service on the other end checks it. Keys depend on a service you may not own, so read its docs for how long it keeps them.
This article describes general patterns with a small simulation. It does not describe any company's private systems. Numbers come from one run on 2026-11-13 with made-up step costs.
Top comments (0)