"Make the agent fault tolerant" sounds like one task. It is three. Your agent can fail loudly, with an error you can see and retry. It can fail silently, when a side effect fires and the process dies before anything records it. And it can fail by waiting, when an approval takes two days and the process holding that wait gets restarted on day one. Each class needs a different mechanism, and most agent frameworks only ship the first. That is why demos look fault tolerant and production agents send the same email twice or lose an approval.
Why fault tolerance is three assignments
When you move an agent from prototype to production, "make it fault tolerant" lands in the backlog as one ticket. You handle the failures you've seen in testing, and those are almost always the loud ones: a tool returns a 500, the model API times out, the LLM sends back malformed JSON.
The other failures rarely show up on your laptop. They show up during a deploy, when the orchestrator kills a pod for using too much memory, when the cloud provider reclaims a spot instance, or when a run sits waiting for a human for three days. Fault tolerance for an agent covers three distinct classes:
- Visible failures, where something errors and you know about it.
- Invisible failures, where work happens but the record of it is lost.
- Long waits, where the agent has to outlive the process running it.
Visible failures and the retry toolbox
A visible failure announces itself. The tool call returns an error, the API times out with a clear exception, the model refuses the request or returns output your parser rejects.
The toolbox for this class is well known:
- Retry with exponential backoff, so a rate-limited or briefly unavailable service gets time to recover.
- Cap the number of attempts, so a permanent error doesn't turn into an infinite loop that burns tokens.
- Dead-letter the requests that keep failing, so a person or a separate process can look at them later.
Almost every agent framework and HTTP client ships some version of this, so it's the part of fault tolerance most teams already have.
Retries have one built-in assumption, though: the failed attempt had no effect. A retry repeats a step; it doesn't know whether the previous attempt already sent the email or charged the card before the error came back. The next class is where it breaks.
Invisible failures and why checkpointing falls short
An invisible failure happens in the gap between doing something and recording that you did it. Your agent calls the payment API and the charge goes through. Before the agent writes "payment done" anywhere, the process disappears. A rolling deploy replaced the pod, the kernel's out-of-memory killer picked your container, or the spot instance was reclaimed. No exception is raised and no retry policy fires, because the code that would catch the error is gone too.
Many agent frameworks answer this with checkpointing. It saves a snapshot of the agent's state after each step, and a restarted agent loads the latest one. If the process dies during a step or right after it, that snapshot is from before the side effect. The restored agent has no idea the charge happened, and it charges again.
Durable execution closes most of that gap by making the record part of execution itself. The engine writes "scheduled" to a journal before a step runs, and "completed" with the result once it returns. On restart it replays the journal, skips every step marked completed, uses the recorded results instead of calling the tools again, and resumes at the first unfinished step. This is the class of failure that separates a demo agent from a production one.
There's still a small window left: the side effect can succeed just before the process dies and before "completed" is written. In that case the engine reruns the step, which is why durable execution engines guarantee that a step runs at least once rather than exactly once. The journal gets you back to the right step. Making the side effect happen once is the job of the step itself: pass an idempotency key to the downstream service so it recognizes the repeat and returns the original result. Dapr Workflow gives every scheduled task a task execution ID that stays the same across retries, so you can use it as that key.
Long waits and durable timers
Agents spend a lot of their lives waiting. A refund over a certain amount needs a manager's approval, which might come in two hours or two days. A procurement agent can't confirm delivery until the container ships on Tuesday.
Processes don't live that long. Over a two-day approval window your agent will probably go through several deploys, a node upgrade or two, and maybe an autoscaler scaling it down overnight. A wait held in process memory, like a sleep call, an in-memory promise or a loop polling an inbox, disappears with the process. Any wait held in memory is a failure scheduled in advance; you just don't know the exact time yet.
Durable execution handles this by moving the wait into the journal as well. A durable timer records "wake me at this time," and an external event records "resume when this approval arrives." While the agent waits, nothing needs to run: the process can stop, restart or move to another node. When the timer fires or the event comes in, the engine loads the journal and continues from the waiting step. Pair the approval event with a timer so that if nobody responds within three days the agent escalates or cancels instead of waiting forever.
The crash test that checks all three
You can check your agent against both of the harder classes with two tests. Both use a force-kill, meaning the process ends right away with no graceful shutdown and no cleanup handlers. For example:
- In a container, use
docker kill, or delete the pod with a zero grace period in Kubernetes. - On Windows, end the process from Task Manager or with
taskkill /F. - On macOS, use Force Quit in Activity Monitor or
kill -9. - On Linux, use
kill -9with the process ID.
For the first test, start a run that makes a side-effecting call, such as sending an email or creating a record in another system. Force-kill the process right after that call succeeds, restart it, and let the run finish. Then count the effects in the downstream system. One is the only right answer. If you see two, the step needs an idempotency key.
For the second test, start a run that waits for an approval or a timer. Force-kill the process during the wait, restart it, then send the approval. The agent should pick up the event and continue.
Most teams skip these tests because they tend to fail the first time. That is exactly why you should run them before production does.
Trade-offs and considerations
Durable execution isn't free. Every step writes to the journal before and after it runs, which adds a little latency and storage per step. For steps with side effects or long waits, that cost is almost always worth paying; for a quick in-memory computation it usually isn't.
The orchestration logic also has to be deterministic, because the engine replays it to rebuild state. Keep LLM calls, tool calls and anything else non-deterministic inside activities, and keep the orchestration code limited to deciding what runs next.
Not every agent needs all three layers either. A stateless chat helper that only reads data is fine with the retry toolbox alone.
Summary
Fault tolerance for AI agents is three separate problems. Retries with backoff and dead-lettering handle visible failures. Invisible failures need more than checkpointing: durable execution writes each step to a journal before and after it runs, so recovery resumes at the right step, and an idempotency key makes sure the side effect lands once. Long waits belong in the journal too, as durable timers and external events, so a restart can't erase them. The two force-kill tests tell you which of these your agent already handles. To see these ideas in a small working example, read The Tiniest Durable Agent. To learn durable timers, external events, and recovery hands-on, take the free Dapr Workflow course on Dapr University.
Top comments (2)
The idempotency-key advice has a lifetime problem that matters most in your third class. Downstream services only remember keys for a limited time (Stripe, for example, says keys may be pruned after at least 24 hours). A task execution ID that stays the same across retries is fine for a retry seconds later, but if the journal gets replayed after a two-day approval wait plus a restart, the key may already be forgotten and the repeat looks like a brand new charge. So for steps that sit after a long wait, I'd check the side effect's own state first (look up the payment by your reference) instead of relying on the key alone.
The other edge worth testing is the timer racing the event: the approval arrives in the same moment the three-day timer fires. Does the engine pick one deterministically and record it, so a replay can't take the other branch? Did you see that case in the Dapr examples?
The force-kill test is the part I'd steal. One case where the idempotency-key answer runs out: steps whose side effect is a UI action rather than an API call. I'm on the Auten team (we build a tool that lets agents drive a real screen), and when the "send" is a click in an app with no API, there's no downstream service to hand a key to. The click either landed or it didn't, and nothing dedupes the second one.
What we've found works is turning the step into check-then-act: on re-run, the activity first reads the observable result (is the message already in Sent, does the record already exist with that title) and only clicks if it's missing. It's still not exactly-once, since the window between the click and the UI reflecting it is real, but it moves the idempotency check from the callee to the caller.
Does Dapr Workflow have a pattern for activities whose only idempotency check is reading external state like that, or do you treat those as "at-least-once, verify after" by design?