Many engineers rightfully view every step in a financial workflow as a potential point of failure, because services go down and networks go offline. Yet there's often an even bigger danger lurking at the spaces in between those steps, because waiting is often where problems accumulate. As the application waits for account verification or settlements, it might restart when a new version is deployed, or its container may crash, or the hardware it's running on might fail. The longer the process runs, the more likely that these or other problems will occur.
AI agents introduce another source of delay: people. Some approvals may arrive in seconds, while others may take hours or days. The agent must track the current state, and to keep the business moving, may escalate if a deadline passes, choose an authorized alternative, or cancel the operation by compensating for previously completed steps.
How can a system wait reliably for as long as required and act immediately when a deadline expires? How can it recover from software or hardware failures without losing known progress or needlessly repeating model calls? How can engineers see what the agent is doing right now? How can the compliance team see what it did last week?
Durable Execution is the foundation that makes all of this possible. A Durable Execution platform, such as Temporal (MIT-licensed open source), records an application's progress and detailed results of every important step. If the application is interrupted, it uses that information to automatically recover the complete application state and then continue from where it left off. Model and tool results become part of that recorded progress, allowing the agent to resume without repeating completed work or wasting tokens unnecessarily. Durable timers allow the agent to wait reliably for any duration, whether seconds or months. The application can also await a specified condition, optionally with a defined time limit, and take action when it receives a message from an external system. These applications are written using normal code, in a standard language such as Python or TypeScript, and can run on standard infrastructure.
If you'd like to find out more about how Durable Execution can help you build reliable agents, stop by the Temporal booth in the Expo Hall for a whiteboarding session with technical staff.
Top comments (4)
"The spaces in between the steps are where problems accumulate" is the line I wish more agent builders internalized. The individual tool calls are the easy part; keeping the run alive across a deploy, an OOM, or a human approval that lands three days later is the real engineering.
The human-as-indefinite-wait case is the one people design around last and get burned by first. We made runs resumable from recorded step results specifically so a restart doesn't replay model calls — the token waste is real, but the bigger win is not re-rolling a non-deterministic step and getting a different answer the second time.
One tension worth naming: durable execution records tool/model outputs as committed progress, but if a recorded result was actually wrong, you don't want to faithfully resume onto bad state. We ended up marking certain steps as "must re-validate on resume" rather than trusted-by-default. With Temporal, how do you model "this recorded result is stale, re-run it" versus ordinary replay? That distinction is where durability and correctness pull against each other.
This is a great question. For the specific case of resuming, Temporal allows you to reset (roll back) a Workflow Execution to some previously completed step. For example, let's say that the third step in your Workflow retrieves the price of a bond, which might be one price when it initially runs and a different value later on. In such a case, you could reset the Workflow Execution back to just before step 3, in which case it would fetch that price again.
More generally, you can handle perishable data by using a Signal (via the CLI, Web UI, or the API provided for any supported language) to notify the Workflow Execution that newer data is available. A Signal is a message-passing mechanism that lets you send data into the running Workflow, and upon receiving the message, the Workflow Execution may change its state (i.e., invalidate the old data and load the new data) or alter the control flow (e.g., to mark something as approved in a human-in-the-loop scenario).
Durable timers solve only part of the approval problem; the resumed workflow also needs to know whether an external side effect may already have happened. I would record an idempotency key and a proof state alongside each wait so a crash after a submit cannot turn a restart into a duplicate action.
Yes, we recommend making side-effecting operations idempotent, for the reason you describe.
The Temporal Workflow API provides a function/method that generates a UUID that will be different for each Workflow Execution but stable across all replays (crash recoveries) of the same Workflow Execution.