A five-minute agent demo can assume that the process stays alive, the network cooperates, and every tool returns a clear result.
A five-hour workflow cannot.
Long-running agents wait on external APIs, process large files, call models repeatedly, trigger side effects, and sometimes pause for a human decision. During that time a worker can crash, a container can be replaced, a request can time out after the remote system has already succeeded, or a workflow can resume under a different prompt or tool version.
At that point, an agent workflow is no longer “a prompt with several tool calls.” It is a distributed workflow with probabilistic steps.
This article presents a generic production pattern for making those workflows recoverable and inspectable. It is a synthesis of durable-execution systems, workflow engines, observability standards, and published engineering cases. It is not a claim that any one agent platform provides every component described here.
Durable does not mean keeping one process alive
The central idea behind durable execution is simple:
Preserve the information required to continue outside the process that is doing the work.
Temporal persists workflow event history so a worker can reconstruct state after failure. LangGraph persistence stores graph-state checkpoints associated with a thread. AWS Step Functions redrive can continue eligible failed executions from unsuccessful steps while retaining successful step results.
The mechanisms differ, but the architectural lesson is consistent: memory in a worker is not durable state.
It also helps to separate three kinds of storage:
- Workflow state — the current node, attempts, decisions, deadlines, and checkpoint pointers for one run.
- Application memory — reusable user or organizational information that can survive across runs.
- Artifacts — large files, generated reports, tool outputs, and datasets stored separately and referenced by hash or URI.
Mixing these lifecycles makes recovery harder. A checkpoint should not need to serialize a two-gigabyte artifact, and long-term memory should not be treated as proof that a specific node completed successfully.
The failures you must model explicitly
“Failed” is too broad to be a useful workflow state. Different failures require different responses.
1. Transient dependency failure
Network interruptions, rate limits, temporary capacity problems, and service restarts may succeed on a later attempt. Retry them with a limit, exponential backoff, jitter, and a total retry budget.
2. Permanent or deterministic failure
Invalid arguments, missing permissions, incompatible schemas, and policy rejections will not improve with another identical request. Route them to a repair path, a human, or a terminal state.
3. Unknown side-effect outcome
This is the dangerous one. Your HTTP request timed out, but the remote service may already have sent the email, charged the card, created the ticket, or changed the database.
Do not classify this as ordinary failure. Use a first-class NEEDS_RECONCILIATION state and verify the external result before retrying.
4. Worker or sandbox loss
Containers are replaceable. The workflow identity, event history, and checkpoints must not be. Anthropic’s description of its managed-agent architecture makes a similar separation: a session is represented as an append-only event log, while harness and sandbox processes can be replaced.
5. Replay incompatibility
A checkpoint may outlive the code, prompt, model route, tool schema, or policy that created it. Resume against a pinned release, or perform an explicit migration. Silent upgrades during recovery turn an incident response into an untested deployment.
6. Retry storm
Retries add load to a system that may already be unhealthy. If every layer retries independently, traffic can multiply rapidly. Pick one retry-owning layer, add backoff and jitter, and use a circuit breaker when a dependency is likely to remain unavailable.
7. Zombie execution
An agent can loop indefinitely, wait forever for an approval, or continue spending tokens after the result has lost value. Every run needs a total deadline plus limits for model calls, tool calls, cost, iterations, and idle time.
8. Duplicate or concurrent resume
The same event can be delivered twice. Two people can approve simultaneously. A retry can race with a manual repair. Stable workflow IDs, optimistic locking, and idempotent resume tokens protect state transitions from duplication.
A minimum production architecture
A practical architecture does not need to be enormous, but it needs explicit boundaries:
Trigger / API
|
v
Durable Orchestrator
- workflow_id
- pinned release manifest
- node states and attempts
- deadlines and budgets
- checkpoint pointers
|
+----> Node Workers / Tool Adapters
| |
| +----> External APIs / Databases
| +----> Artifact Store
| +----> Side-effect Ledger
|
+----> Human Inbox (wait / approve / resume)
|
+----> Trace, Log, and Evaluation Store
The orchestrator owns state transitions. Workers execute bounded steps. The artifact store holds large outputs. The side-effect ledger records the intent, idempotency key, and external receipt for operations that change the world. The human inbox persists waiting states instead of holding an HTTP connection open. The trace store explains what happened across models, tools, queues, and services.
Give each node a real state machine
At minimum, use states that preserve operational meaning:
READY
|
v
RUNNING --------------------------> SUCCEEDED
|
+-- transient -----------------> RETRY_SCHEDULED --> READY
+-- human required ------------> WAITING_HUMAN ----> READY
+-- outcome unknown -----------> NEEDS_RECONCILIATION
+-- permanent -----------------> FAILED_PERMANENT
+-- rollback required ---------> COMPENSATING
|
+--> COMPENSATED / FAILED
NEEDS_RECONCILIATION deserves special emphasis. “The client did not receive success” and “the operation did not happen” are different facts. AWS’s guidance on idempotent APIs recommends client-supplied request tokens and server-side records that associate a token with the original request. If the same token arrives with different parameters, that should be treated as an error rather than a new operation.
Idempotency belongs to the logical operation
An idempotency key should remain stable across attempts of the same logical action.
This is wrong:
key = f"{workflow_id}:{node_id}:{attempt_number}"
Every retry gets a new key, so the downstream system sees a new operation.
Prefer:
key = stable_hash(workflow_id, node_id, logical_operation_id)
Store the key before dispatching the side effect. Record the external receipt when it arrives. If the response is lost, query the downstream service or ledger using that same identity.
Do not assume a workflow framework removes this responsibility. Temporal notes that an Activity can run successfully and then be retried if the worker crashes before reporting completion. LangGraph warns that code before an interrupt may execute again when the node resumes. The application still needs safe side-effect semantics.
A simplified execution loop
The following pseudocode omits framework-specific details, but shows the responsibilities clearly:
async def run_workflow(workflow_id, request):
state = store.load_or_create(
workflow_id=workflow_id,
request=request,
release_manifest=current_release_manifest(),
)
while not state.is_terminal:
enforce_deadline(state)
enforce_cost_and_tool_budgets(state)
node = planner.next_ready_node(state)
if node.requires_human:
token = issue_resume_token(
workflow_id=workflow_id,
node_id=node.id,
state_version=state.version,
)
store.checkpoint(state.wait_for_human(node.id, token))
return {"status": "waiting_human", "token": token}
key = stable_hash(workflow_id, node.id, node.logical_operation_id)
try:
result = await execute_with_policy(
node=node,
state=state,
idempotency_key=key,
timeout=node.timeout,
retry_if=is_transient_and_safe,
backoff="exponential_with_jitter",
)
side_effect_ledger.record_receipt(key, result.receipt)
state = state.apply_result(node.id, result)
store.checkpoint(state)
except UnknownSideEffectOutcome as error:
store.checkpoint(
state.needs_reconciliation(node.id, key, error)
)
return {"status": "needs_reconciliation"}
except PermanentError as error:
state = compensate_or_fail(state, node, error)
store.checkpoint(state)
return state.final_result
Notice what is not left implicit: deadlines, budgets, release versions, human waits, idempotency, reconciliation, compensation, and checkpoint boundaries.
Human-in-the-loop is a durable state, not a modal dialog
Human review is often presented as a UI feature. Architecturally, it is a long-lived workflow state.
Pause before irreversible actions such as sending an external message, deleting data, moving money, changing production systems, or approving access. Persist the WAITING_HUMAN state. Bind the resume token to the workflow, node, and expected state version. Make repeated submissions idempotent. Provide expiration, rejection, cancellation, and escalation paths.
Most importantly, revalidate business conditions when the workflow resumes. Permission, price, inventory, risk, and policy may have changed while the agent was waiting.
Trace the decisions, not only the final answer
OpenTelemetry defines a span as a unit of work with timing, attributes, events, links, and status. That model maps well to agent nodes. A node-level record should include:
- workflow, run, node, attempt, trace, and parent-span IDs;
- workflow, prompt, model, tool, and knowledge-index versions;
- input, output, state, and checkpoint references with hashes;
- idempotency key, side-effect intent, receipt, and reconciliation state;
- timeout, deadline, heartbeat, retry reason, and backoff;
- human request, actor, decision, and timestamp;
- error class, policy result, evaluation score, and final state.
Agent-specific tracing systems can add model generations, tool calls, handoffs, and guardrail events. Trace grading can then identify which step caused a regression. It should be treated as a debugging and evaluation aid, not as proof that the workflow is correct.
Where ZGI fits
We are building ZGI, an open-source, self-hostable Agent Runtime. It brings agents, visual workflows, knowledge, database connections, reusable Skills, model integrations, and tool execution into one workspace. The goal is to reduce repeated integration work for developers and make agent applications easier to assemble and operate.
ZGI is relevant here because production reliability spans the whole agent application, not only the model call. A shared runtime gives teams a place to connect the workflow, tools, knowledge, and human interactions that need to be inspected together.
This article’s checkpoint protocol, retry model, side-effect ledger, state machine, and tracing fields are a general reference architecture. They should not be interpreted as a statement that ZGI currently implements every mechanism exactly as shown. We prefer to keep that boundary explicit while building the project in the open.
The test that matters
The most useful question for a long-running agent is not “Can it complete the happy path?”
It is:
If the worker disappears immediately after step 12 changes an external system, can we determine what happened, avoid repeating the action, restore the correct versioned state, and continue safely?
If the answer is unclear, the workflow is not durable yet.
Long-running agent reliability is mostly the disciplined engineering of state, identity, side effects, time, and evidence. Models make the workflow intelligent. The runtime makes it operable.
References
- Temporal workflow event history
- Temporal Activities: retries and idempotency
- LangGraph persistence
- LangGraph interrupts
- AWS Step Functions redrive
- AWS: Timeouts, retries, and backoff with jitter
- AWS: Making retries safe with idempotent APIs
- OpenTelemetry traces
- Anthropic: Effective harnesses for long-running agents
- Netflix Conductor
- ZGI on GitHub
Top comments (0)