Retries are supposed to make systems more reliable. In AI agents, poorly designed retries can create duplicate actions, unnecessary LLM calls, and a surprisingly expensive feedback loop.
Imagine asking your AI assistant to refund a payment.
The agent calls a tool. The payment provider processes the refund. Everything works — except the response never reaches the agent.
From the agent's perspective, something went wrong.
So it tries again.
The second request reaches the provider. Another refund is processed.
The agent finally receives a successful response and tells you:
"Your refund has been completed."
Technically, that's true. But it has been completed twice.
This is one of the more interesting problems at the intersection of distributed systems and AI agents: the agent doesn't necessarily know whether a tool failed or whether it simply failed to hear back.
And there is another consequence we don't discuss enough: every unnecessary recovery cycle can cost us more than the original tool call.
It can cost us another LLM inference.
The dangerous gap between execution and response
A timeout is an observation made by the caller. It is not a statement about what happened inside the remote system.
Consider this sequence:
sequenceDiagram
participant A as AI Agent
participant R as Tool Runtime
participant P as Payment Provider
A->>R: issue_refund(invoice_842)
R->>P: POST /refund
P->>P: Refund successfully processed
P--xR: Response lost / timeout
R--xA: Timeout error
A->>R: Retry issue_refund()
R->>P: POST /refund
P->>P: Another refund processed
P-->>R: Success
R-->>A: Refund completed
The agent sees a failed request. The provider sees two successful requests.
This is not an unusual edge case. It is a fundamental problem in distributed communication: once a request has been dispatched, the caller may not know whether the remote operation happened.
Brian H. Liou explores this problem in the context of MCP tool calls. His experiments show how retries at different layers of an agent stack can turn one logical invocation into multiple external side effects.
The important part is not simply that retries exist. It's that multiple layers may retry without sharing the same understanding of the operation.
Retry Safety in MCP Tool Calls
The retry problem has more than one layer
When we say "the agent retries", what do we actually mean?
There can be retries in the HTTP client, MCP client, framework, tool runtime, and agent itself.
For example:
flowchart TD
A["Agent / LLM"] --> B["Agent Framework"]
B --> C["MCP Client"]
C --> D["HTTP Client"]
D --> E["MCP Server"]
E --> F["External API"]
B -. "Agent-level retry" .-> B
C -. "Tool-level retry" .-> C
D -. "Transport retry" .-> D
E -. "Provider retry" .-> F
Each layer may have a reasonable reason to retry.
The HTTP client wants to recover from a transient network error. The framework wants to recover from a failed tool invocation. The agent wants to complete the user's request.
Individually, these decisions may look harmless.
Together, they can multiply the number of attempts.
This is why retry ownership matters. If every layer independently decides to retry, the system can produce more requests than anyone intended.
A retry policy should have a clear owner, a bounded attempt budget, and a shared operation identity.
The cost nobody puts on the retry dashboard: tokens
Let's look at the same problem from the model's perspective.
A transport-level retry handled entirely by the HTTP client may not consume any additional LLM tokens.
But many agent frameworks return the tool error to the model and ask it to decide what to do next.
That changes the economics.
The model may need to process the original conversation, tool definitions, previous tool calls, error messages, and tool results before deciding to call the same tool again.
In other words, we may be paying the model to rediscover something our runtime could have handled deterministically.
Consider a simplified example:
flowchart TD
A["Initial LLM inference"] --> B["Tool call"]
B --> C{"Tool response received?"}
C -->|Yes| D["Continue workflow"]
C -->|No| E["Error added to context"]
E --> F["Another LLM inference"]
F --> G["Model decides to retry"]
G --> H["Tool called again"]
H --> I{"Successful?"}
I -->|No| E
I -->|Yes| D
Assume each inference uses:
- 6,000 input tokens (conversation, tool definitions, and execution history)
- 500 output tokens
One successful inference costs 6,500 tokens.
If a timeout causes two additional recovery inferences, the same logical task now costs:
| Input tokens | Output tokens | Total | |
|---|---|---|---|
| Normal execution | 6,000 | 500 | 6,500 |
| One recovery inference | 6,000 | 500 | 6,500 |
| Two recovery inferences | 12,000 | 1,000 | 13,000 |
| Total | 18,000 | 1,500 | 19,500 |
That's three times the token usage for a task that still required only one successful business operation.
And the example is deliberately conservative. In a real agent conversation, the context may grow with every attempt. Error messages, repeated tool results, and additional reasoning can increase subsequent input tokens.
There are two costs here:
Direct cost: additional input and output tokens from repeated model invocations.
Context cost: failed attempts become part of the conversation history, consuming context-window capacity that could have been used for actual work.
This becomes particularly important in long-running agents. A few failed tool calls early in a workflow can make every later inference more expensive.
The model isn't necessarily getting smarter with every retry. Sometimes it's just reading the same history again.
Give an operation an identity, not just an attempt number
One useful idea from distributed systems is idempotency.
Instead of treating every tool invocation as a completely new operation, identify the logical operation and preserve that identity across delivery attempts.
For example:
Logical operation:
refund:invoice-842:approval-17
Attempt 1:
operation_id = refund:invoice-842:approval-17
attempt_id = attempt-001
Attempt 2:
operation_id = refund:invoice-842:approval-17
attempt_id = attempt-002
The operation ID stays the same. The attempt ID changes.
This distinction is important:
- An operation represents what the user wants to happen.
- An attempt represents one effort to deliver that operation.
A simple interface might look like this:
interface ToolOperation {
operationId: string;
attemptId: string;
toolName: string;
payloadHash: string;
status:
| "pending"
| "in_progress"
| "completed"
| "unknown";
result?: unknown;
}
The runtime persists the operation before dispatching it.
When another attempt arrives with the same operation ID, the server checks the operation ledger instead of blindly executing the business action again.
async function executeOperation(
operationId: string,
payload: RefundPayload
) {
const existing = await operationStore.find(operationId);
if (existing?.status === "completed") {
return existing.result;
}
if (existing?.status === "in_progress") {
return { status: "in_progress" };
}
return dispatchRefund(operationId, payload);
}
This is illustrative code, not a complete production implementation. The operation claim must be atomic, and the stored payload hash must match. Reusing an operation ID with different arguments should be rejected.
There is also an important limitation: an internal operation ledger alone cannot guarantee exactly-once execution against an external provider. The provider needs to support idempotency keys, or the runtime must reconcile the external result before dispatching again.
Don't retry an unknown outcome as if it were a failure
This is probably the most important behavioral change.
A tool runtime should distinguish between:
- Failed: The operation is known not to have happened.
- Succeeded: The operation completed and its result is available.
- Unknown: The runtime cannot establish whether the operation happened.
Unknown is not just another error code. It is a different state that requires a different recovery strategy.
stateDiagram-v2
[*] --> Pending
Pending --> Dispatched
Dispatched --> Succeeded: Response received
Dispatched --> Failed: Confirmed failure
Dispatched --> Unknown: Timeout / connection lost
Unknown --> Succeeded: Reconciliation confirms effect
Unknown --> Failed: Reconciliation confirms no effect
Unknown --> Unknown: Still cannot determine
Failed --> Dispatched: Safe retry
Succeeded --> [*]
When the outcome is unknown, the runtime should first try to find out what happened.
For a payment, that might mean querying the provider with an idempotency key or transaction reference.
For a deployment, it might mean checking the deployment ID and current rollout state.
For an email, it might mean querying a delivery record if the provider exposes one.
The idea is simple:
Before repeating an action, check whether the previous attempt already achieved it.
If the result is confirmed, return the existing result to the agent. There is no reason to spend another inference asking the model what to do.
If the operation definitely did not happen, retry using the same operation identity.
If the result remains unknown, stop automatic retries and escalate rather than guessing.
Durable execution: recover state instead of asking the model again
This is where durable execution becomes useful.
In Long-horizon agents and durable execution, Martin Fowler explores how durable workflow systems preserve execution progress across failures.
Instead of reconstructing the entire workflow from scratch, the runtime can resume from persisted state and reuse completed step results.
Compare these two recovery paths.
Without persisted execution state:
Process restarts
|
v
Rebuild agent context
|
v
Load tool definitions
|
v
Ask LLM what to do
|
v
Possibly repeat previous tool call
|
v
Continue workflow
With durable execution state:
Process restarts
|
v
Load workflow checkpoint
|
v
Inspect previous operation
|
v
Reuse completed result
|
v
Continue from next step
The second approach can avoid unnecessary LLM calls because the runtime already knows what happened in previous steps.
But durable execution is not a magic exactly-once switch. A workflow engine may persist its own state while an external service has already performed an action whose response was lost. Idempotency and reconciliation are still necessary at that boundary.
The goal is not to eliminate every retry. It is to avoid replaying work that has already been completed.
So, where should retry logic live?
I'd keep the responsibilities separate:
flowchart TD
U["User request"] --> L["LLM: decide what action is needed"]
L --> R["Tool Runtime"]
R --> O["Operation Ledger"]
O --> P{"Previous result?"}
P -->|Completed| V["Return stored result"]
P -->|In progress| W["Wait / inspect status"]
P -->|Unknown| X["Reconcile external system"]
P -->|New operation| D["Dispatch tool"]
X --> Q{"Outcome established?"}
Q -->|Succeeded| V
Q -->|Not executed| D
Q -->|Still unknown| S["Stop / escalate"]
D --> E["External service"]
E --> F["Persist result"]
F --> V
V --> L
The LLM should decide what needs to happen.
The runtime should decide whether it is safe to execute it again.
That boundary is important. We shouldn't use an expensive probabilistic reasoning step to solve a deterministic execution problem.
Measure retries in tokens, not just requests
Most systems track request failures, retry counts, and latency.
For agentic systems, I'd add a few more metrics:
- Logical operations versus actual delivery attempts.
- Duplicate side effects.
- Operations left in an unknown state.
- Additional LLM calls caused by tool failures.
- Input and output tokens consumed during recovery.
- Context growth caused by failed tool interactions.
One metric I find particularly useful is retry token overhead:
Retry Token Overhead =
Recovery input + output tokens
--------------------------------
Total input + output tokens
For example, if an agent consumes 100,000 tokens and 25,000 of them are spent recovering from tool failures, a quarter of its token budget is going toward recovery.
That is a useful signal because it connects infrastructure reliability directly to model cost.
A system can have a high task-completion rate and still be inefficient. It may eventually finish the work, but only after repeatedly asking the model to process the same context.
The real purpose of a retry
Retries are necessary. Networks fail, services restart, and responses disappear.
But repeating a request is not the same as recovering an operation.
A good agent runtime should know the difference between a new operation, another attempt at an existing operation, and an operation whose outcome is still unknown.
It should also know when not to involve the LLM again.
Because the cost of a poorly designed retry isn't limited to duplicate transactions. It can also be another inference, another growing conversation, another context-window hit, and another unnecessary chunk of your token budget.
A reliable agent shouldn't just know how to try again. It should know when trying again is the wrong thing to do.
Top comments (0)