AI agents are becoming execution systems.
They no longer just answer questions. They send messages, create tickets, issue refunds, make bookings, update customer records, trigger deployments, provision resources, and call tools that change external state.
That creates a failure mode distributed-systems engineers already know well — but agent loops make it unusually easy to trigger:
a tool call can succeed and still look like a failure to the agent.
Consider a refund:
- An agent decides to issue the refund.
- The request reaches the payment provider.
- The provider commits the refund.
- The response is lost, delayed, or times out.
- The agent sees an error.
- The framework retries the tool.
At step 6, the important question is no longer:
Did the request fail?
It is:
Did the external effect already happen?
That distinction is where ordinary retry logic can become dangerous.
A timeout is not proof of failure
A timeout describes what the caller observed. It does not prove what the external provider did.
After a lost acknowledgement, at least two realities may be consistent with the evidence the agent has:
- the refund did not happen, so retrying is necessary;
- the refund did happen, so retrying may create a duplicate.
This is an ambiguous outcome.
If the agent cannot distinguish those realities, blind retry is not merely a reliability mechanism. It can create a second real-world action.
The same pattern applies far beyond payments:
- sending the same email twice;
- creating the same booking twice;
- submitting the same order twice;
- provisioning the same paid resource twice;
- firing the same deployment twice;
- updating an account more than once;
- triggering the same webhook-backed action twice.
The underlying problem is not "AI hallucination." It is distributed-systems uncertainty at the boundary between intent and external effect.
Why agent systems amplify the problem
Traditional applications already retry failed network operations. Agent systems add more ways for repetition to occur.
A tool may be repeated because:
- the model decides to retry after an error;
- the framework has retry logic;
- a workflow resumes from a checkpoint;
- a process crashes and restarts;
- a supervisor redispatches work;
- a sub-agent is re-run;
- an MCP tool call is attempted again;
- a user asks the agent to "try that again."
These mechanisms can all be individually reasonable.
The danger appears when they cross a side-effecting boundary without preserving the identity and outcome of the logical action.
Tool-call IDs are usually the wrong identity
A request ID, tool-call ID, trace ID, retry counter, or timestamp usually identifies an attempt.
But a retry of the same refund is not a new business intention just because it has a new tool-call ID.
For safe retry handling, we need a stable logical operation identity.
For example:
refund / order_123
should identify the same intended refund across all retries of that action.
Attempt 1 might have one request ID.
Attempt 2 might have another.
But if both represent the same intended refund, the logical operation identity should remain stable.
This gives us an important distinction:
transport identity -> which attempt is this?
logical identity -> which real-world action is this?
Those are not the same question.
Identity alone is not enough
There is another failure mode.
Suppose an application reuses the same logical operation ID but changes a value that affects the real-world action.
For example:
operation: send_invoice_4821
attempt 1 destination: alice@example.com
attempt 2 destination: bob@example.com
Those should not be treated as equivalent retries.
The operation identity therefore needs to be bound to the effect-bearing payload.
If a field can change the external effect — amount, destination, message body, booking details, resource configuration, recipient, etc. — changing it should produce a conflict or a new intentional operation.
Otherwise a deduplication mechanism can become a different kind of bug: incorrectly collapsing two distinct actions into one.
The state many systems are missing: UNKNOWN
A useful execution model has at least three outcome states:
| State | Meaning | Safe default |
|---|---|---|
CONFIRMED |
Authoritative evidence says the effect happened | Return/replay the known result; do not execute again |
ABSENT |
Authoritative evidence says the effect did not happen | Execution may proceed |
UNKNOWN |
The effect may have happened, but available evidence cannot prove which state is true | Reconcile or block |
The critical rule is:
UNKNOWN is not permission to execute again.
This sounds conservative because it is.
If duplicate execution could be expensive or irreversible, safety sometimes requires giving up immediate progress.
That is the classic tradeoff between safety and liveness:
- safety: do not accidentally create the duplicate effect;
- liveness: eventually complete the requested action.
If provider truth is unavailable, a high-impact operation may have to remain blocked until a human or a trusted system can resolve it.
Reconciliation: ask reality before repeating it
When an outcome is ambiguous, the strongest recovery path is often reconciliation.
Instead of retrying the mutation, perform a read-only check against an authoritative system.
For a refund, that could mean asking the provider whether the refund exists.
For a booking, check whether the reservation was created.
For a message, query the provider-side message ledger if such a facility exists.
The flow becomes:
external effect may have committed
|
v
UNKNOWN
|
v
authoritative lookup
/ \
/ \
CONFIRMED ABSENT
| |
v v
do not repeat execution may proceed
The phrase authoritative matters.
A missing local database row is not automatically proof that the external effect did not happen.
Neither is a timeout.
Neither is an empty cache.
ABSENT should require evidence strong enough to justify repeating the action.
Not every agent tool needs this
There is an obvious counterargument:
Do we really want every search, calculation, and read operation going through durable execution coordination?
No.
That would add latency and complexity where there is little duplicate-effect risk.
A better model is selective routing.
For example:
| Route | Meaning |
|---|---|
DIRECT |
No consequential external mutation identified |
PROTECT |
Duplicate execution could create an undesirable external effect |
BLOCK |
The system cannot establish that execution is safe |
A web search is usually DIRECT.
A local calculation is usually DIRECT.
A refund, message send, booking, order creation, or deployment trigger may be PROTECT.
A tool with conflicting or insufficient safety information may be BLOCK.
The asymmetry matters:
- a false positive mostly costs latency or integration friction;
- a false negative can permit an unintended duplicate real-world action.
For consequential tools, conservative classification is often the rational choice.
This is not a universal "exactly once" claim
"Exactly once" sounds attractive, but it is easy to overstate when independent systems are involved.
A client generally cannot atomically commit both:
- an arbitrary external provider's state; and
- its own local acknowledgement state
unless the systems share an appropriate transaction, deduplication, or reconciliation contract.
So the defensible target is narrower:
One intended consequential operation should produce at most one corresponding external effect across retries — under explicit assumptions — or the system should block rather than guess.
Those assumptions include:
- stable logical operation identity;
- complete effect binding;
- durable safety state;
- controlled execution concurrency;
- an execution boundary that cannot simply be bypassed;
- authoritative reconciliation where ambiguity cannot otherwise be resolved.
That is a safety property, not a promise that every operation will eventually succeed.
Where Once fits
We have been building Once, an open-source execution-safety layer for AI agent tools and MCP integrations, around this model.
The project separates attempt identity from logical action identity, preserves ambiguous outcomes, binds effect-bearing input to protected operations, and increasingly classifies toolsets so harmless calls can remain direct while consequential calls receive stronger protection.
The current public implementation includes:
- TypeScript and Python SDKs;
- MCP support;
- deterministic cross-language operation identity;
- local durable protection;
- tool discovery and classification;
- selective
DIRECT / PROTECT / BLOCKrouting; - framework integrations;
- hostile-retry evidence across multiple agent frameworks.
The important part is the boundary of the claim.
Once does not claim universal exactly-once execution across arbitrary providers and arbitrary deployments.
If authoritative truth is unavailable, the correct state may remain UNKNOWN.
That limitation is part of the design rather than something to hide.
We published the full systems model
I have now published the deeper technical treatment as a citable technical paper:
Reliable Execution of Consequential AI Agent Actions Under Retries and Ambiguous Outcomes
Jamie Oswald — Once Research
Published: 26 September 2026
DOI: 10.5281/zenodo.22969881
The paper covers:
- the formal one-effect safety property;
- trust assumptions;
- ambiguous-outcome failure states;
- logical identity versus attempt identity;
- effect binding and payload drift;
-
CONFIRMED / ABSENT / UNKNOWN; - reconciliation;
- selective tool protection;
- related techniques including idempotency keys and durable workflow systems;
- hostile-retry evidence;
- threats to validity;
- falsifiable failure criteria.
If you are building agents that can change external state, the question I would ask is simple:
If this tool times out after the external effect commits, what prevents the retry from doing it again?
If the answer is only "the framework retries carefully," there is probably another reliability boundary worth examining.
Links
- Technical paper: https://doi.org/10.5281/zenodo.22969881
- Once research page: https://onceexec.com/research/
- Once: https://onceexec.com/
- GitHub: https://github.com/stringsofthemind-oss/once
Top comments (3)
Really useful comparison. The point about rebuild triggers is especially important—having a technically valid sitemap means little if it’s stale. For large sites, memory usage and URL chunking can become surprisingly tricky.
The UUID trap in agent frameworks is where this usually goes off the rails. Most framework runtimes mint a fresh tool_call_id on every retry turn, so if downstream services deduplicate on the call header, every retry looks like a pristine first attempt.
The other failure mode that bit us in production is argument drift. When a model retries after a timeout error message, it often tries to be helpful by reordering keys, adjusting spacing, or tweaking optional parameter names. That changes the payload hash even when the business intent is identical. Moving the idempotency key out of the model generation loop and deriving it deterministically from the canonical entity ID before the tool touches the network stopped our duplicate external writes.
the logical operation id bound to the payload is the real fix here. most retry bugs I have seen come from teams deduping on the request id alone, which just tracks the attempt not the actual intent. curious if you have seen a good pattern for expiring an UNKNOWN state automatically instead of it sitting blocked forever waiting on a human.