An agent with write tools is coordinating distributed transactions over an unreliable network, and nothing limits when it decides to try again. Without end-to-end idempotency, a retry doesn't make it resilient. It just corrupts data faster than any human could.
The failure needs no jailbreak and no hallucination. A refund tool commits, the 200 OK is lost on the way back, the model sees a 504 and calls again. The API sees a fresh request and refunds a second time.
A timeout means "unknown"
Kleppmann makes the point in Designing Data-Intensive Applications: when a request gets no reply, the sender can't tell whether the request was lost, the server died mid-work, or the work ran and the reply was lost. In a naive agent loop the raw error goes straight back into the prompt and the model guesses. Telling it to be careful in the system prompt doesn't make anything idempotent. MCP won't do it for you either: idempotentHint is only a hint, and the spec has no idempotency-key field.
The mechanism: the gateway owns the key
Every state-changing tool call carries an idempotency key that the orchestrator creates, not the LLM. Build it from durable workflow state ({run_id}:{step_id}) so it survives a crash and resume, store a hash of the arguments next to it, and reject a key that comes back with different arguments. A retry with a known key gets the saved response instead of running again, which is how Stripe's API treats repeated keys on the receiving side.
Provider tool-call IDs don't do this job. They match results to calls in the conversation, and they aren't guaranteed to stay the same across regenerations, provider switches or replays.
What I'd act on
Move retry decisions out of the prompt. One layer per dependency owns retries, usually the gateway. If three layers each retry three times, one logical call becomes 27 attempts. Give the model the final outcome, or an explicit "outcome unknown, reconciling" state instead of a raw 504.
Prefer deep tools where the state is local. One transfer_funds call inside a single ACID transaction beats three tools the model has to call in the right order. When a step spans services, use keys plus a transactional outbox, and remember the relay can deliver twice, so consumers still deduplicate.
Don't mistake exactly-once for correct. Keys expire (Stripe may delete them once they're 24 hours old), legacy ERPs often take no key header, and a compensation is a new action, not a rollback. A key makes the wrong refund happen exactly once. It doesn't replace validation, authorization limits or a person reviewing high-value actions.
Let the model propose. Let the gateway commit, once, with a key.
The full article has the diagrams, the gateway code, the sources and the complete checklist: https://abivarma.medium.com/the-dropped-phone-line-why-ai-agent-tool-calls-are-distributed-transactions-7aa85d600d26
Top comments (2)
This is a great breakdown. "Let the gateway commit, once, with a key" is the exact right architecture.
The hardest part of enforcing this is payload mutation. Even if the orchestrator generates the key {run_id}:{step_id}, what happens if the model alters the payload slightly on the retry? (e.g., adding a new timestamp, or changing the wording in a parameter). The hash changes, the key check fails, and the gateway executes it again.
The gateway needs strict schema validation to strip volatile fields before hashing, or you need a way to semantically match the retry to the original intent. The "outcome unknown" state is crucial too. If the model doesn't know the outcome, it shouldn't be allowed to guess and retry blindly.
A retry is a new model call, so it sends the history again and bills it again. An idempotency key stops the duplicate write, and it also stops the duplicate bill. The second one is easy to miss.