Your AI Agent's Retry Logic Just Charged a Customer Twice

The problem

An agent calls a charge_customer tool. The HTTP request to your payment provider succeeds, the charge goes through — but the response takes 11 seconds to come back, and your orchestration layer has a 10-second timeout on tool calls. The agent sees a timeout, not a success. It doesn't know the charge happened. So it does the only reasonable-looking thing: it retries the tool call.

Now the customer has two charges.

This isn't a hypothetical. It's the single most common production incident I've seen in agent systems that have moved past demos into anything transactional — payments, inventory decrements, email sends, ticket creation, database writes triggered by a tool call. Everyone stress-tests the model's reasoning. Almost nobody stress-tests what happens when a tool call's result gets lost.

Why it happens

LLM-based agents are built on an assumption that quietly breaks down: that a tool call is a single atomic unit — call it, get a result, decide the next step. But a tool call is actually two separate events across a network boundary: "the action happened" and "the agent learned the action happened." Those two events can and do get separated by timeouts, dropped connections, rate-limit retries, or the orchestration process itself crashing mid-call.

When they separate, the agent has no memory of the action — only of the uncertainty. And most agent loops resolve uncertainty the same way: retry. That's the correct instinct for a read (fetching a weather API twice is harmless) and the wrong instinct for a write. The agent framework doesn't know the difference unless you tell it, because from the model's point of view, get_weather and charge_customer are both just "a function I can call."

The deeper issue: idempotency was a solved problem in distributed systems a decade ago — API gateways, payment processors, and message queues all have battle-tested patterns for it. Agent frameworks mostly didn't inherit any of it. The tool-calling layer was built to make function calling easy, not safe, and idempotency doesn't show up in a quickstart demo.

What to do about it

Three things fix nearly all of this:

1. Idempotency keys on every write-capable tool. Generate a UUID when the agent decides to call a mutating tool, before the call goes out, and pass it through. Your backend stores keys it has already processed and returns the original result on a duplicate — the retry becomes a no-op instead of a second charge.

def charge_customer(amount, customer_id, idempotency_key: str):
    existing = db.get_by_idempotency_key(idempotency_key)
    if existing:
        return existing.result  # safe replay, no new charge
    result = payment_provider.charge(amount, customer_id, key=idempotency_key)
    db.save(idempotency_key, result)
    return result

2. Separate your tools into read vs. write, and treat retries differently per class. Reads: retry freely. Writes: retry only with an idempotency key, and cap it — if the key-checked call still fails after 1-2 attempts, surface the uncertainty to a human instead of guessing.

3. Log tool-call intent before the call, not just the result after. If you only log completed calls, a crash between "sent" and "logged" erases the evidence that the call was ever attempted — which is exactly the case you need visibility into. Write the "about to call X with key Y" record first.

The pattern underneath all three: stop treating the tool call as the unit of reliability. Treat the idempotency key as the unit of reliability, and let the tool call be retried as many times as the network wants.

Key takeaways

• A tool-call timeout tells you the response was lost — not that the action didn't happen. Don't conflate the two.
• Any tool that mutates state needs an idempotency key generated before the call, not after.
• Reads and writes need different retry policies; most frameworks default to one policy for both.
• Log call intent before execution, so a mid-call crash doesn't erase the evidence you need to debug it.
• If a mutating call fails even with a key, stop retrying automatically — escalate to a human.