The agent retried the tool call. The customer got charged twice.

An agent will re-issue a tool call for reasons the tool never sees, and if that tool moves money, sends mail, or creates a record, the retry runs the side effect again. Prompting the model to be careful does not close this, because the model is not the layer that decides to retry. The tool contract is where it has to be closed.

The first time this bit us, a customer got charged twice for one order and we spent an afternoon reconstructing why. The trace showed two identical charge_card calls, four seconds apart, same amount, same card. The first call had actually succeeded on the payment side, but the response was slow, our client hit its timeout, and the framework's auto-retry fired a second call. From the model's point of view it made one decision. From the payment processor's point of view it received two charges. Over the following week, before we fixed it, we found 9 duplicate side effects across charges and confirmation emails. Not one of them was a model reasoning error. Every one was a retry the tool had no way to recognize as a retry.

Here is the position I have landed on. Any tool that mutates state has to be idempotent at the tool boundary, and you cannot delegate that to the model, because the model does not know when it is being retried and neither does the layer that retried it. I will walk through where the retries come from, the wrapper I now put on every mutating tool, and the two refinements that decide whether it holds.

Case 1: retries come from three places, and the tool sees none of them

It helps to name where the duplicate call actually originates, because the fix has to sit below all three.

There is the client-timeout retry: the call succeeded server-side, but the response was slow, so the caller gave up and re-sent. The side effect already happened; the caller does not know it. There is the model re-emission: the tool returned something ambiguous (a partial result, an error that was actually a success), and the model, reading the transcript, decides to call the tool again to be sure. And there is the framework auto-retry: many agent runtimes retry a tool call on exception by default, and a timeout is an exception even when the work completed.

The common thread is that in all three, the second call is byte-for-byte a legitimate call. Nothing about it looks wrong. You cannot filter it out by validating arguments, because the arguments are valid. The only thing that distinguishes it from a real second action is that it is the same logical action as one you already performed, and the tool has no memory of that unless you give it one.

Case 2: require an idempotency key per logical action, and dedupe on it (the load-bearing fix)

The mechanism that actually works is the one banks and payment APIs have used for years: every mutating call carries an idempotency key that names the logical action, and the server records the outcome under that key. A second call with the same key does not re-execute, it returns the stored result of the first. The key is not the arguments. It is a stable id for "this specific intended action," generated once, reused across every retry of that action.

Here is the wrapper I put on mutating tools. It stores the result of the first successful execution under the key and short-circuits any duplicate to that stored result instead of running the side effect again.

python
import functools, threading, time

class _Store:
    """Toy store. In production this is Redis/Postgres with a TTL and a real lock."""
    def __init__(self):
        self._data = {}          # key -> (status, result, expires_at)
        self._lock = threading.Lock()

    def begin(self, key, ttl):
        # Returns ("hit", result) if seen, else ("new", None) after reserving the key.
        now = time.time()
        with self._lock:
            row = self._data.get(key)
            if row and row[2] > now:
                if row[0] == "done":
                    return "hit", row[1]
                return "inflight", None      # a duplicate arrived before the first finished
            self._data[key] = ("inflight", None, now + ttl)
            return "new", None

    def finish(self, key, result, ttl):
        with self._lock:
            self._data[key] = ("done", result, time.time() + ttl)

_store = _Store()

def idempotent(ttl=3600):
    """Wrap a mutating tool. Requires an `idempotency_key` kwarg naming the action."""
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, idempotency_key: str, **kwargs):
            if not idempotency_key:
                raise ValueError(f"{fn.__name__} is mutating and needs an idempotency_key")
            state, prior = _store.begin(idempotency_key, ttl)
            if state == "hit":
                return prior                 # duplicate: return first result, do NOT re-run
            if state == "inflight":
                raise RuntimeError(f"action {idempotency_key} already in progress")
            result = fn(*args, **kwargs)     # the side effect runs exactly once
            _store.finish(idempotency_key, result, ttl)
            return result
        return wrapper
    return deco

@idempotent(ttl=3600)
def charge_card(customer_id: str, cents: int) -> dict:
    # the real side effect: hits the payment processor exactly once per key
    return {"charged": cents, "customer": customer_id}

# Same key across retries -> one charge, second call returns the stored result.
key = "order-8842-charge"
a = charge_card("cust_17", 4200, idempotency_key=key)
b = charge_card("cust_17", 4200, idempotency_key=key)   # no second charge
assert a == b

The inflight state matters as much as the done state. If two retries race (the timeout fires while the first call is still running), you do not want both to sail past the check and both hit the processor. Reserving the key before executing, and rejecting a second call that arrives while the first is still in flight, is what closes that window. In production the store is Redis or a row with a unique constraint, and the reservation is a real atomic operation, but the shape is exactly this.

Case 3: the key has to be stable, and reads should stay retryable

Two refinements decide whether this holds up.

First, who generates the key. The safest source is the caller that has the stable notion of the action: your orchestration layer minting one id per logical action (per order, per message, per ticket) and threading it through every retry of that action. You can let the model pass a client-generated action id, but treat it as advisory, because a model asked to "reuse the same key on a retry" will sometimes generate a fresh one, and a fresh key defeats the entire mechanism. If you have no stable id to lean on, a short-TTL dedupe cache keyed on the tuple of (tool_name, normalized_args) is a serviceable fallback: it catches the identical-call retry inside a small window. It is weaker, because two genuinely-distinct identical actions (the same customer legitimately buying the same item twice in a minute) will collide and the second gets silently dropped, so the TTL has to be tuned to your real duplicate-vs-distinct timing.

Second, do not idempotency-gate everything. Reads are naturally safe to retry (fetching a balance twice costs nothing and hides no bug), and wrapping them adds latency and a cache that can go stale. Split the surface at the wrapper layer: mark writes as unsafe and require a key, leave reads retryable and un-keyed. The wrapper should refuse to run a mutating tool without a key, and should not demand one from a read.

The rules I hold myself to on this, with the hedges attached:

1. Every mutating tool carries an idempotency key naming the logical action, and the server dedupes on it. This is the load-bearing fix.
2. Generate the key in the orchestration layer, not the model. Treat a model-supplied key as advisory, never as a guarantee.
3. Reserve the key before executing, so racing retries cannot both run the side effect.
4. A (tool, normalized-args) dedupe cache with a short TTL is a fallback when you have no stable id, with the caveat that it drops genuinely-distinct identical actions.
5. Separate safe (read) from unsafe (write) at the wrapper. Only writes need keys. Do not gate reads.

Where I'd push back on this

The honest counter is that idempotency infrastructure is not free and it introduces its own failure mode: a stale or wrong dedupe entry that suppresses a call that should have run. If your TTL is too long and the store returns a cached result for an action the caller genuinely wanted to repeat, you have now silently dropped a real charge or a real email, and that bug is harder to see than the double-charge, because nothing errored. Add a flaky key store to the picture (the reservation write fails, or the done write is lost after the side effect ran) and you can get the worst of both, a side effect that happened with no record that it did. So this is not "sprinkle a decorator and stop thinking." It is a small distributed-systems problem, and the store's correctness is now part of your tool's correctness.

There is a boundary I will grant. If a tool is naturally idempotent already, either a pure read, or a write whose target already dedupes (an upsert keyed on a natural id, a set-to-value rather than an increment), then adding a key layer on top is redundant machinery that buys you nothing and adds a cache to keep honest. Push the idempotency down to the resource when the resource can carry it. The case I hold firm on is any non-idempotent write with an external side effect: charging a card, sending a message, incrementing a counter, creating a ticket. There, the model and the framework will retry for reasons neither surfaces to the tool, and "the model usually does not double-call" is not a correctness argument, it is a hope. The retry is going to happen. The only question is whether the second one runs the side effect, and that is decided at the tool boundary, not in the prompt.