Retries and per-tool call caps are not exactly-once delivery

The payout tool charged the wallet. The HTTP response never came back. The agent timed out, minted a fresh idempotency key, and fired again. Without a per-tool call cap, that second call is a second dispatch. With a cap of 1, the second key is refused. Neither outcome is exactly-once delivery.

Earlier I wrote about same-key retries with different arguments. That post was about binding the request to the key. This one is about what happens when the agent gives up on the key and tries again under a new one.

Same key means replay. New key means new intent.

An idempotency key is a claim about intent, not a magic "run once forever" flag.

• Same key, same request: return the original result. No second side effect. No second charge.
• Same key, different request: refuse. That is a conflict, not a retry.
• New key: treat it as a new logical action. The server has no duty to notice that the arguments look familiar.

That last rule is the one agent loops keep violating. A timeout feels like failure. Failure feels like "try again." "Try again" in many frameworks means a new UUID. The first call may already have charged the remote tool. The second call does not know that.

Builders who have shipped agent wallets keep hitting the same question: what do you do when the outcome is unknown? Some have not exercised the delivery_uncertain path yet. Peers ask where the call cap should live so a frantic retry cannot spend the budget twice. The pattern below is the boring answer that still gets missed.

The dangerous path: timeout, then new key

Walk the failure the way it shows up in logs:

1. The agent sends pay_invoice under key task-4417-pay.
2. The gateway reserves budget, dispatches to the remote tool, and the tool charges.
3. The response is lost. Network blip, client timeout, worker kill. From the agent's seat the call looks failed.
4. The agent (or its supervisor) starts a new attempt with key task-4417-pay-retry-2.
5. If nothing else stops it, the gateway treats that as a fresh call and dispatches again.

Step 5 is not an idempotency bug. Idempotency did its job for key task-4417-pay. The second key asked for a second intent. The bug is in the control loop: treating silence as "safe to re-issue under a new key."

A same-key retry would have been fine. If the first call finished and a receipt exists, replay it. If the first call is still in flight, wait or get idempotency_in_progress. If the first call left the system in an unknown state after send, a same-key retry still should not fire a second dispatch; it should return the same unknown and make a human or a reconciler check the provider.

Put a per-tool call cap on the permit

When you cannot trust every caller to keep the key, bound how many times a given tool may dispatch under one permit.

max_calls_per_tool belongs on the permit next to budget, expiry, and tool scope. Session policy dies when the session dies; an agent that starts a new session walks right past it. A permit is the durable authorization, so the counter belongs there.

What the cap should mean in practice:

1. Count before dispatch. Reserve a slot when you authorize the call, not only after a success receipt lands. Otherwise two overlapping calls both pass a cap of 1.
2. New key past the cap: deny. Prefer a distinct error such as permit_max_calls_exceeded with the tool name, the limit, and how many calls were already made. No attempt. No charge. No send.
3. Same key still replays. Hitting the cap must not break honest retries. A same-key replay returns the original receipt and does not consume another slot.
4. Decide what keeps the slot. If the tool never dispatched, give the slot back. If the call was sent and the outcome is unknown, keep the slot. Refunding money while returning the count is how you get a second charge on the next attempt.

Here is a sketch written for this post, not code from any product:

def authorize(permit, tool: str, key: str, store):
    existing = store.get(key)
    if existing is not None:
        # Same key: replay. Do not touch the counter.
        return "replay", existing["receipt"]

    limit = permit.max_calls_per_tool.get(tool)
    used = permit.tool_call_counts.get(tool, 0)
    if limit is not None and used + 1 > limit:
        raise Denied("permit_max_calls_exceeded", tool=tool, limit=limit, calls_made=used)

    # Reserve the slot, then dispatch. Slot sticks if send happens.
    permit.tool_call_counts[tool] = used + 1
    receipt = dispatch(tool, key)
    store[key] = {"receipt": receipt}
    return "dispatched", receipt

A cap of 1 on pay_invoice turns the dangerous path into a loud denial on the second key. That is the whole point: stop the blind second send before it leaves.

At most N is not exactly-once

This is the claim people oversell, so say it plainly.

A per-tool call cap bounds the gateway to at most N dispatches for that tool under that permit. It does not make the remote side effect happen exactly once.

Why not:

• After send, the outcome can still be unknown. If the response is lost, the honest record is delivery uncertain: charged, not re-sent by the gateway, and still needing a check against the provider. The cap kept you from firing a second time. It did not finish the first time for you.
• The remote tool must honor its own key. Forwarding an idempotency key only helps if the upstream tool de-duplicates on it. If the tool ignores the key, a single dispatch from your side can still collide with another client talking to the same tool.
• A default "duplicate looking request" log is not a lock. Some systems log when a new key carries the same request hash as a recent attempt. Logging is not enforcement. Unless that guard is set to deny, a new key still dispatches.
• No cap means no bound. If the permit never set max_calls_per_tool for that tool, a new key is simply a new call, limited only by budget and expiry.

So the honest ladder is:

1. Same-key replay protects one intent.
2. A per-tool cap bounds how many intents can leave for that tool.
3. Exactly-once remote effects still require the tool itself to cooperate, plus reconciliation when delivery is uncertain.

If your dashboard says "exactly once" because you set a cap of 1, the dashboard is lying. It means "at most one dispatch from this gateway under this permit." That is still worth having. It is not the stronger promise.

How I am building this in AMW

I build Agent Middleware (AMW), a gateway that puts a permit and receipt boundary in front of MCP tool calls. Agents act under a permit; a same-key retry returns the original receipt, no second call or charge. When a permit sets max_calls_per_tool, that cap is checked before dispatch. A new key past the cap is denied as permit_max_calls_exceeded. A same-key replay still returns the original receipt.

What it does not do:

• No exactly-once remote side effect. After send, a lost response is recorded as delivery uncertain: charged, not re-sent by AMW. Someone still has to check the provider.
• A cap is at most N dispatches, not a guarantee the remote tool ran once. The remote tool must honor its own key.
• The duplicate guard is log-only by default. Identical looking new-key requests are logged and still dispatched unless that guard is set to enforce.

I'm biased: this is the product I am building, and it is pre-revenue. The retry and cap pattern above is older than AMW. Agent loops just make the new-key-after-timeout path common enough that leaving the bound optional is how wallets get drained.

Disclosure

I'm Chris, the founder and sole owner of AMW, which is pre-revenue. This post was drafted with help from an AI assistant; the Python is illustrative and is not AMW's source, and the AMW claims were checked against its code. More at thisisatest.tech.

My question for you: when a tool call times out after you believe it may have charged, do you block new-key retries behind a per-tool cap, or do you force same-key only until a reconciler clears the unknown? I want to hear which control you trust in production.