DEV Community: Chris Sellers

A timeout does not tell you whether the tool ran

Chris Sellers — Tue, 27 Oct 2026 05:30:11 +0000

When an agent calls a tool and the response never arrives, you know one thing: you stopped waiting. You do not know whether the tool did the work.

That gap is where duplicate emails, duplicate tickets and duplicate charges come from. The retry is reasonable. The missing information is the problem.

In AMW (Agent Middleware, Preview), which I am building, a lost response after the send is recorded as delivery_uncertain. AMW never sends that call again on its own. It keeps the state visible so a person or the tool owner can decide.

A same key retry returns the original receipt instead of dispatching again. A new key is a new request. That is the whole contract, and it is narrower than exactly once on purpose.

If you run agents against MCP tools, I would like you to try to break it: github.com/agentmiddleware/agent-middleware

Chris

Try a safe agent retry in two minutes

Chris Sellers — Thu, 22 Oct 2026 03:08:37 +0000

Earlier posts in this series argued about where idempotency keys should come from. This one is hands-on: get a free sandbox key (no account), issue a permit, call a tool, retry with the same key and watch the original receipt come back instead of a new dispatch, then verify the receipt offline. Code: https://github.com/agentmiddleware/amw-demo . If you try it, tell me in the comments which real tool you'd put behind it.

Which agent tool errors are safe to retry? Only one of four

Chris Sellers — Mon, 19 Oct 2026 15:23:33 +0000

An agent gets an error back from a tool call. Should it try again?

Most agent loops answer that with one rule: on error, retry a few times with backoff. I think that rule is wrong for any tool that moves money or changes state, because "error" covers at least four very different situations, and only one of them is safe to retry blindly.

This post is the error map I use in Agent Middleware (AMW), the permit and receipt gateway I am building for agent tool calls. The code below is generic, not lifted from my repo.

Ask one question first: did anything happen?

Before deciding whether to retry, the agent needs to know whether the call reached the point where a side effect could exist. I sort every error by that question.

Refused before anything was reserved. The permit does not allow the call: wrong tool, expired permit, or a constraint the backend cannot enforce. Nothing was written, charged or sent.
Lost a race before anything was written. Two calls tried to update the same permit at the same moment and this one lost. Nothing was written, charged or sent.
Refused because a limit was reached. The permit set a per tool call cap and it is used up. Nothing new was charged or sent, but the earlier calls did happen.
Sent, and the answer never came back. The request left the gateway and the response was lost. The side effect may or may not exist.

Only the second one is a "just try again" error.

What each case should do

Refused before reservation: do not retry. Retrying an identical request gets the identical refusal. The fix lives with whoever issues the permit. In AMW this comes back as a signed denied receipt, so the agent has a record of why it stopped.

Lost a race: retry, with the same idempotency key. AMW answers this with HTTP 409 (or JSON RPC error -32005 on the MCP endpoint). The transaction rolled back, the idempotency record for that attempt was released, and no credits moved. Reusing the same key matters here: if the first attempt had somehow finished, the same key returns its stored receipt instead of dispatching again.

Limit reached: do not retry, and definitely not with a fresh key. A fresh key is a new call, not a repeat. When the permit sets a per tool cap and the cap is used up, a new key gets refused with a cap exceeded error, with nothing dispatched or charged. When the permit sets no cap, a new key simply runs again within the permit's budget and expiry. So the cap only protects you if you set it.

Sent with no answer: park it. I wrote a whole post on this one. The short version: AMW records the attempt as delivery uncertain, keeps the charge, and never resends it on its own. A human or a reconciliation job checks the provider before anything else happens.

A sketch of the agent side

def handle_tool_error(err, call):
    if err.kind == "denied":
        # Permit does not allow it. Retrying changes nothing.
        return stop(reason=err.code, receipt=err.receipt)

    if err.kind == "contended":
        # Nothing was written. Safe to retry with the SAME key.
        return retry(call, idempotency_key=call.idempotency_key,
                     backoff=jittered(base_ms=50, max_tries=3))

    if err.kind == "cap_exceeded":
        # Earlier calls happened. A new key would be a new call.
        return stop(reason="cap reached", receipt=err.receipt)

    if err.kind == "delivery_uncertain":
        # Side effect may exist. Do not retry at all.
        return park(call, for_review=True)

    # Unknown error class: treat it as uncertain, not as safe.
    return park(call, for_review=True)

The last line is the part I care most about. When the agent cannot classify an error, the safe default is to stop and ask, not to loop.

What this does not give you

None of this makes a remote side effect happen exactly once. A cap of 1 means the gateway dispatches at most once for that permit and tool. Whether the provider itself applies the effect once still depends on the provider honoring the idempotency key. The error map just keeps the agent from turning one uncertain call into two certain ones.

Where I am

AMW is early and I am building it alone. I am looking for people building agents that touch payments or other irreversible tools who want to break it.

How does your agent loop decide which errors are safe to retry? I would like to hear where my map is wrong.

More at https://www.thisisatest.tech

When an agent's tool call times out, park it instead of retrying

Chris Sellers — Tue, 13 Oct 2026 19:06:14 +0000

An agent calls a tool that moves money or sends a message. The request goes out. Then nothing comes back: a gateway timeout, a dropped connection, a worker that died mid response.

From the agent's seat, that looks like a failure. Most agent loops treat a failure as a reason to try again. That is the step that charges a customer twice.

This post is about the third state that sits between "it worked" and "it failed", and what an agent should do with it.

A timeout is not a failure

A client timeout tells you one thing: the client stopped waiting. It says nothing about whether the provider did the work.

So every effectful call really ends in one of three states:

Succeeded. You have a response and a receipt.
Failed before send. The request never left your side (validation, auth, a connection that never opened). Safe to retry.
Unknown. The request was sent and the response never arrived. The side effect may or may not have happened.

Most code has branches for the first two and quietly folds the third into "failed". That fold is the bug.

Why "unknown" must not trigger a retry

If you retry an unknown under a new idempotency key, you have asked for a second intent. The provider has no way to know it is the same payment. If the first one landed, you now have two.

If you retry under the same key, you are safer, but only as far as the provider actually de-duplicates on that key. Many do. Some do it only for a limited window. Some tools an agent reaches through MCP never look at the key at all.

Either way, the agent itself is the wrong place to resolve this. It does not have the provider's state, it is under pressure to finish the task, and its default move is to try again.

Park it

The pattern several builders described to me recently is simple: when the outcome is unknown, park the call.

Parking means:

Record the attempt as pending, not failed. Keep the idempotency key, the exact arguments, the tool, the time of send, and whatever partial response you got.
Keep the hold. If you reserved budget or a call slot for this attempt, leave it reserved. Releasing it is how the next attempt gets room to spend again.
Block new keys for the same intent. While an intent is pending, the agent should not be able to issue a fresh key for it. A same key retry can answer "still pending" instead of sending.
Tell the agent something useful. Return a status the loop can act on, not a generic error, so it knows to wait or move on rather than retry.

One reply put it well, paraphrased: treating pending as failed is the expensive move, because failed implies retry or refund, and the charge may well have gone through.

Reconcile outside the agent

A parked call needs something to close it. That is a reconciler, and it should live outside the agent loop.

What it does:

Ask the provider. Many payment and messaging APIs let you look up a request by a client reference you sent with it, and some by the idempotency key itself. That lookup is the real answer.
Back off. Poll on a backoff until the provider gives a final status. Do not hammer it.
Close the record. If the provider says it happened, mark it succeeded and attach what you learned. If the provider says it never arrived, mark it failed, release the hold, and only then let a retry go out.
Escalate when the provider cannot say. Some tools have no lookup at all. Those go to a person with the key, the arguments, and the time of send, so they can check by hand.

The rule underneath: only the reconciler, with evidence from the provider, moves a call from unknown to failed. The agent never does.

What to measure

If you run agents against effectful tools, three numbers tell you whether this is working:

How many calls end in unknown per day. If it is zero, check whether unknowns are being folded into failures.
How long a call stays parked before something closes it.
How many parked calls closed as "it actually happened". That is the number of double charges a naive retry would have caused.

How I handle this in AMW

I'm building Agent Middleware (AMW), a gateway that puts a permit and receipt boundary in front of MCP tool calls. Agents act under a permit; a retry with the same key returns the original receipt instead of dispatching again; a new key is a new request.

For the unknown case, AMW records a lost response after send as delivery uncertain: the call is charged, the caller gets an error rather than a success, and AMW never re-sends it. When a permit sets a per tool call cap, a delivery uncertain call keeps its slot, so a new key past the cap is still denied.

What it does not do: it does not reconcile with the provider for you, and it does not make the remote effect happen exactly once. Someone, or something you run, still has to check the provider and close the record. I'm biased, since this is the product I'm building, and it is pre-revenue. The parking pattern is older than AMW and works without it.

Disclosure

I'm Chris, the founder and sole owner of AMW. This post was drafted with help from an AI assistant, and the AMW claims were checked against its code. More at thisisatest.tech.

My question for you: when a tool call comes back unknown, who closes it in your setup: the agent, a reconciler job, or a person? I'd like to hear what has held up in production.

A traceability import should never guess a missing lot code

Chris Sellers — Mon, 12 Oct 2026 17:10:53 +0000

A supplier sends a CSV of shipping rows. Row 37 has no traceability lot code. The importer has three choices: reject the row, store it with a warning, or fill the gap with something like UNKNOWN so the batch goes through. The third option feels friendliest on upload day and does the most damage during a trace request.

This post is about where to draw that line for FSMA 204 records, the FDA food traceability rule. The pattern applies to any import where a missing field can quietly turn into a wrong answer later.

What a shipping record has to carry

For foods on the Food Traceability List, the rule names key data elements (KDEs) for each critical tracking event (CTE). For shipping, 21 CFR 1.1340(a) lists eight:

Traceability lot code
Quantity and unit of measure
Product description
Location description for the immediate subsequent recipient
Location description for the location you shipped from
Date shipped
Location description for the traceability lot code source, or a traceability lot code source reference
Reference document type and number (a bill of lading, for example)

Receiving under 1.1345(a) mirrors this, with the immediate previous source, the receiving location and the receive date in place of the shipping fields.

Read that list as a data contract. Every field is either there or it is not. The interesting design question is what your system does with the rows where it is not.

Three ways to handle a missing field

Reject. The row never enters the system. The uploader gets an error naming the row and the field. Nothing downstream ever sees a half record.

Store and flag. The row goes in, marked incomplete, with a warning that names the missing element and the section it comes from. Someone can fix it later, and every report that touches it knows it is incomplete.

Fill and continue. The importer writes a placeholder (TLC-UNKNOWN, Unnamed Facility, today's date) and logs a warning nobody reads. The record now looks complete.

Fill and continue is the trap. The placeholder is a value, so every later check that asks "is this field present?" says yes. Worse, placeholders collide. If ten suppliers each upload a row with no lot code and all ten become TLC-UNKNOWN, a trace on that "lot" links ten unrelated shipments. During a recall you have just built a false connection, or hidden a real one.

Where to draw the line

The split I have landed on:

Reject when the row cannot identify what moved. No lot code, no product, no quantity or unit, no date. Without those, the row cannot be joined to anything, so storing it only creates noise. Fail loudly at upload, while the person holding the paperwork is still looking at it.
Store and flag when the row identifies the lot but is missing context. A missing reference document number or location description is a real gap, but the event still happened and the lot is still traceable through it. Keep the row, mark it, cite the section, and make the gap show up in every export.
Never fill. If a default seems tempting, that is usually a sign the field belongs in the reject group or the flag group, not in a third group of invented data.

The flag has to travel. A warning that lives only in an upload log is gone the next day. Put the incomplete state on the record itself so a sortable spreadsheet, an export and a trace query all carry it.

Present is not the same as correct

Even a perfect completeness check only proves a value exists. It does not prove the lot code matches the physical label, that the quantity is right, or that the ship to location is the real recipient. Those need a cross check: the receiving record at the next node should name the same lot code your shipping record did, and the quantities should reconcile within whatever tolerance the business accepts.

So I think of it as two layers:

Completeness: does each required element have a value? Cheap, deterministic, run on every row.
Consistency: do the values agree with the records on either side of the event? Harder, needs both trading partners' data, and is where most real traceability failures hide.

Plenty of systems ship layer one and call it done. It is a good start. It is not the same promise.

A few small things help at the completeness layer: normalize units (cs, case, CS) before checking, warn when a lot code mixes the letter O with zero or the letter I with one, and cap upload size so one bad file cannot bury a reviewer in thousands of warnings.

How I am building this in RegEngine

I'm building RegEngine, an FSMA 204 evidence layer. Today it takes CSV in its own column layout for shipping and receiving events and checks each row against the required KDE lists from 1.1340(a) and 1.1345(a). A row missing the lot code, quantity, unit, date or product is rejected rather than stored. Other missing elements come back as warnings that cite the CFR section.

What it does not do yet:

No automatic bill of lading reading. No PDF, scan, API, EDI or EPCIS intake. If you have a paper BOL, someone types it into the CSV.
Completeness, not correctness. The checks confirm a value is present. They do not prove it is right.
Not a legal determination. A clean check does not mean a record satisfies the rule or any buyer's program. Requirements vary by buyer, so confirm with your retailer.

It is pre-revenue, and I am biased. The FDA compliance date for the rule is July 20, 2028, but some buyers ask sooner, which is why import rules are worth getting right early.

Disclosure

I'm Chris, the founder of RegEngine. This post was drafted with help from an AI assistant, and the RegEngine behavior above was checked against a review of its code. More at regengine.co.

My question for you: when a supplier file arrives with a missing lot code, does your pipeline reject the row, store it flagged, or fill a placeholder so the batch goes through? I would especially like to hear from anyone who has had to untangle placeholder lot codes after the fact.

Retries and per-tool call caps are not exactly-once delivery

Chris Sellers — Sun, 11 Oct 2026 19:25:15 +0000

The payout tool charged the wallet. The HTTP response never came back. The agent timed out, minted a fresh idempotency key, and fired again. Without a per-tool call cap, that second call is a second dispatch. With a cap of 1, the second key is refused. Neither outcome is exactly-once delivery.

Earlier I wrote about same-key retries with different arguments. That post was about binding the request to the key. This one is about what happens when the agent gives up on the key and tries again under a new one.

Same key means replay. New key means new intent.

An idempotency key is a claim about intent, not a magic "run once forever" flag.

Same key, same request: return the original result. No second side effect. No second charge.
Same key, different request: refuse. That is a conflict, not a retry.
New key: treat it as a new logical action. The server has no duty to notice that the arguments look familiar.

That last rule is the one agent loops keep violating. A timeout feels like failure. Failure feels like "try again." "Try again" in many frameworks means a new UUID. The first call may already have charged the remote tool. The second call does not know that.

Builders who have shipped agent wallets keep hitting the same question: what do you do when the outcome is unknown? Some have not exercised the delivery_uncertain path yet. Peers ask where the call cap should live so a frantic retry cannot spend the budget twice. The pattern below is the boring answer that still gets missed.

The dangerous path: timeout, then new key

Walk the failure the way it shows up in logs:

The agent sends pay_invoice under key task-4417-pay.
The gateway reserves budget, dispatches to the remote tool, and the tool charges.
The response is lost. Network blip, client timeout, worker kill. From the agent's seat the call looks failed.
The agent (or its supervisor) starts a new attempt with key task-4417-pay-retry-2.
If nothing else stops it, the gateway treats that as a fresh call and dispatches again.

Step 5 is not an idempotency bug. Idempotency did its job for key task-4417-pay. The second key asked for a second intent. The bug is in the control loop: treating silence as "safe to re-issue under a new key."

A same-key retry would have been fine. If the first call finished and a receipt exists, replay it. If the first call is still in flight, wait or get idempotency_in_progress. If the first call left the system in an unknown state after send, a same-key retry still should not fire a second dispatch; it should return the same unknown and make a human or a reconciler check the provider.

Put a per-tool call cap on the permit

When you cannot trust every caller to keep the key, bound how many times a given tool may dispatch under one permit.

max_calls_per_tool belongs on the permit next to budget, expiry, and tool scope. Session policy dies when the session dies; an agent that starts a new session walks right past it. A permit is the durable authorization, so the counter belongs there.

What the cap should mean in practice:

Count before dispatch. Reserve a slot when you authorize the call, not only after a success receipt lands. Otherwise two overlapping calls both pass a cap of 1.
New key past the cap: deny. Prefer a distinct error such as permit_max_calls_exceeded with the tool name, the limit, and how many calls were already made. No attempt. No charge. No send.
Same key still replays. Hitting the cap must not break honest retries. A same-key replay returns the original receipt and does not consume another slot.
Decide what keeps the slot. If the tool never dispatched, give the slot back. If the call was sent and the outcome is unknown, keep the slot. Refunding money while returning the count is how you get a second charge on the next attempt.

Here is a sketch written for this post, not code from any product:

def authorize(permit, tool: str, key: str, store):
    existing = store.get(key)
    if existing is not None:
        # Same key: replay. Do not touch the counter.
        return "replay", existing["receipt"]

    limit = permit.max_calls_per_tool.get(tool)
    used = permit.tool_call_counts.get(tool, 0)
    if limit is not None and used + 1 > limit:
        raise Denied("permit_max_calls_exceeded", tool=tool, limit=limit, calls_made=used)

    # Reserve the slot, then dispatch. Slot sticks if send happens.
    permit.tool_call_counts[tool] = used + 1
    receipt = dispatch(tool, key)
    store[key] = {"receipt": receipt}
    return "dispatched", receipt

A cap of 1 on pay_invoice turns the dangerous path into a loud denial on the second key. That is the whole point: stop the blind second send before it leaves.

At most N is not exactly-once

This is the claim people oversell, so say it plainly.

A per-tool call cap bounds the gateway to at most N dispatches for that tool under that permit. It does not make the remote side effect happen exactly once.

Why not:

After send, the outcome can still be unknown. If the response is lost, the honest record is delivery uncertain: charged, not re-sent by the gateway, and still needing a check against the provider. The cap kept you from firing a second time. It did not finish the first time for you.
The remote tool must honor its own key. Forwarding an idempotency key only helps if the upstream tool de-duplicates on it. If the tool ignores the key, a single dispatch from your side can still collide with another client talking to the same tool.
A default "duplicate looking request" log is not a lock. Some systems log when a new key carries the same request hash as a recent attempt. Logging is not enforcement. Unless that guard is set to deny, a new key still dispatches.
No cap means no bound. If the permit never set max_calls_per_tool for that tool, a new key is simply a new call, limited only by budget and expiry.

So the honest ladder is:

Same-key replay protects one intent.
A per-tool cap bounds how many intents can leave for that tool.
Exactly-once remote effects still require the tool itself to cooperate, plus reconciliation when delivery is uncertain.

If your dashboard says "exactly once" because you set a cap of 1, the dashboard is lying. It means "at most one dispatch from this gateway under this permit." That is still worth having. It is not the stronger promise.

How I am building this in AMW

I build Agent Middleware (AMW), a gateway that puts a permit and receipt boundary in front of MCP tool calls. Agents act under a permit; a retry with the same key returns the original receipt instead of dispatching again; a new key is a new request. When a permit sets max_calls_per_tool, that cap is checked before dispatch. A new key past the cap is denied as permit_max_calls_exceeded. A same-key replay still returns the original receipt.

What it does not do:

No exactly-once remote side effect. After send, a lost response is recorded as delivery uncertain: charged, not re-sent by AMW. Someone still has to check the provider.
A cap is at most N dispatches, not a guarantee the remote tool ran once. The remote tool must honor its own key.
The duplicate guard is log-only by default. Identical looking new-key requests are logged and still dispatched unless that guard is set to enforce.

I'm biased: this is the product I am building, and it is pre-revenue. The retry and cap pattern above is older than AMW. Agent loops just make the new-key-after-timeout path common enough that leaving the bound optional is how wallets get drained.

Disclosure

I'm Chris, the founder and sole owner of AMW, which is pre-revenue. This post was drafted with help from an AI assistant; the Python is illustrative and is not AMW's source, and the AMW claims were checked against its code. More at thisisatest.tech.

My question for you: when a tool call times out after you believe it may have charged, do you block new-key retries behind a per-tool cap, or do you force same-key only until a reconciler clears the unknown? I want to hear which control you trust in production.

Same idempotency key, different arguments: refuse the mismatch

Chris Sellers — Fri, 09 Oct 2026 22:26:53 +0000

The agent paid the invoice once. Then the model regenerated the tool call for a retry, kept the same idempotency key, and changed the memo field by a few words. The server saw the familiar key, returned the original receipt, and the caller moved on. In the ledger it looks like the corrected call went through. It never did.

Yesterday I wrote about where an AI agent's idempotency key should come from. That post was about minting the key for the logical action and keeping it across retries. This one is about the other half: what the server must do when that same key shows up with a different request.

The key-only matching bug

An idempotency key is not a free pass to replay whatever you stored last time. It is a pointer into a record that also captured what was asked for.

Key-only matching is the bug. The server stores (key → response), looks up by key alone on the next request, and returns the stored response. That is fine when every retry truly repeats the first call. It is a silent lie when the arguments changed.

With ordinary HTTP clients, the body usually stays the same on retry. With LLM agents, a "retry" often means the model writes the tool call again. Amounts get re-parsed. Memo text gets reworded. A recipient id shifts by one digit after a copy from a PDF. The key can still be the stable task key you carefully persisted, and the payload is no longer the same action.

If the server only checks the key, the agent gets the old receipt for a call it never made. That is worse than a clean double charge in one important way: the books say the new intent succeeded.

Builders keep rediscovering this. The usual warning is: if you only check the key, a retry with the same key but different arguments returns the old receipt and the caller thinks the new call went through. Hash the args into the key, or refuse the mismatch loudly.

Refuse the mismatch (hash the request)

The correct server behavior is simple:

When you accept a request under a key, store a hash of the effectful request alongside the response.
On a later request with the same key, compare hashes.
Same key and same hash: return the original result. Skip a second side effect and a second charge.
Same key and different hash: refuse. Prefer not to replay or overwrite. Surface a conflict the client has to handle.

Here is a sketch written for this post, not code from any product:

import hashlib
import json


def request_hash(tool: str, arguments: dict) -> str:
    # Canonicalize so key order does not create false mismatches.
    canonical = json.dumps(
        {"tool": tool, "arguments": arguments},
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(canonical.encode()).hexdigest()


def begin(store, key: str, tool: str, arguments: dict):
    incoming = request_hash(tool, arguments)
    existing = store.get(key)
    if existing is None:
        # First time: reserve the key, run the call, store hash + result.
        return "new", incoming
    if existing["request_hash"] == incoming:
        return "replay", existing["result"]
    # Same key, different request: fail closed.
    raise Conflict("idempotency_key_reused")

Two design notes matter in practice:

Hash the fields that change the effect. Tool name, amount, currency, recipient, and other business fields belong in the hash. Purely cosmetic headers usually do not. If you hash nothing consequential, you are back to key-only matching in disguise.
Make the conflict loud. A soft 200 with a warning header will be ignored by most agent loops. Prefer a distinct error the client must handle, such as idempotency_key_reused or HTTP 409 with a clear body.

There is a second school of thought: bind the arguments into the key itself, so a changed payload naturally becomes a new key. That works too, as long as every caller uses the same recipe. The hash-and-refuse approach is usually safer for a shared API, because clients that already send a stable task key still get protection even when they forget to include the args in the key string.

What the client should do on conflict

When the server says the key was reused with a different request, minting a fresh key and firing again is usually the wrong move. That turns a detected conflict into a second real call.

Treat it as a signal:

Stop the automatic retry. A conflict is not a timeout.
Compare the two intents. Was the change accidental (model reworded a memo) or intentional (amount corrected after review)?
If accidental, resend the original arguments under the same key, so you get a true replay.
If intentional, that is a new logical action. Give it a new key, after you have decided the first action is cancelled, superseded, or left as-is.
Route unclear cases to a person or a planner step. Agent frameworks that retry every non-200 will happily dig a deeper hole unless you catch this error class explicitly.

Filling consequential arguments from task state, not from free-form model text, cuts how often this fires. Let the model decide to pay; let code supply amount, currency, and recipient from the stored task.

How I am building this in AMW

I build Agent Middleware (AMW), a gateway that puts a permit and receipt boundary in front of MCP tool calls. Agents act under a permit; a retry with the same key returns the original receipt instead of dispatching again; a new key is a new request. The request is hashed with the call. Same key with a different payload (or a different permit) is refused as idempotency_key_reused, not replayed. A retry that arrives while the first call is still running gets idempotency_in_progress.

What it does not do:

A new key is a new call. Nothing silently de-duplicates across keys. If you change the key to "get past" a conflict, you asked for a second dispatch.
Remote side effects are a separate problem. AMW bounds its own side: at most one dispatch, one charge, and one receipt per accepted key. Whether the remote tool itself runs once still depends on that tool honoring an idempotency key of its own.

I'm biased: this is the product I am building, and it is pre-revenue. The pattern above is older than AMW. Stripe-style APIs have taught request binding for years. Agents just make the mismatch case common enough that leaving it soft is no longer optional.

Disclosure

My question for you: when your agent hits a same-key conflict because the model rewrote the arguments, do you force a resend of the original payload, or do you treat it as a new action with a new key? I want to hear how people decide that line in production.

Where should an AI agent's idempotency key come from?

Chris Sellers — Thu, 08 Oct 2026 18:45:40 +0000

Your agent's worker sends a payout request, and then the process gets killed before the response comes back. The supervisor restarts it. The agent reloads its plan, sees that the "pay invoice 4417" step never finished, and runs it again. The payment API supports idempotency keys, and the agent sent one both times. The vendor still got paid twice, because the agent generated a fresh UUID on each run.

Yesterday I wrote about gating a risky tool call with a permit and a receipt and ended by asking where an agent's key comes from. This post is my answer, and it applies whatever sits behind the tool.

Two terms

An idempotency key is a value the client sends with a request so the server can recognize a repeat and return the first result instead of acting again. It only works if the repeat carries the same key.

A logical action is the thing you want to happen once in the business sense: pay invoice 4417, delete the staging database created for run 812, send the renewal email for account 77. Attempts are how you try to make it happen. One logical action can take many attempts.

The rule that follows: the key identifies the logical action, not the attempt.

The per request UUID trap

Most retry code mints the key where the request is built. That is the bug.

This is a simplified sketch written for this post, not code from any product:

# Wrong: every attempt looks like a brand new action.
for attempt in range(3):
    try:
        return pay(invoice, idempotency_key=str(uuid.uuid4()))
    except TimeoutError:
        continue

The timeout is exactly the case where the first attempt may have succeeded, and exactly the case where this loop sends a key the server has never seen. Moving uuid4() above the loop fixes the in process retry, but not the restart in the opening story, because the key dies with the process.

Where the key should come from

Here is the flow I use, in order:

Mint or derive the key when the action is planned, not when it is executed. The planner decides "pay invoice 4417 for $250". That decision is the moment the action gets its identity.
Persist the key with the task state before the first attempt. If the step is in your database, queue message, or workflow history, the key goes there too, written before anything is sent.
Every attempt reads the key, never creates it. In process retries, a restarted worker, and a resumed workflow all send the same value.
A new key means a new intent. If the amount or recipient changes, that is a different action and deserves a different key. A transient error is not a reason for a new key.
Treat a key conflict as a signal, not an obstacle. If the server says "this key was used with a different request", stop and look. Do not mint a fresh key to get past it.

The fixed loop, again a sketch:

# Right: the key was created with the task and stored next to it.
key = task.idempotency_key
for attempt in range(3):
    try:
        return pay(invoice, idempotency_key=key)
    except TimeoutError:
        continue

Stored or derived?

There are two reasonable ways to get task.idempotency_key.

Stored: generate a random value when the task row or message is created, and keep it there. Simple, and safe as long as that storage is durable and is the same record every retry reads.

Derived: compute the key from identifiers that are already stable for this action. Also a sketch:

import hashlib


def action_key(run_id: str, step_id: str, business_ref: str) -> str:
    raw = f"payout:v1:{run_id}:{step_id}:{business_ref}"
    return hashlib.sha256(raw.encode()).hexdigest()

Deriving survives even if you lose the stored key, as long as the inputs are truly stable. Choose them carefully:

Include a workflow or run ID, the step ID, and the business reference (invoice number, account ID).
Exclude timestamps, attempt counters, hostnames, and any text the model wrote. Each of those changes between attempts and quietly turns your retry into a new action.
Do not go too coarse. A key of just the invoice number blocks a legitimate second, partial payment on the same invoice. Add whatever makes two real actions different.
Version the format (the v1 above), so changing the recipe later does not collide with old keys.

The agent specific twist

With a normal service, a retry resends the same bytes. With an LLM agent, a retry often means the model generates the tool call again, and it may not produce the same arguments. The memo field gets reworded, or the amount is parsed slightly differently from the invoice text.

If your key is derived from the task, a properly built server will see the same key with a different request and refuse it. That is the system working: it failed closed instead of paying twice. Two things make this smooth in practice:

Fill consequential arguments from task state, not from the model. Let the model decide to pay; let code supply the amount, currency and recipient from the stored task.
Route conflicts to a person or a planner step. The fix is to figure out which version of the action is right, not to retry harder.

How AMW handles keys

I build Agent Middleware (AMW), a gateway that puts a permit and receipt boundary in front of MCP tool calls, so here is how it treats keys today. Agents act under a permit; a retry with the same key returns the original receipt instead of dispatching again; a new key is a new request. Governed calls must carry a key, scoped per wallet. The same key with a different request is refused with idempotency_key_reused, and a retry that arrives while the first call is still running gets idempotency_in_progress.

What it does not do matters just as much for this topic:

A new key is a new call. If the permit sets a per tool call limit and it is used up, the retry is refused. Do not count on anything else to catch it. Where your key comes from is your job, which is why this post exists.
Once only at the remote tool depends on the tool. AMW controls its own side: at most one dispatch, one charge and one receipt per accepted key. Whether the side effect itself happens once at the remote tool depends on that tool de-duplicating on an idempotency key of its own.
A crash in the gateway can make a same-key retry wait. After a crash mid call, a same-key retry can see idempotency_in_progress for about three hours until the stuck call is reconciled. Your agent should treat that as "pending", not as "failed, try a new key".

Disclosure

My question for you: when an agent's plan changes partway through, say the amount gets corrected after a failed attempt, do you treat it as the same action and resolve the conflict, or as a new action with a new key? I go back and forth on where that line belongs.

Gate a risky AI agent tool call with a permit, and keep a receipt

Chris Sellers — Thu, 08 Oct 2026 03:00:32 +0000

Your agent calls a payout tool. The payout goes through on the other side, but the response never makes it back. The client times out, and the agent does the reasonable thing: it retries. Now the vendor has been paid twice.

Swap "payout" for "delete the staging database" or "send the customer email" and the story is the same. The problem is that nothing between the agent and the tool knows the second call is the same action as the first.

This post walks through a pattern for that gap, first in general terms you can build yourself, then how Agent Middleware (AMW), the project I work on, applies it. The Python below is a simplified sketch I wrote for this post, not AMW's source. The one AMW snippet is the request shape from its README.

Two terms first

A permit is a signed, scoped grant that says what an agent may do: which tools, which scopes, how much it may spend, and until when. The gateway checks it before anything costs money or touches the outside world.

A receipt is a signed record of what happened to one governed call: which permit it ran under, which tool, what was charged, and the outcome. A refusal under a valid permit gets one too, so "denied" leaves evidence just like "succeeded".

The general pattern

You can build this in front of any tool that has side effects. The flow:

The agent picks an idempotency key per logical action, before the first attempt, and keeps it across retries. Not per HTTP request. Per "pay invoice 4417".
The gateway hashes the request. Tool name, arguments, who is calling, and under which permit.
The gateway looks up the key. If it already exists with the same hash and a finished result, return the stored response and receipt. Do not call the tool. Do not charge.
If the key exists with a different hash, refuse. A reused key with a new payload is a bug or an attack, not a retry.
If the key exists and the first call is still running, say so. Return "in progress" rather than starting a second call.
Only for a new key: check the permit, reserve budget, call the tool once, and store the response and receipt under that key.

The ordering matters. The replay check comes before the permit check, the charge, and the dispatch, so a retry never gets a chance to spend anything.

A simplified sketch

This is illustrative code to show the shape of the idea. It is not AMW's implementation, and the store, permits, tools and sign_receipt pieces are stand ins for whatever your stack uses.

import hashlib
import json


class KeyReused(Exception):
    pass


class InProgress(Exception):
    pass


def request_hash(tool, args, caller, permit_id):
    canonical = json.dumps(
        {"tool": tool, "args": args, "caller": caller, "permit": permit_id},
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(canonical.encode()).hexdigest()


def governed_call(store, permits, tools, *, key, tool, args, caller, permit_id):
    h = request_hash(tool, args, caller, permit_id)

    # Look up the key before anything can spend money or touch the tool.
    record = store.get(caller, key)
    if record is not None:
        if record.request_hash != h:
            raise KeyReused(key)        # same key, different request
        if record.response is None:
            raise InProgress(key)       # first call still running
        return record.response          # original result and receipt

    # New key only. The store must enforce uniqueness on (caller, key),
    # so two racing first attempts cannot both get past this line.
    record = store.insert(caller, key, h)
    permit = permits.check(permit_id, caller=caller, tool=tool)  # raises on deny
    permit.reserve(tools[tool].price)
    result = tools[tool].call(args, idempotency_key=key)
    response = {"result": result, "receipt": sign_receipt(permit, tool, h, result)}
    store.finish(record, response)
    return response

A real version also needs a signed receipt for denials, a budget release when the call fails before it is sent, and an answer for the case where the tool may have acted but you never heard back. That last one is where most of the difficulty lives.

How AMW applies it

In AMW, agents act under a permit; a retry with the same key returns the original receipt instead of dispatching again; a new key is a new request.

A governed call is an MCP tools/call with the wallet, permit and idempotency key carried in mcpContext. This is the request shape from the AMW README:

curl -sS -X POST "$API_URL/mcp/messages" \
  -H "X-API-Key: $AGENT_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"jsonrpc\": \"2.0\",
    \"id\": \"request-1\",
    \"method\": \"tools/call\",
    \"params\": {
      \"name\": \"$TOOL_ID\",
      \"arguments\": {\"input\": \"hello\"},
      \"mcpContext\": {
        \"wallet_id\": \"$WALLET_ID\",
        \"permit_id\": \"$PERMIT_ID\",
        \"idempotency_key\": \"invoke-1\"
      }
    }
  }"

The behavior follows the same steps as the sketch:

Governed calls require a key. Retry safety is not opt in.
The request hash covers the tool name, the arguments, the wallet and the permit. The key is scoped per wallet and backed by a database uniqueness constraint.
Reusing a key with a different payload fails closed with idempotency_key_reused, and no new receipt. A retry while the first call is still running gets idempotency_in_progress.
A replay returns the stored response before permit validation, the charge, or the dispatch happen. AMW's own test suite sends the same governed call twice and asserts the same receipt ID and a single ledger debit.

On the permit side, each governed call is checked against a signed permit: which wallet and key may use it, which tools and scopes it covers, a credit budget, an expiry, and optional limits such as per tool call counts, an aggregate value cap, forbidden argument fields, and a required recipient domain. Calls outside the permit are refused before any charge.

What the receipt binds

Every governed call, whether it succeeds, is denied, or fails, produces an Ed25519 signed receipt. The receipt binds the permit, wallet, tool, credits authorized and charged, outcome, ledger entry and audit event, plus SHA-256 hashes of the request and the tool's response. A receipt can be exported as a portable bundle and checked by anyone who holds it, with no account, using our published public key and any Ed25519 library, or our SDK verifier (function or CLI), shared on request. Verification trusts the key AMW publishes; there's no transparency log yet.

Limits

A new key is a new call. A retry sent under a new idempotency key is treated as a new call, not a repeat, and it does not get the original receipt back. If the permit sets a per tool call limit and that limit is used up, the retry is refused. Do not count on anything else to catch it. An agent that restarts and mints a fresh key for the same action can still pay twice. Keep the key with the task, not the process.
Not exactly once at the remote tool. AMW makes each accepted key produce at most one call to the tool, one charge and one receipt. Whether the remote side effect itself happens only once depends on the tool honoring the forwarded idempotency key.
Lost responses stay uncertain. For remote tools, AMW records a durable checkpoint right before it calls the tool. If it crashes before that point, the charge is refunded and the budget released. If it crashes or loses the connection after that point, the call is kept as charged and marked delivery_uncertain (the gateway sent it, but cannot say what the tool did) with a signed receipt. AMW never re-sends the call automatically. Someone still has to reconcile it with the tool.
Receipts sign hashes, not content. They do not prove the tool's output was correct, and the signing key is a single injected key with no KMS.
Narrow scope today. MCP is the only governed adapter, upstream execution is limited to one configured tool per deployment, and the supported deployment is vendor managed and single tenant. I am not promising dates for anything beyond that.

Disclosure

I'm Chris, the founder and sole owner of AMW. It is pre-revenue. This post was drafted with help from an AI assistant; the Python sketches are illustrative and are not AMW's source, the curl call is from AMW's README, and every product claim was checked against the AMW code. A sample signed receipt and the public API docs are linked from thisisatest.tech.

The pattern above works without AMW. The hard part in my experience is step 1, so here is my question: when your agent retries a tool call, where does its idempotency key come from, and does it survive a process restart?