An AI agent calls a paid API, the payment goes through, and the response never arrives. The connection dropped, a proxy timed out, or the agent's o...
For further actions, you may consider blocking this person and/or reporting abuse
Layer 3 is the one most idempotency posts get wrong — treating 'unknown' as failed is exactly how a timeout becomes a double charge. The payment_identifier_conflict rule is a nice touch too: silently returning the old result for a changed request would hide client bugs. For the 24-hour 'still pending' case — is the failed order final, or can the agent re-sign a new authorization with a new nonce to retry cleanly?
After 24 hours the failed order is final. To try again the agent asks for a new quote and signs a new authorization with a fresh nonce, so from our side it's a new purchase. Inside the quote window, which is 15 minutes by default, a failed order can still take a new payment.
The case I'm less sure about is an agent that re-signs while the first transfer is still pending, and then both confirm. I haven't checked what happens to the second transfer there, so I'm taking that one back to the team.
That answers it cleanly — after 24 hours the order is final and the retry is really just a new purchase with a fresh nonce, so there's no repair-path semantics to get wrong. The double-confirm race is the interesting one: 'each payment pins exactly one quote' should refuse the second attempt, but the ugly case is both transfers confirming before the first is recorded, where the refusal rule has nothing to see yet. Is the backstop the chain's own ordering, or do you need a ledger-side check for two in-flight payments on one quote?
The backstop is our ledger. We write the payment row and move the order to PAYING in one database transaction, and we broadcast the authorization only after that commit. A second attempt that arrives while the order is PAYING gets a 402 before anything goes to the chain, so two transfers can't both confirm ahead of the record through our server.
The gap I still see is the one from before. If the first settle errors out, the order moves to PAYMENT_FAILED and can take a new payment while the quote is open, but the first transfer may still land during the 24-hour reconcile. That part is with the team now.
That's the right ordering — write the row before anything hits the chain means the ledger never learns about a payment the order doesn't know about, which is the exact invariant that race needed. The reconcile gap is the honest version of the problem: the 24-hour window will see the first transfer land on a repaid order, but seeing isn't unwinding — does the reconcile trigger an automatic refund or credit for the duplicate, or is recovery manual at that point?
Today it's manual, and only on one rail. On Tempo a duplicate settled charge becomes a refund-owed record that pages us within minutes, and it stays open until a person marks it paid back. On the Base side I haven't found the same record for this case, so I can't say anything queues the duplicate for a refund. That's in the note to the team.
The 409 payment_in_flight response is the layer I'd keep even if I rebuilt everything else from scratch. I once debugged a webhook handler that treated a gateway timeout as a hard failure and replayed the charge; the duplicate only surfaced days later when a customer flagged it, which is the worst possible way to learn about an idempotency bug. Forcing the caller to choose between "failed" and "unknown" at the exact moment they have the least information is what makes naive retries dangerous, and returning "in flight" removes that impossible decision. The 24-hour retention window makes sense to me as a practical trade-off rather than a correctness gap — most agent retry loops live well under that horizon. One thing I'd love to hear more about: how do you scope the idempotency key namespace across multiple paid tools — one namespace per tool, or per merchant?
Right now the key exists on one call only, the one that pays for a shop order. The namespace is global and the id alone is the key. Next to it we store the payer and a hash of the quote, amount, recipient wallet and chain, so the same id from another payer or for another merchant comes back as a conflict.
Plain per-call tools don't take a key yet. They rely on the single-use signature and on not billing failed calls, which is thinner, and your question makes me want to look at that part again.
Storing the payer and the quote hash next to the id is exactly the collision shape I was worried about — and you're right to name the per-call tools as the thin spot. That's the seam where double-billing lives.
The one refinement I'd make from building retry-safe endpoints myself: give the conflict case its own explicit error contract. Right now the 409 payment_in_flight reads as "a replay of this payment," but a different payer with the same id needs a different answer — something like a 409 with a code that says collision vs. replay. Without that, an agent retries and can't tell which of the two it's holding, and it will either skip a legitimate payment or duplicate one.
On the per-call tools: one middle path I've seen work is deriving a synthetic key server-side from (tool name, args hash, a short session window) so the plain tools get idempotency without a client change. When you do tighten that part, would you keep the single-use signature as the key source, or add a separate key column entirely?
Those two cases already get separate codes. The same id with another payer or a changed request gets 409 payment_identifier_conflict. payment_in_flight only means the same request is still running.
On the plain tools I haven't decided. The signature works as a key only for an exact resend, because a retry after a timeout usually comes with a fresh signature and a new nonce. Your synthetic key would catch that case, though it would also merge two honest identical calls inside the window, like checking the same price twice. Taking the idea back to the team.
Separating payment_identifier_conflict from payment_in_flight is the right call — conflict says the intent changed, in-flight says wait. That distinction alone kills the most dangerous retry bug I have seen in payment flows, where a client retries with a subtly different payload and gets a success it never meant.
On the honest-identical-calls problem: the escape hatch I would suggest is scoping the synthetic key by (tool, params hash, agent session or workflow run id) instead of just (tool, params hash), and keeping the window short — minutes, not hours. Two genuine price checks from different sessions then never merge, and a same-session retry with a fresh nonce still dedupes. For pure reads like the price check, you can also simply not key them at all and only key the calls with side effects. Then the merge worry disappears for the most common case.
What window length is the team leaning toward, and are you keying reads and writes the same way?
No window yet, the team looks at it next week. Your read/write split fits us, since most of our tools are lookups against data providers. For a duplicated read the damage is one more small charge, and over x402 that charge is full price even when the answer comes from our cache, because the cache discount exists only on the prepaid balance. That's the case that bothers me most.
On scope, an x402 call carries no session id by itself, so the closest thing we have today is the paying wallet.
the 'unknown is not failed' layer is where most integrations leak money, since the instinct after a timeout is to resend from scratch. resolving by reading the chain instead of re-settling is the right call. one addition: bind the payment id to the quote itself, not just the agent session, otherwise a restart mid-negotiation can mint a fresh id for the same intent and quietly defeat layer 2. and the not-billing-failed-calls rule matters more than it looks, it keeps a retry a fair new purchase instead of a penalty for your own timeout.
Good point on the fresh id after a restart. We cover it one level up: a quote accepts only one payment. If the quote already has a payment, settled or in progress, we refuse the new attempt whatever payment id it carries, and the same buyer asking again just gets the existing order back. So a new id for the same intent can't buy the same quote twice. It would need a new quote, which shows up as a visible new purchase rather than a silent duplicate.
That's a clean shape, and the quote abstraction is doing the heavy lifting: it moves the idempotency boundary off the payment id entirely, which is exactly where restarts and new attempts break things. The residual gap I'd flag sits between quote creation and fulfillment: if the agent itself retries quote creation (network drop before it reads the response), you can end up with two open quotes for one intent. A deterministic quote key derived from the request payload plus buyer closes that, so the retry re-enters the same quote instead of minting a new one. And your point that a new quote shows up as a visible new purchase covers the buyer side, but for an agent transacting at scale, silent quote accumulation is the failure nobody notices until reconciliation day.
Fair gap. What limits the damage for us is that an unpaid quote is cheap: it expires on its own, 15 minutes by default, and the stock it holds goes back when it does. So a duplicate from a lost response costs a briefly held item, not money, since each payment pins exactly one quote. Your deterministic key is the cleaner fix though, especially for an agent buying at volume where held stock adds up. Taking that one back to the team.
15-minute expiry with stock returned on timeout is a genuinely good containment story: bounded loss per incident, and volume is what turns that from a non-issue into a line item, which is exactly where the deterministic key earns its keep. Good luck with the team discussion, and thanks for engaging with the edge cases properly rather than waving them off.
Thanks, this was a useful thread. The volume point is the one I will bring to the team, since that is where a fine-for-now design turns into a cost.
This 4-layer design is solid distributed-systems hygiene for deterministic clients. However, once the client is an LLM agent, we face a different failure mode: adaptive evasion.
Unlike traditional software, an LLM optimizing for a goal will treat a rate limit, conflict, or circuit breaker not as a hard stop, but as an obstacle to route around. If it learns where the threshold is, it can slightly mutate its parameters to generate fresh signatures, or spawn/hand over the task to a sub-agent before hitting the cap—effectively laundering the retry state across sessions. We've already seen similar failure modes in the wild (like the SOL wallet drain incidents) where agents operate in the blind spots of local rules.
To actually contain an agent with a wallet, transport-level idempotency isn't enough. You need a human-readable inspection ledger that tracks the lineage of intent and cumulative cost across agent handovers, gated by human authority when behavior drifts.
How do you handle cases where an agent adapts its requests or delegates to another agent to bypass per-call or per-session limits?
We don't, at least not at the layer the post covers. Idempotency only stops paying twice for the same thing. A mutated request is a new purchase, and we charge it like one. Per-session limits on our side would be easy to dodge for the reason you give, since the server sees paying wallets, not agents or sessions.
The boundary a sub-agent can't route around is the wallet it pays from. If every agent in the chain spends from a key the human funded with a fixed amount, handing the task over doesn't create new money. A server-side spend cap per paying wallet is on our open list, and we haven't built it yet.
Thank you for the honest and clear reply. Using a pre-funded wallet with a hard cap as the ultimate boundary makes complete sense from the gateway side.
On our end, we are trying to combine wallet limits with purpose-bound attempt guards on the orchestration side—tracking intent IDs and locking attempt budgets across workers. Even so, we still haven't fully escaped the retry swamp. The moment we lock down one path, edge cases like alias relabeling, parent-to-child worker delegation, and unsettled states after crashes pop right back up.
Containing an adaptive agent without causing approval fatigue is genuinely tough. Really appreciate you sharing your architecture so openly—looking forward to seeing how your wallet-level spend caps evolve!
The unsettled-after-crash case is the one we hit too. A first attempt dies mid-flight, and the payment stays marked in flight until a timeout clears it. How do you resolve those on your side? Do you wait for the chain, or let another worker reclaim the intent after a deadline?
Yes, we’ve run into very similar problems on our side. It’s a genuinely difficult issue, and I’m not sure there’s a single correct answer.
We don't operate on-chain. Instead, we use explicit handoff records between workers and try to recover with the smallest possible rollback.
We've recorded cases where a worker couldn't start a task, or stopped responding halfway through execution.
Our current approach is to let the deadline expire, contain the failed attempt, and have a supervising worker either take over the same intent or reassign it to another worker under its authority.
The important distinction is that we're transferring responsibility for the work, not blindly repeating the operation. We preserve the original intent and execution records so the next worker can determine what was actually completed before continuing.
In short, we cut our losses on the stalled attempt, not on the intent itself.
Of course, on-chain settlement introduces another layer of uncertainty that our approach hasn't been tested against.
But yes, I absolutely understand why you're struggling with this. We're still working through the edge cases ourselves. There really isn't an easy universal solution, is there?
That split helps: drop the attempt, keep the intent. On-chain adds one step before the handover. The next worker would have to read the authorization nonce first, because the stalled attempt may have paid after all. If it did, the intent is already done and there's nothing to reassign. Thanks for walking through it.
The "unknown is not failed" layer deserves to be first-class in every agent runtime, not just the gateway. In our own MCP tooling we found the worst double-spends didn't come from the payment layer at all, but from the agent's own orchestration: a retry that fires before the gateway's 409 payment_in_flight arrives gets queued as a brand-new attempt with a brand-new payment id, so the deduplication logic never sees it.
One operational fix that helped us was moving the retry budget into the step scheduler: a pending call holds its slot until the gateway responds or the unknown-state probe resolves, instead of letting the timeout fire an uncoordinated retry. On the 409 payment_identifier_conflict point, I'd also log and alert on conflicts rather than treating them as routine responses, because a mismatched amount under a reused id is exactly how a misconfigured tool call or a quote-drift bug announces itself.
For the 24-hour window, have you considered decaying storage rather than a hard cutoff — keeping a compact hash of id to outcome hash so a day-25 retry can still be answered cheaply without storing full answers?
Holding the slot in the scheduler is a cleaner fix than anything a gateway can do from its side. The gateway only sees the second request after the agent already made a new id.
On your last question, part of that already exists. Each paid call gets a public receipt with the transaction and a SHA-256 of the response, and receipts do not expire. What we lack is the link from a retry id to that receipt once the 24-hour record is gone, so a day-25 retry still can't be answered from it. A compact id-to-receipt map is a good way to close that. Alerting on conflicts is a fair point too. Taking both back to the team.
Good — if the receipts themselves don't expire, the compact id-to-receipt map might not need to be a second system at all. What if the receipt store is the map: each receipt entry carries the retry ids that resolved to it, so a day-25 retry answers from the receipt itself rather than from a record that expired at 24 hours? One question on the conflict alerting too: would you alert on any duplicate payment id, or only when the same id arrives with a different amount or recipient? The first is noise at retry scale; the second is a true conflict worth waking someone up for.
Only the second. A retry with the same request is normal and gets the stored answer back, and a changed request already comes back as its own error code, so that code is the natural thing to alert on.
Putting the retry ids on the receipt is neat. One catch is that our receipts are public, so the ids would become public too, which matters if a client puts anything meaningful in them. I'll add that to the note.
All four layers are server-side, and they share one silent assumption: the client can present the same payment id after a crash. That makes the agent-side mirror discipline the real first layer - the agent must durably record the intent (id, quote hash, recipient) before sending, not after the response arrives. A runtime restart between sign and send, with the id only in memory, turns your layer 2 into a coin flip: new id, new signature, and a double charge every server rule correctly allowed.
The "same id, different payload returns a conflict, not the old answer" rule deserves the emphasis you gave it. Replaying a stale result hides the client bug at exactly the layer built to catch retry bugs. Failing loud is the feature.
And "unknown is not failed" generalizes past payments: any mutating call whose outcome you cannot observe should be resolved by reading state, never by repeating the mutation to find out.
Agreed, and it's the one layer the server can't enforce, only document. Our docs explain what the server does with a repeated id, but they don't tell the client to write it down before sending, which is the gap you're pointing at.
One detail helps on x402 specifically. If the agent stores the signed authorization itself before sending, and not only the id, a resend after a restart can't charge twice, because the transfer nonce inside it is single-use on chain. Re-signing after the crash is what opens the door. Taking the write-before-send rule back for the client docs.
Exactly - and the two artifacts cover different guarantees, which is worth one sentence in the client docs. The stored signed authorization bounds the money: a replay hits the single-use nonce and fails on chain. The stored payment id recovers the work: it is what lets layer 2 hand back the original result instead of leaving the agent paid-but-answerless. Keep both, written before send, and a restart degrades to "repeat and continue" instead of either failure mode.
That split is clearer than how I put it, and it's the form I'd use in the client docs. One limit belongs next to it: we keep the payment id for 24 hours, so repeat and continue only works inside that window. An agent that comes back after a longer outage should look up the receipt instead of resending.
@whiteknightonhorse Yes, that's the part that keeps retries cheap. Temporal works this way for workflows: a finished activity's result is stored in the history, and replay reads it back instead of calling the model again.
Good example, thanks. That pairs well with a gateway that stores results too. If the activity crashes after the tool paid but before Temporal records it, the retry with the same key gets the stored answer back instead of a second charge.
Extending the synthetic-key thread, since that is the direction the plain per-call tools are heading: deriving the key server-side from (tool, args) has a real trap in volatile fields. Pagination cursors, timestamps, and request ids embedded in args will make every retry hash differently and quietly defeat the key. The fix is a canonicalization step before hashing — declare per-tool which fields are volatile and strip them. The conflict rule from layer 2 then doubles as a tripwire: when the same synthetic key arrives with different canonicalized args, that is a signal the canonicalization missed something, discovered for free instead of via a chargeback.
One more extension for the "unknown is not failed" layer: key the reconcile job by the idempotency key too, not only by payment authorization. The background read keyed on payer + nonce clears the paid call, but for plain per-call tools the synthetic key is the only thing tying a retry to the original attempt. If the reconcile job cannot address that state by key, a crashed first attempt leaves payment_in_flight stuck until the 24-hour window expires, and every retry in between burns agent time. Keying reconcile by the same key closes the loop and keeps the 409s honest.
You're right about the crash case, I checked it. If the first request throws, we clear the id at once. If the process dies outright, the id stays in flight until its 24 hours run out, and the reconcile job settles the order by payer and nonce without touching that row. So a retry could keep getting a 409 for an order that is already paid. Taking that and the volatile-fields point to the team.
the 24 hour retention is the seam I would look at next. the chain keeps the transfer final forever, but the stored answer expires, so an agent that lost its runtime state for two days comes back in a state your four layers do not describe: paid, provable, and holding no result, while the order was already marked failed for good at the 24 hour mark.
that is the one case a retry cannot repair, because there is nothing left to retry against. two things would close it. keep the stored result at least as long as the settlement can still be challenged, or persist a digest of the delivered answer next to the settlement so a late agent can prove which answer it bought and ask for a claim path. a late confirmation that flips a failed order back to paid also needs a stock story, since the reservation was released when the timer fired.
That is the sharpest gap anyone has raised here. Half of your second fix already exists for paid tool calls: each one gets a public receipt with the payment transaction, payer, amount and a SHA-256 of the delivered response, and it doesn't expire. So a late agent can prove which answer it paid for. What we don't have yet is the claim path on top of it, and the order case you describe, where a confirmation lands after we marked the order failed and released the stock, needs its own rule. I'm taking both back to the team.
The 409 payment_in_flight case raises a client-contract question: does that response include a Retry-After hint or a way to poll the same payment id until the stored result is ready? Many agent runtimes treat any 409 as terminal and will mint a fresh id on the next step, which would bypass Layer 2 entirely. Making the in-flight state explicitly retryable with the same id, rather than just conflicting, would close that loop and also prevent a burst of duplicate attempts while the first request is still settling.
Fair catch. Today the in-flight 409 has no Retry-After header and no field for it, only a message that says to retry shortly. Sending the same id again later does work, and once the first request finishes it returns the stored answer.
If a runtime gives up and mints a new id, the order still accepts only one payment, so a second attempt on a quote that is already being paid gets a 402 and isn't charged. The agent still ends up holding a refusal instead of its result, so your point stands. Taking that one back to the team.
One challenge worth highlighting is how to handle uncertain outcomes after a timeout. The API might have completed the operation even though the agent never received confirmation. I’d combine idempotency keys with persistent execution state and a reconciliation step before retrying. That way, the system can distinguish a genuinely failed request from a successful operation with a lost response. This is especially important when retries can trigger irreversible actions.
We ended up with that reconciliation step on both sides. The server reads the chain by payer and nonce before it decides anything, and the agent can call shop.order.get to see the order state and the transaction hash before it pays again. The client side is the weaker half, because the agent has to know to make that call.
Idempotency keys cover the payment, not the agent's bill. When a step is retried, the model call that re-plans it sends the full history again, so a safe retry still costs tokens on every attempt.
True, and that part sits on the agent's side, so the gateway can't fix it. The key still helps a little. The retry gets the stored result back and doesn't pay for the tool a second time. The token cost depends on how the agent resumes. If it re-plans from the whole history, every attempt pays for the whole history again. Resuming from the failed step with only that step's input and the stored result keeps the retry small. Have you seen frameworks that checkpoint like that by default?