An AI agent calls a paid API, the payment goes through, and the response never arrives. The connection dropped, a proxy timed out, or the agent's own runtime restarted. From the agent's side the call failed, so it does the reasonable thing and tries again.
If the API charges per call, that retry is where money goes missing. The first payment may have settled on chain while the agent never saw the result. A naive retry pays a second time for the same work.
We run an MCP gateway where agents pay per call in USDC, over x402 on Base or MPP on Tempo. This post walks through the four layers we use so that a retry never turns into a double charge, and where each layer stops helping.
Layer 1: the payment authorization is single-use
With x402 on Base, the agent does not send money directly. It signs an authorization for a USDC transfer (EIP-3009), and the server submits it. Each authorization carries a nonce, and the token contract accepts a given nonce from a given payer only once.
So if the agent resends the exact same signed payment, the second transfer fails on chain. No server-side bookkeeping is needed for that guarantee, and it holds even if our own records are lost. A new charge always needs a new signature from the agent.
MPP has the same property on its side. A credential that was already used comes back as 403 credential_replayed, and the call is not charged.
Layer 2: a payment id makes the retry return the stored answer
Layer 1 stops the double charge, but it leaves the agent stuck: it paid, it has no result, and resending the payment now fails. To fix that, on our order payments (agents buying from shops through the gateway) the agent can attach a payment id to the payment: the payment-identifier extension of x402, 16 to 128 characters.
The rules we apply:
- Same id, same payment: we return the stored answer from the first request, with no second verify and no second settle. We keep that answer for 24 hours.
- Same id while the first request is still running:
409 payment_in_flight. - Same id with a different quote, amount, recipient or payer:
409 payment_identifier_conflict.
The last rule matters more than it looks. If a changed request with an old id quietly got the old result, idempotency would hide a client bug. The agent would think it bought one thing and actually receive another. A conflict forces the client to notice.
Layer 3: "unknown" is not "failed"
The hardest case is the one in the middle. We broadcast the settlement, and then the facilitator call times out or the transaction is still pending. Did the payment happen?
We treat that state as unknown and never as failed. A background job reads the chain: the authorization state for that payer and nonce, and the USDC transfer itself. If it confirms, the order becomes paid and the result is delivered. If it is still not confirmed after 24 hours, the order is marked failed for good and any reserved stock is released.
What we never do is settle the same authorization a second time to "make sure". That is the classic way a timeout turns into two charges.
Layer 4: failed calls are not billed
For tool calls, we only charge for a delivered response. If the upstream provider rejects the input or fails, the call is not charged. Last week that rule covered 1,905 calls rejected by providers, none of them billed.
This does not prevent retries, but it means a retry after a genuine failure is a fresh, fair purchase rather than a second payment for a call that never produced anything.
Where the layers stop
The 24-hour window is a choice, not a law. An agent that comes back after 25 hours with the same payment id will not get the stored answer. It still cannot be charged twice, because Layer 1 rejects the old signature on chain. What it loses is the cached result, so it has to decide whether to pay again for a fresh call.
That is the trade-off we accepted. The record only needs to cover the agent's realistic retry and recovery period. The guarantee against double charging does not depend on the record at all, because it lives in the payment itself.
A checklist if you build paid tools for agents
- Make the payment instrument single-use (a nonce or a one-time credential), so a resend cannot charge twice even if your database is wrong.
- Give clients an idempotency key and return the stored answer for a repeat, for a stated window.
- Return a conflict when the same key arrives with a different request instead of replaying the old result.
- Treat a timed-out settlement as unknown, resolve it by reading the chain, and never settle again to find out.
- Charge only for delivered results, so a retry after a real failure is a fair new purchase.
None of this is specific to AI. It is distributed-systems hygiene that matters more once the client is a program that retries on its own, at machine speed, with a wallet attached.
Top comments (74)
Layer 3 is the one most idempotency posts get wrong — treating 'unknown' as failed is exactly how a timeout becomes a double charge. The payment_identifier_conflict rule is a nice touch too: silently returning the old result for a changed request would hide client bugs. For the 24-hour 'still pending' case — is the failed order final, or can the agent re-sign a new authorization with a new nonce to retry cleanly?
After 24 hours the failed order is final. To try again the agent asks for a new quote and signs a new authorization with a fresh nonce, so from our side it's a new purchase. Inside the quote window, which is 15 minutes by default, a failed order can still take a new payment.
The case I'm less sure about is an agent that re-signs while the first transfer is still pending, and then both confirm. I haven't checked what happens to the second transfer there, so I'm taking that one back to the team.
That answers it cleanly — after 24 hours the order is final and the retry is really just a new purchase with a fresh nonce, so there's no repair-path semantics to get wrong. The double-confirm race is the interesting one: 'each payment pins exactly one quote' should refuse the second attempt, but the ugly case is both transfers confirming before the first is recorded, where the refusal rule has nothing to see yet. Is the backstop the chain's own ordering, or do you need a ledger-side check for two in-flight payments on one quote?
The backstop is our ledger. We write the payment row and move the order to PAYING in one database transaction, and we broadcast the authorization only after that commit. A second attempt that arrives while the order is PAYING gets a 402 before anything goes to the chain, so two transfers can't both confirm ahead of the record through our server.
The gap I still see is the one from before. If the first settle errors out, the order moves to PAYMENT_FAILED and can take a new payment while the quote is open, but the first transfer may still land during the 24-hour reconcile. That part is with the team now.
That's the right ordering — write the row before anything hits the chain means the ledger never learns about a payment the order doesn't know about, which is the exact invariant that race needed. The reconcile gap is the honest version of the problem: the 24-hour window will see the first transfer land on a repaid order, but seeing isn't unwinding — does the reconcile trigger an automatic refund or credit for the duplicate, or is recovery manual at that point?
Today it's manual, and only on one rail. On Tempo a duplicate settled charge becomes a refund-owed record that pages us within minutes, and it stays open until a person marks it paid back. On the Base side I haven't found the same record for this case, so I can't say anything queues the duplicate for a refund. That's in the note to the team.
The asymmetry is the story here — Tempo-side duplicates page you within minutes, while Base-side ones have no refund record and no queue, so they're only found if someone happens to look. Extending the refund-owed record to the Base path looks like the natural close-out of that team note. Is there a reason the two rails diverge, or is it just that Tempo's was built first?
Mostly the order in which money moves. On Tempo the charge settles before the work runs, so a failed or duplicate call leaves money that has to go back, and we built the refund-owed record for exactly that. On Base we settle only after the result is ready, so a failed call never costs anything and nobody needed a refund queue there.
The duplicate after a failed settle is the one Base case that breaks that assumption, which is why it has no record yet.
That ordering explains the divergence cleanly — the refund queue exists where money moves first, and Base never needed one until this edge case. It also suggests the fix is narrow: a refund-owed record only for that post-failed-settle duplicate, not a whole Base refund pipeline. Is that the shape the team note is converging on, or would they rather unify the rails?
Not decided yet. What's on the table is one refund path for both rails, paid in batches from a wallet set aside for it, so a payer doesn't have to care which rail they used.
The record feeding it would stay narrow on Base though, only the cases where money moved and no result went back. Your narrow version is the fallback if the shared path turns out heavier than it's worth, and I'm taking that argument back with me.
The unified path makes sense from the payer's side — they shouldn't need to know which rail failed them. And keeping the narrow Base record as the fallback means the shared path doesn't have to be perfect on day one to be worth building. If the batch wallet works, does the payer see one refund timeline regardless of rail, or do settlement speeds still leak through?
Settlement speed barely matters here. Both rails confirm a transfer in seconds, so with batched refunds the payer actually waits on the batch schedule.
What could still differ is how fast we notice. On Tempo we write the refund-owed record the moment the call fails, while on Base we would only see the case when a reconcile pass finds it. So one timeline probably has to start at detection, and the Base check needs to run often enough for that to mean something. We haven't built any of this yet.
Detection as the real start is the right correction — the batch schedule is shared, but the clock only starts when the duplicate is noticed, and on Base that notice is a scheduled pass, not an event. So the unified timeline really hinges on reconcile cadence: how often would that pass need to run before a Base payer's wait looks anything like Tempo's minutes? Or does this push the Base check toward a chain watcher rather than a periodic reconcile at all?
Neither, I think. The Base case starts with something we do see as an event: our own settle call coming back as a failure. At that moment we know exactly which authorization to look at, and the USDC contract will tell you whether that nonce has been used.
So the check can be a targeted read, repeated a few times over the next minutes, with the periodic reconcile only as a backstop for whatever slips through. That would put a Base payer close to Tempo's minutes. It's still a design, not code.
That third path is cleaner than either option I offered — the failed settle itself is the event, so the check is targeted, not a watcher scanning everything. The 'repeated a few times over the next minutes' detail is what makes it Tempo-close rather than reconcile-close. The one thing I'd still want pinned down before it becomes code: how many repeats, and what happens if the nonce read is inconclusive after the last one — back to the reconcile, or a human flag?
Not pinned yet, so this is my lean rather than a decision. An inconclusive read after the last repeat stays unknown, not failed, and drops to the reconcile. A human flag makes sense only if the reconcile can't settle it either.
The repeat count I'd rather set from real settle latency on Base than guess now. Taking both back to the team.
That ordering is the honest one — 'unknown' stays unknown rather than being quietly promoted to failed or retried, and the human flag only earns its place if the reconcile can't settle it either. Setting the repeat count from real Base settle latency rather than guessing it is the detail that makes the design testable. When you take it back to the team, is the latency distribution the thing you'd measure first, or the reconcile's own detection rate?
Latency first, I think. It sets the repeat window, and we can collect it from every settle, not only the failed ones. The reconcile's detection rate is harder. Real misses are rare, so we'd have to seed known cases to measure it. That one comes second.
Latency first is the right order — you can collect it from every settle, not just the failures, so the repeat window is set by real data rather than a guess. The seeded-cases point is honest too: with real misses rare, you can't measure detection without manufacturing them. The thing I'd watch is whether the seeded cases stay representative — if they all look like the failure you've already imagined, do they still prove the reconcile catches the one you haven't?
They don't, and that's the limit. Seeded cases only prove the reconcile catches the failures we already thought of. The unknown ones show up as real incidents, so each one we find becomes a new seeded case. Thanks, this thread gave me a lot to take back.
That seeded-case loop is the honest close to the thread — each real incident becomes the next test, so the reconcile gets better exactly where it was blind before. Good luck taking it back to the team; if the latency numbers change the repeat window, I'd be curious what they come out as.
The 409 payment_in_flight response is the layer I'd keep even if I rebuilt everything else from scratch. I once debugged a webhook handler that treated a gateway timeout as a hard failure and replayed the charge; the duplicate only surfaced days later when a customer flagged it, which is the worst possible way to learn about an idempotency bug. Forcing the caller to choose between "failed" and "unknown" at the exact moment they have the least information is what makes naive retries dangerous, and returning "in flight" removes that impossible decision. The 24-hour retention window makes sense to me as a practical trade-off rather than a correctness gap — most agent retry loops live well under that horizon. One thing I'd love to hear more about: how do you scope the idempotency key namespace across multiple paid tools — one namespace per tool, or per merchant?
Right now the key exists on one call only, the one that pays for a shop order. The namespace is global and the id alone is the key. Next to it we store the payer and a hash of the quote, amount, recipient wallet and chain, so the same id from another payer or for another merchant comes back as a conflict.
Plain per-call tools don't take a key yet. They rely on the single-use signature and on not billing failed calls, which is thinner, and your question makes me want to look at that part again.
Storing the payer and the quote hash next to the id is exactly the collision shape I was worried about — and you're right to name the per-call tools as the thin spot. That's the seam where double-billing lives.
The one refinement I'd make from building retry-safe endpoints myself: give the conflict case its own explicit error contract. Right now the 409 payment_in_flight reads as "a replay of this payment," but a different payer with the same id needs a different answer — something like a 409 with a code that says collision vs. replay. Without that, an agent retries and can't tell which of the two it's holding, and it will either skip a legitimate payment or duplicate one.
On the per-call tools: one middle path I've seen work is deriving a synthetic key server-side from (tool name, args hash, a short session window) so the plain tools get idempotency without a client change. When you do tighten that part, would you keep the single-use signature as the key source, or add a separate key column entirely?
Those two cases already get separate codes. The same id with another payer or a changed request gets 409 payment_identifier_conflict. payment_in_flight only means the same request is still running.
On the plain tools I haven't decided. The signature works as a key only for an exact resend, because a retry after a timeout usually comes with a fresh signature and a new nonce. Your synthetic key would catch that case, though it would also merge two honest identical calls inside the window, like checking the same price twice. Taking the idea back to the team.
Separating payment_identifier_conflict from payment_in_flight is the right call — conflict says the intent changed, in-flight says wait. That distinction alone kills the most dangerous retry bug I have seen in payment flows, where a client retries with a subtly different payload and gets a success it never meant.
On the honest-identical-calls problem: the escape hatch I would suggest is scoping the synthetic key by (tool, params hash, agent session or workflow run id) instead of just (tool, params hash), and keeping the window short — minutes, not hours. Two genuine price checks from different sessions then never merge, and a same-session retry with a fresh nonce still dedupes. For pure reads like the price check, you can also simply not key them at all and only key the calls with side effects. Then the merge worry disappears for the most common case.
What window length is the team leaning toward, and are you keying reads and writes the same way?
No window yet, the team looks at it next week. Your read/write split fits us, since most of our tools are lookups against data providers. For a duplicated read the damage is one more small charge, and over x402 that charge is full price even when the answer comes from our cache, because the cache discount exists only on the prepaid balance. That's the case that bothers me most.
On scope, an x402 call carries no session id by itself, so the closest thing we have today is the paying wallet.
That cache-discount asymmetry is the sharpest version of this problem I've seen — a retried read costing full price when the original was discounted makes read dedupe a cost control, not just a correctness nicety. One practical move: since an x402 call carries no session id, scope the dedupe key to the paying wallet address — that address is your session — then key on tool name plus input hash with a short TTL. For lookups that's cheap enough to run before every call. Are you planning to scope keys off the paying address, or mint a per-agent-run session id on your own side?
Not decided yet. The paying address works as a session only if each agent has its own wallet, which is what our buyer docs already recommend. If several agents share one wallet, two of them asking the same thing would collapse into one call, so a run id from the client may still be worth having.
Thanks for staying with this thread, it gave us a lot to work through.
That shared-wallet collapse is the case that would push me to the run id, even if per-agent wallets are the documented default. If two agents share a wallet and ask for the same lookup, a wallet-plus-input key returns one result for both, and the second agent cannot tell whether it was its own call or a reused one. A client-generated run id keeps those two attempts distinct without changing the payment flow. Would you scope that run id to the whole workflow run, or to each individual tool call inside it?
My take is per tool call. A workflow-level id would merge two honest identical lookups inside the same run, which is the case we started out trying to avoid.
Per tool call is the right call, and your honest-identical-lookups point is the counterexample that kills the workflow-level id for me: two agents in the same workflow can legitimately need the same lookup twice, and merging those would hide a real second call. The detail that makes per-tool-call work, though, is minting the id once per logical call and reusing it on retry, so a retry returns the stored result while a genuinely new call gets a fresh id. Would you mint that id in the agent's tool-call step, so a retry reuses it, or in the client wrapper, where a new workflow step always gets a fresh one?
In the tool-call step. The wrapper only sees transport retries, and those should reuse whatever id it was handed, but it can't tell a new logical call from the model asking again after a timeout. So the step mints the id once, tied to the tool call id the framework already assigns, and everything below just passes it through.
The case neither layer catches is the model issuing a fresh tool call for the same lookup because it lost the first answer. I don't have a clean fix for that one beyond a short dedup window on the request content.
That fresh-call case is the one I would treat as a semantic dedup problem, not a transport problem. The wrapper cannot see it, so the check has to sit one layer up, at the agent step itself: before issuing a tool call, fingerprint the tool name plus normalized arguments and compare it against recent calls in the same run. If the same lookup was already issued and answered within a short window, return the cached result instead of calling the paid API again. The failure I would guard against is staleness, so I would only apply that cache to read-only lookups and keep the window short, maybe a minute or two, so a deliberate repeat after the data changed still goes through. Would you scope that dedup to read-only tools, or would you accept the staleness risk for writes too?
Read-only only. For a write, two identical calls a minute apart can be exactly what the user wanted, like two orders for the same item, and content matching would quietly drop the second one. Writes should collapse only on an explicit key the step mints, never on what the request looks like.
For lookups we already cache on the server by tool plus sorted arguments, so part of this happens below the agent anyway. The gap is arguments that change without meaning anything, like a timestamp or a request id the agent adds on its own, which still count as a new call. A per-tool list of fields to ignore would close most of that.
I agree with that hard line on writes: two identical calls a minute apart can be two real orders, so content matching would silently eat one. The read-only ignore list is where I would add a guard, because a timestamp is not always noise. In my own tool work I had a time-window lookup where dropping the timestamp returned the wrong window and looked correct. How do you decide which fields are safe to ignore per tool, and who owns that list when a tool adds a new argument?
Nothing is ignored by default. A field goes on the list only when someone marks it on purpose, and a new argument counts as meaningful until then, so the worst case is a missed cache hit, never a wrong window like the one you hit.
The list belongs with the tool definition itself. We already fingerprint each tool's definition, so adding an argument changes the fingerprint, and that's a natural point to force a look at the ignore list in the same change. We haven't built the list yet, so this is how I'd wire it, not how it runs today.
does the fingerprint cover the whole definition or just the marked fields?
The whole definition: tool id, description and the full input schema, serialized canonically. Price is left out on purpose, so a repricing doesn't look like a new tool.
That means a new argument always moves the fingerprint, which is what you want. It also moves on a reworded description, so as a trigger for the ignore list it's noisy. The review step would compare the schema's properties, not just the hash.
leaving price out while keeping the full schema in makes sense. the reworded-description noise is the part to watch.
the 'unknown is not failed' layer is where most integrations leak money, since the instinct after a timeout is to resend from scratch. resolving by reading the chain instead of re-settling is the right call. one addition: bind the payment id to the quote itself, not just the agent session, otherwise a restart mid-negotiation can mint a fresh id for the same intent and quietly defeat layer 2. and the not-billing-failed-calls rule matters more than it looks, it keeps a retry a fair new purchase instead of a penalty for your own timeout.
Good point on the fresh id after a restart. We cover it one level up: a quote accepts only one payment. If the quote already has a payment, settled or in progress, we refuse the new attempt whatever payment id it carries, and the same buyer asking again just gets the existing order back. So a new id for the same intent can't buy the same quote twice. It would need a new quote, which shows up as a visible new purchase rather than a silent duplicate.
That's a clean shape, and the quote abstraction is doing the heavy lifting: it moves the idempotency boundary off the payment id entirely, which is exactly where restarts and new attempts break things. The residual gap I'd flag sits between quote creation and fulfillment: if the agent itself retries quote creation (network drop before it reads the response), you can end up with two open quotes for one intent. A deterministic quote key derived from the request payload plus buyer closes that, so the retry re-enters the same quote instead of minting a new one. And your point that a new quote shows up as a visible new purchase covers the buyer side, but for an agent transacting at scale, silent quote accumulation is the failure nobody notices until reconciliation day.
Fair gap. What limits the damage for us is that an unpaid quote is cheap: it expires on its own, 15 minutes by default, and the stock it holds goes back when it does. So a duplicate from a lost response costs a briefly held item, not money, since each payment pins exactly one quote. Your deterministic key is the cleaner fix though, especially for an agent buying at volume where held stock adds up. Taking that one back to the team.
15-minute expiry with stock returned on timeout is a genuinely good containment story: bounded loss per incident, and volume is what turns that from a non-issue into a line item, which is exactly where the deterministic key earns its keep. Good luck with the team discussion, and thanks for engaging with the edge cases properly rather than waving them off.
Thanks, this was a useful thread. The volume point is the one I will bring to the team, since that is where a fine-for-now design turns into a cost.
This 4-layer design is solid distributed-systems hygiene for deterministic clients. However, once the client is an LLM agent, we face a different failure mode: adaptive evasion.
Unlike traditional software, an LLM optimizing for a goal will treat a rate limit, conflict, or circuit breaker not as a hard stop, but as an obstacle to route around. If it learns where the threshold is, it can slightly mutate its parameters to generate fresh signatures, or spawn/hand over the task to a sub-agent before hitting the cap—effectively laundering the retry state across sessions. We've already seen similar failure modes in the wild (like the SOL wallet drain incidents) where agents operate in the blind spots of local rules.
To actually contain an agent with a wallet, transport-level idempotency isn't enough. You need a human-readable inspection ledger that tracks the lineage of intent and cumulative cost across agent handovers, gated by human authority when behavior drifts.
How do you handle cases where an agent adapts its requests or delegates to another agent to bypass per-call or per-session limits?
We don't, at least not at the layer the post covers. Idempotency only stops paying twice for the same thing. A mutated request is a new purchase, and we charge it like one. Per-session limits on our side would be easy to dodge for the reason you give, since the server sees paying wallets, not agents or sessions.
The boundary a sub-agent can't route around is the wallet it pays from. If every agent in the chain spends from a key the human funded with a fixed amount, handing the task over doesn't create new money. A server-side spend cap per paying wallet is on our open list, and we haven't built it yet.
Thank you for the honest and clear reply. Using a pre-funded wallet with a hard cap as the ultimate boundary makes complete sense from the gateway side.
On our end, we are trying to combine wallet limits with purpose-bound attempt guards on the orchestration side—tracking intent IDs and locking attempt budgets across workers. Even so, we still haven't fully escaped the retry swamp. The moment we lock down one path, edge cases like alias relabeling, parent-to-child worker delegation, and unsettled states after crashes pop right back up.
Containing an adaptive agent without causing approval fatigue is genuinely tough. Really appreciate you sharing your architecture so openly—looking forward to seeing how your wallet-level spend caps evolve!
The unsettled-after-crash case is the one we hit too. A first attempt dies mid-flight, and the payment stays marked in flight until a timeout clears it. How do you resolve those on your side? Do you wait for the chain, or let another worker reclaim the intent after a deadline?
Yes, we’ve run into very similar problems on our side. It’s a genuinely difficult issue, and I’m not sure there’s a single correct answer.
We don't operate on-chain. Instead, we use explicit handoff records between workers and try to recover with the smallest possible rollback.
We've recorded cases where a worker couldn't start a task, or stopped responding halfway through execution.
Our current approach is to let the deadline expire, contain the failed attempt, and have a supervising worker either take over the same intent or reassign it to another worker under its authority.
The important distinction is that we're transferring responsibility for the work, not blindly repeating the operation. We preserve the original intent and execution records so the next worker can determine what was actually completed before continuing.
In short, we cut our losses on the stalled attempt, not on the intent itself.
Of course, on-chain settlement introduces another layer of uncertainty that our approach hasn't been tested against.
But yes, I absolutely understand why you're struggling with this. We're still working through the edge cases ourselves. There really isn't an easy universal solution, is there?
That split helps: drop the attempt, keep the intent. On-chain adds one step before the handover. The next worker would have to read the authorization nonce first, because the stalled attempt may have paid after all. If it did, the intent is already done and there's nothing to reassign. Thanks for walking through it.
That’s a fantastic point about the stalled attempt having actually gone through—honestly, that was an oversight on my end. Checking the authorization nonce right before handover is a remarkably clean way to bridge that gap. Thank you for pointing that out.
Really looking forward to seeing what you build with this architecture!
The "unknown is not failed" layer deserves to be first-class in every agent runtime, not just the gateway. In our own MCP tooling we found the worst double-spends didn't come from the payment layer at all, but from the agent's own orchestration: a retry that fires before the gateway's 409 payment_in_flight arrives gets queued as a brand-new attempt with a brand-new payment id, so the deduplication logic never sees it.
One operational fix that helped us was moving the retry budget into the step scheduler: a pending call holds its slot until the gateway responds or the unknown-state probe resolves, instead of letting the timeout fire an uncoordinated retry. On the 409 payment_identifier_conflict point, I'd also log and alert on conflicts rather than treating them as routine responses, because a mismatched amount under a reused id is exactly how a misconfigured tool call or a quote-drift bug announces itself.
For the 24-hour window, have you considered decaying storage rather than a hard cutoff — keeping a compact hash of id to outcome hash so a day-25 retry can still be answered cheaply without storing full answers?
Holding the slot in the scheduler is a cleaner fix than anything a gateway can do from its side. The gateway only sees the second request after the agent already made a new id.
On your last question, part of that already exists. Each paid call gets a public receipt with the transaction and a SHA-256 of the response, and receipts do not expire. What we lack is the link from a retry id to that receipt once the 24-hour record is gone, so a day-25 retry still can't be answered from it. A compact id-to-receipt map is a good way to close that. Alerting on conflicts is a fair point too. Taking both back to the team.
Good — if the receipts themselves don't expire, the compact id-to-receipt map might not need to be a second system at all. What if the receipt store is the map: each receipt entry carries the retry ids that resolved to it, so a day-25 retry answers from the receipt itself rather than from a record that expired at 24 hours? One question on the conflict alerting too: would you alert on any duplicate payment id, or only when the same id arrives with a different amount or recipient? The first is noise at retry scale; the second is a true conflict worth waking someone up for.
Only the second. A retry with the same request is normal and gets the stored answer back, and a changed request already comes back as its own error code, so that code is the natural thing to alert on.
Putting the retry ids on the receipt is neat. One catch is that our receipts are public, so the ids would become public too, which matters if a client puts anything meaningful in them. I'll add that to the note.
All four layers are server-side, and they share one silent assumption: the client can present the same payment id after a crash. That makes the agent-side mirror discipline the real first layer - the agent must durably record the intent (id, quote hash, recipient) before sending, not after the response arrives. A runtime restart between sign and send, with the id only in memory, turns your layer 2 into a coin flip: new id, new signature, and a double charge every server rule correctly allowed.
The "same id, different payload returns a conflict, not the old answer" rule deserves the emphasis you gave it. Replaying a stale result hides the client bug at exactly the layer built to catch retry bugs. Failing loud is the feature.
And "unknown is not failed" generalizes past payments: any mutating call whose outcome you cannot observe should be resolved by reading state, never by repeating the mutation to find out.
Agreed, and it's the one layer the server can't enforce, only document. Our docs explain what the server does with a repeated id, but they don't tell the client to write it down before sending, which is the gap you're pointing at.
One detail helps on x402 specifically. If the agent stores the signed authorization itself before sending, and not only the id, a resend after a restart can't charge twice, because the transfer nonce inside it is single-use on chain. Re-signing after the crash is what opens the door. Taking the write-before-send rule back for the client docs.
Exactly - and the two artifacts cover different guarantees, which is worth one sentence in the client docs. The stored signed authorization bounds the money: a replay hits the single-use nonce and fails on chain. The stored payment id recovers the work: it is what lets layer 2 hand back the original result instead of leaving the agent paid-but-answerless. Keep both, written before send, and a restart degrades to "repeat and continue" instead of either failure mode.
That split is clearer than how I put it, and it's the form I'd use in the client docs. One limit belongs next to it: we keep the payment id for 24 hours, so repeat and continue only works inside that window. An agent that comes back after a longer outage should look up the receipt instead of resending.
@whiteknightonhorse Yes, that's the part that keeps retries cheap. Temporal works this way for workflows: a finished activity's result is stored in the history, and replay reads it back instead of calling the model again.
Good example, thanks. That pairs well with a gateway that stores results too. If the activity crashes after the tool paid but before Temporal records it, the retry with the same key gets the stored answer back instead of a second charge.
Extending the synthetic-key thread, since that is the direction the plain per-call tools are heading: deriving the key server-side from (tool, args) has a real trap in volatile fields. Pagination cursors, timestamps, and request ids embedded in args will make every retry hash differently and quietly defeat the key. The fix is a canonicalization step before hashing — declare per-tool which fields are volatile and strip them. The conflict rule from layer 2 then doubles as a tripwire: when the same synthetic key arrives with different canonicalized args, that is a signal the canonicalization missed something, discovered for free instead of via a chargeback.
One more extension for the "unknown is not failed" layer: key the reconcile job by the idempotency key too, not only by payment authorization. The background read keyed on payer + nonce clears the paid call, but for plain per-call tools the synthetic key is the only thing tying a retry to the original attempt. If the reconcile job cannot address that state by key, a crashed first attempt leaves payment_in_flight stuck until the 24-hour window expires, and every retry in between burns agent time. Keying reconcile by the same key closes the loop and keeps the 409s honest.
You're right about the crash case, I checked it. If the first request throws, we clear the id at once. If the process dies outright, the id stays in flight until its 24 hours run out, and the reconcile job settles the order by payer and nonce without touching that row. So a retry could keep getting a 409 for an order that is already paid. Taking that and the volatile-fields point to the team.
the 24 hour retention is the seam I would look at next. the chain keeps the transfer final forever, but the stored answer expires, so an agent that lost its runtime state for two days comes back in a state your four layers do not describe: paid, provable, and holding no result, while the order was already marked failed for good at the 24 hour mark.
that is the one case a retry cannot repair, because there is nothing left to retry against. two things would close it. keep the stored result at least as long as the settlement can still be challenged, or persist a digest of the delivered answer next to the settlement so a late agent can prove which answer it bought and ask for a claim path. a late confirmation that flips a failed order back to paid also needs a stock story, since the reservation was released when the timer fired.
That is the sharpest gap anyone has raised here. Half of your second fix already exists for paid tool calls: each one gets a public receipt with the payment transaction, payer, amount and a SHA-256 of the delivered response, and it doesn't expire. So a late agent can prove which answer it paid for. What we don't have yet is the claim path on top of it, and the order case you describe, where a confirmation lands after we marked the order failed and released the stock, needs its own rule. I'm taking both back to the team.
The 409 payment_in_flight case raises a client-contract question: does that response include a Retry-After hint or a way to poll the same payment id until the stored result is ready? Many agent runtimes treat any 409 as terminal and will mint a fresh id on the next step, which would bypass Layer 2 entirely. Making the in-flight state explicitly retryable with the same id, rather than just conflicting, would close that loop and also prevent a burst of duplicate attempts while the first request is still settling.
Fair catch. Today the in-flight 409 has no Retry-After header and no field for it, only a message that says to retry shortly. Sending the same id again later does work, and once the first request finishes it returns the stored answer.
If a runtime gives up and mints a new id, the order still accepts only one payment, so a second attempt on a quote that is already being paid gets a 402 and isn't charged. The agent still ends up holding a refusal instead of its result, so your point stands. Taking that one back to the team.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.