An AI agent doesn't fill out a checkout form. It hits an endpoint, gets a 402 Payment Required challenge, signs an authorization, and expects a real resource back within seconds. We tested that exact path against our own public endpoint — not a sandbox, real USDC on Base mainnet — and found a real production bug in the process.
What we ran: a batch of GPU jobs through our public x402 endpoint, the same one an external agent would hit. Different GPU classes, different providers (Vast.ai and RunPod), each paid for and torn down automatically.
Results — 7 settled payments, every one with an on-chain hash:
GPU Provider Real cost Charged Time Tx hash
RTX A4000 Vast.ai $0.001215 $0.01 80.1s 0x1aacdc35...05072b
RTX 4090 Vast.ai ~$0.0023 $0.01 ~100s 0xd190f93e...baa5f4
RTX A4000 Vast.ai $0.001215 $0.01 106.1s 0x76318a7e...4630de
RTX 3090 Vast.ai $0.001852 $0.01 109.4s 0x2875dcf1...7b9b8b
A40 RunPod $0.005667 $0.01 63.1s 0x8aea8983...965902
H100 Vast.ai $0.028959 $0.05 109.9s 0x1e595040...65f05ca
A10 (batch inf.) RunPod $0.005667 $0.01 110.7s 0xcfbb0dda...d9514
What broke: 2 of 8 jobs failed on a 120-second payment authorization window. Root cause wasn't price lookup or payment logic — it was our OFAC sanctions-list cache going stale, triggering a live 5MB Treasury re-download on every payment with no timeout. When that download hung, it added up to two minutes inside the payment path.
What we fixed: removed the live download from the payment path entirely, moved refresh to an hourly background job, widened the authorization window from 120s to 600s as a second layer. Re-ran the exact failing config: settled in 18.8 seconds. 0xa7c7ae02e2c4382b748c6736616f5570d7fd72520cb962b4b41434f0fd6f859a.
Independent verification: pulled Vast.ai's own billing ledger directly — every job shows up by contract ID, charge amount, and our own dispatcher's job label embedded in their record. That check also surfaced a second bug: RunPod silently substituted RTX 4090s for two jobs requested as an A40 and an A10, with no disclosure. Currently fixing that one too.
A payment rail is only as good as what happens when it breaks. This is what happened when ours did.
Top comments (6)
I treat payment authorization windows as a dependency budget, not a retry setting: a cold sanctions-list refresh inside the request path can turn a healthy 120-second timeout into a guaranteed failure. Moving that work to a bounded background refresh is the real fix; widening the window alone only hides the tail.
Good catch, and yeah, that's exactly where we landed. The sanctions list check was originally happening inline, which is what turned an occasional cold refresh into a guaranteed timeout under load. We moved it to a bounded background refresh that keeps a warm cache the request path just reads from, so the auth window isn't gated on an external lookup anymore. Widening the timeout was the band aid we tried first, this was the actual fix. Appreciate you calling it out explicitly, it's a distinction worth being precise about.
Paying per call flips the failure model, a retry loop stops being a latency problem and becomes a spend problem, since every failed attempt can cost real money
Curious how you guard against a bug that keeps retrying paid calls, is there a cap per agent or per hour
Good catch, and honestly, no, there's no retry cap or spend ceiling in place yet. We hardened the payment path itself first (the fix in this post), guardrails against runaway retries are the next layer. The plan is a max-retry count per agent plus a hard spend ceiling per agent per time window, so a stuck loop fails safe instead of draining a wallet. Appreciate you flagging it, will post an update here once it's live.
Genuinely good catch, kilawattcloud, this is the kind of bug that only shows up once real money is on the path. The part I'd want to know more about: now that the OFAC refresh is a background hourly job instead of inline, what happens on a cold start or if that background job itself stalls? Does the payment path serve a last-known-good list, or does it fail closed and just not settle? For an ordinary cache serve-stale is the right call, but a sanctions check failing open on a timeout is a different risk entirely, and it's the one direction I wouldn't want the "fix latency" instinct to push toward even by accident.
Genuinely good question, and it turned out to be a real gap, not a hypothetical one.
Cold start is fine, the list lives in the database, not app memory, so it's already there on a fresh start. Confirmed that directly by simulating a true empty list, it blocks with "sanctions list not loaded."
The stalled-refresh case was the real find. The tolerance was 7 days, meaning a silently dead refresh job could keep serving a week-old list and letting payments through. I backdated the cache to 3 days old and ran a live screening check to prove it: it returned "allowed." Exactly the fail-open risk you flagged.
Fixed now. Tightened the ceiling to 48 hours, roughly one full day of consecutive missed refreshes before it stops trusting the list and blocks everything. Re-ran the identical 3-day-old test after the fix, now correctly blocked. Also re-confirmed a real OFAC-listed address still gets caught and a clean one still passes, so detection itself didn't move, just the failure mode around it.
Added to our public incident history with the real before/after. Appreciate you pushing on this, it's a better system because of it.