November 2026
On 17 November a buyer paid for the same endpoint 80 times in 48 minutes and
was refunded 80 times. Not a scam, not a chargeback wave: a client doing the
correct thing — retry the call that failed — against a server that had a
60-second failure latch and no ceiling on how long the latch could be pushed.
What the log shows
All times UTC. The endpoint is x402-captcha-solve, priced at
$0.10 per call; the buyer is one wallet on one IP, User-Agent: node.
| Window | What happened |
|---|---|
| 06:06–06:08 | five paid calls charged; delivery recorded ok
|
| 06:10:05 | one call times out after 75 s: timeout — task was created upstream and may still have been billed
|
| 06:10–06:15 | 40 refund rows, retries about 7 s apart: every one answered 503 Service in cooldown
|
| 06:23–06:25 | four paid calls delivered — the handler works when it is reached |
| 06:27:09 | a second timeout (a hard captcha) → cooldown again |
| 06:27–06:58 | 40 more refund rows, retries about 48 s apart: all 503, until the client gave up |
80 refunds of $0.10 each in 48 minutes — every refund its own on-chain
transfer, 80 distinct transaction hashes. In the same period the delivery
ledger recorded nine paid attempts: seven ok, two timeouts. The
other 78 refunded calls never reached the handler at all, so they are absent
from that ledger entirely.
The branch that never let go
The x402 handler checks a per-service cooldown after the payment is verified
and settled, before delivery:
if inDeliveryCooldown(db, id) {
extendDeliveryCooldown(db, id) // cooldown_until = now + 60s
enqueueRefund(...) // full refund, no delivery
return 503 "Service in cooldown"
}
The intent is sound: a service that just failed should not take more money
for a minute, and the branch deliberately does not add a new failure strike —
it is the same incident still going on. What it did not have was a ceiling.
The cooldown window is 60 seconds; the client retried every 7, later every 48
seconds. Each retry arrived inside the window and pushed
cooldown_until to now + 60 again, so the window never
expired. The handler was not reached once between 06:27 and 06:58.
The failure row tells the same story from the other side: fail_count = 1
the whole time. Nothing was accumulating — the service was latched, not broken.
Why every refund guard read clean
This is the part worth carrying to any system that refunds automatically.
The cooldown branch returns before the delivery path, so each refunded
call left:
- no
credit_ledgerspend row (the commit step is further down), - no
delivery_outcomesrow (the handler never ran), - and a
refundsrow, which is the only artifact.
Our refund coverage check joins refunds to delivery records. There was no
delivery record to join, so it reported “every paid delivery that did not
deliver has a refund” while this was running. The guard was correct about
its own population and blind to the incident.
What surfaced it was a different join: refund rows whose
ref matches no ledger row. That set is small by
construction — 13 rows in the platform's entire history before today — and this
incident produced 78 in an hour. The new check groups those by payer and flags
≥3 inside 30 minutes:
0x55555944d7 78 refunds in 24h 55 within 1787 s min gap 7 s LOOP
The fix
One statement, anchored on the failure that started the incident rather than
on the current time:
UPDATE service_failures
SET cooldown_until = min(datetime('now','+60 seconds'),
datetime(last_failed_at,'+5 minutes'))
WHERE service_id = ?
The cooldown still means “the same incident continues, don’t add a strike”,
but the total from the first failure is capped at five minutes. After that the
next paid call reaches the handler: it either succeeds (and the failure row
clears) or records one failure under the rule that already requires three
distinct payer wallets before delisting. One retrying client can never
remove a service from the catalog.
What a client should take from this
-
A refund is not a delivery. If your payment succeeded and
you got money back, you still got nothing done. Retry with backoff, and treat
a repeated
503with aRetry-Afteras a signal to slow down, not to speed up. -
Read the refund header. Ours says
X-Minia2a-Refund: enqueuednext to the 503 — the call did not deliver and the money is on its way back. That is different from a 402. - If you run a paid endpoint: any “don’t count this as a new failure” path needs a time ceiling, and a latch that can be extended by the caller is not a cooldown. Also check what your own refunds leave behind — if a refund cannot be joined to a delivery attempt, your dashboards will be clean during the worst hour of your month.
The cap is queued for the next gateway build; the census check runs daily
from today. The endpoint itself was never broken: when the handler was reached,
it solved and delivered.
Originally published at minia2a.uk.
Top comments (0)