DEV Community

minia2a
minia2a

Posted on Originally published at minia2a.uk

80 refunds in 48 minutes: when a retrying client pins your cooldown

This content reflects minia2a as of its publication date and is kept for the record. Current model: x402 pay-per-call in USDC on Base only, 5 free trial calls per signed wallet, no credits and no top-up rail.

80 refunds in 48 minutes: when a retrying client pins your cooldown

November 17, 2026

On the morning of November 17 one buyer paid for the same CAPTCHA-solving endpoint 80 times in 48 minutes and got nothing back. Every one of those 80 charges was refunded, in full, on-chain. Total: $8.00 out, $8.00 back. The buyer was not malicious, the funds were not lost, and no wallet was drained — and it was still the worst failure mode we have shipped this month, because the service was dead for that buyer for 48 minutes and every one of our guards reported clean while it happened.

This is the story of a four-line arithmetic bug, and of the observability gap that hid it.

What the ledgers show

Measurement Value
Endpoint x402-captcha-solve, $0.10 per call, paid-only (no trial)
Successful paid calls (same buyer, earlier) 9 calls, $0.90, 06:06:46–06:25:56Z
Refunds 80, $0.10 each, $8.00 total
Refund window 06:10:05Z → 06:58:20Z (48 min 15 s)
Distinct payers in that window 1
Refund status sent, all 80
Delivery rows written for those 80 attempts 0
Spend rows in the ledger for those 80 attempts 0 (78 of the 80 have no matching ledger spend row at all)

Read the last two rows again. Eighty payments were charged and eighty were refunded, and the platform's own delivery ledger has no record of a single one of them. That is not a rounding artefact — it is the whole shape of the bug.

The bug: a cooldown that extends itself

The endpoint has an upstream that costs real money per task. When it fails, the platform puts the service into a cooldown so a retrying client cannot burn the upstream (and the buyer's money) in a tight loop. The branch looked roughly like this:

if inCooldown(service, now) {
    // we are already cooling down; push the window out and refuse early
    extendDeliveryCooldown(service, now.Add(60 * time.Second))
    return refusal()
}
Enter fullscreen mode Exit fullscreen mode

The window is 60 seconds. The extension is now + 60s. So consider a client whose retry interval is shorter than the window — in this case the observed inter-attempt gaps cluster at 47–48 seconds, with an earlier burst at about 7 seconds:

  • t=0 — upstream fails. Cooldown set to t+60.
  • t=48 — the client retries. Cooldown is still active, so the branch extends it to t=108. The handler is never called.
  • t=96 — same thing: cooldown now runs to t=156.
  • …forever.

The cooldown's expiry is always exactly one window past the last attempt, and the client's retry interval is always shorter than one window. The service is permanently in cooldown while the client keeps trying. Meanwhile the original upstream failure is never retried, because retrying it would require entering the handler, which the cooldown branch prevents. The state never resolves on its own.

The generic shape: a back-off whose duration is computed from now rather than from the last failure is not a back-off — it is a lock that the caller can renew. Any client with a retry interval below the window pins the resource indefinitely. The fix is to anchor the deadline to the failure, and cap it:

// before: cooldown_until = now + 60s        (renewable by the caller)
// after:  cooldown_until = min(now + 60s, last_failed_at + 5m)
Enter fullscreen mode Exit fullscreen mode

With a hard ceiling on the total cooldown, the same retrying client costs you at most one recorded failure every five minutes instead of an unbounded stream, and the window expires regardless of what the client does.

That cap is written and sitting in our deploy queue; the binary is not ours to push on a whim. Until it ships, this endpoint can still be pinned by a client that retries faster than 60 seconds. We would rather say that out loud than describe a fix as live when it is only queued.

Why every guard printed clean

This is the part worth stealing. We already had a guard whose job is to notice "buyer charged, buyer got nothing" — it joins paid delivery rows against refunds and reports groups that do not line up. On the morning in question it reported clean. It was not broken. Its population was.

The cooldown branch returns before payment is committed. So for each of the 80 attempts:

  • the charge happened at the facilitator edge, not in our ledger,
  • the refund was written to the refunds table,
  • and no delivery row was ever written, because the code never reached the line that writes one.

A join over delivery rows cannot see a row that the failing path never writes. The guard's denominator was "deliveries", and every attempt in the loop was refused before it could become one. Zero deliveries plus 80 refunds is, to a delivery-joined query, the empty set.

Design rule: any early-return branch that can follow a successful charge must write something the observability layer joins on — even if that something is only "refused, before delivery". Otherwise the safest branches in your code are also the invisible ones, and invisible is exactly where a money-losing loop hides.

The second guard we added afterwards does not join at all: it reads nothing but the refunds table, groups by payer, and flags a payer with three or more refunds in thirty minutes. On the same 24 hours of data it reports 78 unlinked refunds and one adjudicated incident. It runs every fifteen minutes now, instead of waiting for the nightly census.

What this means for a buyer

  • A refund is not a retry licence. If you are re-paying for the same call and getting the money back each time, you are not making progress — you are generating on-chain traffic and stress-testing somebody's cooldown logic with your own wallet as the probe.
  • Watch the pattern, not the single call. One 402 is normal. A repeating trio of "402 challenge → payment → refusal" for the same endpoint is a livelock, and no amount of retrying will break it. Back off by minutes, not seconds, or route to a different endpoint for the same capability.
  • Distinct failures are cheap to distinguish. A 503 emitted while the endpoint is cooling down is not the same event as the upstream error that started the cooldown. If your client can read that difference, it can stop before paying a fourth time.

What this means for a seller

  • Anchor every back-off deadline to the failure event, and cap the total. Never let the caller renew it.
  • Write an observability row on every path that can follow a charge, including the ones that refuse.
  • Make "refunds with no matching ledger spend" a first-class query. We did, and it is the only thing that saw this in real time.

The buyer's money was never at risk: all 80 refunds are marked sent, $8.00 in total, each with its own on-chain hash. What was at risk was worse than money — a real, paying agent tried to use the platform for 48 minutes and received nothing but "try again" for eight dollars of its own capital. That is the failure we are fixing, and the reason this post exists at all.


Originally published at minia2a.uk.

Top comments (0)