We found out about it from a support ticket, not a dashboard. A customer had been billed $340 twice for the same order, four seconds apart, and wanted to know if we'd started double-charging people. We hadn't — or rather, we hadn't meant to. What we'd actually built was a payment flow that assumed retries were free, and that assumption cost us a very uncomfortable afternoon of pulling logs and a much longer week rebuilding the write path to be safe by default.
The setup was ordinary. Our checkout service called out to a payment processor, waited for a response, and marked the order paid. Like any network call, it could time out — and like any reasonable engineer, whoever wrote the original client had added a retry: if the request didn't come back in 5 seconds, try again. That's normal, defensible advice for GET requests and most reads. The problem is that "charge this card $340" is not a read, and a timeout doesn't mean the request failed. It just means we stopped waiting for the answer. The first call had actually gone through on the processor's side; the client's retry was a second, fully independent charge. Same card, same amount, no error on either end — just two successful transactions where we intended one.
The uncomfortable part of debugging it was realizing how much of our system quietly relied on "retry it and see" as a correctness strategy. Webhook handlers retried. Background jobs retried. The checkout client retried. All of that is fine for idempotent operations — re-running a read, or an update that sets a field to the same value, doesn't change the outcome. It's only unsafe for operations with side effects that aren't naturally idempotent, and a payment charge is the sharpest possible example of that category. We had been treating "make it retry-safe" as a network-layer concern instead of an operation-design concern, and it had worked fine right up until it didn't.
The fix was idempotency keys, and the part that took longest wasn't the concept — it's well documented, most payment processors support it natively — it was finding every place in our own code that needed one and didn't have it. The pattern: every write operation that could plausibly be retried gets a client-generated idempotency key, a UUID created once at the start of the user action and reused across every retry of that same action. On our side, before executing the charge, we check a dedupe table keyed on that UUID. If we've seen it before, we return the stored result of the original request instead of re-executing anything. If we haven't, we execute it and store the result, with a TTL long enough to cover realistic retry windows — we settled on 24 hours after finding a few edge cases where mobile clients retried failed requests on next app launch, sometimes the following day.
We didn't stop at payments. Once we went looking, we found three other endpoints — refund issuance, subscription upgrades, and a bulk-invite endpoint — that had the identical shape: a mutating call, a client that retried on timeout, and no dedication mechanism to tell a legitimate retry from an accidental duplicate. Refund issuance was the scariest one to find, because unlike a duplicate charge, a duplicate refund doesn't generate a customer complaint — it just quietly loses the company money until finance notices the numbers don't reconcile.
Since shipping idempotency keys across those four endpoints, duplicate-charge and duplicate-refund tickets have gone from a recurring monthly occurrence to zero. More usefully, we stopped having the argument about whether a given endpoint "needs" one — the checklist for any new mutating endpoint now includes idempotency key support as a default, the same way auth and input validation are defaults, not something you bolt on after the first incident report.
The broader lesson wasn't really about payments. It was that retry logic and idempotency are a matched pair — you can't safely add one without the other, and code that retries a non-idempotent write is a bug waiting for the right timeout to trigger it. We write about the operational lessons like this one as we run into them building Edilec's backend — if you want more of the same, edilec.com has the rest.
Top comments (0)