Why payment retries double-charge people, and where the idempotency key has to come from.
Every payment integration starts the same way. Your service calls the provider, the provider charges the card, you get a response, you mark the order paid. Four hops, and every architecture diagram on the internet draws it exactly like that.
Then one day a customer emails you a screenshot of two identical charges, thirty seconds apart, and you go looking for the bug.
There isn't one. Not in the sense of a line you can fix. What happened is that your service asked a question it couldn't get an answer to, and then guessed.
**
The failure that isn't a failure
**
Your service sends the charge request. The provider receives it, authorizes it, and moves money. Then the response comes back through a load balancer that resets the connection, or the gateway returns a 504, or your pod gets rescheduled mid-flight.
From your side, all of these look identical: an error. No response body, no transaction id, nothing to record.
The critical thing is that an error here does not mean the charge failed. It means you don't know. The card may have been charged. It may not have been. Those two states are indistinguishable from where your code is standing, and any logic that treats a timeout as "it didn't happen" is going to be wrong a predictable percentage of the time.
At low volume you never notice. At scale, "a predictable percentage" is a support queue.
**
Why the retry makes it worse
**
The obvious response to a failed call is to retry it. Most HTTP clients will do this for you, which is part of the problem — the retry often happens somewhere you forgot to look, in a library default or a service mesh policy.
So a second request goes out. The provider receives it, sees a well-formed charge request with a new request id, and does exactly what it was told to do. It charges the card again.
This is not the provider being careless. It's the provider being correct. Two separate requests to charge the same amount are, absent any other information, two separate intents to charge. A payment API that quietly swallowed the second one would be a much more dangerous thing to build on.
**
The misconception worth naming
**
The common assumption is that the provider handles this. It's a payment company, deduplication must be their job.
They do deduplicate, but only their own internal retries. If their gateway retries an authorization against the card network, that's handled inside their boundary and you never see it.
Your retry crosses that boundary. It arrives as a new request from a client that has, as far as they can tell, decided to charge the card again. Nothing in the request says otherwise.
Unless you put something in it that does.
**
The idempotency key
**
The mechanism is a key you attach to the request that identifies the operation, not the attempt. When the provider sees a key it has already processed, it returns the original result instead of performing the charge again.
Most major providers support this. The mechanism is rarely the hard part. Where implementations go wrong is in where the key comes from and how long it lives.
Generate it yourself. Not from the provider, not from a proxy or gateway. If something between your service and the provider is minting the key, it will mint a new one on the retry, which defeats the entire point.
Create it when the order is created, not when the charge is attempted. This is the one that gets missed. If the key is generated at the top of the function that calls the provider, then a retry that re-enters that function generates a fresh key, and the provider sees a brand new operation. The key must be older than the first attempt.
Keep it stable for the lifetime of the order. Same order, same key, on attempt one and attempt nine, today and after a redeploy.
Store it with the order, in the database. A key that lives in a variable dies with the process. A key that lives in a cache dies with the eviction policy. Both failure modes produce exactly the bug you were trying to prevent, and they do it in production under load, which is precisely when the retries are happening.
Deriving the key from the order id is usually enough. It's stable, it's unique, and it survives everything.
**
When you can't add a key
**
Some older or regional providers don't support idempotency keys at all. The fallback is to query before you retry: search the provider for a transaction matching this order, and only charge if nothing comes back.
This is weaker than a key. There's a window between the query and the charge where the first transaction can land, and the search is only as good as the reference you attached to the original request. Treat it as harm reduction, not a solution.
**
The ledger is the source of truth
**
An idempotency key stops the duplicate at write time. It doesn't tell you what actually happened.
For that you need your own ledger — an append-only record of every payment event your system observed — reconciled daily against the settlement file the provider sends. The settlement file is what actually moved. Your ledger is what you think moved. Reconciliation is the process of finding out where those two disagree, and there is always somewhere.
A key prevents. Reconciliation proves. Systems that handle money seriously run both, because the first one only works when your code executes as intended, and the entire premise of this article is that sometimes it doesn't.
**
The short version
**
A timeout tells you nothing about whether the charge succeeded. Retrying without an idempotency key turns that uncertainty into a duplicate charge. Generate the key on your side, at order creation, store it in the database, and send it with every attempt. Then reconcile daily against settlement, because the key only covers the path where your code ran.
If you've shipped payments, you've met this bug. If you haven't yet, now you'll recognize it.
Top comments (0)