We pay our suppliers through a payment provider's payouts API. Every payout request carries an idempotency key built from our payout id, so that if we send the same request twice the provider recognises it and returns the original result instead of paying again.
On a Friday evening in May the provider had a partial outage. For about forty minutes their API accepted payout requests, executed them, and failed to send us a response, so our client timed out after thirty seconds. We retried three times, timed out three times, and moved the requests to a dead letter queue for a person to look at, as designed.
A person looked on Monday morning. The provider's status page said the incident was resolved, so she replayed the queue, three hundred and eighty payouts, with the same keys. Every one was accepted and paid. Two hundred and twenty nine of them had already been paid on Friday.
The provider's documentation says it keeps idempotency keys for twenty four hours. After that a key is just a string it has never seen. Our replay came about sixty hours later. We had read that sentence when we built the integration and treated it as a detail, because our automatic retries all happen within minutes. The dead letter queue had no such limit, and nobody replaying it knew the keys had an expiry date.
The duplicates came to a hundred and ninety thousand pounds. Three weeks of emails to suppliers recovered all but about four thousand.
A request that timed out is now treated as unknown, not failed. Before anything sends it again, we look the payout up through the provider's API by our own reference, which every payout carries in its metadata, and resend only if the provider has no record of it. Every message in the dead letter queue shows its age against the twenty four hour window, and the replay tool refuses anything older than twenty hours that has not passed that lookup. And a daily reconciliation compares the provider's payout list with ours and flags any supplier paid twice for one invoice.
Idempotency is a promise with an expiry date, set by the side that keeps the keys. We had put a queue whose whole purpose was waiting in front of a promise that did not wait.
– Sergey Shinder
Top comments (1)
ouch. I'd rather the provider reject an expired key than treat it as brand new. Has any payments API you've used done that? From my experience building Voiden, the replay request belongs in the repo next to the normal one, with the 24-hour limit written right in it.