A customer was charged twice for one order. We checked the logs and found our payment handler had run twice, three seconds apart, for the same order, and both runs completed successfully. There was no bug in the handler. It did exactly what it was told, twice, because the provider sent the webhook twice.
Their documentation said it plainly, in a paragraph I'd read and not absorbed: delivery is at-least-once, and duplicates are possible on network timeouts. What happened was ordinary. Our endpoint took slightly too long to respond, their side timed out waiting for our 200, marked the delivery as failed, and redelivered. We had in fact processed it the first time. We just hadn't answered fast enough to say so.
That reframed how I think about every integration boundary. At-least-once delivery isn't an edge case of message-based systems, it's the normal guarantee, and it means the receiver is responsible for idempotency. Anyone who assumes exactly-once is relying on a property the network cannot provide. If processing the same event twice does damage, the damage is yours to prevent, not the sender's.
The fix had two halves. First, make the handler idempotent: take the provider's event ID, insert it into a processed-events table with a unique constraint inside the same transaction as the side effect, and let a duplicate key error be the signal that we've already done this work. Not a check-then-act — that's a race that fails under exactly the concurrent redelivery you're defending against. One atomic write that either commits both the effect and the record, or neither.
Second, respond fast. We stopped doing the work inline. The endpoint now validates the signature, writes the event to a queue, and returns 200 in a few milliseconds. Processing happens asynchronously. That alone eliminated the timeout-driven duplicates, and it means a slow database no longer causes a redelivery storm from someone else's retry policy. We then audited every other webhook and queue consumer we had, and found three more that could double-apply, one of them on an email send. Duplicates hadn't happened there yet, which is not the same as being safe.
Assume every message will arrive twice. Design so the second one is boring.
– Sergey Shinder
Top comments (0)