Every webhook arrives twice eventually, so design for it from day one
A unique dedupe key plus a processed-events table kills double-acts
Read current state before acting so a retry is a no-op
Return 2xx fast, do the slow work after you confirm it hasn't run
Every payment provider I've wired up delivers the same webhook at least twice. Not often, but often enough that one duplicate charge is one too many. So I stopped treating retries as an edge case and started assuming they happen on every single event.
The fix isn't clever. It's a dedupe key, a readback, and a couple of state checks.
Here's exactly how I lay them out.
Why Every Webhook Gets Delivered Twice
Webhook senders retry because they can't tell the difference between "your server never got it" and "your server got it but the response got lost on the way back." From the sender's side, those look identical: no 2xx inside the timeout window.
So they send again. And again, on a backoff schedule, sometimes for 24 hours.
That means your endpoint will see the same checkout.completed event twice even when your code worked perfectly the first time. The payload is byte-for-byte identical. The only thing that changed is that the sender got nervous.
There's a second cause that's easier to forget.
Your own server can crash halfway through handling an event. You charged the card, then the process died before you wrote "done" to the database. The sender retries, and now you're about to charge again because nothing in your system remembers the first attempt finished.
A third source is pure infrastructure. A load balancer times out at 10 seconds, your handler takes 11, the balancer returns a 504 to the sender, and the sender retries even though your handler eventually succeeded. I've seen this one bite people who do heavy work inline before responding.
The point is you can't engineer the duplicates away. You can only make the second delivery harmless.
I think of it like the recursion limit problem: you don't argue with the runtime, you design around the hard edge. If you want the deeper version of that mindset, I wrote about it in I Found the Exact Recursion Depth That Crashes Node.
Idempotency means the operation gives the same result whether you run it once or ten times. A retry that lands on an already-processed event should do nothing new and still return a happy 2xx so the sender stops. That's the whole goal. Everything below is just mechanics to get there without race conditions eating you alive.
The Dedupe Key and the Processed-Events Table
Every well-behaved webhook sender puts a unique ID on each event. Stripe calls it event.id, others call it a delivery ID. That ID stays the same across retries of the same event. That's your dedupe key, and it's the foundation of the whole thing.
I keep a single table called processed_events. One column holds the event ID as the primary key. One column holds a status, one holds a timestamp, and that's basically it.
When an event arrives, step one is: try to insert its ID into that table. Don't check-then-insert, that's a race. Insert and let the database tell you if it already exists.
INSERT INTO processed_events (event_id, status, created_at)
VALUES ($1, 'processing', now())
ON CONFLICT (event_id) DO NOTHING;
If the insert affects one row, you're the first to see this event. Go do the work. If it affects zero rows, a previous delivery already claimed it, so you return 200 and move on.
That unique constraint is doing the heavy lifting.
Two retries arriving in the same millisecond both try to insert. The database serializes them. Exactly one wins the insert, the other gets the conflict. No application-level lock needed.
Pick the right key, though. Don't use the payload hash as your dedupe key if the sender adds a fresh timestamp or nonce to each retry, because then two identical events look different. Use the sender's event ID. If they genuinely don't provide one, you'll have to build a stable fingerprint from the business fields that actually matter (order ID plus event type, for example), and that's a worse position to be in.
Keep rows around for at least a week. Senders retry for 24 hours, but I've seen stragglers. Deleting too early reopens the double-act window. A weekly cleanup job that drops anything older than 30 days has never cost me anything.
The lookup is a single primary-key hit, so it stays fast even at scale. If you want to see why index choice matters that much on big tables, Binary Search vs Array.includes() on a 50,000-Row Catalog walks through the same lookup-cost reasoning.
The Readback: Check State Before You Act
The dedupe table handles the clean case. It doesn't handle the ugly one where you claimed the event, started working, and crashed. On the retry, your insert conflicts (the row exists), so you'd skip the work. But the work never finished. Now you've got a stuck order.
That's why I don't trust the dedupe row alone.
I read back the actual business state before I act.
Here's the flow. The event says "mark order 5512 as paid." Before I mark it, I load order 5512 and look at its current status. If it's already paid, I do nothing and return 200. If it's pending, I apply the change.
This readback makes the whole handler safe even if the dedupe table is lying to me. The source of truth isn't "did I see this event," it's "is the world already in the state this event wants."
I write handlers so the action itself is a state transition with a guard:
const order = await getOrder(orderId);
if (order.status === 'paid') {
return res.status(200).send('already paid');
}
await markPaid(orderId, event.id);
For money especially, I lean on the downstream system's own idempotency. Stripe lets you pass an idempotency key on charge creation. I use the webhook event ID as that key. Then even if my readback has a gap, the payment API refuses to run the same charge twice. Belt and suspenders, and worth it when a bug costs a real person real money.
The hardest part is getting the status check and the write to be atomic. If I read "pending" and someone else writes "paid" between my read and my write, I double-act. I solve this with a conditional update that only touches rows still in the expected state:
UPDATE orders SET status = 'paid'
WHERE id = $1 AND status = 'pending';
If that returns zero rows changed, the transition already happened, and I return 200 without doing the side effects again. One statement, no lock, no race.
This is the same "reproduce the exact failing condition before you fix it" discipline from Debugging a Production Bug I Cannot Reproduce Locally. You can't fix a double-act you can't see, so I force duplicates in a test by replaying the same event ID twice and asserting the second run changes nothing.
Respond Fast, Then Do the Slow Work
The last pattern is about the response, not the logic. Senders retry on timeout. So if your handler does slow work inline (sending email, generating a file, calling three other APIs) and that work pushes you past the sender's timeout, you cause the retries you're trying to prevent.
My rule: validate the signature, claim the event in the dedupe table, write the raw payload to a queue, and return 200.
All of that should take under 200 milliseconds.
The actual processing happens off the request. A worker picks the event off the queue and does the slow steps. If the worker crashes mid-job, the queue redelivers, and the same readback checks keep that redelivery safe. Idempotency has to cover your internal queue too, not just the external sender.
Validate the signature first, before anything else.
A webhook endpoint is a public URL. Anyone can POST to it. The signature check (an HMAC of the body against your signing secret) is what proves the sender is real. Skip it and your idempotent handler faithfully processes forged events, which is worse than double-processing real ones.
Don't parse the body before verifying the signature, because most signature schemes sign the raw bytes. Parse-then-verify breaks the check and tempts you to turn verification off. Use the raw request body for the HMAC, then parse.
Order matters inside the handler. Signature check, then dedupe claim, then enqueue, then respond. If you enqueue before the dedupe claim, two retries can both land jobs on the queue, and now your worker has to dedupe instead of your endpoint. Cleaner to stop duplicates at the door.
One thing I got wrong early: returning 500 on an unexpected event type. The sender saw the 500 and retried that event forever on its backoff schedule. Now I return 200 for any event I've safely recorded, even ones I don't act on. A 200 means "received, stop sending." Reserve non-2xx for "I genuinely couldn't accept this, please try later," like a database being down.
I run the scheduling side of my social posts through Buffer, and its webhooks follow the same contract: respond fast, dedupe by delivery ID, process async. The pattern travels across every provider I've touched.
Bottom Line
If you build only one thing from this, build the processed_events table with a unique constraint on the event ID and an ON CONFLICT DO NOTHING insert. That single statement stops most double-acts on its own.
Then add the readback so a mid-flight crash can't leave you stuck, and push slow work to a queue so you never time out into a retry storm. Verify the signature before all of it.
Test it the boring way: fire the same event ID at your endpoint twice in a row and assert the second call changes nothing. If that test passes, retries can't hurt you.
I treat these patterns as a checklist on every webhook I wire into a Shopify store or payment flow now. It's maybe 40 lines of code and it's saved me from the one bug nobody forgives, charging someone twice. If you want the wider system I build these handlers inside, the Claude Blueprint lays out how the pieces fit together.
This article contains affiliate links. If you sign up through them, I may earn a small commission at no extra cost to you. (Ad)
Top comments (0)