Webhooks fail in ways that stay invisible until production traffic hits them: the same event arrives twice, a slow database makes the sender give up and retry, and events land out of order. The fix is not a bigger server — it's treating every webhook as at-least-once delivery and making your handler idempotent, fast to acknowledge, and safe to replay. Do those three things and most 2 A.M. pages disappear.
I've built and debugged webhook receivers for payment providers, Git hosts, and internal event buses. The bugs are almost always the same handful, and they're all preventable. Here's the mental model and the code.
Why do webhooks get delivered more than once?
Because the network is unreliable and senders choose safety over precision. A webhook provider (Stripe, GitHub, Shopify, your own service) sends an HTTP POST and waits for a 2xx within a timeout — often just a few seconds. If your endpoint is slow, returns a 5xx, or the acknowledgment packet gets lost on the way back, the sender assumes failure and retries. Your handler already did the work, but the sender never heard "yes."
This is at-least-once delivery, and it's the correct design on the sender's side. It means the burden of not double-processing lands on you, the receiver. Every serious provider documents this. Stripe, for example, is explicit that you may receive the same event more than once and that handlers must be idempotent.
The mistake I see most is a handler that does real work — charge a wallet, send an email, insert a row — before it returns 200. Any slowness turns one logical event into two side effects.
Takeaway: assume every webhook will be delivered at least twice, and design so the second delivery is a no-op.
How do you make a webhook handler idempotent?
Idempotency means processing the same event twice produces the same result as processing it once. The reliable way to get there is a dedup key plus a uniqueness constraint in your database — not an in-memory Set, which evaporates on restart and doesn't work across multiple instances.
Most providers send a stable event ID (id on Stripe events, the X-GitHub-Delivery header on GitHub). Store it. Let the database reject the duplicate atomically:
CREATE TABLE processed_webhooks (
event_id TEXT PRIMARY KEY,
event_type TEXT NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
// Node + node-postgres. Returns true only the FIRST time an event_id is seen.
async function claimEvent(client, eventId, eventType) {
const res = await client.query(
`INSERT INTO processed_webhooks (event_id, event_type)
VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id`,
[eventId, eventType]
);
return res.rowCount === 1;
}
The ON CONFLICT DO NOTHING ... RETURNING pattern does the check and the claim in one atomic statement, so two concurrent deliveries of the same event can't both win the race. If claimEvent returns false, you've already handled this event — acknowledge with 200 and stop.
For the strongest guarantee, do the claim and the side effect in the same transaction, so a crash between "claimed" and "processed" doesn't strand you:
async function handleEvent(pool, event) {
const client = await pool.connect();
try {
await client.query('BEGIN');
const isNew = await claimEvent(client, event.id, event.type);
if (!isNew) {
await client.query('ROLLBACK');
return { status: 'duplicate' };
}
await applySideEffect(client, event); // same transaction
await client.query('COMMIT');
return { status: 'processed' };
} catch (err) {
await client.query('ROLLBACK');
throw err; // let the caller return 500 so the sender retries
} finally {
client.release();
}
}
If your side effect touches an external system you don't control (sending an email, calling a third-party API), you can't wrap it in the same SQL transaction. There, make the external call idempotent too — many APIs accept an Idempotency-Key header; pass your event ID as that key so the downstream service dedupes on its end.
Takeaway: a database uniqueness constraint is the only dedup that survives restarts, deploys, and horizontal scaling.
Should you process webhooks synchronously or queue them?
Queue them. The single highest-leverage change you can make is to acknowledge fast and process asynchronously. Verify the signature, persist the raw payload, return 200, and let a background worker do the real work.
The reason is the retry timeout. If your handler takes 8 seconds and the sender's timeout is 5, you get a retry even on success — you did the work, but the sender never saw your 200 in time. Under load, this snowballs: slow handlers cause retries, retries add load, more load makes handlers slower.
Here's the split:
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
// 1. Verify signature on the RAW body (see next section)
let event;
try {
event = verifyAndParse(req.body, req.headers['stripe-signature']);
} catch {
return res.status(400).send('bad signature');
}
// 2. Persist raw + enqueue, then acknowledge immediately
try {
await enqueue(event); // durable queue: DB table, SQS, Redis stream...
return res.status(200).send('ok');
} catch {
return res.status(500).send('retry me'); // couldn't even enqueue → let sender retry
}
});
The worker then runs handleEvent from above, with retries and backoff you control. Your "durable queue" can be as simple as an events table with a status column if you're not ready for SQS or a Redis stream — the point is that acknowledgment no longer waits on business logic.
Takeaway: acknowledge within the sender's timeout; do the slow work behind a durable queue you own.
What order do webhooks arrive in, and can you trust it?
You cannot trust order. Retries and parallel delivery mean a subscription.updated can land before the subscription.created it logically follows. Design for it.
Two defenses. First, treat each event as a fact about state at its timestamp, and ignore stale ones. If an event carries an updated_at or a version/sequence number, compare against what you've stored and drop anything older:
// Only apply if this event is newer than what we've already recorded.
const applied = await client.query(
`UPDATE subscriptions
SET status = $2, source_updated_at = $3
WHERE id = $1 AND source_updated_at < $3`,
[sub.id, sub.status, sub.updated_at]
);
if (applied.rowCount === 0) { /* stale or duplicate — safely ignored */ }
Second, when an event references an object you haven't seen yet, re-fetch the current state from the provider's API instead of reconstructing it from event history. The webhook tells you something changed; the API tells you the truth right now. This also rescues you from events you missed entirely during an outage.
Takeaway: webhooks are change notifications, not an ordered event log — reconcile against the source of truth when order matters.
How do you keep signature verification from silently breaking?
Signature verification is where "it works on my machine" bites hardest, because the failure is a 400 the sender sees, not an error you see. Two rules:
-
Verify against the raw request body, byte for byte. If your framework parses JSON before you compute the HMAC, re-serialization changes whitespace and key order, and the signature won't match. In Express, mount
express.raw()on the webhook route only — not a globalexpress.json()that runs first. - Keep the signing secret in config, rotate it deliberately, and support two secrets during rotation so in-flight events signed with the old key still verify.
Never skip verification "temporarily." An unverified webhook endpoint is an unauthenticated write to your database that anyone who learns the URL can call.
Quick reference: the failure modes and their fixes
| Symptom | Root cause | Fix |
|---|---|---|
| Same action happens twice | At-least-once retries | DB uniqueness constraint on event ID |
| Retries even when handler succeeds | Handler slower than sender timeout | Ack fast, process in a background worker |
| "Object not found" mid-handler | Out-of-order delivery | Re-fetch current state from provider API |
| Older data overwrites newer | Order not enforced | Compare timestamps/versions before writing |
| Intermittent 400s from sender | Signature computed on parsed body | HMAC the raw bytes; raw parser on that route only |
| Events lost during an outage | No catch-up path | Reconcile via API; use provider's event replay |
Bottom line
If you build only one thing, build the idempotent claim: a processed_webhooks table with the event ID as primary key and ON CONFLICT DO NOTHING. That single constraint kills the most damaging class of bug — double side effects — no matter how many times an event is redelivered. Add fast acknowledgment with a background worker next, because slow handlers manufacture their own retries. Then handle ordering by reconciling against the provider's API rather than trusting the sequence. Verify signatures on raw bytes, always. Do these four and your webhook receiver stops being the thing that wakes you up.
Top comments (5)
The 2 A.M. bugs are usually the ones where retries and idempotency were designed separately. A retry policy without an idempotency key is a duplicate generator. An idempotency key without a retry model is just a nice database column. The useful design is the pair, plus logs that show which path happened.
Thank you so much for your comment.
Completely agree that the pair is the unit of design — treating them separately is exactly how you end up with a retry policy that's technically "safe" but quietly manufacturing duplicates, and your line about logging which path happened is the part people skip most. One thing I'd add to the idempotency side: the key check has to be atomic against concurrent delivery, not just a read-then-write. Two retries can land within the same few milliseconds and both pass a naive "have I seen this key?" lookup, so the dedup needs to lean on a unique constraint or a conditional insert rather than application-level checking. It also helps to store the original response alongside the key, so a duplicate returns the same result instead of just being dropped — that's what makes it safe for the caller, not only for your database. Curious how you're handling key lifetime on your end — do you expire them, and if so what drove the window?
I usually start key lifetime from the longest realistic retry window, then add enough buffer for queue delays and clock skew. The trap is making it infinite by accident. Past a certain point, old idempotency records become audit/history data, not active dedup state.
I usually start key lifetime from the longest realistic retry window, then add enough buffer for queue delays and clock skew.
That framing of lifetime as "longest retry window + slack for queue delay and clock skew" is exactly the right anchor, and calling out the accidental-infinite trap is the part most people miss. The other thing I'd watch is the storage backing that TTL: if you're leaning on a native TTL (Redis expiry, DynamoDB TTL, a Mongo TTL index), the eviction is best-effort and can lag well past the nominal window, so a key can outlive its "active dedup" role by a good margin while still sitting in the hot path. Since you already separate active dedup from audit/history, it can be worth making that split physical too — expire the dedup record on the short clock, then write a slimmer immutable event to the history store so the two concerns don't fight over one retention number. How do you handle the boundary case where a retry lands right as the key ages out — do you treat a near-expiry miss as "process again" and rely on downstream idempotency, or tighten the buffer to make that vanishingly rare?
Yes, the backing store is where the design either holds or quietly leaks. I like native TTLs for cleanup, but I would still treat the idempotency decision as an explicit state transition with logs. Expiry should remove old keys; it should not be the only thing explaining why a retry was accepted or rejected.