If retries and idempotency were designed separately in your system, you have a duplicate generator. A retry policy without an idempotency key resends work that already happened; an idempotency key without a retry model is a database column nobody's decision depends on. The fix is to treat them as one design — a single path where the retry decision and the dedup decision are made against the same key, and where the logs tell you afterward which branch each event took.
I learned this the slow way, reading production logs at hours I'd rather forget. A commenter on an earlier post put it more sharply than I had: the 2 A.M. bugs are almost always the ones where the retry and the idempotency were built by two different people (or the same person on two different days) and never introduced to each other. This post is the combined design, end to end, with runnable code.
Why do separate retry and idempotency layers still produce duplicates?
Because each layer is locally correct and globally blind. Picture the common shape: a sender POSTs an event, your handler does the work, then something downstream times out. Your retry layer — maybe a queue with a redelivery policy, maybe the upstream provider itself — sees no success and resends. Your idempotency check, if it exists, might live inside the handler, after the expensive work has already run. So the dedup key gets written on attempt one, the work runs on attempt one, and on the retry the key check happens after you've charged the card again because the ordering was wrong.
The two mechanisms have to agree on three things or they leak:
- The same key. The retry must carry the identical idempotency key as the original attempt, or dedup can't recognize it.
- The order of operations. The dedup claim must happen before the side effect, atomically, so a retry that arrives mid-flight can't slip past.
- A shared record of what happened. When you're debugging, you need to know whether a given attempt was a first-run, a deduplicated no-op, or a retry of an in-progress attempt.
Miss any one and you get duplicates that are invisible until real traffic interleaves the attempts.
Takeaway: retries and idempotency are not two features, they're two ends of one control path — same key, claim-before-work, shared audit trail.
What does the combined design look like?
The core is a single state machine keyed by the idempotency key, with a status column. Three states are enough: in_progress, done, failed. The claim, the work, and the retry decision all read and write this one row.
CREATE TABLE webhook_events (
event_key TEXT PRIMARY KEY, -- provider event id or a hash you compute
status TEXT NOT NULL, -- 'in_progress' | 'done' | 'failed'
attempts INT NOT NULL DEFAULT 1,
result JSONB, -- cached response for replays
locked_until TIMESTAMPTZ, -- lease so a crashed worker doesn't wedge the key
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
The claim is a single atomic statement. This is the whole trick: INSERT ... ON CONFLICT lets you either win the claim or discover the current state in one round trip, with no read-then-write race.
// Node + node-postgres. Returns { path, row } describing what this attempt is.
async function claim(pool, eventKey, leaseSeconds = 30) {
const { rows } = await pool.query(
`INSERT INTO webhook_events (event_key, status, locked_until)
VALUES ($1, 'in_progress', now() + ($2 || ' seconds')::interval)
ON CONFLICT (event_key) DO UPDATE
SET attempts = webhook_events.attempts + 1,
locked_until = now() + ($2 || ' seconds')::interval,
updated_at = now()
WHERE webhook_events.status = 'failed'
OR webhook_events.locked_until < now() -- stale lease, safe to retake
RETURNING event_key, status, attempts, result, (xmax = 0) AS inserted`,
[eventKey, String(leaseSeconds)]
);
if (rows.length === 0) {
// Conflict, but the WHERE blocked the update: key is 'done' or actively 'in_progress'.
const { rows: cur } = await pool.query(
`SELECT status, result FROM webhook_events WHERE event_key = $1`, [eventKey]);
return cur[0].status === 'done'
? { path: 'duplicate_done', row: cur[0] } // replay: return cached result
: { path: 'in_progress_elsewhere', row: cur[0] }; // another worker holds the lease
}
return { path: rows[0].inserted ? 'first_run' : 'retry_after_failure', row: rows[0] };
}
Now the handler branches on path, and every branch is named — which is exactly what makes the logs readable later.
app.post('/webhooks/:provider', async (req, res) => {
const eventKey = req.get('X-Idempotency-Key') || req.body.id; // stable per logical event
if (!eventKey) return res.status(400).send('missing event id');
const { path, row } = await claim(pool, eventKey);
req.log.info({ eventKey, path, attempts: row.attempts }, 'webhook.claim');
if (path === 'duplicate_done') return res.status(200).json(row.result);
if (path === 'in_progress_elsewhere') return res.status(409).send('processing'); // let it retry later
try {
const result = await doTheWork(req.body); // the ONE place side effects happen
await pool.query(
`UPDATE webhook_events SET status='done', result=$2, updated_at=now()
WHERE event_key=$1`, [eventKey, result]);
req.log.info({ eventKey, path, outcome: 'done' }, 'webhook.done');
return res.status(200).json(result);
} catch (err) {
await pool.query(
`UPDATE webhook_events SET status='failed', updated_at=now()
WHERE event_key=$1`, [eventKey]);
req.log.warn({ eventKey, path, err: err.message }, 'webhook.failed');
return res.status(500).send('will retry'); // signals the sender/queue to redeliver
}
});
Notice what the two layers now share: the retry decision (500 vs 409 vs 200) is made from the same row the dedup claim wrote. There's no second source of truth.
Takeaway: one atomic claim that returns a named path collapses "should I retry?" and "have I seen this?" into a single decision.
How should the retry policy be configured to match this?
The handler above only decides whether to signal a retry. The when and how many live in whatever redelivers — a queue, a scheduler, or the provider. The pairing rule is that the retry side must (a) preserve the key and (b) back off long enough that an in_progress lease usually clears before the next attempt.
| Concern | Retry-only design | Idempotency-only design | Paired design |
|---|---|---|---|
| Duplicate side effects | Frequent under load | Depends on ordering | Prevented by claim-before-work |
| Lost events | Handled | Not addressed | Handled |
| Crashed mid-process | Wedged or duplicated | Wedged | Lease expires, retry retakes safely |
| Debuggability | "it retried" | "key existed" | Named path per attempt |
| Config coupling | Backoff only | Constraint only | Backoff tuned to lease TTL |
A concrete starting point as of mid-2026: exponential backoff with jitter, base delay of a few seconds, a lease (locked_until) shorter than your first retry interval so a genuinely crashed worker's key frees up, and a cap on total attempts after which the event goes to a dead-letter store for human eyes. If you want the managed version of this, a durable queue like AWS SQS with a redrive policy handles the backoff-and-dead-letter mechanics so your handler only owns the claim logic.
Takeaway: tune the retry backoff to be longer than the idempotency lease, so a legitimate in-flight attempt is never mistaken for a dead one.
How do you make the logs tell you which path happened?
This is the part most teams skip, and it's the part that actually saves the 2 A.M. hour. Because every attempt logs its path and attempts count against a shared eventKey, one query reconstructs the full history of any event:
webhook.claim eventKey=evt_9f3 path=first_run attempts=1
webhook.done eventKey=evt_9f3 outcome=done
webhook.claim eventKey=evt_9f3 path=duplicate_done attempts=1 <- retry, correctly no-op'd
When someone reports "the customer got charged twice," you grep the key and the story is right there: did a first_run run twice (a real bug in the claim), or did a duplicate_done correctly short-circuit (working as designed, the double-charge is elsewhere)? Without the named path, both look identical in the logs and you're guessing. With it, you know within seconds which layer to suspect.
Takeaway: log the decision, not just the event — a path label per attempt turns a duplicate investigation from archaeology into a single grep.
FAQ
Do I need an idempotency key if my webhook provider already retries?
Yes. The provider retrying is exactly why you need the key — its retries are the duplicate source your handler must absorb. The provider guarantees at-least-once delivery; the idempotency key is how you turn that into effectively-once processing.
Where should the idempotency check go — before or after the work?
Before, and atomically. Claim the key in the same statement that would reveal a conflict, then do the side effect only if you won the claim. Checking after the work runs means a concurrent retry can execute the side effect twice before either check sees the other.
What idempotency key should I use if the provider doesn't send an event ID?
Compute a stable hash from the fields that define the event's identity — for example the resource ID plus the event type plus a provider timestamp. Avoid hashing the entire raw body if it contains fields that vary between deliveries of the same logical event.
Bottom line
Build the retry and the idempotency as one path or you'll debug them as one incident. Put a single atomic claim before every side effect, key the retry decision off the same row the claim writes, and set your backoff longer than the claim's lease. Then log a named path for every attempt — first_run, retry_after_failure, duplicate_done, in_progress_elsewhere — so that when a duplicate does slip through, the logs tell you which layer failed instead of leaving you to guess at 2 A.M.
Top comments (0)