Stop Guessing Your Webhook Dedup TTL: Derive It From the Delivery Contract

#api #backend #database #architecture

Your idempotency key's lifetime isn't a number you pick because it feels safe — it's a number you derive from the sender's delivery contract: the provider's retry window, plus queue lag, plus clock skew, plus however long a human might manually replay an event. Add those up and you get a floor, not a vibe. And there's a second, quieter rule most tutorials miss: dedup the delivery event, never the business resource — otherwise your duplicate protection starts blocking legitimate later changes to the same thing.

That second point is the one that bites in production. A commenter on an earlier post about idempotency TTLs put it well: in local-business automation — review replies, listing updates — duplicate prevention has to survive retries without blocking a legitimate later edit. This post is about doing both correctly.

What actually determines the TTL?

An idempotency record does exactly one job: when the same delivery arrives twice, a uniqueness constraint rejects the second copy so your handler doesn't run twice. That job ends the moment the sender stops retrying. So the correct TTL is "how long could a retry of this exact event still show up?" — and that's a sum of four measurable things, not a round number.

TTL = provider_retry_window
    + queue_lag_budget
    + clock_skew_allowance
    + manual_replay_grace

Each term comes from a real source, not intuition:

Input	Where you source it	Why it's in the sum
Provider retry window	The sender's webhook docs (retry schedule / max age)	The dominant term — a retry after this can't reach you
Queue lag budget	Your own p99 time from "received" to "processed"	A delivery can sit in your queue past the provider's window
Clock skew allowance	A fixed pad (minutes) for mismatched clocks	Your `expires_at` and the sender's clock disagree
Manual replay grace	Your ops policy for "resend last N days"	A human replaying old events must still be deduped

The provider retry window is usually the big one, and it varies wildly. As of mid-2026, some providers retry an event over a period of a few days with exponential backoff; others will keep trying for a week or more, and a few let you manually resend events from a dashboard weeks later. Do not trust your memory here — the exact schedule is the kind of thing that changes between API versions, so read the current docs for every sender you integrate and record the number you found.

Takeaway: if you can't point at the doc line and the p99 metric that produced your TTL, you didn't derive it — you guessed it.

A worked example: putting numbers on the contract

Say you ingest webhooks from a payments provider whose docs describe retrying failed deliveries over roughly three days. Your queue occasionally backs up; your p99 processing lag is about six hours. Your ops runbook allows replaying the last two days of events after an incident. Put it together:

provider_retry_window: 3 days
queue_lag_budget: ~6 hours (round up to 1 day)
clock_skew_allowance: 10 minutes
manual_replay_grace: 2 days

That lands around 6 days, so a 7-day TTL is defensible and you can say why. Change providers and the number changes with the contract — a sender that retries for a week pushes you toward a 10–14 day key, and that's correct, not excessive. The mistake is copying one provider's "7 days" onto a different provider whose contract says something else.

Takeaway: the same code deserves different TTLs per sender, because each sender ships a different delivery contract.

The trap: dedup the event, not the resource

Here's where a reasonable-looking design quietly corrupts data. It's tempting to make your idempotency key the thing being changed — reply:review_12345 or update:listing_678. It reads naturally: "only reply to this review once." But that key conflates two different questions:

Is this the same delivery I already processed? (dedup — what the key is for)
Is this the same business intent as before? (not your key's job at all)

If your key is the resource, then the second legitimate edit to that review reply — a correction, an updated listing hour, a new reply after the customer responded — collides with the old key and gets silently dropped as a "duplicate." You didn't prevent a double-process; you blocked a real change.

The fix is to key on the delivery's own unique identifier — the provider's event ID (evt_...), or a hash of the raw payload plus a delivery ID if the sender doesn't give you one. That value is unique per delivery attempt of one event and identical across retries of that same event. A later, genuinely new event carries a different ID and sails through, exactly as it should.

def handle_webhook(event_id, payload, ttl):
    # event_id is the provider's delivery id (evt_...), NOT the resource id.
    cur = db.execute(
        """
        insert into processed_events (event_id, expires_at)
        values (%s, now() + %s)
        on conflict (event_id) do nothing
        """,
        (event_id, ttl),
    )
    if cur.rowcount == 0:
        return  # retry of an event we already handled; safe no-op
    process(payload)   # a later, different event has a different id -> runs

One honest caveat about this shape: there's a crash window between the insert and process. If the worker dies after the row lands but before process finishes, the retry sees the key and skips — you've marked done something never done. If that class of bug matters to you, wrap the insert and the side effect in one transaction (works when the side effect is a DB write), or make process itself idempotent downstream so a re-run is harmless. The uniqueness key protects against duplicate delivery, not against your own partial failure — those are two different problems and it's worth knowing which one you've actually solved.

Takeaway: key on the event to block retries; leave the resource free to change, or your dedup becomes an accidental write-lock on real work.

How do you expire keys without a giant DELETE?

A single DELETE FROM processed_events WHERE expires_at < now() against a table with tens of millions of rows is a long, lock-heavy, bloat-generating scan — the kind of maintenance job that causes the incident it was meant to prevent. Two approaches that hold up:

Batched deletes. Delete in small chunks on a schedule so no single statement holds locks long or generates a giant amount of dead tuples at once.

delete from processed_events
where ctid in (
  select ctid from processed_events
  where expires_at < now()
  limit 10000
);
-- run on a loop until 0 rows affected

Time-partitioning. Range-partition the table by expires_at (say, daily), and reclaim space by dropping whole partitions — a metadata operation, not a row-by-row scan.

create table processed_events (
    event_id    text not null,
    expires_at  timestamptz not null,
    primary key (event_id, expires_at)
) partition by range (expires_at);

Partitioning has a real tradeoff, so don't adopt it blind: a partitioned table's primary key must include the partition column, so your uniqueness is now on (event_id, expires_at), not event_id alone. If the same event could ever be written with two different expires_at values, the uniqueness guarantee leaks. Keep expires_at deterministic for a given event (derive it from the event's own timestamp, not now()), and the leak closes. If you can't guarantee that, stick with batched deletes on a plain unique index.

Takeaway: pick partition-drop for clean, cheap expiry — but only after you've made expires_at deterministic, or you trade a bloat problem for a correctness one.

FAQ

How long should an idempotency key live for webhook deduplication?
Long enough to cover the sender's full retry window plus your queue lag, clock skew, and manual-replay policy — commonly a few days to about two weeks, derived per provider. Never copy one provider's number onto another; read each sender's retry schedule and add your own lag budget.

Should the idempotency key be the event ID or the resource being updated?
The event ID (the provider's delivery identifier), never the resource. Keying on the resource blocks legitimate later changes to that resource as false "duplicates"; keying on the event dedups retries while leaving future genuine updates free to run.

How do I delete expired idempotency keys without locking the table?
Delete in small batches on a schedule, or range-partition by expiry and drop old partitions instead of running one large DELETE. A single delete over a huge table causes long locks and table bloat.

Bottom line

Treat the TTL as derived, not chosen: sum the provider's retry window, your queue lag, a clock-skew pad, and your manual-replay grace, and keep the receipts so the number is defensible per sender. Key your dedup on the delivery event, not the business resource, so retries are rejected while real later edits still get through. Expire with batched deletes or partition drops rather than one giant scan. Do those three things and your idempotency table stays a correctness guarantee instead of turning into the thing that blocks changes and slows every insert.