Set the lifetime of an idempotency key to the sender's longest realistic retry window plus a buffer for queue lag, clock skew, and manual replays — for most providers that lands somewhere between a few days and about two weeks. Keep it shorter and a late retry sneaks a duplicate through; make it effectively infinite and your dedup table quietly grows until the index that guards correctness becomes the thing that slows every insert. The fix is to treat "still preventing duplicates" and "kept for history" as two different jobs with two different lifetimes.
If you store an event ID on first receipt and reject the duplicate on a uniqueness constraint, you have the right shape. The open question nobody answers in the tutorial is: when do you delete that row? Delete too early and you reopen the exact double-processing bug the key existed to prevent. Never delete and you pay for it later in bloat and slow writes. This post is about picking that number deliberately.
Why does an idempotency key even need an expiry?
An idempotency record does one job: when the same event arrives a second time, a uniqueness constraint rejects it so your handler doesn't charge the card twice. That job has a natural end. Once the sender has stopped retrying a given event — because it finally got its 2xx, or gave up — no future delivery of that event will ever arrive. The record can no longer prevent anything. It's inert.
Inert rows are not free. Every insert into a table with a unique index also updates that index. As the table grows into the tens or hundreds of millions of rows, the index gets deeper, cache hit rates drop, and autovacuum has more to chew through. The structure you added to protect correctness gradually taxes throughput on the hot path — every incoming webhook pays for it. And "delete everything older than N days" run against one giant table produces its own long-running, lock-heavy, bloat-generating scans if you do it naively.
So expiry isn't a nice-to-have. An idempotency table without a retention plan is a slow leak that only shows up under the traffic you built it for.
Takeaway: a dedup key stops doing useful work the moment the sender stops retrying — everything you keep past that point is cost, not protection.
How long is the sender's retry window, really?
The floor for your TTL is however long the sender might keep retrying the same event, because a retry that arrives after your key expires is an undetected duplicate. This varies a lot by provider, and the numbers move, so treat any specific figure as "check the current docs" rather than gospel.
As a rough shape, as of mid-2026: Stripe retries failed webhook deliveries with exponential backoff for up to roughly three days. Shopify retries a fixed number of times across a window measured in a couple of days. GitHub does only a small number of automatic attempts and leans on manual redelivery instead. Your own internal event bus retries for exactly as long as you configured it to — which is the one window you can actually look up with certainty.
The trap here is reading the happy-path retry number and forgetting the tail. Providers often distinguish "normal backoff" from behavior during their own incidents, where a backlog can replay events well outside the usual window. When in doubt, anchor to the longest documented figure, not the typical one.
| Sender's realistic max retry window | Example class of sender | Floor for your dedup TTL |
|---|---|---|
| Minutes, a handful of attempts | Lightweight internal hooks | ~24 hours |
| Up to ~24 hours | Many SaaS event senders | ~3 days |
| Up to ~3 days | Stripe-class payment providers | ~7 days |
| Up to about a week | Providers that replay incidents | ~14 days |
Takeaway: your TTL floor is the sender's longest retry window, including incident-recovery replays — not the number from the happy-path docs.
What's the actual formula for a safe TTL?
The retry window is the floor, not the answer. The reader observation that prompted this post is exactly right: start from the longest realistic retry window, then add buffer, and the real danger is making the total infinite by accident. Three things eat into your margin:
- Queue lag. If your endpoint acknowledges fast and pushes the event onto a queue (which is the correct design), the dedup check often happens when the worker processes it — minutes or hours after receipt during a backlog. The key must still be alive then.
-
Clock skew. If
expires_atis computed on one machine and compared on another, or against the database's clock, small drifts matter at the boundary. Compute expiry with the database's ownnow()so there's a single clock. - Operational replays. During incident recovery you may deliberately re-drive a batch of old events from a dead-letter queue. If your TTL already expired those keys, your replay double-processes. Give yourself room to replay.
So:
TTL = longest_retry_window + queue_lag_margin + skew_margin + replay_margin
Concretely, for a Stripe-class three-day window I use seven days: three for retries, and four of slack that has covered every backlog and manual replay I've hit. That's the whole method — pick the window, add honest buffer, and stop well short of infinity.
Takeaway: the buffer isn't padding — it's the queue backlog and the 2 A.M. replay you haven't had yet.
How do you expire the keys without a maintenance headache?
Add an explicit expires_at and let the database own the clock:
CREATE TABLE processed_webhooks (
event_id TEXT PRIMARY KEY,
event_type TEXT NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX processed_webhooks_expires_idx ON processed_webhooks (expires_at);
Claim the event and set its lifetime in the same atomic insert:
const TTL_DAYS = 7;
// Returns true only the FIRST time an event_id is seen.
async function claimEvent(client, eventId, eventType) {
const res = await client.query(
`INSERT INTO processed_webhooks (event_id, event_type, expires_at)
VALUES ($1, $2, now() + make_interval(days => $3))
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id`,
[eventId, eventType, TTL_DAYS]
);
return res.rowCount === 1;
}
Then delete expired rows in bounded batches on a schedule, so cleanup never takes a long lock or generates a giant transaction:
-- Run in a loop until it deletes 0 rows.
DELETE FROM processed_webhooks
WHERE ctid IN (
SELECT ctid FROM processed_webhooks
WHERE expires_at < now()
LIMIT 10000
);
If the table is huge, batched DELETE still churns dead tuples and vacuum. The lower-maintenance option is time partitioning: range-partition by received_at (native, or with pg_partman) and drop whole day/week partitions, which is near-instant and produces no bloat. The honest trade-off: with declarative partitioning the primary key must include the partition key, so a plain PRIMARY KEY (event_id) no longer enforces global uniqueness across partitions. If your event IDs could recur across the partition boundary, you need application-level care or a different uniqueness strategy — partitioning buys cheap cleanup at the cost of the one-line uniqueness guarantee.
Redis is tempting because SET key 1 NX EX 604800 gives you dedup and TTL in a single command. It works, but under a maxmemory eviction policy Redis can drop a key before its TTL, silently reopening the duplicate window — so it's fine as a fast first-line check, risky as your only source of truth for money-moving events.
Takeaway: batch your deletes or partition and drop — a single unbounded
DELETE FROM ... WHERE expires_at < now()on a large table is its own outage.
When does an old key stop being dedup state and become audit history?
This is the distinction that keeps the whole design clean. Past the sender's retry window, an idempotency row can no longer prevent a duplicate — nothing will arrive to be deduped. What people actually want when they hesitate to delete it is history: "did we process event X, and when?" That's a real need, but it's a different table with a different lifetime and different access patterns.
Conflating them is what produces the accidental-infinity bug. You keep dedup rows for a year "just in case," and now your hot correctness-critical table is 300 million rows to answer a question you ask once a quarter. Split them:
- Dedup table: small, hot, TTL sized to retry-window-plus-buffer, aggressively pruned. Its only job is the uniqueness check on the write path.
- Event history / audit log: append-only, retained for whatever compliance or debugging horizon you actually need, queried rarely, and free to live in cheaper storage or a partitioned archive.
Write to the audit log if you need the record, and still let the dedup key expire on the short schedule. The uniqueness constraint stays fast because it only guards the window where duplicates are physically possible.
Takeaway: "we might need it later" is a request for an audit log, not an argument for an immortal dedup key — build the second table and let the first one expire.
FAQ
How long should idempotency keys be stored? Store them at least as long as the sender's maximum retry window, plus buffer for queue lag, clock skew, and manual replays. For most providers that's a few days to about two weeks; for a Stripe-class three-day retry window, seven days is a safe default.
What happens if an idempotency key expires too early? A retry that arrives after expiry is treated as a brand-new event, so your handler processes it a second time — the exact double-charge or double-send the key was meant to prevent. This is why the TTL floor must be the sender's longest retry window, not the typical one.
Should I keep idempotency records forever for auditing? No. Once the retry window passes, the key can't prevent duplicates anymore, so keeping it in the dedup table only adds bloat and slows inserts. If you need long-term history, write it to a separate append-only audit table and let the dedup key expire on its short schedule.
Bottom line
Size the TTL from the sender's longest realistic retry window and add honest buffer for queue backlogs and replays — a week covers most payment-grade providers, and shorter is fine for internal hooks that retry for minutes. Enforce it with an expires_at column plus batched deletes, or time partitioning if the table is large enough to make vacuum hurt. Above all, don't let "we might want it later" turn dedup state into a permanent record: keep the correctness table small and expiring, and put history in its own log. Get those two lifetimes separated and the idempotency layer stays both correct and cheap.
Top comments (1)
The TTL decision should come from the delivery contract, not a neat number. I would include provider retry window, queue delay, clock skew, and manual replay policy. In local-business automation, this matters for things like review replies or GBP updates: duplicate prevention needs to survive retries without blocking legitimate later changes.