Almost every webhook system has a retry policy. Far fewer have a good answer to the question that comes immediately after it: what happens when the retries run out?
That moment is where most delivery systems quietly lose data. Not in a dramatic outage, but in a catch block that logs an error nobody reads, on an endpoint that was down for eleven minutes.
Three endings, and most systems ship two
An outbound event has exactly three terminal states:
- Delivered.
- Still trying.
- Gave up.
In-house delivery code usually models the first two with real care. There is a retry loop, exponential backoff, maybe some jitter. State three is where the design thins out. The loop exits, an error is logged, and the event stops existing in any form the team can act on.
A dead letter queue is what you build so that state three is a place rather than an ending.
A dead letter queue is not a queue of failures
It is a queue of decisions nobody has made yet.
That distinction sounds like semantics and it is not. Every entry in a well built dead letter queue is an event where the system has already exhausted everything it can do automatically, and a human now has to choose one of three things: replay it as is, fix the receiving endpoint and then replay it, or discard it deliberately.
Once you frame it that way, the design follows. A queue of decisions needs enough context for someone to decide, it needs to reach a person who is capable of deciding, and it needs an obvious action to take once they have.
Most dead letter implementations fail on at least one of those three.
Deciding when to stop
Before anything lands in a dead letter queue, you have to decide what "give up" means. Three rules cover most cases.
Separate retryable from non-retryable. A 500, a 502, a 503, a connection reset, a timeout: retry all of those, because the endpoint may well be fine in thirty seconds. A 400, 401, 403 or 422 will not fix itself on the next attempt. Something about the request is wrong, or the credentials are wrong, and hammering it twelve more times over six hours just adds noise. Fail those fast and dead letter them immediately, because the sooner a human sees a 401 the better.
A 429 sits in the middle. Retry it, but honour the Retry-After header rather than your own backoff curve.
Cap by attempts and by wall clock, not just attempts. Exponential backoff compounds quickly. If you retry twelve times with doubling delays, the last attempt lands days after the first. By then the event may be meaningless and the customer has long since noticed.
A front-loaded schedule handles this better than a pure doubling curve. Something like 1s, 5s, 30s, 2m, 10m, 30m, 1h, 6h, 24h catches the common case (an endpoint that blips for a few seconds) almost immediately, while still giving a genuinely down endpoint most of a day to come back. Add roughly ten percent random jitter to every delay, or a thousand retries queued during the same outage will stampede the endpoint the moment it recovers.
Use a circuit breaker so one dead endpoint does not fill your queue. If every delivery to a given endpoint has failed for the last ten minutes, stop trying and start parking. Otherwise a single customer whose server is down for an afternoon generates tens of thousands of doomed attempts, saturates your workers, and slows delivery for everyone else.
There is a fourth case worth handling separately. If an endpoint returns 410 Gone, or 404s consistently for days, the subscription itself is dead. Disable it and notify the account owner rather than dead lettering thousands of events nobody will ever replay.
What to store
The most common mistake in a homegrown dead letter queue is storing the error instead of the event.
A row that says POST failed: 503 is a log line, not a recovery mechanism. You cannot replay it. You have preserved the evidence of the problem and thrown away the thing you actually needed.
A dead letter entry needs the full request as it would be sent again:
- the exact payload body
- the headers, including the signature and its timestamp
- the destination URL and the subscription or endpoint id
- the event id and the idempotency key
- the original event creation time, separate from the time it was dead lettered
- the complete attempt history: timestamp, status code, latency, and response body for every single try
- the reason delivery stopped: attempts exhausted, time budget exceeded, circuit open, non retryable status
The response body is the field teams most often skip, usually for storage reasons, and it is the field that saves the most time. A bare 500 tells you nothing about who owns the problem. A 500 with {"error":"unknown_currency: KES"} in the body tells you this is a broken integration rather than a flaky network, and it tells you in one glance instead of one afternoon.
Truncate the body if you need to. Two kilobytes is usually plenty. Do not drop it.
The dead letter queue nobody looks at
A dead letter queue without alerting is /dev/null with extra steps and a storage bill.
This is the failure mode I would watch for most closely, because it feels like success. The table exists. Rows are being written. Delivery has technically been made recoverable. But nobody is subscribed to it, so events accumulate for weeks and the first person to look is an engineer investigating a customer complaint that has already escalated.
Two rules make it real:
Alert on rate, not on individual entries. One dead letter is noise, and paging on every single one trains people to ignore the channel within a week. Forty from the same endpoint inside ten minutes is an incident. Alert on the second thing.
Alert on age. An entry that has been sitting untouched for seven days is a decision nobody made. That is a different alert with a different urgency, and it should go to whoever owns the integration rather than to whoever is on call.
Replay is harder than it looks
The replay button is the payoff for all of this, and it is also where you can do real damage. Three questions are worth asking before you press it.
Is the event still true? Some events are facts and stay valid forever. payment.succeeded was true when it happened and it is still true four days later. Other events are snapshots of state. order.status.updated from Tuesday may now describe a status that has been superseded twice, and replaying it can walk the receiver's state backwards. Facts are safe to replay indefinitely. Snapshots should either expire or be replaced by a fresh read of current state.
Will the receiver double process it? At least once delivery means duplicates are inevitable and replay makes them likely. The idempotency key you generated when the event was first created has to survive into the dead letter entry and go out with the replay, unchanged. If replaying charges a card twice, the dead letter queue has made things worse rather than better.
Does order matter? This one gets skipped and it is the one that corrupts data. If you replay event 40 after events 41 through 60 have already been delivered, the receiver processes an outdated change on top of newer ones. Where ordering matters for a given entity, replay needs to pause that entity's stream, replay in sequence, then resume. Where it does not matter, say so explicitly in your docs so consumers know not to rely on it.
There is a fourth option worth building, which is to edit before you replay. A surprising share of dead letters are not transport failures at all. The endpoint URL changed, or the payload carries a field the receiver rejects. If the only action available is replay as is, those events are stuck forever and someone ends up writing a one off script. Being able to correct the destination or the body and then resend turns a dead end into a two minute fix.
Poison messages
There is one specific pattern worth designing against.
An event that reliably crashes the receiver will fail every attempt, land in the dead letter queue, get replayed by a well meaning engineer, fail again, and come straight back. Left alone it can loop indefinitely, and if you replay in bulk it can take the receiving service down each time it comes around.
Count replays per event. After two or three, quarantine it and require an explicit override to try again. The loop is almost always a malformed payload or a schema mismatch, and no number of retries will fix either.
What building this actually costs
None of the individual pieces here are hard, and that is exactly why teams underestimate the whole.
The rough shape is an events table with a real state machine, a separate attempts table so history survives, a scheduler that wakes up delivery at the right time, a worker pool that does not starve under load, backoff with jitter, circuit breaker state tracked per endpoint, signature generation and zero downtime secret rotation, SSRF protection on every destination URL your customers can set, an admin interface where a human can actually see and act on dead letters, authentication and an audit trail on the replay action because replay is a write operation with real consequences, and a retention policy so the tables do not grow without bound.
Any competent backend team can build that. The honest cost is not the initial build, it is that you now own a piece of infrastructure that has to keep working correctly while you build the product your customers are actually paying for, and its failure modes are silent by nature. Delivery infrastructure does not page you when it breaks. It just stops mentioning things.
If your event volume is low and your endpoints are internal, building it is a perfectly reasonable call. If you are delivering to endpoints you do not control, the surface area grows faster than most teams expect.
The short version
Retries buy you time. The dead letter queue is what turns the events that time did not save into something recoverable.
Store the whole event, not the error. Alert on rate and on age. Keep idempotency keys intact through replay. Decide up front which of your events are facts and which are snapshots. And make sure a real person receives the queue, because a dead letter queue nobody reads is just a slower way to lose data.
Mittr is the reliable action layer for webhooks and AI agent actions. Every action is written to Postgres before delivery, retried on a front-loaded schedule with jitter behind a per-endpoint circuit breaker, and logged on every attempt with status code, latency, and response. Anything that exhausts its retry budget is dead-lettered rather than dropped, where it can be inspected, edited, and replayed, with alerting on Slack, PagerDuty, or email. More at mittr.io.
Top comments (0)