We found out our webhook delivery was broken the same way most teams do: a customer told us. Not an alert, not a dashboard going red — an email asking why a payment status update from three hours earlier had never shown up in their system. By the time we'd pulled the logs, we counted just over six hours where a downstream endpoint had been failing silently, and every event meant for it was gone. Not queued, not retried. Gone.
The endpoint in question belonged to a mid-sized integration partner whose ops team had rotated a TLS cert and, in the process, briefly served a handshake our client library didn't like. Our webhook sender treated that as a delivery failure, logged it at a level nobody was watching, and moved on. There was no retry logic — we'd built a fire-and-forget publisher because in two years of running it, deliveries had basically always succeeded. That streak was the problem. It meant we'd never had to think about what "basically always" leaves out.
The fix looked simple on a whiteboard and took us about three weeks to get right in production, mostly because the interesting failure modes only show up under real load and real partner flakiness.
Exponential backoff, not fixed intervals
Our first instinct was to retry every 30 seconds for five minutes. That's exactly the pattern that turns one struggling downstream service into a thundering herd — every failed webhook from every affected customer retries in lockstep, which is a great way to keep a recovering endpoint down. We moved to backoff starting at 10 seconds and doubling up to a ceiling of about 30 minutes, with jitter of ±20% on every attempt so retries from different events don't stack on the same tick. Six attempts over roughly two hours, then the event moves to a dead-letter queue instead of disappearing.
The dead-letter queue is the actual fix, not a nice-to-have
The bug wasn't "we don't retry enough times" — it was "when retries are exhausted, the event vanishes." Our DLQ is a plain durable table: event ID, destination, payload, attempt history, and last error. Nothing fancy. What made it useful was building the replay tool at the same time as the queue, not months later. An event sitting in a DLQ that nobody can inspect or resubmit is just a slower way to lose data. We wired ours to page on-call once an endpoint accumulates more than 20 dead-lettered events in an hour, because that's a much stronger signal of "this partner's endpoint is actually down" than any single failed request.
Idempotency keys had to come first, not after
Retries only work if replaying a webhook is safe. We'd been lucky that most of our event handlers were naturally idempotent, but "most" isn't a guarantee, and a payment-status webhook is exactly the kind of event where a duplicate delivery causing a duplicate downstream action is worse than the original failure. Every event we send now carries a UUID in the payload and header, and we document — loudly, in the integration guide — that consumers are expected to dedupe on it. That's a contract change, not just an internal one, and we spent real time making sure existing partners knew before we shipped it.
Distinguishing "retry this" from "don't bother" mattered more than we expected
A 500 from a downstream service is worth retrying. A 400 because the payload is malformed is not — retrying it six times over two hours just delays the DLQ entry and the alert that would have caught the real bug faster. We now classify failures at send time and only apply backoff to the response codes that are actually likely to resolve on their own.
None of this is novel — retry-with-backoff-and-DLQ is a well-worn pattern. What's easy to underestimate is how much of the value is in the boring parts: the replay tooling, the alerting thresholds, the idempotency contract with partners. The queue itself is a day of work. Making failures visible and recoverable instead of silent is the part that actually prevents the next 2 a.m. email from a customer telling you something broke hours ago.
We ended up rebuilding a good chunk of our event-delivery pipeline around these lessons at Edilec, and it's now the default pattern we reach for anytime a service needs to guarantee delivery to something outside our control — you can see more of how we approach this kind of reliability work at edilec.com.
Top comments (1)
The six-hour gap is a good reminder that "delivery succeeded most of the time" is not a reliability strategy. Starting retries at 10 seconds, doubling to a 30-minute ceiling, and adding 20% jitter avoids turning a partner outage into synchronized self-inflicted load, while the durable DLQ and replay tool make failure recoverable instead of merely delayed. The founder-level takeaway is that webhook reliability is part of the integration contract: idempotency keys, documented deduplication, and clear 400-versus-500 retry behavior need to be designed before shipping, not bolted on after the first customer discovers the gap.