Retry Budgets and Dead-Letter Queues: Keeping Digital Delivery Reliable When Suppliers Fail

Instant delivery is a promise that is easy to make and hard to keep. When a customer buys a digital code, they expect it in seconds — not after a support ticket, and not after a manual retry the next morning. The hard part is rarely your own code. It is the chain of third-party suppliers, code distributors, payment processors, and email providers that all have to succeed in sequence.

In this post, I want to share a practical pattern that works well for digital goods: combine a retry budget with a dead-letter queue (DLQ), and treat every failed delivery as data, not as an exception to hide.

Why naive retries make outages worse

The most common retry implementation looks like this: if the supplier call fails, try again immediately, then again after a second, then again after two seconds. That works for a single transient blip. It fails badly when the supplier is actually down.

If 1,000 orders hit a failing supplier at once and each one retries five times, you have just turned 1,000 failures into 6,000 requests against an already struggling dependency. This is the classic retry storm. It delays recovery, burns through rate limits, and can cascade into your own database and queue getting saturated.

A retry budget fixes this by capping retries globally, not just per order.

What a retry budget is

A retry budget is a simple rule: in any rolling window, retries may not exceed a small percentage of successful requests. A common starting point is 20%. If you served 1,000 successful supplier calls in the last minute, you allow at most 200 retries in that same window.

When the budget is exhausted, new failures do not retry immediately. They go straight to a holding state. That sounds harsh, but it protects the system. The alternative — unbounded retries — is what turns a 30-second supplier hiccup into a 20-minute outage.

In practice, you need three pieces:

1. A counter for successes and retries, stored in Redis or in-memory with a rolling window.
2. A decision point in your delivery worker: before scheduling a retry, check the budget.
3. An escape hatch for manual replay when the supplier recovers.

The dead-letter queue is not a trash can

When a delivery exhausts its retries or the budget is spent, it should land in a DLQ with full context: order ID, supplier, request payload hash, response code, timestamp, and retry history. The DLQ is not where orders go to die — it is where they wait for a deliberate decision.

A good DLQ workflow has three paths:

• Auto-replay: A background job replays DLQ items after a backoff, but only if the supplier health check passes. Do not replay blindly on a timer.
• Supplier failover: If Supplier A keeps failing for a specific product, route the replay to Supplier B. This requires your catalog to model substitutable inventory, not hard-coded supplier IDs.
• Human review: If the code was already issued but the email bounced, a human needs to reconcile state before you issue a second code. Double-issuing is worse than a delay.

One detail that matters: make DLQ replay idempotent. Replaying the same delivery twice should never create two codes or two charges. Use an idempotency key tied to the order and the delivery attempt, and store the result of the first successful fulfillment.

Backoff, jitter, and timeouts

Even inside the budget, how you retry matters:

• Use exponential backoff with jitter. Without jitter, thousands of workers retry at the same second and create a thundering herd.
• Set aggressive timeouts for supplier calls. A 30-second timeout that blocks a worker is far more expensive than a fast failure that can be retried.
• Distinguish retryable from non-retryable errors. A 429 or 503 is retryable. A 400 with "invalid product code" is not — retrying it will never succeed.
• Log the supplier error taxonomy, not just the HTTP status. Two suppliers can both return 500 for completely different reasons.

Observability: alert on the DLQ, not just on errors

Most teams alert on error rate. For digital delivery, the more useful signal is DLQ depth and age. A DLQ that grows from 0 to 50 in five minutes tells you something is wrong before customers do. A DLQ item older than 10 minutes tells you a specific order needs attention.

Track these metrics:

• Delivery success rate by supplier and product
• Retry rate vs. retry budget consumption
• DLQ depth, oldest item age, and replay success rate
• Time from order to first successful delivery (p50, p95, p99)

If you run a storefront that sells digital codes, these numbers are your real SLA. Customers do not care that your API returned 200 — they care that the code arrived and worked. Shops like Kadokey live or die on that last mile, because a code that arrives late feels the same as a code that never arrives.

A checklist you can steal

• [ ] Cap total retries with a global budget, not just per-order retries
• [ ] Send exhausted deliveries to a DLQ with full context
• [ ] Make replay idempotent and gated on supplier health
• [ ] Separate retryable from non-retryable supplier errors
• [ ] Add jitter to every backoff
• [ ] Alert on DLQ depth and age, not just error rate
• [ ] Test the failure path with chaos drills: kill a supplier in staging and watch what happens

Conclusion

Reliability for digital delivery is not about never failing. Suppliers will fail, networks will blip, and emails will bounce. Reliability is about failing in a bounded, observable, recoverable way.

Retry budgets keep a bad day from becoming a cascading outage. Dead-letter queues give every failed order a clear next step. Together, they turn "the supplier was down" from an excuse into a handled case.

If you are building checkout or fulfillment for digital goods, start there — before you add more suppliers, more products, or more features. Your future on-call self will thank you.