Short answer: retry only failures that can plausibly clear, use capped exponential backoff with jitter, persist each subscriber delivery as its own delayed message, and move exhausted work to a DLQ that has an explicit replay procedure.
For a shipment update fanned out to many SaaS subscribers, the delivery guarantee should be at-least-once with idempotent consumers, not an informal promise that every webhook runs exactly once. A simple queue retry policy is useful only when it protects the primary queue from a failing destination and still leaves operators a bounded way to recover. Set the attempt cap and delay ceiling from the delivery SLO, then test the resulting retry traffic against capacity before deployment.
What a shipment fan-out incident actually teaches
Use a bounded incident model: shipment SHP-20418 changes to out_for_delivery, 2,000 subscriber deliveries are created, and one destination remains unavailable long enough for retries to overlap the next shipment batch. If every failure returns immediately to the ready queue, the bad destination consumes worker slots, healthy subscribers wait behind it, and retry traffic becomes load created by the recovery mechanism itself. The original update is no longer the hard part; isolation is.
The invariant is small enough to put in a design review: one subscriber's failure must not delay another subscriber's first attempt. That means the fan-out record needs a stable delivery ID, each subscriber gets independent retry state, and delayed messages must not occupy an active worker while waiting. The handler also needs a terminal decision. “Try again” without an attempt budget is an unbounded workload disguised as resilience.
This doesn't require claiming exactly-once delivery. A worker can complete the remote side effect and lose confirmation before it acknowledges the queue message, so a later attempt may repeat the call. Keep a stable idempotency key such as SHP-20418:subscriber-731:out_for_delivery; the subscriber can record it and return the prior outcome for a duplicate. The queue owns redelivery, while the application owns whether redelivery is safe.
Short version: isolate first.
The incident is resolved only when the current backlog and the extra retry arrival rate both fit inside the recovery window. If 20 workers can each attempt 10 deliveries per second, the nominal service rate is 200 attempts per second, but treating that as usable capacity would be reckless — request latency, subscriber limits, and ordinary traffic all consume the same budget. Measure those terms in a load test; don't invent a utilization target in a meeting.
How should SaaS background jobs use a simple queue retry policy?
Classify the outcome before calculating a delay. A success is acknowledged. A permanent failure, such as a rejected subscription configuration, goes directly to a terminal review path. A retryable failure is rescheduled only while both the attempt budget and the elapsed delivery window remain open. This distinction matters more than the exact backoff formula because exponential backoff cannot make a permanent error temporary.
For retryable work, use min(base * 2^attempt, cap) and add jitter. The cap prevents one message from disappearing beyond the delivery SLO; jitter prevents a large failed batch from returning at the same instant. I'm not sure any fixed base delay is defensible without the destination latency distribution and the promised delivery window. Those two measurements decide whether the first retry belongs after seconds or minutes.
The following Go program is a complete policy core. It uses deterministic jitter so the example is reproducible; production code can inject a random source and a durable queue adapter. The values are illustrative inputs, not universal defaults.
package main
import (
"fmt"
"hash/fnv"
"time"
)
type Delivery struct {
ID string
Shipment string
Subscriber string
Attempt int
FirstSeen time.Time
}
type Decision struct {
Action string
RunAt time.Time
Reason string
}
func retryDelay(id string, attempt int, base, cap time.Duration) time.Duration {
delay := base
for i := 0; i < attempt && delay < cap; i++ {
delay *= 2
if delay > cap {
delay = cap
}
}
h := fnv.New32a()
_, _ = h.Write([]byte(fmt.Sprintf("%s:%d", id, attempt)))
// Spread retries across the second half of the capped delay window.
half := delay / 2
return half + time.Duration(h.Sum32()%uint32(half+1))
}
func decide(d Delivery, now time.Time, retryable bool) Decision {
const maxAttempts = 6
const deliveryWindow = 30 * time.Minute
if !retryable {
return Decision{Action: "dead-letter", Reason: "permanent failure"}
}
if d.Attempt >= maxAttempts {
return Decision{Action: "dead-letter", Reason: "attempt budget exhausted"}
}
if now.Sub(d.FirstSeen) >= deliveryWindow {
return Decision{Action: "dead-letter", Reason: "delivery window expired"}
}
delay := retryDelay(d.ID, d.Attempt, 2*time.Second, 5*time.Minute)
if now.Add(delay).After(d.FirstSeen.Add(deliveryWindow)) {
return Decision{Action: "dead-letter", Reason: "next delay exceeds delivery window"}
}
return Decision{Action: "reschedule", RunAt: now.Add(delay), Reason: "retryable failure"}
}
func main() {
now := time.Date(2026, 8, 15, 9, 0, 0, 0, time.UTC)
d := Delivery{
ID: "SHP-20418:subscriber-731:out_for_delivery",
Shipment: "SHP-20418",
Subscriber: "subscriber-731",
Attempt: 2,
FirstSeen: now.Add(-4 * time.Minute),
}
fmt.Printf("%+v\n", decide(d, now, true))
}
Persist Attempt, FirstSeen, and the next eligible time atomically with the message transition. A process-local timer isn't a delayed-message system: a restart loses it, and horizontal workers don't share its state. The queue adapter should expose three operations — acknowledge, reschedule at a timestamp, and dead-letter with a reason — while the policy above remains ordinary application code.
Capacity, SLOs, and the retry budget
Start from the subscriber delivery SLO. If updates must reach eligible subscribers within 30 minutes, the sum of queue wait, attempt duration, and scheduled delays has to fit inside 30 minutes. Six attempts with a five-minute cap do not automatically fit; the exact schedule depends on the base delay, jitter, and time already spent waiting. Test the worst allowed path, not the prettiest average.
A useful capacity model is new attempts + retry attempts < sustainable worker throughput. Track each term separately. Queue depth alone hides whether an increase came from new shipments, slow consumers, or repeat work. At minimum, observe age of oldest ready message, age of oldest delayed message, attempts by ordinal, terminal reasons, DLQ inflow, and successful delivery latency by subscriber. Alert on user impact and exhaustion rate rather than on the mere existence of retries.
Then run a failure drill with a controlled subscriber cohort. Hold one destination unavailable, inject a batch, and verify that healthy subscriber latency stays inside its objective. Restore the destination and measure drain time. Repeat with the worker process restarted while messages are delayed. A design that passes steady-state throughput but cannot drain its own retries before the next peak isn't ready.
Do the arithmetic.
Retry amplification is also a cost and on-call issue. A maximum of six attempts means one original delivery can consume up to six executions, logs, and outbound requests. That upper bound belongs in capacity planning even if normal traffic rarely reaches it. If the bound would overwhelm the system, reduce concurrency per destination, shorten the attempt budget, or reject excess fan-out before accepting a delivery promise the platform cannot keep.
Choosing the queue boundary without buying an incident
The decision is less about a feature checklist than about ownership. AWS SQS documents dead-letter queues through a redrive policy and a maximum receive count. BullMQ documents queue-backed job processing and retry behavior. Those are useful implementation references, but neither changes the application-level need for idempotency, failure classification, a delivery window, and a reviewed replay path.
| Option | Operational ownership | Delivery-control trade-off | Prefer it when | Avoid it when |
|---|---|---|---|---|
| Managed queue | Provider runs queue infrastructure; the team owns policy and consumers | Lower broker on-call load, with service-specific semantics | Broker operation isn't a product differentiator | Portability requirements forbid service-specific redrive behavior |
| Library backed by an existing data store | Team owns the library, workers, and data-store capacity | Fast application integration, but retries share a failure and capacity domain with the store | The team already operates that store and can isolate queue load | Queue traffic could threaten transactional workloads |
| Self-hosted broker | Team owns upgrades, capacity, recovery, and clients | Maximum control over topology and retention | Regulation or workload shape justifies dedicated broker expertise | The on-call team cannot rehearse broker recovery |
The catch is that a DLQ is storage, not recovery. Every terminal message needs the original delivery ID, subscriber, shipment event, attempt history, terminal reason, and payload version. Replay must preserve the idempotency key and create an audit record. Blindly moving an entire DLQ back to the ready queue can recreate the same load spike and the same permanent failures, so replay by reason and bounded batch size.
A managed queue is not suitable when its redrive semantics cannot meet portability or audit constraints; stick with a self-hosted broker when those constraints justify the extra on-call burden. Conversely, self-hosting is a poor bargain for a small team that has no capacity to test upgrades and recovery. The buy-vs-build answer is a staffing and failure-domain decision, not a badge of engineering seriousness.
Deployment and recovery checks
Deploy the retry classifier behind a cohort boundary, then compare first-attempt latency, retry rate, and terminal rate with the prior behavior. Keep message schema changes backward-compatible while old delayed messages remain outstanding. A rollback that restores old code but cannot decode yesterday's scheduled payload is not a rollback. Before broad release, verify four paths: a clean first attempt, a retry followed by success, a permanent failure sent directly to the DLQ, and an exhausted retry budget. Confirm that duplicate delivery produces one subscriber-visible state change. Confirm that a worker restart does not change RunAt. Finally, have someone other than the author execute the replay procedure from a bounded DLQ sample; runbooks that only their author can perform are notes, not operational controls. The recommendation has limits, too: don't add a queue when the caller requires a synchronous answer and cannot tolerate deferred completion, don't use retries for validation errors or revoked subscriptions, and don't promise at-least-once fan-out unless subscriber idempotency, per-destination isolation, and DLQ replay are part of the same design review.
Top comments (0)