For an edtech shipment-update fan-out, use at-least-once delivery with idempotent subscriber sends, a small retry budget, exponential backoff with jitter, and a dead-letter queue (DLQ) for exhausted jobs. That is the least complex policy that keeps a temporary downstream failure from becoming either a lost notification or an infinite retry loop.
The important choice is the unit of work. Create one immutable job per subscriber, not one job for the whole shipment audience. A worker can then acknowledge successful subscribers independently while delaying only the failed ones. The shipment event carries a stable ID; each child job adds the subscriber ID and channel. A unique delivery key ties those values together, so a redelivery can be recognized without pretending the queue provides exactly-once effects.
Split the fan-out early.
How should a simple SaaS queue retry failed background jobs with exponential backoff?
Start with failure classification. A timeout, a connection reset, or an HTTP 429 can be transient, so the job may be tried again after a delay. A malformed destination or a rejected payload is terminal and should move directly to the DLQ. Authentication failures deserve their own operational alert: repeatedly sending the same request with unchanged credentials adds load but does not change the outcome. The exact classification depends on the subscriber channel, and I'm not sure a provider's generic error category is sufficient until its documented status codes have been exercised in an eval fixture.
Keep the retry policy deliberately small: count every delivery attempt, cap the count, calculate a delay, and record the decision. For attempt number n, a useful baseline is min(cap, base * 2 ** (n - 1)). Add jitter so a cohort of jobs that failed together does not wake together. This is a policy recommendation, not a promise about the timing precision of a particular queue. Delayed delivery means "eligible after this time"; workers, queue load, and scheduler resolution can make actual execution later.
The DLQ is part of the normal control flow. AWS's SQS documentation describes a redrive policy whose maxReceiveCount controls when a source-queue message moves to a dead-letter queue. It also warns that a DLQ can break strict ordering, which matters if subscriber updates must be observed in sequence. The general lesson travels well: pick a retry budget from the business deadline, and don't attach a DLQ mechanically when reordering would violate the job's contract.
For shipment updates, ordering is often better expressed in application data. Include a monotonically increasing shipment version, let the subscriber-side delivery record reject an older version, and make the message useful on its own. Then a delayed "packed" update cannot overwrite a newer "shipped" state. This turns queue order from a hidden assumption into something an eval can verify.
A runnable Python retry decision before the worker grows complicated
I like to make the policy a pure function first — notebook-sized, deterministic under a seeded random generator, and cheap to evaluate across thousands of generated cases. The queue adapter can stay boring. The example below models decisions without depending on a commercial SDK; wire publish to the queue already used by the application.
from __future__ import annotations
from dataclasses import dataclass, replace
from enum import Enum
import hashlib
import json
import random
from typing import Protocol
class Action(str, Enum):
ACK = "ack"
RETRY = "retry"
DEAD_LETTER = "dead_letter"
@dataclass(frozen=True)
class ShipmentJob:
shipment_id: str
shipment_version: int
subscriber_id: str
channel: str
attempt: int = 1
@property
def delivery_key(self) -> str:
raw = (
f"{self.shipment_id}:{self.shipment_version}:"
f"{self.subscriber_id}:{self.channel}"
)
return hashlib.sha256(raw.encode()).hexdigest()
@dataclass(frozen=True)
class Outcome:
action: Action
delay_seconds: float = 0.0
reason: str = ""
class Queue(Protocol):
def publish(
self, queue_name: str, body: str, delay_seconds: float = 0.0
) -> None: ...
def retry_delay(
attempt: int,
rng: random.Random,
base_seconds: float = 5.0,
cap_seconds: float = 300.0,
) -> float:
exponential = min(cap_seconds, base_seconds * (2 ** (attempt - 1)))
return rng.uniform(0.0, exponential)
def decide(
job: ShipmentJob,
status_code: int,
rng: random.Random,
max_attempts: int = 6,
) -> Outcome:
if 200 <= status_code < 300:
return Outcome(Action.ACK)
retryable = status_code == 429 or status_code >= 500
if not retryable:
return Outcome(Action.DEAD_LETTER, reason=f"terminal:{status_code}")
if job.attempt >= max_attempts:
return Outcome(Action.DEAD_LETTER, reason="attempts_exhausted")
return Outcome(
Action.RETRY,
delay_seconds=retry_delay(job.attempt, rng),
reason=f"transient:{status_code}",
)
def finish_attempt(
queue: Queue, job: ShipmentJob, outcome: Outcome
) -> None:
if outcome.action is Action.ACK:
return
envelope = {
"job": job.__dict__,
"delivery_key": job.delivery_key,
"reason": outcome.reason,
}
if outcome.action is Action.RETRY:
next_job = replace(job, attempt=job.attempt + 1)
envelope["job"] = next_job.__dict__
queue.publish(
"shipment-delivery",
json.dumps(envelope),
delay_seconds=outcome.delay_seconds,
)
return
queue.publish(
"shipment-delivery-dlq",
json.dumps(envelope),
)
The worker still needs an atomic boundary around its effect. Before contacting an external notification channel, reserve the delivery key in durable storage; after success, mark it complete. If the process stops between the external effect and that final write, the same job may be observed again. There is no generic transaction across a queue, a database, and somebody else's API, so the receiving operation should accept an idempotency key when possible. If it does not, define duplicate notification as an explicit product risk rather than hiding it behind an "exactly once" label.
One detail is easy to miss in the example: republishing a delayed retry and acknowledging the current message must be coordinated. A queue with native retry delay can own that transition. With a generic adapter, persist an outbox entry in the same transaction as the local attempt record, publish the outbox asynchronously, and acknowledge only after the durable handoff. Don't acknowledge first. A process stop in that gap loses the retry.
The code treats 5xx as retryable because it models a generic downstream HTTP service, not because every endpoint has the same semantics. A 501, for example, usually won't improve through repetition. Production classification should be an allowlist derived from the downstream contract, while the broad rule above is kept compact for the policy walkthrough.
Delivery guarantees are a product decision, not a queue checkbox
The right guarantee is the one the subscriber can observe. At-most-once delivery avoids duplicates but may lose an update after a worker failure. At-least-once delivery accepts redelivery and requires idempotent effects. Exactly-once processing is sometimes available inside a bounded system, yet the end-to-end claim becomes much harder once the job calls an external channel. For a shipment notification, at-least-once plus a stable delivery key is usually the honest contract. Consider the awkward boundary: the worker sends a version-8 “shipped” message, the channel accepts it, and the worker stops before acknowledging its queue job. The queue is correct to deliver the job again because it cannot observe the channel's side effect. On the second attempt, the same delivery key lets the channel or the application's delivery ledger suppress another visible notification. If version 7 arrives afterward, the stored shipment version rejects it as stale. These are two different protections — deduplication handles the same event twice, while version comparison handles distinct events arriving out of order — and an eval suite should exercise both.
Name that boundary.
| Concern | Policy | Eval assertion |
|---|---|---|
| Temporary throttling | Retry with jittered backoff | A 429 schedules one later attempt |
| Invalid destination | Dead-letter immediately | A terminal 4xx consumes no retry budget |
| Worker redelivery | Reuse the delivery key | The visible effect is recorded once |
| Stale shipment state | Compare shipment versions | Version 7 cannot replace version 8 |
| Exhausted attempts | Preserve the final envelope in the DLQ | The reason and original identifiers remain inspectable |
A fixed six-attempt budget in sample code is not a universal default. Work backward from the useful delivery window. With a 300-second delay cap, a job may remain retryable long after an update has stopped being useful, especially when queue time is added. Conversely, a subscriber integration with a documented hour-long maintenance window may need a longer budget or a separate recovery workflow. Your mileage may vary — the deadline, not a fashionable attempt count, should settle it.
This is also where prompt-cost awareness helps, even though the worker itself is ordinary infrastructure. If a later stage generates personalized notification copy with a model, do not regenerate it on every transport retry. Store the approved rendered payload or a content hash with the job and retry only the delivery side effect. Otherwise a transient channel error quietly multiplies model calls and makes eval results hard to reproduce.
The catch is that a DLQ is not suitable as long-term business storage, and a retry queue is not a scheduler for every kind of future work. Keep authoritative shipment state in the application database. Use the queue to move bounded work, use the DLQ to isolate jobs requiring inspection or redrive, and use a scheduler when an action is intentionally planned far in the future. If strict ordering across all updates is mandatory, stick with a single ordered stream per shipment or subscriber and accept the throughput and head-of-line-blocking trade-off.
Operate the policy with evals, budgets, and deliberate redrive
Before deployment, build a failure matrix rather than testing only the happy path. Feed the pure decision function success, throttling, terminal rejection, the last permitted attempt, and an old shipment version. Seed randomness in tests and assert delay bounds instead of a single sampled value. Then run an integration test that stops a worker after the external effect but before acknowledgement. That uncomfortable case is where duplicate behavior becomes visible.
Watch attempt count, job age, scheduled delay, terminal reason, and DLQ depth as separate signals. Averages blur the exact failure mode: a low average attempt count can coexist with a small cohort trapped at the cap. Trace one delivery key across the original job and every retry, but avoid putting subscriber contact details into queue names, metric labels, or logs. High-cardinality identifiers belong in traces or structured logs with suitable access controls.
Redrive needs a gate. First correct the terminal condition or update the payload under an auditable rule; next replay a small sample; then compare success rate, duplicate suppression, latency, and downstream throttling against the predeclared eval thresholds. Bulk replaying an untouched DLQ merely recreates the failure burst. AWS also recommends setting DLQ retention longer than source-queue retention, because message age and retention behavior affect how long operators have to investigate.
Keep the runbook short enough to use under pressure. It should state who owns each terminal reason, how to pause a redrive, what evidence permits replay, and how old a shipment update may be before it is discarded. Review retry constants beside product deadlines during deployment, verify that alarms cover job age as well as queue depth, and rehearse one rollback. Done. The queue library can change later; these observable contracts should not.
Top comments (0)