Pick the queue you can drain, not the one with the nicest publish API. Fanout semantics, client ergonomics, the dashboard — those get compared to death in every shortlist thread, and none of them decide how your Tuesday goes. What decides your Tuesday is whether you can look inside a dead letter queue, tell poison messages apart from collateral damage, and push a few thousand failed jobs back through the system at a rate your database survives.
Recovery is the axis. Everything else is taste.
The system I keep coming back to when I explain this is an edtech backend: student uploads, lesson recordings, and a retention policy that says derived artifacts for archived courses disappear after 30 days. That cleanup used to run inside an HTTP handler behind an admin button, which holds up until one course has 90k artifacts and the request dies at the proxy's 60-second limit — halfway through, with no record of where it stopped. Moving it off the request path is the easy half of the fix. The interesting half is what the queue does when the sweep goes wrong at 02:00 and nobody is watching.
The sweep, in one pass
The shape is boring on purpose. A scheduled trigger (a cron entry, a scheduler service, whatever your platform already runs) publishes one message per course into a work queue instead of doing the deletion itself. Workers pull a message, delete one course's expired artifacts inside a transaction, and acknowledge. A failure re-publishes the same unit of work into a delay queue with an attempt counter; once the counter crosses the ceiling, the message lands in the dead letter queue with enough metadata to explain itself later. Nothing in that flow holds a web request open, and the unit of work stays small enough that a partial failure costs one course, not one night.
Here's the consumer, trimmed to the parts that matter:
import random
from dataclasses import dataclass
MAX_ATTEMPTS = 5
BASE_DELAY = 2.0 # seconds
@dataclass
class Job:
key: str # "purge:course-4812:2026-03-01" — derived, not random
body: dict
attempt: int # carried in a header; the protocol only tells you "redelivered"
def backoff(attempt: int) -> float:
ceiling = min(300.0, BASE_DELAY * 2 ** (attempt - 1))
return ceiling * (0.5 + random.random() / 2) # full-ish jitter
def handle(job: Job, db) -> None:
with db.transaction():
if not db.claim(job.key): # INSERT ... ON CONFLICT DO NOTHING
return # already done by an earlier delivery
db.execute(
"DELETE FROM artifact WHERE course_id = %s AND expires_at < %s",
(job.body["course_id"], job.body["cutoff"]),
)
def consume(broker, db) -> None:
for job in broker.receive(queue="cleanup", batch=50):
try:
handle(job, db)
broker.ack(job)
except RateLimited as exc: # upstream answered 429
broker.retry(job, delay=exc.retry_after or backoff(job.attempt))
except Transient:
if job.attempt >= MAX_ATTEMPTS:
broker.dead_letter(job, reason="max_attempts")
else:
broker.retry(job, delay=backoff(job.attempt))
except Permanent as exc:
broker.dead_letter(job, reason=repr(exc))
Two details in there are worth more than the rest. broker.retry re-publishes with a delay rather than sleeping in the worker, because a sleeping consumer still owns its prefetch slot and starves everyone behind it. And job.attempt is something you write into a header yourself: the redelivery flag most brokers hand you is a boolean, so it tells you a message came back but not how many times, and a retry policy needs the count.
How many delayed retries should a failed job get before the dead letter queue?
Fewer than you think, and the number should come from the error class rather than a global constant. Retrying is only useful against failures that heal on their own — a lock timeout, a connection reset, a rate limit. Against a malformed payload or a foreign key that will never exist, five attempts with exponential backoff just means the same exception five times, twenty minutes apart, in five different log lines.
So classify at the raise site and let the classification drive the routing:
| Failure | What it usually is | Where the message goes |
|---|---|---|
429 with Retry-After
|
someone else's budget, not your bug | delayed retry, honoring the header |
| Connection reset, lock timeout | transient contention | delayed retry with jitter |
| Validation error, missing tenant | poison message | dead letter immediately |
| Unknown exception | you don't know yet | two attempts, then dead letter |
The 429 row deserves the extra branch. A rate-limited response can carry a Retry-After value, in seconds or as an HTTP date, and honoring it is strictly better than guessing with your own backoff curve, because the server just told you when it will be ready. Ignore it and your retry storm arrives early and gets rate limited again.
At-least-once delivery is the contract underneath all of this, and it is the honest one to design against. A consumer acknowledgement can be lost after the work committed; a worker can be killed between the delete and the ack; a broker that requeues unacknowledged messages when a channel closes will hand the same job to somebody else. Duplicates are not an edge case in that model. They're the normal operating condition, and the cost of pretending otherwise gets paid at 02:00.
Making duplicate delivery boring
Idempotency is the whole reason at-least-once is tolerable, and the trick is to derive the key from the work instead of generating one. purge:course-4812:2026-03-01 is reproducible by any worker that receives the same message, on the first delivery or the fortieth, which is what makes a claim row with a unique constraint sufficient. A UUID minted by the publisher gives you a different key per publish and quietly buys you nothing on replay.
Cleanup work has a second advantage: deletion is set-shaped, so running it twice converges on the same state. Not every job in your system is that lucky. Sending a parent a "your child's recording was removed" email is not idempotent unless you gate it on a claim row, and incrementing a usage counter is worse — replays inflate it silently, and you'll find out from a billing complaint.
That asymmetry is worth a design rule. Any job that touches money, messaging, or counters gets an explicit claim row before it does anything else; jobs that only converge state can lean on the database. If your worker calls a model API, checkpoint the expensive step. Re-embedding a course catalog on every retry burns real tokens, and a job that retries six times because a downstream write keeps failing pays for that pass six times over. Store the vectors, key them by content hash, and a retry becomes a lookup.
What the drain actually costs
Now the part that made me write this. A dead letter queue is a holding area, not a solution, and draining one is its own workload with its own failure mode: the naive replay hands 40,000 accumulated deletes to a database that already fell over once, at full speed, and the outage repeats with better logging. Replay is a rate-limited operation.
import time
def drain(broker, rate_per_second: float = 20.0, limit: int | None = None) -> int:
interval, sent = 1.0 / rate_per_second, 0
for job in broker.receive(queue="cleanup.dlq", batch=100):
if job.headers.get("reason") == "max_attempts":
broker.publish("cleanup", job.body,
headers={"replay_of": job.key, "attempt": 1})
broker.ack(job)
sent += 1
time.sleep(interval)
else:
broker.park(job) # poison: keep it for a human
if limit and sent >= limit:
break
return sent
Twenty per second is not a benchmark, it's a starting point you tune against the slowest thing the job touches — I'm not sure there's a defensible universal number here, since the ceiling is whatever your storage layer tolerates while still serving traffic. What matters is that the throttle exists, that the replay is bounded by limit so you can drain 500 messages and check the blast radius before committing to the rest, and that poison messages get parked instead of looped. A dead letter queue holding two populations — retryable-later and never-going-to-work — needs the sort before the replay, or every drain cycle re-tests the same broken payloads.
Watch the age, not the depth. Queue depth spikes during normal bursts and tells you very little; the age of the oldest unacknowledged message and the arrival rate into the dead letter queue are the two numbers that actually correlate with "something is wrong right now". A cleanup queue that is 12,000 deep and moving is healthy. One that is 40 deep with a message from nine hours ago is stuck, and only one of those pages you if you alert on depth.
Rehearse the drain like an eval harness
Anything you only exercise during an incident is a plan, not a capability. I run recovery the way I run evals for a RAG pipeline: on a schedule, against fixtures, with a number at the end. Inject 500 synthetic failures into staging, dead-letter them, run the drain, and record three things — wall-clock time to empty, duplicate executions observed, and how many messages needed a human. Do it monthly. First runs tend to be embarrassing, usually because the drain tool doesn't exist yet or the metadata needed to classify a message was never written into the headers.
The catch is that this whole architecture is overkill for a lot of teams, and I'd rather say so than sell you a queue. If your periodic cleanup is one SQL statement that finishes in eight seconds, a scheduled task with an advisory lock and an alert on failure is the better engineering decision — you get the same "not in a web request" property without a broker to operate. Queues earn their keep when the work partitions into independent units that can fail separately, when the tail is long enough that a retry needs to be delayed rather than immediate, or when you need the dead letter queue as an audit trail of what didn't happen. Strict global ordering isn't a good fit for this pattern either: the moment you add delayed retries, messages finish out of order by design, and a system that needs sequence guarantees should stick with a partitioned log where ordering is part of the contract.
For a Node.js SaaS backend the same reasoning holds, and a split stack is common — a JavaScript API tier publishing messages, Python workers consuming them, which is fine as long as both sides agree on the message envelope and the idempotency key format. Pin that envelope in a schema both services import; a queue where the producer and consumer disagree about field names produces the least debuggable class of failure I know.
If you want a checklist to carry into your next design review, it's short. Derive idempotency keys from the work, not from the publisher. Carry the attempt count yourself and let the error class pick the retry policy, honoring Retry-After when the server sends one. Delay retries by re-publishing, never by sleeping in the consumer. Write the failure reason, the first failure timestamp, and the attempt count into the dead-lettered message, because those three fields are the difference between a five-minute drain and an archaeology project. Then rehearse it before you need it.
References
- RabbitMQ, Consumer Acknowledgements and Publisher Confirms — https://www.rabbitmq.com/docs/confirms
- MDN Web Docs, HTTP 429 Too Many Requests — https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Top comments (0)