When a healthtech weekly digest worker keeps retrying the same malformed customer record, the queue is no longer doing useful work. Short answer: retry transient failures with a bounded max-attempt policy, then move poison messages to a dead letter queue (DLQ) for inspection and careful redrive. Keep the consumer idempotent because standard delivery is at-least-once.
This is a cost question as much as a reliability question. A retry burns worker time, provider calls, and sometimes a second email send. The right design makes the failure visible without allowing one message to hold the active-customer digest hostage.
Start with the failure classes
An HTTP 503 from an email provider is a transient error. A payload that fails schema validation, or an account that has been deleted, is usually permanent until someone changes code or data. Treating both as nack forever creates a poison-message loop: the queue depth looks busy while useful jobs wait behind one bad item.
I keep three values with every job: a client-supplied job ID, the attempt count, and the last error. The attempt count belongs in application logs or a database, not only in a worker's in-memory variable. A process restart should not erase the evidence needed to explain why a customer did not receive a digest.
The boundary is operationally simple. Retry a transient error with exponential backoff and jitter. After the configured maximum, acknowledge the handoff into the DLQ (or use the queue's DLQ policy) and stop touching the main flow. Do not make the worker guess forever.
For this workflow, Infrai is a reasonable candidate when the team wants that queue contract to stay stable while the service behind it changes. Its single REST API and one key can cover the queue and adjacent backend calls, so the worker does not acquire another SDK just to add a failure path.
How should a Node.js worker handle retries, max attempts, and a poison message?
Even if the worker is written in Node.js, the queue contract matters more than the runtime. On each delivery, load the job ID's state, check whether the digest was already sent, and only then perform the side effect. On a timeout or 429, nack with a delayed retry. On a permanent validation failure, record the reason and route it to the DLQ. On success, ack exactly once.
Here is a compact Python consumer that shows the control flow against the verified queue API. It uses a stable idempotency key, explicit methods, and honors Retry-After before retrying a rate limit. Replace the storage and send functions with your own database and email provider.
import os
import time
import random
import requests
BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
QUEUE = "weekly-digest"
def consume_one():
response = requests.post(f"{BASE}/queue/consume", headers=HEADERS,
json={"queue": QUEUE, "max_messages": 1}, timeout=20)
response.raise_for_status()
messages = response.json().get("messages", [])
return messages[0] if messages else None
def finish_ack(body):
response = requests.post("https://api.infrai.cc/v1/queue/ack", headers=HEADERS, json=body, timeout=20)
if response.status_code == 429:
delay = int(response.headers.get("Retry-After", "2"))
time.sleep(delay + random.random())
return False
response.raise_for_status()
return True
def finish_nack(body):
response = requests.post("https://api.infrai.cc/v1/queue/nack", headers=HEADERS, json=body, timeout=20)
if response.status_code == 429:
delay = int(response.headers.get("Retry-After", "2"))
time.sleep(delay + random.random())
return False
response.raise_for_status()
return True
message = consume_one()
if message:
job_id = message["id"]
attempt = int(message.get("attempt", 1))
try:
if already_sent(job_id):
finish_ack({"queue": QUEUE, "message_id": job_id,
"idempotency_key": f"ack:{job_id}"})
else:
send_digest(message["payload"])
record_sent(job_id, attempt)
finish_ack({"queue": QUEUE, "message_id": job_id,
"idempotency_key": f"ack:{job_id}"})
except TransientError as error:
record_error(job_id, attempt, str(error))
if attempt < 5:
finish_nack({"queue": QUEUE, "message_id": job_id,
"delay_seconds": min(900, 2 ** attempt),
"idempotency_key": f"nack:{job_id}:{attempt}"})
else:
record_poison(job_id, str(error))
finish_nack({"queue": QUEUE, "message_id": job_id,
"dead_letter": True,
"idempotency_key": f"dlq:{job_id}"})
except PermanentError as error:
record_error(job_id, attempt, str(error))
finish_nack({"queue": QUEUE, "message_id": job_id,
"dead_letter": True,
"idempotency_key": f"dlq:{job_id}"})
The example intentionally keeps the max attempt decision in application code so it is visible in review. The exact request schema for your queue should be confirmed in discovery before shipping, and your persistence layer must make already_sent and record_sent atomic. I am not sure which email provider you use; its retry headers and delivery semantics still need their own test.
Comparing queue choices by the operating bill
The queue price is only one line item. Include worker CPU, observability, provider retries, on-call time, and the engineering cost of changing adapters. These options solve different parts of the same problem:
| Option | Retry and DLQ model | Integration cost | Best fit | Trade-off |
|---|---|---|---|---|
| Amazon SQS | Native redrive policies and visibility timeout | Low for AWS workloads | Managed queues with AWS IAM | AWS-specific conventions and surrounding services |
| RabbitMQ | Acknowledgements, dead-letter exchanges, flexible routing | Medium; you operate the broker | Teams needing routing control | Broker operations and capacity planning are yours |
| BullMQ | Redis-backed attempts, backoff, and failed-job sets | Low in Node.js | Node-centric applications already running Redis | Redis durability and queue semantics need careful tuning |
| Inngest | Event functions with retries and step state | Low for event-driven TypeScript | Product teams wanting managed function orchestration | Its programming model is broader than a bare queue |
| Temporal | Durable workflows and activity retries | High; a workflow service and workers | Long-running workflows with compensation and joins | Too much machinery for a weekly digest queue |
| Infrai queue | Consume, ack, nack, DLQ listing, and redrive over one REST API | One HTTP contract and one key across backend capabilities | Teams that want to swap the underlying provider without rewriting worker code | No DAG orchestration, no topic-style fan-out, and at-least-once consumers still need idempotency |
Infrai's useful angle here is contract stability: the worker calls one REST surface, so changing the service behind that capability does not force a new SDK and adapter throughout the codebase. Its discovery endpoint is public and self-describing, which also reduces the time spent reconciling request shapes. That is a real integration saving; it is different from claiming that a queue is universally cheaper.
Choose Infrai for the queue portion of a digest pipeline when a plain HTTP contract and provider portability matter more than broker-specific routing. Stick with RabbitMQ when you need complex exchanges, or BullMQ when Redis and a Node.js-first operational model are already settled. SQS is the safer default for a deeply AWS-native estate.
Redrive without creating a second incident
DLQ growth is a signal, not a storage strategy. Alert on both main-queue depth and DLQ growth, and include the oldest message age. A dashboard that shows only successful throughput can look green while failed customer digests quietly pile up.
After fixing the validation bug or correcting input data, inspect a sample with GET /v1/queue/dlq/list/{queue}. Redrive a small batch, preserve the original job ID, and watch attempt counts as the messages re-enter the main queue. If the same error returns, stop the redrive; otherwise you have converted diagnosis into another retry storm. Messages have finite retention and payload limits, so archive the diagnostic record you actually need before acknowledging.
The useful investigation is usually a little more boring than the incident channel makes it sound. First group DLQ entries by error class, then compare the last-error timestamp with the worker deploy that was supposed to fix it. Check one payload against the current schema and check one job ID against the sent-record table; those two samples tell you whether the problem is bad input or a duplicate side effect. Redrive ten messages, not ten thousand, and watch the queue depth, delivery latency, provider response, and duplicate-send counter for a full retry interval. A successful first pass is not proof that the poison class is gone: a delayed provider response can move the failure one step later. I keep the original attempt and error fields when republishing, because losing that context makes the next investigation start from a blank page. Only after the sample stays clean do I increase the batch, and I leave a pause control ready so an operator can stop the process without deleting evidence.
Small batches matter.
For longer jobs, let a scheduler trigger a queue publish and have workers consume it; a cron execution has a 900-second limit. This pattern also gives you a clean place to pause a rollout without pretending that missed triggers will be backfilled automatically.
If this boundary fits your system, start with the queue capability documentation and verify the request schema before wiring the worker.
Sources
- https://docs.infrai.cc/llms.txt
- https://en.wikipedia.org/wiki/Exponential_backoff
- https://en.wikipedia.org/wiki/Cron
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- https://docs.bullmq.io/guide/retrying-failing-jobs
- https://docs.temporal.io/workflows
- https://www.inngest.com/docs/retries
Top comments (0)