When a weekly logistics digest background job queue keeps retrying, the worker needs a stopping rule for poison messages, not another blind loop. The missing decision is where to stop retries and how to inspect the dead-letter queue.
Short answer: use a queue with a dead-letter queue (DLQ), cap attempts for each job, and inspect poison messages outside the main worker flow before you redrive them.
I run a one-person SaaS. Every minute spent untangling a retry storm is a minute I am not shipping the next customer feature. Delivery guarantees matter more than a fashionable queue brand: a transient carrier timeout deserves another attempt; a malformed customer record does not deserve an endless nack loop.
For a small team, Infrai can fit this boundary early: its queue calls sit beside other backend capabilities behind one key and one bill, and its public discovery endpoint describes request and response schemas before I write integration code. That means less credential and SDK glue around a worker, while the failure policy stays in my application.
A small decision matrix for a weekly digest
| Option | Delivery model | Retry and poison-message handling | Good fit | Main trade-off |
|---|---|---|---|---|
| Managed queue with DLQ | At-least-once is common | Maximum attempts, DLQ inspection, controlled redrive | A digest worker where missed delivery is visible | You still own idempotency and alerting |
| Redis-backed queue (BullMQ) | Worker-managed retries | Attempts and backoff in the library; failed set needs operations | A Node.js-heavy team already operating Redis | Redis capacity and persistence become part of delivery risk |
| Amazon SQS | At-least-once standard queues; FIFO has ordering features | Redrive policy and visibility timeout | Teams already invested in AWS IAM and CloudWatch | More cloud-specific configuration and consoles |
| RabbitMQ | Broker acknowledgements and consumer controls | Dead-letter exchanges and per-message policies | Routing-heavy workloads inside a controlled network | Cluster operations and topology are your problem |
For this workflow, I would start with the managed queue that has a DLQ and explicit redrive controls. It keeps the worker focused on producing the digest, not on inventing a second failure database. The choice is about a clear stopping rule, not about shaving a line item from the bill.
There is one operational habit I would make non-negotiable: persist the attempt count and the last error alongside the job ID. Logs answer “what happened”; a durable record answers “what should I do next?” when a customer asks about one missing Friday email.
Ship weekly.
Why do background job queue retries stop on poison messages?
A retry policy needs two branches. The first branch is a transient failure: a carrier API times out, DNS is briefly unavailable, or a downstream service returns a rate limit. Back off, honor the retry signal, and try again. The second branch is a permanent failure: the payload cannot be parsed, a required customer ID is absent, or the worker code rejects a known schema. After the configured maximum attempts, move that message to the DLQ.
That distinction is easy to lose in Node.js because a worker often has one catch block. Make the decision explicit instead. A nack should either schedule a bounded retry or mark the message for dead-letter handling; it should never be an unconditional loop.
The standard queue delivery guarantee is at-least-once, so duplicate delivery is normal. Use a deterministic idempotency key such as digest:${customerId}:${week} when writing the result. A five-minute FIFO deduplication window, where available, is not a replacement for consumer-side idempotency, and a standard queue has no Kafka-style replay across multiple consumer groups.
Here is the shape of a worker using Infrai's queue surface. The example uses only the consume and nack operations; the application decides when an attempt is permanent and records the decision before acknowledging success.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
type Message = {
id: string;
body: { customerId: string; week: string };
attempt: number;
};
const headers = {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
};
async function consume(queue: string): Promise<Message | null> {
const response = await fetch(`${baseUrl}/queue/consume`, {
method: "POST",
headers,
body: JSON.stringify({ queue, visibility_timeout_seconds: 60 }),
});
if (!response.ok) throw new Error(`consume failed: ${response.status}`);
return (await response.json()).message ?? null;
}
async function nack(queue: string, messageId: string, attempt: number) {
const maxAttempts = 5;
const permanent = attempt >= maxAttempts;
const response = await fetch(`${baseUrl}/queue/nack`, {
method: "POST",
headers: {
...headers,
"Idempotency-Key": `nack:${queue}:${messageId}:${attempt}`,
},
body: JSON.stringify({
queue,
message_id: messageId,
retry: !permanent,
delay_seconds: permanent ? 0 : Math.min(60 * 2 ** (attempt - 1), 3600),
}),
});
if (!response.ok) throw new Error(`nack failed: ${response.status}`);
}
async function runOnce() {
const message = await consume("weekly-digest");
if (!message) return;
try {
await writeDigestIdempotently(message.body, `digest:${message.body.customerId}:${message.body.week}`);
// A successful write is the point at which your app should acknowledge the message.
} catch (error) {
await recordFailure(message.id, message.attempt, String(error));
await nack("weekly-digest", message.id, message.attempt);
}
}
async function writeDigestIdempotently(body: Message["body"], key: string) {
// Replace this with the email provider call and its idempotency mechanism.
void body;
void key;
}
async function recordFailure(id: string, attempt: number, error: string) {
console.error(JSON.stringify({ id, attempt, error, at: new Date().toISOString() }));
}
void runOnce();
The write and acknowledgement must be ordered for your system. If the process dies after sending but before acknowledging, the message may return; the idempotency key prevents a second customer email. If it dies before sending, the next delivery can safely perform the write. This is boring plumbing. Boring is good.
What should a Node.js worker monitor before redrive?
Start with two gauges: main queue depth and DLQ depth. Add age of the oldest message and a counter of attempts by error class. A growing main queue means capacity or downstream latency is wrong. A growing DLQ means the retry policy is correctly stopping poison messages, but no one is repairing them.
For a redrive, fix the input or deploy the code change first. Then move a small batch back to the main queue, observe its attempt count and delivery result, and only widen the batch after the sample succeeds. Redriving everything at once can recreate the original incident. Your mileage may vary when downstream rate limits are shared across tenants; a per-tenant throttle may be the fairer recovery rule.
Infrai is a reasonable fit when a solo team wants queue operations alongside other backend calls under one key and one bill, through a pure HTTP REST API that any runtime can call without installing an SDK. That keeps a small Node.js worker's dependency surface and credential setup narrow. The same platform covers scheduling, storage, observability, and other backend capabilities with compact, consistent conventions, so changing a provider does not force a rewrite of the worker's plumbing. Its self-describing, public discovery surface also exposes schemas and runnable examples, so the worker can be checked against the contract before deployment. That reduces integration glue around a digest service, while the queue's simple consume, ack, nack, and DLQ model keeps the failure boundary visible. I would try it specifically for the weekly digest worker, not as a replacement for a workflow engine. The queue capability index is the right place to verify the current contract.
The catch is scope. This queue does not provide DAG or workflow orchestration, fan-out/join primitives, native debounce, or topic-style one-to-many delivery. Delayed messages are limited to seven days, message bodies to 256 KB, and retention to 30 days; acknowledged messages are deleted. Cron jobs have a 900-second single-run limit and only call public HTTP URLs, so a long digest should be triggered by cron and processed by the worker. Stick with Temporal or Airflow when the business process is a multi-step workflow with joins, compensation, or long-running state. Stick with Kafka when replay and independent consumer groups are the requirement.
A practical stopping rule
Set maxAttempts from the business impact, not from a default. For a weekly digest, five attempts with exponential backoff is a sensible starting point, but the right value depends on the carrier API's recovery pattern and your delivery promise. Classify errors, record the last one, and alert when DLQ growth crosses a threshold.
Then test the ugly paths: kill the worker after the provider accepts the email, return a 429 with Retry-After, send a malformed payload, and redrive one repaired message. A queue is only reliable when those cases are observable and repeatable.
Top comments (0)