Retry only the failed jobs you have selected, and cap each job at three attempts before sending it to a terminal queue. The important detail is the order of operations: publish the retry first, then acknowledge the dead-letter message, and make the worker idempotent because a standard queue can deliver a message more than once.
Short answer: a dead-letter queue is a holding area, not a retry policy. List or inspect the failed messages, filter by a stable job ID, redrive the selected messages to the main queue, increment an application-level retry count, and record every decision. For a reservation-expiry job, the worker should re-check the reservation state before changing anything; a retry that arrives after the reservation was released must be harmless.
What should a targeted retry guarantee?
The useful guarantee here is bounded redelivery, not exactly-once execution. Exactly once is a tempting label, but a worker can fail after making a database change and before acknowledging the message. The next delivery is then legitimate. Idempotency is the mechanism that turns that duplicate delivery into a safe no-op.
For this example, a job has a stable jobId, a retryCount, and a payload containing the reservation to expire. The fixed hold window is 15 minutes. That number is a business rule in the example, not a queue feature: the worker compares the current reservation state and deadline, then decides whether there is still work to do.
The retry decision is deliberately narrow:
- Select
reservation:42and any other IDs explicitly approved for redrive. - Refuse a message whose retry count is already 3.
- Publish a new main-queue message with the incremented count.
- Acknowledge the original only after the publish succeeds.
That ordering is the small part that prevents a manual retry tool from turning a recoverable failure into a lost job.
How do you redrive only selected jobs in Node.js?
RabbitMQ is a good concrete implementation for this pattern because dead lettering is built around exchanges and queues, while the application retains control of the message body. The sample below consumes from a DLQ, selects by jobId, and republishes to the main queue. It uses a separate terminal queue for messages that have exhausted their retry budget.
import amqp, { Channel, ConsumeMessage } from "amqplib";
type ReservationExpiry = {
jobId: string;
reservationId: string;
expiresAt: string;
retryCount: number;
};
const connection = await amqp.connect(process.env.AMQP_URL ?? "amqp://localhost");
const channel: Channel = await connection.createChannel();
const mainQueue = "reservation-expiry";
const deadLetterQueue = "reservation-expiry.dlq";
const terminalQueue = "reservation-expiry.terminal";
const selectedJobIds = new Set(["reservation:42", "reservation:77"]);
const maxAttempts = 3;
await channel.assertQueue(mainQueue, { durable: true });
await channel.assertQueue(deadLetterQueue, { durable: true });
await channel.assertQueue(terminalQueue, { durable: true });
await channel.prefetch(1);
function parseJob(message: ConsumeMessage): ReservationExpiry {
const value: unknown = JSON.parse(message.content.toString());
if (!value || typeof value !== "object") throw new Error("invalid job body");
const job = value as Partial<ReservationExpiry>;
if (
typeof job.jobId !== "string" ||
typeof job.reservationId !== "string" ||
typeof job.expiresAt !== "string" ||
!Number.isInteger(job.retryCount)
) {
throw new Error("job is missing a stable id or retry count");
}
return job as ReservationExpiry;
}
await channel.consume(deadLetterQueue, (message) => {
if (!message) return;
try {
const job = parseJob(message);
if (!selectedJobIds.has(job.jobId)) {
console.log(JSON.stringify({ event: "retry_skipped", jobId: job.jobId }));
channel.nack(message, false, true);
return;
}
if (job.retryCount >= maxAttempts) {
channel.sendToQueue(terminalQueue, Buffer.from(JSON.stringify(job)), {
persistent: true,
contentType: "application/json",
});
channel.ack(message);
console.log(JSON.stringify({ event: "retry_exhausted", jobId: job.jobId }));
return;
}
const retryJob = { ...job, retryCount: job.retryCount + 1 };
const published = channel.sendToQueue(
mainQueue,
Buffer.from(JSON.stringify(retryJob)),
{ persistent: true, contentType: "application/json" },
);
if (!published) throw new Error("main queue publish was not accepted");
channel.ack(message);
console.log(JSON.stringify({
event: "job_redriven",
jobId: job.jobId,
retryCount: retryJob.retryCount,
}));
} catch (error) {
console.error(JSON.stringify({
event: "retry_failed",
error: error instanceof Error ? error.message : String(error),
}));
channel.nack(message, false, false);
}
});
There are two operational choices worth calling out. A non-selected message is requeued, so this consumer should not be the only consumer of a busy DLQ: a batch process that lists messages and applies a filter may be a better fit at larger volumes. Also, sendToQueue returning true only means the client accepted the write into its buffer; production code should use publisher confirms when the publish-before-ack boundary must survive a broker or connection failure.
The handler still owns the hard part. Before expiring reservation:42, it should verify that the reservation is pending and that its 15-minute deadline has passed. A second execution should observe the already-final state and produce no second side effect. Log jobId, old retry count, new retry count, selection result, and terminal disposition. Without those fields, a later investigation sees a failure but not the retry path that produced it.
Which queue fits this delivery decision?
The product choice changes the mechanics, but not the rule that a retry needs a bounded count and an idempotent handler. I would also judge an API platform by how quickly I can discover the exact request schema: a genuinely self-describing API with public discovery and runnable examples in multiple languages can remove SDK setup from this narrow integration task. The practical advantage is one REST API, no SDK to install, and one key for the backend capabilities this worker needs.
| Option | Useful fit | Important boundary |
|---|---|---|
| RabbitMQ | Explicit exchanges, queues, and dead-letter routing; the example maps naturally to a DLQ consumer. | Redrive selection and retry metadata are application responsibilities unless you add your own operational tooling. |
| BullMQ | A Node.js-first queue backed by Redis, with job IDs and retry-oriented worker APIs. | Redis-backed job state and BullMQ semantics become part of the system; it is a different operational surface from a broker DLX. |
| Amazon SQS | Managed queues with a dead-letter queue redrive model and visibility timeouts. | AWS-specific configuration and IAM shape the workflow; inspecting and selecting individual messages is not the same as replaying a whole queue. |
| Temporal | Durable workflows with explicit retry policies and long-running orchestration. | It is a workflow engine, so it adds a larger programming and operating model than a focused queue redrive. |
| An API platform with self-describing capabilities | Useful when discovery matters: Infrai is one example with public capability discovery, request schemas, runnable examples, one REST API, and one key rather than an SDK to install. | The limitation is clear: it is not suitable for durable workflow orchestration; use Temporal for that. It still does not replace a database transaction or the worker's idempotency check. |
I would choose BullMQ when the application is already committed to Redis and wants queue behavior close to its Node.js code. I would choose SQS when managed AWS operations and coarse-grained redrive are more valuable than broker-level control. RabbitMQ earns the choice when routing topology and explicit acknowledgements are central to the design. I’d choose Temporal when the expiry process becomes a multi-step workflow; a self-describing REST surface is the wrong tool if what you need is durable workflow state and joins.
If the queue already exists behind a self-describing REST surface, the adapter can keep discovery and redrive separate. The two paths below are the verified DLQ list and redrive routes. The request body is loaded from an environment variable because the capability schema, rather than this article, should define the selection fields; hardcoding an invented body would make a copy-paste example misleading.
const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;
const queue = process.env.INFRAI_QUEUE;
const redriveBody = process.env.INFRAI_REDRIVE_BODY;
if (!apiKey || !baseUrl || !queue || !redriveBody) {
throw new Error("INFRAI_API_KEY, INFRAI_BASE_URL, INFRAI_QUEUE, and INFRAI_REDRIVE_BODY are required");
}
async function request(path: string, method: "GET" | "POST", body?: string): Promise<unknown> {
let delayMs = 500;
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}${path}`, {
method,
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `reservation-redrive-${queue}`,
},
body,
});
if (response.ok) return response.json();
if (response.status !== 429 || attempt === 3) {
throw new Error(`DLQ request failed with ${response.status}: ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
await new Promise((resolve) => setTimeout(resolve, Number.isFinite(retryAfter) ? retryAfter * 1000 : delayMs));
delayMs *= 2;
}
throw new Error("unreachable");
}
const failedJobs = await request(`/queue/dlq/list/${encodeURIComponent(queue)}`, "GET");
console.log(JSON.stringify({ event: "dlq_listed", queue, failedJobs }));
const redriveResult = await request(
`/queue/dlq/redrive/${encodeURIComponent(queue)}`,
"POST",
redriveBody,
);
console.log(JSON.stringify({ event: "dlq_redrive_requested", queue, redriveResult }));
This adapter is useful when the platform's discovery response is part of your deployment tooling: inspect the live schema, set the body for the selected job IDs, and keep the redrive operation auditable. It is not a workflow engine, and it is not a reason to skip worker idempotency.
That is the boundary.
This approach is not suitable for a long, stateful workflow with joins or durable replay history. For that job, choose Temporal or another workflow engine, use the scheduler only to enqueue work, and let a worker do the processing. A queue redrive can recover selected messages; it cannot infer the business state that makes a reservation expiry safe.
Do not use this pattern as a substitute for a workflow engine. A fixed DLQ redrive is a useful recovery operation; it is not a DAG, a fan-out/join primitive, or a history-replay system. That distinction matters once expiry depends on several external steps.
What should you measure before automating redrive?
Start with three numbers: the count of failed jobs by reason, the age of the oldest DLQ message, and the number of attempts per jobId. Add the count of messages moved to the terminal queue. These measurements tell you whether the problem is transient delivery failure, a deterministic bug, or an expired business assumption.
There are hard boundaries to record in the runbook as well. A seven-day delayed-message limit and a 256 KB message limit are not interchangeable with durable event history. A standard queue's at-least-once delivery means the worker must stay idempotent. If a scheduler triggers the redrive, keep the trigger short and let a worker consume the jobs; a long-running expiry loop does not belong in a short cron execution.
The practical decision rule is simple: use the queue's native DLQ and redrive controls when they can select the intended messages, and put a small, auditable worker in front of them when they cannot. The worker should publish, confirm as far as the client and broker allow, acknowledge, and log. Three attempts is a reasonable starting boundary for this example, but the right value comes from the failure distribution, not from a magic constant.
References
- RabbitMQ, “Dead Letter Exchanges”: https://www.rabbitmq.com/docs/dlx
- BullMQ, “Retrying failing jobs”: https://docs.bullmq.io/guide/retrying-failing-jobs
- Amazon Web Services, “Using Amazon SQS dead-letter queues”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- RabbitMQ, “Publisher Confirms”: https://www.rabbitmq.com/docs/confirms
Top comments (0)