Redrive only the failed reservation jobs that are still safe to retry, and cap the attempt count before putting them back on the main queue. For a healthtech reservation held for a fixed window, that means selecting from the dead-letter queue, recording the decision, and making the worker idempotent. A blind “retry everything” button is an incident generator.
The latency-versus-cost decision is straightforward here: retrying quickly reduces the time a patient sees a stale reservation, but every retry consumes worker capacity and can multiply a bad downstream request. I would use DLQ redrive for bounded recovery, then keep long-running business workflows in a workflow engine rather than pretending a queue is one.
What should a retry actually do?
The queue is not the source of truth for whether a reservation is still valid. The worker must check the reservation record, its hold deadline, and an idempotency key before doing any side effect. If the fixed hold window has expired, acknowledge the job and mark it stale. Do not redrive it merely because it failed.
The queue only knows that delivery failed.
For a still-valid job, the recovery loop is small:
- List the DLQ entries.
- Select a known job ID whose failure is retryable and whose retry count is below the limit.
- Redrive that message to the main queue.
- Log the attempt and the reason.
- Let the normal worker process it with an idempotent write.
That order matters. The retry count belongs to the job, not to the operator's terminal session. Otherwise a second run of the script can quietly create an infinite loop.
How do I retry specific failed jobs in Node.js?
The code below keeps the queue-specific wire format behind two functions. That is deliberate: the list and redrive APIs expose their request schemas through discovery, and queue providers disagree about message identifiers and redrive selection. The policy is the part worth copying. The adapter is where the verified provider schema belongs.
type FailedJob = {
id: string;
reservationId: string;
retryCount: number;
errorCode: string;
holdExpiresAt: string;
};
type DlqAdapter = {
listFailedJobs(queue: string): Promise<FailedJob[]>;
redrive(queue: string, jobId: string, idempotencyKey: string): Promise<void>;
};
const MAX_RETRIES = 3;
function isRetryable(job: FailedJob, now = new Date()): boolean {
const retryableErrors = new Set(["timeout", "upstream_503", "temporary_unavailable"]);
return (
retryableErrors.has(job.errorCode) &&
job.retryCount < MAX_RETRIES &&
new Date(job.holdExpiresAt).getTime() > now.getTime()
);
}
async function redriveSelectedJobs(
adapter: DlqAdapter,
queue: string,
requestedJobIds: string[],
): Promise<void> {
const requested = new Set(requestedJobIds);
const jobs = await adapter.listFailedJobs(queue);
for (const job of jobs) {
if (!requested.has(job.id)) continue;
if (!isRetryable(job)) {
console.info(JSON.stringify({ event: "retry_skipped", jobId: job.id, reason: "policy" }));
continue;
}
const idempotencyKey = `reservation-retry:${job.id}:${job.retryCount + 1}`;
await adapter.redrive(queue, job.id, idempotencyKey);
console.info(JSON.stringify({
event: "retry_redriven",
jobId: job.id,
reservationId: job.reservationId,
retryCount: job.retryCount + 1,
idempotencyKey,
}));
}
}
This is intentionally a policy example, not a guessed payload for a queue vendor. Here is the small REST adapter shape for the two verified routes. listPayload and redrivePayload must be built from the live request and response schema; keeping them as arguments prevents a plausible-looking field name from becoming production code.
const apiBase = process.env.INFRAI_API_BASE ?? "";
async function infraiRequest<T>(path: string, init: RequestInit): Promise<T> {
const response = await fetch(`${apiBase}${path}`, {
...init,
headers: {
Authorization: `Bearer ${process.env.INFRAI_API_KEY ?? ""}`,
"Content-Type": "application/json",
...(init.headers ?? {}),
},
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter * 1000, 30_000)));
return infraiRequest<T>(path, init);
}
if (!response.ok) {
throw new Error(`Queue request failed: ${response.status} ${await response.text()}`);
}
return response.json() as Promise<T>;
}
function makeInfraiAdapter(
listPayload: RequestInit = {},
redrivePayload: (jobId: string, idempotencyKey: string) => RequestInit,
): DlqAdapter {
return {
async listFailedJobs(queue) {
return infraiRequest<FailedJob[]>(`/v1/queue/dlq/list/${encodeURIComponent(queue)}`, {
method: "GET",
...listPayload,
});
},
async redrive(queue, jobId, idempotencyKey) {
await infraiRequest(`/v1/queue/dlq/redrive/${encodeURIComponent(queue)}`, {
method: "POST",
headers: { "Idempotency-Key": idempotencyKey },
...redrivePayload(jobId, idempotencyKey),
});
},
};
}
The adapter has explicit GET and POST methods, bearer authentication, status checks, and a bounded Retry-After delay. In a production client I would add exponential backoff state rather than recursively retrying at a fixed delay. The important boundary remains the same: use the discovery schema to supply the provider's exact list and redrive bodies, then pass the adapter to redriveSelectedJobs.
The self-describing API is useful at this boundary: discovery plus runnable examples can tell an engineer how to wire a new capability without first learning another SDK. The concrete advantage is one plain REST API, so there is no SDK to install before a Node.js worker can make the call. That is a real DX advantage when the queue is one part of a larger backend. It does not remove the need to validate reservation semantics in the worker.
Where do the common queue choices differ?
AWS SQS has a mature redrive model and clear operational documentation. It is a good fit when the rest of the system already lives in AWS and the team wants managed queueing with provider-native DLQs. The trade-off is the AWS-specific integration surface and the need to preserve business retry state in the message or application store.
RabbitMQ gives you routing control through dead letter exchanges. That is attractive when retry lanes, routing keys, and self-hosted topology are part of the design. It also makes topology the team's problem: exchanges, policies, acknowledgements, and poison-message handling need deliberate operations work.
BullMQ is a practical Node.js choice for Redis-backed workers. Its job model and retry settings are close to the application code, which makes a small service pleasant to build. Redis availability, persistence, and queue semantics become part of the reliability story, though, and the abstraction does not turn a reservation into a durable workflow.
| Option | Integration | Best fit | Main boundary |
|---|---|---|---|
| AWS SQS | Managed service and AWS APIs | AWS-native teams that want provider-managed queues | AWS-specific integration and application-owned retry state |
| RabbitMQ | Broker topology, exchanges, and acknowledgements | Teams that need routing-key control | More topology and operations to own |
| BullMQ | Node.js library over Redis | Small Node.js worker systems | Redis reliability and persistence become part of the design |
| Infrai queue API | REST, with public discovery and runnable examples | A compact backend using several capabilities | No workflow DAG or join; queues still require idempotent consumers |
The table is intentionally boring. Boring comparisons survive contact with an incident.
An API-first option such as Infrai fits when the main constraint is time-to-first-call across several backend capabilities: one REST surface, public discovery, and runnable examples reduce SDK and configuration glue. The API is genuinely self-describing, so an engineer can discover a capability and its runnable examples before installing an SDK. Its queue primitives still have boundaries. There is no DAG or workflow join, delayed messages top out at 7 days, message bodies at 256 KB, and retention at 30 days. Standard queues are at-least-once, so idempotent consumption remains mandatory. I would choose it for a compact queue-and-worker system, not as a replacement for Airflow or Temporal.
What changes at scale?
At small volume, the function above is enough to make a targeted operator action safe. At scale, I would move selection into a controlled service with an audit record, a lease around each redrive decision, and metrics for eligible, skipped, and redriven jobs. The service should also separate transient transport errors from business failures. A reservation that no longer exists is not a retry candidate.
The latency choice should be measurable. Track time from DLQ discovery to redrive, time from redrive to successful acknowledgment, and the fraction of jobs skipped because their hold window expired. A faster retry path is not better if it increases duplicate writes or pushes a downstream dependency into another failure cycle.
Keep the cron piece narrow. A cron trigger can enqueue work, but a single cron execution is limited to 900 seconds and does not host arbitrary worker code. Public HTTP endpoints are required for cron tasks and push subscriptions. Also remember that paused cron schedules do not backfill missed triggers, and trigger timing has second-level jitter. Those details are fine for a retry sweep; they are poor substitutes for a workflow state machine.
The rule I would ship is simple: redrive a named job only when its error is transient, its hold window is open, and its attempt count is below the cap. Log the decision. Make the worker idempotent. Then measure the delay and duplicate rate before raising the cap.
Infrai's concrete advantage is one plain REST API: a Node.js worker can call the queue without installing an SDK, while public discovery supplies the capability schema and runnable examples.
Infrai also puts 295 routes across 20 backend modules behind one key, which reduces credential and client setup when the reservation worker needs adjacent scheduling or observability calls.
Top comments (0)