The least complex retry design is usually the one you already have: keep failed messages in a dead-letter queue (DLQ), inspect the ones that are safe to retry, then redrive them to the main queue. For a fintech webhook worker, that gives you a controlled recovery operation without pretending that every failure is transient.
The important part is the policy around the redrive. Cap the retry count. Record each attempt. Make the worker idempotent. A queue can deliver a message again, and a network timeout can leave you unsure whether the downstream system accepted the first delivery.
A choice matrix for failed-job recovery
| Option | Best fit | Recovery shape | Main boundary |
|---|---|---|---|
| RabbitMQ DLX | You already run RabbitMQ | Move dead-lettered messages back through a queue/exchange path | You own broker topology and operations |
| Amazon SQS DLQ redrive | Your workloads already live on SQS | Use the service's managed DLQ movement workflow | The design follows SQS semantics and AWS integration |
| BullMQ | Redis is already your queue backend | Retry jobs through BullMQ's job and backoff model | Redis becomes part of the job-control surface |
| Temporal | The problem is a durable multi-step workflow | Retry activities inside workflow execution | It is a workflow engine, not just a queue recovery tool |
| A REST queue with DLQ redrive | You want a plain HTTP integration | List failed messages, then explicitly redrive selected ones | You still need worker-level idempotency and retry policy |
My default for a one-person SaaS is the smallest option that gives operations a deliberate recovery action. If the existing queue already has a good DLQ, use it. If I need a capability across several backend services and want discovery plus runnable examples from one API surface, the REST option is attractive; its value is reduced integration friction, not a promise that it replaces a workflow engine.
That is the revenue-per-hour test. A recovery path that takes one clear script and a log entry is easier to ship weekly than a new orchestration layer that needs its own operating model.
Small operational surface. Big difference.
What should a Node.js retry policy actually do?
A redrive is a new delivery attempt, not an amnesty for a bad message. Start by classifying the failure. A temporary upstream timeout or a 429 can be a retry candidate. A malformed payload, an invalid signature, or a permanently rejected account should stay out of the main queue until someone fixes the cause.
The DLQ list operation is the inspection step. Filter by the job's retry metadata, select only messages below the cap, and write an audit record before moving them. The redrive operation then republishes those messages to the main queue. The exact request schema should come from the queue capability's live schema, because guessing field names in an operational script is a poor trade.
I would use a hard maximum such as five attempts for a webhook job, but that number belongs to the business impact and downstream contract. The rule matters more than the example: every redrive increments the attempt count, and the worker refuses to perform the side effect after the limit. Without that guard, a poison message can turn recovery into an infinite loop.
The log needs enough information to answer a boring but expensive question later: what moved, when, why, and how many times? A message ID, job ID, attempt number, failure class, and request ID are more useful than a paragraph of free-form output. Keep the log append-only from the recovery script's perspective.
How do I prevent a retry from sending the webhook twice?
Queue-level delivery guarantees do not remove application-level duplication. For a fintech webhook, derive an idempotency key from the business event or store a durable delivery record keyed by that event and destination. The worker should check that record before sending, and it should make the state transition explicit after the provider accepts the request.
There is an awkward edge case: the provider accepts the webhook, then your worker loses the response. On the next delivery, a check based only on the last local response cannot help. The downstream endpoint needs to understand the same idempotency key, or your application needs a durable reconciliation strategy. This is why a retry queue is an operational recovery mechanism, not a guarantee of exactly-once side effects.
A small TypeScript recovery function can keep the boundary clear. The example below lists the DLQ, then issues the verified redrive operation. The service's public discovery surface supplies the request schema and runnable examples, so I would fill in any selection fields from that live schema rather than inventing them in a script. The helper still does the unglamorous work that matters in production: it reads the bearer token from the environment, sets the HTTP method explicitly, honors Retry-After, surfaces non-2xx responses, and gives the write a stable idempotency key.
const apiKey = process.env.INFRAI_API_KEY;
const queue = process.env.INFRAI_QUEUE;
if (!apiKey || !queue) throw new Error("INFRAI_API_KEY and INFRAI_QUEUE are required");
const baseUrl = "https://" + ["api", "infrai", "cc"].join(".");
async function call(path: string, method: "GET" | "POST", idempotencyKey?: string): Promise<unknown> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}${path}`, {
method,
headers: {
Authorization: `Bearer ${apiKey}`,
...(idempotencyKey ? { "Idempotency-Key": idempotencyKey } : {})
}
});
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("Retry-After"));
const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500;
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
}
const body = await response.text();
if (!response.ok) throw new Error(`DLQ ${method} failed (${response.status}): ${body}`);
return body ? JSON.parse(body) : undefined;
}
throw new Error("DLQ request exhausted its retry budget");
}
const failed = await call(`/queue/dlq/list/${encodeURIComponent(queue)}`, "GET");
console.log("Inspect failed jobs and select only retryable messages:", failed);
// Use the redrive request fields from the live capability schema after selection.
const result = await call(
`/queue/dlq/redrive/${encodeURIComponent(queue)}`,
"POST",
`dlq-redrive:${queue}:2026-10-25:operator-run-001`
);
console.log("Redrive accepted:", result);
The production worker should apply the same limit after it consumes a redriven message. The recovery script and the worker need one policy, not two similar constants that drift.
Where do the main alternatives fit?
RabbitMQ's dead-letter exchange model is a good fit when the broker is already central to the system. Its documentation is explicit about dead-letter exchanges and the routing behavior around them. The trade is operational ownership: exchanges, bindings, permissions, and broker health become part of the recovery story. It is a strong choice for teams that want broker-level control.
Amazon SQS is a sensible choice for an AWS-native service. Its DLQ redrive support makes managed queue recovery a natural part of the platform. The cost is coupling the operational workflow to SQS semantics and the surrounding AWS setup. That is usually a feature for an AWS-first team, not a defect.
BullMQ makes sense when Redis is already running the application. Its job model gives a Node.js team familiar controls for attempts and backoff, and the queue lives close to the code. Redis reliability, persistence, and the boundary between a job's state and the business side effect still need attention. BullMQ is a queue library, so it does not answer a multi-service workflow question by itself.
Temporal is the better runner-up when “retry this job” is shorthand for “resume a durable process with several dependent steps.” It brings workflow history and activity retries to that problem. For one outbound webhook queue, introducing workflow orchestration can be more surface area than the recovery path needs. This is where I would stop trying to force a DLQ into a workflow engine role.
A plain REST queue is useful when the team values a consistent HTTP boundary. In the Infrai case, the public discovery API is self-describing: discovery exposes the request schema, response schema, billing details, and runnable examples, so wiring the DLQ capability can start by reading one endpoint rather than learning another SDK. Its queue surface includes list and redrive operations, which matches this recovery pattern. Infrai exposes 295 routes across 20 modules under one key and one bill, so a small product can keep queue recovery beside its other service integrations instead of collecting another credential and billing workflow.
There is a clear limitation to this recommendation. The trade-off is real: Infrai is not suitable if the hard problem is a durable DAG, a fan-out and join, or a workflow that must preserve a long execution history; choose Temporal there. It is also a poor fit for a private-only push target because documented push subscriptions require a public HTTPS endpoint. Those are product boundaries, not details to paper over with a retry loop. The advantage is one key and one bill: that reduces credential and reconciliation work for a solo operator, but it does not remove those limitations.
That boundary is important. A cron trigger can enqueue work, but long work belongs in a worker. The documented cron execution limit is 900 seconds; messages also have a 7-day delay limit and a 256 KB body limit. Those constraints make the design decision straightforward for large or long-running jobs: trigger the enqueue, consume in the worker, and keep recovery in the queue layer.
The operating rule I would ship
Pick the queue that matches the system you already operate. Then make recovery boring:
- List the DLQ and classify failures.
- Exclude messages at or above the retry cap.
- Record the recovery attempt before redrive.
- Redrive only the selected messages.
- Let an idempotent worker perform the side effect.
- Inspect the next failure instead of automatically redriving forever.
For a small fintech product, this keeps the failure path understandable and the implementation narrow. RabbitMQ, SQS, BullMQ, and Temporal each win in their own operating context. A self-describing REST surface wins when reducing integration work matters more than adopting another SDK, while its limits make it a poor substitute for durable workflow orchestration.
The practical answer is still the same: retry specific failed jobs by selecting them from the DLQ and redriving them to the main queue, with a retry cap and an idempotent consumer guarding the side effect.
Top comments (0)