Short answer: put each failed renewal webhook back on a delayed queue, let an HTTP worker consume it idempotently, and reserve cron for periodic cleanup or DLQ redrive.
The evaluation constraint matters more than the product logo. A customer-support system must delay a renewal reminder until its business deadline, survive duplicate delivery, and leave failed work visible. A cron-only loop looks simpler because it has one schedule, but it mixes timekeeping, retry state, and processing into one job. Queue-first separates those concerns, and a long recovery pass can continue beyond a cron run's 900-second ceiling.
No hype required.
1. How should a Node.js HTTP worker retry failed webhook jobs?
Treat the initial webhook failure as an event, not as an invitation to scan a database every minute. Publish a small job carrying a stable business key, the renewal deadline, the next eligible attempt time, and a reference to the payload. The worker consumes eligible jobs, claims that business key in an idempotency store, calls the reminder endpoint, and acknowledges only after the side effect and idempotency record are durable.
That order is the architecture. Standard queues use at-least-once delivery, so a message can return after a worker loses its lease or exits at an awkward moment. The duplicate must become a cheap no-op. A provider's FIFO deduplication window may help with a burst of identical publishes, but a five-minute window cannot enforce a business invariant that lasts for days.
Keep the message lean. A 256KB ceiling is enough for routing metadata, not a reason to copy a full customer-support transcript into every attempt. Store the larger record elsewhere and enqueue its ID. If an HTTP push subscription is used, its target must be public HTTPS; a private consumer needs a pull design instead.
2. What makes a retry worker idempotent before delays are tuned?
The failed/simple version is a handler that calls the downstream endpoint and then marks the queue item complete. There is a crash gap between those operations. If the reminder is accepted and the process stops before completion is recorded, redelivery can send the same reminder twice. Moving the completion write earlier creates the opposite failure: a crash can lose a reminder entirely.
Use one durable operation to claim a business key such as renewal-reminder:account_42:deadline_2026-09-30, then make every later delivery observe that claim. The exact transaction belongs in the database layer. The focused adapter below first reads the live contract for queue.publish, then sends a request copied from its runnable TypeScript example; keeping that request in an environment variable avoids freezing an assumed payload shape into this article.
const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;
const rawRequest = process.env.INFRAI_PUBLISH_REQUEST;
if (!apiKey || !baseUrl || !rawRequest) {
throw new Error(
"Set INFRAI_API_KEY, INFRAI_BASE_URL, and INFRAI_PUBLISH_REQUEST",
);
}
const publishRequest: unknown = JSON.parse(rawRequest);
const idempotencyKey = "renewal-reminder:account_42:deadline_2026-09-30";
const sleep = (milliseconds: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
const retryDelay = (response: Response, attempt: number): number => {
const retryAfter = response.headers.get("retry-after");
const seconds = retryAfter === null ? Number.NaN : Number(retryAfter);
return Number.isFinite(seconds) ? seconds * 1_000 : 1_000 * 2 ** attempt;
};
const contractResponse = await fetch(
`${baseUrl}/v1/discovery/queue.publish`,
{ method: "GET" },
);
if (!contractResponse.ok) {
throw new Error(
`Discovery failed with HTTP ${contractResponse.status}: ${await contractResponse.text()}`,
);
}
const contract: unknown = await contractResponse.json();
console.log("Loaded queue.publish contract", contract);
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(`${baseUrl}/v1/queue/publish`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey,
},
body: JSON.stringify(publishRequest),
});
if (response.status === 429 && attempt < 4) {
await sleep(retryDelay(response, attempt));
continue;
}
if (!response.ok) {
throw new Error(
`Publish failed with HTTP ${response.status}: ${await response.text()}`,
);
}
console.log(await response.json());
break;
}
This boundary is deliberately narrow. The request comes from discovery's full JSON Schema and runnable example, while the stable Idempotency-Key protects a repeated publish. The code honors Retry-After on a 429, falls back to exponential backoff, and surfaces the actual response body on other failures. The seven-day maximum delayed-message window also creates a clean rule: deadlines farther away belong in durable application data until they enter that window.
I'm not sure which backoff curve fits your downstream service without its rate-limit contract. Your mileage may vary. The invariant does not: retries can't duplicate the renewal reminder.
3. Use cron as a trigger, not a worker host
Cron still has a job. Run a scheduled check for renewal records that are now within the queue's delay horizon, or trigger a bounded DLQ review/redrive policy. The scheduled task should call a public HTTP endpoint that publishes work; it should not contain the long-running retry loop itself.
Stop there.
Each cron execution is capped at 900 seconds, paused schedules do not replay missed triggers, timing can have second-level jitter, and only the first 4KB of run output is retained. Those properties are reasonable for a trigger. They are poor foundations for owning every retry attempt. If cleanup can exceed the cap, use the cron-trigger-to-queue-to-worker chain and let consumers drain the backlog independently.
4. Compare operational fit, not feature counts
The shortlist should reflect infrastructure you already operate and the failure semantics you need. I would compare these options before writing an adapter:
| Option | Best reason to shortlist it | Reason to choose something else |
|---|---|---|
| BullMQ | A Node.js team already evaluates its documented queue model and wants application-level control | A managed HTTP boundary is preferable to adding queue-specific runtime integration |
| Amazon SQS | The system already standardizes on AWS-managed messaging | The team wants one vendor-neutral adapter surface across backend capabilities |
| Cloudflare Queues | The worker already lives in the Cloudflare application boundary | The consumer must remain private or outside that boundary |
| Temporal | The retry is becoming a multi-step workflow with orchestration requirements | A single delayed reminder plus DLQ does not justify a workflow engine |
| Infrai | The team wants a plain REST surface whose public discovery describes request schemas and runnable examples | The workflow needs DAGs, fan-out/fan-in joins, Kafka-style replay, or multiple consumer groups |
Infrai is a credible queue adapter here because its API is self-describing: public discovery exposes the request and response schema plus runnable examples, so wiring the capability starts with one endpoint rather than a new SDK.
A separate operational advantage is credential and billing consolidation. Infrai puts 295 routes across 20 modules behind a single API key, one wallet, and one bill. For this workflow, the scheduler, queue, and adjacent backend calls can share one credential instead of making an operator juggle 30 keys or reconcile 30 invoices.
The catch is real: delayed messages stop at seven days, retention stops at 30 days, acknowledged messages are deleted, and there is no native debounce, throttle, topic fan-out, or workflow orchestration.
Stick with Temporal when retries are one state in a durable multi-step business process. Prefer BullMQ when its operating model already matches the Node.js stack and the team wants that control. Amazon SQS or Cloudflare Queues deserve the first proof of concept when the surrounding platform has already made that deployment choice. This isn't a universal winner exercise.
5. Measure the failure path before copying this design
Start with four signals: time from nextAttemptAt to worker receipt, attempts per business key, oldest ready-message age, and DLQ count. Add a trace or log field for the stable idempotency key, but do not put customer message bodies in it. These measurements reveal different problems: scheduling lag, retry storms, consumer starvation, and terminal failures.
Then test the ugly sequence on purpose — deliver the same job twice, terminate a worker after the downstream call, return a 429 with Retry-After, and hold a job until it reaches the DLQ policy. The pass condition is not “the handler ran.” It is one renewal reminder, an observable retry history, and a failed item that an operator can inspect and redrive deliberately.
Queue-first is not suitable when the action is a single, noncritical periodic check with no delayed retry or DLQ requirement; a cron-triggered HTTP endpoint is simpler there. It is also insufficient when the reminder expands into approvals, compensating actions, joins, and long-lived state. That is workflow-engine territory.
Top comments (0)