Short answer: For a Node.js background job queue, retry failed jobs with capped exponential backoff, persist an idempotency key before the webhook side effect, and move exhausted attempts to a DLQ for reviewed redrive.
Send the weekly digest from a durable queue, and make the job idempotent before adding retries. That ordering is the deciding constraint: a retry can repeat a side effect, while a delayed message only repeats work you have made safe to repeat. This pattern fits a healthtech team processing customer images for a weekly webhook digest, where duplicate delivery is more damaging than a few minutes of latency.
The useful mental model is a small state machine. Before the change, a scheduler called the webhook endpoint directly, an image conversion timed out, and the scheduler had no record of whether the email had already gone out. After the change, the scheduler writes one job, workers lease it, attempts carry an exponential delay, and a dead-letter queue (DLQ) preserves the payload for an intentional redrive. Observability tells you which state needs attention.
How should a Node.js background job queue retry failed webhook jobs?
Start with an idempotency key that names the business event, not the attempt. For an active customer digest, digest:customer_482:2026-W35 is stable even when the queue delivers the message five times. Store that key with the delivery result in the same database transaction that marks the digest complete. A unique constraint turns a duplicate delivery into a harmless read.
Then classify errors. A 429 or a network timeout is usually transient; a 400 caused by an invalid image URL is permanent until data changes. Retrying both classes for the same number of times creates noise and delays diagnosis. The worker below makes that decision explicit and records each transition for metrics.
type DigestJob = {
id: string;
customerId: string;
week: string;
webhookUrl: string;
imageUrl: string;
attempt: number;
};
type DeliveryResult = { status: number; body: string };
const maxAttempts = 5;
const baseDelayMs = 30_000;
function backoffMs(attempt: number): number {
const exponential = baseDelayMs * 2 ** Math.max(0, attempt - 1);
const jitter = Math.floor(Math.random() * 5_000);
return Math.min(exponential + jitter, 15 * 60_000);
}
function retryable(result: DeliveryResult | Error): boolean {
if (result instanceof Error) return true;
return result.status === 408 || result.status === 425 || result.status === 429 || result.status >= 500;
}
async function deliverDigest(job: DigestJob): Promise<void> {
const idempotencyKey = `digest:${job.customerId}:${job.week}`;
const prior = await deliveryStore.find(idempotencyKey);
if (prior) return;
try {
const result = await sendWebhook(job.webhookUrl, {
idempotencyKey,
customerId: job.customerId,
imageUrl: job.imageUrl,
});
if (result.status >= 200 && result.status < 300) {
await deliveryStore.insert({ idempotencyKey, status: result.status });
metrics.count("digest.delivery.success", 1, { attempt: String(job.attempt) });
return;
}
if (!retryable(result) || job.attempt >= maxAttempts) {
await deadLetter.publish({ ...job, reason: `http_${result.status}` });
metrics.count("digest.delivery.dlq", 1, { reason: `http_${result.status}` });
return;
}
await queue.publish({ ...job, attempt: job.attempt + 1 }, { delayMs: backoffMs(job.attempt) });
} catch (error) {
if (job.attempt >= maxAttempts) {
await deadLetter.publish({ ...job, reason: "network_error" });
metrics.count("digest.delivery.dlq", 1, { reason: "network_error" });
return;
}
await queue.publish({ ...job, attempt: job.attempt + 1 }, { delayMs: backoffMs(job.attempt) });
}
}
declare const queue: { publish(job: DigestJob, options: { delayMs: number }): Promise<void> };
declare const deadLetter: { publish(job: DigestJob & { reason: string }): Promise<void> };
declare const deliveryStore: {
find(key: string): Promise<{ status: number } | undefined>;
insert(row: { idempotencyKey: string; status: number }): Promise<void>;
};
declare const metrics: { count(name: string, value: number, labels: Record<string, string>): void };
declare function sendWebhook(url: string, payload: Record<string, string>): Promise<DeliveryResult>;
The attempt number is data, not queue metadata you hope will survive a redrive. Persist it in the message. The cap keeps a failing customer from consuming the whole worker pool, while jitter prevents every customer scheduled at 09:00 from retrying in the same millisecond.
Here is the decision table I keep beside the worker code:
| Failure signal | Action | Why |
|---|---|---|
| Timeout, 429, or 5xx | Retry with backoff | The remote system may recover |
| Invalid image type or 4xx | Publish to DLQ | Repeating the same input cannot fix it |
| Success after duplicate delivery | Return the stored result | Idempotency prevents a second side effect |
That tiny table is also a test plan. Each row should have a fixture, an assertion, and a metric.
What makes exponential backoff and DLQ redrive safe for image processing?
Image work deserves its own boundary. Download the source, validate its content type and size, and write the transformed object to a deterministic path such as digests/482/2026-W35.webp. A second attempt can overwrite that same path, which is safer than creating output-2.webp and losing track of which file the webhook references. Hashing the source bytes gives you a useful dimension for logs without putting protected health information into labels.
The long tail matters here. Imagine Monday's digest has 2,000 active customers. At 09:00, a partner starts returning 429 for ten minutes. Without jitter, all five attempts for those customers line up at 09:00:30, 09:01:30, and 09:03:30. Workers spend their time re-failing the same requests, queue age rises, and a healthy image processor looks broken. With a random five-second spread and a concurrency limit, the queue drains as the partner recovers. The exact values need measurement; I am not sure your partner's rate window matches these numbers, so treat them as starting points and verify against response headers.
Keep the DLQ payload complete enough to replay: job id, customer id, week, source URL, attempt count, first-seen timestamp, and failure reason. Do not put access tokens in it. A redrive command should select a bounded batch, increment a redrive_count, and publish back to the normal queue only after an operator has checked the reason. Redrive is a recovery action, not an automatic sixth retry.
One short rule helps on call: transient transport failures wait; permanent validation failures stop. Your mileage may vary when a partner documents a different retry contract, so put that contract in configuration and test it with a fake clock.
Instrument the queue so a weekly digest is explainable
Logs answer “which customer?” without exposing message bodies. Use a correlation id shared by the scheduler, image worker, and webhook sender. Emit counters for enqueued, started, retry_scheduled, succeeded, and dlq_published; emit a histogram for queue age and processing time. Alert on oldest-message age and DLQ growth, not on a single 500.
I once started with a dashboard showing only successful deliveries. It looked calm while the DLQ grew. Adding a queue-age panel changed the conversation: a stalled consumer became visible before customer support saw a missed digest.
The before/after check is concrete. For one weekly window, you should be able to trace a job id from enqueue to either a single idempotency record or a DLQ item, then explain every delay from the attempt history. If you cannot, the system is making guesses on your behalf.
When should you choose a different retry policy?
Retries are a poor fit for work that is not safe to repeat, has a strict sub-minute deadline, or depends on a partner that forbids duplicate requests. In those cases, use a transactional outbox plus a human-reviewed handoff, or ask the partner for an idempotency protocol before queueing. A DLQ also adds operational work: someone must own triage, retention, and replay permissions.
For a small team with a single consumer, a managed queue can remove broker maintenance; a self-hosted queue can offer more control over retention and locality. The trade-off is staffing, not a universal performance win. Pick the option whose metrics, replay tooling, and access controls your team can operate at 03:00.
Don't hide the policy in a library default. Put the maximum attempts, delay cap, and DLQ retention in versioned configuration so a code review can see a behavior change.
The implementation target is modest: one durable enqueue, one idempotency record, bounded exponential backoff, and a visible DLQ. Build those four pieces first, then tune concurrency with measurements from real digest windows.
Top comments (0)