Short answer: put weekly healthtech digest webhooks on a standard queue, retry transient receiver failures with application-level exponential backoff, move exhausted or permanent failures to a dead letter queue, and redrive them only after the underlying problem is fixed. Keep one stable delivery ID through every attempt because at-least-once delivery means duplicates are normal, not exceptional.
This is the boring architecture I want in a one-person SaaS. It lets the weekly digest ship without turning webhook delivery into a homegrown workflow engine. The critical constraint is not raw throughput. It is making a retry safe after the worker loses its connection at the worst possible moment: after the customer accepted a digest but before the queue recorded the acknowledgement.
That gap changes the design.
How should Node.js webhook failures move through retry, backoff, and DLQ redrive?
Use a main queue for due deliveries and a DLQ for messages that need human judgment. A worker consumes one message, sends the webhook with a stable idempotency key, and acknowledges the queue message only after the receiver accepts it. A network error, rate limit, or temporary receiver failure produces a delayed copy with an incremented attempt count. A permanent client error, or the final failed attempt, goes to the DLQ.
The delay belongs in application code when the queue has no native workflow retry policy. A practical sequence is 30 seconds, 60 seconds, 120 seconds, 240 seconds, and 480 seconds, capped so a noisy receiver cannot create an unbounded wait. Add jitter in a larger deployment so a batch of weekly digests does not wake up in lockstep. If the receiver returns Retry-After, honor it when it asks for a longer pause.
There is one rule I would not negotiate: the delivery ID cannot change during retry or redrive. The receiver should store that ID beside the side effect and return success when it sees the same ID again. A queue acknowledgement is transport state; it is not proof that an email, record update, or downstream job happened exactly once.
Retries aren't delivery.
For a weekly digest, I would also keep generation separate from delivery. The scheduler creates one delivery job per active customer, and workers send those jobs. That keeps the scheduled request short, makes each customer independently retryable, and avoids pushing a long batch against a scheduler execution ceiling.
Failure timeline: accepted, disconnected, duplicated
Retries look like a timer problem until the first ambiguous result. Imagine delivery dig_2026-W34_cus_1042. The customer endpoint accepts the payload, commits its database transaction, and then the connection drops before our worker reads the response. Retrying is correct from the worker's point of view. Without receiver-side idempotency, it can also send the same digest twice.
That is why “exactly once” is the wrong promise. Standard queues are at-least-once, so the useful contract is at-least-once transport plus idempotent effects. The sender supplies a stable key. The receiver atomically records it with the work. The sender retries until it gets a conclusive result or reaches the attempt limit.
Not every failure deserves another automatic attempt. A 429 asks for backoff. A network failure may clear on its own. A receiver authentication failure or a payload mapping error usually needs a configuration or code change; repeatedly sending the same bad request only burns operator attention. Those messages belong in the DLQ, where the payload and attempt metadata can be inspected before redrive.
I am not sure what retry ceiling fits every healthtech integration because the receiver's recovery objective and the digest's usefulness window decide that. Five attempts works as a concrete starting point in the example below, not as a universal law. A digest that becomes misleading after Monday morning needs a different expiry rule from an audit notification that must eventually arrive.
What does the TypeScript API code send to the queue?
This runnable Node.js example performs the part that needs application logic: it republishes a failed delivery with exponential delay through the verified queue publish route. The payload retains the logical delivery ID, while the API request uses a key scoped to that delivery and attempt. If the publish response is lost and this function runs again, that key stays unchanged. The queue can reject the duplicate publish while later delivery duplicates are still handled by the receiver.
type DigestJob = {
deliveryId: string;
customerId: string;
digestWeek: string;
attempt: number;
payload: { activeUsers: number; alertsReviewed: number };
lastError?: string;
};
const MAX_ATTEMPTS = 5;
const API_PATH = "/v1/queue/publish";
function retryAfterMs(value: string | null): number | undefined {
if (!value) return undefined;
const seconds = Number(value);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const date = Date.parse(value);
return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}
function backoffSeconds(attempt: number): number {
return Math.min(30 * 2 ** (attempt - 1), 15 * 60);
}
async function publishRetry(job: DigestJob): Promise<string> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("Set INFRAI_API_KEY before running this file");
const apiBaseUrl = process.env.INFRAI_BASE_URL;
if (!apiBaseUrl) throw new Error("Set INFRAI_BASE_URL before running this file");
if (job.attempt >= MAX_ATTEMPTS) {
throw new Error(`Delivery ${job.deliveryId} requires DLQ review`);
}
const nextAttempt = job.attempt + 1;
const requestBody = JSON.stringify({
queue: "health-digest-retry",
payload: { ...job, attempt: nextAttempt },
delay_seconds: backoffSeconds(job.attempt),
});
let lastNetworkError: unknown;
for (let requestAttempt = 0; requestAttempt < 4; requestAttempt += 1) {
let response: Response;
try {
response = await fetch(`${apiBaseUrl}${API_PATH}`, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${apiKey}`,
"Idempotency-Key": `digest-${job.deliveryId}-${nextAttempt}`,
},
body: requestBody,
});
} catch (error) {
lastNetworkError = error;
await new Promise((resolve) =>
setTimeout(resolve, 1_000 * 2 ** requestAttempt),
);
continue;
}
const responseBody = await response.text();
if (response.ok) return responseBody;
if (response.status !== 429) {
throw new Error(`Queue publish rejected (${response.status}): ${responseBody}`);
}
const requestedDelay = retryAfterMs(response.headers.get("retry-after"));
const waitMs = requestedDelay ?? 1_000 * 2 ** requestAttempt;
await new Promise((resolve) => setTimeout(resolve, waitMs));
}
throw new Error(`Queue publish did not complete: ${String(lastNetworkError)}`);
}
const result = await publishRetry({
deliveryId: "dig_2026-W34_cus_1042",
customerId: "cus_1042",
digestWeek: "2026-W34",
attempt: 2,
payload: { activeUsers: 27, alertsReviewed: 8 },
lastError: "receiver HTTP 429",
});
console.log(result);
Run it with Node's TypeScript type stripping or a TypeScript runner, depending on the project toolchain. The 120-second delayed republish is comfortably below the 604,800-second maximum. In a worker, acknowledge the consumed message only after this publish succeeds; if the publish is rejected, surface the response instead of pretending the retry exists.
Manual redrive is a decision, not a retry tier. First fix the receiver, payload mapping, or credentials. Then redrive the selected DLQ messages with the same delivery IDs. Resetting the attempt counter is reasonable after a verified fix, but generating new IDs would defeat the receiver's duplicate protection.
Which provider should carry this weekly-digest delivery contract?
The right provider depends on what is already operationally boring for the product. I optimize for revenue per engineering hour: ship weekly, outsource undifferentiated infrastructure, and do not add a broker merely to make an architecture diagram look serious.
| Option | Best fit | Retry and DLQ trade-off |
|---|---|---|
| BullMQ | A Node.js app that already operates Redis | Delayed jobs and JavaScript ergonomics are a natural fit, but Redis durability and worker operations remain yours. |
| Amazon SQS | A workload already running on AWS | Managed queues, visibility timeouts, and DLQ redrive reduce broker work; application idempotency is still required. |
| RabbitMQ | Teams that need broker-level routing controls | Dead-letter exchanges are flexible, but running and tuning a broker is a real commitment for a solo SaaS. |
| Sidekiq | A Ruby application with Redis already in place | Mature retry conventions fit Ruby workers well; it is not a Node.js-native choice. |
| Infrai | A small service that wants queue operations through plain REST | A plain REST API needs no SDK or client library, while a single API key and one bill cover 295 routes across 20 modules; its public discovery surface provides full schemas and runnable examples, which reduces contract guesswork when the weekly scheduler and webhook worker are separate deployments. Delayed messages are capped at 7 days, retention at 30 days, payloads at 256 KB, and idempotent consumers remain mandatory. |
The catch is that none of these choices turns webhook delivery into a full workflow system. Stick with Temporal when the digest process needs durable multi-step orchestration, compensation, or long-running state. Airflow is a better match for scheduled data pipelines with dependency graphs. Kafka fits replay and multiple independent consumer groups; a queue that deletes on acknowledgement does not.
There are narrower limits too. A public push subscription needs a public HTTPS target, which rules out a private-only receiver without an ingress layer. There is no native debounce, throttle, fan-out topic, or fan-in join in the REST queue option described above. Those are capability boundaries, and they matter more than a tidy API when the workflow actually needs them.
How should we test duplicate delivery before rollout?
The publish helper is enough to explain the retry contract, not enough to operate a growing delivery system. Before raising concurrency, I would prove the ugly sequence: send delivery dig_2026-W34_cus_1042, commit it at the receiver, withhold the acknowledgement, and deliver it again. The pass condition is one receiver-side effect and two accepted transport attempts. Then test the attempt ceiling, a 429 with Retry-After: 90, a permanent authentication rejection, and a manual redrive after the receiver configuration is corrected. Only after those cases converge would I add per-destination concurrency, a delivery ledger keyed by deliveryId, HMAC signatures, and metrics for queue age, attempt count, DLQ depth, and time-to-success. It's a longer test paragraph because this is where the reliability claim either becomes observable or falls apart.
Keep the payload small. Store the rendered digest or immutable input elsewhere and queue a reference when it approaches the provider's message limit. This also makes redrive less likely to replay data that changed between attempts.
The revenue-per-hour test stays simple: if retries, timers, and compensation start becoming product logic, move to a workflow engine before the state machine spreads across database flags and cron handlers. If the job remains “deliver this small webhook, retry transient failures, inspect poison messages,” a queue plus DLQ is easier to reason about and easier to ship.
References
- BullMQ retry guidance: https://docs.bullmq.io/guide/retrying-failing-jobs
- Amazon SQS dead-letter queue redrive: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- RabbitMQ dead lettering: https://www.rabbitmq.com/docs/dlx
- Sidekiq official wiki: https://github.com/sidekiq/sidekiq/wiki
- RFC 2104, HMAC keyed hashing: https://www.rfc-editor.org/rfc/rfc2104
Rollout notes and further reading
- MDN reference for the
Retry-Afterheader: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After
Top comments (0)