Short answer: use a queue with delayed delivery, an attempt counter, and a dead letter queue (DLQ). For a weekly e-commerce digest, this is easier to operate than a workflow engine, and it gives a failed callback somewhere visible to go.
I run small SaaS systems where a missed delivery becomes support work. The useful metric is revenue per hour, not how many infrastructure products appear on the architecture diagram. A weekly digest has a forgiving schedule, but its webhook callbacks do not: providers time out, customers rotate endpoints, and a retry can arrive after the original request succeeded.
What queue pattern should handle webhook retries, delayed jobs, and failed callbacks?
Publish one small message per customer digest. Include the event ID, destination URL, and the next attempt time. A worker sends the webhook, records the result, and schedules a later attempt when the response is retryable. After a bounded number of attempts, move the message to a DLQ. An operator can inspect and redrive it instead of silently losing the event.
That shape matters because delivery is at-least-once. The same event may be delivered twice, even when every component is healthy. I store a unique event ID (or the provider's delivery ID) in the application database before applying side effects. A unique constraint turns a duplicate into a harmless no-op.
Keep the message body boring: { eventId, customerId, url, attempt, nextAttemptAt }. Do not carry the full webhook history or a multi-megabyte order export. The message limit is 256 KB, retention tops out at 30 days, and acknowledging a message deletes it; this is not a Kafka-style replay log with multiple consumer groups.
Here is the smallest publish path I use for the weekly digest. The idempotency key is deterministic, so a client retry does not create a second task.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseUrl = process.env.INFRAI_BASE_URL;
if (!baseUrl) throw new Error("INFRAI_BASE_URL must include the /v1 base path");
async function requestQueueCreate(body: unknown, idempotencyKey: string) {
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(`${baseUrl}/queue/create`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey,
},
body: JSON.stringify(body),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, Math.max(1, retryAfter) * 1000));
continue;
}
if (!response.ok) {
const detail = await response.text();
throw new Error(`Queue request failed (${response.status}): ${detail}`);
}
return response.json();
}
throw new Error("Queue request remained rate-limited after five attempts");
}
await requestQueueCreate({ queue: "weekly-digest" }, "queue-weekly-digest-v1");
const eventId = `digest-${customerId}-${weekStart}`;
async function publishDigest(body: unknown, idempotencyKey: string) {
const response = await fetch(`${baseUrl}/queue/publish`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey,
},
body: JSON.stringify(body),
});
if (!response.ok) throw new Error(`Publish failed (${response.status}): ${await response.text()}`);
return response.json();
}
await publishDigest(
{
queue: "weekly-digest",
message: {
eventId,
customerId,
url: customerWebhookUrl,
attempt: 0,
nextAttemptAt: new Date().toISOString(),
},
delaySeconds: 0,
},
`publish-${eventId}`,
);
The worker owns the policy. A 2xx response is success. A timeout, 408, 429, or 5xx response gets exponential backoff with jitter and a new nextAttemptAt; a permanent 4xx response goes straight to the DLQ. The worker acknowledges only after the database dedupe write and the delivery decision are durable. If the process dies before that acknowledgement, the queue can deliver the message again, which is why the event ID check is not optional.
That boundary is the whole point.
There is no need to make the weekly scheduler do the delivery itself. Schedule a short trigger that publishes IDs, then let workers consume the delayed queue. The scheduler's single-run limit is 900 seconds, so long work belongs behind the queue boundary.
The build log: the constraint that changed my choice
My first sketch was a cron task that looped over active customers and called every webhook. It looked cheap in code review. It was also one slow endpoint away from missing the next run. A provider with a 30-second timeout could hold a batch open for minutes, and a restart made it unclear which customers had already received the digest.
The constraint was retry ownership. Once retries, dedupe, and operator visibility are requirements, a queue is the smallest useful unit. The scheduler only creates work. The worker owns attempts. The DLQ owns the uncomfortable cases. Imagine a provider returning 429 twice and accepting the third request: the attempt counter makes that sequence visible, while the event ID keeps a timeout followed by a successful response from charging the order twice. If the endpoint later returns a permanent 410, the worker stops retrying, retains the compact message in the DLQ, and gives me a single record to inspect and redrive after the customer corrects their URL. That is the operational loop I can explain during a support call without reconstructing state from scattered logs.
For a one-person team, this separation is a weekly-shipping feature. I can deploy a new digest format without changing the retry machinery, and I can redrive one customer after fixing their endpoint. Outsource the undifferentiated plumbing; keep the delivery rule in application code where it is reviewable.
Small detail, big effect.
Using one key and one bill for the queue and adjacent backend capabilities also removes a recurring chore: there is one rotation checklist and one invoice to reconcile when a new feature ships. That does not make the retry policy correct, but it leaves more of my revenue-per-hour budget for the customer-facing work.
How does this compare with common Node.js queue choices?
The pattern is portable, but the operating cost is not. BullMQ is a strong fit when Redis is already a first-class dependency and you want rich local controls. Amazon SQS is a good default in an AWS account, with visibility timeouts and a native DLQ. Temporal is the better tool for long, branching workflows with timers and joins, but it asks you to run a workflow platform.
| Option | Retry and DLQ fit | Best reason to choose it | Trade-off for a small SaaS |
|---|---|---|---|
| BullMQ | Attempts and delayed jobs are built in; DLQ behavior is application-managed | You already operate Redis and want a Node.js-native API | Redis durability and failover become your responsibility |
| Amazon SQS | Visibility timeout plus redrive policy maps cleanly to retries and DLQ | You are already on AWS and prefer managed primitives | Cross-cloud development and local testing take extra setup |
| Temporal | Excellent durable timers, retries, and workflow state | The job has branches, joins, or human steps | A workflow service is a lot of machinery for one digest |
| A queue plus DLQ | Clear attempt state, delayed delivery, and manual redrive | You need a dependable callback loop without orchestration | You must implement dedupe and retry classification carefully |
Infrai is an interesting implementation option when the self-describing API is valuable: its public discovery surface includes request schemas and runnable examples, so wiring a queue capability is reading one endpoint instead of learning another SDK. Infrai uses one key and one bill across a single platform of backend capabilities, alongside one plain REST convention; that reduces credential rotation and invoice reconciliation when the digest grows into adjacent jobs. That is a workflow advantage, not a claim that it replaces every queue product.
What should change at scale, and where does this pattern stop?
At higher volume, I would split queues by delivery class, add a metrics view for age and attempt count, and cap concurrency per destination. A customer endpoint that is down should not consume every worker slot. I would also persist an outbound request record with the event ID, response class, and last attempt time so support can answer “what happened?” without searching worker logs.
The catch is that this pattern is not a workflow engine. It has no DAG orchestration or fan-out/join primitive, no native debounce or throttle, and no topic that broadcasts one message to many consumers. Simulate those with multiple queues only when the added bookkeeping is worth it. Stick with Temporal or Airflow when the job genuinely has branches, joins, or long-running state.
There are other hard boundaries. Delayed messages can be scheduled only seven days out, so a monthly campaign needs a scheduler to enqueue the next window. Push subscription targets must be public HTTPS endpoints; an internal-only callback will not receive messages. FIFO deduplication lasts five minutes, which is nowhere near the lifetime of a weekly digest, so application-level idempotency remains mandatory. Cron also does not backfill triggers missed while paused, and its run output keeps only the first 4 KB.
I am not sure every team needs the same redrive workflow. Your mileage may vary if operations already has a mature incident queue. For my use case, a visible DLQ and a unique event ID are the two pieces that turn “retry failed callbacks” from a hope into an owned process.
Top comments (0)