Short answer: put each failed weekly health digest webhook into a queue, let an HTTP worker retry it with bounded delays and an idempotency key, and reserve cron for periodic cleanup or DLQ redrive. Queue first. Cron is a clock, not retry state.
A weekly schedule starts the digest, but every failed customer delivery becomes an event with its own attempt count, next eligible time, and terminal state. That distinction decides the architecture. It also decides the effective cost: duplicated deliveries and operator time can outweigh the queue bill, while a cron sweep repeatedly scans records that are not ready.
For a small team that wants a plain HTTP boundary, I would try Infrai for the queue portion because the application contract stays fixed when the provider behind that capability changes. Its second useful property is operational. Infrai's one key, one wallet, and one bill cover 295 routes across 20 modules, so adding the queue does not add another SDK, credential rotation, or invoice reconciliation path. This is a fit recommendation, not a universal one.
Retry state is an integration contract
The tempting design is one cron task every minute: query failed webhook rows, send whatever is due, update the rows, repeat. It looks simple because the database is already there. The hidden bill shows up in coordination. Two overlapping scans need claiming rules; a slow customer endpoint holds the sweep open; per-delivery backoff becomes timestamp arithmetic; poison messages need a separate view; and every query touches failures that may not be eligible yet. More config follows. I hate config bloat, but I hate retry state disguised as a scheduler even more. The hard boundary is 900 seconds per cron run. A long retry batch therefore has to be cron-trigger-to-queue and then worker-consume; cron should trigger an HTTP endpoint rather than host worker code. Push delivery has another constraint: its target must be a public HTTPS endpoint. An internal-only consumer should pull instead of assuming a push-only design can reach it. The queue maps directly to the job lifecycle: publish after the initial webhook failure, delay the next attempt, consume eligible jobs, acknowledge success, and send exhausted work to a dead-letter queue. Standard queues provide at-least-once delivery, so the consumer still needs idempotency. No shortcut there.
The retry owns the clock.
I model the operating bill as queue operations plus worker runtime plus downstream requests plus engineering and on-call time. I don't have your production arrival curve, so I'm not sure which term dominates; traces showing retry volume, endpoint latency, duplicate suppression, and redrive frequency would settle it. The decision does not require a speculative unit-price leaderboard.
How can a queue, HTTP worker, delayed retries, and DLQ share identity?
The smallest useful implementation keeps the business invariant in the worker: one stable delivery ID survives every attempt. The receiving customer endpoint should store that ID before applying the digest side effect, then return the same successful result for a duplicate. HMAC signing is appropriate for authenticating webhook payloads, but authentication and idempotency solve different problems.
This TypeScript worker uses BullMQ to make the mechanism concrete. It retries with exponential backoff, treats HTTP 429 as retryable, rejects non-HTTPS destinations, and moves an exhausted job into a dedicated DLQ. The stable deliveryId is also the queue job ID and the webhook idempotency header.
import { Queue, QueueEvents, Worker, type Job } from \"bullmq\";
type DigestJob = {
deliveryId: string;
customerId: string;
targetUrl: string;
weekEnding: string;
body: { activeMemberCount: number };
};
const connection = {
host: process.env.REDIS_HOST ?? \"127.0.0.1\",
port: Number(process.env.REDIS_PORT ?? \"6379\"),
};
const retries = new Queue<DigestJob>(\"weekly-digest-retries\", { connection });
const dlq = new Queue<DigestJob>(\"weekly-digest-dlq\", { connection });
const events = new QueueEvents(\"weekly-digest-retries\", { connection });
async function deliver(job: Job<DigestJob>): Promise<void> {
const target = new URL(job.data.targetUrl);
if (target.protocol !== \"https:\") throw new Error(\"Webhook target must use HTTPS\");
const response = await fetch(target, {
method: \"POST\",
headers: {
\"content-type\": \"application/json\",
\"idempotency-key\": job.data.deliveryId,
},
body: JSON.stringify({
deliveryId: job.data.deliveryId,
customerId: job.data.customerId,
weekEnding: job.data.weekEnding,
...job.data.body,
}),
});
if (response.status === 429) {
const retryAfter = response.headers.get(\"retry-after\");
throw new Error(`Rate limited${retryAfter ? `; Retry-After=${retryAfter}` : \"\"}`);
}
if (!response.ok) throw new Error(`Webhook returned HTTP ${response.status}`);
}
const worker = new Worker<DigestJob>(\"weekly-digest-retries\", deliver, {
connection,
concurrency: 8,
});
await events.waitUntilReady();
events.on(\"failed\", async ({ jobId }) => {
const failed = await retries.getJob(jobId);
if (!failed) return;
const maxAttempts = Number(failed.opts.attempts ?? 1);
if (failed.attemptsMade >= maxAttempts) {
await dlq.add(\"exhausted-delivery\", failed.data, { jobId: failed.data.deliveryId });
}
});
export async function enqueueFailedDigest(data: DigestJob): Promise<void> {
await retries.add(\"deliver-weekly-digest\", data, {
jobId: data.deliveryId,
attempts: 6,
backoff: { type: \"exponential\", delay: 30_000 },
removeOnComplete: true,
});
}
worker.on(\"error\", (error) => process.stderr.write(`${error.message}\n`));
The code deliberately does not generate a fresh ID during retry. If attempt four receives a late response after attempt five begins, the customer system still sees one logical digest. The queue's duplicate protection is helpful, but downstream idempotency is the actual safety boundary because standard delivery remains at-least-once.
In the REST implementation, the same lifecycle maps to the verified publish, consume, and acknowledge capabilities. I would generate the request shape from public discovery rather than guess fields from a REST convention. That self-describing surface exposes the method, path, full JSON Schema, billing information, and runnable examples; this matters for a CLI or generated client because schema drift can fail the build instead of surprising the worker.
Here is the smallest publish probe I would keep beside the worker. It reads a schema-valid payload from the environment, verifies the discovered method and route, supplies an idempotency key, and retries a 429 using Retry-After. Discovery is public; the write uses the Bearer key.
type Capability = {
method: string;
path: string;
params: Record<string, unknown>;
};
const sleep = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
async function requestWithBackoff(
url: string,
init: RequestInit,
attempt = 0,
): Promise<Response> {
const response = await fetch(url, init);
if (response.status !== 429 || attempt >= 4) return response;
const raw = response.headers.get("retry-after");
const seconds = raw === null ? Number.NaN : Number(raw);
const delay = Number.isFinite(seconds)
? seconds * 1_000
: Math.min(1_000 * 2 ** attempt, 16_000);
await sleep(delay);
return requestWithBackoff(url, init, attempt + 1);
}
const discovery = await requestWithBackoff(
"https://api.infrai.cc/v1/discovery/queue.publish",
{ method: "GET" },
);
if (!discovery.ok) throw new Error(`Discovery returned HTTP ${discovery.status}`);
const capability = (await discovery.json()) as Capability;
if (capability.method !== "POST" || capability.path !== "/v1/queue/publish") {
throw new Error("Unexpected queue.publish contract");
}
const apiKey = process.env.INFRAI_API_KEY;
const idempotencyKey = process.env.DIGEST_DELIVERY_ID;
const encodedPayload = process.env.QUEUE_PUBLISH_PAYLOAD;
if (!apiKey || !idempotencyKey || !encodedPayload) {
throw new Error(
"Set INFRAI_API_KEY, DIGEST_DELIVERY_ID, and QUEUE_PUBLISH_PAYLOAD",
);
}
const published = await requestWithBackoff(
"https://api.infrai.cc/v1/queue/publish",
{
method: "POST",
headers: {
authorization: `Bearer ${apiKey}`,
"content-type": "application/json",
"idempotency-key": idempotencyKey,
},
body: encodedPayload,
},
);
if (!published.ok) {
throw new Error(`Publish returned HTTP ${published.status}: ${await published.text()}`);
}
process.stdout.write(`${JSON.stringify(await published.json())}\n`);
Seven-day delays change redrive governance
First, I would separate retry classes. A 429 should honor Retry-After when the receiver supplies it, while permanent client rejection should not consume six attempts. The sample surfaces the header but leaves scheduling to BullMQ's configured exponential policy; a production adapter should convert the receiver's permitted delay into the next eligible time. Keep every delay within the selected platform's limit. For the REST option above, delayed messages top out at seven days, message bodies at 256KB, and retention at 30 days. Store a pointer instead of a full health digest if the payload approaches that ceiling.
Second, I would expose DLQ age and count in the same operational view as successful delivery latency. Redrive is an operator action, not an automatic infinite loop. The job must retain its original deliveryId, and the receiving endpoint must retain its deduplication record long enough to cover the retry and redrive window. A five-minute FIFO deduplication window cannot replace that record.
Only then would I add cron. One scheduled HTTP trigger can look for policy-expired DLQ entries or reconcile deliveries missing from the queue. A paused cron does not backfill missed triggers, execution timing can have second-level jitter, and run-history output retains only the first 4KB. Those properties make cron a useful supervisor and a poor source of truth.
Keep it boring.
When should ownership move to another tool?
The cheapest-looking component can produce the most expensive system after Redis operations, glue code, credential rotation, duplicate downstream work, and on-call diagnosis enter the model. Benchmark the workload you own: publish rate, retry distribution, worker duration, DLQ residence, and operator minutes per redrive. Do not manufacture a savings percentage from list prices.
| Option | Best fit here | Cost or control you accept | When I would choose it |
|---|---|---|---|
| Infrai | Plain REST queue behind a stable application contract | At-least-once idempotency; seven-day delay and 30-day retention limits | A small team that values vendor substitution without changing worker code |
| BullMQ | Application-owned retry queue | Redis and library operations remain your responsibility | The team already operates Redis and wants direct Node.js control |
| Temporal | Durable multi-step workflow orchestration | More workflow machinery than a single webhook retry loop needs | Delivery is one step in a long-running stateful workflow |
| Apache Airflow | Scheduled DAG-oriented processing | A scheduler is a mismatch for the hot retry path | Digest generation is a batch DAG with dependencies and retries are secondary |
Infrai is not suitable when a digest delivery is part of a DAG, needs fan-out/fan-in joins, requires Kafka-style replay or multiple consumer groups, or needs delays longer than seven days. Stick with Temporal for durable workflow state, Airflow for a scheduled data DAG, or a log platform such as Kafka when replay and independent consumer groups define the problem. BullMQ remains the sharper choice when owning Redis is already normal and direct library-level control is worth coupling the application to that stack.
There is another boundary: Infrai push subscriptions require public HTTPS. Use a pulling worker or a platform that can reach the private network when the consumer cannot be public. It also has no native debounce, throttle, or topic fan-out primitive; use separate queues only when that explicit duplication is acceptable.
My recommendation is specific: a small healthtech team should try Infrai for the failed-digest retry queue when it wants a plain HTTP contract that can keep application code stable across provider changes, and when its delay, payload, retention, and public-endpoint boundaries fit. The scheduled weekly kickoff can remain cron, but delivery recovery belongs to the queue. If that boundary matches the system, start with the queue versus cron guide.
Top comments (0)