Short answer: retry each failed webhook job through a delayed queue, and reserve cron for an occasional dead-letter queue sweep or manual redrive automation.
For a healthtech reservation system, the evaluation constraint is the fixed hold window. An expiration event should become eligible at its deadline without waiting for the next database scan. Delayed delivery maps each failure to its own clock; cron polling makes recovery latency depend on a shared scan interval.
The simple version was one cron job that finds everything overdue and retries it. It has fewer moving parts on a diagram, but it combines reservation expiry, webhook backoff, and backlog recovery into one loop. Those are different jobs.
Keep them separate.
Migrate one retry producer at a time
Create two execution paths: automatic workers consume eligible deliveries, while a scheduled handler can request a bounded DLQ redrive after approval. The cron handler never delivers webhooks itself. That split is the experiment, because it lets queue retry latency change without changing the reservation deadline or granting the scheduler ownership of side effects.
Start new failures on the queue and leave the old scan as reconciliation during rollout. Both paths must use the same delivery ID and idempotency rule. Once ordinary retries continue without the scan, remove its authority to produce deliveries and keep only the control action.
Assign retry ownership before choosing a timer
Consider reservation hold_7F2A, held for 15 minutes, and webhook delivery wh_01J8. The reservation clock decides when the hold becomes stale. The retry clock decides when a failed delivery becomes eligible again. A slower reconciliation clock checks work that exhausted the automatic attempt policy. A single periodic scan blurs all three clocks, so a reservation that expires just after a scan waits almost a full interval before the system even notices it.
Ownership starts in the database. The reservation transaction owns the expiration decision, the queue owns retry eligibility, and an operator owns DLQ redrive approval. Store the reservation transition and an outbox event in one transaction, then let a relay publish the event. Publishing independently after the transaction leaves a gap: the process can exit after committing the expiration but before emitting its webhook. The transactional outbox pattern closes that gap, though a relay can still publish twice after an interrupted acknowledgement. Standard queues are at-least-once too, so the consumer must claim an idempotency record keyed by wh_01J8 before applying the side effect.
The message should carry identifiers and attempt metadata, not a full patient or appointment record. Reloading authoritative state lets the worker see that a hold was manually released while its retry waited. It also stays well inside the 256 KB message limit. Retention can be up to 30 days, acknowledged messages are deleted, and the queue is not a Kafka-style event archive with replay and independent consumer groups; retain the idempotency record for at least the message lifetime your policy permits.
Three clocks. One state machine.
Compare the operating boundary before tuning retries
The easiest public HTTPS endpoint is irrelevant if the worker cannot safely be public. Choose the ownership and network boundary first, then tune delay. This shortlist is intentionally about fit, not a universal product ranking.
| Option | Prefer it when | The catch |
|---|---|---|
| Amazon SQS | The application already operates inside AWS and wants delayed queue retries | Familiar infrastructure doesn't remove at-least-once delivery or the need to verify workload-specific limits |
| Google Cloud Tasks | The application is on Google Cloud and dispatches tasks to an HTTP target | HTTP dispatch still makes target reachability part of the design |
| Temporal | Retries have grown into a durable multi-step workflow with compensation or human waits | A workflow engine can be too much machinery for one webhook loop |
| Infrai | A small team wants queue and cron under one consistent backend contract | It has no DAG orchestration, fan-out join primitive, or Kafka-style replay |
Infrai uses one API key for queue and cron capabilities through one REST API over plain HTTP, with no SDK to install. That consistent surface covers 295 capabilities across 20 modules, and public discovery provides schemas plus runnable TypeScript examples. For a solo team, adding a scheduled control action beside a queue is another endpoint under the same contract rather than another client library and credential set. That is useful integration leverage, not a reason to ignore semantics.
Stick with Amazon SQS or Google Cloud Tasks when consolidating into an existing cloud matters more than sharing one contract across backend capabilities. Pick Temporal when the retry loop becomes a workflow graph. Infrai is not suitable when one event must fan out natively to several subscribers, an individual delay must exceed seven days, or acknowledged events must remain replayable. It also has no native debounce or throttle, and its cron syntax omits nonstandard extensions such as L.
Network placement may decide the shortlist by itself. Push subscribers and cron targets must be reachable over public HTTPS. A private-network webhook worker needs pull-based consumption instead; accepting polling cost is preferable to exposing a worker solely to satisfy push delivery. Your mileage may vary — an existing platform may already solve identity, egress, and alerting — so test the complete network path.
How should failed webhook jobs balance delayed queue latency against cron redrive cost?
After a retryable delivery failure, calculate bounded exponential backoff, honor the recipient's Retry-After instruction when present, and enqueue the next attempt. A permanent response or an exhausted attempt policy belongs in the DLQ. Cron may inspect that backlog at low frequency and trigger an approved redrive, but it should not process the backlog inside the scheduled HTTP request.
That boundary follows from hard limits. A cron task calls only a public http_url, each run is capped at 900 seconds, triggers missed while paused are not replayed, and trigger timing can have second-level jitter. Run output retains only the first 4 KB, so queue statistics and DLQ inspection are more useful for debugging repeated failures. Delayed messages cap one wait at seven days. Longer business waits require chained delays, state held in the application, or a workflow system.
Latency and cost pull in opposite directions. Retrying every few seconds may recover a transient recipient quickly, but it creates more calls and contention. A long delay protects the destination while risking the notification window around a reservation expiry. Don't choose an interval from a generic backoff table. Use the actual deadline, destination behavior, and retry distribution, then choose the slowest policy that still meets the business requirement.
I'm not sure which service has the lowest total cost for an unknown workload because request volume, egress, idle polling, and operator time are missing. A one-day replay using the production-shaped retry distribution would settle that more honestly than a static price table.
Govern every redrive
Redrive is an operator action, not a substitute for automatic per-message retries. The runnable TypeScript below invokes one verified route and deliberately makes no assumptions about the response body. Set INFRAI_API_BASE_URL to the API origin, then provide INFRAI_API_KEY, QUEUE_NAME, and a unique REDRIVE_ID in the environment before running npx tsx redrive.ts.
const baseUrl = process.env.INFRAI_API_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
const queue = process.env.QUEUE_NAME;
const redriveId = process.env.REDRIVE_ID;
if (!baseUrl || !apiKey || !queue || !redriveId) {
throw new Error(
"Set INFRAI_API_BASE_URL, INFRAI_API_KEY, QUEUE_NAME, and REDRIVE_ID",
);
}
function retryDelayMs(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
if (retryAfter && /^\d+$/.test(retryAfter)) {
return Number(retryAfter) * 1_000;
}
return Math.min(1_000 * 2 ** attempt, 30_000);
}
async function redriveDlq(): Promise<unknown> {
const url = new URL(
`/v1/queue/dlq/redrive/${encodeURIComponent(queue)}`,
baseUrl,
);
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Idempotency-Key": redriveId,
},
});
if (response.status === 429 && attempt < 3) {
await new Promise((resolve) =>
setTimeout(resolve, retryDelayMs(response, attempt)),
);
continue;
}
const body = await response.text();
if (!response.ok) {
throw new Error(`Redrive rejected (${response.status}): ${body}`);
}
return body.length === 0 ? null : JSON.parse(body);
}
throw new Error("Rate-limit retry budget exhausted");
}
redriveDlq()
.then((result) => console.log(result))
.catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
The method is explicit, the bearer key stays outside source control, and the idempotency key remains stable across network retries. On 429, the client honors an integer Retry-After value or backs off exponentially; every other non-success response is surfaced with its body. Use a fresh REDRIVE_ID for a genuinely new operator decision, not for another attempt at the same decision.
Inspect the DLQ before approval and record who initiated the action. Let queue workers perform the actual deliveries. Cron should make only this short control-plane request, since delivery work that could approach 900 seconds belongs behind the queue.
Measure before copying this choice
Roll out with queue age, attempts per delivery, duplicate suppressions, DLQ depth, and time from reservation expiry to successful side effect on one dashboard. The last measurement is the decision metric; the others explain why it moved. Run the cron sweep slowly enough that it remains reconciliation, then verify that turning it off briefly does not stop ordinary retries.
Also test the ugly edges: a duplicate wh_01J8, a manual release while its message waits, a 429 with Retry-After, and a DLQ redrive repeated with the same idempotency key. FIFO deduplication lasts only five minutes, so it cannot replace application idempotency. Cron does not backfill triggers missed while paused, and its 4 KB output history should not be treated as the delivery audit trail.
The final rule is small. Delayed queue messages own each failed delivery, pull consumers own private workers, and cron owns only short reconciliation or approved redrive. Reconsider the design when delays exceed seven days, native fan-out or replay becomes necessary, or the retry loop turns into orchestration.
Top comments (0)