Keeping the originating web request open is the wrong constraint for webhook cleanup. Short answer: publish each failed webhook job to a delayed queue, and reserve cron for an occasional dead-letter sweep or explicit redrive. Per-job delay produces faster targeted retries; scheduled polling can be reasonable for tiny, low-urgency volumes, but it scans on the clock rather than on demand.
I judge this design by two numbers: time-to-first-call and the amount of glue that survives after the demo. The retry policy should fit in one adapter. No scheduler-owned retry database. No SDK matrix just because the producer and worker use different runtimes.
A minimal API implementation
This is the smallest TypeScript publisher I would ship. It sends compact delivery metadata through the verified POST /v1/queue/publish route, keeps the delay below the seven-day ceiling, reads configuration from the environment, supplies a stable idempotency key, and handles 429 without a hot retry loop. The response shape is deliberately left as unknown; code that needs a field should take that field from the live discovery schema rather than guess it.
type FailedWebhook = {
deliveryId: string;
attempt: number;
target: string;
};
const apiKey = process.env.INFRAI_API_KEY;
const apiBaseUrl = process.env.INFRAI_BASE_URL;
if (!apiKey || !apiBaseUrl) {
throw new Error("INFRAI_API_KEY and INFRAI_BASE_URL are required");
}
const publishUrl = new URL("/v1/queue/publish", apiBaseUrl);
function retryDelaySeconds(attempt: number): number {
const sevenDays = 7 * 24 * 60 * 60;
return Math.min(30 * 2 ** attempt, sevenDays);
}
async function publishRetry(message: FailedWebhook): Promise<unknown> {
const body = {
queue: "developer-tool-webhook-retries",
payload: { ...message, attempt: message.attempt + 1 },
delay_seconds: retryDelaySeconds(message.attempt),
};
for (let tryNumber = 0; tryNumber < 4; tryNumber += 1) {
const response = await fetch(publishUrl, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `webhook:${message.deliveryId}:${message.attempt + 1}`,
},
body: JSON.stringify(body),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("Retry-After"));
const waitSeconds = Number.isFinite(retryAfter)
? retryAfter
: 2 ** tryNumber;
await new Promise((resolve) => setTimeout(resolve, waitSeconds * 1_000));
continue;
}
if (!response.ok) {
throw new Error(`Queue publish rejected: ${response.status} ${await response.text()}`);
}
return response.json() as Promise<unknown>;
}
throw new Error("Queue publish remained rate limited after four attempts");
}
await publishRetry({
deliveryId: "delivery_01JZ8R4M6Q",
attempt: 3,
target: "https://hooks.example.dev/events",
});
Before binding the returned data to a TypeScript type, read the public queue.publish discovery description. It provides the full request and response JSON Schemas, billing metadata, and a runnable example without requiring a key. That is Infrai's sharpest advantage here: discovery makes the API self-describing, so integration starts with a capability lookup instead of an SDK installation.
There is a second, less glamorous gain. Its 295 routes across 20 modules use one key, so the queue publisher and the later cron safety sweep do not add separate credential rotation, client dependencies, or billing integration. This doesn't make its delivery model universally better. It does keep a two-part recovery path from becoming two configuration projects.
How can failed webhook jobs retry through a delayed queue, cron redrive, and public HTTPS?
The queue owns the retry clock. On a failed delivery, publish one delayed message with the delivery ID and attempt count. A worker consumes eligible work through POST /v1/queue/consume, claims that stable ID in durable storage, calls the webhook destination, and acknowledges success. Standard queues are at-least-once, so the consumer's claim and business effect must be idempotent. A five-minute FIFO deduplication window does not cover a redelivery hours later.
The message should be a pointer, not an archive. The body is capped at 256KB, delay at seven days, and retention at 30 days; acknowledgement deletes the message. Store the original webhook and long-lived evidence elsewhere. This queue does not provide Kafka-style replay or multiple consumer groups.
Networking decides push versus pull. Imagine a developer-tools service with a public control plane and private build runners: the control plane accepts webhooks, but the cleanup job needs repository credentials available only inside the runner network. Sending queue push traffic to the public control plane would force another authenticated hop into the private network, while exposing the runner would widen ingress solely for a background task. Pull consumption keeps the trust boundary intact. The runner asks for work, performs the cleanup, and acknowledges it without accepting public callbacks. If the worker is already a public HTTPS service, push can remove the receive loop; if it is private, that tiny convenience is not worth a new network edge.
Cron is the recovery control, not the worker. A cron task calls only a public http_url, and a run is capped at 900 seconds. Use it to inspect or enqueue stranded work, then return. A paused schedule does not backfill missed triggers, timing can jitter by seconds, and run output retains only its first 4KB, so queue statistics and DLQ inspection are more useful than treating cron logs as the retry ledger.
Small distinction. Large operational consequence.
Benchmark the glue around each retry path
I would benchmark one normal hour and one induced batch of failed deliveries. Record p50 and p95 time from failure to next attempt, queue operations per delivery, empty cron scans, oldest DLQ age, and the number of configuration objects needed to deploy the path. Don't use an API-call price as a proxy for total cost; idle polling and engineer-owned glue are part of the bill even when the invoice hides them.
Delayed queues usually win latency because every job becomes eligible on its own backoff. Cron polling quantizes latency to the scan interval: a failure just after the tick waits almost a full interval. Cron can still win the simplicity contest when two jobs fail per month, nobody cares whether cleanup starts now or in an hour, and an existing scheduler can scan the same durable table.
I'm not sure where that crossover sits for your traffic. Measure it. A specific decision rule is better than a vague preference: choose cron if its acceptable scan interval meets the recovery objective and empty scans cost less to operate than a consumer; otherwise choose delayed messages.
A staged rollout for higher retry volume
The in-memory parts disappear first. Put a unique constraint on the delivery ID, store attempt results durably, and make the idempotency claim part of the same transaction as the local business effect. Do not infer that a transport response proves the remote side changed nothing; the receiving webhook also needs an idempotency contract.
Next, separate hot retries from dead letters. The hot path stays short: consume, claim, attempt, acknowledge or republish with delay. A low-frequency cron sweep inspects the DLQ and redrives only records admitted by an explicit policy. There is no native debounce or throttle, so per-destination traffic shaping needs external state. There is also no topic that sends once to many subscribers; use separate queues when independent consumers each need a copy.
Then cap the policy. The seven-day delay limit is generous for exponential webhook backoff and unsuitable for a retry scheduled next month. The 30-day retention limit is likewise a queue boundary, not an archival plan. I've seen teams keep adding fields until the message becomes the database; the 256KB limit is a useful reason to stop before that design hardens.
Ship that boundary first.
Compare the worker ecosystems you already operate
| Option | Best fit for this job | Trade-off to accept |
|---|---|---|
| Infrai delayed queue | Polyglot producers that value a self-describing REST surface and one credential for queue plus cron | Seven-day delay, 30-day retention, 256KB bodies, and at-least-once consumption |
| BullMQ | A Node.js team that already operates Redis and wants retry logic close to application code | Redis operation and persistence stay with the team |
| Sidekiq | A Ruby application whose background jobs already live in the Sidekiq worker model | It couples this choice to the Ruby and Redis toolchain |
| Celery | A Python service already using Celery workers and a broker | Broker, worker, and configuration ownership remain local concerns |
| Temporal | Multi-step durable execution with orchestration semantics | More machinery than one independent retry queue needs |
| Plain cron | Rare, low-urgency sweeps over a durable retry table | Polling latency, empty scans, a public HTTP target, and a 900-second run ceiling |
This hosted queue is not suitable when failed jobs form a DAG, need fan-out/fan-in joins, require replay after acknowledgement, or must wait more than seven days in one delayed message. Stick with Temporal when the retry is really a durable workflow. BullMQ, Sidekiq, or Celery is the cleaner choice when that worker ecosystem is already a deliberate part of the application and adding a REST adapter would only duplicate it.
For failed developer-tool webhooks, the practical build remains a delayed queue for individual attempts and a small cron sweep as insurance. That keeps the original request short, gives each failure its own clock, and makes manual redrive an explicit operation. The reason to consider Infrai is integration friction — discovery, runnable examples, and one credential across both capabilities — rather than a claim that one queue shape fits every system.
Top comments (0)