| Recovery path | Pick it when | Main trade-off |
|---|---|---|
| Delayed queue | A failed delivery needs a precise next attempt | More moving parts and per-message state |
| Cron redrive | A sweep every few minutes is acceptable | Coarser latency and duplicate-scan work |
| Hybrid | Most retries are timely, but old records need repair | Two operating paths to observe |
Short answer: use a delayed queue for the first retries of a failed weekly digest webhook, then run a small cron redrive for records that outlive the queue's retry window. That split keeps customer-facing latency predictable while giving operators a cheap, inspectable repair path.
This is a property-management workflow. Every Monday, an application assembles a digest for active customers: new leases, maintenance updates, and open balances. The digest is stored before delivery begins. A partner's public HTTPS endpoint receives it. A timeout, a DNS problem, or a partner deploy can leave the delivery unfinished, so the scheduler needs a decision rule rather than a button marked “retry.”
How should property teams retry failed webhook jobs with delayed queues or cron?
Start with a durable delivery record. The transaction that publishes a digest also writes an outbox row containing a stable delivery ID, endpoint, payload hash, attempt count, and next-attempt time. A worker claims that row, sends HTTPS, and records the response class. The transactional outbox pattern keeps the business change and the message intent together; a separate publisher can safely pick up rows later.
Use the delayed queue for attempts one through three. A 30-second delay handles a transient timeout; a five-minute delay gives a partner deployment room to settle; a longer delay prevents a busy endpoint from being hammered. Those intervals are policy, not a promise that every endpoint will recover.
Keep it boring.
Consider Monday's run when one regional property partner returns HTTP 429 for 18 minutes while another partner times out at 10 seconds. The first delivery should wait for the partner's rate-limit window, while the second can follow exponential delay. If both records share one retry bucket, the noisy partner hides the timeout signal and the queue grows for no useful reason. Store the response class and Retry-After value on each delivery, cap the number of claims per endpoint, and let the redrive query select only records whose next-attempt time has arrived. By the time the weekly digest is due again, an operator can explain which customers were delayed, which endpoint accepted a request, and which records still need review. That audit trail is more valuable than a single global retry count.
Cron is the broom.
The redrive job queries only rows whose nextAttemptAt is in the past and whose state is retryable. It should claim a bounded page, enqueue each delivery, and move the next scan cursor forward. If a worker and the redrive job see the same row, the delivery ID and an idempotency key prevent a second accepted digest from becoming a second customer notification.
Here is a small TypeScript worker. It treats network timeouts and HTTP 5xx responses as retryable, while a 4xx response is a review state unless the partner contract says otherwise.
type Delivery = {
id: string;
url: string;
body: string;
attempt: number;
};
function nextDelay(attempt: number): number {
const seconds = Math.min(60 * 60, 30 * 2 ** attempt);
return seconds * 1000;
}
async function sendDelivery(delivery: Delivery): Promise<"sent" | "retry" | "review"> {
try {
const response = await fetch(delivery.url, {
method: "POST",
headers: {
"content-type": "application/json",
"idempotency-key": delivery.id,
},
body: delivery.body,
signal: AbortSignal.timeout(10_000),
});
if (response.ok) return "sent";
if (response.status >= 500) return "retry";
return "review";
} catch {
return "retry";
}
}
async function process(delivery: Delivery): Promise<void> {
const result = await sendDelivery(delivery);
if (result === "retry") {
await scheduleAt(delivery.id, Date.now() + nextDelay(delivery.attempt));
} else {
await markResult(delivery.id, result);
}
}
declare function scheduleAt(id: string, timestamp: number): Promise<void>;
declare function markResult(id: string, result: "sent" | "review"): Promise<void>;
The code deliberately leaves persistence behind interfaces. The database must update the attempt and claim atomically; otherwise two workers can send the same digest. The endpoint should verify a signature and treat the delivery ID as idempotent. A 200 response is not proof that the downstream job finished, only that the endpoint accepted the request.
What should observability measure before a redrive?
Measure the queue age, not just queue depth. For each delivery, emit attempt number, outcome class, endpoint host, and time from nextAttemptAt to claim. A dashboard can then show “95th-percentile retry age” beside the weekly digest's customer impact. Alert on an old row that has not moved, a rising timeout ratio, and a growing review state.
Logs need the delivery ID and a correlation ID, but never the full customer payload. Keep the payload hash so an operator can tell whether a retry is the same digest. Metrics should distinguish a partner's 429 from a local timeout; their remedies are different. I once treated both as “webhook failed,” and the alert looked calm while one partner was rate-limiting every building in a region. The lesson was simple: classify first.
For a cron redrive, record the scan duration, rows examined, rows claimed, and rows skipped because another worker owned them. A sweep that appears green while examining zero rows is not useful evidence. Add a synthetic endpoint in a staging account and check that a known delivery crosses the entire path.
Which path is easiest to operate over a month?
Delayed queues make timing explicit. They are a good fit when a customer expects a repair within minutes and the queue service already exposes visibility timeouts, dead-letter handling, and metrics. They become awkward when retention policies differ by partner or when operators need to edit thousands of scheduled messages.
Cron redrive is easier to inspect in a database. It fits low-volume digests, strict change control, and teams that want one repair query. The catch is latency: a five-minute sweep can add nearly five minutes before the next attempt, and a wide query can create load even when no delivery needs work. Stick with a queue when the endpoint has a tight recovery objective; choose a sweep when a bounded delay is acceptable.
The hybrid has a clear boundary. The queue owns fresh failures for a fixed number of attempts. Cron owns aged records, dead-letter review, and a manual “replay this delivery” action. Give both paths the same state machine and event names so an operator can follow one delivery from queued to sent, retryable, or review.
Limits and a practical rollout
This design is not suitable when a partner requires exactly-once delivery; HTTPS plus retries cannot provide that guarantee, so the receiving contract must be idempotent. It is also a poor fit for multi-megabyte reports that should be downloaded rather than pushed. Your mileage may vary on delay values: measure endpoint recovery and rate limits before changing them.
Roll out in four steps: write the outbox row in the digest transaction, ship the worker with a bounded timeout, add the redrive query in read-only mode, then enable claims with a small page size. Keep a runbook for stuck rows and a retention rule for old payloads. The success signal is not “the cron ran.” It is a traceable delivery ID, a classified outcome, and a customer who receives one digest.
Top comments (0)