Retry failed webhook jobs with a delayed queue, not a cron polling loop: each failed delivery keeps its own attempt state and can become eligible again without waiting for the next scan.
Short answer: use a delayed queue for each failed webhook job, apply bounded backoff, and make the consumer idempotent; reserve cron for an occasional DLQ sweep or manual redrive trigger.
This is the operational distinction that matters. The initial reminder may be scheduled against the renewal deadline. After an attempted delivery fails, the retry belongs to the delivery's state: attempt number, next eligible time, and stable delivery ID. A queue can carry that state with the individual message. Cron has to rediscover it by scanning shared storage.
Why should failed webhook jobs keep identity across delayed queue retries?
Publish one message per reminder delivery. On a retryable result, negatively acknowledge or republish that message with a delay derived from its attempt count. On success, acknowledge it. After the retry budget is exhausted, leave it in a dead-letter queue for inspection rather than resetting its history and starting another silent loop.
That mapping is direct:
| Event | Queue action | Required guardrail |
|---|---|---|
| Renewal deadline reached | Publish one delivery job | Stable delivery ID |
| Retryable webhook result | Delay the next attempt | Bounded exponential backoff |
| Successful delivery | Acknowledge the message | Durable idempotency record at the receiver |
| Retry budget exhausted | Move to or retain in the DLQ | Alert and inspect before redrive |
| Operator approves recovery | Redrive selected work | Preserve the original delivery ID |
Standard queues are at-least-once. Duplicates aren't an edge case to wish away; they are part of the contract. The webhook receiver should atomically record a stable delivery ID with the business effect, then return the same successful outcome when that ID arrives again. An idempotency header helps only when the receiver actually enforces it.
I've been paged by both missed jobs and duplicate deliveries. The postmortem lesson is usually less dramatic than the page: a retry counter was local to a process, an acknowledgement raced a crash, or a redrive generated fresh IDs. Keep the identity stable. Persist the effect before acknowledging.
For the concrete media workflow, a useful identity is derived from the subscriber, renewal period, and reminder kind. Don't derive it from the attempt number. Attempt 1 and attempt 5 are the same intended effect.
Inspect the dead-letter queue through plain HTTP
The worker needs a small, explicit state machine. A successful response acknowledges. A response that asks the caller to slow down, a timeout, or a temporary destination failure schedules another attempt. A permanent rejection goes to review without burning the full retry budget. The exact classification depends on the webhook contract; I'm not sure a generic worker can safely decide every 4xx case without that contract.
Keep it boring.
The following Go program is a runnable DLQ inspection command for Infrai. It uses one verified read route and deliberately treats the response as opaque JSON; inspection should not mutate or redrive work. Set INFRAI_BASE_URL to the API's versioned base URL, and keep the key outside source control.
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func retryAfter(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
delay := time.Second << attempt
if delay > 30*time.Second {
return 30 * time.Second
}
return delay
}
func main() {
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
key := os.Getenv("INFRAI_API_KEY")
queue := os.Getenv("QUEUE_NAME")
if baseURL == "" || key == "" || queue == "" {
fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL, INFRAI_API_KEY, and QUEUE_NAME are required")
os.Exit(2)
}
endpoint := baseURL + "/queue/dlq/list/" + url.PathEscape(queue)
client := &http.Client{Timeout: 20 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
cancel()
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
cancel()
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
cancel()
if readErr != nil {
fmt.Fprintln(os.Stderr, readErr)
os.Exit(1)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryAfter(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "request failed: status=%d body=%s\n", resp.StatusCode, body)
os.Exit(1)
}
fmt.Println(string(body))
return
}
fmt.Fprintln(os.Stderr, "rate limit retry budget exhausted")
os.Exit(1)
}
For Infrai, delayed messages are capped at 7 days, message bodies at 256 KB, and retention at 30 days. Its plain REST interface is the useful fit here: a Go worker can call it without installing or tracking a provider SDK, and the same key covers the queue and cron capabilities. The live discovery schema, available without authentication, is the authority for write request bodies.
Compare operational ownership, not sticker price
Cron is still useful as a reconciliation control. Run a low-frequency check that finds dead-lettered reminders requiring attention, or use it to invoke an approved redrive process. Do not make the cron handler process the backlog itself. A cron run is capped at 900 seconds and calls a public http_url, so the safe pattern for long work is cron trigger, queue publish, then worker consumption.
There are two other limits worth putting in the runbook. Pausing cron does not backfill missed triggers when it resumes, and cron output history retains only the first 4 KB. Those properties make cron a poor system of record for delivery recovery. Queue statistics, stable delivery records, and DLQ inspection carry the evidence an incident review needs.
Network placement changes the design too. Push subscribers must be reachable over public HTTPS. If the webhook worker is private, use pull consumption from inside the private network. Don't open an internal worker to the internet merely to preserve a push topology.
The catch is that a delayed queue is not a workflow engine. It is not suitable when a renewal process needs DAG dependencies, joins, durable multi-step orchestration, or compensation across several systems. Stick with Temporal for that workflow-shaped problem, or evaluate Airflow when the work is a scheduled data pipeline. Kafka belongs in the evaluation when retained replay and multiple consumer groups are requirements; a queue with acknowledgement deletion and at most 30 days of retention does not provide Kafka-style replay. For a small team already operating Redis, BullMQ may fit its existing operational model. AWS SQS and Google Cloud Tasks are also reasonable managed candidates to compare against the same retry, DLQ, network, and idempotency checklist.
| Candidate | Strongest reason to evaluate it here | Reason to choose something else |
|---|---|---|
| Infrai queue plus cron | Plain HTTP integration and one credential across both controls | Need delays beyond 7 days, payloads beyond 256 KB, or Kafka-style replay |
| AWS SQS | Existing AWS operational ownership | The team does not want another cloud-specific queue integration |
| Google Cloud Tasks | Existing Google Cloud operational ownership | The team needs a provider-neutral HTTP control surface |
| BullMQ | The team already owns its Redis operations | The team wants a managed service instead of owning queue storage |
| Temporal | The job is really a durable multi-step workflow | A single webhook retry state machine does not justify workflow orchestration |
Provider features and limits change. Verify each candidate's current documentation before committing; your mileage may vary with network topology and the team's on-call experience. The decision should come from failure semantics and operational ownership, not a headline price.
Roll out with a reversible redrive policy
Before enabling automatic retries, test duplicate delivery on purpose. Submit the same stable delivery ID twice and confirm that the renewal reminder produces one business effect. Then interrupt a worker after the receiver commits but before the queue acknowledgement; the repeated message should still be harmless. This is the crash window that exposes pretend idempotency.
Watch counts by attempt and outcome, queue age, DLQ depth, and the age of the oldest message. Alert on business risk: a reminder approaching its deadline deserves attention even when queue depth is low. A raw failure count without age or deadline context can look healthy while one important job waits.
Redrive in bounded batches after identifying the failed dependency and preserving the original IDs. Stop if the same failure class rises again. Never purge the DLQ as a recovery step; it destroys the evidence needed to decide whether redrive is safe.
Rollback is simple because the responsibilities are separate. Pause new automatic redrives, leave queued messages intact, and keep the initial renewal schedule running. Operators can inspect and selectively redrive after the destination recovers. If delayed retry behavior itself must be disabled, route new failures to the DLQ and page on their deadline age. No job should disappear merely because automation was rolled back.
The final acceptance check is blunt: one intended reminder, one durable business effect, even across a worker crash and a redrive.
Top comments (0)