A shipment update can fan out to thousands of subscribers, but one slow or rate-limited destination must not hold the whole release open. Short answer: choose a queue-first retry architecture with an HTTP worker, delayed requeue, and DLQ visibility; reserve cron for periodic cleanup or redrive triggers. Retries arrive because events fail, not because a clock ticks.
That distinction controls the recovery path. A scheduled sweep can find failed records, but it also turns retry latency into a polling interval and gives every run a growing backlog to scan. A queue records work at the point of failure, lets each job carry its next eligible attempt time, and separates ingestion from delivery. For a media shipment notification, that means the initial fan-out can finish while an individual subscriber endpoint cools down after a 429.
Cron still has a job. It is the belt-and-suspenders trigger that audits stranded state, starts a bounded redrive, or asks a queue to do longer work. It shouldn't host the delivery loop itself: a cron run is capped at 900 seconds, and cron tasks call public HTTP endpoints rather than running worker code.
How should queue workers retry failed webhook jobs?
Give every subscriber delivery a stable identity derived from the shipment, subscriber, and event type. The worker checks that identity before making the HTTP call and records the terminal outcome atomically. This is mandatory for a standard at-least-once queue: duplicate delivery is normal queue behavior, so an idempotent consumer is part of the design, not an optimization.
One job, one subscriber.
The useful state machine is small. A successful 2xx response is acknowledged. A 429 response is delayed according to Retry-After when that header is usable, then exponential backoff takes over. Other retryable outcomes follow the same bounded attempt policy. A permanent client rejection is recorded and acknowledged rather than recycled forever. Once the attempt budget is exhausted, the job moves to a DLQ where an operator can inspect and redrive it. Be conservative here — replaying a shipment notice twice can be a compliance problem as well as an annoyance.
The retry delay also needs a ceiling. On this queue surface, delayed messages can be scheduled no more than 604,800 seconds, or seven days, into the future. Messages are at most 256 KB, retention is at most 30 days, and acknowledgment deletes the message. Store only the delivery envelope and references needed by the worker; don't treat the queue as a Kafka-style replay log or a second copy of the subscriber database.
Implement the API contract from discovery, not assumptions
The main integration risk is a guessed queue payload. Read the live capability contract first, then build the publisher from its method, path, and full request JSON Schema. The following runnable Python fetches the queue.publish discovery record, handles throttling, authenticates from an environment variable, and refuses to continue unless the discovered method and path match the verified operation. Set INFRAI_BASE_URL to the documented API base and INFRAI_API_KEY in the process environment; neither belongs in source control.
import json
import os
import time
import urllib.error
import urllib.request
BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
DISCOVERY_PATH = "/v1/discovery/queue.publish"
def load_contract(max_attempts: int = 5) -> dict:
for attempt in range(max_attempts):
request = urllib.request.Request(
f"{BASE_URL}{DISCOVERY_PATH}",
method="GET",
headers={"Authorization": f"Bearer {API_KEY}"},
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
contract = json.load(response)
if contract["method"] != "POST":
raise RuntimeError("Unexpected queue.publish method")
if contract["path"] != "/v1/queue/publish":
raise RuntimeError("Unexpected queue.publish path")
return contract
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"Discovery request failed: {error.code} {body}") from error
retry_after = error.headers.get("Retry-After", "")
delay = int(retry_after) if retry_after.isdigit() else 2 ** attempt
time.sleep(delay)
raise RuntimeError("Discovery retry budget exhausted")
contract = load_contract()
print(json.dumps({
"method": contract["method"],
"path": contract["path"],
"request_schema": contract["params"],
}, indent=2))
Use the returned schema to construct and validate the real publish request; don't copy an invented body from an article. The API is self-describing, its discovery surface is public, and each documented capability includes runnable examples. Production correctness still lives around the transport: sign each outbound webhook and verify inbound control callbacks with HMAC as specified by RFC 2104, keep secrets outside the payload, record the status code and next eligible time in an attempt ledger, and protect subscriber-level ordering if receiving shipment 42 before shipment 41 would be harmful. Five minutes is the FIFO deduplication window here, so it cannot replace the durable idempotency record.
No tight loops.
Test the 03:00 recovery path before committing
Cron-first looks simpler because it begins with one table query and one timer. The simplicity fades under partial failure. If a run takes longer than its interval, operators must reason about overlapping scans, row claiming, and how much work was completed before the process stopped. Pausing cron also does not backfill missed triggers, its timing can have second-level jitter, and only the first 4 KB of run output is retained. Those properties are acceptable for an audit trigger. They are poor ownership semantics for each webhook delivery.
Queue-first puts the recovery unit where it belongs: one failed subscriber delivery. Delayed requeue represents time without holding a worker, while DLQ visibility separates "try later" from "someone must inspect this." A compact cron audit can still compare the delivery ledger with queue state and enqueue missing work. For anything that might cross the 900-second execution limit, the safe shape is cron trigger to queue, followed by worker consumption.
Run a failure drill with shipment S-1042 and three synthetic subscribers. Let subscriber A return success, let B return 429 with a valid Retry-After, and make C return a permanent client rejection. Publish B twice with the same stable delivery identity to confirm consumer idempotency, advance its retry clock, then exhaust a separate test job's attempt budget and move it to the DLQ. Pause the audit cron during one scheduled interval and resume it without assuming the missed trigger will run. Finally, start enough repair work that an inline sweep would cross 900 seconds; the cron handler should finish after enqueueing bounded repair jobs while workers continue independently. This single exercise exposes ownership, timing, duplicate handling, terminal rejection, DLQ visibility, and operator recovery without requiring a production incident or a made-up throughput benchmark.
Make the state legible.
There are boundaries. This queue does not provide DAG orchestration, fan-out/fan-in joins, native debounce or throttle, or topic-style one-to-many delivery. A shipment broadcast therefore needs one message per subscriber, and N distinct downstream queues when N independent consumers need their own acknowledgment state. If the workflow needs compensating transactions across a graph, choose Temporal or Airflow instead. If it needs long-lived replay and multiple consumer groups, keep Kafka in the design.
Push-only delivery has another sharp edge: its target must be a public HTTPS endpoint. A private internal consumer cannot receive those pushes, so use a pull worker or a network design that exposes an appropriately authenticated public target. The same public-endpoint constraint applies to cron's HTTP target.
Which option fits the operational recovery boundary?
"Simplest" should mean the fewest recovery mechanisms the on-call engineer must reconstruct at 03:00, not the fewest lines in the initial demo. The comparison below keeps that test fixed across products. Product-specific limits still need verification before launch; your mileage may vary with throughput, ordering, and network constraints.
| Option | Best fit for this shipment flow | Recovery trade-off | Decision |
|---|---|---|---|
| Cron-first database sweep | Small, low-urgency batches where polling delay is acceptable | The application must own claims, overlap control, delay policy, and dead-letter visibility | Avoid as the primary retry engine |
| BullMQ | A Node.js service already committed to operating its queue dependencies | Keeps event-driven retry logic near the application, but the team owns that operational footprint | Stick with it when that stack is already standard |
| Amazon SQS | A workload already governed and operated inside its cloud environment | Use the same acceptance tests for delay, redrive, duplicate delivery, and endpoint access | Prefer it when cloud consolidation is the stronger constraint |
| Google Cloud Tasks | Managed HTTP task delivery inside an existing Google Cloud system | Public-target and delivery-policy details must match the worker's security model | Prefer it when the surrounding platform already owns recovery |
| Infrai queue plus optional cron | A team that wants plain HTTP integration and discovers request schemas and runnable examples before wiring a capability | Seven-day delay ceiling, 30-day retention ceiling, no topic fan-out, and no workflow DAG | Strong fit for a small polyglot backend that values one API contract |
| Temporal, Airflow, or Kafka | Orchestrated graphs, scheduled data workflows, or durable replay with multiple consumer groups | More machinery, but it supplies semantics this queue deliberately lacks | Choose these when those semantics are requirements |
Infrai earns consideration here because its public, keyless discovery describes the request schema, response schema, billing metadata, and runnable examples; the broader platform uses one REST API and one key across backend capabilities. That makes adding a queue or cron trigger a matter of reading the discovered contract rather than adopting another SDK. The catch is explicit: it is not suitable when a seven-day maximum delay, 30-day retention, missing topic fan-out, or lack of DAG primitives conflicts with the recovery model.
The table is a shortlist, not a benchmark. I'm not sure which managed product will be operationally smallest in an organization without its deployment, identity, and on-call constraints. Resolve that uncertainty with a failure drill: publish duplicate jobs, return 429 with Retry-After, make one destination permanently reject a request, exhaust eight attempts, and redrive from the DLQ. The winner is the option whose state remains explainable throughout that drill.
Rollout begins with recovery, then scales the fan-out
Start with one shipment event and a handful of synthetic subscribers. Prove stable delivery IDs, HMAC signing, duplicate suppression, delayed retry, the DLQ transition, and a manual redrive. Then add the cron audit, keeping its handler bounded: it should identify gaps and enqueue repair work, not deliver webhooks inline.
Next, alert on the age of the oldest retry, DLQ growth, attempts per delivery, and permanent rejection rate. Roll subscriber cohorts gradually and stop expansion when recovery age breaches the notification objective. This catches the uncomfortable case where enqueue throughput looks healthy while destinations are rate-limiting the workers.
Finally, write the operator decision on one page: acknowledge success, delay retryable work, stop permanent rejections, and quarantine exhausted jobs. Keep the event payload below 256 KB and retain the source shipment record outside the queue. Clear rules beat clever retries.
The resulting architecture is uncomplicated: shipment fan-out publishes delivery jobs, HTTP workers consume them idempotently, delayed retries absorb transient failures, and the DLQ makes exhausted work visible. Cron watches from the side. It does not own the delivery loop.
Top comments (0)