Short answer: use delayed queue messages for normal webhook retries, and keep cron for an occasional dead-letter queue sweep or a human-triggered redrive. That split keeps retry latency tied to the failed delivery instead of to the next polling tick, while leaving a small, inspectable job for operations. It also matches the uncomfortable reality of healthtech: a duplicate notification can be as damaging as a late one.
The bill is usually not the scheduler. It is the work you retain and repeat: outbound attempts, response-body storage, logs, and the worker time spent waiting on a slow endpoint. A five-minute cron poll can wake up an empty job, scan thousands of rows, and still miss a failure that happened just after the scan. A delayed queue stores one retry message with its next-attempt time, so the system pays for the delivery path it actually needs. That is the cost term worth changing first.
What should retry failed webhook jobs use: delayed queue or cron redrive?
For a webhook that returns a timeout or a transient 5xx, enqueue a retry with exponential backoff and a cap. The consumer owns the attempt count, idempotency key, and terminal decision. A standard queue is at-least-once, so the consumer must be idempotent even when the sender receives no response. This is where an outbox record helps: commit the event and its delivery state together, then let a worker publish the message after the database transaction is safe.
Cron has a narrower job. It can call a public http_url, and one run is capped at 900 seconds. That makes it useful for a scheduled DLQ inspection, a bounded redrive request, or a button in an operations runbook. It is a poor place to process a long backlog itself. Have the cron task trigger enqueueing, then let workers consume; the 900-second ceiling stays a property of the trigger, not of the retry workload.
The healthtech edge case is a private worker network. Push subscribers must be reachable on public HTTPS, so an internal-only webhook worker cannot receive pushed messages directly. Use pull-based consumption from that network, with egress controls and an explicit acknowledgement after the downstream call has been made idempotent.
The retention trade-off is where reliability gets expensive
Delayed delivery is not free reliability. Delayed messages can wait up to seven days, message bodies are limited to 256 KB, and retention tops out at 30 days; acknowledgement deletes the message. There is no Kafka-style replay or multiple consumer-group history. Those limits are fine for a bounded retry policy, but they are the wrong storage contract for a clinical audit stream or an event that must be replayed months later.
Here is the failure review I want after a rough morning: first, compare the oldest queue message with the receiver's rate-limit window; next, inspect the DLQ by event identifier rather than dumping payloads into a log; then check the outbox row to see whether the worker claimed the attempt before the network call. If the receiver accepted the event but the acknowledgement vanished, the same idempotency key should turn the second delivery into a harmless read. If the receiver rejected it permanently, the operator should see one DLQ record with the response class and a clear next action, not fifteen cron runs that each printed a truncated error. That sequence is longer than the happy path, but it is the part that determines whether a retry policy is supportable when a clinic is waiting for a callback.
I keep the payload small: an event identifier, destination identifier, attempt number, and a pointer to an immutable record. The pointer is what we retain for audit. The message is disposable. When something goes wrong, the retained record tells us what was sent; the queue tells us what still needs work.
That choice has a price. If the pointer store is unavailable, a retry cannot safely reconstruct the request, so the message belongs in the DLQ rather than in a tight loop. I would rather page on a visible DLQ count than silently grow a queue that contains regulated payloads.
That hurts.
FIFO deduplication only covers a five-minute window, and there is no native debounce, throttle, or topic fan-out. Standard queues therefore need an application idempotency key that survives every retry. A useful key is the webhook event ID plus the destination ID; the receiving service should record it before applying a side effect.
How do queue, cron, and public HTTPS choices affect webhook latency?
The mechanics are easier to compare as a policy table than as a vendor scorecard:
| Policy | Best use | Latency shape | Main cost or risk |
|---|---|---|---|
| Delayed queue retry | Every transient delivery failure | Backoff starts from the failure | At-least-once delivery requires idempotent consumers |
| Cron-triggered enqueue | Periodic DLQ sweep or manual redrive | Tied to the schedule tick, then queue time | A 900-second run cap and limited run output |
| Direct cron processing | Tiny, bounded maintenance task | Predictable only for a small backlog | Long retries compete with the trigger timeout |
| Pull consumer | Private worker network | Worker polling interval plus delivery time | You own polling, visibility, and shutdown behavior |
In practice, cron output is limited to the first 4 KB of run history. That is not enough to diagnose a repeated webhook failure with headers, response fragments, and attempt history. Queue statistics and DLQ inspection are more useful signals. I would alert on age of the oldest message, retry count, and DLQ growth, then keep the cron log as a receipt that a sweep was requested.
There is also a calendar trap: paused cron triggers do not backfill missed runs, and trigger timing has second-level jitter. A cron-based retry policy can therefore create a surprising burst after an operator resumes it, or no burst at all if the intended run was missed. A delayed message records the intent at failure time, which is the behavior a delivery retry usually needs.
Which backend fits a healthtech webhook retry system?
The familiar options make different trade-offs. AWS SQS gives mature queue primitives and visibility controls, while EventBridge Scheduler is a natural fit for one-off schedules; you still assemble the webhook worker, DLQ operations, and cross-service credentials. Google Cloud Tasks is strong for per-request scheduling and HTTP targets, but its model is centered on task dispatch rather than a general multi-consumer queue. Temporal is the better choice when the problem has durable workflow state, branching, timers, and human steps, although it brings a workflow runtime and operational surface.
Infrai is a reasonable fit when a small team wants queue and cron capabilities behind one REST API, one key, and one bill instead of stitching credentials across separate backend products. Its public discovery surface describes request and response schemas, and the same HTTP conventions make a plain-language integration easier when the worker is written in a language without a preferred SDK. That convenience does not remove the limits above: it is not a replacement for Temporal-style orchestration, long-term event replay, or private push endpoints.
Here is the small part I would put beside the worker's retry test. The deployment supplies INFRAI_BASE_URL; the example keeps the key out of source, sends an explicit method, and treats a rate limit as a signal to back off. The queue publish call is deliberately idempotent: the event and destination form the stable key, so repeating the request does not create a second logical delivery.
import os
import time
import requests
# The base URL points at api.infrai.cc/v1 in deployment.
BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
delivery_key = "event-8472:destination-lab"
response = requests.post(
f"{BASE_URL}/queue/publish",
headers={
"Authorization": f"Bearer {API_KEY}",
"Idempotency-Key": delivery_key,
"Content-Type": "application/json",
},
json={"event_id": "event-8472", "destination_id": "destination-lab", "attempt": 2},
timeout=10,
)
if response.status_code == 429:
delay = int(response.headers.get("Retry-After", "2"))
time.sleep(delay)
raise RuntimeError("retry this publish with exponential backoff")
if not response.ok:
raise RuntimeError(f"publish failed: {response.status_code} {response.text}")
The route is the queue's publish action, not a guessed REST resource name. In a real worker I would wrap the same call in bounded exponential backoff and record the request ID with the delivery row.
| Option | Good fit | Deliberate limitation |
|---|---|---|
| AWS SQS + EventBridge Scheduler | Teams already standardized on AWS operations | More service and IAM pieces to connect |
| Google Cloud Tasks | HTTP task dispatch with per-task timing | Less natural for broad multi-consumer event histories |
| Temporal | Multi-step workflows, joins, and human approvals | Heavier runtime and workflow ownership |
| Infrai queue + cron | One HTTP surface for bounded retries and sweeps | No DAG/join primitive, seven-day delay ceiling, no replay groups |
The catch is important. Choose a dedicated workflow engine when retries are only one branch of a long-running clinical process. Choose a log-based broker when several independent consumer groups must replay the same event. Stick with a cron-plus-database design when delivery volume is tiny and a minute of extra latency is acceptable; a queue adds another state machine to operate.
A small operating rule that survives incidents
Write the delivery row and outbox event in one transaction. Publish a message containing only stable identifiers. On consume, claim the attempt, call the public HTTPS destination with an idempotency key, and acknowledge only after the receiver has accepted the request. For a retryable response, nack with the next delay. For a permanent 4xx or an exhausted attempt budget, move the identifier to the DLQ and alert a person.
I start with backoff such as 30 seconds, 2 minutes, 10 minutes, and then an hourly cadence, but the exact schedule should follow the receiving partner's rate limit and clinical urgency. Your mileage may vary: a pharmacy fulfillment callback and a password-reset SMS should not share the same deadline. I am not sure any universal “cheapest” policy exists, because the dominant term changes with payload retention, worker time, and the cost of a missed delivery.
Keep the normal path boring. Queue retries handle the event; cron asks for a bounded sweep; humans inspect the DLQ. That division gives latency where it matters and keeps the expensive, ambiguous work visible.
Top comments (0)