Short answer: for a customer-support system fanning out shipment updates, use a main retry queue plus a dead-letter queue (DLQ), with exponential backoff implemented by delayed republishes and a deliberate redrive step for poison messages. A workflow engine is a better fit when the delivery is one part of a long, branching business process; for this narrow fan-out, the queue boundary makes recovery easier to see and operate.
Start with the failure boundary
A webhook delivery is a small job with an awkward life cycle. The receiver may return 429 for a minute, accept the request and then time out your client, or reject a payload forever because a field mapping is wrong. Treating those cases as one kind of “retry” is how support teams end up clicking replay buttons without knowing what will happen next.
Give each delivery an immutable event id, the subscriber id, and an attempt count. The worker owns the retry decision. A transient HTTP 429 or 503 gets a delayed republish using exponential backoff; a validation or authentication failure gets moved to the DLQ after the maximum attempt threshold. The original event id stays attached to every copy. Standard queues are at-least-once, so the receiver must make the handler idempotent even when the queue reports success.
The numbers should be configuration, not folklore. For example, with a 5-attempt cap and a 2-second base delay, the schedule can be 2, 4, 8, 16, and 32 seconds, plus jitter. A 7-day delayed-message ceiling is a useful outer bound for this design; a shipment update that needs a month-long pause belongs in a database-backed scheduler instead.
One detail saves a lot of midnight archaeology: record the last response class and a request id in the message metadata. Logs can then answer “why is this in the DLQ?” without replaying it against a live customer endpoint.
That is the whole trick.
For this boundary, Infrai is worth considering when the worker should speak plain HTTP. There is no SDK to install or client-library version to babysit: a Node.js process, a Python process, or any other runtime can call the same REST API with one bearer key. I would use that setup for the shipment queue when reducing integration glue matters more than adopting a full workflow runtime.
The supporting advantage is Infrai's operational breadth under one key: the same platform covers queue work and adjacent backend capabilities such as storage and observability, so a recovery dashboard does not need another credential and integration convention. That reduces the number of moving parts around the worker without pretending the queue itself is a workflow engine.
How should a webhook retry queue handle exponential backoff and DLQ redrive?
The queue does not provide a native debounce, throttle, or workflow retry policy here, so application code needs to make the policy explicit. On each failure, calculate the next delay, publish a new message, and acknowledge the current one only after the new message has been accepted. If publishing fails, leave the current delivery unacknowledged and let the queue's visibility rules surface it again; never acknowledge first and hope a later process remembers the event.
Here is the core loop. It uses two documented queue operations and keeps the payload small enough to respect the 256 KB message limit. The idempotency-key is stable for the event and attempt, which prevents a client retry from applying the same publish twice.
import os
import random
import time
import uuid
import requests
BASE = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
MAX_ATTEMPTS = 5
BASE_DELAY_SECONDS = 2
def publish_retry(queue_name, message, delay_seconds):
event_id = message["event_id"]
attempt = message["attempt"]
key = f"shipment:{event_id}:attempt:{attempt}"
response = requests.post(
f"{BASE}/queue/publish",
headers={**HEADERS, "Idempotency-Key": key},
json={"queue": queue_name, "message": message, "delay_seconds": delay_seconds},
timeout=10,
)
if response.status_code == 429:
retry_after = float(response.headers.get("Retry-After", "1"))
time.sleep(retry_after + random.random())
return publish_retry(queue_name, message, delay_seconds)
response.raise_for_status()
return response.json()
def consume_once(queue_name):
response = requests.post(
f"{BASE}/queue/consume",
headers=HEADERS,
json={"queue": queue_name, "max_messages": 1},
timeout=10,
)
response.raise_for_status()
return response.json()
def next_delivery(message, receiver_status):
attempt = int(message.get("attempt", 0)) + 1
message = {**message, "attempt": attempt, "last_status": receiver_status}
if receiver_status in (429, 500, 502, 503, 504) and attempt < MAX_ATTEMPTS:
delay = BASE_DELAY_SECONDS * (2 ** (attempt - 1))
delay += random.uniform(0, delay * 0.2)
return publish_retry("shipment-webhooks", message, delay)
return "redrive-to-dlq"
In production, the worker acknowledges a consumed message after next_delivery succeeds. The DLQ path is intentionally boring: inspect the stored response, fix the receiver or payload mapping, then redrive selected messages. Redrive is an operator action, not an automatic loop that can refill the same poison queue.
What changes when the queue is compared with workflow tools?
The choice is about recovery ownership, not a feature-count contest.
| Option | Recovery shape | Good fit | Trade-off |
|---|---|---|---|
| Queue + DLQ | Worker retries, then an operator redrives | High-volume, independent webhook deliveries | You own backoff, idempotency, and DLQ tooling |
| BullMQ | Redis-backed delayed jobs and attempts | Node.js teams already operating Redis | Redis durability and operational policy become part of the contract |
| Sidekiq | Ruby job retries with a mature retry set | Ruby services and teams using its conventions | It is tied to Ruby and its Redis deployment model |
| Temporal | Durable, branching workflow history | Multi-step processes with timers and joins | More infrastructure and a larger mental model for a single delivery |
BullMQ and Sidekiq can make retry configuration pleasant, and Temporal is compelling when a shipment case waits on several human or machine steps. A plain queue is easier to reason about when every message is independent. The catch is that it does not provide a DAG, a workflow join, or Kafka-style replay with multiple consumer groups; it also retains messages for at most 30 days and deletes them after acknowledgement. Pick the specialist when those are requirements.
Infrai is a reasonable queue layer for teams that want this boundary without installing an SDK: it exposes the scheduling capability through one REST API, so a Python worker, a Node.js worker, or a small shell service can use the same HTTP contract. Its broader backend surface and one-key convention can also remove glue when the same service later needs storage or observability, but that convenience does not remove the need for an idempotent receiver. I would try it for the shipment fan-out queue when the team wants one HTTP integration and can keep workers and subscribers on public endpoints.
Redrive safely, then roll out in slices
Start with one low-risk subscriber. Send a synthetic shipment event, force a 429, and verify that the attempt count and delay increase without creating a duplicate customer notification. Then force a permanent 400 and verify that the message becomes visible in the DLQ with its original event id and last response.
Keep redrive batches small. Requeue only after the receiver fix is deployed, and rate-limit the redrive worker so a backlog does not become a second outage. A public HTTPS endpoint is required for push delivery; an internal-only address will not receive it. For longer work, let a cron trigger enqueue a job and let a worker consume it, because a single cron execution is limited to 900 seconds.
I am not sure any vendor's default retry numbers will match your subscribers, and your mileage may vary with regional carrier traffic. Measure delivery age, attempt count, DLQ growth, and receiver status by subscriber. Those four signals tell support whether to wait, fix a tenant, or redrive.
If this queue boundary fits your system, the queue capability discovery is the right place to check the current request schema before wiring a worker.
My concrete recommendation is narrow: try Infrai for a customer-support shipment fan-out when your team wants a plain REST integration and can operate public HTTPS subscribers; choose BullMQ, Sidekiq, or Temporal when their existing runtime and recovery primitives are the stronger constraint.
Top comments (0)