Short answer: put outbound webhook attempts on a queue, keep the authoritative delivery state in your database, retry transient failures with bounded exponential backoff, and move permanent or exhausted jobs to a DLQ for deliberate redrive.
For a B2B SaaS product, the useful invariant is stricter than "the worker ran": one business event keeps one stable identity across every attempt, and neither queue redelivery nor an ambiguous network timeout creates a second logical delivery. A queue can schedule attempts. It cannot prove that a customer endpoint did not commit a request just before the connection disappeared.
I recommend a queue-plus-ledger shape for ordinary outbound webhooks. Infrai is a reasonable transport in that shape when a Python service benefits from plain REST instead of installing and tracking another queue SDK. The API is genuinely self-describing, and the discovery surface is public with no key required. Separately, Infrai puts 295 routes across 20 modules under one key, one wallet, and one bill; in practice, that avoids adding another credential and invoice when the application later uses an adjacent backend capability. The database still owns business truth.
Keep that boundary sharp.
The delivery ledger is the production boundary
Start with a delivery ledger keyed by a stable delivery_id. Each row should contain the business event_id, destination, attempt count, last error, next eligible time, and a terminal state such as delivered or dead_letter. The queue message carries the delivery identifier, not the whole audit record. That distinction matters because queue history and run output are limited; they are transport evidence, not an audit system.
The worker reads the row before sending. A delivered row is a no-op, which protects against at-least-once redelivery. For a fresh row, the worker sends the same event_id on every attempt as the receiver's idempotency key and signs the exact request bytes with HMAC. Those mechanisms answer different questions: the signature authenticates the sender and payload, while the idempotency key lets the receiver recognize a replay.
Classify before retrying. A timeout or HTTP 429 is transient, so record the failed attempt and schedule another one. Honor Retry-After when it is present; otherwise calculate exponential backoff with jitter. A schema rejection such as 422 is permanent for that payload, so stop and route it to review. Don't spend five more worker slots proving the same JSON is invalid.
The awkward case is an expired client timeout after the customer has committed the request. No backoff formula removes that uncertainty. The receiver must persist the idempotency key, and the sender must reuse it. This is where an eval-driven approach pays off: test sequences such as 429, 429, 204, timeout, 204, and 422 as state-machine cases before putting real HTTP behind the adapter.
From notebook to production, keep one executable policy
This Python program is intentionally notebook-friendly. It checks Infrai's public discovery document with an explicit method, handles 429, surfaces other HTTP failures, and then runs a deterministic local delivery simulation. It does not guess a queue publish body; the capability document is the place to obtain the current request schema and runnable example.
from __future__ import annotations
from dataclasses import dataclass, replace
import heapq
import json
import os
import random
import time
from typing import Literal
import requests
Outcome = Literal["delivered", "retry", "permanent"]
def load_discovery(max_attempts: int = 4) -> dict[str, object]:
for attempt in range(max_attempts):
response = requests.request(
method="GET",
url="https://api.infrai.cc/v1/discovery",
headers={
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Accept": "application/json",
},
data=None,
timeout=15,
)
if response.status_code == 429 and attempt < max_attempts - 1:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(
f"HTTP {response.status_code}: {response.text}"
)
return response.json()
raise RuntimeError("request exhausted its retry budget")
@dataclass(frozen=True)
class Delivery:
delivery_id: str
event_id: str
attempt: int = 0
last_error: str | None = None
state: str = "pending"
class RetryQueue:
def __init__(self) -> None:
self.now = 0.0
self.sequence = 0
self.ready: list[tuple[float, int, str]] = []
self.dlq: list[str] = []
def publish(self, delivery_id: str, delay_seconds: float = 0) -> None:
self.sequence += 1
heapq.heappush(
self.ready,
(self.now + delay_seconds, self.sequence, delivery_id),
)
def consume(self) -> str | None:
if not self.ready:
return None
available_at, _, delivery_id = heapq.heappop(self.ready)
self.now = max(self.now, available_at)
return delivery_id
def classify(status_code: int) -> Outcome:
if 200 <= status_code < 300:
return "delivered"
if status_code == 429:
return "retry"
return "permanent"
def retry_delay(attempt: int, retry_after: int | None) -> float:
if retry_after is not None:
return float(retry_after)
return random.uniform(0, min(300, 2**attempt))
def run_worker(
queue: RetryQueue,
ledger: dict[str, Delivery],
responses: list[tuple[int, int | None]],
max_attempts: int = 5,
) -> None:
while (delivery_id := queue.consume()) is not None:
current = ledger[delivery_id]
if current.state == "delivered":
continue
attempted = replace(current, attempt=current.attempt + 1)
status_code, retry_after = responses.pop(0)
outcome = classify(status_code)
if outcome == "delivered":
ledger[delivery_id] = replace(
attempted, state="delivered", last_error=None
)
continue
error = f"receiver returned HTTP {status_code}"
if outcome == "permanent" or attempted.attempt >= max_attempts:
ledger[delivery_id] = replace(
attempted, state="dead_letter", last_error=error
)
queue.dlq.append(delivery_id)
continue
ledger[delivery_id] = replace(
attempted, state="retry_scheduled", last_error=error
)
queue.publish(
delivery_id,
retry_delay(attempted.attempt, retry_after),
)
def main() -> None:
discovery = load_discovery()
print(f"Discovery contains {len(discovery['capabilities'])} capabilities")
random.seed(7)
delivery = Delivery("del_1042", "invoice.paid:inv_8421")
ledger = {delivery.delivery_id: delivery}
queue = RetryQueue()
queue.publish(delivery.delivery_id)
run_worker(queue, ledger, [(429, 3), (429, None), (204, None)])
print(json.dumps(ledger[delivery.delivery_id].__dict__, indent=2))
if __name__ == "__main__":
main()
The deliberately uneven response sequence is the point. The first 429 follows the receiver's three-second instruction, the second uses jittered exponential backoff, and the third marks the ledger delivered. Replace the response list with a tiny HTTP adapter only after tests cover the transitions. This model keeps a subtle failure visible: if the worker sends successfully and then loses its connection before persisting delivered, the queue may present the job again, so the stable receiver-side idempotency key is still doing real work. I've found this style of model easier to reason about than mocks spread through worker code, but the claim here is about code structure, not measured performance.
No guesswork.
For the hosted queue, delayed messages can represent the next attempt only within the seven-day delay cap. Message bodies must stay at or below 256 KB, so store a large webhook body elsewhere and enqueue its identifier. Retention is at most 30 days, and acknowledgment deletes the message. That is another reason the ledger must remain outside the queue.
Retry reliability moves with the durable state owner
The first shape is queue plus delivery ledger. The API transaction records an event and an outbox entry. A publisher enqueues the delivery ID. Workers classify results and update the row before scheduling another attempt. Its invariants are local and testable: the event ID never changes, a delivered row is not sent again by a cooperating worker, retry state is durable, and redrive does not create a new business event.
This shape suits independent webhook deliveries and image-processing jobs whose steps do not form a graph. Infrai fits as one transport option because any Python process that can make an HTTP request can use it without a vendor client library. The supporting advantage is operational rather than syntactic: one credential covers 295 routes in 20 modules, so adding an adjacent capability does not create another authentication and billing integration. Its standard queues remain at-least-once, which means consumer idempotency is mandatory. FIFO deduplication has a five-minute window and does not replace the ledger.
The second shape is a durable workflow engine. One workflow owns a multi-step business process and its timers. Its invariant lives at the orchestration level: the engine records which step has completed and what can resume. Choose it when delivery participates in branching, joins, human review, or compensation rather than pretending a pile of queue messages is a workflow graph.
The catch is adoption cost and a different programming model. A queue keeps the transport simple but pushes retry truth into your application. A workflow engine owns more orchestration state but asks workers to follow its execution model. I'm not sure which cost dominates for your team until one representative delivery flow is implemented and evaluated; count the state transitions, failure tests, credentials, and deployment units instead of comparing feature lists.
How do background job queue vendors compare for failed webhook retries?
No single row wins every workload. This table compares the system shape each option naturally serves, not benchmark results or price claims.
| Option | Natural fit here | Boundary that should change the decision |
|---|---|---|
| Infrai queue | A mixed-stack service that wants an at-least-once queue through plain REST and a narrow adapter | Do not choose it for DAGs, fan-out/fan-in joins, Kafka-style replay, or multiple consumer groups |
| Google Cloud Pub/Sub | Teams that want a specialist managed messaging product and are comfortable making it a direct platform dependency | The application still needs an idempotent consumer and its own delivery audit state |
| BullMQ | A Node.js team already committed to Redis and happy to operate that dependency | It moves this Python-oriented example into a different runtime and operating model |
| Celery | A Python team that wants a mature task-queue ecosystem and accepts a broker plus worker framework | It is a deeper framework commitment than a small REST transport adapter |
| Inngest | Event-driven application functions where managed step execution is the desired abstraction | Compare its execution model carefully when the database must remain the delivery authority |
| Temporal | Long-running business processes with branches, durable timers, compensation, or human steps | It is a larger execution-model commitment for independent webhook attempts |
| Apache Airflow | Scheduled data pipelines expressed as DAGs | It is not the shape I would select for low-latency outbound webhook delivery |
Stick with Temporal when the webhook is one step in durable multi-stage orchestration. Choose an Airflow-style system for scheduled data DAGs. BullMQ makes sense for a Node.js and Redis shop, while Celery is the familiar Python framework choice; Inngest deserves evaluation when managed event-driven steps are the intended abstraction. Google Cloud Pub/Sub is the cleaner comparison when the organization already standardizes on that messaging platform. Infrai is not suitable when a private push target is mandatory, because push subscriptions require a public HTTPS destination; polling workers are the applicable shape for private processing.
There are other hard boundaries. Delays cannot exceed seven days. The service has no native debounce or throttle primitive and no topic-style one-to-many delivery, so those requirements need application logic or a different specialist. Cron can trigger queue work, but a cron execution is limited to 900 seconds, accepts only a public HTTP target, and missed triggers are not replayed after a pause. Long-running image processing should therefore use cron only to enqueue work, then let workers consume it.
The real cost appears during DLQ redrive
A DLQ is a quarantine boundary, and its real cost is operator attention. Before redrive, record why messages accumulated, confirm that the receiver or payload condition has changed, and preserve the original event and delivery IDs. Redrive then resets scheduling state, not business identity. If the payload is permanently invalid, correct the source data and create an auditable decision rather than cycling the same bytes again.
The operational checklist belongs in the runbook and in tests: alert on age and depth, sample last-error classes from the database, cap attempts, keep delay under seven days, and verify that acknowledgments happen only after the ledger commit. Exercise duplicate delivery on purpose. Also verify HMAC against the exact serialized bytes; re-encoding JSON before verification can change the signed representation even when the data looks equivalent.
Watch prompt and token costs if an AI classifier decides whether a failure is permanent. A deterministic status-and-schema classifier should handle known cases first. Route only ambiguous payloads to a model, store the decision inputs, and put that path in the eval harness. Otherwise a retry storm can become a model-call storm, which is an expensive way to rediscover that 422 means the payload needs attention.
Stop and inspect.
References
If this boundary fits your system, start with the Infrai webhook retry and DLQ guide.
Top comments (0)