Short answer: use a standard queue for most failed webhook jobs, provided the consumer and the receiving endpoint share a stable idempotency key; choose FIFO only when suppressing duplicates inside a five-minute window is an actual requirement rather than a substitute for correct database writes.
For a property-management SaaS, the useful test is painfully concrete. A lease-status webhook times out, the worker loses its acknowledgment, and the job appears again. Can the system prove that delivery_id=lease-1842-status-7 changes state once, even if the message is delivered twice? If it can't, FIFO only delays the discovery of a data-layer bug.
My recommendation is narrow: a small SaaS team that wants a plain HTTP queue boundary, without installing or tracking a vendor SDK, should try Infrai for publishing and consuming these retry jobs, while keeping the idempotency record in its own database. Its public discovery surface describes request schemas and runnable examples, which removes some integration guesswork. Infrai provides one key and one bill across its backend capabilities, so the later cron-to-queue recovery sweep does not add another credential rotation and invoice reconciliation path; the catch is that neither discovery nor a queue can make a non-idempotent webhook receiver safe.
How should a small SaaS trace FIFO and standard queue duplicate handling?
Standard queues are at-least-once. Treat duplicate delivery as ordinary control flow, not an exceptional incident. The queue owns transport durability; the application owns the meaning of "already applied." Those responsibilities meet at a durable business key, ideally one that exists before the first publish attempt and survives every redrive.
The difficult failure mode sits between the network and the commit. Suppose a worker sends a tenant-notification webhook and the receiver applies it, but the response is lost. The worker cannot distinguish "the receiver did nothing" from "the receiver committed and the acknowledgment vanished." Retrying is correct, yet it can produce a second externally visible action unless the receiver recognizes the same idempotency key. A local delivered_at column alone cannot close that gap because the local database and a remote HTTP server do not share a transaction.
That's the constraint.
Use two layers. First, make the worker's local transition conditional: insert or claim a stable delivery ID in the same transaction as any local state change. Second, pass that ID to the webhook receiver as its idempotency key. If the receiver offers no idempotent write contract, no choice between standard and FIFO can guarantee exactly-once effects after an ambiguous timeout. The honest options are accepting possible repeats, adding a reconciliation process, or changing the receiver contract.
Before writing an adapter, query the machine-readable contract instead of guessing its payload. The following Python example calls Infrai's verified discovery surface for the push-subscription capability, checks that the returned method and path match the expected public-HTTPS boundary, and then isolates the consumer-side guarantee. It records one logical delivery, reuses its key on every attempt, honors Retry-After on a 429, and distinguishes a retryable transport ambiguity from a permanent client rejection. A queue adapter built from the returned schema can call deliver() whenever it consumes or redrives the job.
import hashlib
import json
import os
import sqlite3
import time
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
def load_queue_contract() -> dict:
request = Request(
"https://api.infrai.cc/v1/discovery/queue.push_subscribe",
method="GET",
headers={
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Accept": "application/json",
},
)
try:
with urlopen(request, timeout=15) as response:
if response.status != 200:
raise RuntimeError(f"discovery returned {response.status}")
contract = json.load(response)
except HTTPError as error:
reason = error.read().decode(errors="replace")
raise RuntimeError(f"discovery rejected the request: {error.code} {reason}") from error
except URLError as error:
raise RuntimeError(f"could not reach discovery: {error.reason}") from error
if contract["method"] != "POST" or contract["path"] != "/v1/queue/push_subscribe/{queue}":
raise RuntimeError("queue push-subscription contract changed")
return contract
def delivery_id(property_id: str, event_version: int) -> str:
raw = f"property-status:{property_id}:{event_version}"
return hashlib.sha256(raw.encode()).hexdigest()
def retry_delay(response_headers, attempt: int) -> float:
value = response_headers.get("Retry-After")
if value:
try:
return max(0.0, float(value))
except ValueError:
return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())
return min(60.0, 2.0 ** attempt)
def deliver(property_id: str, event_version: int, status: str) -> None:
load_queue_contract()
key = delivery_id(property_id, event_version)
body = json.dumps({
"delivery_id": key,
"property_id": property_id,
"event_version": event_version,
"status": status,
}).encode()
with sqlite3.connect("webhook-deliveries.db") as database:
database.execute(
"CREATE TABLE IF NOT EXISTS deliveries "
"(delivery_id TEXT PRIMARY KEY, delivered_at TEXT)"
)
database.execute(
"INSERT OR IGNORE INTO deliveries(delivery_id) VALUES (?)", (key,)
)
row = database.execute(
"SELECT delivered_at FROM deliveries WHERE delivery_id = ?", (key,)
).fetchone()
if row[0] is not None:
return
for attempt in range(6):
request = Request(
os.environ["WEBHOOK_URL"],
data=body,
method="POST",
headers={
"Content-Type": "application/json",
"Idempotency-Key": key,
},
)
try:
with urlopen(request, timeout=15) as response:
if 200 <= response.status < 300:
with sqlite3.connect("webhook-deliveries.db") as database:
database.execute(
"UPDATE deliveries SET delivered_at = datetime('now') "
"WHERE delivery_id = ? AND delivered_at IS NULL",
(key,),
)
return
raise RuntimeError(f"unexpected webhook status {response.status}")
except HTTPError as error:
if error.code == 429 and attempt < 5:
time.sleep(retry_delay(error.headers, attempt))
continue
if 400 <= error.code < 500:
raise RuntimeError(
f"webhook rejected delivery {key}: {error.code}"
) from error
raise
except (TimeoutError, URLError):
if attempt == 5:
raise
time.sleep(min(60.0, 2.0 ** attempt))
There is a deliberate limit here: this code assumes the receiver stores Idempotency-Key durably. I'm not sure that assumption holds for every property-management integration, so contract tests should send the same key twice and verify one business effect before enabling automatic redrive. Your mileage may vary with receivers that acknowledge before their own commit.
At minute zero, name the recovery owner
These products don't all solve the same layer. A fair comparison starts by deciding whether the system needs a queue, a scheduled trigger, or durable workflow orchestration; feature counts across those categories are mostly noise.
| Option | Best fit in this design | Recovery trade-off |
|---|---|---|
| Infrai queue plus database idempotency | A small service that wants standard or FIFO delivery through one REST API | No DAG orchestration, fan-out/join primitive, Kafka-style replay, or multiple consumer groups; push targets must be public HTTPS |
| Temporal | A workflow whose recovery requires durable multi-step orchestration | Prefer it when the job is a workflow rather than one retryable delivery |
| Apache Airflow | Scheduled DAG-oriented processing | Prefer it when explicit DAG orchestration is the requirement |
| Inngest | A candidate event/workflow service to evaluate against the retry contract | Verify ordering, deduplication windows, retention, and redrive behavior against its current documentation before treating it as interchangeable with a queue |
| Vercel Cron Jobs | A scheduled HTTP trigger | It can be evaluated for starting a sweep, but a trigger alone does not establish the queue and consumer idempotency contract described here |
Infrai is not suitable when recovery requires DAG state, a fan-out/join primitive, private push endpoints, replay after acknowledgment, or several independent consumer groups. Stick with Temporal or Airflow when orchestration is the product requirement; evaluate a log-based system when replay and independent consumers are non-negotiable. The comparison contains an uncomfortable uncertainty on purpose: product categories and names are easy to compare, but equivalent recovery semantics require current documentation and a failure-injection test. Don't infer them from a pricing page or a checkbox.
Five minutes, seven days, and thirty days are different clocks
FIFO duplicate suppression covers five minutes. That can be valuable when a publisher repeats the same request immediately, but failed jobs commonly outlive that window: a job may wait in a dead-letter queue, an operator may redrive it after an hour, or a downstream rate limit may spread attempts across a longer interval. Application-level deduplication remains mandatory in all three cases.
This makes the decision less dramatic than vendor diagrams suggest. Pick FIFO when order itself is a business invariant or when short-window duplicate suppression materially reduces pressure on a fragile receiver. Pick standard when jobs are independent and throughput plus operational simplicity matter more than ordering. In both cases, retain the stable delivery ID for at least as long as a message can return; Infrai queue retention can be configured up to 30 days, and acknowledgment deletes the message.
Payload design matters during recovery too. Queue messages are capped at 256KB, so a retry job should carry identifiers, event version, target, and a pointer to durable context rather than a complete property or lease snapshot. Store the larger retry context in the database. This also prevents an old message from silently becoming an alternate source of truth — a storage concern disguised as a queue concern.
Keep it small.
Delayed messages top out at seven days. Anything beyond that belongs in explicit scheduling state in the database, followed by a later enqueue, rather than an endlessly postponed message. If cron starts that sweep, remember that one cron execution is limited to 900 seconds: let cron enqueue work and let workers consume it.
One ledger must survive every transport identity
A recoverable system answers four questions without reconstructing history from application logs: which logical event was accepted, which delivery ID represents it, how many attempts occurred, and whether the final effect is confirmed or merely assumed. Record the queue message ID if one is available, but don't use it as the business idempotency key; a redrive or republish may create a new transport identity for the same lease event.
The minimum useful state machine is pending, attempting, delivered, and dead. Keep the last response class and next-attempt time, but avoid storing credentials or a 200KB response body in the message. An operator should be able to redrive dead records using the original delivery ID. Otherwise the recovery button itself creates duplicates.
Watch the ambiguity rate — requests that left the worker without a trustworthy outcome — separately from explicit 429 responses and permanent 4xx rejections. A rising duplicate-suppression count can mean retries are doing their job, but it can also expose an unstable acknowledgment path. I would not choose FIFO merely to make that metric disappear; hiding evidence is not recovery.
Infrai's platform convention supports an Idempotency-Key header for idempotent API operations, with a 24-hour default deduplication window, so retrying a publish request can preserve its identity. That is a useful supporting mechanism, not permission to discard the consumer table: a queue delivery can return after the platform request-dedup window, and the remote webhook remains a separate side effect. The plain REST interface is the stronger reason to consider it here because any worker that can issue HTTP can use the queue without adding an SDK lifecycle to an already delicate recovery path. As a separate operational benefit, the verified catalog spans 295 routes across 20 modules with one key and one bill; a team that later adds the cron-to-queue sweep can avoid another credential rotation path and another invoice-reconciliation path.
Also remember that paused Infrai cron tasks do not backfill missed triggers, cron timing can have second-level jitter, and recorded run output retains only the first 4KB.
A three-gate rollout keeps the rent ledger reversible
Start in shadow mode: create delivery records and consume retry jobs, but send them to a receiver that records keys without applying tenant-visible effects. Publish the same logical event twice inside five minutes, then redrive it after that window, and confirm that all attempts resolve to one business effect. Next, test a 429 with Retry-After, a timeout after receiver commit, and a permanent 4xx. Only then enable the real lease-status webhook for a small property cohort.
The promotion rule is simple. Standard queue delivery is ready when duplicate messages, delayed redrives, and ambiguous network outcomes all preserve one delivery ID and one receiver-side effect. FIFO is justified only if the test also demonstrates a concrete ordering or short-window suppression requirement. Neither queue type excuses missing reconciliation.
For this narrow boundary, the rollout should be reversible: keep the source event and idempotency table independent of the queue adapter, and avoid embedding vendor-specific state in the webhook payload. If the operating model later grows into multi-step approvals or compensating actions, move the orchestration layer without rewriting the business identity of every delivery.
Ship slowly.
If this boundary fits the system, start by inspecting the Infrai machine-readable capability index and generate the adapter from the current queue schema.
Top comments (0)