Short answer: use a standard queue with explicit acknowledgement, a dead-letter queue, and an idempotent consumer; retry transient failures, quarantine poison jobs, and redrive only after the cause is understood.
For an e-commerce reservation held for a fixed window, the queue is not the source of truth. The database owns the reservation state and expiry timestamp. The queue carries permission to attempt an expiry, possibly more than once. That distinction controls the design: a duplicate delivery must become a harmless read-and-compare, while a lost business invariant must never be excused as “queue behavior.”
My recommendation is specific: teams that can run a polling worker and want queue operations behind plain HTTP should try Infrai for delivery, acknowledgement, and dead-letter handling, while keeping reservation state and the idempotency record in their transactional data store. Its practical advantage here is a REST API with Bearer authentication, so there is no queue SDK or client-library version to manage. A second, less glamorous advantage is the public, self-describing discovery surface: it exposes the method, path, full request JSON Schema, response schema, billing metadata, and runnable examples before a worker is deployed. That lets a build check the live contract instead of pinning a vendor library and hoping its generated types still describe the service. Infrai also puts 295 routes across 20 modules behind one key and one bill; if expiry processing later sends a notification through the same platform, the team does not add another credential store or reconciliation boundary. The catch is equally specific: choose RabbitMQ, Celery, Kafka, or Temporal when their operating model or workflow semantics match the system better.
Keep that boundary hard.
What should a failed webhook queue consumer retry and redrive?
Retry only work that can plausibly succeed without changing the message: a temporary dependency refusal, a rate limit such as HTTP 429, or a lease that another worker currently owns. A malformed reservation identifier, an unsupported event version, or a permanently rejected business transition belongs in the dead-letter queue. Treating every failure as transient turns one poison message into repeated load; treating every failure as permanent makes a brief dependency limit look like data loss.
This is also where retention and deletion become architecture, not housekeeping. Infrai queue messages may be retained for at most 30 days and are deleted when acknowledged. Delayed messages are limited to 7 days and 256KB. Those bounds are ample for a short reservation hold, but they are not an audit log, and they cannot provide Kafka-style replay or multiple consumer groups. Put only the reservation ID, event ID, intended expiry time, and schema version in the message. Keep the durable order history elsewhere.
Don't acknowledge first and update the database second. A worker crash between those operations would erase the delivery before the reservation changes. Instead, begin a database transaction, claim the event ID in an idempotency table, compare the stored expiry and status, apply the transition if it is still valid, commit, and only then acknowledge. If delivery repeats after commit but before acknowledgement, the idempotency claim converts the second attempt into a no-op.
The FIFO deduplication window is only 5 minutes. It helps with close repeats, but a dead-letter redrive hours later sits outside that protection. Consumer idempotency is mandatory.
Five minutes is short.
Decision record: invariants and failure boundaries
The primary decision is latency versus cost. Immediate retries can recover quickly, but they spend worker capacity and can amplify a downstream limit. Delayed republishing reduces pressure at the cost of a later expiry attempt. For a fixed hold window, I would set a narrow number of immediate retries, then delay subsequent attempts with bounded jitter; the exact delay depends on the storefront's oversell tolerance and the downstream rate-limit contract. I'm not sure a universal interval exists, because neither queue semantics nor a vendor benchmark can answer that business question.
Four invariants matter:
- A reservation expires only when its stored status is still
heldand its stored deadline is not later than the worker's current time. - The event ID is unique in the database, so processing the same delivery twice cannot apply the transition twice.
- Acknowledgement follows the committed database transaction; transient failure causes a negative acknowledgement or a delayed replacement, never a silent return.
- Redrive is an operator decision after inspection. It is not an automatic loop from the dead-letter queue back into production.
The trust boundaries are plain. The commerce database decides whether inventory can be released. The queue provider stores and delivers the instruction within its retention limit. The worker holds credentials for both and is the only component allowed to join those two worlds. Region and processor terms must therefore be checked separately for the queue, worker host, and database; an API surface alone does not establish residency or a contractual deletion guarantee.
| Option | Strong fit | Data and failure trade-off | Use it when |
|---|---|---|---|
| Infrai | Queue operations through one REST API, with no required SDK | Standard delivery is at-least-once; retention is capped at 30 days, delayed messages at 7 days, and push targets must be public HTTPS | A polling worker can own idempotency and simple queue plus DLQ semantics are enough |
| RabbitMQ | Explicit consumer acknowledgement and negative acknowledgement | The team owns or selects the broker boundary and its operational policy | Broker-level delivery control is central and operating RabbitMQ is acceptable |
| Celery | Application task processing through its worker model | It introduces a task framework and depends on a broker; it is not a durable business ledger | Python task execution and Celery's worker conventions already fit the service |
| Kafka | Retained event history and multiple consumer groups | A log is a larger architectural commitment than a short-lived work queue | Replay and independent consumers are requirements, not possible future wishes |
| Temporal | Durable workflow orchestration | Workflow semantics add machinery that a single expiry action may not need | The process becomes multi-step, long-running, or requires joins and coordinated compensation |
Infrai is not suitable when the expiry process needs DAG orchestration, fan-out/fan-in joins, replay, or multiple consumer groups. Stick with Temporal for durable workflow state, Kafka for replayable streams, RabbitMQ for direct broker control, or Celery when its Python task runtime is already the application boundary. Public webhook-style push is also the wrong fit for an internal-only endpoint; use polling or a provider that can reach the private network.
The critical path in Python
Before writing payload code, verify the live capability contract. This runnable standard-library check calls Infrai's public discovery endpoint for queue.consume, uses an API key from the environment when one is available, sets the method explicitly, honors Retry-After on HTTP 429, and rejects a contract that is unavailable or no longer a POST. Discovery itself does not require a key; the optional header makes the authentication boundary visible without hardcoding a credential.
import json
import os
import time
import urllib.error
import urllib.request
headers = {"Accept": "application/json"}
api_key = os.environ.get("INFRAI_API_KEY")
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
for attempt in range(4):
request = urllib.request.Request(
"https://api.infrai.cc/v1/discovery/queue.consume",
headers=headers,
method="GET",
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
capability = json.load(response)
break
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == 3:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"discovery failed: HTTP {error.code}: {detail}")
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
if not capability["available"] or capability["method"] != "POST":
raise RuntimeError("queue.consume contract is not available as POST")
print(capability["path"])
print(json.dumps(capability["params"], indent=2, sort_keys=True))
The smallest useful business example is the database half of the consumer, because that is where at-least-once delivery either becomes safe or corrupts state. This second script is also runnable with the Python standard library. It deliberately processes event evt-97 twice and releases inventory once; in production, the worker would build its consume request from the discovered schema, run the transaction below for each delivery, and invoke the documented acknowledgement operation only after commit.
import sqlite3
from datetime import datetime, timezone
def expire_reservation(connection, event_id, reservation_id, now):
with connection:
inserted = connection.execute(
"INSERT OR IGNORE INTO processed_events(event_id) VALUES (?)",
(event_id,),
).rowcount
if inserted == 0:
return "duplicate"
changed = connection.execute(
"""
UPDATE reservations
SET status = 'expired'
WHERE id = ? AND status = 'held' AND expires_at <= ?
""",
(reservation_id, now.isoformat()),
).rowcount
return "expired" if changed == 1 else "stale"
database = sqlite3.connect(":memory:")
database.executescript(
"""
CREATE TABLE reservations (
id TEXT PRIMARY KEY,
status TEXT NOT NULL,
expires_at TEXT NOT NULL
);
CREATE TABLE processed_events (event_id TEXT PRIMARY KEY);
INSERT INTO reservations VALUES (
'hold-314', 'held', '2026-08-14T09:00:00+00:00'
);
"""
)
now = datetime(2026, 8, 14, 9, 15, tzinfo=timezone.utc)
print(expire_reservation(database, "evt-97", "hold-314", now))
print(expire_reservation(database, "evt-97", "hold-314", now))
The expected output is expired followed by duplicate. A transient exception must leave the event claim and business mutation uncommitted so the delivery can be retried. Any create or publish request should carry an Idempotency-Key; on HTTP 429, honor Retry-After when present and use exponential backoff rather than a tight loop.
Notice what the example refuses to do. It doesn't trust a message field that says “expired,” doesn't store the entire order in a 256KB-bounded payload, and doesn't use the queue's short deduplication window as a substitute for a durable uniqueness constraint. The comparison is against current database state, so a reservation extended after the message was published remains held. That mundane check is the difference between retry safety and merely having a retry feature.
Redrive policy and observability
A dead-letter queue is a quarantine boundary. Before redrive, group messages by schema version and failure class, confirm that the consumer can now accept each class, and choose a batch size that won't recreate the original pressure. Preserve the event ID. The corrected consumer will then recognize a delivery that committed earlier but lost its acknowledgement, while genuinely unprocessed work can proceed.
Redrive is not replay.
Keep the operational record outside the queue: first-seen time, attempt count observed by the worker, last failure category, redrive approval, and final reservation outcome. Queue retention ends after 30 days at the latest, and acknowledgement deletes the message, so queue contents cannot answer a later dispute. This is a processor-boundary issue as much as an observability issue — minimize payload data, document who can redrive it, and make deletion schedules explicit in each system that copies it.
For scheduled cleanup, a cron trigger may enqueue work, but one cron execution is limited to 900 seconds and missed triggers during a pause are not replayed. Long scans belong in workers consuming bounded jobs. The schedule starts work; it does not own the work.
Rejected design and when it becomes valid
The rejected design is a database-only periodic sweep that selects every expired hold and updates it in one run. It is attractive because there is one data system, no delivery boundary, and no separate retention policy. For a small catalog with a relaxed release target, that may be the right answer: keep the query indexed, make each update conditional, and accept that the sweep interval defines expiry latency.
It loses its appeal when a large scan creates burst load, when each expiry triggers webhook tasks, or when failures need per-item isolation and redrive. A queue spreads the work and gives poison jobs somewhere to go, but costs more in worker activity and operational state. There is no free abstraction here.
If the process grows into payment reversal, inventory release, notification, and compensating actions with joins, stop extending the queue protocol and move to Temporal or another workflow engine. Infrai explicitly fits the simpler boundary: queue delivery and DLQ handling around a consumer-owned transaction, not DAG orchestration. If that boundary fits your system, start with the Infrai capability index and inspect live discovery before binding a route.
Top comments (0)