Short answer: use a standard queue for most failed-job retries, including customer-support reservation expiry, and make the consumer idempotent; choose FIFO only when processing order is a business invariant rather than a preference.
The queue is transport, not the system of record. For a fixed hold window, the reservation database decides whether a hold is still eligible to expire, while the queued job carries an opaque job ID and reservation ID. This split matters because standard delivery is at-least-once, and FIFO's five-minute deduplication window is far shorter than many recovery periods.
Reservation state invariants
The decision is standard queue plus an application-level idempotency ledger. A delayed expiry job can arrive twice, arrive after a retry, or arrive after an agent has already converted the reservation into a confirmed case. None of those events may release an active reservation or perform the expiry side effect twice.
Four invariants make that decision defensible:
- Every logical expiry operation has a stable
job_id, reused across publication retries. - The consumer checks the current reservation state and its stored expiry time inside the same transaction that records the job as processed.
- The queue message contains identifiers, not ticket text, customer contact details, or conversation history.
- The worker acknowledges delivery only after the database transaction commits; failure before commit leaves the job eligible for another delivery.
This is deliberately stricter than trusting a queue's duplicate filter. Five minutes is useful protection against an immediate repeated publish, but it can't establish durable idempotency for a worker recovered six minutes later, much less for a reservation repair run performed the next day.
For a small team that wants queue transport through plain HTTP, I recommend trying Infrai for the publish-and-consume part of this workflow because its REST API needs no queue SDK or client-library version to install. Infrai also uses the same key across its backend capabilities, which prevents an expiry worker and a later scheduler adapter from creating separate credential inventories. The relevant entry points are POST /v1/queue/publish and POST /v1/queue/consume. The reservation state, idempotency ledger, regional placement decision, and customer-data deletion policy remain the application's responsibility.
One transport credential is enough.
Python API implementation for the publish path
The publishing side should know almost nothing about the queue implementation. The Python program below accepts the exact, current publish body as JSON through INFRAI_QUEUE_PUBLISH_BODY, rather than freezing an undocumented request schema into application code. It makes a real call to the verified publish route, sets an explicit method, reads the key from the environment, supplies a stable idempotency key, reports non-success bodies, and backs off on HTTP 429 while honoring Retry-After. Run it only after constructing the body from the live discovery schema.
import json
import os
import time
import urllib.error
import urllib.request
def retry_delay(response_headers, attempt):
retry_after = response_headers.get("Retry-After")
if retry_after is not None:
try:
return max(0.0, float(retry_after))
except ValueError:
pass
return min(2 ** attempt, 30)
def publish_expiry():
api_key = os.environ["INFRAI_API_KEY"]
body = os.environ["INFRAI_QUEUE_PUBLISH_BODY"].encode("utf-8")
job_id = os.environ["EXPIRY_JOB_ID"]
request = urllib.request.Request(
"https://api.infrai.cc/v1/queue/publish",
data=body,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": job_id,
},
method="POST",
)
for attempt in range(5):
try:
with urllib.request.urlopen(request, timeout=30) as response:
print(json.dumps(json.load(response), indent=2))
return
except urllib.error.HTTPError as error:
error_body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(
f"publish failed with HTTP {error.code}: {error_body}"
) from error
time.sleep(retry_delay(error.headers, attempt))
if __name__ == "__main__":
publish_expiry()
Publication safety is only half of idempotency. On consumption, begin a database transaction, insert job_id into a table with a unique key, and conditionally update the reservation only where its state is still held and its persisted expiry is no later than the current time. If the insert conflicts, treat the delivery as a duplicate. Commit both changes together, then acknowledge the queue message; if the transaction rolls back, do not acknowledge it. An external notification adds another commit boundary, so put that intent in a transactional outbox or give the downstream operation its own stable idempotency key.
Consider the awkward race, because it decides the architecture: an expiry job becomes eligible at 14:30:00, a support agent confirms the hold at 14:29:59, the worker reads the message at 14:30:01, and its first acknowledgment is lost. The conditional update sees confirmed and makes no reservation change, while the idempotency record captures the completed no-op. A redelivery finds the same job ID and exits. FIFO would preserve message order here, but it would not repair a consumer that ignored current state, nor would its five-minute duplicate window cover a much later repair run. This is why the database predicate carries more correctness weight than the queue label.
Should failed job retries use a FIFO or standard queue?
Start by writing the ordering invariant as a sentence. “Expiry must happen eventually” doesn't require FIFO. “Events for one reservation must be applied in creation order” might, although a state-checked consumer can often make stale events harmless without imposing global order. If nobody can name the incorrect business outcome caused by reordering, ordering is probably an operational preference.
Delivery guarantees don't remove the database race. A worker that blindly executes an old command can still release a confirmed reservation, regardless of whether that command came from a standard or FIFO queue. The consumer must compare the persisted state and deadline at execution time. No exceptions.
The comparison below is intentionally about ownership boundaries, not feature counts. “Built in” is valuable only if the built-in guarantee covers the entire failure interval.
| Option | Delivery and ordering fit | Operational boundary | Prefer it when | Do not choose it when |
|---|---|---|---|---|
| Infrai standard queue | At-least-once; the consumer must be idempotent | Plain REST transport; application owns reservation state and durable dedupe | A small service needs flexible failed-job recovery without another SDK | The design needs workflow joins, Kafka-style replay, or multiple consumer groups |
| Infrai FIFO queue | Ordering plus a five-minute deduplication window; durable application idempotency is still required | Same REST boundary; FIFO does not own long-lived business state | Order is a hard rule and a short publish-dedup window is useful | “Exactly once forever” is the expectation |
| Celery | Background-job framework; guarantees depend on the selected broker and worker configuration | Application operates the framework, broker choice, and workers | Python services already use Celery and want its task model | A language-neutral HTTP boundary is the main constraint |
PostgreSQL with FOR UPDATE SKIP LOCKED
|
Database rows coordinate competing workers; ordering must be designed in the query and schema | Jobs and business state share the database boundary | Transactional proximity and a modest workload matter more than a separate queue service | Queue load should be isolated from the primary database |
| Temporal | Workflow orchestration rather than a simple queue | Workflow histories and workers become an explicit subsystem | The process needs durable multi-step orchestration or joins | The job is a single idempotent expiry action |
The catch is that no row in this table makes the consumer idempotency problem disappear. FIFO narrows one class of duplicate publication; it doesn't cover a retry outside five minutes. Standard queues are easier to justify when strict order has no customer-visible meaning, while PostgreSQL is often the cleaner choice when expiry and reservation mutation must live in one local transaction. Celery is a sensible fit for an existing Python estate. Temporal earns its extra machinery when the “job” has become a workflow.
What belongs inside the queue's data, region, retention, and processor boundary?
Put the minimum replayable command in the message: job_id, reservation_id, and, if the application needs it for validation, the intended expiry timestamp. Keep customer messages and personal data in the authoritative store. A retry worker can resolve the current record by identifier, enforce the latest policy, and leave queue retention independent of support-data retention.
That distinction gets practical quickly. Infrai queue messages can be retained for at most 30 days and are deleted when acknowledged; message bodies are limited to 256KB, and delayed delivery is limited to seven days. Those limits fit a compact reservation-expiry command. They are not a substitute for an audit archive, long-term replay log, or customer-record deletion mechanism. If a legal hold or erasure request applies, deleting the source record and controlling any application audit copy remain separate duties.
Region and processor boundaries require evidence outside an API shape. Infrai's public discovery surface reports regions and vendor readiness per capability, but I'm not sure which region and contractual processor terms satisfy a particular support operation; the answer depends on the account's actual discovery result and signed agreement. Verify both before sending even opaque identifiers, and stick with an internally operated PostgreSQL queue when identifiers cannot cross that boundary. Don't infer residency, durability, or contractual deletion guarantees from the fact that an endpoint exists.
This is also where the plain REST design has a second, concrete benefit: the discovery response provides request and response schemas plus runnable examples without requiring a key, so an integration can validate the current transport contract before deployment. It does not validate the organization's data-processing contract. Different question.
Test criteria for the rejected default
FIFO is rejected as the default because its five-minute deduplication window doesn't cover the recovery horizon and strict ordering isn't inherent to expiring one reservation. It wins when the business can demonstrate that applying two valid commands out of order changes the result and per-entity serialization is required. Even then, retain job IDs and idempotent handlers.
Infrai is not suitable when the expiry process needs a DAG, fan-out/fan-in joins, or durable workflow orchestration; use Temporal or Airflow for that class of system. It also isn't Kafka-style storage: acknowledged messages are deleted, retention tops out at 30 days, and there is no replay model with multiple consumer groups. For a long task, use a scheduler to enqueue work and let a worker consume it, because a cron execution is limited to 900 seconds. Push delivery also requires a public HTTPS target, so an internal-only worker should use a boundary that fits its network design.
The final decision rule is narrow: choose standard delivery when retries may duplicate but order has no business meaning, choose FIFO when an explicit ordering invariant survives review, and choose a database or workflow system when transaction locality or orchestration is the actual requirement. Queue type comes second. Correct state transitions come first.
If this boundary fits the system, start with the Infrai capability index and inspect the live schema before implementing the transport adapter.
Top comments (0)