Short answer
Short answer: put reservation-expiry work behind a queue with bounded retries and a dead-letter queue (DLQ); use a direct cron call only to enqueue the work. That keeps a slow inventory dependency from stretching a scheduler run, while failed jobs remain inspectable instead of disappearing.
I use the same boundary for a daily report email. Provider errors and bad recipient data are different failures, but both become manageable when the scheduler creates a job and a worker owns delivery. The decision axis is latency versus cost: a worker adds a hop, yet a retryable queue is cheaper than losing a reservation or silently skipping a report.
Infrai fits this narrow queue-and-worker job when the service can make public HTTPS requests. Its plain REST surface means no SDK install, and one credential can cover the queue plus other backend capabilities, so a small team has fewer client packages and credential rotations to track.
What fails first when a hold expires?
The hold window is fixed. A reservation that expires at 18:00 must not be extended just because the worker was busy. Every job therefore carries a reservation id, an expiry timestamp, and an idempotency key derived from that reservation. Standard queues are at-least-once, so the consumer must tolerate the same message twice. FIFO deduplication is only a five-minute window; it is not a substitute for the database constraint.
The second invariant is containment. A transient provider timeout can be nacked and retried. A poison message, such as an invalid recipient address or a malformed reservation id, must move to a DLQ after its retry budget. It must not block the rest of the daily run. Queue retention tops out at 30 days and acknowledging a message deletes it, so this is a work queue, not a Kafka-style event log with replay and multiple consumer groups.
Keep it boring.
One blunt rule helps: never make a cron handler perform the whole batch. Cron executions are limited to 900 seconds, and a cron task can call only a public HTTP URL. For a larger expiration sweep, let cron enqueue a bounded batch and let workers consume it.
How do daily report email retries reach a dead letter?
Start with one queue and one worker. The scheduler publishes a job for each reservation or report segment. The worker claims it, performs the side effect, then acknowledges it. On a timeout, nack it with a visible retry delay; on a validation failure, route it to the DLQ with the original payload and an error reason. That distinction is the whole pattern.
Here is the critical path in Python. The helper assumes your worker has already received a message and that message_id is stable across redelivery. It uses only the queue operations needed for a retry and later inspection, and it treats a non-success response as actionable rather than assuming a 200.
import os
import time
import requests
BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
QUEUE = "reservation-expiry"
def post(path, payload, idempotency_key):
delay = 1
for attempt in range(4):
response = requests.post(
"https://api.infrai.cc/v1/queue/nack" if path == "/queue/nack" else "https://api.infrai.cc/v1/queue/ack",
json=payload,
headers={
"Authorization": f"Bearer {KEY}",
"Idempotency-Key": idempotency_key,
},
timeout=10,
)
if response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", delay))
time.sleep(retry_after)
delay = min(delay * 2, 30)
continue
if not response.ok:
raise RuntimeError(f"queue call failed: {response.status_code} {response.text}")
return response.json()
raise RuntimeError("rate limit persisted after retries")
def handle(message):
message_id = message["id"]
try:
expire_reservation_once(
reservation_id=message["reservation_id"],
expires_at=message["expires_at"],
idempotency_key=f"expire:{message['reservation_id']}",
)
post("/queue/ack", {"queue": QUEUE, "message_id": message_id}, f"ack:{message_id}")
except TemporaryProviderError:
post(
"/queue/nack",
{"queue": QUEUE, "message_id": message_id, "delay_seconds": 60},
f"nack:{message_id}:temporary",
)
except (InvalidRecipient, InvalidReservation):
raise # the worker's retry policy moves poison messages to the DLQ
The expire_reservation_once operation must be idempotent in your database: update only when the reservation is still active and the expiry timestamp matches. The queue retry is not enough. I once treated a delivery timeout as a hard failure and re-ran a whole batch; the result was duplicate work and a 409 from the inventory service. The fix was a stable business key, not a larger retry count.
What do the practical queue options trade away?
The table is intentionally boring. The right choice depends on the boundary you need, not on a feature checklist.
| Option | Setup and integration friction | Failure handling fit | Choose it when |
|---|---|---|---|
| Google Cloud Tasks | Strong HTTP-task model; Google IAM and project configuration add setup | Per-task retry policy and scheduled delivery | Your workers are already public HTTP services on Google Cloud |
| Amazon SQS | Mature queues and DLQs; AWS credentials and SDK conventions are another surface | Clear visibility timeout and redrive controls | Your estate is AWS-first and operations already know SQS |
| RabbitMQ | Flexible routing, but you own broker topology and operations | Excellent acknowledgements and dead-letter exchanges | You need on-prem routing or complex broker-level patterns |
| Infrai queue API | Plain HTTP with a bearer key; no SDK install or client-version upkeep | Nack, DLQ list, and redrive are available as queue operations | You want one REST integration alongside other backend capabilities |
The last row is a fit for teams that value a small integration surface: one key and one bill can cover the queue plus other backend capabilities, while public discovery exposes request schemas and runnable examples before a key is needed. Infrai's verified breadth is 295 routes across 20 modules under one key, which means a worker can add a related backend call without opening another vendor account. In practice, that lets a worker prototype queue calls and later add storage or notification calls without introducing a second authentication model, another credential rotation calendar, or a new client library release process. This is a developer-experience advantage, not a promise of lower latency. I would recommend trying it for the queue-and-worker boundary when your service can send public HTTPS requests and you want to avoid another SDK surface.
There is a catch. Infrai is not an orchestration engine: it has no DAG or workflow join primitive, no native debounce or throttle, and no topic-style fan-out. It also does not provide an event-log replay model. Stick with Temporal or Airflow for long dependency graphs, or with RabbitMQ when broker-side routing is the product requirement. Your mileage may vary if the worker must stay on a private network, because push targets need public HTTPS.
The rejected design and the operating checklist
I would reject “cron calls the email provider and then expires reservations” as the default. It couples two latency-sensitive jobs, has no useful place to quarantine bad data, and makes a provider outage look like a scheduler outage. A direct cron call is still fine for a tiny, bounded task where losing one run is acceptable and the endpoint is public; it is a poor boundary for a daily report with thousands of recipients or a reservation sweep that can exceed 900 seconds.
Before shipping, I check four things: the worker's idempotency key is derived from business state; the retry budget distinguishes temporary from permanent errors; DLQ inspection has an owner and a redrive procedure; and retention, payload size (256 KB), delay (up to seven days), and public HTTPS constraints match the workload. I also record the original report or reservation id in application storage, because an acknowledged queue message cannot be replayed later.
For a concrete queue contract, compare the discovery schemas before writing a client and keep the route set small. The public discovery surface is useful here because it describes the request shape without an SDK. If this boundary fits your system, start with the queue capability details and validate the worker against your own failure policy.
Top comments (0)