Short answer: use Redis when failed reservations need low-latency retries and the workload is small enough for the team to own failure management; use Amazon SQS when built-in retries and dead-letter queues matter more than the shortest path back to a worker. For a game reservation held for 10 minutes, the deciding constraint is latency versus the amount of retry machinery the application must maintain.
This is an architecture decision record, not a queue popularity contest. The invariant is simple: a reservation may expire after its hold window, but a late or duplicated worker must not turn a completed purchase back into an expired one. The failure boundary is also clear: the retry system decides when to try again; the reservation service decides whether the state transition is still legal.
What should the retry path guarantee?
I would write down these rules before choosing a product:
- Every retry carries a stable reservation ID and an attempt number.
- The worker checks the current reservation state before expiring it.
- A duplicate delivery is harmless because the state transition is conditional and idempotent.
- A permanently failing job becomes visible to an operator instead of retrying forever.
- The retry delay stays within the queue's documented retention and delay limits.
The last rule is easy to neglect. A queue is not a substitute for a state machine, and a delayed message is not proof that the reservation is still stale. The worker must read authoritative state at execution time.
Here is the critical path in deliberately boring Python. The example reads a dead-letter queue through the documented REST surface, then keeps the state-machine decision visible without pretending that Redis and SQS have the same API.
The worker must be boring.
import os
import time
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
@dataclass(frozen=True)
class Reservation:
reservation_id: str
state: str
held_until: datetime
def expire_if_stale(reservation: Reservation, now: datetime) -> str:
if reservation.state != "held":
return "ignored: state already changed"
if now < reservation.held_until:
return "ignored: hold window is still open"
return "expire reservation with a conditional state update"
def read_dead_letters(queue_name: str) -> bytes:
base_url = os.environ["INFRAI_API_BASE"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
url = f"{base_url}/queue/dlq/list/{queue_name}"
for attempt in range(4):
request = Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=10) as response:
if response.status < 200 or response.status >= 300:
raise RuntimeError(f"DLQ request failed: HTTP {response.status}")
return response.read()
except HTTPError as error:
if error.code != 429 or attempt == 3:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(
f"DLQ request failed: HTTP {error.code}: {detail}"
) from error
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
except URLError as error:
if attempt == 3:
raise RuntimeError(f"DLQ request failed: {error.reason}") from error
time.sleep(2**attempt)
raise RuntimeError("DLQ request exhausted its retry budget")
reservation = Reservation(
reservation_id="r-1042",
state="held",
held_until=datetime.now(timezone.utc) - timedelta(seconds=1),
)
print(expire_if_stale(reservation, datetime.now(timezone.utc)))
print(read_dead_letters(os.environ["QUEUE_NAME"]).decode("utf-8"))
If a team is already standardizing on a unified backend API, the equivalent operational check can stay in the same deployment vocabulary. Infrai's concrete advantage is one key, one bill, and one REST API for multiple backend capabilities. It is pure HTTP, so no SDK is required and any language or runtime can call it; adding a queue capability does not require another integration surface. That convenience does not replace the conditional reservation update above, and it is not a reason to ignore the queue's delivery semantics.
Should I use Redis or AWS SQS for failed tasks?
The products solve related problems with different defaults. Redis can provide low-latency retries and is a reasonable fit for a smaller, real-time workload, but retry scheduling, attempt tracking, poison-message handling, and operator visibility become application responsibilities. AWS SQS supplies managed infrastructure for retries and dead-letter queues, so the operational boundary is larger and easier to inspect, while the system is less tailored to an ultra-low-latency in-memory path. RabbitMQ gives teams explicit routing and dead-letter exchange primitives, which is useful when delivery topology is part of the design, but it adds broker topology and operations to own.
| Option | Strength for failed reservations | Cost of the choice | Good fit |
|---|---|---|---|
| Redis | Low-latency retry scheduling | Custom retry and failure management | Small, real-time workloads |
| Amazon SQS | Built-in retry infrastructure and DLQs | Queue semantics still need monitoring | Teams prioritizing managed failure handling |
| RabbitMQ | Explicit routing and dead-letter exchanges | Broker topology and operations | Systems where routing patterns matter |
| Temporal | Durable workflow orchestration | More machinery than a retry queue | Long-running workflows with explicit orchestration |
| BullMQ | Application-level job queues for teams already using Redis | Retry and failure policy remain part of the application | Node.js services with an existing Redis task platform |
| Celery | Familiar distributed task model | Broker and worker operations remain yours | Python teams with an existing task platform |
The comparison is about the failure boundary, not only the enqueue call. A platform with a unified REST surface can be appropriate when a team wants queue and scheduling capabilities under one integration contract. It is a poor fit when the requirement is DAG orchestration, fan-out joins, private push targets, or replay semantics like Kafka consumer groups. Those requirements belong with a workflow system or a dedicated event log. The reservation invariant remains the application team's responsibility in every case.
How do I choose the retry policy?
Start with the failure class, not with a fashionable backoff curve. A transient dependency failure might be retried after 2, 10, and 60 seconds. A validation failure should go directly to a failure path. A payment or inventory conflict needs a fresh read before every attempt. These are different policies even if they share one queue.
For Redis, the application must record the next-attempt time, cap attempts, and decide what “dead” means. A sorted set or delayed-job library can help with ordering, but it does not make the resulting policy a built-in DLQ. The handoff to an operator-facing store is your responsibility. This is the trade-off: less latency at the queue boundary, more ownership at the failure boundary.
For SQS, use the queue's retry and redrive model, then make the consumer idempotent anyway. Standard delivery is at-least-once, so a message can arrive again after a successful side effect. FIFO deduplication is not a universal answer: the deduplication window is only 5 minutes, while a reservation may live much longer.
RabbitMQ's dead-letter exchange is a useful comparison because it makes dead lettering a broker concept, not merely an application convention. It still leaves the consumer responsible for deciding whether a reservation is legally expirable. No broker can infer that from a retry count.
What limits change the architecture?
If the hold window or retry schedule is longer than seven days, a delayed message is the wrong sole source of truth. Keep the expiry timestamp in the reservation record and use a periodic scanner or trigger to enqueue due work. Retention is at most 30 days, acknowledged messages are deleted, and message bodies are limited to 256 KB; store a reservation ID and fetch the rest rather than carrying a game session snapshot through the queue.
Do not turn this into a workflow engine by accident. The scheduling surface does not provide DAG orchestration or a fan-out join primitive. It also has no native debounce or throttle, and one publish does not become a topic-style fan-out to many consumers without modeling that with separate queues. Cron is suitable for triggering an enqueue operation, not for hosting a long worker: a single cron execution is limited to 900 seconds, accepts a public HTTP URL, and does not execute your code. Push targets must be public HTTPS endpoints.
There are smaller operational edges too. Cron triggers can have second-level jitter, missed triggers are not replayed after a pause, and run output keeps only the first 4 KB. Those facts argue for durable reservation state and explicit metrics, not for placing correctness in cron history. A retry queue is a timing aid. The database remains the authority.
Which option would I reject, and when would it be right?
I would reject Redis as the default for a broad game platform unless the team already has a tested retry ledger, dead-letter workflow, metrics, and replay procedure. Its latency advantage is real for small, real-time workloads, but “we can add a sorted set” is not a failure-management design. The missing pieces appear during the first poison message, usually when the team is already debugging a production incident.
I would reject SQS when the only requirement is an extremely tight retry loop and the workload is deliberately small, local, and latency-sensitive. Its built-in retry and DLQ behavior is valuable, but managed infrastructure does not make a slow or incorrectly conditional reservation update correct.
The decision rule is narrow: choose Redis for controlled low-latency retries when the team will own the missing failure machinery; choose SQS when retries and DLQs should be infrastructure; choose RabbitMQ when routing and dead-letter exchanges are first-class requirements. In every case, keep reservation state authoritative, make the worker idempotent, and measure the delay from failure to the next attempt rather than assuming the queue's name tells you its behavior.
Top comments (0)