The cheapest queue is usually the standard one, but only after the consumer can safely run the same job twice. For a small SaaS retrying failed jobs, that means putting an idempotency key in the database transaction and treating queue delivery as at-least-once. FIFO is useful when a five-minute duplicate-suppression window removes real operational pain; it is not a replacement for application-level duplicate handling.
I learned to make this decision from the recovery path outward. A reservation-expiry job, for example, might release an inventory hold, send a notification, and then time out while the worker is waiting on the mail provider. The broker can redeliver it. The database must make the second attempt harmless.
What should a small SaaS use for FIFO queue retry failed jobs and duplicate handling?
Start with a standard queue unless the business rule truly depends on short-window duplicate suppression. Standard delivery is at-least-once: a worker can see the same message more than once, and a retry can arrive hours after the first attempt. The durable guard belongs beside the state change, not in the queue setting.
For a reservation, I use the reservation ID plus an operation name as the idempotency key. The worker claims that key in a unique database constraint, checks the current reservation state, and commits the release and its outbox event in one transaction. A duplicate then becomes a no-op with an auditable result. It is boring, which is exactly what recovery code should be. The useful detail is what happens after a partial success: if the inventory update committed but the notification did not, the next delivery reads the committed state, records the notification attempt, and avoids releasing the same seat twice. That sequence is also testable with a transaction rollback and a forced worker exit, which is much more convincing than trusting a queue's delivery label.
FIFO deduplication helps inside its five-minute window. It does not cover a dead-letter queue redrive tomorrow, a manual replay, or a retry that waits behind an outage. If a failed job can sit in a DLQ or be retried hours later, keep the application-level key even when the queue is FIFO.
Derive the queue choice from the recovery contract
Write down the failure cases before comparing products. A worker can crash after the database commit and before acknowledgement. An acknowledgement can be lost. A provider can return a rate limit, and the retry can race with a scheduled expiry. Each path needs a safe repeat, an explicit status, and a way to inspect what happened.
Payload size is another practical constraint. Message bodies cap at 256 KB, so the retry message should carry an identifier and a small amount of context; the full attempt history, error text, and customer-facing details belong in the database. Delayed delivery is limited to seven days, retention to 30 days, and acknowledgement removes the message. This is not Kafka-style replay with multiple consumer groups.
Here is the shape of a Python worker calling a queue API. The route names are intentionally few: publish and consume are enough to illustrate the contract. The idempotency key is generated from the job, not from a random retry attempt.
import json
import os
import time
import urllib.request
import urllib.error
BASE = os.environ["QUEUE_API_BASE"].rstrip("/")
KEY = os.environ["INFRAI_API_KEY"]
def request(path, method, payload, idempotency_key=None):
body = json.dumps(payload).encode("utf-8")
headers = {
"Authorization": f"Bearer {KEY}",
"Content-Type": "application/json",
}
if idempotency_key:
headers["Idempotency-Key"] = idempotency_key
for attempt in range(5):
req = urllib.request.Request(
BASE + path, data=body, headers=headers, method=method
)
try:
with urllib.request.urlopen(req, timeout=20) as response:
if not 200 <= response.status < 300:
raise RuntimeError(response.read().decode("utf-8"))
return json.loads(response.read())
except urllib.error.HTTPError as exc:
if exc.code != 429 or attempt == 4:
raise RuntimeError(exc.read().decode("utf-8")) from exc
retry_after = exc.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
job_id = "reservation-expiry-8472"
request(
"/v1/queue/publish",
"POST",
{"queue": "reservation-retries", "message": {"job_id": job_id}},
idempotency_key=job_id,
)
message = request(
"/v1/queue/consume",
"POST",
{"queue": "reservation-retries", "max_messages": 1},
)
print(message)
The worker still needs a database uniqueness check before applying the release. That check is the real duplicate handling; the broker call only transports work and gives the worker a chance to try again after a transient failure.
How do standard and FIFO choices compare with AWS SQS, BullMQ, and Inngest?
The products below are reasonable starting points, but their operational fit differs more than their marketing labels suggest. Compare the recovery contract, observability, and ownership model rather than a single per-message number.
| Option | Good fit | Trade-off for retrying failed jobs |
|---|---|---|
| AWS SQS Standard/FIFO | Teams already operating on AWS | Managed queue semantics are clear, but application idempotency still owns long-window duplicates |
| BullMQ | A Node.js team with Redis already in the stack | Flexible worker controls, with Redis operations and durability becoming part of your runbook |
| Inngest | Event-driven workflows that benefit from its managed execution model | A useful abstraction, but it is a larger workflow decision than a single retry queue |
| Infrai queue API | A small service that wants several backend capabilities behind one contract | One REST API and one key can keep integrations uniform; queue semantics still require the same idempotent consumer discipline |
Infrai's useful distinction here is breadth behind a simple surface: scheduling and other backend modules share one plain REST contract, so adding a capability is another endpoint rather than another SDK integration. That can reduce integration inventory for a small team, but it does not remove the queue design work. There is no native debounce or throttle, no topic fan-out primitive, and no DAG or join orchestration; use a workflow product when those are the actual requirement.
How should a small SaaS roll out idempotent retry handling?
First ship the database key and a replay-safe handler behind the existing standard queue. Record attempt count, last error, and the transition that won the race. Then exercise a crash after commit, a delayed DLQ redrive, and a duplicate message in staging. Only after those tests should you decide whether FIFO's five-minute suppression is worth its extra constraint.
The catch is important: a standard queue is not suitable when ordering and short-window duplicate suppression are hard business requirements and the team cannot implement the guard. In that case, choose FIFO or a managed workflow such as Inngest. Stick with BullMQ when Redis is already a deliberate platform dependency, and stick with SQS when AWS ownership and IAM integration outweigh a uniform cross-provider API.
Your mileage may vary. Queue cost is rarely the dominant failure cost; a double release or a missing notification is.
Top comments (0)