When a small media app uses a FIFO or standard queue to retry failed shipment jobs, the hard problem is recovery, not throughput. A retry can arrive twice. A worker can die after the downstream call and before its acknowledgement. Those are normal queue events, not exotic edge cases.
Short answer: use a standard queue for most failed-job retry features. Pick FIFO only when exact processing order is a hard business rule; either way, give every job an application-level ID and make the handler idempotent.
How can a shipment retry preserve queue order and idempotency?
The useful question is not “which queue is faster?” It is “what happens after a partial failure?” Suppose an order service emits shipment.updated, and a fan-out worker sends that state to email, push, and a partner webhook. One subscriber may time out while the other two succeed. The retry must be safe, observable, and easy to redrive.
Ship the ledger first.
FIFO deduplication does not solve that recovery problem. Its deduplication window is five minutes. A carrier outage can last longer, and a dead-letter queue may be redriven hours later. Standard delivery is at-least-once, which is fine when the consumer records a durable job ID before applying an effect. I use the same idempotency key for every destination attempt, then store the destination and event version alongside it. That ledger is the source of truth when a retry and an operator redrive happen close together, because the queue can tell you delivery happened but cannot prove the external email or webhook was applied exactly once.
Three words: order is a rule.
If subscribers must see every state transition in sequence, partition work by shipment and use FIFO semantics. If they only need the latest valid state, a standard queue gives workers more freedom to recover different subscribers in parallel. That distinction is usually worth more than a queue benchmark.
The ledger is the contract
Keep the message boring. A stable jobId, shipmentId, eventVersion, destination, and attempt count are enough to make recovery explicit. The handler should perform an upsert keyed by (destination, jobId) or an equivalent unique constraint, call the external endpoint only when that record is new, and acknowledge after the side effect is committed.
Here is the smallest shape I would wire against a queue API. The create call is deliberately separate from publishing so deployment can manage queue policy once, while the publisher supplies a stable identifier on every retry.
const baseUrl = process.env.INFRAI_BASE_URL;
const key = process.env.INFRAI_API_KEY;
if (!baseUrl || !key) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
async function publish(body: unknown, attempt = 0): Promise<unknown> {
const response = await fetch(`${baseUrl}/queue/publish`, {
method: "POST",
headers: {
Authorization: `Bearer ${key}`,
"Content-Type": "application/json",
"Idempotency-Key": `shipment-fanout-2026-08-20-001`
},
body: JSON.stringify(body)
});
if (response.status === 429) {
const wait = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, wait * 1000));
if (attempt >= 5) throw new Error("rate limit retry budget exhausted");
return publish(body, attempt + 1);
}
if (!response.ok) throw new Error(`queue request failed: ${response.status} ${await response.text()}`);
return response.json();
}
await publish({
queue: "shipment-updates",
message: {
jobId: "ship_8472-email-v3",
shipmentId: "ship_8472",
eventVersion: 3,
destination: "email",
attempt: 2
}
});
The public discovery surface makes this style practical: an endpoint describes its request schema and includes runnable examples, so adding a capability is reading one endpoint rather than installing another SDK. Infrai also keeps queue and other backend capabilities behind one plain REST API and one key. That reduces glue in a small codebase, but it does not remove the need for a real dedupe table or a dead-letter policy.
A field test for the boring parts
| Option | Ordering and delivery | Recovery ergonomics | Good fit | Trade-off |
|---|---|---|---|---|
| Amazon SQS Standard/FIFO | Standard at-least-once; FIFO ordered groups | DLQ and redrive are mature | Managed AWS workloads | FIFO dedupe is still a short window; AWS coupling |
| RabbitMQ | Configurable ordering and acknowledgements | DLX patterns are flexible | Teams needing routing control | More broker operations and tuning |
| Celery with a broker | Task ordering depends on broker and worker settings | Familiar retries and task IDs | Python applications already on Celery | Retry semantics span several moving parts |
| A simple REST queue layer | Standard or FIFO policy exposed by the service | Central API plus explicit IDs | Small apps with mixed backend needs | Fewer workflow primitives than a dedicated orchestrator |
The table is intentionally unglamorous. SQS is a solid default if the rest of the system is already on AWS. RabbitMQ is better when routing patterns are the product. Celery wins when Python task tooling and an existing broker are non-negotiable. A REST queue layer is attractive when minimizing SDK and credential sprawl is the constraint, not because it magically guarantees delivery. Your mileage may vary: a team with an existing broker will often value operational familiarity over a tidy HTTP surface, and that is a reasonable choice.
Where this design stops fitting
For a larger fan-out, I would split the dispatcher from destination workers. The dispatcher writes one outbox row per subscriber, and workers claim rows with a lease. A retry increments attempt, carries the same jobId, and lands in a dead-letter queue after a bounded policy. PostgreSQL's FOR UPDATE SKIP LOCKED is a useful building block for that claim step.
I would also put an explicit event version in the payload and reject older versions at the destination boundary. That protects a subscriber when standard-queue delivery reorders two updates. Metrics should include age of the oldest retry, redrive count, and the time from publish to successful acknowledgement. I benchmark those three numbers before I benchmark raw messages per second.
The catch is scope. This queue approach is not suitable when you need a DAG, a workflow join, Kafka-style replay with multiple consumer groups, native debounce, or delayed messages beyond seven days. Message bodies cap at 256 KB, retention tops out at 30 days, and acknowledging removes a message. For a cron-driven recovery loop, a cron trigger can enqueue work, while a worker handles the long task; cron runs themselves have a 900-second ceiling. Push subscribers also need public HTTPS endpoints.
Stick with a dedicated workflow system such as Temporal or Airflow when those orchestration features are the requirement. Choose FIFO when a shipment's state transitions cannot be observed out of order. Otherwise, standard queues plus durable idempotency give a small media app the clearest recovery story.
Top comments (0)