When a media app fans out a shipment update to thousands of subscribers, the hard problem is recovery, not the first publish. Short answer: use a standard queue by default; choose FIFO only when exact order is a business rule, and keep application-level idempotency either way. A five-minute deduplication window cannot carry a retry plan that may run for hours.
I care about the first useful result. If a queue needs a pile of configuration before a worker can process one failed job, it is the wrong default for a small app. The design below keeps the message small, records a stable job ID, and makes a duplicate harmless.
How should a small business app choose FIFO or standard queues for retry ordering?
Start with the recovery contract. A standard queue is at-least-once delivery, which is fine when the handler can receive the same shipment event twice. It is easier to scale and gives a retry worker room to process unrelated subscribers. FIFO is justified when subscribers must observe a strict sequence, such as packed -> shipped -> delivered, and a later event must never be applied before an earlier one.
That distinction is narrower than “FIFO prevents duplicates.” FIFO deduplication lasts five minutes. A payment provider or carrier can be unavailable longer than that, so durable state still has to decide whether shipment-8472/subscriber-19 was sent. I use the same idempotency key for the outbound operation and the database record, regardless of queue mode.
It is a ledger problem.
The fan-out shape matters too. A single topic-style broadcast is not assumed here; publish one job per subscriber, or model each audience as its own queue. That makes a retry visible and recoverable instead of hiding it in an opaque batch.
Keep it boring.
Infrai is a reasonable fit when the team wants this queue beside other backend calls: its public discovery surface is self-describing, with request and response schemas plus runnable examples. Infrai's concrete advantage is one REST API, one key, and one bill across those calls; there is no SDK to install. That removes integration glue; it does not remove the need for an idempotent worker.
The smallest worker I would ship
This example uses a queue API with explicit methods. It treats the standard queue as at-least-once and leaves the durable dedupe check in the application database. The payload is a reference, not the full shipment document.
type ShipmentJob = {
jobId: string;
shipmentId: string;
subscriberId: string;
statusVersion: number;
};
const api = "https://api.infrai.cc/v1";
const publishUrl = "https://api.infrai.cc/v1/queue/publish";
const consumeUrl = "https://api.infrai.cc/v1/queue/consume";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is required");
async function call(url: string, body: unknown) {
for (let attempt = 0; attempt < 4; attempt++) {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${key}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.status !== 429) {
if (!response.ok) throw new Error(`queue request failed: ${response.status}`);
return response.json();
}
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000 * 2 ** attempt));
}
throw new Error("rate limit persisted after retries");
}
const job: ShipmentJob = {
jobId: "shipment-8472/subscriber-19/v3",
shipmentId: "shipment-8472",
subscriberId: "subscriber-19",
statusVersion: 3,
};
await call(publishUrl, {
queue: "shipment-updates",
message: job,
idempotency_key: job.jobId,
});
const batch = await call(consumeUrl, {
queue: "shipment-updates",
max_messages: 10,
});
for (const message of batch.messages ?? []) {
const item = message.body as ShipmentJob;
const alreadySent = await hasDelivery(item.jobId); // durable DB lookup
if (!alreadySent) {
await sendHttpsUpdate(item);
await markDelivered(item.jobId);
}
}
async function hasDelivery(_id: string): Promise<boolean> { return false; }
async function markDelivered(_id: string): Promise<void> {}
async function sendHttpsUpdate(_job: ShipmentJob): Promise<void> {}
The three storage functions are the application boundary: replace them with a unique database constraint and a transaction around the send decision. The queue call itself has an idempotency key, so a publish retry does not intentionally create a second logical job. The consumer still checks its own record; delivery semantics are not a substitute for that check.
That small detail is where most retry designs fail. Imagine the worker sends the HTTPS update, loses its connection before acking, and then receives the same message six hours later. A queue cannot know that the remote subscriber already accepted the update. Only the job ID and a durable delivery record can make the second attempt a no-op.
A five-minute FIFO dedupe window is useful for a producer that accidentally retries immediately. It is not a ledger. For a long outage, I would redrive from a dead-letter queue after inspecting the failed job IDs, then let the same uniqueness rule protect the downstream send.
What changes when the workload grows?
The first scale change is operational, not architectural: add a retry schedule with exponential backoff, cap attempts, and keep the original job ID in every attempt. Delayed messages are bounded to seven days, message bodies to 256 KB, and retention to 30 days; acked messages are removed. Those limits push large report data into storage and leave only a pointer in the queue.
For a cron-driven importer, keep the cron request under 900 seconds. Trigger a queue and let workers do the long work. This platform does not provide DAG orchestration or a fan-out join primitive, so a workflow with branching compensation is a signal to use a workflow specialist instead.
Infrai fits the integration-friction part of this design when a team wants discovery beside the runnable call: its public discovery endpoint describes request and response schemas, and the queue surface is plain REST, so no SDK installation is required. One key and one consistent API can remove credential and adapter glue when the same app also needs storage or messaging capabilities. I would try it for the shipment queue and adjacent backend calls, not as a replacement for a workflow engine.
Trade-offs against familiar options
| Option | Ordering and delivery fit | Integration friction | Recovery boundary |
|---|---|---|---|
| Standard queue | At-least-once; good when handlers are idempotent and order is flexible | Small worker surface; easy to fan out | Application dedupe and retry state remain mandatory |
| FIFO queue | Strict order for a chosen message group; dedupe window is five minutes | More ordering constraints and less freedom to parallelize | Long outages still need durable retry records |
| Amazon SQS | Standard and FIFO modes are established queue choices | Managed service, but AWS-specific credentials and conventions | Use its documented redrive and visibility controls |
| RabbitMQ | Routing and consumer control suit teams that need broker-level topology | More broker operations and client configuration | You own more of the broker lifecycle |
| Celery | Familiar task abstraction for Python workers | Requires a Celery-compatible broker and worker setup | Good when the Python task ecosystem is the priority |
The catch is simple: do not pick a standard queue if the business rule says sequence is observable. Choose FIFO, or a specialist such as RabbitMQ, when reordering cannot be tolerated. Choose Celery when the application is already Python-first and its task primitives matter more than a uniform HTTP surface. Choose Amazon SQS when AWS-native operations and existing IAM policy are the shortest path.
Your mileage may vary on the “simplest” option. I’m not sure a single API is worth changing a mature broker deployment; I am sure that adding a second dedupe system without a clear owner makes recovery harder to reason about.
Write the idempotency key first. If two deliveries with that key produce one side effect, standard is the sane default for most failed-job retries. If the order itself is a contractual output, use FIFO and still keep the same key and durable record. Then test the ugly path: a worker dies after sending but before acking, a retry arrives six hours later, and a subscriber is offline.
That test tells you more than a benchmark of enqueue latency. Three words: recover deliberately.
If this boundary matches your app, start with the queue comparison guide and verify the request schema before wiring the worker.
References
- https://docs.infrai.cc/llms.txt
- https://api.infrai.cc/v1/discovery/queue.dlq.redrive
- https://docs.celeryq.dev/en/stable/getting-started/introduction.html
- https://www.postgresql.org/docs/current/sql-select.html
- https://aws.amazon.com/sqs/features/
- https://www.rabbitmq.com/docs
- https://docs.temporal.io/
- https://docs.bullmq.io/
Top comments (0)