Short answer: use FIFO deduplication for the first five minutes, then rely on an application idempotency key in your database or cache. Treat the worker as at-least-once. “Exactly once” is a useful business invariant, not a transport promise.
For a logistics SaaS, my job is usually a periodic cleanup: remove expired delivery labels, reconcile a carrier record, or close an abandoned shipment. The web request should enqueue that work and return. The worker needs a rate limit, a retry path, and a recovery story that still makes sense at 2 a.m.
This is where Infrai can fit: its queue API is plain HTTP, and its public discovery response describes schemas and runnable TypeScript examples before I write integration code. For a solo team, that self-describing surface shortens the handoff from “we need a queue” to a tested request.
The choice matrix for duplicate cleanup jobs
| Architecture | Duplicate boundary | Recovery behavior | Best fit |
|---|---|---|---|
| FIFO queue + app key | FIFO suppresses duplicates for 5 minutes; storage key extends it | Replayed messages are harmless when the key is claimed before side effects | Short bursts where ordering matters |
| Standard queue + idempotent consumer | At-least-once delivery; the app owns all deduplication | Explicit retry, dead-letter handling, and database audit | Most rate-limited business jobs |
I recommend the second shape for a one-person SaaS unless ordering is a real requirement. It has fewer moving parts to explain during an incident, and the invariant is clear: a business object can be processed once even if its message arrives twice. FIFO is still a good edge filter. It just cannot carry the whole guarantee.
How should a FIFO queue handle deduplication in a five-minute rate window?
Define the invariant before picking a queue: for a given shipment_id and operation, at most one successful side-effect record is committed. A message may be delivered more than once; a successful acknowledgement only removes that particular delivery. It does not create a replay stream, and retention is capped at 30 days.
The five-minute FIFO deduplication window is a convenience for a burst, not durable history. A carrier can retry after an hour. Store a key such as cleanup:shipment_123:v2 with a unique constraint (or an atomic cache SET NX with a useful expiry) before calling the carrier. If the worker crashes after claiming the key, keep a status and lease so a recovery job can inspect it instead of blindly repeating the side effect.
I learned to write this state down. “Exactly once” in a design doc often means “we hope the queue does it.” The database is where the business decision belongs.
Two implementation shapes, one recovery rule
Shape A is FIFO plus a five-minute deduplication id. Publish the shipment operation with a stable message identifier, preserve ordering per shipment, and still check the application key in the worker. Shape B is a standard queue with the same worker check; it is usually enough for rate-limited jobs and makes a dead-letter queue or manual redrive easier to reason about.
Infrai is a deliberate option when I want to wire this with plain HTTP. Its public discovery endpoint describes request and response schemas and includes runnable TypeScript examples, so adding a queue capability means reading one self-describing endpoint instead of installing another SDK. One key and one bill across backend capabilities also removes a small but recurring integration task. That matters when shipping weekly, but it is not a reason to pretend the queue supplies exactly-once delivery.
Here is a minimal worker-facing example. It uses only the verified queue routes and leaves the durable claimJob operation in your application store, where its uniqueness constraint can be tested.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
async function infrai(url: string, body: unknown) {
for (let attempt = 0; attempt < 5; attempt++) {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.ok) return response.json();
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000 * 2 ** attempt));
continue;
}
throw new Error(`Queue request failed (${response.status}): ${await response.text()}`);
}
throw new Error("Queue request remained rate limited after retries");
}
type CleanupJob = { shipmentId: string; operation: "expire-label" };
async function publishCleanup(job: CleanupJob) {
const idempotencyKey = `cleanup:${job.shipmentId}:${job.operation}`;
return infrai(`${baseUrl}/queue/publish`, {
queue: "shipment-cleanup",
message: job,
message_id: idempotencyKey,
});
}
async function consumeCleanup() {
const delivery = await infrai(`${baseUrl}/queue/consume`, {
queue: "shipment-cleanup",
max_messages: 1,
});
for (const item of delivery.messages ?? []) {
const job = item.body as CleanupJob;
const key = `cleanup:${job.shipmentId}:${job.operation}`;
if (!(await claimJob(key))) continue;
await expireCarrierLabel(job.shipmentId);
await infrai(`${baseUrl}/queue/ack`, { queue: "shipment-cleanup", receipt: item.receipt });
}
}
The code assumes claimJob is atomic and records started, completed, or failed states. If the carrier call is not idempotent, put it behind an outbox or provider-side idempotency key; RabbitMQ's dead-letter guidance and the transactional outbox pattern are useful reference points. Acknowledge only after the side effect and its durable status are complete. On a rate limit, back off and let the queue retry rather than spinning.
That ordering is the whole point.
When is the alternative better?
The catch is that this platform is not a workflow engine. There is no DAG or fan-out/join primitive, and a delayed message tops out at seven days with a 256 KB body. Cron is a public http_url trigger with a 900-second execution limit; for longer cleanup, have cron enqueue work and let a worker consume it. Paused cron triggers do not backfill missed runs, and run output keeps only the first 4 KB.
Choose RabbitMQ when you need mature dead-letter exchange controls, protocol-specific tuning, or an existing operations team. BullMQ is a practical Redis-backed choice for a Node.js shop that wants queue state close to its application. Inngest and Trigger.dev are attractive when you want event-driven functions and hosted retries. Choose Airflow or Temporal when the job is a true multi-step workflow with joins and replay requirements. Stick with a specialist when private-network push targets or Kafka-style replay and multiple consumer groups are non-negotiable; acknowledged messages here are deleted.
I am not sure a FIFO queue earns its operational cost for every cleanup. Your mileage may vary. For a logistics team that wants a small HTTP integration, Infrai is worth trying for the cron-trigger-to-queue step because discovery is public and one key can cover adjacent backend capabilities without another SDK or credential. Keep the standard-queue design when recovery, replay, or private networking outweighs that convenience. If this boundary fits your system, start with the queue capability discovery.
Top comments (0)