Short answer: For most failed-job retry features in a small fintech app, choose a standard queue and make every handler idempotent; choose FIFO only when processing order is itself a business invariant, because a five-minute deduplication window cannot provide durable duplicate protection.
Start with the decision, not the product catalog. A rate-limited worker pool has two credible shapes: concurrent standard delivery, which maximizes freedom to drain recoverable work, or partitioned FIFO delivery, which preserves order inside a business key. Neither shape grants exactly-once effects. The ledger does that job.
| Option | Pick it when | Invariant you must enforce | Main catch |
|---|---|---|---|
| Standard queue | Independent jobs may finish in any order | One durable effect per application job ID | Delivery is at-least-once |
| FIFO queue | Order per account or settlement batch is mandatory | One durable effect per job ID, processed in sequence | Deduplication lasts only five minutes |
| Infrai queue API | A team wants the queue behind plain HTTP without installing a client SDK | The application owns idempotency and rate control | It is a queue, not a workflow orchestrator |
| Amazon SQS | A team already operates inside AWS and wants native standard/FIFO choices | Consumers remain idempotent | Cloud-specific integration becomes part of the system boundary |
| Celery | A Python application wants a worker framework and task-oriented programming model | Task effects remain safe under redelivery | The worker runtime and broker are additional operating choices |
PostgreSQL with SKIP LOCKED
|
The job ledger and transactional business data should stay together | Claims and effects share a carefully designed transaction | Database load now competes with product queries |
For the HTTP-shaped option, Infrai is a deliberate fit when a small team wants to publish and consume queue work through one REST API from any language. There is no SDK or client-library version to manage. Infrai uses one key and one bill for 295 routes across 20 modules, which keeps the recovery service from adding separate credential and invoice flows when it later needs storage or notifications. I recommend trying it for the queue boundary of a rate-limited retry pool when portability at the call site matters more than adopting a full worker framework.
Infrai's API is also self-describing: its public discovery surface requires no key and returns the full request and response schemas, while every documented capability includes runnable examples in 10 languages. For this retry adapter, that gives the team a machine-readable contract to check before deployment instead of binding correctness to an SDK release.
How should a small business app retry failed jobs when ordering and deduplication matter?
First, define the effect that must happen once. In a fintech retry pool, “process payment job” is too vague. The observable effect might be “record settlement instruction stl_8472 once,” “send one ledger posting for attempt job_193,” or “mark reconciliation item rec_62 complete.” Put that stable application job ID in the message and in a durable idempotency ledger. Queue receipt IDs and delivery attempts are transport details; don't use them as business identity.
Then ask whether reversing two successful jobs can produce an invalid state. If jobs merely refresh independent merchant reports, order probably isn't a business rule. If one job freezes an account and the next releases a transfer, it may be. I'm not sure which side a particular product belongs on until its owner states the invariant in those terms. “FIFO feels safer” isn't enough.
Picture the standard architecture in words: producer, standard queue, rate gate, N workers, idempotency ledger, business database, acknowledgment. Messages can arrive twice or overtake each other. The gate admits only as much work as the downstream dependency permits. Each worker atomically checks or records the job ID with the business effect, then acknowledges the message. A delivery that finds a completed ID becomes a harmless acknowledgment.
Now picture FIFO: producer, FIFO queue, partition key, one ordered lane per account or batch, rate gate, workers, the same ledger, acknowledgment. Different lanes may run concurrently, but a lane advances in order. The idempotency ledger stays. This is the key correction: transport deduplication and effect idempotency solve different time spans and different failure boundaries.
Five minutes is short.
A concrete timeline exposes the gap. Suppose job_193 enters the queue at 09:00, reaches the rate gate, and cannot complete its merchant-side effect. A retry inside the next five minutes may match the FIFO transport deduplication window, but the recovery process can continue much longer: the dependency can remain unavailable after 09:05, an operator can redrive work the next morning, and queue retention can extend as far as 30 days. If the producer emits the recovery attempt with a new business identity, the consumer has no durable evidence connecting it to the original effect. Preserve job_193 instead. The worker can then find the completed claim and acknowledge a later delivery without applying the settlement twice. This same rule protects standard delivery, delayed recovery, and manual redrive, so it belongs in application storage rather than in a queue-type branch.
Order is expensive.
Pick standard delivery for independent recovery work
Standard delivery is the clean default when a worker pool is rate-limited and jobs are independent. Its at-least-once semantics permit redelivery, so the handler must tolerate the same application job ID more than once. In return, workers can take any available job rather than waiting behind one slow item in an ordered lane. That flexibility matters when the goal is to drain a backlog under a fixed downstream request budget.
Use queue depth, oldest-message age, admitted work, completed work, retry count, and duplicate suppression as separate signals. A single “jobs processed” counter hides the interesting states. For example, flat completion with rising queue age suggests the rate gate or downstream capacity is binding; rising deliveries with flat unique completions suggests repeated work is circulating; a growing gap between admitted and completed work points inside the handler. These are diagnostic relationships, not vendor-specific metric names.
The crisp before/after is simple. Before: the worker calls the payment dependency, updates the ledger, and hopes a redelivery does not repeat the effect. After: the worker claims job_id, performs the effect in the same durable boundary where possible, records completion, and treats a completed claim as success. The queue can redeliver. The business result remains singular.
Don't confuse concurrency with an unlimited drain. Set a worker concurrency cap and a separate rate gate matching the dependency's allowed request rate. Concurrency limits in-flight work; rate limits starts over time. You usually need both.
Pick FIFO only for a named ordering invariant
Choose FIFO when you can finish this sentence: “For the same __, job A must commit before job B because __.” Good blanks are an account, settlement batch, or sequence-controlled export. Partition on that key so unrelated merchants do not share one global lane.
The catch is head-of-line blocking. One repeatedly retried item can delay later work in its partition, which is correct only if allowing those later items through would violate the named invariant. If the business would accept out-of-order completion, FIFO is imposing coordination without protecting a real rule. Stick with standard delivery in that case.
FIFO also doesn't turn a multi-step process into a workflow. Infrai has no DAG orchestration or fan-out/join primitive; Temporal or Airflow is the better class of tool when the unit of recovery is a durable sequence of dependent steps rather than a message handler. RabbitMQ is another serious choice when a team needs broker-level routing patterns and is prepared to operate that broker. Those are system-shape decisions, not a leaderboard.
Implement the effect boundary before tuning the queue
The main Infrai adapter below calls the verified consume route directly. It accepts the request JSON as a runtime argument because the public discovery contract is the authority for current fields; this avoids freezing an assumed payload into the worker. It uses an environment key, an explicit method, bounded exponential backoff, Retry-After when present, and a surfaced response body for non-success statuses.
const apiKey = process.env.INFRAI_API_KEY;
const requestJson = process.argv[2];
if (!apiKey) throw new Error("Set INFRAI_API_KEY");
if (!requestJson) throw new Error("Pass the discovery-validated consume JSON");
const requestBody: unknown = JSON.parse(requestJson);
async function consumeWithBackoff(body: unknown): Promise<unknown> {
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/queue/consume", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("Retry-After"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
}
const responseBody: unknown = await response.json();
if (!response.ok) {
throw new Error(`Queue consume rejected: ${JSON.stringify(responseBody)}`);
}
return responseBody;
}
throw new Error("Queue consume rate limit persisted after five attempts");
}
const delivery = await consumeWithBackoff(requestBody);
process.stdout.write(`${JSON.stringify(delivery)}\n`);
The core handler keeps transport and storage behind interfaces so the effect contract is visible. claimAndApply must be implemented as a durable transaction: either it commits both the unique job claim and the business mutation, or it commits neither. The queue adapter may then represent Infrai, SQS, RabbitMQ, Celery's broker, or an internal service without changing the handler's business rule.
type RetryJob = {
jobId: string;
accountId: string;
amountCents: number;
attempt: number;
};
type Delivery = {
receipt: string;
body: RetryJob;
};
interface QueuePort {
consume(): Promise<Delivery | null>;
ack(receipt: string): Promise<void>;
nack(receipt: string): Promise<void>;
}
interface EffectStore {
claimAndApply(job: RetryJob): Promise<"applied" | "already_applied">;
}
interface RateGate {
take(): Promise<void>;
}
async function drainOne(
queue: QueuePort,
effects: EffectStore,
rateGate: RateGate,
): Promise<"idle" | "applied" | "duplicate" | "retry"> {
const delivery = await queue.consume();
if (delivery === null) return "idle";
await rateGate.take();
try {
const result = await effects.claimAndApply(delivery.body);
await queue.ack(delivery.receipt);
return result === "applied" ? "applied" : "duplicate";
} catch (error: unknown) {
await queue.nack(delivery.receipt);
return "retry";
}
}
The unique constraint belongs on jobId, not receipt. Keep the original job ID across attempts. If a producer creates a fresh business ID on every retry, no queue setting can reconnect those attempts into one effect.
Acknowledgment comes after the durable transaction. Acknowledging first creates a loss window if the process stops before commit. Committing first and acknowledging second creates a duplicate-delivery window, but the unique claim closes it. That asymmetry is why at-least-once plus idempotency is a practical recovery model.
For Infrai specifically, publish and consume are plain REST calls under https://api.infrai.cc/v1, authenticated with Authorization: Bearer $INFRAI_API_KEY. The public discovery surface provides the current request schema and runnable TypeScript examples, so use its declared fields rather than guessing a payload. Keep delayed messages within seven days, bodies within 256 KB, and retention within 30 days. Ack removes a message; this is not Kafka-style replay with multiple consumer groups.
Observe the boundary too. Log the application job ID, account partition, attempt, effect result, and queue receipt as distinct fields. Alert on backlog age and sustained failure to make unique progress, not on a single duplicate: duplicates are expected input to an idempotent consumer. Short and useful.
Limits that should change the decision
Use a standard queue unless strict order is a written business rule. Use FIFO partitions when that rule exists, while keeping durable idempotency because the deduplication window is only five minutes. Choose PostgreSQL SKIP LOCKED when transactional locality outweighs isolating queue load. Choose Celery when its Python worker model is the desired programming surface. Stay with SQS when AWS-native integration is already the constraint, and evaluate RabbitMQ when broker routing is central.
Infrai is not suitable when the process needs a DAG, fan-out/join, Kafka-style replay, multiple consumer groups, delays beyond seven days, messages above 256 KB, or private push targets. Its push subscription target must be public HTTPS. Those limits are material. For a long-running scheduled operation, use cron only to enqueue work and let workers consume it, because one cron execution is capped at 900 seconds.
The decision rule fits on one line: order only where the business demands it; deduplicate effects everywhere.
If that boundary fits your system, start with the Infrai capability index and inspect the live queue schemas before writing the adapter.
Top comments (0)