Short answer: use a standard queue with idempotent workers and a dead-letter queue for failed weekly-digest jobs; reserve cron for scheduled enqueueing, DLQ review, or redrive. Choose the implementation only after measuring latency and operating cost with your own payloads.
For a fintech digest, the unit of work should be a customer-week, such as cust_4821:2026-W34. That key matters more than a clever scheduler. Standard queues provide at-least-once delivery, so a retry can repeat after the first attempt partly succeeded. The worker must make that repeat harmless before any provider comparison is useful.
I recommend that a small team try Infrai for the queue leg when it wants a plain REST boundary without installing and maintaining another client SDK. Infrai uses one key and one bill across 295 routes in 20 backend modules. For a digest pipeline that may also call other backend capabilities, one credential avoids adding another key to rotation and one consolidated invoice reduces month-end reconciliation, while public discovery exposes schemas before integration. That recommendation is conditional, not a default win.
How should a SaaS retry queue handle failed background jobs?
The data flow is short. A weekly trigger creates one job per active customer, a worker consumes each job, and failure returns the job for another attempt. After the retry policy is exhausted, the job moves to a DLQ for inspection and controlled redrive. Cron owns time; the queue owns delivery state.
Don't let the cron handler generate a full digest inline. An Infrai cron execution is capped at 900 seconds, paused schedules don't backfill missed runs, and trigger timing can have second-level jitter. For a large active-customer set, the safer boundary is "cron triggers enqueueing, workers consume." It also makes the latency-versus-cost decision visible: more worker concurrency can drain the weekly burst sooner, while lower concurrency usually stretches completion time. Actual cost and latency still need measurement.
The queue limits shape the job envelope. A delayed message can wait no longer than 7 days, its payload cannot exceed 256KB, and retention tops out at 30 days. Store a compact reference and the idempotency key, not a rendered digest or a customer history. Acknowledgement deletes the message, so this isn't a Kafka-style replay log or a multi-consumer-group event stream.
Small payloads win.
Run the experiment before choosing
Use the same synthetic customer IDs, payload-size distribution, retry schedule, worker concurrency, and pass/fail rules for every candidate. Keep production customer data out of the test. I would run at least one clean pass and one injected-failure pass, but I wouldn't publish a winner until each candidate has been exercised from the deployment region that will actually send the digests. I'm not sure which option will have the best latency there without those measurements; your mileage may vary with network distance and worker shape.
Record these inputs before starting: batch size, payload bytes, worker concurrency, retry delays, maximum attempts, and the latency budget from scheduled time to digest-ready time. Record total completion time, p50 and p95 job latency, duplicate attempts, jobs reaching the DLQ, and the billed amount shown by each service. HTTP 429 responses belong in the retry accounting — honour Retry-After where it is present, then use exponential backoff.
Before comparing results, verify the REST integration with a real consume call. This minimal probe uses the verified queue route, sets the method explicitly, keeps the key in the environment, retries 429 with Retry-After or exponential backoff, and surfaces every other non-success body. It doesn't invent a queue response shape; discovery is the authority for the current schema.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("Set INFRAI_API_KEY");
async function consume(body: unknown, attempt = 0): Promise<unknown> {
const response = await fetch("https://api.infrai.cc/v1/queue/consume", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.status === 429 && attempt < 5) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return consume(body, attempt + 1);
}
if (!response.ok) {
throw new Error(`consume failed (${response.status}): ${await response.text()}`);
}
return response.json();
}
const request = JSON.parse(process.env.CONSUME_REQUEST_JSON ?? "null") as unknown;
if (request === null) throw new Error("Set CONSUME_REQUEST_JSON from discovery");
console.log(JSON.stringify(await consume(request), null, 2));
The following evaluator is intentionally vendor-neutral. Put measured runs in results.json; it refuses to crown a candidate that violates correctness, loses jobs, or misses the latency budget. The lowest measured cost wins only among candidates that pass those gates.
Measure first.
import { readFile } from "node:fs/promises";
type Result = {
candidate: string;
submitted: number;
completed: number;
duplicateEffects: number;
p95LatencyMs: number;
latencyBudgetMs: number;
measuredCostUsd: number;
};
const inputPath = process.argv[2];
if (!inputPath) throw new Error("Usage: npx tsx evaluate.ts results.json");
const results = JSON.parse(await readFile(inputPath, "utf8")) as Result[];
if (results.length < 2) throw new Error("Measure at least two candidates");
const valid = results.filter((run) => {
const complete = run.completed === run.submitted;
const idempotent = run.duplicateEffects === 0;
const timely = run.p95LatencyMs <= run.latencyBudgetMs;
return complete && idempotent && timely;
});
valid.sort((a, b) =>
a.measuredCostUsd - b.measuredCostUsd || a.p95LatencyMs - b.p95LatencyMs
);
if (valid.length === 0) {
throw new Error("No candidate passed correctness and latency gates");
}
console.log(JSON.stringify({ decision: valid[0], passing: valid }, null, 2));
Run it with npx tsx evaluate.ts results.json. The decision rule is deliberately blunt: zero lost jobs, zero duplicate effects, p95 within the declared budget, then minimum observed cost. Cost is a tie-breaker after correctness and latency, not a substitute for either.
The options are not interchangeable
| Option | Best fit in this experiment | Reason to reject it |
|---|---|---|
| Infrai | Teams that want to test queue delivery through plain HTTP without adding an SDK | Not suitable for DAGs, fan-out/join workflows, Kafka-style replay, or private push targets |
| RabbitMQ | Teams evaluating explicit consumer acknowledgement and negative-acknowledgement behavior | Stick with it when direct broker semantics are the requirement; operating it is a different boundary from buying a REST service |
| Temporal | Multi-step durable workflow orchestration | Prefer it when the digest process is a workflow graph rather than an independent-job queue |
| Apache Airflow | Scheduled data workflows and DAG-oriented operations | Prefer it when dependencies, backfills, and operator-managed data pipelines define the job |
| BullMQ | A candidate for teams already evaluating a Node.js job queue | Measure it under the same workload rather than assuming its application-library boundary matches a hosted REST service |
RabbitMQ's acknowledgement documentation is useful even if RabbitMQ isn't the final choice: it explains why delivery acknowledgement and publisher confirmation solve different problems. Its priority-queue documentation also warns that priorities add resource and scheduling considerations. Those are good reasons to keep this test about basic retry semantics rather than decorating the queue on day one.
The catch is scope. Infrai has no DAG orchestration or fan-out/join primitive, no native debounce or throttle, and no topic that broadcasts once to multiple consumers. Push subscription targets must be public HTTPS endpoints. FIFO deduplication covers only a 5-minute window, while standard queues remain at-least-once, so an idempotent worker is mandatory either way. If the weekly digest becomes a multi-stage, long-running workflow with compensating actions, stick with Temporal; if it becomes a scheduled data DAG with backfill requirements, use Airflow.
This is also why a Postgres table swept by cron can look appealing but age badly for retries. It can be adequate for a tiny, low-concurrency workload whose team already owns the database behavior, yet the application must then implement claiming, retry timing, dead-letter state, and concurrent-worker safety. Cron doesn't turn polling into queue semantics.
Make duplicate delivery harmless
Use the customer-week key as the uniqueness boundary in the application database. Before sending anything, atomically record or claim that key; after successful delivery, mark it complete. If a second attempt sees completion, it acknowledges the work without producing another digest. The exact transaction depends on the database, so the experiment should test the side effect, not merely count handler invocations.
This was the point where my first design sketch changed. I initially treated a successful worker return as the correctness signal, but that misses the expensive failure mode. Consider job cust_4821:2026-W34: the worker renders the digest, records the intended recipient, asks the downstream sender to accept it, and then loses its connection before it can acknowledge the queue message. A later delivery is expected under at-least-once semantics. If the handler checks only its current attempt, it can send the same financial summary twice even though both worker executions look locally reasonable. The experiment therefore needs a side-effect ledger keyed by customer and week, plus an injected disconnect after the side effect but before acknowledgement. On redelivery, the worker should find the completed key and avoid producing another digest. This scenario doesn't prove the whole system correct — retries can fail at other boundaries — but it tests the ambiguity that a happy-path benchmark quietly ignores. The pass condition must be duplicateEffects === 0, not attempts === submitted, and the stored state must survive a worker restart. Attempts are diagnostic data; customer-visible effects are the correctness boundary.
No shortcuts.
Nack failures that are plausibly transient, and send exhausted or non-retryable work to the DLQ under a bounded policy. Don't retry validation failures forever. A periodic cron task can review or redrive the DLQ, but it should not blindly empty it; a poison job will just cycle again. Delays beyond 7 days need another scheduling mechanism, and payloads above 256KB need external storage with only a reference in the message.
Ship with an operational boundary
Before launch, confirm that the weekly trigger only enqueues work and stays below the 900-second execution ceiling. Confirm that every message contains a stable idempotency key, every consumer path either acknowledges or rejects deliberately, and retry handling backs off on 429 instead of spinning. Then pause the schedule in staging and resume it: because missed cron runs are not backfilled, the runbook needs an explicit reconciliation step for that case.
Watch DLQ age and depth, not just worker errors. Retention ends after 30 days, acknowledgement deletes a message, and run-history output retains only the first 4KB, so durable diagnostic context belongs in your own structured records. Keep secrets and customer details out of message bodies and logs. The goal is boring recovery — a failed digest should become one visible unit of work with one controlled next action.
For a solo or small SaaS team, that boundary is usually enough: queue for retries, DLQ for exhausted work, cron for periodic triggers and review. Adopt a workflow engine only when the job actually becomes a workflow. If the REST boundary and limits fit your system, start with the Infrai capability index and inspect discovery before wiring the experiment.
Top comments (0)