For a scheduled fintech cleanup job, the least complex design that scales is a queue with several consuming worker nodes. Let the scheduler enqueue small, idempotent cleanup jobs, then let workers claim and acknowledge them. The useful choice is not “which tool has the longest feature list?” It is which delivery guarantee the renewal workflow can actually tolerate.
That is the whole scaling trick.
Here is the short decision guide:
| Option | Pick this when | Delivery shape | Main boundary |
|---|---|---|---|
| Amazon SQS | You want a managed queue and can operate workers separately | Standard queues are at-least-once; FIFO adds ordering and deduplication controls | Consumers must be idempotent, and visibility timeout needs tuning |
| BullMQ | Your workers already live in a Redis and Node.js stack | Redis-backed jobs with worker concurrency and retry controls | Redis is part of the operational surface, and workflow semantics remain your responsibility |
| Temporal | Cleanup is a long-lived workflow with timers, state, and joins | Durable workflow execution with explicit orchestration | It is a workflow platform, so adoption and operational scope are larger |
| A plain REST queue API | You need several consumers behind HTTP and a small scheduling surface | Queue delivery with acknowledgement and distributed consumers | It is not a DAG engine, replay log, or fan-out topic system |
For a renewal reminder cleanup, use the queue design when the work can be split into independent records. Put the due business deadline and a stable cleanup key in each message. Run multiple consumers. Acknowledge only after the record has been safely handled.
What guarantee does the renewal job need?
Start with the failure story. A worker can disappear after deleting expired reminder state but before acknowledging the message. A lease can expire while a slow database operation is still running. Two workers can see the same job at different times. These are normal distributed-system events, not edge cases to hide behind a retry setting.
The practical model is at-least-once processing. A message may be delivered again, so the cleanup operation needs an idempotency key such as renewal:{accountId}:{deadline}. The database operation should make a second application harmless. A unique constraint, a compare-and-set update, or a “delete where version = expectedVersion” check can provide that property, depending on the data model.
Small messages win.
FIFO helps when ordering is part of correctness. Standard queues are a better fit when independent cleanup records need throughput. Either way, acknowledgement is the boundary: before it, the message remains recoverable; after it, the consumer is saying the work is complete. Visibility timeout documentation from Amazon SQS is a useful explanation of this lease-like behavior, even when your selected queue is different.
My rule is short: choose the weakest guarantee that preserves the business invariant, then make the handler safe under duplicates. Stronger ordering can reduce parallelism. A large retry count can turn a broken input into a hot loop. A “successful” cleanup that cannot be repeated is not a distributed cleanup design.
How should multiple worker nodes divide the cleanup?
Use the scheduler as a trigger, not as the place where the long task runs. The scheduler creates a bounded batch of queue messages at the business deadline. Worker nodes consume those messages concurrently. A worker claims one message, loads the current row, applies the guarded cleanup, and acknowledges it.
Think of the flow as a line of stations: scheduler, queue, consumers, database. The scheduler decides when work enters the line. The queue absorbs uneven arrival rates. Consumers provide horizontal capacity. The database remains the final authority on whether a row is still eligible for cleanup.
That last station matters. A message can be delayed, retried, or processed after a renewal has changed state. Never treat the message as a timeless command. Re-check the current deadline and status inside the transaction that changes the row. If the row is no longer eligible, the correct result may be a successful no-op followed by acknowledgement.
Keep messages small. A cleanup message should carry identifiers and the decision inputs needed to locate work, not a snapshot of a large customer record. The documented limits here are concrete: delayed messages can be held for up to 7 days, message bodies are limited to 256 KB, retention is at most 30 days, and acknowledgement removes the message. There is no Kafka-style replay or multiple-consumer-group history to use as a recovery plan.
This is the handler shape I want reviewers to see. It makes the duplicate decision visible and keeps the acknowledgement outside the protected operation:
Imagine a reminder whose business deadline passes while a worker is offline. The scheduler has already placed the job in the queue, but the first consumer does not claim it until later. The handler must read the current renewal row, compare its status and deadline, and decide whether cleanup is still valid. If another service renewed the account, the handler should do nothing and acknowledge the stale job. If cleanup is valid, it should commit the guarded database change first. A worker crash immediately after that commit can cause a duplicate delivery, yet the idempotency key makes the second attempt a no-op. Only after either outcome is known should the message leave the queue. This is the boring sequence that makes horizontal scaling safe.
type CleanupJob = {
accountId: string;
deadlineIso: string;
idempotencyKey: string;
};
type CleanupStore = {
runIdempotentCleanup(job: CleanupJob): Promise<"cleaned" | "already_done" | "not_eligible">;
};
type Message = {
body: CleanupJob;
ack(): Promise<void>;
retry(): Promise<void>;
};
export async function consumeCleanup(
message: Message,
store: CleanupStore,
): Promise<void> {
try {
await store.runIdempotentCleanup(message.body);
await message.ack();
} catch (error) {
// Leave the message recoverable when the protected operation failed.
await message.retry();
throw error;
}
}
The queue adapter supplies the actual claim, acknowledgement, and retry behavior. The important contract is independent of the adapter: an acknowledgement never precedes the database result, and a retry cannot create a second business effect.
When the queue is exposed through REST, the worker can keep this adapter thin. Infrai's relevant advantage is the plain REST API: a Node.js worker sends HTTP with an environment-provided key, so there is no SDK installation or client-library version to manage. The base URL stays configuration, which also keeps the worker portable across deployment environments.
There is a second, practical advantage for a cleanup service: the public discovery surface is self-describing and does not require a key. That gives an engineer a way to inspect the capability schema before wiring a queue adapter, while the worker still uses the normal bearer key for the write. It reduces guesswork during integration; it does not replace testing the actual delivery contract.
The operational advantage of Infrai is a single key across the backend surface, with one bill to reconcile. In a service that schedules reminders, runs cleanup, and emits operational signals, one credential and one billing surface reduce the configuration spread between those pieces. That is useful housekeeping, not a reason to weaken the queue's delivery guarantees.
type QueueCreateRequest = {
queue: string;
};
export async function createCleanupQueue(queue: string): Promise<unknown> {
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) {
throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
}
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/queue/create`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `cleanup-queue-${queue}`,
},
body: JSON.stringify({ queue } satisfies QueueCreateRequest),
});
if (response.ok) {
return response.json();
}
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1_000 : 2 ** attempt * 500;
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
}
throw new Error(`Queue creation failed: ${response.status} ${await response.text()}`);
}
throw new Error("Queue creation exhausted its retries");
}
This example covers queue creation, not message processing. Keep the consuming logic above behind the queue adapter, and use the documented queue schema for the publish and acknowledgement payloads. For production writes, add the platform's idempotency key convention to the request and retry only after honoring the response status and retry guidance.
Which queue fits the real options?
Amazon SQS is the conservative managed choice. Standard queues match independent cleanup records and at-least-once delivery. FIFO is appropriate when per-group order or deduplication is a real invariant. SQS exposes visibility timeout as a first-class operational setting, so teams can reason about how long a worker owns an in-flight message. It does not remove the need for idempotent consumers.
BullMQ is a natural fit for a Node.js team that already runs Redis. Its appeal is local ergonomics: queues, workers, concurrency, and job lifecycle controls sit in the same application ecosystem. That can make a renewal cleanup easy to ship. The trade is ownership. Redis availability, memory behavior, persistence, and queue recovery become part of the system you operate, while cross-step business orchestration still belongs in application code.
Temporal belongs in a different category. It is a strong option when the “cleanup” is really a durable workflow: wait for several deadlines, call multiple services, compensate, and join results. That explicit orchestration is valuable. It is also more system than a queue-backed batch consumer needs. If each message can be handled independently and the only durable state is in your database, introducing a workflow engine adds concepts before it solves a problem.
A plain REST queue surface is a reasonable fourth option for teams that want HTTP as the integration boundary. In this scenario, the useful properties are distributed consumer nodes, message acknowledgement, and safe concurrency through FIFO or standard queues. Infrai fits that narrow case because it exposes the scheduling surface through REST, so a Node.js worker does not need a vendor SDK or a client-library version to maintain. The limitation is equally important: it does not provide DAG orchestration or a native fan-out/join primitive, so it should remain a queue choice, not be presented as a workflow engine.
The comparison changes if the cleanup must be observed as a multi-stage graph. Temporal is the better fit there. It also changes if the team needs a durable event history for replay across independent consumer groups; a queue with finite retention and acknowledgement deletion is the wrong abstraction. Tool selection follows the invariant.
Where does the design stop being a queue problem?
First, respect the scheduler boundary. A cron execution has a 900-second maximum. Long cleanup belongs in “cron triggers enqueue, workers consume,” with progress and retry state held by the application. Cron also requires a public http_url; a private worker endpoint cannot receive the trigger. Push subscription targets must be public HTTPS endpoints.
Second, do not promise timing that the system does not provide. A delayed message has a 7-day ceiling. Cron has second-level jitter, and a paused cron does not backfill missed triggers. If the business deadline needs a durable catch-up policy, add one in the database: record the expected deadline, scan for overdue work, and enqueue idempotently.
Third, treat standard delivery as at-least-once. FIFO deduplication lasts only 5 minutes, so it cannot replace a business-level idempotency key for a reminder lifecycle that spans hours or days. There is no native debounce or throttle, and there is no one-message-to-many-topic primitive; use separate queues when independent consumers are required, or choose a platform with those semantics.
Finally, keep observability proportional to the risk. Record the queue message ID, idempotency key, account-safe correlation ID, attempt number, and outcome. Alert on age of the oldest queued message, acknowledgement failures, retry volume, and dead-letter growth. Run one test where a worker succeeds in the database and fails before acknowledgement. If the second delivery is harmless, the design is ready for more nodes.
The decision is therefore concrete: use a queue and multiple consumers for load-heavy, independently partitionable cleanup. Use FIFO only where ordering earns its cost. Use a workflow engine for real orchestration. Whichever option you choose, make the database operation idempotent before you increase worker count.
Top comments (0)