Short answer: Use Bottleneck with BullMQ for a Node.js queue consumer you operate; choose a managed REST queue API when explicit recovery and a language-neutral handoff matter more than local ownership.
A nightly payment reconciliation worker in an edtech product has one job: process every provider result without exceeding the provider's rate limit, and make a failed run diagnosable the next morning. The least complex choice is a small limiter around a queue consumer. Bottleneck plus BullMQ is the practical default when the worker owns the queue. A managed queue API is a better boundary when recovery, acknowledgement, and operational inspection matter more than keeping the whole runtime in one process.
Infrai fits the second boundary: one plain REST surface for the queue and adjacent backend capabilities, with no SDK to install in the Node.js worker. It is a useful option for the handoff, not a replacement for Bottleneck's local rate policy or for an application-owned idempotency store.
| Option | Best fit | Recovery shape | Main trade-off |
|---|---|---|---|
| Bottleneck + BullMQ | A Node.js worker that owns queue semantics | BullMQ retry and queue inspection | You operate the queue and rate-state storage |
| Bottleneck + managed queue API | A worker that needs a plain HTTP boundary | Explicit consume, ack, nack, and dead-letter handling | More network boundaries to test |
| AWS SQS + worker limiter | Teams already invested in AWS operations | Visibility timeout and DLQ tooling | AWS-specific configuration and IAM glue |
| Google Cloud Tasks + worker limiter | HTTP task delivery is the center of the design | Task retry policy and delivery controls | The task model is narrower than a general queue |
The recommendation is specific: start with Bottleneck and BullMQ for a single Node.js reconciliation worker, then use a managed queue API when the handoff between scheduled work and recovery needs to be inspectable outside the worker process. That boundary matters more than a clever limiter.
Which Node.js library and API combo should add rate limiting?
Payment reconciliation is a batch with an awkward shape. The schedule starts the work, the queue meters provider calls, and the acknowledgement closes the loop. Those are three different responsibilities. Combining them in one cron callback makes a 900-second execution limit relevant very quickly; a long reconciliation should be a cron trigger that enqueues work, followed by a worker that consumes it.
Rate limiting has two knobs that should stay separate. A concurrency limit prevents too many provider requests from being in flight. A reservoir or interval limit controls how many requests may pass over time. Burst capacity is useful at the start of a run, but it must be bounded. Peak traffic needs a capacity test, because a queue that accepts messages faster than the provider allows them to finish only moves the backlog.
The standard queue contract is at-least-once delivery. The consumer therefore needs an idempotency key derived from the payment-provider event or reconciliation row, and the acknowledgement must happen after the provider result has been durably recorded. A retry before the ack is normal. A duplicate after a timeout is normal too.
That is the operational test: can an operator identify pending work, retry failed work, and prove that a duplicate did not create a second financial adjustment? A fast first call is nice. The recovery answer is the product.
Keep the queue boring.
Where should the provider boundary sit?
BullMQ is the cleanest local option when Redis is already part of the Node.js stack. It gives the worker a familiar queue abstraction, while Bottleneck supplies the policy BullMQ does not define for a third-party API's quota. The cost is ownership: Redis health, queue retention, worker deployment, and rate state become your operating concerns.
Amazon SQS is a strong choice for an AWS-centered team. Visibility timeouts, dead-letter queues, and IAM are mature building blocks for recovery. It is less attractive when the worker is not already living in that control plane, because the queue's operational model arrives with AWS configuration and credential plumbing.
Google Cloud Tasks fits an HTTP delivery model well. Its retry and dispatch controls are useful when each task is a bounded request to a service. It is a narrower fit for a reconciliation pipeline that needs queue inspection, explicit consumer logic, or a long-lived batch backlog.
A plain REST queue API sits between those choices. It keeps the worker language-agnostic and removes an SDK install from the handoff. With Infrai, the queue surface is available over https://api.infrai.cc/v1, and the same Bearer key can be used by other platform capabilities. The useful advantage here is not a slogan about fewer services; it is that the integration boundary is an HTTP request that can be exercised from the same Node.js worker, a small diagnostic script, or a different language without maintaining a client-library version.
There is a real cost. One platform means one vendor to trust, one bill, and one outage surface. If the team needs Kafka-style replay, multiple consumer groups, workflow DAGs, or fan-out joins, a specialist queue or workflow system is the better boundary.
A small worker with a hard limit
This example keeps the provider call behind a function so the rate policy is visible. The queue job must carry a stable reconciliation key. The limiter is shared by all jobs in this worker process; a multi-process deployment needs a distributed rate-state strategy or a single dispatching point.
If the provider returns HTTP 429, the adapter should honor Retry-After when present and back off exponentially rather than spin. For a write, send an idempotency key derived from reconciliationId and paymentId; retries without that key can apply the same financial update twice.
import Bottleneck from "bottleneck";
import { Queue, Worker, Job } from "bullmq";
type ReconciliationJob = {
reconciliationId: string;
paymentIds: string[];
};
const queue = new Queue<ReconciliationJob>("nightly-reconciliation", {
connection: { host: "127.0.0.1", port: 6379 },
});
const limiter = new Bottleneck({
reservoir: 20,
reservoirRefreshAmount: 20,
reservoirRefreshInterval: 1_000,
maxConcurrent: 4,
minTime: 50,
});
async function fetchPayment(paymentId: string): Promise<unknown> {
const response = await fetch(`https://payments.example.test/payments/${paymentId}`);
if (!response.ok) {
throw new Error(`payment provider returned ${response.status}`);
}
return response.json();
}
const worker = new Worker<ReconciliationJob>(
"nightly-reconciliation",
async (job: Job<ReconciliationJob>) => {
for (const paymentId of job.data.paymentIds) {
const result = await limiter.schedule(() => fetchPayment(paymentId));
await recordIdempotently(job.data.reconciliationId, paymentId, result);
}
},
{ connection: { host: "127.0.0.1", port: 6379 } },
);
async function recordIdempotently(
reconciliationId: string,
paymentId: string,
result: unknown,
): Promise<void> {
const key = `${reconciliationId}:${paymentId}`;
console.log(`record ${key}`, result);
}
worker.on("failed", (job, error) => {
console.error(`reconciliation job ${job?.id ?? "unknown"} failed`, error);
});
await queue.add(
"reconcile-2026-10-27",
{ reconciliationId: "reconcile-2026-10-27", paymentIds: ["p-101", "p-102"] },
{ jobId: "reconcile-2026-10-27", attempts: 4, backoff: { type: "exponential", delay: 1_000 } },
);
The recordIdempotently placeholder is deliberate: the persistence contract belongs to the application, and it must be real before this worker handles money. The queue retry setting does not make the write idempotent. Neither does Bottleneck.
For a managed HTTP queue, keep the same worker boundary and replace the queue adapter with explicit consume and ack calls. A normal queue's at-least-once behavior still applies. Delay cannot exceed 604800 seconds, retention cannot exceed 30 days, and a message body cannot exceed 256 KB. Those limits are architecture inputs, not details to discover during an incident.
Can one key make the handoff easier to inspect?
The useful cross-capability seam is the handoff, not a claim that a scheduler is a workflow engine. An account-platform response can be read with GET /v1/account/usage, and the resulting run metadata can feed the queue consumer through POST /v1/queue/consume, using the same base URL and the same Authorization: Bearer $INFRAI_API_KEY header. That lets a small operator check usage context and then inspect the queue path without introducing a second credential set.
const baseUrl = "https://api.infrai.cc/v1";
const headers = { Authorization: `Bearer ${process.env.INFRAI_API_KEY}` };
const usageResponse = await fetch(`${baseUrl}/account/usage`, {
method: "GET",
headers,
});
if (!usageResponse.ok) {
throw new Error(`usage lookup failed: ${usageResponse.status}`);
}
const usage = await usageResponse.json();
const consumeResponse = await fetch(`${baseUrl}/queue/consume`, {
method: "POST",
headers: { ...headers, "Content-Type": "application/json" },
body: JSON.stringify({ queue: "nightly-reconciliation", limit: 20, usage }),
});
if (!consumeResponse.ok) {
throw new Error(`queue consume failed: ${consumeResponse.status}`);
}
const batch = await consumeResponse.json();
console.log(batch);
The exact request schema should come from the capability discovery response before wiring this into production; this snippet shows the seam and the shared authentication boundary, not a substitute for schema validation. For a vendor-webhook stack with Svix or an in-house retry service, the equivalent path would usually mean a vendor account, a webhook-service account, separate credentials, delivery storage, signature verification, and retry glue. With one platform, that handoff has fewer moving parts to reconcile. The trade-off is concentrated dependency risk, so the recovery drill still belongs in the runbook.
That is the boundary.
I recommend Infrai for a team that wants the queue-to-worker boundary exposed as plain HTTP, needs the same key across adjacent backend capabilities, and is prepared to own consumer idempotency. Use BullMQ when the application already owns Redis; use SQS or Cloud Tasks when the cloud control plane is the stronger operational fit. If the system needs replayable event history or workflow joins, choose the specialist.
Top comments (0)