DEV Community

PeterParker8991
PeterParker8991

Posted on

Node.js Rate Limiting Library and API Combo for Recoverable Nightly Queue Consumers

#node #queues #ratelimiting #logistics

A nightly payment reconciliation job has one constraint that changes the answer: a provider throttle must slow the consumer without losing the shipment record that still needs checking tomorrow morning. For a one-person SaaS, I would start with Bottleneck around a queue consumer, then use a queue with explicit acknowledgement and a dead-letter path. BullMQ is the practical Node.js pairing when Redis is already part of the system. A managed queue API becomes attractive when recovery controls matter more than keeping the worker stack in-process.

The goal is controlled backpressure, not a faster loop. Let the queue hold work, let Bottleneck control calls to the payment provider, and acknowledge a message only after reconciliation succeeds. That gives me a retry boundary I can reason about while still leaving time to ship the next feature.

What does recovery need to preserve?

Imagine 18,400 shipment-payment records arrive during the nightly run. The provider accepts 20 requests per second, but the consumer can pull much faster. A naive Promise.all turns that difference into 429 responses, a retry storm, or both. The dangerous part is not the throttle. It is losing the relationship between a payment record and its next attempt.

I want four properties:

  • The queue remains the source of pending work.
  • The limiter controls both burst size and concurrent provider calls.
  • A failure leaves the message unacknowledged or explicitly negative-acknowledged.
  • A successful result is idempotent before acknowledgement.

Standard queues are at-least-once. Duplicate delivery is normal, so the reconciliation write needs a stable key such as shipment_id + settlement_date. A limiter cannot provide that guarantee. It only decides when work may start.

This is also where a short cron invocation stops being the right worker. A cron trigger can enqueue the reconciliation batch, but a long run belongs in a worker. A 900-second execution ceiling and a 7-day maximum message delay make those boundaries concrete: schedule the trigger, enqueue records, and let consumers drain the work.

What library and API combo should I use for adding rate limiting?

Bottleneck handles the local timing decision. BullMQ handles durable jobs when Redis is acceptable. The important order is schedule the provider call, persist the idempotent result, then return successfully so the queue can acknowledge the job.

import Bottleneck from "bottleneck";
import { Queue, Worker, Job } from "bullmq";

const connection = {
  host: process.env.REDIS_HOST ?? "127.0.0.1",
  port: Number(process.env.REDIS_PORT ?? 6379),
};

type PaymentRecord = {
  shipmentId: string;
  settlementDate: string;
  providerPaymentId: string;
};

const queue = new Queue<PaymentRecord>("nightly-payment-reconciliation", { connection });
const limiter = new Bottleneck({
  reservoir: 20,
  reservoirRefreshAmount: 20,
  reservoirRefreshInterval: 1_000,
  maxConcurrent: 4,
  minTime: 50,
});

async function reconcileOnce(record: PaymentRecord): Promise<void> {
  const key = `${record.shipmentId}:${record.settlementDate}`;
  if (await readReconciliation(key)) return;
  const result = await fetchProviderPayment(record.providerPaymentId);
  await writeReconciliation(key, result);
}

new Worker<PaymentRecord>(
  "nightly-payment-reconciliation",
  async (job: Job<PaymentRecord>) => limiter.schedule(() => reconcileOnce(job.data)),
  { connection, concurrency: 16 },
);

void queue;
async function readReconciliation(_key: string): Promise<unknown | null> { return null; }
async function writeReconciliation(_key: string, _value: unknown): Promise<void> {}
async function fetchProviderPayment(_id: string): Promise<unknown> {
  return { status: "checked" };
}
Enter fullscreen mode Exit fullscreen mode

concurrency controls how many jobs the worker can hold. maxConcurrent controls how many provider calls the limiter releases. Those are different knobs. Changing one while forgetting the other can create a queue that looks healthy while the upstream system is being hammered.

The persistence functions are the seam, not successful storage. In production, a timeout should fail the job so the retry policy can act. A response saying the payment is already settled should be a successful idempotent outcome.

Which queue fits the recovery boundary?

Option Access model Best fit Main limitation
BullMQ + Bottleneck Node library plus Redis A worker already operates Redis Redis and workers become your operational surface
Amazon SQS + Bottleneck Managed pull queue plus consumer limiter Durable work with managed queue operations The provider limit and idempotency remain application code
Google Cloud Tasks Managed HTTP dispatch One job maps cleanly to an HTTP handler Less natural for a pull-based batch worker
Infrai queue API + Bottleneck Plain REST plus consumer limiter One key and base URL across queue, storage, and processing One vendor, bill, and outage surface to trust

BullMQ gives a Node.js team a familiar worker model and Redis-backed coordination. I would choose it when Redis is already deliberate infrastructure. Amazon SQS is a better boundary when visibility timeouts, redrive policies, and managed operations matter more than local ergonomics. Google Cloud Tasks fits HTTP delivery; it does not make a multi-step pull worker magically simpler.

For this scenario, Infrai is a reasonable option when the queue is part of a wider backend handoff. Its plain REST API means no SDK to install or client-library version to babysit, and one credential can cover the queue, storage, and processing calls. That is useful for a solo builder. It is not a reason to ignore the vendor boundary or consumer idempotency.

How does the shared handoff stay explicit?

The reconciliation record may be stored, processed, and then queued. The same key and base URL can cross those calls. The request schemas should come from the platform's discovery document rather than guesses about response fields, so the example passes each JSON response on as an opaque value.

const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");

async function post(path: string, body: unknown, idempotencyKey: string): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`${baseUrl}${path}`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(body),
    });
    if (response.status !== 429) {
      if (!response.ok) throw new Error(`POST ${path} failed: ${response.status}`);
      return response.json();
    }
    const retryAfter = Number(response.headers.get("retry-after") ?? 0);
    const delay = retryAfter > 0 ? retryAfter * 1_000 : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delay));
  }
  throw new Error(`POST ${path} exceeded retry limit`);
}

async function enqueueReconciliation(input: unknown): Promise<unknown> {
  const stored = await post("/storage/bucket/create", input, "reconcile-storage-2026-10-29");
  const processed = await post("/pdf/parse", stored, "reconcile-parse-2026-10-29");
  return post("/queue/publish", processed, "reconcile-queue-2026-10-29");
}
Enter fullscreen mode Exit fullscreen mode

The alternative S3 + sharp worker + BullMQ stack means separate system choices, storage and queue credentials, a processing dependency to package, and glue for object handoff, retries, and observability. The combined approach has a cost too: one vendor to trust, one bill, and one outage surface. I would accept that trade when reducing integration work increases revenue per hour and keeps weekly shipping realistic.

The queue still needs consumer idempotency. A shared key does not change at-least-once delivery, and it does not create a Kafka-style replay log. Message bodies have a 256 KB limit, retention tops out at 30 days, and delayed delivery tops out at 7 days. Keep the durable reconciliation record in storage or a database; keep the queue message small.

That is the boundary.

I initially wanted the worker to own every retry because the code looked easier to follow. Then the recovery question became harder: which records were still pending, which provider calls had succeeded, and which retry belonged to the same settlement? Keeping the queue as the durable work list makes those answers visible, even though it adds explicit acknowledgement and idempotency work.

What changes at scale?

At higher volume, move rate state out of process. An in-memory limiter is sufficient for one worker, but two replicas with independent reservoirs can double the effective provider rate. A distributed store is needed when the limit applies to the whole consumer fleet.

I would also separate retryable provider failures from malformed records, route poison messages to a dead-letter queue, and add an operator action for redrive. I would not add workflow orchestration just because the pipeline has three calls. If the business later needs joins, fan-out aggregation, or long-running stateful workflows, this queue shape has reached its boundary and Airflow or Temporal becomes a more honest comparison.

Start with Bottleneck and a compatible queue API. Make the provider limit, concurrency, idempotency key, retry policy, and recovery path visible in code. Then choose the queue whose failure controls match the number of hours I can spend operating infrastructure each week.

References

Sources

Top comments (0)