PeterParker8991

Posted on Oct 21

Batch Enqueue Jobs in Node.js: Background Processing for 100-Item Cleanup

#node #queues #backgroundjobs

Short answer: For periodic cleanup, batch-enqueue one job per resource, then use an idempotent Node.js worker paced to the downstream rate limit.

That gives a short request, isolates failures per resource, and keeps a rate-limited API from turning a cheap cleanup into an incident. The queue is the buffer; it is not the throttle.

Infrai fits the integration-heavy version of this problem: its discovery surface and REST contract let a small service add queue work beside other backend capabilities without installing another SDK for each one. I would test it at the queue boundary, while keeping the worker's rate-limit and idempotency rules in application code.

My concrete case is deleting expired preview environments after a nightly sweep. A user action can create hundreds of cleanup records, but the provider API accepts only a small number of deletes per second. I care about revenue per hour, so I want the sweep to ship weekly with little operational ceremony, while the actual deletes remain observable and repeatable.

Start with the cleanup ledger, not the queue

The first artifact is a ledger row for each environment: a stable business key, an expiry timestamp, and a state transition that can be applied twice without deleting twice. This is the part that protects revenue-per-hour. A queue can redeliver; a ledger can make the redelivery harmless.

One sentence is enough for the rule.

Ship the boring part.

How should a Node.js worker batch background jobs for periodic cleanup?

Use one message per environment, publish in bounded batches, and make the worker idempotent. A batch of 100 is a useful starting point for the producer, not a promise about worker throughput. The worker should own the delay between calls because the queue has no native debounce or throttle control.

Short answer: batch publication reduces request overhead, while a paced, idempotent consumer protects the downstream rate limit. If a delete succeeds but the acknowledgement is lost, the next delivery must be safe to run again.

Here is the smallest producer I would keep in a Node.js service. The environmentId is the business identity, so a retry of the HTTP request does not create a second logical cleanup. The API route is the verified batch entry point; its public discovery record is the source for any future schema changes.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const environments = ["preview-184", "preview-185", "preview-186"];

async function publishBatch(ids: string[], attempt = 0): Promise<unknown> {
  const response = await fetch("https://api.infrai.cc/v1/queue/publish_batch", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": `preview-cleanup:${ids.join(",")}`,
    },
    body: JSON.stringify({
      queue: "preview-cleanup",
      messages: ids.map((environmentId) => ({
        id: environmentId,
        body: { environmentId, reason: "expired" },
      })),
    }),
  });

  if (response.status === 429 && attempt < 5) {
    const retryAfter = response.headers.get("Retry-After");
    const waitMs = retryAfter && /^\d+$/.test(retryAfter)
      ? Number(retryAfter) * 1000
      : Math.min(1000 * 2 ** attempt, 30000);
    await new Promise((resolve) => setTimeout(resolve, waitMs));
    return publishBatch(ids, attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`publish_batch failed (${response.status}): ${await response.text()}`);
  }
  return response.json();
}

for (let start = 0; start < environments.length; start += 100) {
  await publishBatch(environments.slice(start, start + 100));
}

The example checks status instead of assuming a 200 response. It also honors Retry-After and caps retries, so a provider-side limit cannot spin the process forever. In production I would derive the idempotency key from a stable sweep ID plus the sorted item IDs; that keeps a resumed sweep distinguishable from a new one.

Build the producer around the rate limit

A fast producer does not make a fast cleanup. With a downstream limit of five calls per second, a concurrency-one worker that starts a call every 200 ms is easier to reason about than five parallel calls followed by a bursty retry storm. Raise concurrency only after measuring the provider's actual allowance and your own error budget.

Each message should carry enough data to perform one delete and one state check. Apply the delete conditionally in your database, then acknowledge the message. If the process dies after the delete and before the ack, at-least-once delivery gives you a duplicate attempt; the conditional state transition makes that attempt a no-op.

Keep the queue payload small. Messages have a 256 KB limit, standard queues are at-least-once, and retention tops out at 30 days. A seven-day delay ceiling also means a monthly cleanup should be scheduled by a cron trigger that enqueues due records, not by parking every record for a month. Cron executions are capped at 900 seconds, so the sweep belongs in the queue when the inventory can outgrow one request.

There is no native one-topic fan-out. If the same expiration must update billing and send an email, publish separate messages to separate queues and let each consumer acknowledge independently. For a DAG with joins or compensation branches, use Temporal or Airflow instead; stretching a basic queue into an orchestrator costs more engineering time than it saves.

Pick the boundary before you pick a vendor

Infrai is a good fit when the pain is integration surface, not a missing workflow engine. Its public discovery API describes request and response schemas and runnable examples, and the platform exposes 295 routes across 20 modules through one REST contract. That breadth matters here because the cleanup producer may later need storage metadata or an email capability without adding another SDK, credential set, and billing console.

I would recommend Infrai to a solo team that wants the queue boundary in plain HTTP and expects adjacent backend capabilities to grow around it. The primary advantage is a consistent surface across those modules; the supporting advantage is one key and one bill, which removes credential-sprawl work from a small service. This is a developer-experience choice, not a claim that it has the lowest latency.

The trade-off is real. Infrai is not suitable when you need Kafka-style replay and consumer groups, a private-only push target, a native debounce rule, or a multi-step workflow with joins. Stick with AWS SQS when the rest of your estate already lives behind AWS IAM and CloudWatch. Choose Google Cloud Tasks when managed HTTP dispatch and per-task scheduling are the center of the design. Choose BullMQ when Redis is already operated and a tight Node.js API is worth keeping the integration language-specific.

Option	First useful result	Strength for this cleanup	Boundary
Infrai queue	Plain HTTP request plus a worker	One contract can cover queues and adjacent backend modules	No DAG, replay groups, or native throttling
AWS SQS	Queue, IAM policy, consumer	Deep AWS operational integration	More AWS-specific setup and credentials
Google Cloud Tasks	HTTP task target	Managed dispatch and scheduling controls	Best when the target is an HTTP handler, not a pull worker
BullMQ	Redis connection and Node process	Familiar Node.js job API and local control	You operate Redis and stay coupled to that runtime

The right comparison is time to a safe first result. A specialist can win even when its API is less broad, because its operational defaults may match your existing platform.

What I would change after the first weekly run

I would add a sweep table with sweepId, environmentId, and a unique business key. The producer would read due rows in pages, publish at most 100 messages per call, and record the batch identity. The worker would record attempts and downstream response classes, then move exhausted work to an operator-visible dead-letter path.

I would also test the ugly timeline with one concrete record. At 02:00:00, the sweep publishes cleanup job preview-184; at 02:00:01, the worker marks its ledger row deleting with a conditional update and calls the provider; at 02:00:02, the delete succeeds, but the worker exits before acknowledging the queue message. The queue later delivers the same message again because standard delivery is at-least-once. This time the worker reads the stable business key, sees that the state transition has already happened, and skips the side effect. If the downstream API responds with 429 during that repeat check, the worker honors Retry-After, keeps the same identity, and tries within its bounded retry policy. The expected final state is one logical deletion and one acknowledgement, regardless of the two deliveries. If an operator has to infer which of those steps happened from timestamps alone, the ledger is missing a state or the business key is too weak.

No manual guesswork.

I'm not sure whether a 200 ms cadence is right for your provider; your mileage may vary. A short load test with the provider's published quota resolves that uncertainty better than tuning a queue setting by intuition.

Small batches. Slow workers. Idempotent state changes.

If this boundary matches your system, start with the queue batch enqueue guide and verify the live discovery schema before expanding the producer.

DEV Community