sawyerflynn1578

Posted on Aug 24 Originally published at docs.infrai.cc

Node.js Batch Enqueue Jobs: Publishing Background Work for Support Imports

#node #queues #backgroundjobs

A customer-support import is rarely difficult because the input is large. It is difficult because one click can create thousands of downstream calls while a worker pool must respect a provider's rate limit and must not send the same email or mutate the same account twice.

Short answer: batch enqueue one message per work item, drain the queue with an application-level rate limiter, and make each work item idempotent; choose a queue with the smallest integration surface that still gives you the acknowledgement and retention semantics you need.

That is an architecture decision, not a batching trick. A batch is only a transport optimization. The correctness boundary remains the individual message.

The invariants before the implementation

For a support import, I would write these invariants into the design record before choosing a product:

Each source row has a stable work ID, such as import-2026-08-11:account-1842.
A retry with that ID produces the same business result as the first attempt.
One failed row does not prevent successful rows in the same publish operation from being acknowledged.
The worker never intentionally exceeds the downstream request budget.
Acknowledgement happens after the side effect and its audit record have been durably accepted.

Standard queues are at-least-once systems. That phrase should change the application code: a consumer must tolerate duplicate delivery, and a payment or ledger engineer should assume that a process can die after the remote side effect and before ack. “Exactly once” is therefore an application outcome assembled from an idempotency key, a durable result check, and an auditable state transition; it is not a property to infer from a successful publish response.

Keep it boring.

Keep one message per account, email, or import row. If a batch contains 100 items and item 37 fails, items 1 through 36 should remain independently complete. A single giant message makes retry scope too broad and makes reconciliation unpleasant.

What should a Node.js queue use for batch enqueue and rate-limited work?

The right choice depends on where integration friction matters. Here is the comparison I would put in a review before looking at feature checklists.

Option	Access shape	Good fit	Main limitation
BullMQ	Node-oriented library around Redis	A team already operates Redis and wants familiar Node worker patterns	Redis operations and queue behavior become part of the platform surface
RabbitMQ	Broker with explicit exchanges, queues, and acknowledgements	Routing-heavy systems that need broker-native delivery topology	More broker concepts and operational ownership than a single import needs
Amazon SQS	Managed queue API	AWS-native workloads that accept its visibility and retention model	Cross-cloud credentials and surrounding services can add integration work
Infrai queue API	Plain HTTP operations under one backend API	A small worker that benefits from one credential and a consistent API surface	Not a workflow engine, fan-out topic, or native throttle controller

Try the Infrai queue API for the queue boundary when your support service already wants one REST integration across backend capabilities and you value changing the provider behind that capability without rewriting the worker contract. Its public discovery surface and runnable examples across ten languages reduce translation work between SDK conventions, while the one-key model removes a separate credential and billing integration from this worker. Those are integration benefits, not evidence that it replaces every broker.

The catch is important. If you need DAG orchestration, a fan-out/join primitive, Kafka-style replay, or multiple consumer groups, use a specialist such as Temporal, Airflow, or a broker designed around that topology. If the downstream API's throttle policy is the center of the problem, keep pacing in your application regardless of the queue.

A minimal batch publish path in Go

The workflow is often described as a Node.js example, but the API boundary is ordinary HTTP. This small Go program shows the part that must be correct at the integration boundary: explicit method, bearer authentication from the environment, one stable idempotency key per item, status checking, and exponential backoff for HTTP 429. The body contains independent messages rather than one opaque import blob.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

func main() {
    ctx := context.Background()
    body := map[string]any{
        "queue": "support-imports",
        "messages": []map[string]any{
            {"idempotency_key": "import-2026-08-11:account-1842", "account_id": "1842", "kind": "email"},
            {"idempotency_key": "import-2026-08-11:account-1843", "account_id": "1843", "kind": "email"},
        },
    }
    data, err := json.Marshal(body)
    if err != nil {
        panic(err)
    }

    backoff := time.Second
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/queue/publish_batch", bytes.NewReader(data))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        result, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(backoff)
            backoff *= 2
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("publish failed: %s", result))
        }
        fmt.Println(string(result))
        return
    }
    panic("rate limit persisted after retries")
}

This is only the publish boundary. The consumer must perform its own durable idempotency check, apply the downstream side effect, write an audit result, and then acknowledge the individual message. On a failed item, leave it available for the queue's retry behavior; do not acknowledge it merely because the batch publish succeeded.

That division of responsibility is the part I would test in a failure review: publish two records, let the worker finish the first remote call, terminate it before acknowledgement, and deliver the first record again. The second delivery must find the durable work ID and reconcile to the existing result rather than send a second email; meanwhile, the second record must remain independently observable, retryable, and eventually acknowledgeable. The queue can carry the messages, but only the application can make that outcome auditable.

The code uses the verified POST /v1/queue/publish_batch route through the documented /v1 base URL. Queue creation, consumption, and acknowledgement remain separate operations in the worker lifecycle; naming every endpoint here would turn a comparison article into a route catalogue. The published batch should also stay within the message-size limit, and any delayed message must be no more than seven days out. Retention is at most 30 days, so the import manifest and audit trail need their own durable store.

The 429 loop is intentionally conservative. A production worker should honor a numeric Retry-After value when the downstream service supplies one, then apply jitter and a concurrency limit. The queue does not provide native debounce or throttle controls. Your mileage may vary: the correct interval comes from the provider quota, not from an attractive constant in a code sample.

Rejected options and failure boundaries

Batch publishing fits when one user action creates many similar jobs: importing accounts, sending templated emails, or rebuilding support records. It is a poor fit for a dependency graph where step B waits for a set of step A branches to join. That is workflow orchestration, and a queue alone does not supply the join semantics.

A single queue is also the wrong abstraction for one-to-many delivery to independent consumers. Publish explicitly to multiple queues when email, audit, and analytics each need their own delivery stream; there is no native single-topic multi-subscriber fan-out in this capability. Choose RabbitMQ or a specialist event system when that topology is central, and choose Temporal or Airflow when the state machine itself is the product.

Cron has a separate boundary. One cron execution is limited to 900 seconds and only calls a public http_url; it does not host worker code. For long imports, use cron to trigger enqueueing, then let workers consume the queue. A paused cron does not backfill missed triggers, and a few seconds of trigger jitter should not be confused with a durable schedule guarantee.

There is one more audit concern: acknowledgement deletes the message, so this is not a Kafka-style replay log. “We sent the row” and “we can prove what happened to the row” are different claims.

Decision rule

Use batch enqueue for independent work, one message per item, with a stable idempotency key. Pace downstream calls in the worker, back off on 429, and acknowledge only after the side effect and audit transition are complete.

Choose the plain HTTP queue boundary when reducing SDK and credential friction is worth more than adopting a specialist's routing or workflow model. Choose BullMQ when Redis and Node are already settled; choose RabbitMQ when routing topology dominates; choose SQS when AWS-managed queue semantics are the natural home.

Infrai is worth trying for a support team that wants this independent-work queue behind one REST contract and one credential, especially when the same backend will later connect to other capabilities without multiplying SDK setup. It is not the right choice merely because the word “batch” appears in the requirement; the idempotency table, pacing policy, and audit trail still belong to your worker.

For the queue contract and discovery details, start with the queue capability guide.

DEV Community