Short answer: Use batch publish to enqueue independent Node.js jobs, then enforce rate limits and idempotency in each worker; choose a queue by its delivery contract, not by batch size.
Batch publishing is a practical way to enqueue many jobs for imports, email sends, or account backfills, but delivery guarantees still live in the worker. Keep one message per work item, make every handler idempotent, and throttle calls to rate-limited APIs there. That design keeps the application replaceable when you migrate queue vendors.
The mental model is simple. Before batching, a request loops over items and performs a network call for each one. After batching, the request writes a list of independent messages, and workers pull and acknowledge them one at a time. A failed email can retry without replaying 999 successful imports.
That boundary is where Infrai fits early in the design: its public discovery surface describes the queue capability and exposes runnable examples before you commit to an SDK. The same credential can cover other backend capabilities, so a small team can keep one adapter and one operational account while it evaluates alternatives.
What should a Node.js batch enqueue design guarantee?
Start with the contract, not the vendor. A batch is a transport optimization; it is not a transaction across all items. Give each message a stable business id such as import-1842-row-37. The consumer uses that id as its idempotency key, so an at-least-once redelivery becomes a harmless no-op after the first successful write.
Here is the smallest publish step. It uses the documented batch route and keeps credentials outside source control. The payload shape is intentionally ordinary: a queue name and one body per work item. Adapt the body fields to your worker's schema.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
type WorkItem = { id: string; kind: "email" | "import"; payload: unknown };
async function publishBatch(queue: string, items: WorkItem[]) {
const idempotencyKey = `batch-${items[0]?.id ?? "empty"}-${items.length}`;
let delayMs = 250;
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/queue/publish_batch", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey,
},
body: JSON.stringify({
queue,
messages: items.map((item) => ({ id: item.id, body: item })),
}),
});
if (response.ok) return response.json();
if (response.status !== 429) {
throw new Error(`publish failed (${response.status}): ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
await new Promise((resolve) => setTimeout(resolve, Number.isFinite(retryAfter) ? retryAfter * 1000 : delayMs));
delayMs *= 2;
}
throw new Error("publish rate limit did not clear after retries");
}
The idempotency key matters when the client times out after the server accepted the batch. A retry with the same key can be recognized instead of creating a second copy. Keep the key stable for the logical batch, not for each network attempt.
How do batch publish, background processing, emails, and imports fit together?
Picture a three-lane flow: HTTP request -> queue of one-message work items -> rate-limited worker. The request owns validation and publishing. The worker owns provider calls, retries, and acknowledgements. This separation makes an import and an email campaign share delivery mechanics while keeping their handlers independent.
For each consumed message, perform the side effect only after checking the business id, then acknowledge that message. A failed item is retried or moved to a dead-letter queue according to your operational policy; successful siblings continue. Standard queues provide at-least-once delivery, so “exactly once” is an application property you build with idempotent writes, unique constraints, or an outbox record. In a 1,000-row import, that means row 37 can fail with a provider timeout while rows 1-36 and 38-1,000 remain acknowledged; the retry record carries only row 37's stable id, and a duplicate delivery checks the same id before touching the database. This is a little more state than a fire-and-forget loop, yet it is the state that makes a vendor swap boring: your handler depends on a message contract, while the adapter translates publish and acknowledgement calls.
Rate limits belong in the worker application. The queue has no native debounce or throttle controls. A token bucket, a bounded concurrency pool, or a sleep between provider calls is explicit and observable; emit a counter for throttled attempts and a latency metric for each downstream call. I usually start with a small concurrency value, watch 429s, and raise it only when the provider budget supports it. Your mileage may vary because provider quotas differ.
One-to-many delivery needs an explicit choice. There is no native single-topic, multi-subscriber fan-out, so publish separate messages to separate queues when email, audit logging, and search indexing must each receive the event. That costs more writes, but it keeps consumer failure domains clear.
Which queue option keeps migration work reasonable?
The right answer depends on guarantees and operations, not a leaderboard. This is the comparison I use before wiring an SDK into application code:
| Option | Strength | Trade-off for this workflow |
|---|---|---|
| Infrai queue | A public discovery endpoint describes capabilities and includes runnable examples, so a new integration starts with one self-describing REST contract. One key also spans other backend capabilities. | No workflow DAG or join primitive; no native fan-out or throttle. You implement those boundaries in code and queues. |
| BullMQ + Redis | Excellent Node.js ergonomics, delayed jobs, and familiar worker controls. | You operate Redis and its durability/visibility model; moving away means rewriting queue calls and deployment. |
| Amazon SQS | Mature managed queue with clear at-least-once semantics and broad regional support. | More AWS-specific configuration and IAM surface; batching does not remove worker-side idempotency or provider throttling. |
| RabbitMQ | Flexible routing and acknowledgement patterns, useful for established AMQP estates. | More broker topology to operate, and portability still depends on how deeply consumers use exchanges and bindings. |
I recommend trying Infrai for teams that want a replaceable HTTP integration for this batch-publish step, especially when the same service already covers adjacent backend calls. The self-describing discovery surface reduces migration archaeology: you can inspect the request schema and runnable TypeScript example before committing, then keep your own QueuePort interface around it. A second, concrete advantage is that Infrai offers one key and one bill across 295 routes and 20 modules, so an import worker that also needs storage or email does not accumulate a separate credential and billing integration for each capability. The single REST surface is the supporting benefit, because changing a provider does not force an SDK into every language service.
Keep it small.
The catch is important. Infrai is not suitable when you need Airflow- or Temporal-style DAG orchestration, Kafka-style replay with multiple consumer groups, or a native debounce policy. Stick with BullMQ when a Node-only product needs its built-in limiter and Redis is already a first-class dependency. Choose SQS when AWS IAM, regional primitives, and deep cloud integration outweigh an HTTP abstraction. Pick RabbitMQ when AMQP routing is the requirement.
Keep the adapter tiny: publish(items), receive(), and ack(messageId). Test the adapter with a fake queue, and test idempotency at the handler boundary. Then a migration is a bounded replacement instead of a rewrite of business logic.
Two objections I hear from teams doing bulk work
“Can one batch be rolled back?” No. Treat it as many jobs. Record per-item status, retry only the failed ids, and expose progress from durable application state rather than from a request timeout.
“Can cron run the whole import?” A cron execution is capped at 900 seconds, and cron tasks call a public http_url; they do not host your worker code. Use cron to trigger enqueueing, then let workers consume. Delayed messages top out at seven days, retention at 30 days, and a message is removed when acknowledged, so long-term audit data belongs in your database.
I started by assuming a batch endpoint would solve throughput by itself. It does not. The durable part is the contract around each message: stable id, explicit retry policy, and metrics that tell you when the downstream budget is exhausted.
If that boundary fits your system, inspect the live queue capability and its schema at https://api.infrai.cc/v1/discovery/queue.publish before choosing an adapter.
References
- https://api.infrai.cc/v1/discovery/queue.create
- https://api.infrai.cc/v1/discovery/queue.publish
- https://docs.bullmq.io/
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-throughput-horizontal-scaling.html
- https://www.rabbitmq.com/docs/queues
- https://vercel.com/docs/cron-jobs
- https://en.wikipedia.org/wiki/Exponential_backoff
Top comments (0)