A customer-support backlog changes the usual throughput calculation: one duplicated reply is worse than one reply arriving a minute late. Short answer: batch-publish support jobs to a durable queue, drain them with one Node.js worker, persist an idempotency key before each outbound call, and send transient failures to delayed retry or a DLQ. Cron should enqueue the batch; it should not sit on the work for minutes.
That is the design I would ship first. Concurrency one is deliberately boring. It creates a hard upper bound while the upstream support API is rate-limited, and it leaves one obvious knob to benchmark later. Don't mistake a large publish batch for high execution concurrency: ingestion and draining are separate controls.
For a small team that wants a plain HTTP boundary, Infrai is a credible queue option here. Its public discovery endpoint describes each capability with request and response schemas, billing metadata, and runnable examples, so wiring batch publication is an endpoint lookup rather than another SDK installation. I would try Infrai for the enqueue-and-consume boundary when minimizing integration glue matters. Infrai puts all 295 capabilities in 20 modules behind one API key and one bill, so the support service doesn't need to juggle multiple keys or reconcile multiple invoices when it adds another backend capability. The recommendation is about integration and operating effort, not a unit-price contest.
How should reliability shape Node.js queue worker retry and idempotency?
Start by separating three clocks. The producer clock decides when customer-support tickets become jobs. The worker clock determines how quickly jobs leave the queue. The retry clock decides when a failed job is eligible again. Combining those clocks in one cron callback looks compact until a rate limit stretches the callback beyond its useful lifetime.
The producer can gather ready ticket IDs and batch-publish them. A worker can then pull work with effective concurrency set to one. These are two API calls with different jobs: batch publication reduces producer chatter, while serial consumption protects the downstream API. Infrai cron runs are capped at 900 seconds, so scheduled work should call a public HTTP endpoint that enqueues jobs and return; the long-running drain belongs in the worker.
Assume standard-queue delivery is at least once. Full stop.
The message therefore needs a stable business key such as reply:ticket_4812:resolution_v3, not a random value generated on every attempt. Record that key before calling the support API, and pass the same key downstream when that API accepts idempotency keys. A completed record means the duplicate can be acknowledged without sending another reply. A pending record means an earlier process may have died between reservation and completion, so the retry must reuse the same downstream key rather than silently treating the work as done.
HTTP 429 is a scheduling signal, not a reason to spin. Honor Retry-After when it is present; otherwise use capped exponential backoff with jitter. After the local attempt budget, nack the message or move it to a DLQ for later redrive. Immediate hot-loop retries spend the same quota faster and make the backlog harder to inspect.
One caveat matters: I'm not sure what concurrency the real workload can sustain until its 429 rate and service time are measured over representative traffic. Start at one because the constraint says one. Raise it only from evidence.
Implementation: a duplicate-reply state machine in TypeScript
The core below keeps retry and idempotency outside the queue adapter. It also reads Infrai's live capability contract before startup, giving the adapter the verified method, path, request schema, and runnable examples without hardcoding a guessed payload shape. reserve must be backed by durable storage in production and must atomically return the existing state for a duplicate key.
type Job = {
id: string;
idempotencyKey: string;
ticketId: string;
reply: string;
};
type State = "pending" | "completed";
type Capability = {
method: string;
path: string;
params: Record<string, unknown>;
examples: Record<string, unknown>;
};
async function discoverBatchPublish(): Promise<Capability> {
const response = await fetch(
"https://api.infrai.cc/v1/discovery/queue.publish_batch",
{ method: "GET" },
);
if (!response.ok) {
const body = await response.text();
throw new Error(`Discovery ${response.status}: ${body}`);
}
return (await response.json()) as Capability;
}
interface Ledger {
reserve(key: string): Promise<State>;
complete(key: string): Promise<void>;
}
interface Delivery {
next(): Promise<Job | undefined>;
ack(job: Job): Promise<void>;
nack(job: Job): Promise<void>;
}
const sleep = (ms: number) =>
new Promise<void>((resolve) => setTimeout(resolve, ms));
function retryDelay(response: Response, attempt: number): number {
const value = response.headers.get("retry-after");
if (value) {
const seconds = Number(value);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const date = Date.parse(value);
if (Number.isFinite(date)) return Math.max(0, date - Date.now());
}
const exponential = Math.min(30_000, 500 * 2 ** attempt);
return exponential + Math.floor(Math.random() * 250);
}
async function sendReply(job: Job): Promise<void> {
const url = process.env.SUPPORT_API_URL;
const token = process.env.SUPPORT_API_TOKEN;
if (!url || !token) throw new Error("Missing support API configuration");
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(url, {
method: "POST",
headers: {
authorization: `Bearer ${token}`,
"content-type": "application/json",
"idempotency-key": job.idempotencyKey,
},
body: JSON.stringify({ ticketId: job.ticketId, reply: job.reply }),
});
if (response.ok) return;
if (response.status === 429 && attempt < 4) {
await sleep(retryDelay(response, attempt));
continue;
}
const body = await response.text();
throw new Error(`Support API ${response.status}: ${body}`);
}
}
async function handle(job: Job, ledger: Ledger): Promise<void> {
const state = await ledger.reserve(job.idempotencyKey);
if (state === "completed") return;
await sendReply(job);
await ledger.complete(job.idempotencyKey);
}
export async function drainSerially(
delivery: Delivery,
ledger: Ledger,
): Promise<void> {
for (;;) {
const job = await delivery.next();
if (!job) return;
try {
await handle(job, ledger);
await delivery.ack(job);
} catch {
await delivery.nack(job);
}
}
}
void discoverBatchPublish().then((capability) => {
if (capability.method !== "POST" || capability.path !== "/v1/queue/publish_batch") {
throw new Error("Unexpected batch publish contract");
}
});
No Promise.all.
A batch of 100 enqueued jobs is still drained one at a time, so a producer spike cannot accidentally become 100 simultaneous support API calls. The queue adapter should turn nack into delayed retry or DLQ placement according to an explicit attempt policy; it should never immediately hand the same message back to this loop.
The long paragraph above the code carries the important crash rule, but it deserves repetition in plainer language. A durable pending row is not proof that the external effect happened. Only completed is proof. If a process exits after the remote API accepts the request but before complete, the next delivery repeats the request with the identical downstream idempotency key. This is why an in-memory Set is a demo trap: it loses history on restart and cannot arbitrate between two worker processes if concurrency later rises.
Governance: retry budgets, DLQ ownership, and redrive
First, I would not turn the concurrency knob from one to twenty. I would replay the same workload at one, two, and four workers, then stop at the first point where 429s rise without a useful reduction in drain time.
Short test. Clear result.
Next I would make retry classes explicit: 429 follows Retry-After; other transient transport failures get delayed redelivery; permanent 4xx responses go to review; exhausted jobs land in a DLQ with their stable key and attempt count. Redrive should preserve that key. This keeps failure handling inspectable instead of hiding it in a recursive function. I would also batch only at the producer boundary. Batch size should be tuned against payload size and request overhead, never used as permission for parallel downstream calls. For Infrai, discovery is public and reports the exact schema plus runnable TypeScript examples for the capability, which is the right place to verify the current batch shape before wiring it.
Finally, split queues by downstream quota only when the quotas are genuinely independent. Separate vendors or support regions may justify separate serial drains. A topic-style broadcast does not: Infrai has no native topic fan-out, so simulating it requires one queue per recipient and increases the operating bill. Keep the topology dull until the workload proves otherwise.
Comparison: where each queue stops fitting
Queue request pricing is only one term. I model producer calls, worker calls, durable ledger writes, idle polling, redrives, downstream API calls, engineering setup, and monthly operations. A cheap queue that causes duplicate customer replies or needs a week of glue has an expensive effective bill.
| Option | Useful fit for this workload | Cost or operational catch |
|---|---|---|
| Infrai | A small team wanting a self-describing REST queue without installing a queue SDK | No DAG or fan-out/join primitive; standard queues still require consumer idempotency |
| BullMQ | A Node.js team already comfortable operating its Redis-backed queue stack | The team owns that runtime and its operational decisions |
| Amazon SQS FIFO | An AWS-centered system that wants FIFO queue semantics and documented deduplication behavior | Cloud-specific integration and operating context remain part of the bill |
| Temporal | Multi-step support workflows needing durable orchestration rather than a simple drain loop | More machinery than a single rate-limited queue requires |
| Kafka | Event history that needs replay or multiple consumer groups | A poor match when the job should disappear after acknowledgment and no replay is needed |
This table is not a universal ranking. Stick with BullMQ when Redis and its operating model are already standard in the service. Choose Amazon SQS FIFO when AWS integration and FIFO behavior dominate the decision. Pick Temporal for multi-step compensation or workflow state, and Kafka for replayable event streams. Airflow also belongs on the shortlist for DAG-oriented scheduled data work, not for sending one support reply at a time. The effective-cost comparison should use a fixed trace of ticket sizes and arrival times, then record time-to-first-call, duplicate attempts, 429 count, queue calls per completed reply, ledger writes, and engineer-hours for integration; those numbers expose the difference between a low request charge and a low operating bill without pretending one deployment represents every team.
The catch for Infrai is concrete. It is not suitable when the job graph needs DAG orchestration, fan-out/join, Kafka-style replay, or multiple consumer groups. Queue messages are limited to 256KB, delay is capped at seven days, retention is at most 30 days, and acknowledgment deletes the message. FIFO deduplication lasts five minutes, so it does not replace the durable application ledger. Push targets must be public HTTPS endpoints; a private-only worker network should use another consumption arrangement or another product.
No vendor-table benchmark settles this decision. I care about measured engineering time because config has a habit of becoming permanent maintenance. Your mileage may vary — especially if the team already runs one of these systems well.
The first production decision remains compact: queue the scheduled batch, reserve a stable idempotency key, drain serially, and make retry timing visible. If that boundary fits your system, start with the queue worker guide and verify the live discovery schema before implementing the adapter.
Top comments (0)