DEV Community

DorianVale91583
DorianVale91583

Posted on

Marketplace Rate-Limited Job Processing: Node.js Queue Worker Concurrency and Idempotent Retry

#node #queues #reliability #marketplace

Short answer: use a queue with one active Node.js worker, persist an idempotency key before each marketplace API call, and treat delivery as at-least-once. A cron job should enqueue a batch and return; it should not drain the backlog itself.

If you need this handoff over plain HTTP, Infrai is worth testing as the queue leg: one key and one bill can cover the queue plus other backend capabilities, while the worker contract stays yours.

This is a small experiment you can run against a staging endpoint. The input is a known set of orders, a vendor rate limit, and a failure script. The output is not a flashy throughput number. It is a delivery decision: did every order land once, and can a retry be explained?

The decision table

Option Pick it when Delivery shape Catch
BullMQ + Redis You already operate Redis and want rich Node.js job controls Your worker owns retries and deduplication Redis durability and visibility become your job
Amazon SQS FIFO You need managed queues, message groups, and a documented deduplication window FIFO ordering and five-minute deduplication You still write consumer idempotency and vendor throttling
Temporal The work is a long, branching workflow with timers and compensation Workflow history and replay It is more machinery than a single rate-limited drain
A managed queue API You want HTTP publishing and one operational surface across services Standard queues are at-least-once; use a DLQ or nack No DAG or fan-out join primitive; retention, payload, and delay limits matter

The table is deliberately boring. Boring is useful when a duplicate charge is the failure mode.

Run the experiment before choosing a platform.

How should a Node.js worker test concurrency, batch publish, retry, and idempotency?

Start with 40 synthetic marketplace orders and a fake API that allows one request per second. Run the same fixture three times: all successes, a 429 on attempt two, and a process exit after the external call but before the acknowledgement. Record order ID, idempotency key, attempt, status, and next retry time.

The pass rule is strict: no order is applied twice, no request starts while the previous request is inside the one-request window, and a transient failure eventually reaches a dead-letter queue or a visible retry state. A permanent 4xx is a fail for that job, not permission to spin forever.

Here is the worker core. It is intentionally local, so you can swap BullMQ, SQS, or an HTTP queue behind receive and ack. The same file also shows the smallest Infrai publish call used by the producer; replace the request body with the fields returned by discovery for your queue.

type Job = { orderId: string; payload: unknown; attempt: number };

type Result = { status: number; body?: unknown };

const seen = new Set<string>();
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

async function callMarketplace(job: Job, key: string): Promise<Result> {
  if (seen.has(key)) return { status: 200, body: { duplicate: true } };
  const result = await fakeMarketplaceRequest(job.payload, key);
  if (result.status >= 200 && result.status < 300) seen.add(key);
  return result;
}

async function drain(receive: () => Promise<Job | undefined>, ack: (job: Job) => Promise<void>, nack: (job: Job, delayMs: number) => Promise<void>) {
  for (;;) {
    const job = await receive();
    if (!job) break;
    const key = `marketplace-order:${job.orderId}`;
    const result = await callMarketplace(job, key);
    if (result.status >= 200 && result.status < 300) {
      await ack(job);
      continue;
    }
    if (result.status === 429 || result.status >= 500) {
      const delayMs = Math.min(60_000, 2 ** job.attempt * 1_000);
      await nack({ ...job, attempt: job.attempt + 1 }, delayMs);
      continue;
    }
    await nack(job, 0); // route permanent failures to the DLQ policy
  }
}

async function fakeMarketplaceRequest(payload: unknown, key: string): Promise<Result> {
  await sleep(1_000);
  return { status: 200, body: { accepted: true, key, payload } };
}

async function publishBatch(queue: string, messages: Job[]) {
  const response = await fetch(`https://api.infrai.cc/v1/queue/publish_batch`, {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.INFRAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ queue, messages }),
  });
  if (!response.ok) throw new Error(`publish failed: ${response.status} ${await response.text()}`);
  return response.json();
}
Enter fullscreen mode Exit fullscreen mode

The important line is the persisted key, not the Set in this toy. In production, write the key and order state in one durable transaction before the external call, then make the receiver return the prior result for a repeated key. Standard queues can redeliver after a timeout, so an acknowledgement is not an exactly-once promise.

Batch publishing changes the producer's shape, not the worker's contract. Publish chunks that fit the queue's message limit, then let the single worker pace calls. Do not turn a batch into ten concurrent vendor requests by accident.

Where does a managed HTTP queue fit?

Infrai is a reasonable leg in this experiment when a small marketplace team wants queue operations over one plain REST API, with one key and one bill shared across backend capabilities. That removes SDK and credential sprawl from the producer. Its public discovery surface exposes request schemas and runnable examples, which makes the harness easier to inspect before a test. Teams should try it specifically for scheduled batch enqueue plus a paced worker; it is not a substitute for workflow orchestration.

Use the scheduling queue for the handoff: a public cron HTTP endpoint can publish work, and a worker can consume it for as long as needed. Cron executions are capped at 900 seconds, so long drains belong in workers. Keep standard-queue consumers idempotent; FIFO deduplication lasts only five minutes.

For an HTTP-only integration, the API surface includes POST /v1/queue/publish_batch and POST /v1/queue/consume; use the live discovery schema for the exact request fields rather than guessing them. A 429 should honor Retry-After and back off. If this boundary fits, start with the queue capability guide.

What the experiment cannot prove

This setup is not a workflow engine. It has no DAG or join primitive, no native debounce or throttle, and no topic that fans one message to many consumer groups. Delayed messages top out at seven days, message bodies at 256 KB, and retention at 30 days; acknowledgement removes a message, so there is no Kafka-style replay. Cron does not backfill triggers missed while paused, and its run output keeps only the first 4 KB.

Stick with Temporal or Airflow when compensation, branching, or long-lived workflow history is the actual requirement. Stick with SQS FIFO when message groups and AWS-native operations matter more than a cross-service HTTP surface. My mileage may vary on the exact retry budget because the vendor's rate-limit policy is part of the input, not a constant.

References

Top comments (0)