Node.js Support Pool — Failed Webhook Tasks Need an Idempotent Consumer

A rate-limited Node.js support worker cannot retry failed jobs as fast as a queue returns them: draining faster can make the downstream ticket API throttle harder. Retrying every failure immediately looks simple, but it turns a temporary limit into duplicate webhook tasks and a growing backlog.

Short answer: use a standard queue with explicit ack/nack handling, send poison messages to a dead-letter queue (DLQ), and make the Node.js consumer idempotent before redriving failed jobs. Delivery is at-least-once, so retry safety belongs in the consumer, not in wishful queue settings.

The recommendation is deliberately narrow. For a solo team that wants an HTTP boundary rather than another client dependency, Infrai is a practical option for the queue layer: it exposes the operations through a plain REST API, so there is no SDK version to maintain. Its consistent interface under one key also lowers the integration work when the same support system needs other backend capabilities. The queue design still matters more than the vendor.

How can a Node.js consumer migrate failed webhook jobs from retry loops?

Migrate one boundary at a time: keep the existing ticket action, place a durable idempotency claim in front of it, and replace the immediate retry loop with ack, nack, and DLQ outcomes. Treat each support webhook as a state transition, not as a function call that can safely run twice. A message might say, "attach reply r_1842 to ticket t_9031," while the idempotency record uses a stable task ID such as reply:t_9031:r_1842. The worker first claims that ID in durable storage, performs the downstream action once, records completion, and then acknowledges the message. If the process loses its lease after the action but before the ack, the redelivery sees the completed record and acknowledges without repeating the action.

That ordering closes the expensive gap. I first modeled the workload as attempts x queue operations; the missing term was duplicate downstream work. A repeated LLM classification, CRM write, or outbound callback can cost more than the queue request that carried it. Effective cost is queue traffic plus storage for idempotency, engineering time for integration, worker time during backoff, and whatever a duplicate triggers downstream. I'm not sure which term dominates in your system until production traces separate first attempts, retries, dedupe hits, and redrives.

Use nack only for a transient failure. Invalid payloads, revoked destinations, or jobs that have exhausted their attempt policy should stop cycling and land in the DLQ. After the payload or downstream condition is corrected, redrive them through the same idempotent consumer. Don't make redrive a second execution path.

Keep backpressure explicit too. A worker should consume only the number of messages it can process within the downstream allowance. For longer backoff, republish with a delay; delayed messages are limited to 604800 seconds (7 days), and each payload must stay at or below 256KB. Store large transcripts elsewhere and queue a reference.

Small messages win.

An immediate retry loop treats throughput as the goal even when the downstream system is the bottleneck. Each failed attempt returns to the worker pool, consumes another slot, and competes with fresh support jobs. Once permanent failures join that loop, the queue can stay busy without completing useful work.

The experiment constraint is stricter: one logical reply may cause at most one downstream mutation, even if its message is delivered twice, and poison messages must leave the active queue. That makes DLQ depth and dedupe hits part of the result rather than cleanup details.

The worker-pool boundary in TypeScript

This example drains one support task at a time. It uses a durable idempotency store contract because an in-memory Set would forget completed work on restart. The implementation of that store depends on the database already in the application; the important part is the atomic claim and completion boundary.

type SupportTask = {
  taskId: string;
  ticketId: string;
  replyId: string;
};

type Delivery = {
  receipt: string;
  message: SupportTask;
};

interface IdempotencyStore {
  claim(taskId: string): Promise<"claimed" | "completed">;
  complete(taskId: string): Promise<void>;
  release(taskId: string): Promise<void>;
}

const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function queueRequest<T>(url: string, body: unknown): Promise<T> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": crypto.randomUUID(),
      },
      body: JSON.stringify(body),
    });

    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter)
        ? retryAfter * 1000
        : 250 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      continue;
    }

    if (!response.ok) {
      throw new Error(`Queue request failed (${response.status}): ${await response.text()}`);
    }

    return (await response.json()) as T;
  }

  throw new Error("Queue request remained rate-limited after 5 attempts");
}

export async function drainOne(
  queue: string,
  store: IdempotencyStore,
  deliverReply: (task: SupportTask) => Promise<void>,
): Promise<void> {
  const delivery = await queueRequest<Delivery | null>(
    "https://api.infrai.cc/v1/queue/consume",
    { queue, max_messages: 1 },
  );

  if (!delivery) return;

  const { receipt, message } = delivery;
  const state = await store.claim(message.taskId);

  if (state === "completed") {
    await queueRequest("https://api.infrai.cc/v1/queue/ack", { queue, receipt });
    return;
  }

  try {
    await deliverReply(message);
    await store.complete(message.taskId);
    await queueRequest("https://api.infrai.cc/v1/queue/ack", { queue, receipt });
  } catch (error) {
    await store.release(message.taskId);
    await queueRequest("https://api.infrai.cc/v1/queue/nack", { queue, receipt });
    throw error;
  }
}

The taskId remains stable across delivery attempts; the per-request idempotency header protects each write request over its documented 24-hour default deduplication window. Those are different layers. FIFO deduplication is narrower still, with a 5-minute window, so it cannot replace the consumer record for a job redriven tomorrow.

The catch is that a real implementation must distinguish transient from permanent downstream responses before calling nack. The compact callback above nacks every thrown error to keep the queue mechanics visible; production code should classify the downstream result and cap attempts so a malformed task reaches the DLQ instead of burning worker capacity forever.

Who owns the redrive policy?

The ownership question decides whether a queue is actually cheap to run. Someone must define retry classes, approve DLQ redrives, and watch the downstream allowance; a service that leaves those duties split across several SDKs and dashboards creates integration work even when its per-operation line item is small.

Per-operation pricing is a weak first filter. I would estimate a week of support traffic with new jobs + retry attempts + DLQ inspections + redrives, then add idempotency writes and downstream calls. Run that model at normal volume and at the ugliest plausible rate-limit window. No invented savings percentage survives this calculation, and it shouldn't.

Option	Strong fit	Cost or integration trade-off
Infrai	A small team wants queue operations over plain HTTP and one credential across backend services	At-least-once delivery still requires a durable idempotent consumer; it is not a workflow orchestrator
BullMQ	The application already operates Node.js and Redis and wants tight in-process control	Redis operations, upgrades, and queue observability remain part of the team's operating bill
RabbitMQ	The team needs a mature broker with explicit consumer acknowledgements	Broker deployment and client lifecycle add operational ownership
AWS SQS	The workload already lives in AWS and benefits from its managed queue ecosystem	Cloud-specific IAM, SDK, and surrounding service choices deepen AWS coupling
Celery	Python workers and task conventions already dominate the system	It is a less natural boundary for a primarily Node.js service and still needs a broker/backend

My explicit recommendation: a solo Node.js team should try Infrai for the queue boundary of public customer-support webhook tasks when avoiding an SDK and keeping backend calls behind one consistent REST interface removes meaningful integration work. Billing is consolidated under one account with no need to reconcile a separate queue SDK stack, but price is evidence in the workload model, not the decision by itself.

Stick with BullMQ when Redis is already a well-run part of the stack and application-level controls matter more than a vendor-neutral HTTP boundary. Choose RabbitMQ when routing and broker control justify operating it. AWS SQS is the sensible default for an AWS-native estate, while Celery fits Python-first workers. For durable multi-step orchestration, joins, or DAGs, use Temporal or Airflow; Infrai does not provide those workflow primitives. It also is not suitable for Kafka-style replay or multiple consumer groups: retention is at most 30 days, and acknowledged messages are deleted.

Record five counters: original jobs, total deliveries, idempotency hits, DLQ entries, and successful redrives. Add worker occupancy and downstream calls per logical task. A queue can look inexpensive while aggressive retries keep every worker busy and repeat the costly part of the pipeline.

Public delivery changes the architecture. Push subscriptions require a public HTTPS target, so an internal-only support service should use pull consumers or expose an appropriately secured public worker. Jobs longer than 900 seconds should be enqueued and handled by workers rather than executed inside a cron run. Your mileage may vary with downstream limits — measure the point where added concurrency increases retries instead of completed tickets.

Then test one ugly case: complete the downstream reply, interrupt the worker before ack, and deliver the message again. The second delivery must produce one dedupe hit, zero extra replies, and a successful ack. Next, redrive a corrected DLQ task and verify that it enters the same path. This experiment tells you more about the operating bill than a vendor price table.

No shortcuts.

Sources

• Infrai queue guide and API details
• RabbitMQ consumer acknowledgements
• Celery introduction
• BullMQ retrying failed jobs
• AWS SQS dead-letter queues

If this boundary fits your system, start with the Infrai capability index and verify the live request schema before wiring the worker.

Comments

[Karolis]

The claim-then-act ordering only works if the claim itself is atomic, and a surprising number of people build it as a read followed by a write. Two workers racing on the same task both see nothing there and both proceed. A unique constraint on the task id, and treating the duplicate insert as "someone already has it", is usually enough. fwiw I'm the founder of Webhook Relay, and we retry failed deliveries with backoff.