In-Queue Retry vs DLQ Redrive — 3 Node.js Rules for Rate-Limited Renewal APIs

#node #queues #observability

Short answer: for a customer-support renewal reminder that must wait for a business deadline, retry a transient 429 in the worker, then move a repeatedly failing job to a dead-letter queue (DLQ) and redrive it after the upstream API recovers. Watch both the main backlog and DLQ volume; make the reminder operation idempotent before any retry can happen.

The before picture is one worker hammering a rate-limited API until the queue stops moving. The after picture is a paced worker, a separate failure lane, and an operator who can see whether the deadline is approaching. That separation is the decision.

What should a Node.js queue do with failed, rate-limited API calls?

Keep the first few failures close to the job. A 429 normally means “try later,” so the worker should honor Retry-After when it exists and use capped exponential backoff when it does not. A network timeout can take the same path. A malformed customer record or a permanent authorization failure should not consume the same retry budget forever.

After the retry limit, nack the message into the DLQ. Poison messages leave the normal lane, which protects other renewal reminders from one broken account. Redrive is a deliberate operation: wait for the external dependency to recover, inspect the DLQ, then move a bounded batch back to the main queue. Endless in-place retries hide the queue's real health.

There is a small but important detail here. Standard queues are at-least-once, so a successful API call followed by a lost acknowledgement can deliver the same job again. The reminder endpoint therefore needs a stable idempotency key, such as renewal:${accountId}:${deadline}. Store the side-effect record outside the queue before acknowledging. Queue retention is limited, and ack deletes the message; it is not an audit database.

Short version: retry transient pressure, isolate poison work, and make replay boring.

A copyable Node.js worker for retry and idempotency

The example below uses the public queue statistics and DLQ routes to make the control loop visible. It treats the queue as a transport, not as the source of truth for the reminder ledger. Set INFRAI_API_KEY and QUEUE_NAME; the worker can run beside any HTTP client for the renewal provider.

const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
const queue = process.env.QUEUE_NAME;

if (!baseUrl || !apiKey || !queue) {
  throw new Error("INFRAI_BASE_URL, INFRAI_API_KEY, and QUEUE_NAME are required");
}

async function request(path: string, method: "GET" | "POST", body?: unknown) {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`${baseUrl}${path}`, {
      method,
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: body === undefined ? undefined : JSON.stringify(body),
    });

    if (response.status !== 429) {
      if (!response.ok) {
        throw new Error(`${method} ${path} failed: ${response.status} ${await response.text()}`);
      }
      return response.json();
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const delaySeconds = Number.isFinite(retryAfter) && retryAfter > 0
      ? retryAfter
      : Math.min(2 ** attempt, 60);
    await new Promise((resolve) => setTimeout(resolve, delaySeconds * 1000));
  }
  throw new Error(`rate limit persisted for ${method} ${path}`);
}

async function inspect(queueName: string) {
  const statsPath = "/v1/queue/stats/{queue}".replace("{queue}", encodeURIComponent(queueName));
  const stats = await request(statsPath, "GET");
  return { stats };
}

const snapshot = await inspect(queue);
console.log(JSON.stringify(snapshot));

The route names matter. They come from discovery, so this code does not invent a REST-shaped /queue/jobs endpoint. In a production worker, call redrive only after an operator or a scheduled policy has checked the upstream health. Keep the batch small enough that a recovered API is not immediately rate-limited again.

How do retry, DLQ redrive, monitoring, and backlog processing fit together?

Think of four signals flowing left to right:

publish reminder -> consume -> call API -> ack

Transient 429s loop through bounded backoff. Permanent or repeatedly failing jobs branch down to DLQ -> inspect -> redrive. Metrics sit above both lanes: main backlog age, oldest message age, attempts per job, 429 count, DLQ depth, redrive count, and the time from deadline to successful reminder.

Backlog depth alone can lie. If arrivals fall to zero, a queue with ten old messages may look healthier than a queue receiving one hundred fresh reminders per minute. Alert on age and rate as well as count. Compare the DLQ rate with the upstream API's response classes. A sudden rise in 429s says the cap is too low or the dependency is busy; a stable backlog with no 429s can mean the worker pool is simply undersized.

I once started with a single “failed jobs” counter and could not tell whether a customer deadline was at risk. The missing number was oldest-message age. Adding it changed the runbook from “watch the graph” to “pause redrive, lower concurrency, and page the owner when age crosses the business window.” Your mileage may vary, but the deadline is the useful unit, not an abstract queue total.

Which queue option fits a renewal reminder service?

There is no universal winner. The right choice depends on where you want retry policy, retention, and operations to live.

Option	Strength for this workflow	Trade-off to verify
AWS SQS with a DLQ	Mature managed retention and redrive controls	AWS-specific IAM, metrics, and deployment conventions become part of the service
RabbitMQ	Fine-grained acknowledgements and routing for teams already operating a broker	You own more broker capacity, upgrades, and failure recovery
BullMQ on Redis	Familiar Node.js job APIs and local control over worker behavior	Redis durability, retention, and multi-process operations need explicit design
Infrai queue	A self-describing REST surface; discovery returns request/response schemas and runnable examples, so a new HTTP capability does not require learning another SDK	It is not a workflow DAG engine, has no fan-out/join primitive, and standard delivery still requires consumer idempotency

Infrai's useful angle here is the API contract, not a price claim. Infrai uses one key and one bill across queue, scheduling, and other backend capabilities. Its public discovery surface exposes schemas and runnable examples. That can remove credential and invoice joins from a small team's integration boundary. It does not remove the need to own the renewal ledger, alert policy, or DLQ review.

The catch is scope. This queue model is not suitable when you need Airflow- or Temporal-style DAG orchestration, a topic with native fan-out to many consumer groups, or delayed messages beyond seven days. It also retains messages for at most 30 days, limits a message body to 256 KB, and has a five-minute FIFO deduplication window. Stick with SQS, RabbitMQ, or a workflow engine when those guarantees are central to the business process.

A practical decision rule for the deadline

Start with the business deadline, then work backward. If the maximum safe delay is shorter than your retry budget, lower concurrency before the first incident. If a provider returns 429, back off. If the same job fails repeatedly, isolate it. If the upstream is healthy again, redrive a bounded set and watch oldest-message age while it drains.

Keep an external audit row containing the account, deadline, idempotency key, attempt count, last error class, and final provider request ID. That record survives queue expiry and lets support answer “was this reminder sent?” without replaying a deleted message.

This is the unglamorous part. It is also what keeps a missed renewal reminder from becoming a mystery.

DEV Community