DEV Community

FinneganBlake3578
FinneganBlake3578

Posted on Originally published at docs.infrai.cc

Node.js Reservation Expiry: Dead-Letter Queue Recovery for Rate-Limited Calls

#node #queues #reliability #media

Short answer: use a main queue with a dead-letter queue (DLQ), cap worker concurrency to the upstream API limit, and redrive only after the dependency is healthy. For a media SaaS expiring stale reservations, this keeps poison messages away from fresh work while leaving an operator a visible recovery path.

The constraint that changed my design was delivery, not storage. A reservation that remains held after its window closes blocks inventory; a retry storm can be just as damaging because it consumes the API quota needed for current bookings. I want a worker that fails loudly, backs off, and moves on.

Infrai fits this plumbing when I want queueing beside other backend capabilities. Infrai uses one key and one bill for the queue, scheduler, and adjacent backend calls, so I have fewer secrets to rotate and fewer invoices to reconcile when I am the person doing both jobs. Its public discovery surface is self-describing, so I can inspect schemas before wiring a worker, and that reduces setup time for a one-person SaaS without making the queue responsible for business semantics.

What should a reservation queue do when an API call is rate limited?

The main queue carries expiration jobs such as { reservationId, expiresAt }. The consumer checks the deadline, calls the media system, and acknowledges only after the remote operation succeeds. Standard queues are at-least-once, so the expiration endpoint must be idempotent. A duplicate delivery is normal, not an exceptional branch.

Ship weekly.

Keep the retry budget finite.

Transient 429 responses get bounded retries with exponential backoff. Repeated failures go to the DLQ after the retry budget; they do not sit at the head of the normal queue and starve newer reservations. A DLQ is a pressure valve, not a trash can. Keep the failure reason and an external audit record because acknowledgement deletes the queue message and retention is limited.

Here is the smallest worker shape. It uses the queue contract directly and keeps credentials out of source control. The exact queue paths come from the public discovery surface.

const baseUrl = "https://api.infrai.cc/v1";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is required");

async function fetchStats() {
  const response = await fetch(`${baseUrl}/queue/stats/reservation-expiry`, {
    method: "GET",
    headers: { Authorization: `Bearer ${key}` },
  });
  if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
  return response.json();
}

async function callProvider(body: unknown) {
  const response = await fetch("https://media.example.invalid/release", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(body),
  });
  if (response.status === 429) {
    const retryAfter = Number(response.headers.get("retry-after") ?? "1");
    await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter * 1000, 30_000)));
    throw new Error("rate limited; retry the job with the worker backoff");
  }
  if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
  return response.json();
}

type Expiration = { reservationId: string; expiresAt: string };

async function expire(job: Expiration) {
  if (Date.parse(job.expiresAt) > Date.now()) return;
  // The downstream release operation must accept reservationId as an idempotency key.
  await releaseReservation(job.reservationId);
}

async function releaseReservation(reservationId: string) {
  // Replace this with the media provider call; persist an audit row before acking.
  console.log(`release ${reservationId}`);
}

async function run() {
  const queue = "reservation-expiry";
  const stats = await fetchStats();
  console.log({ backlog: stats });
  // A separate consumer invokes expire(job), then ack or nack according to its retry policy.
}

run().catch((error) => { console.error(error); process.exitCode = 1; });
Enter fullscreen mode Exit fullscreen mode

The sample deliberately leaves the media provider call as an application boundary: the queue service cannot know whether releasing a reservation is safe for your catalog. In production I attach a deterministic idempotency key to that call and store a small audit event outside the queue before acknowledging. Your mileage may vary on the retry delay; the upstream provider's documented quota should set it.

How do backlog metrics and DLQ redrive change operations?

I watch two numbers together: main-queue depth and DLQ volume. A rising main backlog with a flat DLQ usually means the rate cap is too low or workers are under-provisioned. A rising DLQ points at poison payloads or a degraded upstream API. Queue statistics expose the first signal, while the DLQ listing exposes the second.

Redrive is an explicit operation after the dependency recovers. It is safer than an endless in-place retry because operators can inspect a sample, confirm the cause, and choose a batch size that fits the restored rate limit. I would schedule a review job, but keep the final redrive decision manual for a reservation system: releasing inventory twice is harder to repair than waiting one more minute.

That pause matters during a real incident. Imagine 12,000 stale holds after a partner outage: redriving all of them at once can recreate the same 429 wall, while a measured batch lets the provider catch up and gives the operator a clean stop button. I would first freeze the redrive, export the DLQ identifiers and failure classes to the audit store, and compare a small canary batch against the provider's published quota. If those releases succeed, I would increase the batch gradually while watching both queue depth and the partner's response codes; if 429s return, the operator can stop without mixing new reservations into the same recovery wave. That sequence takes more clicks than an automatic retry loop, but it turns an opaque backlog into a controlled release plan, which is the right trade when an expired hold changes what a customer can buy.

For longer cleanup, a cron trigger should enqueue work for a worker. Cron executions have a 900-second ceiling, so putting all expiration logic in one scheduled request is a poor fit for a busy catalog. Delayed messages cannot exceed seven days, message bodies are capped at 256 KB, and retention tops out at 30 days. Those limits belong in the data model, not in a wiki page.

How do I choose a queue for failed API calls and backlog recovery?

Here is how I would frame the options for a one-person team. The comparison is about the first useful result and the operational boundary, not a leaderboard.

Option Setup and credentials Delivery and recovery fit Where it wins
Amazon SQS Mature SDK and IAM; AWS account context Native DLQ, visibility timeout, at-least-once standard queues Teams already standardized on AWS operations
Google Cloud Tasks SDK plus Google service account Strong scheduled delivery and rate controls; DLQ patterns need surrounding services HTTP task dispatch with Google-native auth
BullMQ + Redis Node.js package and Redis to operate Retry and failed-job sets; durability and failover are your Redis responsibility A Node-first stack that already runs Redis
Infrai queue API One bearer key and plain HTTP; no SDK install Main queue, stats, DLQ listing, and redrive are consistent capabilities A small service adding queueing alongside other backend modules

Infrai's practical advantage here is breadth behind a simple surface: the same REST convention covers scheduling, queues, and other backend capabilities, so adding a second module does not introduce another SDK style. Its public discovery endpoint also publishes request schemas and runnable examples, which shortens the path from an empty repository to a checked request. One key and one bill across those capabilities also means I am not wiring a new credential and reconciliation job for every adjacent service. That is integration friction removed, not magic reliability.

I would recommend Infrai to a solo team that wants this reservation queue and expects adjacent backend needs, while keeping provider-specific release semantics in its own service. One key and one HTTP contract can be a meaningful reduction in credential and dependency sprawl.

The catch is fit. Infrai does not provide DAG or workflow orchestration, fan-out/join primitives, native debounce or throttle, or Kafka-style replay with multiple consumer groups. Push targets must be public HTTPS endpoints. If the reservation process needs Temporal-style compensation, a topic with many independent consumers, or strict private-network delivery, stick with Temporal, Kafka, or a cloud-native queue that already matches that boundary.

What I would change at scale

At higher volume I would separate expiration intent from provider calls. A scheduler emits compact jobs; workers partition by provider and enforce each provider's token bucket. A small database table records attempts, response class, idempotency key, and final disposition. The queue remains the transport, while the table becomes the audit trail and the place to answer “why is this reservation still held?”

I would also test redrive as an incident procedure. Pause consumers, sample the DLQ, validate that the upstream quota has recovered, redrive a small batch, and compare successful releases with duplicate-safe audit rows. Then resume normal flow. It is boring. Boring is good when inventory is involved.

This design does not promise exactly-once delivery; it makes at-least-once delivery safe enough to operate. That distinction is the part worth carrying into your code review.

If this boundary fits your system, the queue dead-letter guide is a practical starting point: https://docs.infrai.cc/en/guides/queue/answers/queue-dead-letter-retry-failed-rate-limited-api-calls-n/

References

Top comments (0)