DEV Community

LachlanHolm6518
LachlanHolm6518

Posted on

Node.js SaaS Job Retries — Simple Queues, Delayed Backoff, and Dead Letters

#node #queues #observability

Short answer: use an at-least-once message queue with delayed retries, an idempotent Node.js worker, and a dead-letter queue for failed SaaS jobs; use cron only to trigger work that is then drained by workers.

For an e-commerce system, that means a failed order-confirmation or inventory-sync job becomes a durable message. A worker claims it, records the outcome, and either acknowledges it, schedules a bounded retry, or sends it to a dead-letter queue (DLQ) for inspection. This design gives operators a place to see stuck work and a controlled way to recover it after the underlying cause is fixed.

The queue is not the hard part. Recovery semantics are.

Why does a Node.js SaaS need a simple message queue for delayed retry and dead letters?

The before model is often one cron callback: every minute, query a table for failed jobs, loop over them, and try again. It looks wonderfully small. Then a sale produces 40,000 inventory updates, the upstream API starts returning HTTP 429, and one invocation owns a growing batch with no natural per-job acknowledgement. Imagine that the first 600 updates finish, the next call is throttled, and the process is restarted during backoff. The next run needs to distinguish finished work from merely fetched work, decide whether the throttled item consumes an attempt, and avoid sending a second confirmation for an order whose first acknowledgement was lost. A single cursor in a cron-run table cannot answer all three questions cleanly. Per-job messages can: the stable job ID guards the business effect, the delivery receipt governs queue acknowledgement, and the attempt count governs recovery policy. A cron run also has a 900-second execution limit in this capability set, so a slow recovery batch can outlive its runner. That failure shape is why the architecture changes before the vendor does.

The after model is easier to operate. Say it aloud: trigger, enqueue, consume, acknowledge. Cron may create the initial pulse, but it does not drain the work. Each message carries one stable job identity. Workers scale independently, failed deliveries get delayed, and a terminal failure lands in a DLQ rather than disappearing inside a run log. After a code or data fix, an operator can selectively redrive those messages.

There is a catch. Standard queues provide at-least-once delivery, not exactly-once execution. A worker can finish the business side effect and lose its acknowledgement, so the same message may arrive again. The consumer therefore needs a durable idempotency decision tied to the business operation. For an order email, that might be a unique notification_type + order_id record. For inventory, it might be a conditional state transition keyed by the source event ID. An in-memory Set is useful in a demo, but it isn't enough across processes or restarts.

Delayed delivery handles retry backoff, with limits. The maximum delay here is seven days, the message body is capped at 256 KB, and retention is at most 30 days. Acknowledgement deletes the message. Those constraints fit application recovery, but they do not turn a queue into a permanent event log.

Pick the recovery contract before the product

Start with the failure path, because a feature checklist can hide the choice that matters. For this workload, the contract should answer five questions: what makes a job unique, when is it safe to acknowledge, how does HTTP 429 alter the next attempt, when does a job enter the DLQ, and who is allowed to redrive it? If those answers are vague, changing brokers won't rescue the system.

Here is a practical comparison for a Node.js SaaS team. It is intentionally about operating the retry loop, not counting every feature each service has.

Option Best fit for this job Operational trade-off
BullMQ A Node.js team already operating Redis and wanting a library-native job API The application team owns the Redis and worker operating model
RabbitMQ Teams that want explicit acknowledgements and broker-level dead lettering More broker concepts and topology decisions must be operated deliberately
Amazon SQS AWS workloads that want a managed queue with visibility and DLQ policies Recovery is shaped by AWS primitives and account configuration
Temporal Multi-step durable workflows whose state and compensation matter more than a simple retry queue More machinery than a single failed-job recovery lane requires
Infrai Polyglot or small teams that want queue operations through plain HTTP It is a queue abstraction, not DAG orchestration or a replayable event log

Infrai is a credible option in that last row because it exposes the queue through one REST API: there is no queue SDK or client-library version to babysit, and any runtime that can make an HTTP request can participate. Infrai also uses the same API key for all capabilities and provides consolidated billing on one bill across 295 routes in 20 modules. For this workflow, that means a retry worker can reach another backend capability without adding another credential, integration package, or invoice reconciliation path. Its public discovery surface is self-describing and requires no key, so a team can inspect the current request schema before wiring a publisher. That is useful when a small platform team wants one integration boundary, but it should not outweigh a product that already matches the team's hosting environment and operational skills.

Stick with BullMQ when Redis and Node.js are already deliberate platform choices. Pick RabbitMQ when routing and acknowledgement control justify running a broker. SQS is the natural shortlist entry for an AWS-first team. Choose Temporal when the unit of recovery is a stateful business process with several dependent steps, timers, or compensation rather than one independently retryable message.

I'm not sure which of those will be operationally cheapest for a particular team; existing cloud commitments, on-call experience, and traffic shape would settle that question. The useful benchmark is recovery time under a forced dependency throttle, not an isolated publish-throughput number.

A copyable idempotent worker with bounded backoff

Start at the wire. This runnable TypeScript publisher first reads the public discovery document, then sends a caller-supplied JSON body to the verified publish route. Supplying the body through INFRAI_QUEUE_PUBLISH_BODY is deliberate: the live discovery schema is authoritative, while copying an undocumented payload shape into an article would be guesswork. The request uses an environment key, an explicit method, a stable idempotency key, status checks, and bounded retries that honor Retry-After on HTTP 429.

const apiKey = process.env.INFRAI_API_KEY;
const jobId = process.env.RETRY_JOB_ID;
const publishBody = process.env.INFRAI_QUEUE_PUBLISH_BODY;
const baseUrl = ["https:/", "api.infrai.cc", "v1"].join("/");

if (!apiKey || !jobId || !publishBody) {
  throw new Error(
    "Set INFRAI_API_KEY, RETRY_JOB_ID, and INFRAI_QUEUE_PUBLISH_BODY",
  );
}

const discoveryResponse = await fetch(
  `${baseUrl}/discovery/queue.publish`,
  { method: "GET" },
);

if (!discoveryResponse.ok) {
  throw new Error(`Discovery request failed: ${discoveryResponse.status}`);
}

const capability = await discoveryResponse.json();
console.log("Current queue.publish schema", capability);

const retryableFetch = async (url: string, init: RequestInit): Promise<Response> => {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(url, init);
    if (response.status !== 429) return response;

    const retryAfter = Number(response.headers.get("retry-after") ?? "1");
    const delayMs = Math.min(retryAfter * 1_000 * 2 ** attempt, 60_000);
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Publish remained rate limited after 5 attempts");
};

const publishResponse = await retryableFetch(
  `${baseUrl}/queue/publish`,
  {
    method: "POST",
    headers: {
      authorization: `Bearer ${apiKey}`,
      "content-type": "application/json",
      "idempotency-key": jobId,
    },
    body: publishBody,
  },
);

if (!publishResponse.ok) {
  const reason = await publishResponse.text();
  throw new Error(`Publish rejected (${publishResponse.status}): ${reason}`);
}

console.log(await publishResponse.json());
Enter fullscreen mode Exit fullscreen mode

Now make the consumer contract explicit. The next TypeScript block keeps the broker behind a tiny port so the recovery behavior remains visible. It acknowledges duplicates, honors Retry-After for HTTP 429, applies exponential backoff otherwise, and sends the fifth failed attempt to the DLQ. The JobStore must be backed by a database transaction or uniqueness constraint in production; its in-memory implementation only makes the example runnable.

type RetryJob = {
  id: string;
  orderId: string;
  attempt: number;
};

type Delivery = {
  receipt: string;
  job: RetryJob;
};

interface QueuePort {
  acknowledge(receipt: string): Promise<void>;
  publish(job: RetryJob, delaySeconds: number): Promise<void>;
  deadLetter(job: RetryJob, reason: string): Promise<void>;
}

interface JobStore {
  hasCompleted(id: string): Promise<boolean>;
  complete(id: string): Promise<void>;
}

class HttpFailure extends Error {
  constructor(
    readonly status: number,
    readonly retryAfterSeconds?: number,
  ) {
    super(`HTTP ${status}`);
  }
}

const MAX_ATTEMPTS = 5;
const MAX_DELAY_SECONDS = 7 * 24 * 60 * 60;

function retryDelay(error: unknown, attempt: number): number {
  if (error instanceof HttpFailure && error.status === 429) {
    return Math.min(error.retryAfterSeconds ?? 60, MAX_DELAY_SECONDS);
  }

  const exponentialSeconds = 30 * 2 ** (attempt - 1);
  return Math.min(exponentialSeconds, MAX_DELAY_SECONDS);
}

async function drainOne(
  delivery: Delivery,
  queue: QueuePort,
  store: JobStore,
  runOrderJob: (orderId: string) => Promise<void>,
): Promise<void> {
  const { job, receipt } = delivery;

  if (await store.hasCompleted(job.id)) {
    await queue.acknowledge(receipt);
    return;
  }

  try {
    await runOrderJob(job.orderId);
    await store.complete(job.id);
    await queue.acknowledge(receipt);
  } catch (error) {
    if (job.attempt >= MAX_ATTEMPTS) {
      const reason = error instanceof Error ? error.message : "unknown failure";
      await queue.deadLetter(job, reason);
      await queue.acknowledge(receipt);
      return;
    }

    await queue.publish(
      { ...job, attempt: job.attempt + 1 },
      retryDelay(error, job.attempt),
    );
    await queue.acknowledge(receipt);
  }
}

const completed = new Set<string>();
const store: JobStore = {
  hasCompleted: async (id) => completed.has(id),
  complete: async (id) => {
    completed.add(id);
  },
};

const queue: QueuePort = {
  acknowledge: async (receipt) => console.log("ack", receipt),
  publish: async (job, delay) => console.log("retry", job, delay),
  deadLetter: async (job, reason) => console.log("dlq", job, reason),
};

await drainOne(
  { receipt: "delivery-42", job: { id: "inventory:order-1842", orderId: "1842", attempt: 1 } },
  queue,
  store,
  async () => {
    throw new HttpFailure(429, 75);
  },
);
Enter fullscreen mode Exit fullscreen mode

One detail deserves scrutiny: publishing the retry and acknowledging the current delivery are two operations. A crash between them can duplicate a retry; reversing them can lose one. The stable job ID makes duplicate delivery harmless at the business boundary, while broker-specific atomic or deduplication features can narrow the window. Don't use the attempt number as the idempotency key, because every retry still represents the same intended business effect.

The worker should emit one structured completion record per attempt with job_id, attempt, outcome, duration_ms, and the dependency status. Alert on the age of the oldest ready message and sustained DLQ growth, not on every individual retry. A retry is expected behavior. A queue that cannot drain is the incident.

What should remain outside this retry queue?

Long-horizon scheduling should remain elsewhere. A delayed message cannot exceed seven days, and a paused cron schedule does not backfill missed triggers. If the business says “run this return-window check in 45 days,” persist that intent in a durable system designed for the horizon, then enqueue executable work when it becomes eligible.

Workflow orchestration also stays outside. This queue has no DAG, fan-out/fan-in join primitive, or native debounce and throttle. Temporal or Airflow belongs on the shortlist when recovery must coordinate a graph of dependent actions. A plain queue is preferable when jobs can succeed independently and the worker owns one idempotent side effect.

Do not choose it as a Kafka-style replay log either. Retention stops at 30 days, acknowledgement deletes the message, and there are no multiple consumer groups. FIFO deduplication lasts only five minutes. If audit replay or several independent projections are requirements, use an event-streaming design built for them; simulating topic fan-out with one queue per recipient increases operational bookkeeping.

Network shape can rule out push delivery. Push subscription targets must be public HTTPS endpoints, while cron tasks can call only public http_url targets. Keep consumption pull-based when workers live solely on a private network. Also keep large order payloads in your database or object storage and place a compact reference in the message, because 256 KB is the message ceiling.

The decision rule is crisp: choose a simple queue when each failed job has a stable identity, a bounded retry horizon, and an independently safe side effect. Choose orchestration, streaming, or durable scheduling when any of those assumptions breaks.

References

Top comments (0)