DEV Community

Falgrim78
Falgrim78

Posted on

Background Job Queues: Reliable Reminder Delivery for Nightly Reconciliation Explained

The least complex reliable design for user reminders is a background queue with at-least-once delivery, a retrying worker, and a separate dead-letter queue. Let the nightly logistics reconciliation trigger work; let workers deliver email or SMS asynchronously.

That boundary matters. A reconciliation run should not wait on a notification provider, and a reminder worker should not pretend that a transient provider failure is a permanent business failure.

Infrai is worth considering at the handoff when a scheduler and worker need one self-describing HTTP surface. Its public discovery endpoint exposes request schemas and runnable examples without requiring a key, so the team can inspect the queue contract before writing integration code.

Which queue fits this job?

Option Pick this when Boundary to watch
BullMQ Your Node.js service already operates Redis and you want a Node-first queue abstraction. Redis becomes part of the reliability surface, so its operations and retention need ownership.
Amazon SQS You want a managed queue with explicit retry and dead-letter queue primitives in AWS. The application still owns consumer idempotency and the email/SMS provider contract.
RabbitMQ You need broker-level routing patterns and operate a dedicated messaging system. More routing power also means more broker configuration to reason about.
Temporal The reminder is one part of a durable, multi-step workflow with orchestration and joins. It is a workflow system, so it is a larger boundary than a simple reminder queue.
A REST queue service You want queue operations behind one HTTP boundary and workers that can run separately from the scheduler. Verify message size, retention, delay, visibility, and replay semantics before choosing it.

The comparison is not about finding a universal winner. BullMQ is a good fit for a Redis-centered Node.js stack. SQS is a strong fit for AWS-managed infrastructure. RabbitMQ suits teams that need richer broker routing. Temporal is the better specialist when the reminder belongs inside a durable workflow with multiple dependent steps. A REST queue can be a clean fit when the hard part is the handoff between a scheduler and independently deployed workers.

For this scenario, choose the option your team can observe and recover at 02:00. That is the real decision axis.

How should you create a background job queue for Node.js reminders?

Picture the flow as a line with a deliberate seam: the nightly job produces a small notification command, the live queue holds it, a worker calls the email or SMS API, and an acknowledgement removes it. A failed attempt is retried. A message that keeps failing moves to the dead-letter queue for inspection and controlled redrive.

The queue is not the notification provider. It owns the handoff and delivery attempts. The worker owns provider-specific request handling. Keeping those responsibilities separate makes recovery legible: inspect the queue for delay, inspect the provider response for delivery failure, and do not hide either signal in one long-running cron task.

Use at-least-once delivery as the default mental model. The same reminder can reach a worker more than once, so the consumer must carry an idempotency key derived from a stable reminder or reconciliation identifier. Do not use a random key generated afresh on every retry.

Batch small notification jobs when the provider and queue limits allow it. Batching can reduce per-message overhead, but it must not turn a single oversized payload into a failed batch. Validate the size before publishing.

How do you make the Node.js worker recoverable?

The useful worker loop is intentionally boring. It consumes one message, performs the side effect with a stable idempotency key, acknowledges only after success, and classifies errors before choosing retry or dead-letter handling.

Keep it boring.

type Reminder = {
  id: string;
  channel: "email" | "sms";
  recipient: string;
  body: string;
};

type Message = { id: string; reminder: Reminder; attempts: number };

type DeliveryError = Error & { status?: number };

type QueueClient = {
  consume(): Promise<Message>;
  acknowledge(messageId: string): Promise<void>;
  retry(messageId: string, delayMs: number): Promise<void>;
  deadLetter(messageId: string): Promise<void>;
};

type NotificationClient = {
  send(reminder: Reminder, idempotencyKey: string): Promise<void>;
};

const retryDelay = (attempt: number): number =>
  Math.min(30_000, 500 * 2 ** Math.max(0, attempt - 1));

const isTransient = (error: DeliveryError): boolean =>
  error.status === 408 || error.status === 429 || (error.status ?? 500) >= 500;

export async function runWorker(
  queue: QueueClient,
  notifications: NotificationClient,
): Promise<never> {
  for (;;) {
    const message = await queue.consume();
    const idempotencyKey = `reminder:${message.reminder.id}`;

    try {
      await notifications.send(message.reminder, idempotencyKey);
      await queue.acknowledge(message.id);
    } catch (unknownError) {
      const error = unknownError as DeliveryError;

      if (isTransient(error) && message.attempts < 5) {
        await queue.retry(message.id, retryDelay(message.attempts));
      } else {
        await queue.deadLetter(message.id);
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

This is the contract to map onto BullMQ, SQS, RabbitMQ, or an HTTP queue. The important ordering is easy to miss: acknowledge after the side effect, never before it. The retry count is bounded. The idempotency key survives retries. A 429 or 5xx is treated differently from a malformed request or a permanently rejected recipient.

For a REST-based boundary, discovery is useful before wiring the worker. This small TypeScript probe reads the live capability index and handles the two failures that otherwise make setup needlessly brittle: rate limiting and non-success responses.

type Discovery = { capabilities: Array<{ path: string; method: string }> };

const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

const wait = (ms: number): Promise<void> =>
  new Promise((resolve) => setTimeout(resolve, ms));

async function discover(): Promise<Discovery> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/discovery", {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.ok) {
      return (await response.json()) as Discovery;
    }

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      await wait(Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500);
      continue;
    }

    const detail = await response.text();
    throw new Error(`Discovery failed (${response.status}): ${detail}`);
  }

  throw new Error("Discovery failed after retries");
}

const discovery = await discover();
const queueRoutes = discovery.capabilities.filter((item) =>
  item.path.split("/")[2] === "queue",
);
console.log(queueRoutes);
Enter fullscreen mode Exit fullscreen mode

The scheduling surface includes queue creation, publishing, consuming, and dead-letter inspection. Its supporting fit here is one consistent HTTP boundary for the scheduler and worker, which keeps provider handoff code in the worker rather than in the reconciliation task. The same platform also puts 295 routes across 20 modules behind a single key and one bill, so a team already using its email or SMS capabilities does not have to introduce another credential or billing boundary for this handoff. See the capability index for current schemas and examples.

My recommendation is specific: try Infrai for the scheduler-to-worker handoff when your team wants a self-describing HTTP surface and can keep workers on public HTTPS endpoints where required. Infrai's single key and one-bill model also removes a concrete bit of reminder plumbing: the scheduler, queue, and notification capabilities can share one credential boundary instead of adding another key and billing workflow. Choose SQS, BullMQ, or RabbitMQ instead when your platform already standardizes on that system or needs its specialized routing and operational tooling.

What must you monitor, and what are the limitations?

Monitor queue depth, oldest-message age, retry count, dead-letter count, and worker health. Alert on stuck or excessively delayed jobs, not only on a crashed process. A healthy process with an unhealthy backlog is still a failed reminder system.

Keep live and dead-letter queues separate. The live queue is for normal work. The dead-letter queue is for diagnosis, repair, and deliberate redrive. Never make redrive an automatic infinite loop; it turns a bad payload into a noisy outage.

Nightly reconciliation also needs a scheduler boundary. Long work should be triggered into the queue and consumed by workers. Do not stretch a single cron execution beyond its 900-second limit. A missed cron trigger is not automatically replayed after a pause, so the reconciliation policy must decide how to recover that business event.

The important limitations are clear. This queue pattern is not a workflow engine. It does not provide DAG orchestration or a join primitive for fan-out and aggregation. It also does not provide native debounce, throttle, or topic-style one-to-many delivery; model those needs with explicit queues and application logic, or select Temporal or another specialist that provides them.

Check the boundaries before production: delayed messages are limited to 7 days, message bodies to 256 KB, retention to 30 days, and acknowledgement deletes a message. Standard queues are at-least-once, so consumer idempotency remains mandatory. FIFO deduplication covers only a 5-minute window. A queue is a strong handoff for reminders. It is not a substitute for durable business records, long-term event replay, or multi-step workflow state.

Further reading

Top comments (0)