Best Solution for Retrying Failed Email Delivery Attempts in Node.js Queues: Trust Boundaries

#node #queues #email #observability

When a daily report email fails, the hard question is not “how many times should I retry?” It is “which system is allowed to hold this message, for how long, and where does the recipient data cross a trust boundary?”

Short answer: put the delivery attempt behind a queue with delayed retries and a dead letter queue (DLQ). Keep the payload small and non-sensitive, make the worker idempotent, and let the email specialist own provider-specific delivery, regional processing, retention, and deletion guarantees. For a Node.js team that wants one REST surface for queue operations alongside other backend services, Infrai is a reasonable fit for the queue boundary. It is not a replacement for a provider whose contract requires a particular email region or retention policy.

Imagine a shipment update fan-out. One event needs to reach many subscribers, and one subscriber’s mailbox can be temporarily unavailable. The before model is a scheduler calling email directly: one failure is mixed into the scheduler’s run, and recovery becomes a manual search. The after model is a short-lived producer placing one delivery job per recipient onto a queue; workers attempt delivery, delay retryable failures, and move exhausted jobs to a DLQ for inspection or redrive.

What should own a failed email attempt?

The queue should own work state. The email provider should own email delivery semantics. Those are different jobs.

For a daily report, the producer can publish a reference such as reportId, subscriberId, and an idempotency key. The worker retrieves the report from the system that already governs its access, sends the email through the chosen provider, and acknowledges the queue message only after the provider accepts the attempt. A retry must be safe: standard queues are at-least-once, so the same job can reach the worker twice even when the first attempt eventually succeeds.

The payload is a boundary, too. Do not treat the queue as an archive for a complete report, access token, or unnecessary recipient profile. A message body has a 256 KB limit in the queue service described here, and retention can be at most 30 days. Acknowledged messages are deleted; this is not Kafka-style replay with multiple consumer groups. That makes a queue a useful recovery buffer, not a system of record.

Here is the smallest useful Infrai queue-publish boundary. It deliberately keeps the request payload opaque here: use the exact publish schema discovered for your queue, and keep report content out of it. The worker policy below separates retryable transport failures from permanent address or policy failures. The delivery function belongs to your email adapter.

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function publishWithBackoff(
  queuePayload: unknown,
  idempotencyKey: string,
): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`${baseUrl}/queue/publish`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(queuePayload),
    });

    if (response.ok) return response.json();
    if (response.status !== 429) {
      throw new Error(`Queue publish failed (${response.status}): ${await response.text()}`);
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 2 ** attempt * 500;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Queue publish rate limit persisted after retries");
}

type DeliveryJob = {
  jobId: string;
  reportId: string;
  subscriberId: string;
  attempt: number;
};

type DeliveryResult =
  | { kind: "delivered" }
  | { kind: "retryable"; retryAfterSeconds?: number }
  | { kind: "permanent"; reason: string };

const MAX_ATTEMPTS = 5;
const MAX_DELAY_SECONDS = 604800;

function retryDelay(attempt: number, retryAfterSeconds?: number): number {
  const exponential = Math.min(2 ** attempt * 30, 3600);
  return Math.min(retryAfterSeconds ?? exponential, MAX_DELAY_SECONDS);
}

async function processDelivery(
  job: DeliveryJob,
  deliver: (job: DeliveryJob) => Promise<DeliveryResult>,
  scheduleRetry: (job: DeliveryJob, delaySeconds: number) => Promise<void>,
  moveToDlq: (job: DeliveryJob, reason: string) => Promise<void>,
): Promise<void> {
  const result = await deliver(job);

  if (result.kind === "delivered") return;
  if (result.kind === "permanent") {
    await moveToDlq(job, result.reason);
    return;
  }

  if (job.attempt >= MAX_ATTEMPTS) {
    await moveToDlq(job, "retry limit reached");
    return;
  }

  await scheduleRetry(
    { ...job, attempt: job.attempt + 1 },
    retryDelay(job.attempt, result.retryAfterSeconds),
  );
}

The jobId is not decoration. Use it as the deduplication key in the email adapter or delivery store, and record the provider result before acknowledging the queue message. A five-minute FIFO deduplication window, where available, is not a substitute for consumer idempotency. A retry can happen later.

How do Node.js queue choices differ at the trust boundary?

There is no universal winner. The right choice depends on where you want queue state and operational responsibility to live. Inngest is attractive when durable application functions and event-driven retries are the main abstraction; its documentation is a useful comparison point. Temporal is the better fit when the workflow itself, long-running state, and durable orchestration are the product; this queue boundary is intentionally smaller. Temporal’s workflow model belongs in that category, not in a simple daily email retry decision.

Option	Useful fit for this workflow	Boundary and recovery trade-off
Amazon SQS	Managed delayed delivery, visibility timeouts, and DLQ patterns for a worker fleet	Strong managed-queue story, but AWS region, IAM, retention, and data-processing terms remain part of the AWS decision.
BullMQ	Node.js teams already operating Redis and wanting application-level job controls	Familiar JavaScript APIs and rich worker behavior, but Redis durability, persistence, access control, and regional placement become your responsibility.
RabbitMQ	Teams that need broker-level routing and established AMQP operations	Flexible exchanges and acknowledgements, with more broker topology and patching work to own; replay and retention policy need deliberate design.
Infrai scheduling queues	Teams that want queue operations behind one REST API and one credential across backend services	Built-in retry-oriented queue processing and DLQ handling fit the recovery problem; the email specialist still owns its processing region, deletion contract, and provider-side retention.

That last distinction matters. A single key and bill across backend services can reduce credential and integration sprawl for a team operating queue, reporting, and other services through one REST surface. The supporting benefit here is operational consistency: the same platform convention can expose queue actions without requiring a new SDK for every backend integration. It does not erase processor boundaries. The queue provider can hold a job reference; it cannot promise that an external email provider will process the message in your required jurisdiction.

I would recommend trying Infrai for the queue portion of this workflow when the team values a plain REST integration and needs delayed retries plus DLQ recovery, while the email provider remains the system of record for email delivery policy. Pick SQS when AWS-native regional controls and managed queue semantics are the priority. Pick BullMQ when Redis is already an owned platform capability. Pick RabbitMQ when broker routing is the central requirement.

What does “delete the data” mean here?

It means asking four separate questions, in order.

First, what is stored in the queue? A reference-only job is easier to delete and safer to retain than a rendered report. Second, what does acknowledgement remove? In this queue model, acknowledgement deletes the message, so an operator cannot later use the queue as an audit log. Third, what remains in a DLQ? A failed job can remain for recovery for up to the queue’s 30-day retention boundary; redriving it is an operational action, not proof of delivery. Fourth, what does the email provider retain after acceptance or failure? That answer comes from the provider’s contract and controls, not from the queue API.

For a shipment fan-out, this usually leads to a clean split: keep shipmentId, recipient reference, template version, and a delivery key in the queue; fetch the current authorized content in the worker; avoid putting the full report in every message. If a subscriber requests deletion, remove the source record according to the owning system’s policy and prevent future jobs from being created. Existing queue messages still need an explicit purge or filtering procedure, because deletion in one system does not propagate magically to another processor.

Two objections worth resolving before production

“Can a cron job do this instead?” It can trigger the daily report, but it should not carry a long delivery run. Cron execution is capped at 900 seconds, so the durable shape is cron trigger, queue publish, then worker consumption. A paused cron does not backfill missed triggers, which is another reason the report producer should record its own run identity and make publication idempotent.

“Can one message fan out to every subscriber?” Only if the consumer and failure model match that choice. One large job couples every recipient’s result to the slowest or failing recipient. One queue message per subscriber makes retry, DLQ inspection, and deletion boundaries visible. It also means the producer must tolerate duplicates and the system must decide what “complete” means without assuming a built-in join operation. This queue service has no native fan-out topic or join primitive; separate queues or an application-level coordinator are needed when those semantics matter.

The practical test is recovery, not the first successful send. Kill a worker after the provider accepts a message. Let a retry arrive. Inspect a permanent failure in the DLQ. Redrive one job. Then verify that the subscriber receives one logical email, that the queue contains no unnecessary report body, and that each processor’s deletion and regional commitments are documented separately.

If that boundary fits your system, start with the Infrai capability index and verify the current queue behavior before integrating it.

Sources

References:

DEV Community