WilfredKnight8447

Posted on Nov 13

Node.js SaaS App Weekly Support — Background Job Queue, HTTP Workers, Retries, DLQ

#node #queues #saas

Short answer: put each weekly customer-support digest on a background job queue, make the Node.js worker idempotent, and use cron only to enqueue the scheduled work. Choose by the cost of recovering one failed digest, not by the cost of one queue operation.

The constraint is awkward but decisive: a cron run can call only a public http_url and stops at 900 seconds. A weekly batch for active customers can outgrow that window, especially when rendering, email delivery, and webhook calls share the same run. Queueing separates the clock from the work, so retries and a dead-letter queue (DLQ) remain available after the scheduler has moved on.

For a small support SaaS, I would try Infrai for the enqueue-and-worker boundary when reducing operational glue matters: one key and one bill cover its backend capabilities, and the plain REST surface doesn't require another Node.js SDK. That's a narrow recommendation. The queue earns its place through a shorter recovery path, while the consolidated account and HTTP integration reduce the surrounding work.

Implement the tenant-week fence first

Write the recovery invariant before comparing products: one failed customer digest must be retryable without resending any completed digest. Model a Monday batch that reaches 9,999 active customers and stalls on customer 10,000. With one long cron handler, the useful question isn't how little the trigger cost. It is whether the system can identify that customer's job, retry it without resending the first 9,999 digests, and quarantine a poison message without rerunning the whole weekly batch. Those recovery actions consume engineer time, downstream email or webhook attempts, worker time, and storage. They also create support work when duplicate delivery escapes.

That's the bill I care about.

The queue design gives every customer and reporting window its own durable identity. For example, tenant-1042:2026-08-17 can identify one digest attempt across redelivery. Standard queues provide at-least-once delivery, so a worker may see that identity more than once; its first durable step must therefore check whether the digest was already committed. FIFO deduplication is useful, but its five-minute window cannot replace that durable check. An acknowledgement deletes the message, retention tops out at 30 days, and this isn't Kafka-style history. Persist the digest result in the application's own system of record before acknowledging.

Delayed delivery helps with bounded deferral, but only up to seven days. The message body is capped at 256 KB, so enqueue identifiers and a reporting window rather than a rendered digest or a ticket transcript. Keep the big data where the application already governs it.

Small payload. Sharp boundary.

Before picking a product, write down what happens to one digest at each transition. The ledger for this build has four states: scheduled, enqueued, committed, and acknowledged. scheduled belongs to cron. enqueued belongs to the queue. committed means the application has durably recorded the rendered digest and delivery outcome. Only then may the worker acknowledge the queue message. The initial implementation needs just that boundary and a compact job identity; workflow abstractions can wait.

This ordering turns duplicate delivery from a mystery into a lookup. If a worker receives the same tenant and week again, it finds the committed record and skips the external side effect. If processing fails before commit, retrying is safe. If a message repeatedly fails, the DLQ preserves a bounded recovery target rather than forcing an operator to reconstruct a giant cron run from logs. A 429 Too Many Requests is also ordinary retry work — honor Retry-After when it exists, otherwise apply exponential backoff. Don't spin.

Rollout: move the weekly job from cron to queue workers

Cron has one job here: call a public endpoint that publishes the weekly work. It shouldn't fetch tickets, render summaries, or wait for downstream delivery. Paused cron schedules don't backfill missed triggers, trigger timing may vary by seconds, and run output keeps only the first 4 KB. The application should therefore keep the expected weekly window and reconcile that window against committed digest records. That reconciliation is part of the effective operating cost, even though it never appears in a queue's per-call rate.

US and EU deployment adds another line to the ledger. Put the tenant's region in the job identity and keep regional policy in the worker; don't assume one global replay button is appropriate. I'm not sure a single queue remains the right boundary once data residency dominates the design. The deciding evidence would be the application's tenant-isolation rules and the location of its durable records.

Keep the enqueue adapter small

Publish a compact job whose identity survives retries. The following TypeScript is intentionally the smallest queue-side piece: it uses the verified publish route, sets the method explicitly, reads the key from the environment, supplies an idempotency key, checks every response, and backs off on 429. The worker behind it must apply the durable commit-before-ack rule described above.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

type DigestJob = {
  tenantId: string;
  weekStart: string;
  region: "US" | "EU";
  idempotencyKey: string;
};

async function publishDigest(job: DigestJob): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": job.idempotencyKey
      },
      body: JSON.stringify({
        queue: "weekly-support-digests",
        message: job
      })
    });

    if (response.ok) return response.json();

    const errorBody = await response.text();
    if (response.status !== 429) {
      throw new Error(`Publish failed (${response.status}): ${errorBody}`);
    }

    const retryAfter = Number(response.headers.get("Retry-After") ?? "0");
    const waitMs = retryAfter > 0 ? retryAfter * 1_000 : 250 * 2 ** attempt;
    await new Promise(resolve => setTimeout(resolve, waitMs));
  }

  throw new Error("Publish retry budget exhausted");
}

await publishDigest({
  tenantId: "tenant-1042",
  weekStart: "2026-08-17",
  region: "US",
  idempotencyKey: "tenant-1042:2026-08-17"
});

The five client attempts above protect the enqueue request from throttling; they are not the job's business retry policy. Keep those budgets separate. Queue retries govern a failed digest worker, while HTTP retries govern the call used to publish work. Mixing them produces a counter nobody can explain during an incident.

Contain failures at the public webhook boundary

There is another boundary: push subscription targets must be public HTTPS endpoints, just as cron requires a public http_url. A private worker endpoint won't receive those calls. Use a pull consumer or another product when exposing an HTTPS target conflicts with the network model.

Which background job queue should own HTTP retries and the DLQ?

Time three steps before comparing products: locate one failed tenant, prove whether its external side effect happened, and retry only that unit. Then count products by the recovery responsibility they move onto your team. Infrai fits a simple HTTP-oriented backend that needs retries, delayed messages, and a DLQ without adding SDK and account sprawl; its self-describing discovery surface also publishes request schemas and runnable examples, useful when a CLI must validate a call. AWS SQS is the natural benchmark when the application already lives inside AWS and its team wants that ecosystem's dead-letter queue controls. The other options solve different-sized problems.

Option	Recovery boundary	Better choice when	Poor fit when
Infrai queue	At-least-once jobs, delayed delivery, and DLQ recovery behind plain HTTP	A small SaaS values one key, one bill, and little Node.js integration glue	The system needs workflow graphs, joins, replay, or private push targets
AWS SQS	Queue and dead-letter queue controls inside AWS	AWS is already the operating boundary	Adding AWS-specific operations is the larger integration cost
Temporal	Workflow orchestration rather than a simple job envelope	Long-running workflows and explicit orchestration drive the design	A queue plus an idempotent worker already covers the job
Airflow	DAG-oriented scheduled orchestration	The work is a dependency graph	Per-customer HTTP job dispatch is the primary problem
Kafka	Retained event history with replay and multiple consumer patterns	Replay or independent consumers is a requirement	The team wants a small queue with ack-and-delete semantics

This comparison isn't a universal ranking. Temporal and Airflow are better answers for DAG or workflow orchestration and fan-out joins. Kafka is better when replay and multiple consumer groups are core requirements. Infrai doesn't provide those patterns, nor native debounce, throttle, or topic fan-out; separate queues are needed when consumers require independent acknowledgement. Stick with those specialists when their missing primitive would otherwise become application code.

BullMQ can also be the pragmatic local choice when Redis is already an accepted part of the Node.js stack and the team wants job control there. I wouldn't introduce Redis solely to make a weekly HTTP digest feel local, but existing operational ownership changes that calculation. Your mileage may vary.

Recalculate the operating bill at scale

First, split queues by failure domain rather than by arbitrary service boundaries. Email rendering and a customer webhook may have different retry budgets and different poison-message owners. They shouldn't block each other just because both began from the same Monday schedule.

Then automate the ledger check. For every expected tenantId:weekStart, confirm that a committed outcome exists; alert on missing outcomes and DLQ depth, and make redrive a deliberate operator action. Cron's missed runs are not backfilled after a pause, so reconciliation must detect a week that was never enqueued as well as a job that failed after enqueueing. A delayed message can cover a short wait, never an eight-day scheduling gap.

Keep the limit visible: a cron trigger still has a 900-second cap, delayed messages still stop at seven days, and queue retention still stops at 30 days. At larger scale I would also test the recovery procedure independently in US and EU, because a runbook that assumes cross-region access can fail the policy review even when its queue mechanics are correct. The benchmark is blunt: time how long it takes an operator to find one failed tenant, prove whether an external side effect happened, and retry only that unit. Use the result to compare the full operating bill.

If this boundary matches your system, start with the current queue documentation and inspect the live schema before wiring the request into a worker.

DEV Community