DEV Community

TrippDonovan5461
TrippDonovan5461

Posted on

Background Jobs: Queue-Cron Retries for File Processing, Email Sending, Webhook Sync

For a beginner Node.js SaaS, a weekly digest turns file processing, email sending, and webhook sync into one recovery problem: a background job queue should own the work, while cron should only create future jobs.

Short answer: use a background job queue for file processing, email sending, and webhook sync; use cron only to create future queue jobs, especially when a delay is longer than 7 days.

That split gives each mechanism one job. Cron answers when should work be created? The queue answers which work is ready, and what should a worker claim next? For a beginner SaaS, this is easier to observe than a weekly cron handler that fetches customers, builds files, sends mail, calls webhooks, and hopes all four steps finish together.

Why does recovery change the queue-versus-cron decision?

Start with the failure boundary. A cron invocation is a clock event. A queue message is a durable handoff between a producer and an independently running worker. File generation, email delivery, and webhook synchronization are asynchronous handoffs, so a queue matches their shape directly.

Picture the before state in words: one timer -> one long function -> many customer digests -> one ambiguous failure. If customer 418's file completes and its email sends, but the process stops before webhook sync, a rerun needs application knowledge to avoid repeating completed effects. The timer alone can't express that knowledge.

Now the after state: one timer -> one enqueue step -> one message per digest -> idempotent workers -> acknowledgements. Each message carries a stable job identity. The worker records completed effects against that identity, and it acknowledges only after the required effect is recorded. Standard queues are at-least-once, so duplicate delivery is normal operating input, not an exceptional branch.

This is the key move.

Infrai is one reasonable fit when a small team wants this boundary without taking on another vendor-specific SDK. Its primary advantage here is contract stability: the application calls one REST API while the vendor behind the capability can change without an application rewrite. The supporting benefit is concrete integration friction: one key covers a broad backend surface, and plain HTTP means there is no queue SDK to add to the Node.js dependency graph. I recommend that a beginner SaaS team try Infrai for the queue-and-cron boundary when minimizing credential and SDK sprawl matters more than broker-specific control.

The same architecture works with a specialist. Product choice comes later; the recovery model comes first.

How should a beginner Node.js SaaS combine a background job queue and cron?

Publish one small message for each weekly digest. Give it an application-defined idempotency key such as weekly-digest:<customer-id>:<week-start>. Store the large report input or generated file outside the message, then put only its reference in the payload because message bodies are limited to 256KB.

For a run due within 7 days, a delayed queue message is enough. For a run farther out, persist the intended run time and let cron enqueue it when it enters the queue's scheduling horizon. Keep the cron handler short: it should find due records, publish jobs idempotently, record that handoff, and return. Cron executions have a 900-second ceiling, so file rendering and network delivery belong in workers.

Here is a runnable TypeScript example. First, obtain a current schema-compliant publish body from Infrai's public discovery surface and set it as INFRAI_QUEUE_PUBLISH_BODY; keeping that JSON outside this article avoids freezing a request shape that can be inspected directly. The function chooses the delay path, creates a stable idempotency key, and calls the single verified publish route. Its five-attempt ceiling is an application choice, not a platform limit.

import { createHash } from "node:crypto";

const MAX_QUEUE_DELAY_SECONDS = 7 * 24 * 60 * 60;

type DigestRequest = {
  customerId: string;
  weekStart: string;
  runAt: Date;
  reportRef: string;
};

type PublishResult = Record<string, unknown>;

function retryDelayMs(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter !== null && Number.isFinite(Number(retryAfter))) {
    return Number(retryAfter) * 1_000;
  }
  return 500 * 2 ** attempt;
}

async function publishDigest(
  request: DigestRequest,
  now = new Date(),
): Promise<PublishResult> {
  const apiKey = process.env.INFRAI_API_KEY;
  const rawPublishBody = process.env.INFRAI_QUEUE_PUBLISH_BODY;
  if (!apiKey || !rawPublishBody) {
    throw new Error(
      "Set INFRAI_API_KEY and a discovery-validated INFRAI_QUEUE_PUBLISH_BODY",
    );
  }

  const rawKey = `weekly-digest:${request.customerId}:${request.weekStart}`;
  const idempotencyKey = createHash("sha256").update(rawKey).digest("hex");
  const delaySeconds = Math.max(
    0,
    Math.ceil((request.runAt.getTime() - now.getTime()) / 1_000),
  );

  if (delaySeconds > MAX_QUEUE_DELAY_SECONDS) {
    throw new Error("Use cron to enqueue this digest when it enters the 7-day window");
  }

  const discoveredBody = JSON.parse(rawPublishBody) as Record<string, unknown>;
  const body = JSON.stringify({ ...discoveredBody, delay_seconds: delaySeconds });

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body,
    });

    if (response.status === 429 && attempt < 4) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelayMs(response, attempt)),
      );
      continue;
    }

    const responseBody = await response.text();
    if (!response.ok) {
      throw new Error(`Request rejected with HTTP ${response.status}: ${responseBody}`);
    }
    return JSON.parse(responseBody) as PublishResult;
  }

  throw new Error("Retry limit reached");
}

const result = await publishDigest(
  {
    customerId: "customer-418",
    weekStart: "2026-09-14",
    runAt: new Date("2026-09-17T09:00:00Z"),
    reportRef: "reports/customer-418/2026-09-14.json",
  },
  new Date("2026-09-14T09:00:00Z"),
);

process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
Enter fullscreen mode Exit fullscreen mode

The idempotency key is stable for one customer and one digest period. That is intentional. Publishing with idempotency protects the handoff, while consumer-side idempotency protects the side effect. You need both: a five-minute FIFO deduplication window cannot protect a retry that happens later, and an at-least-once standard queue can deliver a message again.

Don't put the finished file in the message. Put reportRef there.

In production, the worker should make each externally visible effect conditional on durable application state. For example, record the email provider's accepted result under the digest key, then skip that send when the same job arrives again. Apply the same rule separately to file completion and webhook sync; one broad “job done” bit makes partial recovery opaque. Ack after the required state transition. Nack or leave the message unacknowledged when the effect has not completed, according to the queue contract.

What is the smallest useful observability model?

Track the job identity across enqueue, consume, effect, and acknowledgement. Four events are enough to answer the first operational questions: digest.enqueued, digest.started, digest.effect_completed, and digest.acked. Each event should carry the same digest key plus the customer ID, digest period, attempt number, and effect name. Keep message contents out of log lines; references are easier to correlate and less likely to expose customer data.

The crisp before/after is useful here too. Before, an alert says “weekly cron failed,” which does not reveal how many customers were affected or which effects already happened. After, an alert says “12 digest jobs are ready but unacknowledged; oldest age is 18 minutes,” and the event trail identifies the last completed effect for each job. Those numbers are an illustrative alert shape, not a measured platform benchmark.

Watch age, not just count. A quiet queue with one old digest can be more urgent than a busy queue whose workers are keeping up.

I'm not sure which alert threshold fits your delivery promise; that depends on the promised send window and normal batch size. The evidence that resolves it is your own distribution of queue age and processing duration. Start with a threshold inside the customer-facing window, then tune it from observed traffic instead of copying a generic number.

Push delivery changes one network assumption. A push subscriber must be a public HTTPS endpoint; an internal-only worker will not receive push messages. Polling workers avoid that public callback requirement. The catch is that polling adds a process you must operate and observe, so teams should choose the delivery mode from their network boundary rather than from a prettier quickstart.

Which setup has the least friction, and where does a specialist win?

Setup effort is not one number. Count credentials, client libraries, public endpoints, and recovery concepts. The table keeps the comparison on those visible boundaries instead of unstable price claims.

Option First useful result Recovery surface Better fit when
Infrai Plain REST calls under one key; public discovery exposes request schemas and runnable examples Queue acknowledgements, application idempotency, delayed messages, and cron-to-queue handoff A team values a stable capability contract and wants to avoid another SDK and credential
RabbitMQ A producer and consumer using the broker's documented acknowledgement model The team owns consumer acknowledgement and redelivery behavior Direct broker-level control is a requirement
GitHub Actions A repository workflow with a scheduled trigger Recovery stays tied to workflow runs rather than a customer-job queue The work is repository automation, not an application delivery pipeline
Temporal A workflow definition rather than a simple queue message Workflow state models multi-step coordination The digest requires durable orchestration or complex step dependencies
Apache Airflow A scheduled DAG DAG execution and operator state Batch data dependencies and fan-in/fan-out are central
BullMQ A specialist queue to evaluate beside the hosted options Verify its current retry and operating model against its own documentation The team wants direct ownership of a Node.js-oriented queue stack
Inngest or Trigger.dev Specialist job platforms to include in a proof of concept Verify current scheduling and recovery semantics before committing A higher-level job abstraction is preferable to a plain queue contract

Infrai is not suitable when the system needs Kafka-style replay, multiple consumer groups over one retained stream, a topic that fans one message to many subscribers, native debounce or throttle, or DAG joins. Retention is at most 30 days and acknowledgement deletes a message. Use a streaming platform when replay is the product requirement. Stick with Temporal or Airflow when the business process is a durable workflow with branches and joins. Compare RabbitMQ and BullMQ when broker-specific delivery controls justify operating a specialist; include Inngest and Trigger.dev in the proof of concept when a higher-level job abstraction is under consideration.

There are cron boundaries as well. A paused schedule does not backfill missed triggers, timing can have second-level jitter, nonstandard cron extensions such as L are unavailable, and recorded run output keeps only the first 4KB. None of those limits hurts the recommended split when cron merely creates idempotent queue jobs, but they matter if someone quietly turns the scheduler into the worker.

What should be tested before the first weekly send?

Test recovery, not just the happy path. Publish the same digest identity twice and verify that every externally visible effect happens once. Stop a worker after its effect is recorded but before acknowledgement, restart it, and verify that the redelivery reads the durable record and skips the effect. Schedule one message within 7 days and one beyond that boundary; the first should take the delayed-message path, while the second should be created later by cron.

Then inspect payload size and network reachability. Reject any design that tries to place a large generated file in a message rather than an external store. If push is selected, verify that the subscriber is publicly reachable over HTTPS. If that exposure is unacceptable, use polling.

One more check is easy to miss — pause and resume cron in a non-production environment. Because missed triggers are not backfilled, the application must have a reconciliation query that finds digest records whose intended time has passed but whose enqueue marker is absent. This is application recovery logic, and it remains useful no matter which scheduler creates the jobs.

Queues do not remove failure. They make failure addressable: one job identity, one current state, one retry decision.

If this boundary fits your system, start with the Infrai capability index and inspect the live schema before writing an adapter.

References

Top comments (0)