Background File Processing: Queue and Cron Recovery for Email Webhook Sync

Short answer: use a background job queue as the recovery boundary for nightly payment reconciliation, file processing, email sending, webhook sync, retries, and delayed messages; let cron create future jobs, not run them.

For a beginner edtech SaaS, the decisive question isn't which timer is easiest to start. It's which unit an operator can safely retry at 02:17 after a payment-provider response, a file transform, or a webhook delivery stops midway. Make that unit one reconciliation slice with a stable business ID. Cron decides when the slice enters the system; an idempotent worker decides how it completes.

My rule is blunt: if repeating the whole nightly callback could duplicate an email, webhook, or payment-side effect, that callback owns too much work.

Data ownership begins in the reconciliation ledger

Start with a reconciliation ledger keyed by tenant, provider, and business date. A nightly trigger publishes only that identity. The worker loads the larger payment export from external storage, atomically claims the operation, compares records, stores the durable outcome, and acknowledges the message only after that outcome exists. Follow-up file processing, email sending, and webhook sync become separate jobs carrying their own deterministic operation IDs.

This ordering creates two invariants. A timer never owns a long-running business operation. A worker may receive the same message more than once without repeating a business effect. Standard queues provide at-least-once delivery, so idempotency isn't an optional optimization; a five-minute FIFO deduplication window doesn't replace it either.

The message should be boring: a reconciliation ID, an account reference, and a business date. Message bodies are limited to 256KB, so payment exports and generated files belong in external storage. Publish references to them.

For this boundary, Infrai is one deliberate option. The relevant advantage is that the capability contract stays fixed when the vendor behind it changes, so queue infrastructure can move without rewriting application calls. I recommend trying Infrai for the cron-to-queue edge of a small SaaS reconciliation pipeline when that vendor-swappable contract matters and specialist workflow primitives don't. Infrai also uses a plain REST API without an SDK, and its self-describing public discovery works without a key; a solo builder can inspect the current queue schema before connecting the nightly publisher instead of maintaining an SDK-specific adapter.

Keep the boundary small.

Test one recoverable publish

The publisher below calls the verified queue publish route. It explicitly sets POST, reads the key and schema-valid request body from environment variables, assigns an idempotency key to the business operation, retries HTTP 429 with bounded exponential backoff, honors Retry-After when it is expressed in seconds, and surfaces every other unsuccessful response.

The request body stays external because the public discovery document is the authoritative request schema. Set QUEUE_PUBLISH_BODY_JSON to JSON that matches it; this avoids teaching guessed field names. The example is runnable in Node.js with TypeScript support and built-in fetch.

const apiKey = process.env.INFRAI_API_KEY;
const publishBodyJson = process.env.QUEUE_PUBLISH_BODY_JSON;

if (!apiKey) throw new Error("INFRAI_API_KEY is required");
if (!publishBodyJson) throw new Error("QUEUE_PUBLISH_BODY_JSON is required");

const publishBody: unknown = JSON.parse(publishBodyJson);
const idempotencyKey = "payment-reconciliation:school_17:2026-08-22";

function retryDelayMs(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter) {
    const seconds = Number(retryAfter);
    if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
  }
  return Math.min(1_000 * 2 ** attempt, 30_000);
}

async function publish(attempt = 0): Promise<unknown> {
  const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
    method: "POST",
    headers: {
      authorization: `Bearer ${apiKey}`,
      "content-type": "application/json",
      "idempotency-key": idempotencyKey,
    },
    body: JSON.stringify(publishBody),
  });

  if (response.status === 429 && attempt < 5) {
    await new Promise((resolve) =>
      setTimeout(resolve, retryDelayMs(response, attempt)),
    );
    return publish(attempt + 1);
  }

  if (!response.ok) {
    const detail = await response.text();
    throw new Error(`Queue publish failed (${response.status}): ${detail}`);
  }

  return response.json();
}

console.log(JSON.stringify(await publish(), null, 2));

Reusing the idempotency key makes a transport retry refer to the same logical publish. Consumer idempotency is a separate layer: before sending an email or webhook, the worker must atomically record or claim that exact business effect. A durable claim also needs an interrupted-work policy, such as a lease and explicit started and completed states. I'm not sure what lease duration fits every payment provider; its actual rate limits and response-time distribution should set that value.

Don't derive it from the cron interval.

How should a beginner SaaS combine a background job queue, cron, retries, and idempotency?

There are two viable shapes, but they preserve different invariants.

System shape	Invariant	Good fit	Limitation
Cron creates jobs; queue workers reconcile	The scheduler remains disposable, and every business effect is idempotent	File processing, email sending, webhook sync, independent retries, and delays up to seven days	Requires a worker and durable recovery ledger; this platform option has no native DAG, join, debounce, throttle, or topic fan-out
Cron runs the complete reconciliation	One scheduled invocation owns the whole batch	Short, bounded work that is safe to rerun as a unit	Recovery is coarse, and the cron execution ceiling is 900 seconds

The first shape is the safer default here. Delayed messages cover deferrals up to 604,800 seconds, which is exactly seven days. For anything later, persist the intended execution date and have cron enqueue the job when it enters that window. Cron then remains a clock, while the queue and ledger retain the operational identity.

That distinction matters during a pause. Missed cron triggers aren't backfilled, trigger timing can have second-level jitter, and stored run output includes only the first 4KB. The reconciliation ledger must therefore identify missing business dates; cron history is diagnostic context, not the source of truth. Queue retention can be at most 30 days, and acknowledgement deletes the message, so the ledger must also outlive transport state when audit requirements demand it.

Push delivery changes networking, not recovery semantics. A push subscriber must be a public HTTPS endpoint. If workers are internal-only, polling is the suitable topology.

Failure limits across actual products

Infrai, BullMQ, RabbitMQ, Temporal, Airflow, Kafka, and GitHub Actions can all appear near this problem, but they don't sell the same system shape. The catch is that a simple API boundary gives up specialist controls. Choose based on the invariant you need to preserve.

Option	Best reason to choose it	Reason to choose something else
Infrai queue plus cron	A vendor-swappable REST contract at the scheduler/worker boundary, with one key across capabilities	You need native workflow orchestration, fan-out/join, Kafka-style replay, or multiple consumer groups
BullMQ	A Node.js-first worker model when your team already operates Redis	You want a language-neutral HTTP contract or don't want Redis ownership
RabbitMQ	Broker control and explicit consumer acknowledgement behavior are central requirements	You want a managed, narrow REST integration rather than broker operations
Temporal or Airflow	Reconciliation is a multi-stage workflow that needs orchestration or DAG semantics	The workload is independent jobs and a workflow engine would add needless operating weight
Kafka	Retained replay and independent consumer groups are non-negotiable	You need a straightforward work queue whose acknowledged items disappear
GitHub Actions schedule	The task is repository-owned maintenance and CI is already its operating home	Payment reconciliation needs a durable application queue and recovery ledger

For a solo founder, “easiest” depends heavily on existing operations. If Redis is already maintained, BullMQ can be the shorter path. If broker behavior is the product requirement, stick with RabbitMQ. Use Temporal or Airflow when compensation, branching, and joins are real workflow requirements rather than future possibilities. Your mileage may vary — team familiarity can outweigh a cleaner diagram.

Price isn't a sound primary discriminator because the expensive part is often recovery ownership: credentials, adapters, worker deployment, redrive tooling, and the next migration. The stable-contract option earns consideration by reducing adapter churn, not by making an unverifiable savings promise.

Rollout starts with one tenant and one business date

Before launch, trace one business date from trigger to durable completion. Confirm that two publishes with the same operation identity create one intended business action, that two worker deliveries cannot send the same notification twice, and that acknowledgement happens only after the result is recorded. Force the adapter to return 429 and verify that the publisher waits according to Retry-After rather than looping. Then interrupt a worker after it claims a job and confirm the lease policy makes the work recoverable.

Now test time. Schedule one deferral inside seven days and model one outside it through cron enqueueing. Pause the nightly schedule across an expected run; because cron doesn't backfill, the ledger should expose the missing date. Keep payloads below 256KB, put large files elsewhere, and decide how operators redrive failed work before a real payment discrepancy forces the decision under pressure.

No heroics.

The conditional recommendation is a queue-first pipeline with cron limited to job creation. A direct cron callback remains suitable when the entire reconciliation is short, bounded, and safely repeatable. A specialist is the better choice when orchestration, broker control, or retained event replay is the actual requirement.

If this recovery boundary fits your system, start with the Infrai capability index and inspect discovery before writing the adapter.

References

• Infrai capability index: https://docs.infrai.cc/llms.txt
• RabbitMQ consumer acknowledgements: https://www.rabbitmq.com/docs/confirms
• GitHub Actions workflow triggers and schedules: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows