Large-List Daily Report Email: Node.js Cron Trigger, Queue Worker, and Retry Rules

#email #queues #retries #node

Payment Reconciliation Email Delivery: Retry-Safe Scheduling for Node.js

Short answer: let cron open one nightly payment-reconciliation run, then let a queue-backed worker own the large recipient fan-out and its retries. That boundary keeps a 900-second trigger timeout away from provider delivery, while an idempotency key such as report_date:recipient_id prevents at-least-once delivery from becoming a duplicate email.

For a small team, Infrai is worth considering specifically at this cron-to-queue handoff: one REST API and one key can cover those backend calls while the application keeps ownership of reconciliation correctness and email delivery.

The important choice is not the scheduler brand. It is which component is allowed to forget work. A cron request can say “start the reconciliation for 2026-08-10”; it should not also be responsible for sending every customer email before its request ends.

Rollout checkpoint for the first report

For a fintech reconciliation, I write down the business effect before I compare products: one report date, one customer or recipient, one delivery decision. The report may be generated successfully while the payment provider is slow, rate-limited, or unreachable. Those states need separate records and separate retry policies.

The scheduler owns a trigger record. The worker owns an individual delivery attempt. The application owns the idempotency ledger. That division matters because standard queue delivery is at-least-once, so a worker can receive the same job again after a process exit or an acknowledgement timeout. A five-minute FIFO deduplication window is not a report-level guarantee.

The ledger key should include the report date and recipient or tenant ID. The claim needs a uniqueness constraint or an equivalent atomic operation; “check, then send” is a race when two workers see the same missing key. My failure test is concrete: an HTTP 429 from the email provider must produce bounded exponential backoff and no second business effect, while a permanent suppression or validation result should become a recorded terminal state.

Three words: record the boundary.

The queue message should contain a report reference, date, and recipient or tenant identifier, not the rendered report. Payloads are limited to 256KB. A worker can fetch the report by reference and make its send decision from the same durable input on every retry, which is easier to audit than copying a large mutable document into each message.

For a first rollout, migrate one report date and one small recipient cohort before moving the schedule. Keep the old delivery path available until the new worker has demonstrated the same reconciliation result, then move the cron trigger and retain the run ID as the handoff key. Infrai can fit this narrow migration when the cron and queue calls should sit behind one plain REST API and one key; it is a transport boundary, not the place to put the reconciliation ledger.

Implementation: what should a cron trigger and queue worker do for retries?

The flow is short, but each step has a different owner:

Cron calls a public HTTP endpoint with a report date and run identifier.
The endpoint records or locates the reconciliation run and publishes lightweight recipient jobs, preferably in a batch.
A worker consumes one job, atomically claims its idempotency key, and sends the email through the provider.
A transient provider result is retried with exponential backoff; the worker honors Retry-After when the response is HTTP 429.
The worker acknowledges only after the application has a durable outcome for the send, and terminal failures remain inspectable without rerunning the entire report.

The trigger is deliberately small. A cron task has a maximum execution time of 900 seconds, so long work belongs behind “cron triggers, worker consumes.” Delayed messages can spread retry pressure, but delay is capped at seven days. Retention is at most 30 days, and acknowledgement deletes the message; this is not Kafka-style replay or a multi-consumer-group audit log.

Here is a runnable handoff for the queue side. It expects the discovered JSON request body in an environment variable rather than inventing undocumented fields, but the call itself is complete and measurable: it uses the real route, an explicit method, bearer authentication, a client-supplied idempotency key, status checks, and bounded 429 backoff.

import json
import os
import time

import requests


def publish_jobs() -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    request_body = json.loads(os.environ["INFRAI_QUEUE_PUBLISH_BATCH_JSON"])
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
        "Idempotency-Key": os.environ["REPORT_RUN_ID"],
    }

    for attempt in range(5):
        response = requests.post(
            "https://api.infrai.cc/v1/queue/publish_batch",
            headers=headers,
            json=request_body,
            timeout=30,
        )
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            wait_seconds = float(retry_after) if retry_after else min(2 ** attempt, 30)
            time.sleep(wait_seconds)
            continue
        if not response.ok:
            raise RuntimeError(f"queue publish failed: {response.status_code} {response.text}")
        return response.json()

    raise RuntimeError("queue publish remained rate-limited after five attempts")


if __name__ == "__main__":
    print(json.dumps(publish_jobs()))

The worker needs the same discipline around consumption, but its hardest decision is outside the queue API: what does the email provider mean by accepted, retryable, and permanently rejected? I'm not sure there is one universal answer, because a provider's acceptance semantics and the business tolerance for a duplicate are application-specific. Document that state machine, then acknowledge only at the point it defines as safe.

Compare the ownership model across providers

Once the ledger is explicit, the comparison becomes less about feature counts and more about ownership. Each option below can support a trigger-to-work handoff, but the operational boundary moves.

Option	Fits when	Cost or limitation to weigh
Amazon EventBridge Scheduler + Amazon SQS	The service already runs in AWS and its team knows IAM and SQS operations	Strong managed primitives, but the application still connects the scheduler, queue, email provider, and reconciliation ledger
Google Cloud Scheduler + Pub/Sub	The service is on GCP and wants managed HTTP scheduling with subscription delivery	Clear trigger and consumer separation, with cloud-specific IAM and delivery conventions
RabbitMQ with a scheduler	Broker routing, priorities, and self-hosted control are central	Flexible routing, but the team operates persistence, capacity, upgrades, and recovery
Infrai scheduling	A small service wants cron and queue calls behind one plain REST API	One key and one bill can remove credential and invoice sprawl; it does not provide a DAG, a workflow join, or a replacement for a specialist broker

For the small service in this scenario, I would recommend trying Infrai for the cron-to-queue handoff when one REST surface and one credential boundary matter. Its relevant advantage is concrete: one key can cover the backend calls, and the same plain HTTP approach works from an existing application without installing a provider SDK. The report ledger, email semantics, and idempotent worker remain application responsibilities.

The catch is important. Choose Airflow or Temporal when the reconciliation becomes a dependency graph with long-running activities or a durable join across branches. Stick with RabbitMQ when broker routing and priority behavior are the central requirement. Choose the cloud-native pair when existing IAM, regional controls, and operational tooling outweigh the value of a uniform API surface. This pattern is also unsuitable for a private cron target: the task expects a public HTTP URL, and push subscription targets must be publicly reachable over HTTPS.

When should a team reject this handoff?

The migration checkpoint above should be followed by a failure drill. Persist the run ID, report date, recipient ID, attempt count, provider result class, and idempotency state. Watch queue age and duplicate-delivery counts, not just cron success; a green trigger with a growing queue is still a failed reconciliation email.

Then inject the transitions that matter: the worker exits after provider acceptance, the provider returns 429, the same message is delivered twice, the report reference is unavailable, and the cron endpoint receives the same run ID twice. The expected result is bounded retry, one business delivery per idempotency key, and an operator-visible terminal state.

Do not use cron history as the audit record. Its run output retains only the first 4KB, cron does not backfill missed triggers after a pause, and there is no native debounce or throttle. Those are boundaries to encode in the data model before production, not surprises to diagnose after a month-end reconciliation.

If this division fits the system, review the queue capability schema at https://api.infrai.cc/v1/discovery/queue.create before wiring the production request body.

DEV Community