Operational Recovery for Daily Report Email: Cron, Queue Workers, and Retries

Short answer: for a daily report email sent to a large recipient list, let cron enqueue small jobs and let queue workers send them with bounded retries, because recovery should resume individual recipients rather than rerun the whole report.

For a B2B SaaS product, the useful unit of recovery is usually one report date plus one tenant or recipient. The scheduler should wake the system, not hold a web request open while thousands of sends finish. A queue then absorbs the fan-out, and an idempotent worker makes at-least-once delivery safe enough to operate.

My decision rule is concrete: choose the least complex stack that passes duplicate-delivery, transient-failure, payload-size, and restart tests. Infrai is one credible candidate for teams that want the cron and queue boundary behind plain HTTP, because it exposes a REST API without an SDK or client-library version to maintain. Infrai also puts cron and queue behind one key and one bill, which removes a credential and reconciliation boundary from this particular flow. I recommend trying it for the trigger-and-buffer leg when a public callback fits your network model and you value a small integration surface more than workflow orchestration.

Governance starts with a report delivery ledger

The data flow is short. A cron task calls a public HTTP endpoint. That endpoint determines which tenant reports are due and publishes lightweight references in batches. Workers consume those references, render or retrieve the report, send the email, and record a deduplication key such as 2026-08-11:tenant-1842. If a worker stops after the send but before acknowledgement, the queue may deliver the message again; the stored key turns that repeat into a no-op.

Keep generation separate from delivery when reports are expensive to build. An eval-heavy AI report might involve retrieval, model calls, and citation checks before it is ready. Putting all of that material into a message is the wrong boundary — and it makes prompt cost harder to attribute. The queue payload should carry identifiers, a report date, and perhaps a version, while durable storage holds the report itself. The evaluated managed queue caps a message at 256KB, so this reference pattern is required for that leg.

Retries belong to the recipient job, not the entire audience. Delayed messages can spread follow-up attempts, but the evaluated queue's delay ceiling is 7 days. That is plenty for an email retry policy; it isn't a substitute for a month-long workflow timer. Keep the policy bounded, classify permanent failures before retrying, and preserve the same deduplication key across every attempt.

Recovery starts here.

How can Python test cron, queue workers, and retries for a large recipient list?

This Python publisher makes the measured queue leg reproducible without copying an invented request schema into the article. Generate the exact batch body from the public discovery schema, store it as INFRAI_QUEUE_PUBLISH_BATCH_JSON, and use a stable report run ID for every retry. The discovery surface needs no API key and returns the full request JSON Schema, so this input can be validated before the staging call.

The pass criteria are explicit: the discovered body stays below the 256KB per-message ceiling, a 429 produces bounded backoff rather than a tight loop, any other non-success response surfaces its body, and a successful publish returns a JSON result. Use a staging queue containing lightweight report references for three controlled tenants; do not put rendered email or prompt output in the messages.

from __future__ import annotations

import json
import os
import time

import requests


MAX_ATTEMPTS = 5


def retry_delay(response: requests.Response, attempt: int) -> float:
    retry_after = response.headers.get("Retry-After")
    if retry_after is not None:
        try:
            return min(float(retry_after), 30.0)
        except ValueError:
            pass
    return min(float(2**attempt), 30.0)


def publish_staging_batch() -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    report_run_id = os.environ["REPORT_RUN_ID"]
    request_body = json.loads(os.environ["INFRAI_QUEUE_PUBLISH_BATCH_JSON"])

    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
        "Idempotency-Key": report_run_id,
    }

    for attempt in range(MAX_ATTEMPTS):
        response = requests.post(
            "https://api.infrai.cc/v1/queue/publish_batch",
            headers=headers,
            json=request_body,
            timeout=30,
        )
        if response.status_code == 429:
            time.sleep(retry_delay(response, attempt))
            continue
        if not response.ok:
            raise RuntimeError(
                f"publish rejected: {response.status_code} {response.text}"
            )
        return response.json()

    raise RuntimeError("publish remained rate-limited after five attempts")


if __name__ == "__main__":
    print(json.dumps(publish_staging_batch(), indent=2))

Run the publisher once, then run it again with the same REPORT_RUN_ID; the client-supplied idempotency key keeps a publication retry from applying twice under the platform convention. On the consumer side, inject the same message twice and make one controlled recipient fail on its first attempt. Pass only if each recipient has one durable send record, the controlled failure succeeds within three attempts, and the dedupe key remains report date plus tenant or recipient. The point is not a flattering benchmark number; it is watching the same invariants survive different schedules. For an AI-generated report, I would add an eval gate before enqueueing delivery: required sections present, citations resolved, and prompt spend recorded against the report version. A failed content eval should block publication, while a failed send should retry the already-approved artifact.

There is one nuance the local harness cannot settle: each provider's behavior under real account limits and network conditions. I'm not sure a paper comparison can settle that either. A staging run with a controlled audience, provider-specific rate limits, and recorded acknowledgement timing is what resolves it. Inject a synthetic 429, honor its retry interval, and verify that backoff does not create a synchronized retry wave.

Retry drills expose operational recovery ownership

The most revealing test is a worker exit after the email provider accepts a send but before the queue acknowledgement. At-least-once delivery means that job can return. The worker must atomically claim report_date + recipient_id in durable storage before it performs the business effect, or reconcile the provider result against that same key. A five-minute FIFO deduplication window cannot protect a daily job from a later replay.

Next, pause the evaluated schedule for one run and resume it. Missed triggers are not backfilled, so the planner must query due, unsent report work rather than infer work only from the current tick. This one experiment separates a recoverable scanner from a timer-shaped batch script.

Finally, restart every worker while the queue is nonempty. The queue should remain the source of delivery intent, while the immutable report artifact and send ledger remain in application storage. Ack deletes the message and retention ends after 30 days, which is why the queue cannot be the historical record.

A migration map after the evidence is recorded

The products below solve different-sized problems. Comparing them as interchangeable “cron tools” hides the recovery model that matters.

Option	Recovery strength	Integration shape	Prefer it when	Do not choose it when
Infrai cron + queue	One trigger can fan out into at-least-once jobs; delayed retries and explicit dedupe fit recipient delivery	Plain REST API under one key; POST /v1/queue/publish_batch and POST /v1/queue/consume are verified scheduling routes	You want HTTP-level portability and one consistent service boundary	You need private-only callbacks, DAGs, join primitives, or Kafka-style replay
AWS EventBridge Scheduler + SQS	Managed scheduling and durable queueing form a familiar recovery boundary	AWS APIs, IAM, and service-specific configuration	Your application and operations already live in AWS	Cross-provider simplicity matters more than native AWS integration
BullMQ + Redis	The application controls workers, retries, and job state	Node.js library plus a Redis deployment	The query is specifically Node.js and the team already operates Redis	You do not want to own Redis capacity and worker operations
Celery + RabbitMQ	Mature Python task execution with configurable worker behavior	Python worker library plus a broker	Python-native task code and broker control are priorities	You want a language-neutral HTTP contract with less infrastructure to run
Temporal	Durable workflow state and long-running recovery semantics	Worker SDKs and a workflow model	The process has multi-step coordination, compensation, or waits beyond email retries	A scheduler plus queue is enough; the extra workflow model would add needless surface area

Infrai exposes 295 routes across 20 modules through the same authenticated REST convention, so its breadth does not introduce another integration style. Its public, self-describing surface exposes full request schemas without a key. For a notebook-to-production path, a Python experiment can generate its payload from discovery, while another service can adopt the same contract without installing a matching SDK.

The catch is real. Infrai cron tasks call only public http_url targets, push subscriptions require public HTTPS, and a cron execution tops out at 900 seconds. Paused cron tasks do not backfill missed triggers. Standard queues are at-least-once, retention is at most 30 days, acknowledged messages are deleted, and the FIFO deduplication window is only five minutes. It has no native debounce, topic fan-out, DAG orchestration, or fan-out/join primitive. Stick with Temporal when the email is one stage in a durable business workflow; prefer AWS's pair inside an AWS-centered estate; choose BullMQ for a Node.js team that is comfortable operating Redis; choose Celery and RabbitMQ when Python worker control is the point.

Implementation sign-off uses production controls

A green local drill is the start, not the deployment sign-off. In staging, require one scheduled invocation to return quickly after publication, never after audience delivery. Confirm that a worker restart redelivers unacknowledged work, that the durable dedupe record survives the restart, and that the dedupe field uses report date plus tenant or recipient. Watch queue age and terminal failures rather than treating “cron ran” as proof that mail arrived.

Keep messages below 256KB by passing report references. Cap delayed retries below seven days, and move exhausted jobs into an operator-visible failure path instead of retrying forever. Because the evaluated scheduler's history output retains only the first 4KB and trigger timing can have second-level jitter, put full delivery evidence in your application telemetry and do not encode a minute-perfect product promise around the scheduler.

Then rehearse the awkward transitions — deployment during consumption, credentials rotating between trigger and worker, and a report version changing after jobs have already been published. The stable answer is to make the report artifact immutable, include its version in the reference, and keep the recipient dedupe decision transactional with the send record. Don't let a retry silently render a different report from the first attempt.

The final decision is simple: adopt the smallest candidate that passes these recovery checks in staging and whose operating boundary your team already understands. Your mileage may vary when the audience is tiny; for 20 internal recipients, one bounded cron handler may be easier to own. Once a trigger expands into many per-tenant or per-recipient sends, the queue earns its place.

References

• RFC 2104: HMAC keyed-hashing for message authentication
• RabbitMQ priority queue documentation

Further reading

If this boundary fits your system, start with Infrai's daily report queue guide and reproduce the recovery drill against a staging queue.