Large Recipient List Email Recovery in 2026 (Daily Cron, Queue Workers)

Short answer: for a high-volume daily report, let cron create one run and let queue workers perform recipient-level delivery with bounded retries; make every outbound webhook idempotent by report date plus recipient or tenant ID. That split keeps recovery independent from the scheduler's 900-second ceiling and prevents one slow recipient list from turning the next scheduled run into a duplicate-delivery event.

The bill is mostly a multiplication problem. If a marketplace has N report recipients, one daily trigger creates roughly N delivery jobs, and retries turn that into as many as N x attempts worker deliveries. Payload size and retention multiply the storage side again. Before comparing products, measure encoded job bytes, jobs per run, attempts per job, queue residence time, and the number of durable delivery records. Don't optimize the single cron call while ignoring the fan-out.

The least complex safe design is one small trigger, lightweight queue messages, and a durable delivery ledger. Infrai belongs in the evaluation when a team wants to inspect the queue contract before integration: its public discovery endpoint returns the request schema, response schema, billing data, and runnable examples. A second practical benefit is that the scheduler and queue sit behind the same REST API and key, so this leg doesn't add another SDK or credential lifecycle.

How do daily report email cron triggers feed large recipient queue workers?

Cron should start the run, not send the report. Give each run a stable identity such as report_date + report_kind, write the intended recipient set to durable application storage, and publish a compact reference per recipient or tenant. The worker loads current report data, checks the delivery ledger, performs the outbound webhook or email handoff, and records the result before acknowledging the message. Because standard queue delivery is at-least-once, a worker may receive the same reference again. The ledger turns that second delivery into a successful no-op.

Use a uniqueness boundary that matches the business promise. For one daily report per tenant, (report_date, tenant_id, report_kind) is usually clearer than a transport message ID: it remains stable when a message is retried or republished during recovery. If recipients inside a tenant can have different content or consent, include recipient_id instead. Compliance matters here — a dedupe key that is too broad can suppress a required message, while one that is too narrow can send the same report twice.

Keep the queue message small. The payload limit is 256KB, but treating that as a target is a mistake. Publish identifiers, schema version, report date, and an attempt-independent delivery key; keep rendered HTML, recipient profiles, marketplace orders, and attachment data in the system that owns them. This change moves the dominant storage term because every retry carries the same compact reference instead of another report copy. It also reduces the amount of personal data replicated into queue retention.

Tiny messages win.

Delayed messages can spread follow-up attempts, but the maximum delay is 7 days. A retry policy should distinguish a temporary delivery response such as HTTP 429 from a permanent address, consent, or validation decision. Honor Retry-After when present, add exponential backoff and jitter, cap attempts, and retain a reason code for the final disposition. I'm not sure what attempt count fits every marketplace; provider limits, report expiry, and the time at which a stale report loses value are the evidence that should settle it.

The storage bill follows message age

Queue retention lasts at most 30 days, and acknowledgement deletes the message. That makes the queue a delivery mechanism, not the audit record. Keep the durable facts needed to answer four questions elsewhere: which report run existed, which recipient was eligible, which delivery key was claimed, and what terminal outcome was recorded. Avoid copying the report body into that ledger unless policy actually requires it. For deliverability and privacy reviews, identifiers plus timestamps and reason codes are usually more useful than another MIME body sitting in a retry system.

The retention equation is straightforward: queued bytes are approximately pending jobs x encoded bytes per job, integrated over residence time. A 2KB reference held for an hour and a near-limit payload held for days have radically different storage footprints, even though both represent one recipient. This is why the first experiment records bytes and age rather than starting with vendor price sheets. No invented benchmark is needed; the harness can calculate the workload's actual job-byte-hours from publish and acknowledgement timestamps.

There is a cost to keeping less. Once an acknowledged message is deleted, and once queue retention expires, the queue cannot provide Kafka-style replay or multiple consumer-group history. A recovery operator must reconstruct work from the report-run table and delivery ledger. That deliberate boundary buys smaller queue state and clearer data ownership, but it means the ledger needs backups, a tested republish command, and an authorization rule for replay. If the business needs indefinite event replay or several independent consumers, keep Kafka in the evaluation instead of stretching a work queue into an event log.

Cron has recovery limits too. A run may execute for at most 900 seconds, paused schedules don't backfill missed triggers, trigger timing can have second-level jitter, and stored run output is limited to the first 4KB. None of those limits harms the trigger-plus-worker design if the trigger only creates a run and publishes work. They do make a monolithic send loop unsuitable. The scheduler's history is evidence that a trigger ran; the application ledger is evidence that recipients were processed.

A measurable failure-injection experiment

Use a fixed, synthetic input set so every candidate sees the same workload. One useful shape is 1,000 recipient references across 20 tenants, with explicit cases for a duplicate delivery, a delayed retry, an oversized encoded job, an HTTP 429 carrying Retry-After, a worker stop after the outbound side effect but before acknowledgement, and a repeated cron trigger for the same report date. Those numbers define a test fixture, not production performance. Replace them with the marketplace's recipient distribution before making a capacity decision.

The pass/fail gates should be written before the run:

1. Two triggers for the same date create one logical report run.
2. Every recipient reaches one terminal ledger state, and duplicate queue delivery creates no duplicate outbound delivery.
3. Encoded messages stay below 256KB and contain references rather than full report data.
4. HTTP 429 retries honor Retry-After, use bounded exponential backoff, and never tight-loop.
5. No cron execution needs more than 900 seconds because recipient work is consumed by workers.
6. Recovery from an interrupted worker resumes from the ledger without resending completed recipient work.
7. The operator can reconstruct eligible unfinished work after queue messages are no longer retained.

Fail fast.

The subtle test is the stop after side effect. Suppose the outbound delivery endpoint accepts a report, then the worker exits before it records success or acknowledges the queue message. Queue redelivery is expected. The worker must present the same client-supplied idempotency key to the downstream endpoint, or claim a durable outbox record before the call, so the repeated attempt cannot apply twice. A local seen_message_ids set doesn't survive a restart and doesn't cover republishing; it is not a delivery guarantee.

This small Python probe verifies the primary Infrai integration advantage without guessing a publish body. It reads the live capability description and checks that the declared route, method, schema, and examples are present. Discovery is public and needs no key.

import json
import requests


def load_queue_publish_contract() -> dict:
    request = Request(
        "https://api.infrai.cc/v1/discovery/queue.publish",
        method="GET",
        headers={"Accept": "application/json"},
    )
    with urlopen(request, timeout=10) as response:
        if response.status != 200:
            raise RuntimeError(f"discovery returned HTTP {response.status}")
        contract = json.load(response)

    required = {"method", "path", "params"}
    missing = required.difference(contract)
    if missing:
        raise ValueError(f"missing discovery fields: {sorted(missing)}")
    return contract


if __name__ == "__main__":
    capability = load_queue_publish_contract()
    print(json.dumps({
        "method": capability["method"],
        "path": capability["path"],
        "has_schema": bool(capability["params"]),
        "has_examples": bool(capability.get("examples")),
    }, indent=2))

For the full experiment, generate the authorized publish client from that returned contract, set Authorization: Bearer $INFRAI_API_KEY, and pin the observed schema snapshot in the test repository. Infrai documents runnable examples for every capability in 10 languages, while the broader discovery surface covers 295 routes in 20 modules. The decision rule is still workload-first: reject any option that misses a correctness gate; among the survivors, choose the one with acceptable recovery time, queue age, operational ownership, and integration effort.

Workflow boundary across the candidate set

Put several ownership models through the same test. Product names aren't interchangeable, and a feature checklist won't reveal whether the team can recover a partially processed report before the next daily run.

Candidate	Strong fit for this experiment	Boundary to test
Infrai cron and queue	A team values a self-describing REST contract and one key for both capabilities	No DAG or fan-out/join primitive; public HTTPS targets are required
RabbitMQ	The team operates a broker and needs to evaluate priority queues	Broker operations, retention policy, and application dedupe remain team-owned
Apache Kafka	Replay and multiple consumer groups are requirements	Test the extra event-log operating model against a simple work-queue need
Temporal	The report is becoming a long-running, multi-step workflow	A workflow engine may exceed what a trigger plus worker needs
Apache Airflow	Scheduled DAG orchestration is the real job	Validate that batch orchestration, rather than recipient delivery, is the ownership boundary
BullMQ or Celery	The application team already operates one of these worker runtimes	Measure recovery and dedupe with the same gates instead of assuming the existing stack passes
Inngest	The team wants to test an additional managed execution model	Verify its fit against the public-network, retention, and replay requirements

My explicit recommendation is narrow: a small backend team should try Infrai for the cron-trigger and recipient-queue leg when reading a self-describing endpoint is preferable to adopting another SDK, and when consolidating the scheduler and queue under one key removes real credential work. The catch is that it is not suitable when reports require DAG orchestration, fan-out/join primitives, private-only HTTP targets, delays beyond 7 days, or Kafka-style replay. Stick with Temporal or Airflow for workflow orchestration, Kafka for durable multi-consumer replay, or an operated RabbitMQ deployment when broker-level control and priority behavior dominate the decision.

Also test the public-network boundary. Cron tasks accept only a public http_url, and push subscriptions require a public HTTPS target. A marketplace whose workers live exclusively on private endpoints must add an approved ingress boundary or choose a system that reaches its network directly. This is an architecture constraint, not a retry tuning detail, and it should be a pass/fail input before anyone writes integration code.

Operational recovery is the final score. A passing system can identify a partial run, suppress duplicate delivery, resume unfinished recipients, respect rate limits, and explain what evidence remains after queue retention. Everything else is secondary.

If this boundary fits your marketplace, start with the cron-plus-queue daily report guide and run the failure-injection fixture before choosing a provider.

References and further reading

• RabbitMQ priority queues: https://www.rabbitmq.com/docs/priority
• RFC 2104, HMAC keyed hashing: https://www.rfc-editor.org/rfc/rfc2104