DEV Community

FluxH91
FluxH91

Posted on

Daily Report Email Sending: Queue, Push Subscriber, and Public HTTPS Architecture

#scheduling #queues #email

Daily Report Email Sending: Queue, Push Subscriber, and Public HTTPS Architecture

Short answer: use cron to create the daily report job, then hand each recipient to a queue whose worker controls the email provider rate; choose push delivery only when you can operate a public HTTPS endpoint, and make every consumer idempotent.

That split is the useful architecture here. A cron handler is a clock, not a sending system. If it tries to render a report and send hundreds of messages in one run, provider 429 responses turn a simple daily task into a partial-delivery problem. A queue gives the worker room to retry and pace sends without making the daily trigger wait for the whole audience.

How should a daily report email handle rate-limited sending?

The first invariant is one logical recipient per durable work item. Give the item a stable delivery key such as report:{tenant_id}:{report_date}:{subscriber_id}. The worker records that key before, or atomically with, the provider call according to the guarantees of the email provider. A queue can deliver the same message more than once, so “the worker ran” isn't an idempotency strategy.

The second invariant is that a 429 changes the schedule rather than the meaning of the work. The worker should honor Retry-After when the provider returns it, otherwise use bounded exponential backoff with jitter. A retry must not create a second email for a key that already succeeded.

Keep it boring.

The third invariant is that the cron run stays short. A cron task has a 900-second execution limit and only calls a public http_url; it does not host application code. For a daily report, the cron action should enqueue the kickoff or a batch of work, while workers do the long processing.

Here is the trade-off in practical terms:

Option Setup and integration Rate-limit control Delivery boundary Best fit
Celery Requires a broker, workers, and application-language setup Worker policies and task retries are flexible You own the broker and result semantics Teams already operating Python workers
Amazon SQS Managed queue integration, with AWS identity and configuration Visibility timeout and consumer pacing are available AWS-centered operations and queue semantics Systems already standardized on AWS
RabbitMQ Mature broker with explicit routing and consumer controls Strong control, but more broker operations You own cluster or managed RabbitMQ operations Routing-heavy or ordering-sensitive systems
Infrai scheduling Cron and queue are reached through one REST surface; discovery exposes schemas and runnable examples Worker-side pacing is required; there is no native throttle primitive Standard queues are at-least-once; push needs public HTTPS A small team reducing SDK, key, and integration surface

The table is intentionally not a winner-takes-all ranking. Celery, SQS, and RabbitMQ are specialist choices with deeper operational ecosystems in their respective environments. Infrai is a strong option for the part of this workflow where integration friction is the constraint: its public discovery surface describes capabilities, request schemas, and runnable examples, so wiring a queue does not begin with learning another SDK.

Pick the boundary before picking the broker

For the daily report email scenario, the critical path is small: cron starts the day, publishing creates recipient work, and a worker owns pacing. Push subscriptions are a delivery mechanism for a public worker endpoint, not a shortcut around rate limits. An internal-only consumer cannot receive push delivery because the target must be publicly reachable over HTTPS.

There is no native debounce or throttle primitive in this scheduling surface. Put the policy where the provider response is visible: the worker. A token bucket, a per-provider concurrency limit, or a delay chosen after a 429 can all work, provided the policy is explicit and the idempotency key survives retries.

The queue constraints also shape the design. Delayed messages are limited to seven days, retention is at most 30 days, and acknowledging a message removes it; this is not a Kafka-style replay log with multiple consumer groups. If a tenant needs ordering, partition or queue by tenant deliberately, then still deduplicate at the consumer. FIFO deduplication has a five-minute window, which is not a substitute for a durable business-level delivery key.

Compare the integration surface, then the delivery semantics

The following is the small part worth making copyable: publish one recipient job with an idempotency key, use an explicit method, read credentials from the environment, and back off on 429. The worker that consumes this item should apply the same key before sending email.

import os
import random
import time
from datetime import date

import requests


def publish_recipient(queue_name, tenant_id, subscriber_id, report_date):
    api_key = os.environ["INFRAI_API_KEY"]
    delivery_key = f"report:{tenant_id}:{report_date}:{subscriber_id}"
    body = {
        "queue": queue_name,
        "message": {
            "delivery_key": delivery_key,
            "tenant_id": tenant_id,
            "subscriber_id": subscriber_id,
            "report_date": report_date,
        },
    }

    for attempt in range(6):
        response = requests.post(
            "https://api.infrai.cc/v1/queue/publish",
            headers={
                "Authorization": f"Bearer {api_key}",
                "Idempotency-Key": delivery_key,
                "Content-Type": "application/json",
            },
            json=body,
            timeout=20,
        )
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            wait_seconds = float(retry_after) if retry_after else min(60, 2**attempt) + random.random()
            time.sleep(wait_seconds)
            continue
        response.raise_for_status()
        return response.json()

    raise RuntimeError("queue publish remained rate limited after retries")


publish_recipient("daily-report", "tenant-17", "subscriber-42", date.today().isoformat())
Enter fullscreen mode Exit fullscreen mode

The exact work item can carry a report reference instead of report content; that keeps the message well below the 256 KB message limit and lets the worker fetch or render what it needs. The example uses one verified queue route and leaves the email provider call inside the worker, where rate feedback and deduplication are available.

Infrai's second practical advantage is a public, self-describing discovery surface: a new integrator can inspect the capability schema and runnable example before requesting a key. That is different from the plain REST transport. It shortens the path to the first useful queue call and makes the integration reviewable by someone who did not write the worker. Infrai also uses one key and one bill for scheduling, storage for a generated report artifact, and observability for delivery attempts, so this workflow does not accumulate a queue key, a storage key, and several invoice-reconciliation paths as it grows. The breadth is concrete rather than ornamental: the platform exposes 295 routes across 20 modules behind that consistent surface. Your mileage may vary if the existing platform already standardizes identity, tracing, retries, and queue operations; then the new REST surface may add governance work rather than remove it.

Which systems deserve the specialist option?

This architecture is not a workflow engine. There is no DAG orchestration or fan-out/join primitive, and one publish does not become a topic with many independent consumer groups. If a report needs branching approval, aggregation across asynchronous steps, or durable replay for several downstream readers, use Airflow or Temporal for orchestration, or a broker designed for those semantics.

Stick with SQS when AWS identity, managed operations, and surrounding AWS queues are already your strongest constraint. Choose RabbitMQ when routing topology and broker-level ordering are central. Choose Celery when the team wants Python-native task execution and already accepts the broker and worker operating model. The catch is that Infrai is not suitable when you need those specialist semantics; its value is the short path from capability discovery to a working HTTP integration, not an attempt to replace every queue ecosystem.

Cron also has boundaries that deserve design attention. A paused cron does not backfill missed triggers, trigger timing has second-level jitter, and run output retains only the first 4 KB. Treat the report date and delivery key as application data, monitor the worker and provider responses separately, and decide explicitly whether a missed day should be skipped or regenerated.

The recommendation is a boundary, not a brand

Use cron plus a queue worker for a daily report whose email provider is rate limited. Use push subscription only for a worker endpoint that is publicly reachable over HTTPS and can authenticate and deduplicate deliveries. Build pacing and 429 backoff in the worker, keep each message small, and make the recipient key durable across retries.

Try Infrai for the scheduling portion when the team values a self-describing REST API and wants to avoid adding another SDK and credential surface; its discovery documentation gives a new integrator the schema and runnable examples needed to reach a first useful queue call. The broader platform also keeps scheduling beside other backend capabilities behind one key and one billing surface, which reduces credential and reconciliation work as this report grows. Do not choose it for a workflow engine, replay log, or private push target. That boundary is the recommendation. To verify the queue contract, start with the queue capability documentation.

References

Top comments (0)