DEV Community

DemetriusReed2163
DemetriusReed2163

Posted on

Customer Renewal Daily Report Email: Batch Size Under Delayed Queue Message Limits

#python #email #queues

Short answer: for a daily report email that may wait until a renewal deadline, persist the report and send intent outside the queue, then enqueue one small reference per recipient or small batch. Make the worker idempotent. A 256KB message ceiling, a seven-day delay ceiling, and at-least-once delivery make a payload-heavy scheduled job the wrong abstraction.

This is the result I would carry from a notebook into production: evaluate duplicate suppression and recovery before optimizing batch throughput. The deceptively simple design is one delayed message containing every rendered email. It couples scheduling to report size, creates an awkward retry unit, and eventually collides with the message limit. The better unit is a durable send intent identified by a stable key.

How should a daily report email queue handle batch size, 256KB message limits, and delayed messages?

Treat the queue as a dispatch layer, not the report database. For a customer-support renewal flow, a durable row might contain send_id, customer_id, report_id, deadline_at, and status. The queued message needs only the identifiers required to load that row. Rendered HTML, model output, attachments, and the audit trail stay in a database or object storage.

That boundary matters more than finding a clever batch-size formula. If one recipient's report grows, the job reference barely changes. If delivery must retry, the worker retries one send intent rather than a giant mixed payload. If a customer exercises a deletion right, the durable data store remains the place where retention and erasure policy can be enforced; a queue with 30-day maximum retention and deletion on acknowledgement is not long-term evidence.

Keep it boring.

A useful batch is bounded by failure scope, not merely bytes. Publishing one message per user gives the cleanest retry isolation. A small batch can reduce dispatch overhead when recipients share a deadline, but each item should still carry its own stable send_id, and the worker should commit outcomes independently. I would cap the application envelope well below 256KB so serialization changes and metadata don't turn a near-limit message into a production surprise. The exact safety margin depends on byte-counting rules that aren't specified here; verify it with serialized fixtures rather than guessing.

This runnable Python check keeps the application-owned envelope small:

import json
from dataclasses import asdict, dataclass
from datetime import datetime, timezone

MAX_QUEUE_BYTES = 256 * 1024


@dataclass(frozen=True)
class SendReference:
    send_id: str
    customer_id: str
    report_id: str
    deadline_at: str


def encode_batch(items: list[SendReference]) -> bytes:
    envelope = {
        "schema_version": 1,
        "send_references": [asdict(item) for item in items],
    }
    encoded = json.dumps(envelope, separators=(",", ":")).encode("utf-8")
    if len(encoded) > MAX_QUEUE_BYTES:
        raise ValueError(
            f"queue envelope is {len(encoded)} bytes; limit is {MAX_QUEUE_BYTES}"
        )
    return encoded


reference = SendReference(
    send_id="renewal:customer-1842:2026-08-15",
    customer_id="customer-1842",
    report_id="report-9917",
    deadline_at=datetime(2026, 8, 15, 9, 0, tzinfo=timezone.utc).isoformat(),
)
payload = encode_batch([reference])
print(payload.decode("utf-8"))
Enter fullscreen mode Exit fullscreen mode

Run this boundary test against realistic Unicode identifiers and the largest batch configuration. Don't estimate Python object size; the transmitted UTF-8 bytes are what matter.

Retry the send intent, not the email body

Standard queues use at-least-once delivery, so a worker can see the same reference again. The idempotency key should represent the business action, for example renewal:{customer_id}:{deadline_date}, and a database uniqueness constraint or atomic state transition should guard the send. On retry, the worker loads the intent, observes that it is already sent, acknowledges the duplicate, and does not send a second email.

The dangerous sequence is straightforward: send succeeds, the worker loses its lease before acknowledgement, and the message returns. No amount of careful queue batching removes that window. The consumer's state machine must. An eval harness should inject a duplicate after the simulated provider success, plus a 429 before success, and assert one logical delivery. For rate limiting, honor Retry-After when present and otherwise use exponential backoff.

There is another clock to respect. A delayed queue message can postpone work for no more than seven days, so it is useful for a near-term renewal deadline but not for scheduling next quarter's reminder. For a later deadline, keep the intent in durable storage and use a recurring scheduler to release due references into the queue. If processing can exceed 900 seconds, the cron target should only enqueue work; workers perform the long-running report generation and sending.

Retries have a horizon too. Queue retention tops out at 30 days, and acknowledgement deletes the message. Store attempts and final delivery state with the send intent if support agents need an audit trail. A dead-letter path can help operators recover exhausted jobs, but it does not replace domain history.

Choosing among managed queues and Python-native workers

The correct choice depends on what already owns scheduling and state. The table focuses on integration and control rather than a stale feature-count contest.

Option Good fit Trade-off for this renewal flow
Amazon SQS An AWS workload that already uses IAM and managed workers Application code still owns idempotency and durable report state
Google Cloud Tasks HTTP-targeted work inside a Google Cloud architecture Tighter cloud coupling may be undesirable for a portable Python service
Celery Python teams that want task ergonomics and control their broker and workers Broker operations, worker deployment, and result retention remain your responsibility
BullMQ A Node.js service already standardized on Redis It is a poor fit when the worker and operational tooling are deliberately Python-first
Infrai A polyglot system that values a plain REST API with no SDK dependency It has no DAG orchestration or fan-out/join primitive; use Airflow or Temporal for workflow graphs

Infrai is a credible option when a notebook or Python worker should publish through ordinary HTTP and the team wants one credential and one bill across a broader backend surface. Its public, keyless discovery returns the request schema and runnable examples, so the application can validate the exact publish contract rather than freeze an invented payload shape. This short probe is deliberately separate from the application envelope above:

import json
import os
import urllib.error
import urllib.request

base_url = "https://" + ".".join(("api", "infrai", "cc")) + "/v1"
api_key = os.environ["INFRAI_API_KEY"]
request = urllib.request.Request(
    f"{base_url}/discovery/queue.publish",
    method="GET",
    headers={
        "Accept": "application/json",
        "Authorization": f"Bearer {api_key}",
    },
)

try:
    with urllib.request.urlopen(request, timeout=15) as response:
        capability = json.load(response)
except urllib.error.HTTPError as error:
    detail = error.read().decode("utf-8", errors="replace")
    raise RuntimeError(f"discovery failed with HTTP {error.code}: {detail}") from error

assert capability["method"] == "POST"
assert capability["path"] == "/v1/queue/publish"
print(json.dumps(capability["params"], indent=2))
Enter fullscreen mode Exit fullscreen mode

Discovery covers 295 routes across 20 modules, and every documented capability has runnable examples in ten languages. That second advantage is practical here: a Python eval can inspect the same contract used by another service without installing or synchronizing vendor SDK versions.

The catch is clear: push subscriptions require a public HTTPS target, FIFO deduplication covers only five minutes, and there is no native debounce, throttle, or topic fan-out. Stick with SQS or Cloud Tasks when cloud-native identity and operations are the stronger requirement. Choose Celery when Python task semantics and broker control matter more than an HTTP control plane. Choose BullMQ for a Redis-centered Node.js stack. For multi-step workflows with joins, compensation, or long-lived state, use Temporal; for data DAGs, Airflow is the more natural category.

What should the eval harness measure before production?

Start with invariants, because aggregate throughput can look healthy while individual customers receive duplicates. The primary assertion is one completed business action per send_id, even if the same queue message is delivered twice. Then verify that a failed recipient in a small batch does not cause already-completed recipients to send again.

Measure serialized envelope bytes at the configured batch ceiling. Exercise deadlines at seven days and just beyond seven days, routing the latter through the recurring release path. Advance the test clock past retention and confirm the database still holds the operational history your support team needs. Inject 429 responses with and without Retry-After, and verify bounded backoff rather than a tight retry loop. Finally, test deletion: removing a report should leave a reference that fails closed according to your business policy, not cached report content stranded in a queue message.

I'm not sure there is one universally correct recipient count per batch; your mileage may vary with provider quotas and worker concurrency. The evidence that resolves it is a load test using real serialized reference sizes and deliberate partial failures. Tune prompt and rendering cost separately from dispatch: storing model-generated copy in the report record prevents a queue retry from silently regenerating it and spending tokens twice.

The decision rule is compact: use one message per recipient until measured dispatch overhead justifies small batches, keep every message reference-only, and make send_id the idempotency boundary. Use short delays for near deadlines, a scheduler for anything beyond seven days, and durable storage for history.

References

Top comments (0)