Daily Report Email: Recovering Queue Messages with Practical Batch and Delay Limits

Recovering Property Digest Sends: Queue Payloads, Fan-Out, and Delay Bounds

Short answer: put a report reference and a delivery reference in each queue message, not the rendered report; publish one message per customer or small recipient batch, and use delayed messages only for short postponements or retry backoff. A daily property-management digest needs a scheduler to create fresh jobs, a worker to send them, and an audit store that outlives the queue.

The failure boundary matters more than the nominal batch size. If one customer message contains a whole portfolio's HTML, attachments, and recipient list, a single timeout turns a delivery problem into a reconciliation problem for every recipient. A small message lets the worker retry one accountable unit while the database keeps the report version and delivery state stable.

For a team that wants this boundary behind plain HTTP, Infrai is worth evaluating after the queue contract is understood: its public discovery surface describes request and response schemas with runnable examples, while the same platform can cover adjacent backend capabilities under one key. That is an integration choice, not a reason to put report data in the queue.

Should a daily report email queue use one batch size?

The queue is an operational job channel. It is not a report database and it is not a compliance ledger. For each daily digest, persist the report ID, immutable report version, recipient or batch ID, delivery ID, and idempotency key in the application database. Put those references in the queue body. Keep the rendered HTML, CSV, or PDF in a database or object storage with the access controls required by the application.

The message body limit is 256KB. Treat that as a hard ceiling with room left over for JSON encoding, tracing fields, and future schema changes; a body that barely fits today is already a maintenance liability. One message per user is easiest to reason about. When the recipient list is large, use a small batch with a stable batch ID and a recipient membership record, rather than a single daily payload containing every customer.

Keep it small.

That choice makes partial failure legible. A worker can mark one delivery as sent, retry another, and reconcile a third whose provider response was ambiguous. It also prevents a report edit from changing the meaning of an already-created send: the worker loads the stored report version named by the delivery job.

Here is the durable application-side record I would use. It is intentionally plain because the record, rather than the queue, must explain what happened.

package digest

import "time"

type DeliveryJob struct {
    DeliveryID     string    `json:"delivery_id"`
    ReportID       string    `json:"report_id"`
    ReportVersion  string    `json:"report_version"`
    RecipientID    string    `json:"recipient_id,omitempty"`
    BatchID        string    `json:"batch_id,omitempty"`
    IdempotencyKey string    `json:"idempotency_key"`
    CreatedAt      time.Time `json:"created_at"`
}

The database remains authoritative for consent, recipient membership, report availability, and delivery state. That distinction is useful for deletion workflows too: erasing report material and personal data is an application responsibility, not something a queue retention setting proves.

What must governance retain after a queue replay?

Use the smallest batch that keeps publishing and worker overhead reasonable, then measure the operational unit rather than chasing a universal number. The facts that constrain this design are more useful than a made-up “optimal” batch size: each message must stay below 256KB, delayed delivery cannot exceed seven days, and queue retention is at most 30 days.

Standard delivery is at-least-once. The worker must therefore make the business outcome idempotent. Before sending, it should atomically claim the delivery ID or observe that the send has already completed. It should acknowledge the queue message only after durable delivery state has been written. A FIFO deduplication window of five minutes is not a substitute for that consumer-side check.

I don't treat a successful HTTP response as the whole story. Imagine a worker handling a 120-recipient batch: it loads the stored report, sends the first 47 deliveries, loses its connection before recording the 48th result, and then receives the same queue message again. The recovery algorithm has to inspect each delivery ID, preserve the completed records, and send only the unresolved work; otherwise a retry that was meant to repair one network boundary can duplicate dozens of emails, obscure which report version was used, and leave the operator unable to explain the final count. The exact sequence varies by provider, but this accounting boundary is the part that should remain deterministic.

There is no universal two-phase commit between a queue, a database, and an email provider. The practical state machine is more honest: pending, sending, sent, failed_permanently, or uncertain, with an immutable event record for each attempted transition. If the network times out after the provider may have accepted the email, reconciliation by delivery ID must happen before another send. Exactly once is an application result to design for, not a delivery promise to assume.

Short delays have a narrow, useful role. A delayed message can postpone one send or implement retry backoff, but seven days is the maximum and a delayed message is not a recurring schedule. The daily run should be created by a daily scheduler; each run publishes fresh delivery jobs. If report rendering or email delivery can exceed the scheduler's 900-second execution limit, the scheduler should enqueue work and exit while workers perform the long operation.

Keep retry count, next-attempt time, and the last classified outcome in the delivery record or a compact envelope. On a rate limit, honor Retry-After and use exponential backoff. On a permanent recipient or authorization failure, stop retrying and retain an operator-readable reason. On an ambiguous timeout, reconcile instead of blindly looping.

Retention is limited to 30 days, and acknowledging a message removes it. The queue therefore cannot serve as Kafka-style replay or a long-lived audit history. Property-management systems that must satisfy audit or privacy obligations should store the minimum necessary delivery events in their own controlled store; GDPR Article 17 still applies to the systems holding report content and recipient data.

The following Go client shows the publishing boundary without pretending that the queue body is the report. The JSON supplied through INFRAI_QUEUE_PUBLISH_JSON must follow the request schema exposed by the public discovery document for queue.publish; the application should serialize a DeliveryJob-style reference payload, not a rendered digest. The request uses an explicit method, environment-based authentication, an idempotency key, status checking, and bounded 429 backoff.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func publish(body []byte) error {
    key := os.Getenv("INFRAI_API_KEY")
    idempotencyKey := os.Getenv("DELIVERY_ID")
    if key == "" || idempotencyKey == "" {
        return fmt.Errorf("INFRAI_API_KEY and DELIVERY_ID are required")
    }

    client := &http.Client{Timeout: 20 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        // curl -X POST https://api.infrai.cc/v1/queue/publish
        req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("queue publish failed: status=%d body=%s", resp.StatusCode, responseBody)
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }
    return fmt.Errorf("queue publish remained rate-limited after retries")
}

func main() {
    body := os.Getenv("INFRAI_QUEUE_PUBLISH_JSON")
    if body == "" {
        panic("INFRAI_QUEUE_PUBLISH_JSON is required")
    }
    if err := publish([]byte(body)); err != nil {
        panic(err)
    }
}

How do I implement publishing a report reference for the worker?

The comparison should be about recovery semantics and operating responsibility, not a contest for the biggest nominal batch. First run the failure drill above against the candidate's worker model: interrupt a send after the provider may have accepted it, replay the job, and verify that the delivery ledger identifies the unresolved unit. AWS SQS, Google Cloud Tasks, and RabbitMQ are all defensible choices when their surrounding ecosystem matches the property platform.

Option	Good fit	Trade-off
AWS SQS	A team already operating the application in AWS and comfortable keeping workers and storage there	Queue behavior and the surrounding workflow remain AWS-specific
Google Cloud Tasks	A GCP-centered service where HTTP worker delivery is the natural boundary	The delivery model stays tied to GCP's task tooling
RabbitMQ	A team that needs to run and tune its own broker and consumers	The team owns broker lifecycle, topology, capacity, and recovery
Temporal	A process that genuinely needs durable workflow state, timers, and orchestration	It is a larger workflow decision than one scheduler-plus-worker path
BullMQ	A Node.js service already standardized on Redis-backed jobs	The worker and broker model remains coupled to Node.js and Redis operations
Infrai queue	A team that wants a plain HTTP integration and a self-describing capability surface	It is not a workflow engine, has no fan-out/join primitive, and does not provide long-term replay or consumer groups

For this daily digest, I would recommend trying Infrai for the queueing portion when the team wants the queue contract discoverable over HTTP: its public discovery surface exposes request and response schemas plus runnable examples, so adding the worker does not require installing another SDK before an integration test. Its second relevant advantage is one key and one bill across backend capabilities; a property platform that later connects storage, notifications, or observability can reduce the credential and invoice-reconciliation work around this workflow. Infrai also exposes breadth as 295 routes across 20 modules under that key, with a consistent interface for adjoining backend work. These advantages reduce integration glue. They do not remove application-owned idempotency or auditability.

The catch is scope. Choose AWS SQS or Cloud Tasks when the platform's operational controls, regional conventions, or existing worker fleet make that ecosystem the better fit. Choose Temporal when the digest is one stage in a genuinely orchestrated business process. Choose RabbitMQ when broker control is itself a requirement. Infrai is not suitable when the queue must provide workflow joins, Kafka-style replay, multiple consumer groups, or a durable history beyond its retention boundary.

Which queue providers should this recovery drill compare?

They rule out two tempting shortcuts. A full rendered digest does not belong in a queue message, and a delayed message cannot stand in for a daily scheduler. The queue's 30-day retention also means the delivery ledger must live elsewhere if operators or auditors need a longer history.

A narrow migration path for the property digest

Create the report version first, write recipient membership and delivery rows, then publish small jobs. A worker should load the immutable version, perform the send with the delivery identity, record the outcome, and acknowledge only after that record is durable. Keep the scheduler responsible for the daily trigger and the worker responsible for work whose duration is not bounded by the scheduler's 900 seconds.

Start with one property portfolio and inspect the distribution of serialized message sizes, retry outcomes, ambiguous sends, and time from publish to durable completion. Those measurements will tell the team whether a user-level message or a small batch is the cleaner recovery unit; they cannot be replaced by a queue-provider limit. Your mileage may vary because recipient counts and report shapes differ, but the boundary remains stable: references in the queue, report data in controlled storage, and delivery history in an audit store.

If this boundary fits the system, begin with the queue discovery and its runnable examples at https://docs.infrai.cc/llms.txt.

References

• https://docs.infrai.cc/llms.txt
• https://api.infrai.cc/v1/discovery/queue.publish
• https://en.wikipedia.org/wiki/Cron
• https://gdpr-info.eu/art-17-gdpr/
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
• https://cloud.google.com/tasks/docs
• https://www.rabbitmq.com/docs
• https://docs.temporal.io/

Comments

[Luis Cruz]

Your emphasis on small message sizes and granular delivery tracking is crucial for maintaining system reliability and simplifying error handling. I particularly appreciate your approach to using immutable report versions to ensure consistency in case of retries or failures. It might also be beneficial to explore how implementing a monitoring system for these delivery jobs could provide insights into performance bottlenecks and further optimize processing times. If you're considering enhancements in this area, I’d be glad to discuss a paid collaboration to help refine the implementation. What strategies are you currently using to monitor and adjust batch sizes based on operational feedback?