DEV Community

CalderHayes9638
CalderHayes9638

Posted on

Daily Report Email Delivery Under Rate Limits: Queue or HTTPS Push?

Short answer: use a daily scheduler to create the report run, then put one small delivery job per recipient into a queue; let a rate-aware worker control admission to the email provider, and use a public HTTPS push subscriber only when exposing that ingress is acceptable. The queue is the pacing boundary, while the application database is the source of truth for idempotency and audit.

The important distinction is between starting work and completing work. A scheduled callback can say that the day's report should begin. It should not own thousands of provider calls, transient failures, or the question of whether a timeout happened before or after the provider accepted a message. Those are delivery concerns, and they need durable state. You don't want a green scheduler run to mean “some messages were probably sent.”

Make that boundary explicit.

The ledger comes before the transport

Start with the invariant, not the queue brand: one report identity may be delivered more than once by the transport, but it must produce at most one accepted business effect. For a logistics report, a useful operation key is (report_date, tenant_id, recipient_id). Store that key with the run ID, current state, attempt count, provider response ID, and timestamps. A unique constraint on the operation key prevents two workers from creating two send decisions at the same time.

This is an exactly-once mindset applied to a system that is usually at-least-once. It does not claim that the network can deliver exactly once. It says that the application has an idempotency boundary and an audit trail strong enough to reconcile an ambiguous attempt. That distinction matters when a worker sends successfully, loses its acknowledgement, and receives the same job again.

The scheduler should create a run record, determine the intended recipient set, and enqueue compact commands. It should finish after admission work is durable. The worker should claim a command, apply the provider's rate policy, call the mail API, and write the resulting state before acknowledging the queue message. If the provider supports an idempotency key, pass the operation key through; if it does not, the ledger still prevents a second attempt after a recorded acceptance, but an ambiguous network timeout needs explicit review rather than a comforting guess.

The dangerous interval is small but consequential. Suppose the worker sends the 08:00 dispatch for tenant 7 and recipient 41, receives a provider acceptance, and crashes before committing accepted or acknowledging the queue. The broker redelivers the command. A naive consumer sees “not accepted” and sends again; a ledger-aware consumer marks the operation as unresolved, checks the provider's idempotency or delivery record, and only then decides whether another attempt is permissible. That extra state is not ceremony: without it, a timeout becomes a duplicate email, while with it, an operator can explain the run to a customer or an auditor. The same reasoning applies when two workers race, when a deployment interrupts a request, and when the provider returns 429 after the application has already consumed its retry budget. I've found the useful design question is not “can this queue retry?” but “what durable fact lets the next process decide?”

Keep payloads boring.

type DeliveryJob struct {
    RunID         string
    OperationKey  string
    RecipientID   string
    ReportObject  string
}

type DeliveryState string

const (
    Queued    DeliveryState = "queued"
    Accepted  DeliveryState = "accepted"
    Retryable DeliveryState = "retryable"
    Failed    DeliveryState = "failed"
)
Enter fullscreen mode Exit fullscreen mode

The queue message needs a reference to the rendered report, not the entire report. A reference also lets the worker load the same content on a retry and makes the send record inspectable. Validate that the report object belongs to the run and recipient before sending; a validly authenticated request with the wrong tenant context is still a data leak.

How should a daily report email queue handle provider limits?

A pull worker makes the control loop explicit: it takes a job only when its concurrency and token budget permit, waits after a throttle response, and retries the same operation key. A push subscriber moves job admission to an HTTP endpoint. That can fit a service that already has authentication, observability, and a durable send ledger, but the endpoint must be reachable over public HTTPS and must return success only after the job reaches a durable state.

The public endpoint is not a substitute for the ledger. Authenticate the request, validate the queue and job identity, enforce a bounded body size, and make repeated deliveries harmless. If the handler has accepted the job into a durable internal work table, it can acknowledge the push and let a private worker perform the rate-limited send. Sending inside the HTTP request is possible, but it couples provider latency to the push timeout and makes capacity harder to reason about.

Rate limiting is flow control. On 429, honor Retry-After when the provider supplies it. Otherwise use exponential backoff with jitter, cap the delay, and cap the total attempts. A retry after a process restart must read its decision from durable state, not from an in-memory counter. The following worker sketch keeps the transport generic and shows the part that should be tested: the same operation key travels through every attempt.

package main

import (
    "context"
    "errors"
    "math/rand"
    "time"
)

type DeliveryJob struct {
    OperationKey string
    RecipientID  string
}

var ErrRateLimited = errors.New("provider rate limit")

func backoff(attempt int, retryAfter time.Duration) time.Duration {
    if retryAfter > 0 {
        return retryAfter
    }
    base := time.Second * time.Duration(1<<min(attempt, 6))
    return base + time.Duration(rand.Int63n(int64(base/2)))
}

func min(a, b int) int {
    if a < b {
        return a
    }
    return b
}

func deliver(ctx context.Context, job DeliveryJob, maxAttempts int) error {
    for attempt := 0; attempt < maxAttempts; attempt++ {
        // claimOrSkip is a transaction against the durable send ledger.
        if claimOrSkip(job.OperationKey) {
            err, retryAfter := sendOnce(ctx, job.OperationKey, job.RecipientID)
            if err == nil {
                markAccepted(job.OperationKey)
                return nil
            }
            if !errors.Is(err, ErrRateLimited) {
                markFailed(job.OperationKey, err)
                return err
            }
            wait := backoff(attempt, retryAfter)
            timer := time.NewTimer(wait)
            select {
            case <-ctx.Done():
                timer.Stop()
                return ctx.Err()
            case <-timer.C:
            }
        }
    }
    return errors.New("retry limit reached")
}

func claimOrSkip(string) bool                         { return true }
func markAccepted(string)                              {}
func markFailed(string, error)                         {}
func sendOnce(context.Context, string, string) (error, time.Duration) {
    return nil, 0
}
Enter fullscreen mode Exit fullscreen mode

This is deliberately a sketch: the ledger functions must be implemented with the database's transactional guarantees, and the provider call must define what an ambiguous timeout means. I am not sure any generic queue can resolve that ambiguity by itself; the missing evidence is the provider's idempotency contract and a reconciliation endpoint or report. That uncertainty belongs in the design review.

Test the boundary, not just the happy path

The first failure is a retry storm. A daily batch released at once can exceed the provider's quota, and immediate retries amplify the overload. A worker pool with bounded concurrency, a shared rate budget, jitter, and a dead-letter path is more useful than a large timeout around the scheduler. Test with synthetic 429 responses and confirm that queue age rises predictably instead of worker count rising without limit.

The second failure is a false success. A queue acknowledgement means the transport no longer owns the message; it does not prove that an email was accepted. Acknowledge after the durable transition, and distinguish accepted, retryable, failed, and needs_review. A needs_review state is valuable for timeouts after a provider request has been written, because an operator can reconcile the provider response before allowing another send.

The third failure is an incomplete run. An empty queue is not a completion certificate if messages disappear after acknowledgement. Compare the intended recipient count with accepted, failed, and unresolved ledger rows. Keep the run ID, operation key, attempt number, response identifier, and timestamps. These fields turn a support question into a query.

The fourth failure is a security mistake at the public boundary. A public HTTPS subscriber should use request authentication and replay protection, and it should reject unknown run or tenant identifiers. Do not put secrets, full report contents, or mutable authorization decisions in a message that can be replayed. The endpoint can be public without making the report public.

There is also a compliance limit: email addresses and report contents may be personal or operational data. Retention, access logging, deletion, regional processing, and incident response should be requirements of the ledger and object store, not afterthoughts in the queue configuration. Standards and provider contracts determine the exact obligations; the architecture cannot waive them.

Which ingress model fits the security boundary?

The catch is that public push delivery expands the ingress boundary. It is unsuitable when policy requires the consumer to remain private, when the team cannot operate authenticated HTTP ingress, or when provider latency makes the push timeout difficult to keep within its delivery contract. In those cases, use a private worker that pulls from the queue and shares the same idempotency ledger.

The queue pattern is also a poor fit for long-lived branching workflows, durable joins, or months of replay. A work queue is for admitting work; it is not automatically an event log, a workflow engine, or a compliance archive. If those are governing requirements, choose an orchestration or event-log architecture whose retention, replay, and state model are explicit. If the workload is only one daily report, a scheduler plus a queue may be needless machinery: a single application job with a durable ledger can be simpler when provider limits and retry volume are small.

The selection rule is therefore conditional. Use a pull worker for private networks and visible pacing. Use push when public HTTPS is acceptable and an existing service can make the boundary small and auditable. Use neither as a replacement for a database-backed send ledger. That is the part that protects the business effect.

Decision question Pull worker Public HTTPS push
Network boundary Private consumer is possible Public authenticated endpoint is required
Pacing Easy to coordinate in the worker pool Must be enforced after HTTP admission
Acknowledgement After the ledger transition After durable admission or send state
Best reason to choose it Security and operational visibility Existing service and simpler inbound delivery

The table is a decision aid, not a product ranking. Your mileage may vary with the provider's timeout and idempotency contract.

A rollout that leaves an audit trail

Begin with one tenant and a conservative rate. Create a deterministic run ID and operation keys, enqueue references rather than report bodies, and assert that duplicate commands produce one accepted ledger row. Then inject a 429, a worker restart, a lost acknowledgement, and an ambiguous provider timeout. Each test should end with a reconciliation query, not just a green process exit.

Add metrics for queue age, admission rate, provider response classes, attempt count, unresolved sends, and intended-versus-accepted totals. Alert on age and reconciliation drift, because a queue can be healthy while the report promise is being missed. Keep a manual review path for ambiguous sends; silently replaying them is how a retry feature becomes a duplicate-delivery feature.

Finally, document the boundary conditions in the runbook: who may expose public HTTPS, how credentials rotate, how long report objects and ledger rows remain, what happens after the retry cap, and how an operator proves that a day's audience was accounted for. The simplest architecture is the one whose failure state can be explained.

References

Top comments (0)