Short answer: a daily report email for a large recipient list should use cron to create one logical batch, queue workers to send recipient-sized jobs, and an idempotency record to make retries safe. Keep the time decision separate from the delivery decision.
That boundary gives an on-call engineer a useful answer when the morning report is late: was the batch created, how many recipients are unfinished, and can a redelivery send the same mail twice? A successful cron invocation alone cannot answer any of those questions.
How should cron, queues, and Node.js workers split a daily report email?
The trigger should create or find a batch keyed by a report type and logical reporting date. Give that key a database uniqueness constraint so two trigger attempts converge on one batch. Store the expected recipient count with it. A number such as planned=48,912 has operational meaning; “the schedule ran” does not.
After the batch exists, enqueue small messages containing a batch ID, recipient ID, attempt number, and schema version. Do not put a complete audience or rendered report in every message. Referencing durable data keeps replay small and lets retention policies act on the source records without digging through a queue.
For recovery, make the batch record the source of truth rather than treating queue depth as a ledger. Suppose an operator pauses expansion halfway through the intended audience. They should be able to ask for recipients with no terminal delivery record, enqueue only those stable IDs, and see the batch counters converge again. Replaying every message from a broker is a different operation: it can include entries already accepted, entries whose audience membership has since changed, and messages encoded by an older schema. The distinction is easy to miss during a quiet test and expensive to reconstruct during an incident. A useful runbook says who may replay, which batch state permits it, how long the logical report remains useful, and what evidence must be captured before the action. It also says when to stop. A mailout that is hours past its business deadline needs an explicit expiry decision, not an endlessly growing retry count.
The worker then claims one recipient delivery, renders the message, and calls the sender with a deterministic idempotency key. HMAC is a reasonable way to derive a non-revealing key from stable inputs when recipient identifiers should not appear in operational systems. RFC 2104 defines the keyed-hash construction. A hash by itself does not coordinate concurrent workers, though, so the claim must be protected by a unique record or comparable atomic operation.
package mailout
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
)
type Store interface {
Claim(context.Context, string) (bool, error)
MarkAccepted(context.Context, string, string) error
}
type Sender interface {
Send(context.Context, string, string, string) (string, error)
}
func deliveryKey(secret []byte, date, kind, recipientID string) string {
mac := hmac.New(sha256.New, secret)
mac.Write([]byte(date + "\x00" + kind + "\x00" + recipientID))
return hex.EncodeToString(mac.Sum(nil))
}
func Deliver(ctx context.Context, store Store, sender Sender, secret []byte, date, kind, recipientID, batchID string) error {
key := deliveryKey(secret, date, kind, recipientID)
claimed, err := store.Claim(ctx, key)
if err != nil || !claimed {
return err
}
receipt, err := sender.Send(ctx, recipientID, batchID, key)
if err != nil {
return err
}
return store.MarkAccepted(ctx, key, receipt)
}
The queue message is acknowledged only after the delivery contract's send outcome has been recorded. A process can stop after an external send but before acknowledgement. Redelivery is normal in an at-least-once system. The idempotency decision belongs beside the external side effect.
Why do warm-worker timeouts create duplicate mail?
The risky moment is usually the first delivery window. Fresh workers load templates, establish outbound connections, and render messages together. If a lease or visibility timeout was chosen from warm median latency, it can expire while the first worker is still making progress. The queue offers the job to another worker, and both have a plausible claim on the same recipient.
Measure the end-to-end tail, not the average.
Set the timeout from cold and warm tail latency, with room for runtime pauses and downstream response time. A Node.js worker also needs event-loop-delay telemetry: a live process can miss a lease renewal while CPU-heavy rendering holds the loop. Preload templates before claiming work, reuse connections where the sender permits it, and put a cap on concurrency during rollout so a new worker pool does not create a connection surge.
Shutdown belongs in the same runbook. Stop claiming, permit a bounded drain, then leave unfinished messages for the queue's documented redelivery behavior. Don't acknowledge work merely to make a deployment look tidy. Correct ownership matters more.
What retry and backpressure rules keep a large recipient list bounded?
Retries need a budget, not optimism. Classify results as accepted, permanently rejected, or transiently unresolved. A malformed address leaves the active retry path. A timeout with an unknown external outcome first needs reconciliation against the delivery record; an automatic resend converts uncertainty into duplicate mail.
| Decision | Safer default | Reason |
|---|---|---|
| Fan-out unit | One recipient | Isolates failure and makes progress countable |
| Retry timing | Exponential backoff with jitter | Avoids synchronized retry bursts |
| Concurrency | Global and downstream caps | Limits pressure on workers and senders |
| Terminal work | Quarantine with a reason code | Keeps poison jobs out of active delivery |
Use a finite attempt ceiling and a maximum job age tied to the report's value. Yesterday's operational report can be less useful than no report. Backpressure starts before the queue is full: slow batch expansion when oldest-job age, throttling, or failure rate threatens the completion window. Don't build an unbounded ready backlog and hope autoscaling catches it.
Priority can help express preference, but it cannot create downstream capacity. RabbitMQ documents that priority queues have resource and scheduling costs and recommends a small range of priority levels. When an urgent class has a hard delivery objective, reserve capacity or isolate it from routine mail.
Your mileage may vary on the thresholds. They should come from the deadline and a production-sized, non-delivering load test.
What must the daily report email runbook prove before it is healthy?
Track planned, enqueued, accepted, permanently failed, and expired counts for every batch. Pair those with queue depth, oldest-job age, delivery-attempt latency, and structured fields for batch ID, recipient ID, job ID, attempt, and an idempotency-key fingerprint. Avoid raw recipient addresses in logs just because they are convenient.
Alert on impact: no batch after its creation window, oldest-job age threatening the delivery objective, or terminal counts no longer converging on the planned count. A retry-rate dashboard is useful; a page on every retry is noise.
Before release, simulate duplicate trigger acquisition, worker termination during a send, delayed downstream responses, poisoned payloads, and secret rotation. Verify that one logical report date produces one batch and one recipient key produces one committed delivery. Recovery should be a documented command path to pause expansion, drain workers, quarantine a job, replay a stable subset, and expire work that has lost its value.
The catch is the privacy and operating cost of per-recipient records. This design is not suitable for a tiny internal list where duplicate mail has little impact and a transactional outbox plus one process meets the recovery objective. Stick with that simpler model until audience growth, runtime, or recovery requirements justify the extra on-call burden. If a mail provider has a bulk operation with recipient-level status and idempotency, use that boundary rather than recreating it in workers.
References
- RFC 2104: HMAC keyed-hashing for message authentication: https://www.rfc-editor.org/rfc/rfc2104
- RabbitMQ priority queue documentation: https://www.rabbitmq.com/docs/priority
Top comments (0)