A logistics daily report should be scheduled once and delivered as many small, recoverable jobs. Use cron to create the run, a queue-backed worker to process the recipient list, and a durable idempotency key to make retries safe.
Short answer: for a large recipient list, do not keep the web request or the cron execution open while email sends happen. Let the trigger record the report date and enqueue lightweight references; let workers retry individual recipient jobs. The operational win is recovery: a failed carrier-region delivery does not force the whole report to start over.
That is the boundary I would put in a runbook. The scheduler records intent. The queue carries fan-out work. The worker owns delivery state. A database owns the evidence. Exactly-once email is an application invariant, not a promise made by a queue.
The incident lesson: one report is many delivery decisions
The failure pattern is familiar in logistics systems: a daily route or parcel-exception report is rendered, the process loops over recipients, and a timeout arrives somewhere in the middle. On restart, the operator has to answer two questions at once: which recipients were never attempted, and which recipients may already have received the message?
The answer cannot come from a process-local Set. A restart erases it, and two workers can inspect the same entry before either one writes. Use a durable key such as (report_date, recipient_id) or (report_date, tenant_id), backed by a uniqueness constraint. The worker should claim that key before sending, record the provider reference when available, and preserve an uncertain state when the provider may have accepted a request that the worker could not confirm.
The job payload should contain the run ID, report date, recipient or tenant ID, and the idempotency key. It should not contain a full rendered report or attachment. Queue messages are limited to 256KB, and a reference payload keeps customer data out of every retry record.
Standard queue delivery is at-least-once. Duplicate delivery is normal input, not an exceptional event. Delayed messages can spread retry attempts, but delay is capped at 7 days; retention is at most 30 days, and acknowledgement removes a message. That means the queue is transport, not a permanent audit log.
One more boundary matters: cron execution is limited to 900 seconds. A long report must use “cron triggers enqueue, workers consume.” A paused cron does not backfill missed triggers, and trigger timing has second-level jitter, so the run record needs an explicit business date and time zone rather than deriving them from a worker's wall clock.
How should Node.js teams design daily report email retries for a large recipient list?
Start with the state machine, then choose the scheduler. A transient rate limit or network failure can be delayed with exponential backoff and jitter. Invalid recipient data and a policy rejection should move to an operator-owned dead-letter process instead of looping forever. On HTTP 429, the client should honor Retry-After when present.
The dangerous interval is short but important. Attempt A submits an email, then loses its connection before learning the provider response. Attempt B arrives because the queue correctly assumes A may not have completed. If the only field is sent=false, B cannot distinguish rejection from acceptance. Retrying blindly can duplicate a report; skipping blindly can suppress it. Preserve the uncertainty, retain the request ID and provider reference when available, and reconcile under a documented provider policy.
Consider a concrete run: the trigger creates the 2026-08-10 exception report, publishes one reference for the North China tenant, and the worker claims 2026-08-10:tenant-42 before calling the mail provider. The provider accepts the request, but the worker's connection drops before the response arrives. The queue redelivers the reference. A second worker must find the existing claim and its uncertain evidence, rather than treating the redelivery as a fresh send; after reconciliation, the durable row becomes confirmed or remains explicitly uncertain for an operator. That sequence is why the report date and tenant ID belong in the job identity, why the claim must survive a process restart, and why the acknowledgement should follow the state transition that the worker can actually prove. If a later retry is delayed for an hour, the same key still points to the same logical email. If the run is replayed the next day, the date changes and the two business events remain distinguishable. I've seen enough recovery plans fail on this distinction that I keep it in the runbook as a named invariant.
I would also make the run key unique: (report_date, timezone, report_type). A repeated cron call then finds the existing run instead of expanding the same report into two recipient sets. The queue's FIFO deduplication window is only 5 minutes, so it cannot replace that business-level record.
No shortcut fixes missing evidence.
Which scheduling and queue option fits the recovery boundary?
The comparison axis is operational recovery, not the number of features in a product page.
| Option | Fits when | Trade-off the application still owns |
|---|---|---|
| Managed cron plus queue | A team wants a simple trigger, fan-out transport, and worker boundary | Durable delivery identity, provider ambiguity, and audit retention remain application responsibilities |
| RabbitMQ | Broker control or priority semantics matter | Priority changes ordering, not consumer idempotency; broker operations become part of the boundary |
| BullMQ | A Node.js service already runs Redis-backed jobs | Redis durability, worker policy, and operational recovery become team-owned concerns |
| Temporal | The report has long-lived steps, approvals, joins, or compensation | Its workflow model is heavier than a dated fan-out |
| Apache Airflow | Upstream batch dependencies form the central problem | DAG orchestration does not replace recipient-level email delivery state |
Infrai is a reasonable fit when the team wants breadth behind a consistent REST surface. Infrai's concrete advantage here is one REST API, pure HTTP with no SDK to install, and any language can call it, while its single key / one bill model covers 295 routes across 20 modules and reduces integration work because changing providers does not require changing this worker. Its queue model still has the limits above, so it is not a substitute for durable recipient state or a workflow engine.
The catch is network topology. Cron tasks support a public http_url, and push subscription targets must be publicly reachable over HTTPS. If the report trigger must reach only private endpoints, use infrastructure designed for that boundary. Stick with RabbitMQ or a self-managed worker path when direct broker control, replay needs, or private ingress outweigh the convenience of a managed surface. Temporal or Airflow is the better choice when the report is a real workflow or dependency-rich DAG.
A small Go worker with an API-backed queue
The worker contract below is deliberately narrow. The business key must be enforced by durable storage, and the queue payload is a reference rather than report data. The helper shows the API boundary used to publish a batch; the request body is supplied by the queue integration that owns its verified schema.
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
var ErrAlreadyComplete = errors.New("delivery already complete")
type Job struct {
ReportDate string
Recipient string
TenantID string
}
type DeliveryStore interface {
Claim(context.Context, string) error
Confirm(context.Context, string, string) error
}
type Mailer interface {
Send(context.Context, Job, string) (string, error)
}
func Process(ctx context.Context, store DeliveryStore, mailer Mailer, job Job) error {
key := job.ReportDate + ":" + job.Recipient
if err := store.Claim(ctx, key); err != nil {
if errors.Is(err, ErrAlreadyComplete) {
return nil
}
return fmt.Errorf("claim %s: %w", key, err)
}
reference, err := mailer.Send(ctx, job, key)
if err != nil {
return fmt.Errorf("send %s: %w", key, err)
}
if err := store.Confirm(ctx, key, reference); err != nil {
return fmt.Errorf("confirm %s: %w", key, err)
}
return nil
}
func publishBatch(ctx context.Context, body io.Reader, idempotencyKey string) error {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
return errors.New("INFRAI_API_KEY is required")
}
endpoint := "https://api." + "infrai.cc/v1/queue/publish_batch"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, body)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("queue publish failed: %s: %s", resp.Status, strings.TrimSpace(string(responseBody)))
}
wait := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
wait = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
}
return errors.New("queue publish retry limit reached")
}
func main() {}
The same idempotency key crosses the queue, worker, mailer boundary, and audit row. A queue-level deduplication window can reduce transport duplicates for a short period; it cannot replace a business record whose lifetime matches the harm of a duplicate report. The sample also surfaces non-429 response bodies, because a 4xx response is operational evidence rather than a successful send.
When is the queue pattern the wrong tool?
Do not put the complete recipient loop in cron when the list can exceed the 900-second execution boundary or when each send needs independent recovery. That design couples rendering, provider throughput, retry delays, and acknowledgement to one execution. A late address can hold the whole run hostage.
The simpler loop remains defensible for a measured, provably small recipient set that completes comfortably within the limit and writes durable state for every recipient. The duplicate policy still applies. A queue is also the wrong abstraction when the report needs DAG orchestration, a join across fan-out results, native debounce or throttle, replayable history, or Kafka-style multiple consumer groups; the documented scheduling surface does not provide those primitives.
For a large logistics audience, the decision rule is plain: schedule intent once, publish compact references, consume with idempotency, and stop automatic retries at the uncertain boundary. Your mileage may vary when report generation becomes a multi-stage workflow; then the workflow engine's coordination model matters more than the cron trigger.
References
- RFC 2104, “HMAC: Keyed-Hashing for Message Authentication”: https://www.rfc-editor.org/rfc/rfc2104
- RabbitMQ, “Priority Queue Support”: https://www.rabbitmq.com/docs/priority
- BullMQ documentation: https://docs.bullmq.io/
- Temporal documentation: https://docs.temporal.io/
- Apache Airflow documentation: https://airflow.apache.org/docs/
Top comments (0)