Short answer: keep a daily report email schedule in durable storage, put only references in queue messages, and choose recipient batch size from measured bytes and processing time rather than from a fixed count. The deciding constraint is recoverability: seven days is long enough for cancellations, changes, and deletion requests to matter before a message is delivered.
The tempting implementation is a scheduled process that renders a report, loops through every recipient, and sends email before its next tick. It is short code, and it works right up to the point where a deploy interrupts it, a recipient changes preference, or a retry repeats an external side effect. A queue helps with fan-out, but it does not replace a record of the work that is due.
The queue is not the calendar.
How should a daily report email queue handle 256 KB message limits and seven-day delayed batches?
Treat 256 KB as a hard boundary on the serialized message, not as a target. A recipient count cannot establish payload size: identifiers vary, JSON punctuation adds bytes, and UTF-8 text is not one byte per character. The safe check is to serialize the object the queue adapter will publish, measure its UTF-8 byte length, and split before it crosses a deliberately lower application budget.
The message should name work, not carry the work's private output. For a daily report, that normally means a schedule ID, report date, artifact ID, batch partition, and recipient IDs. Rendered HTML, attachments, full profiles, and an LLM response belong in controlled storage with retention rules. This makes a retry cheaper and makes later deletion tractable; a delayed message containing copied personal data creates another place to find and erase it. GDPR's right to erasure applies in particular circumstances, with exceptions that need legal review, but the system design consequence is clear: minimize copies before a seven-day wait turns them into forgotten state.
Delay is a separate concern. Keep deliverAt, a stable logical key, current status, and cancellation state in a queryable schedule record. A recurring trigger can wake a dispatcher and find due rows; cron is a convention for recurring scheduled execution, not proof that a particular email was sent. The dispatcher claims a small window of due work and then publishes compact batches. A broker's delayed-message feature can be useful transport, yet the schedule record remains the source of truth for work that must be inspected, cancelled, or rescheduled.
The experiment: why the direct loop loses its appeal
For a practical design, compare behavior under interruption rather than the number of lines in the happy path. The direct loop combines time calculation, report creation, recipient selection, email delivery, and recovery in one execution. Its failure boundary is vague: after a process stops, the system may know the report was generated but not which recipients received it. A restart then has two bad guesses available: send the whole list again and risk duplicate mail, or skip the list and risk a silent omission. Tracking only a queue acknowledgement does not settle the question, because the acknowledgement describes transport acceptance rather than the recipient-visible side effect. Tracking only a provider response is also insufficient when the process cannot durably associate that response with the intended recipient and logical report date. The useful design work is deciding which state transition establishes ownership of the next action, recording it before crossing the relevant boundary, and making the next attempt consult that record instead of reconstructing history from logs.
A two-stage design separates those decisions. First, create or locate one report artifact for the logical report date. Then claim recipient deliveries using a stable key such as tenant/report-date/recipient/template-version. A queue batch is only a convenient carrier for references to those delivery attempts. If the same batch appears twice, the delivery ledger decides whether there is work left to do.
That distinction changes testing. Run two dispatchers at the same time. Publish the same partition twice. Stop processing after recording intent and before recording the final outcome. None of these tests should produce two logical sends for one recipient. Email providers and databases have different atomicity boundaries, so exactly-once delivery is not a promise a generic queue snippet can make. The target is narrower and useful: one durable identity, one winning claim, and an explicit reconciliation path for uncertain outcomes.
Here is the focused guard I would put at the publish boundary. It begins with an arbitrary group of recipient IDs, measures the finished JSON representation, and recursively creates smaller batches when necessary. 240 * 1024 is headroom for an application whose queue contract limits messages to 256 KB; the adapter still needs to account for any attributes it adds.
type DeliveryBatch = {
scheduleId: string;
reportArtifactId: string;
recipientIds: string[];
partition: string;
};
interface QueuePort {
publish(batch: DeliveryBatch): Promise<void>;
}
const applicationBudgetBytes = 240 * 1024;
function encodedBytes(batch: DeliveryBatch): number {
return Buffer.byteLength(JSON.stringify(batch), "utf8");
}
async function publishBatch(queue: QueuePort, batch: DeliveryBatch): Promise<void> {
if (encodedBytes(batch) <= applicationBudgetBytes) {
await queue.publish(batch);
return;
}
if (batch.recipientIds.length < 2) {
throw new Error("Recipient reference exceeds the application payload budget");
}
const midpoint = Math.ceil(batch.recipientIds.length / 2);
await publishBatch(queue, {
...batch,
recipientIds: batch.recipientIds.slice(0, midpoint),
partition: `${batch.partition}-a`,
});
await publishBatch(queue, {
...batch,
recipientIds: batch.recipientIds.slice(midpoint),
partition: `${batch.partition}-b`,
});
}
The catch is that this is more machinery than a tiny internal mailing list needs. A single scheduled process can be suitable when rerunning the whole list is harmless and one owner can inspect the result. Conversely, a branching process with approvals, human waits, and compensating actions is not suitable for a small schedule table plus queue; use a workflow engine designed to model that state. The right boundary comes from the cost of being wrong, not from the appeal of a particular queue API.
What should determine the batch size after the queue message is valid?
Byte size only answers whether a batch can enter the queue. Processing capacity answers how many recipients should share a unit of work. Start with a deliberately modest group, then measure queue age, serialized bytes at p50 and p95, worker duration, claim contention, retry outcomes, delivery latency, and the number of duplicate claims blocked by the ledger. A batch that fits comfortably can still be too large if its worker lease expires, one slow recipient delays the rest, or replaying a partial failure creates an expensive blast radius.
Use the measurements to change one variable at a time. If fixed per-message overhead dominates and workers stay well inside their execution limit, increase the candidate group while retaining the byte guard. If a batch approaches the worker's time budget, reduce it. Keep report-generation cost apart from delivery cost: generate a report artifact once for its intended logical date, then make delivery retries read that artifact rather than regenerate variable content.
I'm not sure a universal starting count exists because recipient ID lengths, provider latency, and the work done per email all change the result. This is why a literal 100 in sample code should be treated as a test input, not policy.
Operational checks for work delayed up to seven days
A seven-day horizon puts data lifecycle and calendar behavior on the critical path. Store enough state to answer four ordinary support questions: what is due, what was cancelled, what was sent, and which retained copies belong to a person. Give schedule rows, artifacts, delivery records, logs, and dead-letter records explicit owners and retention dates. Dead-letter entries should contain identifiers and failure categories, not rendered email bodies.
Time-zone tests deserve the same attention as retry tests. Run the scheduler using a documented clock basis, calculate each promised local time explicitly, and exercise daylight-saving transitions. Replay a missed tick, cancel an item scheduled for next week, and run concurrent dispatchers against the same due record. Also test payloads containing long identifiers and multibyte text just below and above the application budget. These are small tests. They expose big assumptions.
Before copying the pattern, measure the final queue envelope, the oldest due schedule record, claim failures, worker duration, and deletion completion time. The outcome should remain understandable even when a process restarts: future work is visible in storage, queue messages stay small, retries preserve a logical identity, and every data copy has a lifecycle.
Top comments (0)