delay.run

Posted on Nov 7

Before a scheduled job emails 50,000 people: a 12-point checklist for bulk sends that can't be unsent

#email #backend #checklist #devops

Most scheduled jobs fail quietly. A report is late, a cache is stale, somebody notices on Tuesday.

A scheduled bulk email is the opposite. When it goes wrong, it goes wrong loudly, instantly, in thousands of inboxes, and there is no undo. "Hi {{first_name}}" to your whole customer list. The trial-ending email to people who already paid. The same newsletter, three times, because a retry fired after a timeout.

Here's the checklist I'd want before any job that sends email (or SMS, or push) to a large audience at a scheduled time.

Before it ships

1. The audience query is reviewed, not just the template

Template bugs are embarrassing. Audience bugs are expensive. Review the SQL or filter that selects recipients as carefully as any migration. Look for:

missing WHERE unsubscribed = false
joins that can duplicate a user (one row per order, not per customer)
time comparisons in the wrong time zone

Add a SELECT count(*) of the audience to the job's logs before anything is sent.

2. The audience is frozen at a known moment

If the job selects recipients and sends in a long loop, someone who signs up or unsubscribes mid-send gets inconsistent treatment. Better: snapshot the recipient IDs into a send table first, then send from that table.

INSERT INTO send_recipients (send_id, user_id)
SELECT :send_id, id FROM users
WHERE marketing_opt_in AND NOT unsubscribed
ON CONFLICT DO NOTHING;

This also gives you an exact record of who was meant to get it.

3. Every recipient has a "sent" marker, and the job checks it

The single most important line. If the job crashes at recipient 31,402 and restarts, it must not start again from 1.

for r in pending_recipients(send_id):          # WHERE sent_at IS NULL
    msg_id = provider.send(r.email, render(r), idempotency_key=f"{send_id}:{r.user_id}")
    mark_sent(send_id, r.user_id, msg_id)

If your provider supports idempotency keys, use one per recipient per send. If not, the sent_at marker is your only protection.

4. Retries are per recipient, not per job

A job-level retry ("the send failed, run it again") is how people get three copies. Retry the individual failed recipients with backoff; never re-run the whole batch blindly.

5. Missing personalisation fails closed

Decide what happens when first_name is empty. A fallback ("Hi there") is fine. Rendering the raw placeholder is not. Make the template renderer raise on undefined variables, and test it with a user who has every optional field empty.

6. There's a kill switch you can reach in under a minute

A feature flag, a row in a sends table with status = 'paused', an environment variable the loop checks every few hundred recipients. Something that doesn't require a deploy.

if i % 200 == 0 and send_is_paused(send_id):
    log.warning("send paused at %d", i)
    return

If you can't stop a send halfway, you can't limit the blast radius of a mistake.

Before it runs

7. Seed list first

Send to a small internal list (with real-looking test accounts across mail providers) through the same code path. Not the preview button: the actual job, with a filter.

8. Rate limits are respected on purpose

Your provider has sending limits, and receiving mail servers dislike sudden spikes from one sender. Throttle deliberately rather than letting the provider reject you halfway through.

9. The schedule is unambiguous

"Tuesday 9am" in whose time zone? Store the scheduled time as UTC plus the intended zone, and log both. If you're sending in each recipient's local time, test someone in UTC+13 and someone in UTC−10.

10. Nothing else big is running at the same time

A bulk send hits your database (audience query, rendering, marking sent) and your web servers (people clicking through at once). Check it doesn't coincide with backups, reindexing or another campaign.

While and after it runs

11. Progress is visible

Log sent/failed/remaining counts at intervals. Somebody should be able to answer "how far along is it?" without reading code.

12. You can answer "who got what" afterwards

When a customer says "I got this twice" or "I never got it", you need the send table: recipient, timestamp, provider message ID, result. Keep it.

Bonus: a dry-run mode that does everything except send

The cheapest safety net of all is a --dry-run flag that runs the real audience query, renders every message, writes the results to a file and stops short of calling the provider. Run it the day before. Then open the file and look: sort by length to find empty renders, grep for {{, count unique email addresses against the row count. Ten minutes with a dry-run output has caught more bad sends than any amount of template previewing, because it exercises the data you'll actually send to, not the one test account everyone uses.

The three that matter most

If you only do three: per-recipient sent markers (3), a kill switch (6), and a seed send through the real path (7). Those turn a disaster into an incident you can stop at 2%.

Scheduling the moment a job runs is the easy part; tools such as delay.run handle the "run this at 09:00 Tuesday" piece. What happens once it's running is still your code, and that's where this checklist lives.

DEV Community