Email deliverability in CI: what you can actually automate, and what you can't

"Did that email arrive?" is two questions wearing one coat. One of them automates cleanly. The other does not automate at all, and most of the frustration around deliverability testing comes from not separating them.

• Did we send a well-formed, authenticated message? Deterministic. Test it in CI.
• Did a receiver put it in the inbox? A judgement made by someone else's classifier, using your history. Not a test. A measurement, and a lagging one.

Automating the first is worth doing and is mostly ignored. Trying to automate the second produces a suite that fails for reasons nobody can action.

The part that belongs in CI

DNS invariants

These are pure functions of your DNS, so they can assert:

def test_exactly_one_spf_record(resolver, domain):
    txt = resolver.resolve(domain, "TXT")
    spf = [r for r in txt if r.to_text().strip('"').startswith("v=spf1")]
    assert len(spf) == 1, f"{domain} publishes {len(spf)} SPF records; must be 1"

def test_spf_lookup_budget(resolver, domain):
    used = count_lookups(domain, resolver)   # walks includes recursively
    assert used <= 10, f"{domain} uses {used} of 10 DNS lookups"

The second one is the test that earns its keep. The lookup limit counts the whole include tree, so a record that passes today breaks when a vendor adds an include inside their record — a change you will never be notified about. That is exactly the class of failure a scheduled job should catch and a human never will.

Run these nightly against production DNS, not just on deploy. The failure arrives on a day you did not ship anything.

Message structure

Assert on your own output before it goes anywhere:

• A List-Unsubscribe header, and List-Unsubscribe-Post for one-click
• A Message-ID with your sending domain in it
• A plain-text alternative that is not empty and not just stripped HTML
• From: aligning with the domain you intend to sign with
• No unsubscribe link pointing at localhost — this ships more often than anyone admits

Authentication, end to end

The strongest automated check sends one real message to a mailbox you control, then reads the Authentication-Results header the receiver stamped on it:

Authentication-Results: mx.google.com;
  spf=pass smtp.mailfrom=yourdomain.com;
  dkim=pass header.i=@yourdomain.com;
  dmarc=pass header.from=yourdomain.com

Assert all three pass — and assert on the d= / header.i= value, not just the verdict. dkim=pass with your ESP's domain there is a pass that aligns with nothing, and DMARC fails anyway. That assertion catches a misconfiguration that every "is DKIM working?" check reports as fine.

The part that does not belong in CI

Inbox placement. Whether Gmail files you under Primary, Promotions or Spam depends on your domain's history, the recipient's own behaviour, and a classifier that changes without notice. Seed-list tools sample it, and the sample is drawn from mailboxes with no real engagement history — which is precisely the input the classifier cares most about. Useful as a trend. Not a pass/fail gate.

Reputation. A lagging aggregate. There is no assertion to write.

Spam-word scoring. Content is real but it is the smallest factor, and the scores are not the receiver's scores. A well-written message from an unauthenticated domain still lands in spam, which is why rewriting subject lines is the first thing people try and the last thing that helps.

Gate your build on none of these. Put them on a dashboard with a date on it.

Dev and staging, without becoming a spammer

The cheapest mistake in this whole area is a staging environment that can reach the open internet.

• Point non-production at a catching SMTP server — MailHog, Mailpit, or your provider's sandbox. Make it the default, so reaching real recipients requires an explicit override rather than the reverse.
• If production credentials exist anywhere in a non-production environment, one seeded fixture with real addresses is all it takes. Do not rely on the seed data being fake.
• Never warm a domain with traffic generated by tests. Warm-up is a claim about human sending behaviour; synthetic volume against seed mailboxes teaches the receiver something, and it is not what you want it to learn.

A shape that works

Stage	Frequency	Gates the build?
DNS invariants — one SPF record, ≤10 lookups, DKIM resolves, DMARC present	Nightly + deploy	Yes
Message structure assertions	Every commit	Yes
Live send → assert Authentication-Results incl. d= alignment	Nightly	Yes
Bounce-code classification tests (fixtures)	Every commit	Yes
Inbox placement sample	Weekly	No — trend only
Complaint rate vs the 0.1% threshold	Continuous	Alert, not a gate

Everything in the top four is a property of your own system and fails for reasons you can fix. Everything below is somebody else's verdict on your history, and belongs on a graph rather than in a red build.

If you want the DNS half checked right now without wiring anything up, this runs the same lookups against live DNS, and spf-audit is the CI-shaped version — it walks the full include tree and exits non-zero over the limit.