Test your email deliverability from a script (and fail the build when SPF or DKIM breaks)

Serhii Tanichev — Fri, 18 Sep 2026 13:24:47 +0000

Last spring a transactional template that had passed every review for two years started landing in spam at Gmail. Nobody had touched it. What had changed was the sending platform: the team migrated to a new relay, the platform signed with its own DKIM domain by default, and the DMARC alignment that used to hold quietly stopped holding. It took nine days to notice, because "delivered" in the dashboard stayed at 99 percent the whole time.

I have spent years on the sending side of email, and I see this shape of failure more than any other. Authentication does not break in the template. It breaks in DNS, in the relay, in the migration nobody thought was mail-related. So the check has to run where the migration runs: in CI, against the real message, through the real relay.

This is the setup I use. It needs no account and no API key, because the tester I built for it hands out addresses to anyone.

The loop

Email Spam Tester works the way a receiving mail server works. You ask it for a disposable address, you send a message there, and it tells you what the receiving side saw: SPF, DKIM, DMARC with alignment, reverse DNS, blocklists, two spam engines, the Gmail and Yahoo bulk sender rules. Forty-one checks the day I write this. Each warning quotes the RFC section it rests on.

That number moves. It was thirty-nine in August, and it goes up whenever a receiver publishes a rule worth testing against or a report I read by hand turns up something the tool should have caught and did not. Two consequences for a script. Read checks_total off the status endpoint instead of writing the number into your code, and expect a finding you have never seen before to show up in a nightly run one day, on a template nobody touched.

From a script the whole thing is three HTTP calls and one SMTP session.

POST https://email-spam-tester.com/api/v1/inbox reserves an address. The answer has address, slug and expires_at. The address accepts exactly one message and expires in an hour.
Send the message over SMTP to that address. Use the address the API returned. It lives on a subdomain, and the domain is not part of the contract, so do not hardcode it.
GET https://email-spam-tester.com/api/v1/tests/{slug}/status answers 202 while nothing has arrived and 200 with progress afterwards. analysis_status runs received, analyzing, checks_ready, failed. The checks are final at checks_ready.
GET https://email-spam-tester.com/api/v1/tests/{slug} is the report.

Here is the client, standard library only. Python 3.11.

#!/usr/bin/env python3
"""Reserve an address, send a message to it, wait, return the report."""
import json
import sys
import time
import urllib.error
import urllib.request

API = "https://email-spam-tester.com/api/v1"


def call(method: str, path: str) -> tuple[int, dict]:
    req = urllib.request.Request(
        API + path,
        method=method,
        headers={"Accept": "application/json"},
        data=b"" if method == "POST" else None,
    )
    try:
        with urllib.request.urlopen(req, timeout=60) as resp:
            return resp.status, json.load(resp)
    except urllib.error.HTTPError as err:
        # 202, 404, 410 and 429 all arrive with a JSON body.
        return err.code, json.load(err)


def reserve() -> dict:
    status, body = call("POST", "/inbox")
    if status != 200:
        raise SystemExit(f"could not reserve an address: {status} {body}")
    return body


def wait_for_report(slug: str, timeout: int = 300) -> dict:
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        status, body = call("GET", f"/tests/{slug}/status")
        if status == 202:
            time.sleep(5)
            continue
        if status == 410:
            raise SystemExit("the address expired before the message arrived")
        if status != 200:
            raise SystemExit(f"unexpected answer: {status} {body}")
        print(f"{body['analysis_status']} "
              f"({body['checks_done']}/{body['checks_total']} checks)", file=sys.stderr)
        if body["analysis_status"] == "failed":
            raise SystemExit("the analysis failed on the server side")
        if body["analysis_status"] == "checks_ready":
            break
        time.sleep(3)
    else:
        raise SystemExit("no report within the timeout")
    status, report = call("GET", f"/tests/{slug}")
    if status != 200:
        raise SystemExit(f"could not read the report: {status} {report}")
    return report

Nothing surprising in there. The one detail worth a second look is the 202: urlopen raises on it, so it is caught in the same place as real errors and handed back as a status code, which keeps the polling loop flat.

Sending the real message

The point of the exercise is to send what you would actually send. A two-line "test" from your laptop tells you about your laptop. Half of the checks read headers the relay adds, so the message has to go through the relay, with the From address and the template that production uses.

import os
import smtplib
from email.message import EmailMessage


def send(to_address: str, template_path: str) -> None:
    msg = EmailMessage()
    msg["From"] = os.environ["MAIL_FROM"]          # the real sender, e.g. billing@example.com
    msg["To"] = to_address
    msg["Subject"] = "Your invoice for September"    # a real subject, not "test"
    with open(template_path, encoding="utf-8") as fh:
        html = fh.read()
    msg.set_content("Your invoice is attached. Plain-text part for clients that want one.")
    msg.add_alternative(html, subtype="html")

    with smtplib.SMTP(os.environ["SMTP_HOST"], int(os.environ.get("SMTP_PORT", "587")),
                      timeout=60) as smtp:
        smtp.starttls()
        smtp.login(os.environ["SMTP_USER"], os.environ["SMTP_PASSWORD"])
        smtp.send_message(msg)

If your template has links, keep them. Link reputation and the mismatch between visible text and the href are part of what gets scored, and a stripped-down copy hides exactly the things you want caught.

One decision to make early: which relay. The staging relay is convenient and tells you almost nothing, because it usually signs with a different key, sends from a different IP and sometimes from a different domain. I point the job at the production relay with a dedicated sender account that can only send, and I accept that one real message a night leaves the building. That message goes to an address that expires in an hour and is read by nobody, which is a cheaper price than the alternative I described at the top.

Deciding pass or fail

The report has two scores. score_ours runs from 0 to 100 and weights authentication and infrastructure heavily, because those decide delivery before any filter reads a word. score_compat is the classic 0 to 10 SpamAssassin-style number, there so you can compare with older tools. There is also checks[], one entry per check, each with id, category, status, title, summary, and citations.

For a build gate I use two rules. The overall score has to clear a threshold, and nothing in the auth category may be fail. A message can score well on content while DKIM is signed with the wrong domain, and that is the case I want to stop at the door.

def judge(report: dict, minimum: int = 80) -> int:
    problems = []
    for check in report["checks"]:
        if check["category"] == "auth" and check["status"] == "fail":
            problems.append(f"[auth] {check['title']}: {check['summary']}")
    score = report["score_ours"]
    if not report["complete"]:
        print("note: some checks could not run; the score is optimistic", file=sys.stderr)
    for check in report["checks"]:
        if check["status"] in ("warn", "fail") and check["category"] != "auth":
            print(f"[{check['status']}] {check['title']}: {check['summary']}", file=sys.stderr)
    print(f"score {score}/100, classic {report['score_compat']}/10", file=sys.stderr)
    print(f"report: {report['report_url']}", file=sys.stderr)
    if problems:
        print("\n".join(problems), file=sys.stderr)
        for check in report["checks"]:
            if check["category"] == "auth" and check["status"] == "fail":
                for source in check.get("citations", {}).get("standards", []):
                    print(f"    {source['title']}: {source['url']}", file=sys.stderr)
        return 1
    if score is None or score < minimum:
        print(f"score below {minimum}", file=sys.stderr)
        return 1
    return 0


def main() -> int:
    template = sys.argv[1] if len(sys.argv) > 1 else "templates/invoice.html"
    reservation = reserve()
    send(reservation["address"], template)
    report = wait_for_report(reservation["slug"])
    return judge(report)


if __name__ == "__main__":
    sys.exit(main())

Two notes on judge. A status of skip means the check did not apply; it is not a pass and not a failure, and it is left alone here. complete: false means something could not be checked, and I print it rather than fail on it, because a blocklist that timed out on the tester's side is not your problem to fix at two in the morning.

Each check also carries citations. For a DKIM failure that is the section of RFC 6376 with the sentence quoted verbatim, and where Google publishes its own requirement, the page where it says so. I print those for the failures, because the person fixing DNS at nine in the morning should not have to take the tool's word for anything.

The report_url line matters more than it looks. When the job goes red, the person reading the log gets a link to a page with every check, the quoted RFC text and a fix plan. Hand people the page, not the JSON.

The workflow

The job runs on a schedule and on changes to the templates directory. Nightly is enough for DNS drift; on push catches template edits. SMTP credentials go in repository secrets. The tester itself needs nothing, since there is no key.

name: email-deliverability

on:
  schedule:
    - cron: "17 6 * * *"
  push:
    paths:
      - "templates/**"
      - ".github/workflows/email-deliverability.yml"
  workflow_dispatch:

jobs:
  spam-test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Send the invoice template through the relay and read the report
        env:
          SMTP_HOST: ${{ secrets.SMTP_HOST }}
          SMTP_PORT: "587"
          SMTP_USER: ${{ secrets.SMTP_USER }}
          SMTP_PASSWORD: ${{ secrets.SMTP_PASSWORD }}
          MAIL_FROM: billing@example.com
        run: python3 scripts/spam_test.py templates/invoice.html

Fifteen minutes is generous. The checks are usually done a couple of minutes after the message lands; the timeout is there for the day your relay queues outbound mail.

What it has caught so far

The migration story from the top is the obvious one. The job would have gone red on the first nightly run after the switch, with [auth] DKIM alignment in the log and the report explaining that the signature's d= was the platform's domain rather than ours. Nine days become one night.

The one I did not expect was SPF. Somebody added a survey tool to the SPF record, the eleventh include: in a chain that was already at ten lookups, and the record went to permerror for every receiver on the planet. Nothing bounced. Mail kept arriving, slightly worse each week. The nightly job flagged it as an auth failure on the next run, with RFC 7208 section 4.6.4 quoted in the report, and the fix was a five-minute DNS edit instead of a month of falling open rates.

There is also a quieter benefit. Once the check is in CI, DNS changes stop being invisible to the people who write templates. The failure shows up next to their commit.

Where to take it from here

Run the same script against every From domain you send from; they fail independently. Keep one message per test address, since the address accepts exactly one and then closes. And if you would rather have an agent run the loop than a cron job, the same service exposes an MCP server with four tools, described at https://github.com/serg-tanichev/email-spam-tester-mcp.

The threshold of 80 is mine, for transactional mail from domains I look after. Pick yours by running the script against a message you know lands well, and set the bar a little under that.

About the author. Email Spam Tester is a free, independent email deliverability testing tool built by Serhii Tanichev. I came up through sending infrastructure and built the tester for the checks I kept doing by hand. It is still being added to: what changed and on which day is at https://email-spam-tester.com/changelog/. Site: https://email-spam-tester.com/. LinkedIn: https://www.linkedin.com/in/tanichev-sergey/.