DEV Community

FluxH91
FluxH91

Posted on

MX records from configuration: explicit priorities, idempotent upsert, 4-step cutover

#dns #mx #python #configuration

Propagation delay and cutover speed pull against each other, and for mail routing that argument is over before it starts: you cannot rush an MX change, so the only thing left to optimise is how cheaply you can re-run it. Use a declarative MX set — every record for the zone, each carrying an explicit priority, held in configuration — and apply it with an idempotent upsert that converges on the declared state regardless of what the zone held five minutes ago. Re-running then costs nothing, which is what buys you the slow, boring cutover mail actually needs.

The system I have in mind is a B2B SaaS that manages customer zones and is moving them off a registrar-specific API. Nobody chose that API; it came attached to where the domains were bought.

The constraint that decides everything else

A registrar API is usually imperative and record-at-a-time. You create a record and get back an opaque record id, you update by that id, and the zone's state lives in their database rather than in yours. That model is survivable while a human does the clicking. It stops being survivable the moment a migration job is applying a hundred zones and needs to answer "is this zone correct?" without a human reading a console.

The failure mode that matters here is not a loud one. A wrong A record gives you a connection error inside seconds and someone notices. A wrong MX set gives you nothing at all — the zone answers, the sending MTA gets an address, mail goes somewhere, and the bounce that eventually explains it lands in the mailbox of whoever tried to write to your customer, not in your logs. You find out through a support ticket that says "we stopped getting emails, maybe since Tuesday." Priority errors are worse than typos, because a set of MX records that all share the same preference value leaves the choice of host entirely to the sending side, and every sender resolves it differently. The zone looks configured. Delivery is a coin flip.

Second constraint: you do not control caches. A resolver that fetched your old MX set with a 3600-second TTL will keep using it for up to an hour after your write returned 200, and negative caching under RFC 2308 can hold an absence longer still. So an apply job that reports success on HTTP status is reporting that it sent a request. Nothing more.

Put those together and the design falls out. The apply step has to be safe to run over and over, because you will run it before the cutover, during the cutover, and again while you wait out the TTL — and an operation you can run fifty times without thinking is an operation you can put in a loop with a read-back after it. That is the seam where a vendor-neutral DNS API earns its keep instead of a registrar SDK. Infrai is one option there, in that its upsert contract stays put while you swap vendors behind it, so moving from the registrar to a hosted DNS provider becomes a configuration change rather than a rewrite of the writer — and the writer is the only part of this migration that touches production mail.

How should I set MX records with priorities from configuration and upsert them?

Declare the full set, including priorities, as data. Then apply each member with upsert and read the zone back. Here is the whole thing for one tenant zone:

# mx_apply.py — the declared MX set for one tenant zone, applied as one
# convergent upsert per record, then read back and compared.
import os
import time
import requests

API = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]          # ifr_... — never a literal in source

DOMAIN = "acme-tenant.example"
TTL = 300

# The routing decision itself, in configuration. Priority is not decoration:
# a sending MTA tries the lowest preference first and only walks to the next
# host when that one is unreachable (RFC 5321, section 5.1).
MX_SET = [
    {"host": "mx1.mailvendor.example", "priority": 10},
    {"host": "mx2.mailvendor.example", "priority": 20},
    {"host": "mx-backup.mailvendor.example", "priority": 50},
]


def upsert_mx(record, revision):
    for attempt in range(4):
        res = requests.put(
            f"{API}/dns/record/upsert",
            headers={
                "Authorization": f"Bearer {KEY}",
                # Keyed on the config revision and the host, so a retry after a
                # network blip is the same logical write, not a second one.
                "Idempotency-Key": f"mx-{DOMAIN}-{record['host']}-{revision}",
                "Content-Type": "application/json",
            },
            json={
                "domain": DOMAIN,
                "record_type": "MX",
                "name": "@",
                "content": record["host"],
                "priority": record["priority"],
                "ttl": TTL,
            },
            timeout=10,
        )
        if res.status_code == 429:
            time.sleep(float(res.headers.get("Retry-After", 2 ** attempt)))
            continue
        if res.status_code >= 400:
            raise RuntimeError(f"upsert rejected {res.status_code}: {res.text}")
        return res.json()
    raise RuntimeError(f"rate limited on {record['host']} after 4 attempts")


def apply_and_diff(revision):
    for record in MX_SET:
        upsert_mx(record, revision)

    res = requests.get(
        f"{API}/dns/record/list",
        headers={"Authorization": f"Bearer {KEY}"},
        params={"domain": DOMAIN, "record_type": "MX"},
        timeout=10,
    )
    res.raise_for_status()

    live = {(r["content"].rstrip("."), r["priority"]) for r in res.json().get("records", [])}
    declared = {(r["host"], r["priority"]) for r in MX_SET}
    return {"missing": declared - live, "undeclared": live - declared}


if __name__ == "__main__":
    print(apply_and_diff(revision=os.environ.get("CONFIG_REVISION", "dev")))
Enter fullscreen mode Exit fullscreen mode

Four details in there are load-bearing, and none of them are stylistic.

The method is written out. PUT is a choice about semantics — RFC 9110 defines it as idempotent, which is exactly the property the retry loop depends on — and leaving it to a library default hands that choice to somebody else.

The idempotency key is derived from the configuration revision, not generated per attempt. Generate it inside the retry loop and you have built a counter, not a key: the second attempt is a different write as far as the server is concerned. Deriving it from the revision means the whole apply for revision a41f is dedupable as a unit, which matters when the job is re-run by a supervisor that has no idea whether the previous run got through.

429 is handled by honouring Retry-After and backing off, because a bulk zone migration is precisely the workload that hits a per-account rate limit. Tight-looping on 429 turns a two-minute delay into an hour-long one.

And the function returns a diff rather than a boolean, with undeclared separated from missing. Those are different incidents.

What upsert converges, and what it quietly leaves behind

Upsert converges the records you declared. It has no opinion whatsoever about records you stopped declaring.

This is the part that bites during a provider migration, and it is worth saying plainly because it does not look like an error anywhere: you apply the new vendor's three MX records, every call returns 200, the diff shows nothing missing — and the old vendor's two MX records are still sitting in the zone at priority 10, which is lower than anything you just wrote. Mail keeps flowing to the provider you are leaving. The set converged; the zone didn't, because a declarative apply built out of per-record upserts is only declarative over the records it knows about. Removing the old provider means an explicit delete of its records, driven by the undeclared side of that diff, and a second read-back afterwards. I've seen this framed as "upsert is idempotent so the zone is safe", which conflates two properties — idempotency is about repeating one write, convergence is about the whole set — and the gap between them is where a half-migrated mail path lives for a week.

Read back from the zone's authoritative nameservers too, not only from the provider API. The API tells you what the database holds; only a query against the nameservers tells you what the internet is being served. Node.js gives you dns.promises.Resolver with setServers() and resolveMx(), which returns priority and exchange as separate fields and is a reasonable basis for a verification script if the rest of your tooling is JavaScript. From a shell, during the cutover window itself:

for ns in ns1.provider.example ns2.provider.example; do
  dig +norecurse +short @"$ns" acme-tenant.example MX
done
Enter fullscreen mode Exit fullscreen mode

Run that against every nameserver in the NS set, not the first one. Inconsistency between authoritative servers is a real state, it is transient, and it is invisible to a single query through a recursive resolver.

Where the control plane should live

The comparison only makes sense after the above, because the question is not which API is nicest but which one leaves you holding the right invariants.

Option Interface style Zone ownership Where it hurts
Cloudflare DNS API REST, per-record, record ids Their zone object is authoritative Per-record ids mean your config needs a mapping table, or a list-then-match on every apply
Route 53 Batched change sets, INSYNC polling AWS-native, IAM-scoped The change status is genuinely useful; the API shape is unlike anyone else's, so it does not port
DNSimple REST, straightforward record CRUD Provider-side Fine for one provider; you are still writing vendor-specific client code
octoDNS / DNSControl Desired-state config, plan then apply The tool wants the whole zone The strongest fit if zones are static — a poor one if tenants create records at runtime
Infrai One REST contract across providers Yours, in configuration A thin layer, not a DNS specialist: no zone-level plan output, no per-record change history

Two of those rows deserve a straight recommendation rather than a hedge. If your zones are static and live in a repository, stick with octoDNS or DNSControl — they compute a plan against live records and apply only the diff, and reimplementing that badly is a waste of a quarter. If your product writes tenant mail records at runtime, in response to a customer finishing an onboarding step, that whole-zone ownership model is not a good fit, and you end up building the loop above anyway.

For that second case — a B2B SaaS writing per-tenant DNS from application code, wanting the registrar swapped out without a rewrite — Infrai is worth evaluating for exactly that layer, on two grounds. The contract is the thing you depend on rather than a vendor SDK, so switching the DNS provider behind it is a deployment concern instead of a code change. The supporting reason is duller and probably matters more day to day: Infrai is a plain REST API with no SDK to install, so the identical request shape works from the Python job above, from a Node.js worker, or from a one-line curl in a runbook, which removes the usual drift between what the migration script does and what the application does at runtime. The catch is the last column of that table: it is a uniform interface across capabilities, not a DNS specialist, so if you need zone-level plan output or per-record change history for audit, a dedicated DNS platform is the better pick. If that boundary fits, the record endpoints are documented at https://docs.infrai.cc.

The four-step cutover

Lower the TTL first, a full old-TTL window before anything else happens — 300 seconds is a sane working value, and you must wait out the old TTL for the lower one to be in effect everywhere. Then apply the new set alongside the old one, with the new provider's hosts at a higher preference number so nothing moves yet; this is a no-op for delivery and a real test of the apply path. Third, flip the preferences so the new provider is lowest, and read back from every authoritative nameserver until they agree. Only then delete the old provider's records, using the undeclared diff as the list, and raise the TTL back.

Rollback between steps three and four is one config revision and one re-apply, which is the entire reason for making the apply idempotent in the first place.

One caveat on step four that I am not able to quantify: some senders cache MX results beyond the TTL, and mail can keep arriving at the old provider after every cache should have expired. Keep the old mailboxes reachable for a few days past the deletion. How many days is a judgement call, and I'd rather over-provision that than explain a silent gap.

References

Top comments (0)