DEV Community

DemetriusReed2163
DemetriusReed2163

Posted on

Per-Tenant Subdomain Zones: Lowering DNS TTL Early for a Planned Cutover

#dns #ttl #python #architecture

Pick the zone owner before you pick a TTL number. Use a long DNS TTL for a tenant subdomain that isn't moving — 3600 seconds is a sane steady-state default — then drop it to 300 seconds one full TTL period ahead of a planned cutover, and restore the long value once traffic looks right. Lowering the TTL on the morning of the cutover changes nothing for a resolver that already cached the old answer under the old lifetime.

That part gets paid for twice.

I ship tenant-facing features for property management software, where every property manager who finishes onboarding gets its own subdomain automatically, no ticket, no human in the loop. harborview.rentals.example.com on day one. Six months later their marketing lead decides the resident portal should live at portal.harborviewproperties.example, and now the hostname sits in a zone I have no authority over at all — same application, same routing table, a completely different change process.

TTL selection is downstream of that.

The data flow is short. Onboarding writes a tenant row; a DNS worker renders that row into a record in whichever zone the tenant belongs to; a verification step polls the authoritative nameserver until the new answer shows up; the edge starts accepting the hostname. The TTL gets written in step two and decided in step one.

A pre-flight that reads the TTL you actually have

Most cutover checklists start by looking up the record. The trouble is that a recursive resolver hands back the remaining lifetime of whatever it cached, not the value configured in the zone, so a pre-flight that asks a public resolver can report a comfortable two-digit number while the zone itself is still publishing 86400. Ask the zone's own nameserver instead.

import dns.message
import dns.query
import dns.rdatatype
import dns.resolver


def authoritative_ttl(fqdn: str, rdtype: str = "CNAME") -> int | None:
    """Read the configured TTL straight from the zone's nameserver."""
    zone = dns.resolver.zone_for_name(fqdn)
    ns_name = dns.resolver.resolve(zone, "NS")[0].target
    ns_ip = dns.resolver.resolve(ns_name, "A")[0].address

    want = dns.rdatatype.from_text(rdtype)
    reply = dns.query.udp(dns.message.make_query(fqdn, want), ns_ip, timeout=5)
    for rrset in reply.answer:
        if rrset.rdtype == want:
            return rrset.ttl
    return None


TENANT_HOSTS = {
    "harborview": "harborview.rentals.example.com",   # platform-owned zone
    "oakline": "portal.oaklineproperties.example",    # customer-owned zone
}

if __name__ == "__main__":
    for tenant, host in TENANT_HOSTS.items():
        print(f"{tenant:12} {host:42} ttl={authoritative_ttl(host)}")
Enter fullscreen mode Exit fullscreen mode

That runs on dnspython 2.7 and takes about a second per tenant. I run it as a nightly job over the whole tenant table and store the result next to the tenant row, because the number I care about on cutover day is the one from three days ago — the one that tells me whether the pre-lowering actually landed.

Zone ownership decides how much TTL control you have

Here's the uncomfortable bit. The TTL you can change is the TTL on a record you publish, and in a multi-tenant product a large fraction of your hostnames are records somebody else publishes.

Zone model Who sets the record TTL Realistic lead time for pre-lowering
Platform-owned per-tenant record under your apex You One TTL period, scheduled by your own worker
Customer-owned zone with a CNAME pointing at you The customer's registrar or DNS account Days, and only if someone answers the email
Delegated subzone, customer publishes NS records You, below the delegation point One TTL, plus the NS record TTL they chose

The third row is the one worth arguing for during sales calls. If a property manager delegates portal.theirdomain.example to your nameservers with an NS record, every record underneath becomes yours, including its cache lifetime, and the customer's only remaining lever is the delegation itself. The trade-off is real: you now operate DNS for a name that carries their brand, and an outage there reads as their outage.

For the customer-owned CNAME row, the honest engineering answer is that you do not have a TTL strategy. You have a request. Build the change calendar around a lead time you can't compress, and stop pretending the number is under your control.

Should I pre-lower the DNS TTL for a planned tenant subdomain cutover?

Yes, for any zone you publish — and the mechanism is the reason, not the ritual. RFC 1035 defines the TTL as the interval in seconds that a record may be cached before it is discarded, so the earliest a resolver can learn about your new 300-second policy is the next time it refetches the record, which can be a full old-TTL away. Pre-lowering buys that refetch in advance. Skip it and your 300-second promise silently becomes an 86400-second one for every resolver that fetched an hour before you started.

Three things make the real propagation window messier than the arithmetic.

Resolver software clamps. Unbound exposes cache-min-ttl and cache-max-ttl, and an operator who sets a floor of 300 or 600 seconds will hold your 60-second record for their floor instead — you cannot make a resolver poll faster than its own policy allows. RFC 8767 goes further and blesses serving stale data past expiry when the authoritative servers are unreachable, which is excellent for resilience and terrible for your mental model of a clean cutover. And RFC 2181 requires every record in an RRSet to share one TTL, so a per-record override inside a set does not survive; the set moves together or not at all.

Mail is the other trap. A tenant subdomain that sends rent reminders drags TXT records into the cutover alongside the CNAME — SPF, a DKIM selector, and the DMARC record at _dmarc.<host>. RFC 7489 defines an sp tag that sets policy for subdomains of the publishing domain, which means a tenant subdomain with no DMARC record of its own inherits from the organizational domain. Cut the subdomain over to a new sending path without moving those records in the same window and the first thing you notice is a quarantine rate, not a 404.

The catch with short TTLs is that they are only cheap while they're temporary. I'm not convinced there's a universally correct steady-state number — the resolver mix in front of a residential rental portal looks nothing like the mix in front of a commercial property back office, and your mileage may vary. What I'd defend is the shape: long for stability, short for the window, explicit in configuration either way, never inherited from a provider default.

What a short TTL costs when nothing is changing

Arithmetic, mostly. A record published at 60 seconds can be refetched up to sixty times more often by the same caching resolver than one published at 3600, and each miss adds a full resolution round trip to the first request of a session. For a worker pool that opens fresh connections per job — the way most background pipelines do — that surfaces as p99 latency you will spend an afternoon misattributing to whatever API sits at the other end. Keep the window short, then close it.

Running the cutover as a graded change

I treat a tenant cutover the way I treat a model change: it doesn't ship until a gate passes, and the gate is code, not a person squinting at dig output. Two days out, the worker lowers the TTL on every platform-owned record in the batch and opens a task for each customer-owned zone, with a restore step written into the same ticket so nobody forgets to raise it back. One day out, the gate runs against several public resolvers and has to come back empty.

import dns.resolver

PUBLIC_RESOLVERS = ["9.9.9.9", "1.1.1.1", "8.8.8.8"]


def pre_cutover_gate(fqdn: str, current_target: str, max_ttl: int = 300) -> list[str]:
    """Empty list means every resolver is ready for a fast change."""
    failures = []
    for ip in PUBLIC_RESOLVERS:
        resolver = dns.resolver.Resolver(configure=False)
        resolver.nameservers = [ip]
        answer = resolver.resolve(fqdn, "CNAME")
        seen = str(answer[0].target).rstrip(".")
        if seen != current_target.rstrip("."):
            failures.append(f"{ip}: unexpected target {seen}")
        elif answer.rrset.ttl > max_ttl:
            failures.append(f"{ip}: ttl {answer.rrset.ttl}s above {max_ttl}s")
    return failures
Enter fullscreen mode Exit fullscreen mode

On the day itself the order matters more than the speed: publish the new record, keep the old target serving for at least one old-TTL period so a rollback is a DNS change rather than a restore, watch per-tenant request volume on both targets until the old one flatlines, then raise the TTL back and close the ticket. The rollback target and the restore time belong in the change record before anyone touches production, because the half-state where one engineer has raised the TTL while another is still watching stale caches is where the confusing incidents come from.

Two edges are worth naming. A short TTL is not a good fit as a permanent posture for a high-volume apex record, and it won't help at all during an unplanned failover — by then the caches are already full, and the only lever left is how long you're willing to serve both targets. If your cutovers are genuinely unpredictable, stick with a longer overlap window and dual-serving instead of chasing propagation speed you can't buy after the fact.

DNS is shared infrastructure that other people's software caches on your behalf. Treat the cache lifetime as part of the release plan and most of the drama disappears.

References

Top comments (0)