DEV Community

CrimsonWave9361502
CrimsonWave9361502

Posted on

Tenant Subdomain Provisioning: Five Rules for Managing Vendor TXT Records and Zone Drift

#dns #txt #multitenancy #python

Creating a tenant subdomain is fast. Publishing the third-party verification TXT records that make it useful is not, and the gap between those two facts is where automated onboarding quietly breaks. If a provisioning job writes a vendor's verification entry and then immediately asks that vendor to check it, go with a pre-stage-then-verify flow instead: write every TXT record the tenant zone needs, wait for your own authoritative servers and a handful of public resolvers to agree, and only then hand the tenant a button that triggers someone else's checker.

The rest is bookkeeping. Careful bookkeeping, but bookkeeping.

The system I'm describing is a healthtech scheduling platform where every clinic gets clinic-042.example.net automatically at signup — appointment reminders, a vanity login host, an SSO tie-in, a status page. Four separate vendors, each wanting proof of control over that name, each with its own checker and its own retry etiquette. The obvious implementation is a single job: call the DNS API, then call each vendor's verify endpoint in the same function. It works on a laptop against a fresh name. It fails in production against a name a resolver has already seen.

1. The negative cache decides your cutover, not your DNS API

Here's the mechanism people skip. When a resolver is asked for _acme-challenge.clinic-042.example.net and the authoritative server answers NXDOMAIN, that "no" gets cached like any other answer. RFC 2308 ties the lifetime of that negative answer to the SOA MINIMUM field (bounded by the SOA record's own TTL), and recommends it stay under three hours. Plenty of zones ship with a SOA minimum of 3600 s because nobody ever revisited the default.

So the sequence that bites you is: health check or preflight probe asks for the name → gets NXDOMAIN → resolver caches the negative answer → your job creates the record 200 ms later → the vendor's checker, resolving through the same public resolver, is told the name does not exist for the next hour. The record is there. The API returned success. The verification still fails, and your retry loop cheerfully burns through its budget against a cached "no".

Two changes fix this and they're both boring. Drop the SOA minimum to something in the 60–300 s range before you automate anything, and stop probing names you are about to create. I'd rather have a provisioning job that writes blind and then polls than one that "checks first" and poisons every cache on the path.

Positive TTLs matter less than people assume during first provisioning, because nothing has cached a positive answer yet. They matter enormously during rotation, which is rule four's problem.

2. Should third-party verification TXT records live in the tenant subdomain or the shared parent zone?

Delegate when the vendor's lookup is anchored at the tenant name, keep it flat when the lookup climbs.

That's the whole decision rule, and it isn't arbitrary — it follows from how each protocol resolves. ACME's dns-01 challenge looks for a TXT record at _acme-challenge.<the name on the certificate>, per RFC 8555, so a per-tenant wildcard cert needs that entry under the tenant's own label. DMARC goes the other way: RFC 7489 has receivers query _dmarc.<domain> and, on failure, fall back to the organizational domain, which means clinic-042.example.net inherits your parent policy unless you publish a record for it or set sp= upstream. CAA climbs too. SPF doesn't climb at all — it's evaluated against whatever domain appears in the envelope sender, so a per-tenant sending subdomain needs its own SPF entry or it has none.

Delegating the tenant label to its own zone buys real isolation: a bad record written for one clinic can't collide with another clinic's RRset, and you can hand a tenant's zone to a different provider later without touching the parent. The catch is that delegation adds a lookup hop and an NS TTL of its own to every cutover, and NS records tend to be published with long TTLs precisely because they're not supposed to change. For a platform onboarding a few clinics a week, one flat parent zone with tenant-prefixed names is easier to reason about and faster to change. Above roughly a few hundred tenants, or when tenants bring their own domains, delegation stops being optional — at that point the shared zone's record count becomes a governance problem in its own right.

Layout Cutover cost Blast radius Fits when
Flat parent zone, tenant-prefixed names One record write, no NS hop Shared RRsets and one SPF budget Tens of tenants, all on your domain
Delegated per-tenant zone Extra NS lookup, NS TTL on every move Isolated per tenant Hundreds of tenants, or tenant-owned domains

Managing other vendors' records in a zone you own is the uncomfortable part of either layout. You're the operator of record for strings whose meaning is defined by someone else's control panel, and nobody sends you a deprecation notice when an account is closed.

3. Treat the zone as a fixture and assert against it

This is where my day job leaks in. I build eval harnesses for RAG pipelines, and the habit that transfers cleanest to DNS is this one: write down the expected output, check the real system against it on a schedule, and make the diff the artifact you review. A zone file is a fixture. Vendor TXT strings are expected values with an owner and an expiry.

# zone_expectations.py — the tenant zone as a test fixture
import dns.resolver  # dnspython 2.x

PUBLIC_RESOLVERS = ["1.1.1.1", "8.8.8.8", "9.9.9.9"]

# Every record carries the team that owns it and the system that depends on it.
EXPECTED = {
    "_acme-challenge.clinic-042.example.net": {
        "owner": "platform", "prefix": "", "purpose": "wildcard cert renewal",
    },
    "clinic-042.example.net": {
        "owner": "billing-ops", "prefix": "mail-vendor-verify=", "purpose": "sending domain proof",
    },
    "_dmarc.clinic-042.example.net": {
        "owner": "platform", "prefix": "v=DMARC1", "purpose": "reporting policy",
    },
}


def txt_values(name: str, server: str, timeout: float = 5.0) -> list[str]:
    r = dns.resolver.Resolver(configure=False)
    r.nameservers = [server]
    r.lifetime = timeout
    try:
        answer = r.resolve(name, "TXT")
    except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
        return []
    # A TXT RRset can hold several records, each split into 255-octet chunks.
    return ["".join(chunk.decode() for chunk in rr.strings) for rr in answer]


def audit(expected: dict) -> list[tuple[str, str]]:
    problems = []
    for name, meta in expected.items():
        seen = {s: txt_values(name, s) for s in PUBLIC_RESOLVERS}
        if any(not v for v in seen.values()):
            problems.append((name, f"missing from at least one resolver: {seen}"))
            continue
        if meta["prefix"] and not any(
            v.startswith(meta["prefix"]) for values in seen.values() for v in values
        ):
            problems.append((name, f"no value with prefix {meta['prefix']!r} ({meta['owner']})"))
    return problems
Enter fullscreen mode Exit fullscreen mode

Run it nightly against every live tenant, fail the build on a non-empty list, and the zone stops drifting in the dark. The unglamorous payoff is orphan detection: any TXT string resolving in production that has no entry in EXPECTED is either an undocumented dependency or garbage from a vendor you stopped paying, and both deserve a ticket.

Zone-as-code tooling — octoDNS and DNSControl are the two most commonly cited open-source options — gives you the other half, rendering the zone from a reviewed repo so nobody hand-edits a record at 2am. The audit checks what resolvers actually return; the repo checks what you meant. They catch different failures and I'd keep both.

4. Budget the record count before you accept another vendor

A shared apex is a finite resource and TXT is where it gets spent. Every record in a TXT RRset at one name is returned together, and RFC 2181 requires all records in that set to carry the same TTL, so a vendor entry you want to rotate in minutes is stuck with whatever TTL its neighbors need. Each TXT string is capped at 255 octets by RFC 1035; longer values get split into chunks that some naive parsers concatenate wrong.

The sharper limit is SPF. RFC 7208 caps evaluation at ten DNS-lookup-causing terms and calls a domain publishing two SPF records a permanent error — one deployment adding a second include: to a domain that already had nine will silently break mail authentication for every tenant sharing that sending domain. Per-tenant sending subdomains fix this by giving each clinic its own budget, which is the strongest practical argument for the per-tenant layout in a healthcare product where a missed reminder is a clinical event and not a growth metric.

Before onboarding vendor number five, write down the record count you'll have at the apex, the SPF lookup count, and which records share a TTL. Ten minutes of arithmetic.

5. Measure three numbers, then decide how fast you can promise

Propagation delay versus cutover speed is the axis this whole design turns on, and it's measurable rather than mystical. Time-to-authoritative is the interval from a successful write to your own nameservers returning the record — seconds on most managed platforms, and the number your DNS vendor's status page actually covers. Time-to-agreement is the interval until a fixed set of public resolvers all return it, which is the one that governs third-party verification. Retry-window fit is whether that second number lands inside the checker's retry policy, which varies wildly between vendors and is usually documented badly.

import time

def wait_for_agreement(name, predicate, deadline_s=600, interval_s=15):
    """Block until every probe resolver returns a value the predicate accepts."""
    started = time.monotonic()
    while time.monotonic() - started < deadline_s:
        results = {s: txt_values(name, s) for s in PUBLIC_RESOLVERS}
        if all(any(predicate(v) for v in values) for values in results.values()):
            return time.monotonic() - started  # log this; it's your propagation budget
        time.sleep(interval_s)
    raise TimeoutError(f"{name} not visible on all probe resolvers in {deadline_s}s")
Enter fullscreen mode Exit fullscreen mode

Log that returned duration for every provisioning run and you get a distribution instead of a folk belief. If the p95 sits comfortably under the strictest vendor's retry window, you can trigger verification automatically and tell the tenant onboarding takes a few minutes. If it doesn't, don't paper over it with a longer sleep — make the tenant-facing step asynchronous, show a pending state, and let the checker be retried by a queue.

A few honest limits. Three public resolvers are a sample, not a guarantee; a tenant's clinic network may run its own forwarder with its own caching behaviour, and nothing you do in your zone fixes that. This approach also assumes you control the parent domain — if tenants bring their own domains, you're back to emailing someone a CNAME and waiting, and no amount of automation on your side changes their registrar's propagation. And if you only onboard a handful of tenants a month, skip the harness; stick with a checklist and a calendar reminder, because a nightly audit job nobody reads is worse than no job at all.

I'm not sure there's a universally correct TTL here. Lower TTLs buy cutover speed and cost query volume; higher ones do the reverse, and where you land depends on how often your vendor set churns. What I am confident about is the ordering: fix the negative cache first, decide the zone layout from how each protocol resolves, then automate — in that order, because automating on top of a 3600 s SOA minimum just makes the wrong thing happen faster.

Sources

Top comments (0)