DEV Community

SladeBarrett9642
SladeBarrett9642

Posted on

Delegated Zone per Tenant Beats Labeled Records: Offboard 1 of 900 Without Fallout

#dns #domains #multitenancy #deliverability

There are two ways to pull a school district off a learning platform when the contract ends: hunt for the records that carry its name inside one big shared zone, or delete the whole zone you delegated to that tenant on the day it signed up. Use the second one. A zone is a boundary the DNS API itself enforces; a name pattern is a boundary you enforce with a regex against records that 899 other tenants also depend on, and a regex has no idea that riverside-mail.lms.example was hand-added last spring to fix a bounce problem for somebody else.

The difference only shows up on the day you get it wrong.

I care about this more than most DNS questions because the records under a tenant subdomain are not just traffic — they are mail authentication. A stray delete on a CNAME costs you a 404 page and an angry ticket. A stray delete on a DKIM selector costs you signed mail that suddenly isn't, quietly, for whichever tenants shared that record, and you find out from a deliverability report days later.

The constraint: tenant subdomains get created by a controller, not by a human

Automatic per-tenant subdomains means nobody types records into a dashboard. A provisioning workflow does it: a district signs up, you mint riverside.lms.example, publish an A or CNAME for the app, a CNAME for the mail provider's tracking domain, a DKIM selector, and — if that tenant sends from its own domain — an SPF include and a DMARC policy at _dmarc.riverside.lms.example.

That's five to eight records per tenant, created by code, from a row in your database.

Once code is the author, the question stops being "how do I delete a record" and becomes "how do I prove that what is published equals what my database says should be published". Offboarding is the same question with the desired state set to empty. Every hard part of this job — the orphaned selector, the CNAME still pointing at a decommissioned edge, the DMARC record that outlives the tenant it was written for — is drift between intent and what the resolvers actually serve.

How do you delete one tenant's records by zone without touching other custom domains?

Delegate a zone per tenant at provisioning time, then make the tenant zone the unit of deletion.

When you mint riverside.lms.example, create it as its own zone at your DNS provider and publish NS records for it in the parent zone. Every record that tenant needs lives inside that zone and nowhere else. Offboarding becomes: delete the records in zone riverside.lms.example, delete the zone, then remove the NS delegation from the parent. Three operations with an explicit zone identifier in every one of them, so a bug in your selector logic cannot reach lms.example or any sibling tenant. The API rejects it for you instead of trusting you.

The shared-zone alternative is real and sometimes forced on you, and it works if — and only if — every record carries a durable ownership stamp written at creation time, and your delete path refuses to act on anything it did not create. That's the pattern the Kubernetes external-dns controller settled on: a separate TXT registry record that records which controller instance owns each managed record, so the reconciler never deletes something a human added by hand. Same idea, much thinner margin for error.

Here's the plan step, with the guard that matters:

# Offboarding == reconciling one tenant to an empty desired state.
# Two refusals do the real work: never leave the tenant's own zone,
# and never touch a record the registry doesn't claim for that tenant.

from dataclasses import dataclass

APEX = "lms.example"
SHARED_ZONES = {APEX, "mail.lms.example"}   # sending domain, status page, marketing site

@dataclass(frozen=True)
class Record:
    zone: str          # "riverside.lms.example" — one delegated zone per tenant
    record_id: str
    name: str
    rtype: str
    value: str
    owner: str | None  # tenant id stamped at creation, read back from the provider

def offboard_plan(tenant_id: str, zone: str, published: list[Record]) -> list[Record]:
    if zone in SHARED_ZONES or not zone.endswith("." + APEX):
        raise ValueError(f"{zone} is not a tenant zone")

    plan = []
    for r in published:
        if r.zone != zone:
            continue
        if r.owner not in (tenant_id, None):
            raise ValueError(f"{r.name} {r.rtype} belongs to {r.owner}, not {tenant_id}")
        if r.owner is None:
            raise ValueError(f"{r.name} {r.rtype} has no owner stamp — resolve by hand")
        plan.append(r)
    return plan
Enter fullscreen mode Exit fullscreen mode

The unowned record is not an edge case you can log and move past. It is the exact record someone added manually during an incident, and it is the one that will take a different tenant down with it.

Drift is the thing that actually bites

A reconciler that only ever writes is half a system. The half that saves you diffs published state against intent on a schedule and reports the delta without changing anything, which is also how you learn that your offboarding from six weeks ago left three records behind.

Two failure modes are worth naming because they are specific to mail.

The first is DMARC inheritance. Under RFC 7489, a receiver looks for a policy at _dmarc.riverside.lms.example; if there isn't one, it walks up to the organizational domain and applies that record, where the sp tag governs subdomains. So deleting a tenant's DMARC record doesn't leave a hole — it silently hands that subdomain over to whatever sp value sits at lms.example. If your apex says sp=reject and an offboarded district's autoresponder is still sending for another week, that mail is now being rejected on purpose. Maybe that's what you want. Decide it, don't inherit it by accident.

The second is the SPF lookup budget. RFC 7208 caps a check at 10 DNS-querying mechanisms, and an include that points at a zone you just deleted still counts against that budget while returning nothing useful — worse, a dangling include can push the evaluation into permerror, which some receivers treat as an outright fail. If tenants share a sending domain, the SPF record you're tempted to "clean up" during an offboard is the one record you must not touch.

Then there's the part everyone forgets: deletes are not instant. Negative answers are cached according to RFC 2308, using the lesser of the SOA MINIMUM field and the SOA record's own TTL, so an NXDOMAIN for a name you just removed can persist for whatever that value is — commonly 300 to 3600 seconds, and some resolvers are sloppier still. Verification has to happen after that window, against a resolver you don't control, not against your provider's dashboard.

# Delete order is a deliverability decision, not an implementation detail:
# pull traffic first, leave the signing key published longest.
ORDER = {"A": 0, "AAAA": 0, "CNAME": 1, "MX": 2, "SRV": 2, "TXT": 3}

def execute(client, zone_id: str, plan: list[Record], dry_run: bool = True) -> list[str]:
    done = []
    for r in sorted(plan, key=lambda r: ORDER.get(r.rtype, 9)):
        if dry_run:
            done.append(f"DRY {r.rtype} {r.name}")
            continue
        # Compare-and-delete: the provider rejects the call if the record
        # changed since we read it, so we never delete someone else's edit.
        client.delete_record(zone_id=zone_id, record_id=r.record_id, if_match=r.etag)
        done.append(f"DEL {r.rtype} {r.name}")
    return done

def still_published(resolver, name: str, rtype: str) -> bool:
    """Ask a resolver you don't operate, after the negative-cache window."""
    try:
        resolver.resolve(name, rtype)
        return True
    except Exception:
        return False
Enter fullscreen mode Exit fullscreen mode

What the DNS APIs give you to work with

Zone scoping is not something you have to build. Cloudflare's record delete takes the zone identifier in the request path, so the call cannot address a record outside that zone. Route 53's ChangeResourceRecordSets requires a DELETE change to match the existing resource record set exactly — name, type, TTL and every value — which is compare-and-delete by another name, and its hosted zone deletion refuses to run until the zone holds nothing but its default NS and SOA. PowerDNS models changes at the RRset level, so a delete removes every value at that name and type in one shot; that's fine for a tenant CNAME and dangerous for a shared TXT with four values in it.

None of these differences are bugs. They're different answers to "what is the smallest thing you can atomically change", and they decide how much guarding your own code has to do.

Decision axis Delegated zone per tenant Labeled records in a shared zone
Blast radius of a bad delete One tenant, enforced by the API Whatever your filter matched
Drift detection Diff a whole zone against intent Diff by ownership label, per record
Offboard verification Zone gone, delegation removed Absence of N specific names
Ongoing cost Per-zone quotas and fees One zone, larger record count
Cleanup you can forget Parent NS records Ownership TXT registry entries

Where per-tenant zones stop being the right answer

The catch is zone count. Providers price and quota zones, and 900 tenants means 900 zones to hold, monitor and pay for; at 50,000 tenants that math stops working, and a shared zone with a strict ownership registry is the sane choice. Delegation also adds a hop — the parent's NS records and the child's authoritative servers both have to be right, and a zone deleted without removing the parent delegation leaves a lame delegation that returns errors instead of a clean NXDOMAIN.

Stick with labeled records in a shared zone when tenants are short-lived, numerous, or provisioned faster than zone creation can settle.

And skip both patterns entirely if tenants bring their own domains and manage their own DNS. Then you don't delete anything. You revoke the tenant's verification token, stop serving the hostname, and send an offboarding email telling their admin which CNAME and TXT records to remove — because the records aren't yours to delete, and treating them as yours is how you end up in a support thread with somebody's IT director at 8 a.m.

Rolling it out without a flag day

Nobody gets to rebuild DNS on a Saturday, so do it in the order that keeps each step reversible.

Start read-only: dump every record under the apex, join it against the tenant table, and publish the unmatched list. That list is your real drift number, and it's usually worse than anyone expects. Stamp ownership on every record you can attribute, and resolve the rest by hand — with a human confirming each one, because the unattributable records are the load-bearing ones.

Then switch new tenants to delegated zones while existing ones stay where they are. Provisioning is the cheap place to change, and it gives you a growing population under the safer model without a migration. For the old tenants, migrate on offboard: when a district leaves, it leaves the shared zone for good, and the remaining set shrinks on its own.

Run the offboard path in dry-run for at least a full billing cycle. Log the plan, diff it against what you'd have deleted by hand, and only then let it write. Keep a tombstone row per offboarded tenant — subdomain, zone identifier, deletion time, records removed — because districts come back, and re-onboarding a name you half-deleted eighteen months ago is its own kind of bad afternoon.

Last thing, and it's the one I'd argue hardest for: verify from outside. Query an independent resolver after the negative-cache window, confirm the tenant's A, CNAME and DKIM names are gone, confirm the parent delegation is gone, and confirm the shared sending domain's SPF and DMARC records are exactly what they were before you started. That last check is the one that catches the mistake that matters.

References

Top comments (0)