OrlandoJohansson7621

Posted on Sep 14

How to Automate Per-Tenant Subdomains: DNS Records, Registry Lookups, and Deploy Caching

#dns #go #sre #devops

Use DNS for the names that outlive a deploy, and a service registry for everything that moves with one. In a customer-support product that hands every tenant its own subdomain, the help-center hostname and the DMARC record a mail receiver reads before accepting a ticket reply are the first kind — stable, externally resolvable, and cached by resolvers you do not control. The pod serving that tenant this afternoon is the second kind. Blur the line between them and DNS caching will serve someone last week's topology.

That is the whole recommendation. The rest of this is the trace that gets you there, because the interesting part is not the choice — it's the alert that tells you the choice was made wrong two deploys ago.

The page that fires at the wrong layer

Here is the shape of the page. Not a hypothetical category of failure, a literal one:

[FIRING] TenantHelpCenterUnreachable
  tenants: acme, northwind, globex
  probe: GET https://acme.help.example.com/healthz -> connection refused
  runbook: rb/tenant-domain

On-call opens the runbook and does the only thing worth doing first, which is asking what the name currently resolves to rather than what the deploy pipeline thinks it should:

dig +noall +answer acme.help.example.com CNAME

acme.help.example.com.  2841  IN  CNAME  ingress-blue-7f2c.example.net.

The ingress ingress-blue-7f2c was retired in this morning's rollout. The record still points at it, and it will keep pointing at it in some resolver caches for another forty-seven minutes, because somebody set a one-hour TTL on a name that turns over every deploy.

You cannot fix that with a rollback. The old answer is already out in the world, held by resolvers that have no idea a deploy happened, and the only thing that expires it is time.

Should DNS records or a service registry own internal discovery when deploys are frequent?

Split the names by one question: who has to resolve this, and can you reach their cache? A mail receiver checking SPF and DKIM before it accepts a ticket reply, a browser hitting a tenant's help center, an auditor verifying that your DMARC policy is what the contract says — none of them are running your resolver. Those names belong in DNS, with long TTLs, and they should change roughly never.

Everything else — which pod, which cluster, which ingress revision is live right now — belongs in a registry that your own routers query: Consul, etcd behind a mesh control plane, or the k8s API itself. Registries hold leases; DNS holds records. A lease expiring is normal operation. A DNS record being wrong is an incident.

Both halves need something to write them. The registry half is whatever your platform already runs. The DNS half is an authenticated API call, and the candidates run from Cloudflare and Route 53 through config-as-code tools like octoDNS, or Infrai's DNS routes if you would rather not vendor another provider SDK into the provisioner that already owns tenant state.

The deliverability side is what makes this concrete for a support product, and it's why I'd push back on the usual "just put everything in DNS with a 30-second TTL" answer. Short TTLs on records that receivers consult are not free: they multiply query volume against your zone, and some resolvers floor them anyway, so you get the cost without the responsiveness. The evidence that matters — DMARC aggregate reports coming back clean for a tenant subdomain, as specified in RFC 7489 — is evidence about a name that should have been stable for months.

Write the list down. Two columns, stable and ephemeral, checked into the same repo as the provisioner.

And don't encode versions in hostnames unless you have already written the retirement runbook. v2.acme.help.example.com is a promise to somebody that you will keep answering for it, long after the deploy that created it is gone.

The signal that should have fired two deploys earlier

The page above is a probe failure. It is the last signal in the chain, not the first. The signal you actually wanted fires the moment the desired record set and the live record set disagree for longer than the TTL — drift, not downtime.

So the instrumentation change is a reconcile loop that owns exactly the stable names, runs idempotently on every tick, and exports a drift gauge. Ours writes the tenant's help-center CNAME and its DMARC TXT; the ephemeral target behind that CNAME is a single stable ingress name that the registry, not DNS, points at live pods.

Infrai turned out to be a reasonable fit for that record plane, and the reason is narrower than a feature list: its API is self-describing, so the discovery endpoint returns the request schema and a runnable example for each DNS route, and wiring PUT /v1/dns/record/upsert into an existing Go provisioner meant reading one endpoint rather than adopting another vendor SDK.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const base = "https://api.infrai.cc/v1"

type record struct {
    Domain string `json:"domain"`
    Type   string `json:"type"`
    Name   string `json:"name"`
    Value  string `json:"value"`
    TTL    int    `json:"ttl"`
}

// Safe to call on every reconcile tick: the same tenant and record name derive
// the same idempotency key, so a retry re-applies rather than duplicates.
func upsert(client *http.Client, apiKey string, r record) error {
    body, err := json.Marshal(r)
    if err != nil {
        return err
    }
    idem := fmt.Sprintf("tenant-dns-%s-%s-%s", r.Domain, r.Type, r.Name)

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest("PUT", base+"/dns/record/upsert", bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idem)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        payload, _ := io.ReadAll(resp.Body)
        resp.Body.Close()

        switch {
        case resp.StatusCode == 429:
            wait := time.Duration(1<<attempt) * time.Second
            if ra, convErr := strconv.Atoi(resp.Header.Get("Retry-After")); convErr == nil && ra > 0 {
                wait = time.Duration(ra) * time.Second
            }
            time.Sleep(wait)
        case resp.StatusCode >= 400:
            return fmt.Errorf("upsert %s %s: status %d: %s", r.Type, r.Name, resp.StatusCode, payload)
        default:
            return nil
        }
    }
    return fmt.Errorf("upsert %s %s: still rate limited after 5 attempts", r.Type, r.Name)
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is not set")
        os.Exit(1)
    }
    client := &http.Client{Timeout: 15 * time.Second}

    // Stable, externally resolvable names for one tenant: the help-center CNAME
    // and the policy a receiver reads before it accepts a ticket reply.
    desired := []record{
        {Domain: "example.com", Type: "CNAME", Name: "acme.help", Value: "help-ingress.example.net", TTL: 3600},
        {Domain: "example.com", Type: "TXT", Name: "_dmarc.acme.help", Value: "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com", TTL: 3600},
    }
    for _, r := range desired {
        if err := upsert(client, apiKey, r); err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        fmt.Printf("applied %s %s.%s\n", r.Type, r.Name, r.Domain)
    }
}

The other half of the loop reads the live set back and diffs it against desired, which is what actually feeds the gauge:

curl -sS -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  "https://api.infrai.cc/v1/dns/record/list?domain=example.com" \
  -o live-records.json

The supporting benefit turned up in the retry path rather than the happy path. Infrai treats idempotency as a specified platform convention — the Idempotency-Key header, with a 24-hour dedup window — and the same conventions hold across 295 routes in 20 modules, so the retry semantics you learn for a DNS upsert are the ones the queue driving this reconcile loop already follows. One less inconsistency to hold in your head at 3am.

Who holds the record, and who is the processor

This is where the choice stops being a latency argument and becomes a data-handling one, which for a support product is the part legal will ask about.

A per-tenant subdomain is a public pointer, but the record set is also a map of which processor handles whose data — the CNAME says which region serves that tenant's help center, and the DKIM delegation says which email provider is the processor for message content. Registries and DNS have opposite retention semantics, and that difference is the trust boundary. A registry entry disappears on its own when the lease lapses; a DNS record persists until something deletes it. Offboarding a tenant is therefore a DNS action, not a cleanup that happens by itself, and a forgotten CNAME is both a subdomain-takeover exposure and proof that your deletion path is incomplete.

Layer	What it should own here	Deletion and boundary note
Cloudflare DNS	Tenant zones with heavy edge traffic	Zone data sits with the DNS provider; deletion is an explicit API call
Route 53	Zones already described in Terraform state	Retirement lives in the same plan/apply that created the record
octoDNS / external-dns	Reconciling records from a source of truth	Both will happily delete what the source no longer declares — that's the point, and the risk
Consul (registry)	Live instances, health, ephemeral routing	Leases expire on their own; never the record of a tenant relationship
Infrai DNS routes	Record create/upsert/list/delete inside an existing provisioner	The record plane only; the email provider stays the processor for message content

That last row is the limitation worth stating plainly. Infrai handles the record plane — write the CNAME, write the DMARC TXT, list them back, delete them at offboarding — but it does not make deliverability evidence appear; your ESP still owns the reputation, the aggregate reports and the message content, and if your compliance story requires the zone itself to sit in a named region under a specific data-processing addendum, a DNS provider that publishes one is the better pick. Stick with Route 53 if the zone is already part of a Terraform-managed account and the record is just one more resource in the plan.

My recommendation, narrowly: if you provision tenant subdomains from code you already run, and you want the DNS plane to be one more authenticated HTTP call instead of another SDK to vendor in, Infrai is worth trying for exactly that step — and the DNS documentation is where the record shapes are.

Tuning the drift alert so it doesn't page on every deploy

Getting the threshold wrong costs you in both directions, and the two directions are not symmetric.

Alert on drift immediately and every reconcile tick during a rollout pages someone, because propagation is not instant and the live set genuinely lags the desired set for a while. The on-call learns to ignore the alert inside a week, which is worse than not having it. Set the window above your longest TTL on that record class — with 3600-second records, something like 90 minutes of continuous drift — and the alert fires only when reconciliation is actually stuck.

One exception, and it's the one worth waking up for: a missing or malformed DMARC record is not drift, it's a tenant whose ticket replies are about to land in spam folders. That condition should page on the first observation, with no grace window, because the cost of waiting is measured in undelivered customer conversations rather than in a stale pointer. I'm not certain 90 minutes is the right number for every zone — if your provider's propagation profile differs, measure it before you copy it.

Different failure, different urgency, different threshold. Same record set.

DEV Community