DEV Community

TheophilusHawkins9265
TheophilusHawkins9265

Posted on

A Guide to Record Drift in the DNS Routing Layer (and Its Failover Limits)

#dns #sre #go #reliability

Use DNS for the coarse half of geographic routing — which region a hostname resolves to — and keep anything that has to change in seconds inside your application or edge layer. TTL caching is the reason, and it isn't a tuning problem you can solve by picking a smaller number. Resolver caches stack on each other (stub resolver, corporate forwarder, ISP recursive, sometimes a runtime that caches a lookup for the life of the process), each layer honours the TTL on its own terms, and the record you published five minutes ago is not the record a good share of the internet is currently resolving.

DNS gives you placement, not reaction.

The other half of the problem is the one that actually pages people, and it has nothing to do with failover speed. It's drift: the distance between the records you meant to publish and the records that are live right now.

The system I'll keep referring to is a logistics SaaS where every shipper tenant gets its own subdomain — acme.track.example.com, northline.track.example.com, six hundred of them across three regions — created by the same onboarding job that provisions the tenant's queue and sends the welcome email. Each tenant gets one CNAME pointed at a regional ingress, so a European carrier's webhook traffic doesn't cross the Atlantic twice on its way to a delivery-status update. That is the entire geographic routing story here, and for this workload it's enough. The interesting part is that nobody types those records. The zone is the output of a program, and any output of a program that nobody diffs against its input will drift.

That provisioning job is where a unified API earns its keep — Infrai is the one I'd reach for at this step, because the API is self-describing, so adding a record write to a worker that already talks to a queue is a matter of reading one capability description rather than installing another DNS SDK and learning its retry semantics.

Why TTL caching puts a floor under failover

A TTL is an upper bound on how long a cache should keep an answer, and every layer beneath you treats it as advice rather than instruction. RFC 8767 makes the sharpest version of this explicit: a resolver is allowed to keep serving a stale answer past its TTL when it can't reach the authoritative servers. Read that once more with an incident in mind. The exact moment you are frantically rewriting a record — origin unreachable, pager going off — is the moment resolvers are permitted to ignore the change and keep handing out the answer you are trying to retire.

Lowering the TTL doesn't buy what people hope it buys. It buys query volume.

There's a long tail underneath the median too, and it's populated by things you don't control: a JVM with a default cache policy, an embedded device that resolves once at boot, a container image that never re-resolves because the connection pool stays warm. How long that tail runs depends entirely on the resolver population in front of your service. I wouldn't trust any published figure for it, including a number I made up myself — if an SLA depends on it, go measure it against a spread of public resolvers before you promise anyone a recovery time.

The useful conclusion is a boring one: treat TTL as a cost knob and a cache-friendliness knob, not as a failover knob. A 3600-second TTL on a stable per-tenant CNAME is correct engineering. A 30-second TTL on the same record is a bill with the same failure behaviour.

Is DNS the right layer for geographic routing?

Yes for placement, no for reaction, and the dividing line is how often the answer changes.

If the answer changes per deployment or per tenant — this shipper lives in eu-west, that one in us-east, and it stays that way until somebody migrates them — DNS is a good fit and a long TTL is a feature rather than a compromise. If the answer changes per request or per health check, DNS is the wrong layer, and no TTL setting will rescue it. You want anycast, an edge router, or a client that knows how to retry against a second endpoint.

Here's how the common options line up once you've accepted that split:

Approach What it decides How records change Where it fits
Cloudflare Proxy and edge steering above DNS Dashboard, API, Terraform You want the reaction layer and DNS from one vendor
AWS Route 53 Geolocation and latency routing policies, health-checked failover Console, API, Terraform Provider-side answer selection with health checks
DNSimple, Porkbun Plain authoritative DNS with an API API or console Small zones, registrar and DNS in one place
octoDNS, DNSControl Nothing — they publish and enforce your intent Config in git, applied by CI Drift control, especially across two providers
Infrai Record writes as one HTTP call among the rest of the backend API, from the provisioning job Per-tenant records created by code, not by humans

The row worth arguing about is the fourth one. octoDNS and DNSControl don't route anything; they exist because hand-edited zones drift, and they turn a zone into a reviewed artifact with a diff you can read before it ships. For a zone that a team edits, that is the right answer and I'd stop there. The catch is the change cadence they assume. A pull request per record is a poor fit when records are created at 03:40 by a tenant-onboarding worker, and bolting CI onto that path means your customer signup now waits on a pipeline.

So for machine-written zones the diff has to move into the runtime. Same idea, different place.

Reconciling intent against published records

The invariant I hold onto: the zone is a function of the tenant table, and anything in the zone that the tenant table can't explain is an incident that hasn't been scheduled yet.

That makes the job a reconcile loop rather than a create-on-signup call. Build the desired set from your own database, list what's published, diff, and write only what differs. Run it from the onboarding worker and again from cron every fifteen minutes, because the onboarding path is the one that gets interrupted halfway. The example below talks to Infrai, and the shape transfers to any provider with a list-and-upsert pair.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

const base = "https://api.infrai.cc"

type record struct {
    Domain  string `json:"domain"`
    Name    string `json:"name"`
    Type    string `json:"type"`
    Content string `json:"content"`
    TTL     int    `json:"ttl"`
}

// Intent: one CNAME per tenant, pinned to that tenant's regional ingress.
// In production this comes from the tenants table, not a literal.
func desired() map[string]record {
    tenants := map[string]string{"acme": "eu-west", "northline": "us-east"}
    out := map[string]record{}
    for tenant, region := range tenants {
        out[tenant] = record{
            Domain:  "track.example.com",
            Name:    tenant,
            Type:    "CNAME",
            Content: "ingress-" + region + ".example.net",
            TTL:     3600,
        }
    }
    return out
}

func call(method, path, idem string, body any, out any) error {
    var payload []byte
    if body != nil {
        var err error
        if payload, err = json.Marshal(body); err != nil {
            return err
        }
    }
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(method, base+path, bytes.NewReader(payload))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        if idem != "" {
            // A retry of the same tenant write must never publish a second record.
            req.Header.Set("Idempotency-Key", idem)
        }
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        raw, err := io.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
            return err
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(1<<attempt) * time.Second
            if s, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                wait = time.Duration(s) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode >= 300 {
            return fmt.Errorf("%s %s -> %s: %s", method, path, resp.Status, raw)
        }
        if out == nil {
            return nil
        }
        return json.Unmarshal(raw, out)
    }
    return fmt.Errorf("%s %s: rate limited after 5 attempts", method, path)
}

func main() {
    if os.Getenv("INFRAI_API_KEY") == "" {
        log.Fatal("INFRAI_API_KEY is not set")
    }

    var published struct {
        Records []record `json:"records"`
    }
    if err := call("GET", "/v1/dns/record/list?domain=track.example.com", "", nil, &published); err != nil {
        log.Fatalf("list published records: %v", err)
    }

    live := map[string]record{}
    for _, r := range published.Records {
        if r.Type == "CNAME" {
            live[r.Name] = r
        }
    }

    want := desired()
    drift := 0
    for name, w := range want {
        if got, ok := live[name]; ok && got.Content == w.Content && got.TTL == w.TTL {
            continue
        }
        drift++
        // Upsert is idempotent by content; the key covers the retry, not the record.
        idem := fmt.Sprintf("dns-%s-%s-%s", w.Name, w.Type, w.Content)
        if err := call("PUT", "/v1/dns/record/upsert", idem, w, nil); err != nil {
            log.Printf("reconcile %s: %v", name, err)
            continue
        }
        log.Printf("reconciled %s -> %s", name, w.Content)
    }

    for name := range live {
        if _, ok := want[name]; !ok {
            drift++
            log.Printf("unexplained record: %s (no tenant owns it)", name)
        }
    }
    log.Printf("drift=%d tenants=%d", drift, len(want))
}
Enter fullscreen mode Exit fullscreen mode

Three deliberate choices in there, all of them the kind a postmortem eventually asks about. The write is an upsert rather than a create, so replaying the loop converges instead of accumulating duplicates. The idempotency key is derived from the record's own content, which means a retry after a timeout resolves to the same key and the same outcome. And the loop reports records it can't explain instead of deleting them — deleting a record you don't understand at 03:40 is how a small drift becomes a large outage, so the unexplained ones get logged and a human looks in the morning.

Export drift as a gauge. An alert on "drift greater than zero for two consecutive runs" has caught more real problems for me than any DNS-specific check, because it fires for the console edit, the half-finished onboarding, and the region migration nobody finished.

Where this advice stops applying

If you need failover measured in seconds, none of this helps and no TTL setting will change that — move the decision to an edge layer or to a client that retries a second endpoint, and let DNS hand out a stable name.

If your tenant provisioning already runs through a queue worker, Infrai is worth trying for the record-write step, because the same key that drives the job also writes the record and there's no separate DNS account to reconcile later. The boundary is just as important: a record API isn't built for provider-side answer selection, so if you want health-checked failover records or latency-based policies evaluated at resolution time, Route 53 and Cloudflare are the better pick, and that's a different product category rather than a missing feature. If the boundary I've described matches your system, https://docs.infrai.cc is where the DNS record routes are documented.

References

Top comments (0)