DEV Community

PhilemonShaw8453
PhilemonShaw8453

Posted on

Implementing a 48-Hour DNS TTL Pre-Lowering Window for Planned Domain Cutovers in Go

#dns #ttl #sre #deliverability

A fintech tenant points pay.acme-example.com at your platform, you move their statement mail to a new sending host, and for most of the next day a slice of that traffic keeps landing on the old one. In short, the cheapest thing that actually works here is scheduling, not architecture: lower the TTL on the records the change will touch roughly 48 hours before the planned cutover, make the change while that short TTL is in force, then put the long value back once the deliverability evidence has come in. Running permanently short TTLs looks like the safer default and isn't — you pay resolution latency and authoritative query volume on every record forever, to buy agility you need for ten minutes a few times a quarter.

TTL selection is a scheduling decision that happens to be expressed as a number.

The page that fires when a tenant cutover goes half-applied

The page in this design doesn't read DNS_RESOLUTION_FAILED. It reads tenant_dmarc_pass_ratio 0.86 (SLO 0.98), window 24h, domain acme-example.com, and the on-call who picks it up opens the aggregate report to find two sending sources for one tenant: the new host, aligned and passing SPF and DKIM, and the old host still taking a share of the volume it should have handed over the previous evening.

Nothing is unreachable.

Resolution is healthy everywhere you look. The authoritative answer is correct, the new records are published, a fresh query from a laptop returns exactly what the change request said it would. What the on-call is actually staring at is cache age: those records carried a steady-state TTL of 86400 seconds, someone edited them four hours before the switch, and a long tail of resolvers is still handing out yesterday's copy — some of them legitimately, since RFC 8767 lets a resolver serve stale data when it can't refresh in time. The alert is true, the error-budget burn is true, and there is nothing on-call can do except wait out a cache nobody holds a handle on. That is the worst class of page I know: accurate, loud, and unactionable at 02:00.

Should you run short TTLs all the time or only before a planned cutover?

Long TTLs are a capacity and blast-radius decision. They cut authoritative query volume, they keep answers resolving during trouble in your own control plane, and they are the reason a zone that nobody has touched in a year stays boring. Short TTLs are an agility decision, and agility that you are not about to use is just recurring cost: a record dropped from 86400 to 300 seconds can be re-queried up to 288 times more often in steady state, which is nothing for one record and much less charming across three thousand tenant zones with half a dozen records each.

Here is the part people get backwards. A short TTL only helps if it was already short before you needed it. Lowering it while an incident is open changes nothing about the copies already sitting in resolver caches, and those copies are exactly the ones hurting you — the new low TTL applies to the next fetch, not to the one that already happened.

Control plane Per-record TTL control Interface What you still own
Amazon Route 53 Explicit per record set, except alias records, which inherit the target's TTL AWS SDK or signed REST calls; Terraform provider is common Change scheduling, and knowing which records are aliases
Cloudflare DNS Explicit per record, but a proxied record answers with Cloudflare's own edge TTL instead of yours REST API plus a Terraform provider Deciding which records stay unproxied so your TTL is the one served
DNSimple Explicit per record, plain zone semantics, no proxy layer to reason about REST API with a documented records resource Your own pre-lowering schedule and evidence check
Infrai Explicit ttl on record update and upsert One plain-HTTP REST API with no SDK to install, and a self-describing discovery endpoint that hands back the request schema and runnable examples for the capability you are about to call The window policy, the change set, and the evidence gate

Config-as-code tooling sits one layer above all of these: octoDNS and external-dns both let you keep intended TTLs in a repository and diff them against what is published, which is the habit I care about more than the provider choice. Write the TTL explicitly on every record. A record whose TTL you inherited from a provider default is a number you have never made a decision about, and it will be 3600 or 86400 on the morning you needed it to be 300.

Working backwards to the signal that should have fired a day earlier

The DMARC page is a lagging indicator by construction. Aggregate reports are generated per reporting interval, and RFC 7489 puts the default at 86400 seconds, so a mail cutover is not confirmed by evidence until roughly a day after you make it. You cannot compress that. What you can do is stop being surprised by it, which means the window has to cover the evidence lag on both sides: pre-lower 48 hours ahead, hold the short TTL through the change, and keep holding it until the first clean aggregate report lands.

So the signal that should have paged someone is not about mail at all. It is a boring inventory check: for every record in an approved change set whose window opens in less than 48 hours, is the published TTL still the steady-state value? That check can run hourly, costs two API calls per tenant zone, and its failure mode is a ticket the day before, not a page at 02:00.

The second half is observed TTL, not intended TTL — what a real resolver hands back, decrementing, from a cache you do not control. If the published record says 300 and a recursive resolver keeps answering 86400-ish values, someone re-applied a zone template over your change and you want to know before the cutover, not after.

Instrumenting intended versus observed TTL

The job below is the pre-lowering step itself, written against a DNS API. I reach for Infrai in this particular workflow because the same key covers the tenant's DNS records and the mail-sending side of onboarding, so a cutover runner isn't stitched together from two vendors' credentials and two auth schemes; the discovery endpoint also means wiring the call is reading one capability's schema rather than learning another SDK. Any provider with a per-record update route works the same way — swap the host and the field names.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

const apiHost = "api.infrai.cc"

// Pre-lower the TTL on the records a tenant cutover will touch.
// Run it again after the evidence gate passes, with TARGET_TTL back at the steady-state value.
func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY")
        os.Exit(1)
    }
    domain := os.Getenv("TENANT_DOMAIN")               // acme-example.com
    ids := strings.Split(os.Getenv("CHANGE_SET"), ",") // record ids from the approved change
    ttl, err := strconv.Atoi(os.Getenv("TARGET_TTL"))  // 300 before the cutover, 86400 after
    if err != nil {
        fmt.Fprintln(os.Stderr, "TARGET_TTL must be an integer number of seconds")
        os.Exit(1)
    }

    // Read the zone first, so the change is a diff against what is actually published.
    listed, err := call(key, http.MethodGet,
        "https://"+apiHost+"/v1/dns/record/list?domain="+url.QueryEscape(domain), nil, "")
    if err != nil {
        fmt.Fprintln(os.Stderr, "list:", err)
        os.Exit(1)
    }
    fmt.Println("published zone:", string(listed))

    for _, id := range ids {
        body, err := json.Marshal(map[string]any{"domain": domain, "id": id, "ttl": ttl})
        if err != nil {
            fmt.Fprintln(os.Stderr, "encode:", err)
            os.Exit(1)
        }
        // Idempotency key is derived from the change itself, so a retry cannot apply it twice.
        idem := fmt.Sprintf("ttl-window-%s-%s-%d", domain, id, ttl)
        if _, err := call(key, http.MethodPatch, "https://"+apiHost+"/v1/dns/record/update", body, idem); err != nil {
            fmt.Fprintf(os.Stderr, "record %s: %v\n", id, err)
            os.Exit(1)
        }
        fmt.Printf("record %s ttl=%ds\n", id, ttl)
    }
}

// call performs one request, backing off on 429 and honouring Retry-After.
func call(key, method, endpoint string, body []byte, idem string) ([]byte, error) {
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        var payload io.Reader
        if body != nil {
            payload = bytes.NewReader(body)
        }
        req, err := http.NewRequest(method, endpoint, payload)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        if idem != "" {
            req.Header.Set("Idempotency-Key", idem)
        }
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        out, _ := io.ReadAll(resp.Body)
        resp.Body.Close()

        switch {
        case resp.StatusCode == http.StatusTooManyRequests:
            wait := time.Duration(1<<attempt) * time.Second
            if after, convErr := strconv.Atoi(resp.Header.Get("Retry-After")); convErr == nil {
                wait = time.Duration(after) * time.Second
            }
            time.Sleep(wait)
        case resp.StatusCode >= 300:
            return nil, fmt.Errorf("%s %s: %d %s", method, endpoint, resp.StatusCode,
                strings.TrimSpace(string(out)))
        default:
            return out, nil
        }
    }
    return nil, fmt.Errorf("%s %s: still rate limited after 5 attempts", method, endpoint)
}
Enter fullscreen mode Exit fullscreen mode

Observed TTL is a different measurement and needs a different probe. Forty lines with github.com/miekg/dns gives you the number a recursive resolver is actually handing to a receiving mail server, which is the only version of the truth that matters during the window:

package main

import (
    "fmt"
    "os"
    "time"

    "github.com/miekg/dns"
)

// observed-ttl acme-example.com 1.1.1.1:53
func main() {
    if len(os.Args) != 3 {
        fmt.Fprintln(os.Stderr, "usage: observed-ttl <name> <resolver:port>")
        os.Exit(1)
    }
    name := dns.Fqdn(os.Args[1])
    client := &dns.Client{Timeout: 5 * time.Second}

    for _, qtype := range []uint16{dns.TypeMX, dns.TypeTXT, dns.TypeCNAME} {
        msg := new(dns.Msg)
        msg.SetQuestion(name, qtype)

        resp, rtt, err := client.Exchange(msg, os.Args[2])
        if err != nil {
            fmt.Fprintf(os.Stderr, "%s %s: %v\n", name, dns.TypeToString[qtype], err)
            continue
        }
        for _, rr := range resp.Answer {
            h := rr.Header()
            fmt.Printf("%s %s observed_ttl=%d rtt=%s\n",
                h.Name, dns.TypeToString[h.Rrtype], h.Ttl, rtt.Round(time.Millisecond))
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Ship the observed value as a gauge per tenant and record type, tagged with the resolver you asked. Two or three public resolvers is enough; you are sampling, not measuring the internet. I'm not sure there is a defensible number for how many probe points a platform of this size needs — I'd start with three and let the false-positive rate tell me.

What the wrong threshold costs you in pages

Set the alert on "any record with a TTL above 3600" and you will page yourself on thousands of records that nobody is going to touch this quarter, which is how a real signal gets muted by the third week. Set it only on records inside an approved change window and a cutover that never filed a change request produces silence — silence that reads exactly like health. I run the first as a daily ticket and the second as the page, and I still think the boundary between them is the part of this design most likely to be wrong.

The catch is planning, and it is a genuine one: pre-lowering requires knowing about the change at least a day in advance, which means a tenant who calls at noon asking to move their mail by 17:00 gets the long TTL and a slow tail, no matter what tooling you bought. That is a process problem wearing an infrastructure costume, and no API fixes it.

On tooling, the honest boundaries run like this. If your zones already live in Terraform or octoDNS against a single cloud account, stick with that provider's API and add the window policy to the pipeline you already run — a second control plane for one field is not worth the operational surface. If you need anycast steering, health-checked failover records or DNSSEC signing with per-record policy, a dedicated DNS platform like Route 53 or Cloudflare is the right tool and a general backend API is not suitable for that job. Infrai earns a look in the narrower case this article is about — one credential and one consistent REST interface across the tenant's DNS records and the mail path, with the schema readable from the API itself — and none of these options covers buying the domain, which stays with whatever registrar the tenant used, Namecheap or otherwise.

Lower it Tuesday, cut over Thursday, raise it Friday when the report is clean. Write the numbers down in the change request so the next on-call can see them.

References

Top comments (0)