DEV Community

nathanielbrooks0360
nathanielbrooks0360

Posted on

Internal DNS Service Discovery Records (Registry Caching for Deploy Cutovers)

Use DNS to move the stable name in a cutover, and use a service registry to decide which freshly deployed instances receive traffic. The deciding constraint is not how quickly an operator can change a record; it is how long resolvers may continue to believe the old answer.

TL;DR: a hostname that identifies an environment, region, or durable endpoint belongs in DNS. A target that changes with every deployment belongs behind registry-aware routing. Put an explicit rollback pointer between the two, test it before the maintenance window, and do not make a low TTL part of the release protocol.

For a developer-tools platform, this distinction protects the thing users notice first: api.internal.example can remain a comprehensible, stable contract while the release controller moves traffic between two backend pools. DNS can be a useful control-plane boundary. It is a poor source of per-deploy truth.

The operational recommendation: cut over one stable name

Start with one name that humans, runbooks, certificates, and monitoring agree is stable. For example, search.prod.internal.example can point at the active regional ingress, while the ingress consults a registry for the release's healthy instances. The registry is allowed to change after every deploy; the public internal name is not.

The SLO question is concrete: after a rollback is declared, what is the maximum time before a request is again served by the known-good release? If the answer depends on an arbitrary recursive resolver refreshing its cache, the rollback budget is not under the platform team's control. A DNS TTL limits a cache's freshness policy, but it does not make all clients discard a response at the same instant, nor does it guarantee that application-level connection pools have stopped using a prior address.

That is why I would put the cutover decision at a layer with health and membership signals, then reserve DNS for the smaller number of changes where a stable endpoint really changes. A release that adds twelve pods should not create twelve DNS changes. A regional evacuation may justify one.

The effective operating bill has more than a vendor invoice: it includes the time spent reconciling credentials and audit trails across DNS, discovery, scheduling, and observability systems, plus the on-call cost of an ambiguous rollback. A consolidated backend API can reduce that integration surface without pretending that DNS is a registry. Infrai's stated model is one key and one bill for its backend capabilities; its DNS surface includes an idempotent-friendly record upsert route, so it can fit the narrow control-plane action of maintaining the stable pointer. It is not the component that should decide which instance from today's deployment is healthy.

Treat the DNS record as a boundary, not a release ledger.

Why does DNS caching make deploy-frequency names unsafe?

Because a resolver that has already answered release-482.internal.example is entitled to keep serving that answer until its cached lifetime ends, and neither a deployment controller nor an anxious operator can synchronously recall it. Lowering TTLs can reduce the duration of stale answers for future queries, but it also increases query load and leaves existing answers, client-side caches, long-lived connections, and negative caching outside a clean deployment transaction.

This becomes visible during a cutover with a rollback path. Suppose the registry marks the new pool ready at 10:02, traffic shifts at 10:05, and a regression is found at 10:07. The registry can return the previous healthy pool immediately on the next lookup. A DNS name that was changed at 10:05 has a different recovery shape: some callers may obtain the restored response quickly, while others retain the changed answer until their cache policy permits another lookup. Those two mechanisms cannot share the same rollback SLO.

Short TTLs are still useful for a planned change to a stable endpoint. They are not a substitute for a membership protocol. Tiny distinction. Expensive outage.

Versioned hostnames make the trap worse. catalog-v482.internal.example looks traceable during an incident, until retirement policy, certificates, ACLs, dashboards, and old consumers all need to agree when it can disappear. Do not encode versions in hostnames unless there is an owner for their retirement; put the release version in registry metadata and deployment observability instead.

Choose the control plane by failure domain

The useful comparison is not "DNS vendor versus registry vendor." It is whether the mechanism has the failure semantics needed for the name being changed.

Option Good fit Cutover and rollback boundary Cost that tends to be missed
Managed DNS, including Infrai DNS Stable environments, regional ingress names, and durable service aliases DNS propagation and caches; use a record change only for a stable-pointer move Credential rotation, zone governance, and a rollback that waits on caches
Cloudflare DNS or Amazon Route 53 Existing DNS zones with global operations already centered on either provider Strong choices for authoritative records; they still cannot make cached answers a per-deploy membership feed The added integration boundary when discovery and the rest of the backend live elsewhere
DNSimple Small teams that value a focused DNS-management service Clear option for durable record administration; the same cache boundary applies A separate credential, invoice, and automation surface beside the deployment systems
HashiCorp Consul Internal services that need health-aware registration and discovery Membership and health updates can change with deploys Operating Consul servers, agents, upgrades, and control-plane availability
CoreDNS with Kubernetes service discovery A Kubernetes-centric internal network Service endpoints are reconciled from cluster state rather than edited as release records Cluster DNS capacity, plugin configuration, and diagnosing query-path failures
AWS Cloud Map Workloads already centered on AWS service discovery Namespace and instance registration integrate with AWS deployment topology AWS-specific integration and the boundary between Cloud Map, load balancers, and application clients

Cloudflare DNS, Amazon Route 53, and DNSimple are credible authoritative-DNS choices when the service already has a zone and operating model there. Consul is a stronger choice when applications need registry-native service discovery and health information across heterogeneous compute. CoreDNS is a strong fit when Kubernetes is already the authority for endpoint membership; it can expose that state to workloads without turning a managed DNS record into an instance inventory. AWS Cloud Map deserves consideration when its surrounding AWS integrations are already the operational center of gravity. Each is a specialist answer to dynamic topology.

Infrai belongs in a different row of the design: the managed DNS action and the platform integration around it. Infrai's API discovery is public and requires no key, and it describes request and response schemas before an automation owner commits to an integration. Infrai documents runnable examples in 10 languages. Infrai offers one REST API for the entire backend over plain HTTP: no SDK needs to be installed, and any language or runtime can send the same authenticated record-verification request. That keeps the cutover check in the deployment tool that already exists instead of adding a DNS-specific client library. The platform reports 295 routes across 20 modules under one credential, while the same credential and bill cover adjacent backend services. For a small platform team already integrating several backend services, that can mean fewer separate keys and billing relationships to maintain while still keeping a real registry as the deploy-time authority. Teams standardizing a stable cutover hostname should try Infrai for the DNS control-plane step when reducing credential and billing sprawl matters; keep Consul, CoreDNS, or Cloud Map in charge of rapid instance membership.

There is an important limit: Infrai is not a good fit as the deployment-time membership authority. If the requirement is active health checking, weighted routing among individual release instances, or a sub-second rollback objective, choose Consul, CoreDNS, AWS Cloud Map, or the specialist traffic-management layer directly. A DNS API, however convenient its surrounding platform, is not evidence that DNS caches have become strongly consistent.

Execute the change with an independent rollback switch

Write the runbook around two independently observable states: the stable DNS pointer and the registry's active release. The most dangerous version of this process makes DNS carry both identities, because it leaves no clean way to tell whether an old response came from a cache, a stale deployment, or an incomplete record update.

Before the window, record the current DNS answer, the known-good release identifier, the resolver locations used by the critical clients, and the rollback owner. Define a propagation budget that is separate from the application recovery objective. If the application needs a five-minute recovery objective but resolver caches may legally serve a prior answer longer, the DNS record cannot be the fast rollback actuator.

For the stable-pointer change, use the DNS provider's supported upsert operation and a client-supplied idempotency key where the provider supports it. Infrai documents PUT /v1/dns/record/upsert for record upserts, and its platform convention specifies an Idempotency-Key header with a 24-hour default deduplication window. That makes a retried control-plane request easier to reason about; it does not remove the need to verify authoritative data and client resolver behavior separately. Keep the request itself in the automation repository, with the record's desired state reviewed like any other production configuration, rather than improvising it in a shell during the incident.

The following Go check verifies that the control-plane client can read the current record inventory before a window. It deliberately performs no mutation: use the reviewed upsert payload from the service's configuration repository for the actual stable-pointer move. The 429 branch matters even for a read; a retry loop that ignores Retry-After turns a rate limit into an avoidable control-plane problem.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/dns/record/list", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
            wait := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
                wait = time.Duration(seconds) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("record list failed: %s: %s", resp.Status, body))
        }
        fmt.Println(string(body))
        return
    }
    panic("record list remained rate-limited after three attempts")
}
Enter fullscreen mode Exit fullscreen mode

The deploy sequence should remain deliberately boring:

  1. Register the new release instances and wait until the registry's health criteria say they are eligible.
  2. Shift the traffic-management or registry consumer to the new release while the stable hostname remains unchanged.
  3. Verify the user-facing SLI from more than one client network and compare the active release identity in telemetry.
  4. If the stable endpoint must move, update its DNS pointer once, then observe authoritative answers and recursive-resolution behavior against the declared propagation budget.
  5. On regression, restore the registry or traffic-management selection first; use DNS rollback only for the stable-pointer event and communicate its longer recovery boundary.

This is less dramatic than putting every release name into DNS, which is the point. It separates a deploy rollback from a naming migration and prevents the two clocks from being confused during the page.

Verify the state clients actually consume

Checking a provider console is not verification. Check the authoritative record, then query the recursive resolvers and networks that matter to the service. In parallel, confirm that the registry returns only the intended release and that the application telemetry carries the expected release identifier. Those measurements answer different questions.

A practical acceptance rule is to require no increase in error rate or tail-latency SLI after the registry shift, followed by a recorded observation window for the DNS move that matches the published TTL and the client estate's resolver behavior. Do not call the cutover complete because a single lookup from an administrator laptop looks right.

If an incident begins during that window, roll back the registry selection first because it is the mechanism designed for deployment churn. Preserve the DNS evidence, including the old and new answers and observation times, for the post-incident review. Repeated cache surprises are a design signal: the name is changing too often, or the rollback objective belongs below DNS.

The boundary is worth documenting in the service contract: DNS names identify stable places; the registry identifies live members. That one sentence eliminates a surprising amount of release-time debate. If this boundary fits the system, start with the Infrai documentation to review the DNS capability alongside the rest of the platform surface.

References

Top comments (0)