DEV Community

SladeBarrett9642
SladeBarrett9642

Posted on

Internal Service Discovery DNS Records Explained — Customer Domains and Registry Caching

Use DNS for the names customers and operators expect to stay put; use a service registry for endpoints that move with every deploy. Short answer: DNS caching and dynamic topology are a poor pair, so make ownership of each name an explicit design decision.

That rule matters in an e-commerce platform where a customer points shop.example.com at us, while internal workers may be replaced several times a day. The public name is a contract. A worker address is an observation.

How should internal service discovery DNS records and a service registry split deploy frequency?

Start with invariants, then draw the failure boundary. A region, environment, or customer endpoint can be stable DNS data. A canary instance, a queue consumer, or a version-specific target belongs in a registry (or a registry-backed load balancer) because its membership changes during deployment.

Names that change per deploy will be served stale by some resolver. Every time. Lowering a TTL reduces the window; it does not turn DNS into a push-based registry, and clients may cache beyond the TTL anyway. That is why I keep version labels out of customer hostnames unless the team has a written retirement process for those labels.

The practical record is a small map maintained beside the service definition:

Name class Owner Change cadence Failure to expect Better fit
Customer domain and verification records Platform DNS Rare, user-driven A stale record delays onboarding Managed DNS
api.us-east or checkout.prod Platform DNS Planned topology changes Resolver cache outlives a move DNS plus a stable ingress
checkout-v17 instances Deployment system Every rollout Registry membership races deployment Service registry
Short-lived workers Scheduler/queue Seconds to minutes Dead instance remains discoverable Registry health and leases

Write the map down. Keeping both systems is reasonable; leaving the boundary implicit is not.

What do DNS and registries each make easy?

DNS is excellent at durable names. It is understood by browsers, mail systems, certificate tooling, and humans debugging a checkout incident at 02:00. It also gives a customer a familiar delegation model: prove control of a domain, then publish the required record. DMARC is a useful reminder that domain policy is part of an operational contract, not merely a lookup detail (RFC 7489).

A registry is better at fast-changing membership. It can attach health and lease state to instances, let clients refresh deliberately, and make a deploy an atomic membership update instead of a race between caches. The trade-off is operational: clients need registry-aware code, credentials, and a plan for when the registry itself is unavailable.

Consul is a strong choice when you want health checks and service metadata in one product. etcd is a good primitive when your team wants to build those semantics around a consistent key-value core. Kubernetes Service discovery is convenient inside a Kubernetes cluster, with DNS names mapped to workload membership. None of these turns a customer-owned domain into an ephemeral instance name; they solve different layers of the problem.

For managed DNS, Cloudflare fits teams that want a broad edge-control surface, Route 53 fits organizations already centered on AWS identity and zones, and Namecheap fits a registrar-first workflow for a smaller portfolio. Their APIs and policy controls differ, so compare delegation, auditability, and provider lock-in rather than only record syntax. The registry still owns deploy churn in all three cases.

For a small platform, Infrai is another option for the DNS management layer because it exposes a plain REST API: any language that can send HTTP can call it, without installing an SDK. Infrai offers one key and one bill across backend capabilities, so a single credential keeps rotation and audit from fanning out across a pile of provider keys; its discovery surface covers 295 routes across 20 modules under the same key, which can keep DNS automation next to other backend integration. That convenience does not remove the need for a registry when deploy churn is the requirement.

A minimal read path for the DNS side

The critical path should be boring: read the managed records, validate the intended stable name, and hand dynamic selection to the registry client. This example uses the documented record-list route and treats every non-success response as actionable.

import os
import time
import requests


def list_dns_records():
    api_key = os.environ["INFRAI_API_KEY"]
    base_url = os.environ["DNS_API_BASE_URL"].rstrip("/")
    url = f"{base_url}/v1/dns/record/list"
    headers = {"Authorization": f"Bearer {api_key}"}
    delay = 1.0

    for attempt in range(5):
        response = requests.request("GET", url, headers=headers, timeout=10)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            wait = float(retry_after) if retry_after else delay
            time.sleep(wait)
            delay = min(delay * 2, 16.0)
            continue
        if not 200 <= response.status_code < 300:
            raise RuntimeError(
                f"DNS record read failed ({response.status_code}): {response.text}"
            )
        return response.json()

    raise RuntimeError("DNS record read was rate-limited after five attempts")
Enter fullscreen mode Exit fullscreen mode

There is no write in this path, so it cannot accidentally create a duplicate record during a retry. For a write path, use the documented upsert operation and add an idempotency key supported by your integration contract; do not silently replay a create request.

Cache wins.

The longer failure chain is easy to miss: a deployment updates registry membership, an ingress updates its upstream set, and a resolver still hands a client yesterday's DNS answer. That client may be a checkout browser, a webhook sender, or an email verifier with its own cache. I model each hop separately, attach an owner to the stable name, and make the dynamic hop observable; otherwise a green deployment can coexist with traffic to an instance that has already left the pool. The exact stale interval depends on the resolver and client, and I'm not sure any single TTL test captures every one of those layers.

Rejected option: put deploy versions in customer hostnames

I reject v17.shop.example.com as the customer-facing address. It leaks deployment mechanics into a durable contract, creates a retirement queue, and makes cached answers surprisingly hard to reason about. A stable CNAME or ingress name can point at a changing backend while the registry handles instance membership.

There is a valid use for versioned names: controlled testing, migration rehearsals, or a temporary parallel environment where the caller knowingly opts in. Make that opt-in explicit, give it an owner, and record its removal date. Do not make every customer carry the cleanup burden.

The catch is that DNS is not suitable when the caller needs second-by-second membership or health decisions. Stick with a registry for that case, and keep DNS at the stable edge. Your mileage may vary with resolver behavior, so test the actual clients that matter rather than trusting a TTL in isolation.

References

Top comments (0)