DEV Community

Rasika Dangamuwa
Rasika Dangamuwa

Posted on

Why DNS Propagation Lies to You: 5 Resolver Traps That Break Production Migrations

#networking #devops #webdev #programming

You cut over your apex domain or primary API endpoint to a new server cluster. You update the DNS record in your cloud console, run dig api.example.com +short in your local shell, and see the new IP address. You announce that migration is complete.

Ten minutes later, your incident channel lights up.

A fifth of your global users are still hitting the old servers. Webhook callbacks from payment providers are failing, and mobile clients report intermittent ERR_NAME_NOT_RESOLVED errors.

To most engineers, "DNS propagation" sounds like a gradual wave washing over global networks. In reality, DNS is an asynchronous hierarchy of independent caches governed by differing RFC implementations and ISP behaviors.

Here are five real-world resolver traps that cause production traffic splits during migrations, and how to avoid them.

1. The Pre-Migration TTL Illusion

The most frequent migration mistake is changing the record and reducing the Time-to-Live (TTL) in the exact same step:

# Old record:
api.example.com.   86400   IN   A   198.51.100.10

# Migration change:
api.example.com.   300     IN   A   203.0.113.50
Enter fullscreen mode Exit fullscreen mode

Reducing the TTL to 300 seconds during cutover does nothing for resolvers that already cached the previous record. Any resolver that queried api.example.com five minutes prior will continue routing traffic to the old IP (198.51.100.10) for the remaining 23 hours and 55 minutes of the original 86,400-second window.

Fix: Lower your TTL to 300 seconds at least 48 to 72 hours before your scheduled maintenance window. Only execute the cutover once existing caches have expired under the old TTL.

2. Negative Caching Traps (RFC 2308)

Suppose you are launching a new endpoint, such as v2-auth.example.com. While preparing the release an hour before launch, an engineer tests a deployment script or a CI job curls the URL to verify TLS certificates.

The authoritative nameserver correctly returns NXDOMAIN (Non-Existent Domain).

Under RFC 2308, recursive resolvers cache negative responses. The duration is dictated by the MINIMUM field in your zone's SOA record:

example.com.  3600  IN  SOA  ns1.example.com. hostmaster.example.com. (
                2026092601 ; serial
                7200       ; refresh
                3600       ; retry
                1209600    ; expire
                3600       ; negative caching TTL (1 hour)
              )
Enter fullscreen mode Exit fullscreen mode

Even if you publish the new A record seconds later, resolvers that saw that early NXDOMAIN will reject all queries for the next hour without querying authoritative nameservers again.

3. Anycast PoP Divergence and Local Resolver Drift

Major public DNS providers (like Cloudflare 1.1.1.1 and Google 8.8.8.8) use BGP Anycast to announce the same IP from hundreds of Points of Presence (PoPs) worldwide.

When querying 1.1.1.1 from London, you hit a different node and cache than someone querying 1.1.1.1 in Tokyo or Frankfurt. Anycast providers do not synchronize cache entries globally in real time. One edge node may fetch the fresh record upon cache eviction, while another edge node in another region still has 15 minutes remaining on its TTL.

When verifying whether your DNS changes have reached global consensus, querying a multi-location DNS Propagation Checker allows you to inspect resolver status, response latencies, and regional IP discrepancies side-by-side rather than trusting your own shell's single hop.

4. EDNS0 Client Subnet (ECS) Routing Mismatches

If your infrastructure relies on GeoDNS or latency-based routing (Route 53, Cloudflare Load Balancing), your authoritative nameserver returns different IP addresses based on client geography.

Under RFC 7871 (EDNS Client Subnet), recursive resolvers send a truncated version of the client's IP (/24 for IPv4) to the authoritative nameserver to return the closest server.

However, privacy-conscious resolvers often strip ECS data entirely. In those cases, the authoritative server routes traffic based on the Anycast resolver's datacenter IP, not the end user's location. Testing DNS locally often hides routing mismatches that your production clients encounter abroad.

5. Application and OS-Level Stub Caching

Even when upstream resolvers update, internal application runtimes often hold onto stale IPs:

  • JVM: Historically cached successful DNS lookups forever (networkaddress.cache.ttl = -1). In modern runtimes, it defaults to 30 seconds unless a security manager locks it indefinitely.
  • Node.js: Standard http.get calls delegate to getaddrinfo synchronously in libuv, inheriting OS caching from systemd-resolved or nscd.
  • Docker / Kubernetes: Internal CoreDNS forwarders maintain their own upstream TTL rules.

Configure explicit DNS cache limits in long-running container runtimes before performing zero-downtime host migrations.

Safe Migration Protocol

To ensure clean zero-downtime cutovers:

  1. Lower TTLs in advance: Set TTLs to 300 seconds at least 48 hours prior to maintenance.
  2. Never pre-query unreleased subdomains: Prevent negative cache poisoning on recursive resolvers.
  3. Keep the old origin active: Maintain reverse proxy routing or a 301 redirect on the old cluster for at least 48 hours post-cutover.
  4. Monitor global propagation: Verify worldwide consistency with tools like Nutilz before shutting down legacy infrastructure.

Top comments (0)