DEV Community

Cover image for How DNS resolution works under the hood: a step by step guide for high availability infrastructure
binadit
binadit

Posted on Originally published at binadit.com

How DNS resolution works under the hood: a step by step guide for high availability infrastructure

DNS resolution: the seven hops nobody thinks about until failover breaks

You push a config change, wait for propagation, and traffic still hits the dead server. Sound familiar? Most DNS incidents come down to not knowing what actually happens between a browser typing a domain and a TCP handshake starting. Let's fix that.

This is a practical walkthrough of the resolution chain, plus the config patterns that turn DNS into an active part of your high availability setup instead of a silent single point of failure.

Assumptions

  • You control a domain through Cloudflare, Route 53, or similar, with dashboard or API access
  • You have a Linux box with dig, nslookup, and tcpdump/ngrep available
  • You know the basics of IP, TCP, and UDP
  • You're comfortable running commands against a real or disposable test domain

We'll use example.com everywhere below. Swap it for your own.

The resolution chain, hop by hop

1. Browser cache

Before anything hits the network, Chrome checks its own DNS cache:

chrome://net-internals/#dns
Enter fullscreen mode Exit fullscreen mode

If there's a live entry, resolution stops right there. This is why a DNS change can look "stuck" in your own browser even after the TTL has expired everywhere else.

2. OS stub resolver

Next stop is the OS-level resolver, usually systemd-resolved or nscd on Linux:

resolvectl status
resolvectl statistics
Enter fullscreen mode Exit fullscreen mode

It checks /etc/hosts, then its own cache, before forwarding anything upstream.

3. Recursive resolver

No local hit means the query goes to a recursive resolver: your ISP's, or a public one like 1.1.1.1 or 8.8.8.8. This is where +trace becomes your best debugging friend:

dig @1.1.1.1 example.com +trace
Enter fullscreen mode Exit fullscreen mode

4. Root servers

The recursive resolver asks a root server who's authoritative for .com. Roots don't know about example.com, only about the TLD:

com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
Enter fullscreen mode Exit fullscreen mode

5. TLD servers

The TLD server hands back your domain's authoritative nameservers, the ones set at your registrar:

example.com. 172800 IN NS ns1.yourdnsprovider.com.
example.com. 172800 IN NS ns2.yourdnsprovider.com.
Enter fullscreen mode Exit fullscreen mode

6. Authoritative nameserver

Finally, the actual record comes back:

example.com. 300 IN A 203.0.113.42
Enter fullscreen mode Exit fullscreen mode

This is the step that matters for HA: which IP gets returned, how fast it can change, and what happens when that IP goes dark.

Configuring DNS for actual failover

Understanding the chain is step one. Making it work for you is step two.

Drop your TTLs on failover-critical records. A 3600s TTL means an hour-long tail of stale traffic during failover. Use 60-300s instead:

example.com. 60 IN A 203.0.113.42
Enter fullscreen mode Exit fullscreen mode

You'll pay for it in query volume against your authoritative nameservers. Worth it.

Attach health checks. Route 53 and Cloudflare can pull an unhealthy origin out of the response set automatically:

aws route53 change-resource-record-sets \
  --hosted-zone-id Z1PA6795UKMFR9 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "example.com",
        "Type": "A",
        "SetIdentifier": "primary",
        "Failover": "PRIMARY",
        "TTL": 60,
        "ResourceRecords": [{"Value": "203.0.113.42"}],
        "HealthCheckId": "abcd1234-healthcheck-id"
      }
    }]
  }'
Enter fullscreen mode Exit fullscreen mode

This is the same mechanism behind a clean zero-downtime migration: DNS shifts traffic instead of forcing a hard cutover.

Return multiple A records. Clients can fall back to a second IP:

example.com. 300 IN A 203.0.113.42
example.com. 300 IN A 203.0.113.43
Enter fullscreen mode Exit fullscreen mode

Fine for stateless services behind a load balancer, but not a real substitute for health-checked failover; clients cache order and don't always retry smartly.

Verifying it actually works

Don't trust the dashboard. Check every layer.

dig example.com +noall +answer
dig example.com @8.8.8.8 +noall +answer
dig example.com @1.1.1.1 +noall +answer
Enter fullscreen mode Exit fullscreen mode

Lower-than-configured TTLs from a resolver mean the change is propagating. Original TTLs mean you're still hitting a cached answer.

Check resolution latency directly:

dig example.com | grep "Query time"
Enter fullscreen mode Exit fullscreen mode

Consistently above 100-150ms? Look at resolver placement or a provider with more edge presence. This happens before the TCP handshake even starts, so it's pure overhead on TTFB.

Simulate failover instead of waiting for an incident to test it for you:

iptables -A INPUT -s <health-check-ip> -j DROP
watch -n 5 'dig example.com +short'
Enter fullscreen mode Exit fullscreen mode

Confirm the response flips to your secondary within the configured TTL, then remove the rule.

Add DNS resolution time to your monitoring as its own metric, separate from full page load. A spike there with stable backend times points straight at a resolver or nameserver problem, not your app.

Pitfalls that keep coming back

  • Default TTLs. 3600 or 86400 seconds quietly kills any failover plan built on top.
  • Single nameserver provider. One outage there takes your domain down regardless of server health.
  • DNSSEC misconfiguration. Broken signing chains fail silently for validating resolvers while looking fine in tools that skip validation.
  • Testing only from your machine. Your local cache hides what real users see. Always check against multiple public resolvers.
  • Deep CNAME chains. Each extra hop adds latency and another point of failure.

Understand the seven hops, and DNS stops being a mystery box that occasionally ruins your day.

Full original writeup: How DNS resolution works under the hood

Originally published on binadit.com

Top comments (0)