DEV Community

jun liu
jun liu

Posted on Originally published at ops.jiunotes.top

DNS Debugging: Six Symptoms, One Decision Tree, and a +cd Trick That Saves an Hour

#dns #linux #networking #debugging

DNS gets blamed first and debugged last. The symptom is always the same — the page won't load, the API times out, the container can't reach its database — and the cause can be anywhere between your laptop's cache and a TLD server. So people run every command they remember, in random order, and learn nothing.

The move that actually works is duller: classify the symptom first, then pick one command for that class. Six symptoms cover almost every DNS failure I have run into in twenty-odd years of operations work.

"DNS is broken"
  │
  ├─ NXDOMAIN ──────────────── The name doesn't exist, or the delegation hasn't propagated
  ├─ SERVFAIL ──────────────── Some server in the chain refused to answer
  ├─ Timeout ───────────────── The recursor is down, or port 53 is blocked on the path
  ├─ CNAME won't resolve ───── A loop, or a CNAME sitting on top of an A record
  ├─ Record won't change ───── TTL hasn't expired yet
  └─ DNSSEC fails ──────────── The signature chain is broken upstream
Enter fullscreen mode Exit fullscreen mode

Start at the top of that list, not at the top of your memory.

SERVFAIL: try +cd before you chase anything

This is the one worth reading the article for.

SERVFAIL is DNS's version of a 500. The server refuses to answer and won't say why. Half the time it is not a broken server at all — it is DNSSEC validation failing, and the resolver is being honest about it. Two commands tell them apart:

dig @8.8.8.8 example.com        # SERVFAIL
dig @8.8.8.8 example.com +cd    # validation off: NOERROR, correct IP
Enter fullscreen mode Exit fullscreen mode

If the second one returns the right answer, the record is fine and the chain is broken. Look at the two ends of the chain:

dig example.com DS +noall +answer        # at the parent (registrar / TLD)
dig example.com DNSKEY +noall +answer    # at the authoritative
Enter fullscreen mode Exit fullscreen mode

Different key IDs mean the registrar never synced the DS record after a key rotation. That gets fixed at the registrar, not on your server. You can spend an afternoon restarting BIND and never touch the actual problem.

If both queries return SERVFAIL, the authoritative really is down and you should go chase connectivity instead.

NXDOMAIN: three questions, in order

NXDOMAIN means "this name does not exist". Ask it three times, from the outside in:

dig example.com                          # your local recursor
dig @ns1.example.com example.com         # the authoritative, cache bypassed
dig @a.gtld-servers.net example.com NS   # the delegation at the TLD
Enter fullscreen mode Exit fullscreen mode
  • NOERROR with a non-empty answer section → the name exists and your resolver was lying. Flush it.
  • NOERROR with an empty answer section → the name exists but has no A/AAAA record. Add one.
  • NXDOMAIN from the authoritative → the name genuinely doesn't exist. Check the registrar.

The classic false alarm is a domain registered ten minutes ago. The registrar console says "registered", the authoritative returns NXDOMAIN, and nothing is actually wrong: the delegation to the TLD hasn't been published yet. That has taken anywhere from 15 minutes to six hours in cases I have handled. Don't touch your resolver config; check the TLD first.

Timeout: compare resolvers, don't guess

One timeout tells you nothing. Three timeouts tell you where the problem is:

dig @8.8.8.8 example.com +time=3 +tries=1
dig @1.1.1.1 example.com +time=3 +tries=1
dig @223.5.5.5 example.com +time=3 +tries=1
dig example.com +time=3 +tries=1        # whatever your host uses
Enter fullscreen mode Exit fullscreen mode
  • All four fail → the path is blocked, or port 53 is filtered somewhere.
  • Only the public ones fail → your internal recursor is down. Check systemctl status systemd-resolved or systemctl status bind.
  • Public works, internal is slow → the internal recursor is misconfigured, not dead.

When exactly one machine is affected and its neighbours on the same VLAN are fine, stop looking at the network. Check the local resolver service. A segfaulted systemd-resolved looks exactly like a network fault from the application's point of view, and sudo systemctl restart systemd-resolved fixes it in two seconds:

systemctl status systemd-resolved
sudo systemctl restart systemd-resolved
sudo resolvectl flush-caches
Enter fullscreen mode Exit fullscreen mode

CNAME loops, and the trap next to them 

dig example.com +noall +answer +comments
Enter fullscreen mode Exit fullscreen mode

A loop looks like this, and resolvers give up after eight CNAME hops:

example.com.      300 IN CNAME www.example.com.
www.example.com.  300 IN CNAME example.com.
Enter fullscreen mode Exit fullscreen mode

The neighbouring trap is quieter: a CNAME cannot share a name with any other record. This is invalid and some providers will happily save it anyway.

example.com.      300 IN CNAME cdn.example.com.
example.com.      300 IN A     1.2.3.4
Enter fullscreen mode Exit fullscreen mode

Split it into two names — app.example.com pointing at lb.example.com, with the A record on lb.example.com — and the problem disappears.

"I changed the record and nothing happened"

That's TTL, almost every time. Read it out of the answer section:

dig example.com | head -20
# example.com.  86400 IN A 1.2.3.4
#               ^^^^^ 86400 seconds = 24 hours. You wait 24 hours.
Enter fullscreen mode Exit fullscreen mode

The change procedure that avoids this, in this order:

  1. Lower the TTL to 60 seconds.
  2. Wait 60 seconds.
  3. Change the record.
  4. Wait for propagation.
  5. Put the TTL back.

Do steps 1 and 3 in the other order and the change is a no-op for a day. Then compare two independent resolvers to see who is stale:

dig @8.8.8.8 example.com
dig @1.1.1.1 example.com
Enter fullscreen mode Exit fullscreen mode

Same answer from both → wait out the TTL. Different answers → one of them is stale, flush it. Both stale → the authoritative never changed; go back to the registrar.

Cache flush commands, by platform:

sudo systemctl restart systemd-resolved    # Linux, systemd-resolved
sudo kill -SIGHUP $(pgrep dnsmasq)         # Linux, dnsmasq
ipconfig /flushdns                         # Windows
sudo dscacheutil -flushcache               # macOS
sudo killall -HUP mDNSResponder            # macOS
Enter fullscreen mode Exit fullscreen mode

DNSSEC: read the flags, not the error message 

dig example.com +dnssec +noall +comments
dig example.com +cd     +noall +comments
Enter fullscreen mode Exit fullscreen mode

An ad bit in the flags line means validation succeeded. SERVFAIL with +dnssec and NOERROR with +cd means the answer is valid but unsigned somewhere up the chain. If the DS and DNSKEY key IDs don't match, the break is at the delegation — escalate to the registrar with the command output rather than rebuilding the zone.

Cheat sheet

Command What it gives you
dig +short example.com The answer, one line
dig example.com A A record
dig example.com AAAA IPv6 record
dig example.com MX Mail routing
dig example.com TXT SPF / DMARC / DKIM
dig example.com NS Authoritative nameservers
dig @8.8.8.8 example.com Ask a specific resolver
dig @ns1.example.com example.com Ask the authoritative, cache bypassed
dig example.com +trace Iterative walk from the root
dig example.com +noall +answer Answer section only
dig example.com +dnssec Request signatures
dig example.com +cd Skip validation

dig covers about 90% of this. host is fine in scripts. nslookup is what Windows gives you and it shows you the least — if you can dig, don't nslookup.

Still broken? Drop to packets 

sudo tcpdump -nn -i any -s 0 port 53 -c 50
sudo tcpdump -nn -i any port 53 and host 8.8.8.8
Enter fullscreen mode Exit fullscreen mode

Query leaves, no response comes back → the path or a middlebox is dropping it. Response comes back but the application never sees it → it's the local resolver, not the network.

I keep a longer version of this with the full decision tree, conditional forwarding configs for BIND and dnsmasq, the Docker 127.0.0.11 resolver behaviour, and three production post-mortems here: DNS troubleshooting: six-symptom decision tree. It also covers the DNSSEC caveat I skipped above — Cloudflare, Route 53 and Namecheap each handle key rotation slightly differently.

One more thing worth bookmarking: dig example.com +trace is the single most informative command in this article, and almost nobody runs it before restarting something. The trace tells you which hop is broken. Restarting tells you nothing.

Top comments (0)