DEV Community

jun liu
jun liu

Posted on Originally published at ops.jiunotes.top

When "Network Is Down" but the Server Is Fine: Debugging DNS Resolution Failures

#cli #debugging #devops #networking

You ping the server. It answers. You open the app in a browser. Connection refused — or worse, it hangs for thirty seconds and then times out. The server is up, the cable is plugged in, yet nothing resolves. Nine times out of ten, the problem is not the network at all. It is DNS.

I have spent more than a few late nights on this exact symptom. Here is the field-tested path I now follow every time, so you do not have to reinvent it at 2 a.m.

Start with the cheapest test: nslookup / dig

Before touching configs, confirm whether name resolution itself works:

dig +short example.com
nslookup example.com
Enter fullscreen mode Exit fullscreen mode
  • Empty output or SERVFAIL → the resolver is the problem, not the remote host.
  • A correct IP but the app still fails → resolution is fine; look at the app or firewall next.
  • NXDOMAIN → the name genuinely does not exist (typo, or the record was deleted).

On Linux, systemd-resolved often listens on 127.0.0.53. If you point tools at the wrong resolver you get confusing results, so check /etc/resolv.conf first.

The usual suspects

  1. /etc/resolv.conf got overwritten. Cloud-init, DHCP leases, and VPN clients all rewrite this file. If nameserver points at a dead IP, every lookup fails while pings to raw IPs still work.
  2. A flaky internal resolver. Pointing everything at one corporate DNS server that intermittently drops queries produces the classic "works, then doesn't" pattern.
  3. systemd-resolved stub confusion. 127.0.0.53 is the stub. Query it directly with dig @127.0.0.53 example.com to isolate whether the stub or the upstream is at fault.
  4. nscd / dnsmasq cache poisoning or staleness. A cached NXDOMAIN can linger. Flush with sudo systemd-resolve --flush-caches (or restart dnsmasq).
  5. Firewall blocking UDP 53. Outbound DNS is UDP/53 (and TCP/53 for large responses). A strict egress rule that allows TCP but not UDP will let dig sometimes work and the app sometimes fail.

A 5-minute triage checklist 

# 1. Does raw resolution work?
dig +short example.com

# 2. Which resolver are we using?
cat /etc/resolv.conf

# 3. Is the stub resolving, or just the upstream?
dig @127.0.0.53 example.com
dig @8.8.8.8 example.com

# 4. Is UDP 53 actually open outbound?
nc -vzu 8.8.8.8 53

# 5. Clear any poison cache
sudo systemd-resolve --flush-caches
Enter fullscreen mode Exit fullscreen mode

If step 2 shows a dead nameserver, fix that first — it explains everything.

The fix that actually sticks

For servers, I stopped relying on DHCP-assigned resolvers. I pin a known-good pair in /etc/systemd/resolved.conf:

[Resolve]
DNS=8.8.8.8 1.1.1.1
FallbackDNS=9.9.9.9
Domains=~.
Enter fullscreen mode Exit fullscreen mode

Then sudo systemctl restart systemd-resolved. The Domains=~. line is the part people miss: it tells the resolver to use these servers for every domain, not just ones matching a search list.

Why this matters beyond the outage

DNS failures are silent and asymmetric. The database host resolves, the cache host does not, and suddenly one service throws timeouts while dashboards stay green. Baking a deterministic resolver config into the base image — and alerting on dig latency, not just on ping — has saved me more 2 a.m. pages than any other single change.

If you want the full postmortem write-up with packet captures and the exact tcpdump filter I use to catch dropped UDP/53 queries, it lives on my ops notes site: ops.jiunotes.top/en. The same notes are mirrored on the main site: jiunotes.top/en.

Top comments (0)