DEV Community

devtocash
devtocash

Posted on Originally published at devtocash.com

Kubernetes DNS Resolution Failures: How to Debug and Fix CoreDNS Issues

#sre #kubernetes #devops #reliability

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

What a Kubernetes DNS failure actually looks like

Your app logs say dial tcp: lookup payments-api.prod.svc.cluster.local: i/o timeout, or Temporary failure in name resolution, or an NXDOMAIN for a service you can see with kubectl get svc. Nothing was deployed, nothing crashed — but suddenly half your services can't find each other. What happened: every DNS lookup in the cluster flows through CoreDNS (behind the kube-dns Service), and something between your pod and that Service — or inside CoreDNS itself — is broken.

DNS failures are the nastiest entry in this troubleshooting series because the symptom shows up in your app's logs, several layers away from the cause. The error is in the checkout service; the root cause is a conntrack race on one node, or a NetworkPolicy someone tightened yesterday. Before touching anything, do what works for every failure in this series: find the first failure, not the loudest one.

Step 1: Reproduce it from inside a pod

Don't debug DNS from your laptop — your laptop doesn't use the cluster's resolvers. Get a shell with real DNS tooling in the same namespace (and ideally on the same node) as the failing workload:

kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 \
  -n prod --restart=Never -- sleep infinity
kubectl exec -it dnsutils -n prod -- sh
Enter fullscreen mode Exit fullscreen mode

Then run the three lookups that partition the problem space:

nslookup kubernetes.default.svc.cluster.local   # cluster-internal, always exists
nslookup payments-api.prod.svc.cluster.local    # the name that's failing
nslookup google.com                             # external, tests upstream forwarding
Enter fullscreen mode Exit fullscreen mode

And read the resolver config the kubelet injected:

cat /etc/resolv.conf
Enter fullscreen mode Exit fullscreen mode 
nameserver 10.96.0.10
search prod.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
Enter fullscreen mode Exit fullscreen mode

That file is the whole client side of Kubernetes DNS. 10.96.0.10 is the ClusterIP of the kube-dns Service — not a CoreDNS pod. The search list plus ndots:5 explains most of the weird behavior you'll meet in Step 3. The three lookups tell you where to go next:

  • All three fail → the path from this pod to CoreDNS is broken (Step 3.2, 3.4) or CoreDNS is down (Step 3.1).
  • Internal works, external fails → CoreDNS is fine; upstream forwarding is broken (Step 3.5).
  • Everything works from your test pod but the app still fails → it's per-node or intermittent — the conntrack race (Step 3.3) — or the app's base image resolver behaves differently (musl vs glibc, Step 3.3 again).

Step 2: Establish scope — one pod, one node, or the whole cluster 

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide   # CoreDNS up? which nodes?
kubectl get endpoints kube-dns -n kube-system                  # does the Service have backends?
Enter fullscreen mode Exit fullscreen mode

An empty ENDPOINTS column is a smoking gun: the kube-dns ClusterIP exists, kubelet keeps writing it into every pod's resolv.conf, but there is nothing behind it — every lookup in the cluster times out. That's CoreDNS pods down, unschedulable, or failing readiness.

If CoreDNS looks healthy, check whether failures cluster on specific nodes — run the same nslookup from test pods pinned to a suspect node and a healthy one. DNS-broken-on-one-node is almost always kube-proxy or conntrack on that node, and the fastest business-hours fix is cordon, drain, and recycle it — same cattle-not-pets logic as a NotReady node.

Step 3: Fix the actual cause

1. CoreDNS is crashing — usually the loop

If CoreDNS is in CrashLoopBackOff, check its logs before anything else:

kubectl logs -n kube-system -l k8s-app=kube-dns --previous
Enter fullscreen mode Exit fullscreen mode

The classic message is:

[FATAL] plugin/loop: Loop (127.0.0.1:53076 -> :53) detected for zone "."
Enter fullscreen mode Exit fullscreen mode

This is CoreDNS refusing to forward queries to itself. On nodes running systemd-resolved, /etc/resolv.conf points at the local stub 127.0.0.53 — CoreDNS inherits that as its upstream, sends external queries to the node, which sends them right back. The fix is to point the kubelet at the real upstream file so CoreDNS inherits actual resolvers:

# /var/lib/kubelet/config.yaml
resolvConf: /run/systemd/resolve/resolv.conf
Enter fullscreen mode Exit fullscreen mode

Restart the kubelet, then restart the CoreDNS pods. For any other crash loop reason (bad Corefile edit, OOMKilled after someone shrank its limits), the general CrashLoopBackOff playbook applies unchanged — CoreDNS is just a Deployment.

2. A NetworkPolicy is silently eating port 53

If lookups from your test pod time out but CoreDNS is healthy and other namespaces resolve fine, check for egress policies in the failing namespace:

kubectl get networkpolicy -n prod
Enter fullscreen mode Exit fullscreen mode

The moment any policy with policyTypes: [Egress] selects a pod, everything not explicitly allowed is dropped — including DNS. Every egress-restricted namespace needs this allowance, and it needs both UDP and TCP (responses over 512 bytes and several mitigations below use TCP):

egress:
  - to:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system
        podSelector:
          matchLabels:
            k8s-app: kube-dns
    ports:
      - port: 53
        protocol: UDP
      - port: 53
        protocol: TCP
Enter fullscreen mode Exit fullscreen mode

This is the single most common way DNS breaks in one namespace right after a security hardening PR merges. If you run Cilium, Hubble shows the drop directly (hubble observe --verdict DROPPED --port 53) — the same flow data I used to generate NetworkPolicies safely with an agent, where DNS was the first thing the generated policies had to get right.

3. Intermittent 5-second timeouts: the conntrack race and ndots amplification

The most reported Kubernetes DNS symptom ever: lookups usually work, but a few percent take exactly 5 seconds (the glibc retry interval) or time out entirely. Two mechanisms stack on each other.

The conntrack race. glibc resolvers fire the A and AAAA queries in parallel, over UDP, from the same socket. Two packets racing through Linux conntrack insertion at the same instant can trigger a kernel race where one is dropped — the client waits 5 seconds and retries. You can see the evidence on a node:

conntrack -S | grep -v "insert_failed=0"    # non-zero insert_failed = the race
Enter fullscreen mode Exit fullscreen mode

ndots:5 amplification. With ndots:5, any name with fewer than five dots gets the search list applied first. A lookup of api.stripe.com becomes up to four wrong guesses — api.stripe.com.prod.svc.cluster.local, .svc.cluster.local, .cluster.local, then finally the real name — each one an A+AAAA pair. One external lookup = up to 10 queries, which multiplies both CoreDNS load and your exposure to the race.

Fixes, in order of leverage:

Deploy NodeLocal DNSCache. A DaemonSet cache on every node at a link-local IP (169.254.20.10); pods talk to their local node agent (no conntrack for that hop, thanks to NOTRACK rules), and it upgrades to TCP toward CoreDNS. This is the structural fix for the race and what managed clusters at any real scale should run.

Cut ndots for external-heavy workloads. Per-pod, no cluster-wide risk:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
Enter fullscreen mode Exit fullscreen mode

Or cheaper still: use FQDNs with a trailing dot in config — api.stripe.com. skips the search list entirely, and payments-api.prod.svc.cluster.local. does the same for internal calls.

Know your libc. single-request-reopen (serialize A/AAAA on separate sockets) is a glibc option — Alpine/musl images silently ignore it, which is why "the fix worked in staging (Debian) but not prod (Alpine)". musl also handles TCP fallback poorly; for Alpine-based apps, NodeLocal DNSCache is effectively the only reliable fix.

4. kube-proxy isn't programming the DNS Service on that node

If one node can't reach 10.96.0.10 at all (even nc -u -z -w2 10.96.0.10 53 from a pod there fails) but can reach CoreDNS pod IPs directly, the ClusterIP translation is broken on that node. Check kube-proxy:

kubectl logs -n kube-system -l k8s-app=kube-proxy --field-selector spec.nodeName=<node>
iptables-save | grep -c KUBE-SVC    # on the node; ~0 means rules never programmed
Enter fullscreen mode Exit fullscreen mode

Restarting the kube-proxy pod on that node forces a full resync. If this recurs, look for conntrack table exhaustion (conntrack -C vs conntrack_max in sysctl) — a node swallowing packets at capacity produces exactly this shape of "DNS is down but only here."

5. Internal resolves, external doesn't: upstream forwarding

If kubernetes.default resolves but google.com returns SERVFAIL or times out, CoreDNS can't reach its upstream. The forwarding rule lives in the Corefile:

kubectl get configmap coredns -n kube-system -o yaml | grep -A2 forward
Enter fullscreen mode Exit fullscreen mode 
forward . /etc/resolv.conf {
   max_concurrent 1000
}
Enter fullscreen mode Exit fullscreen mode

/etc/resolv.conf here is the node's resolv.conf (via the kubelet setting from Step 3.1). So external DNS breaks when the node's resolvers are wrong, when a firewall/security group blocks outbound 53 from nodes, or when your VPC resolver rate-limits (AWS caps at 1024 packets/sec per ENI — CoreDNS behind heavy ndots amplification can genuinely hit it). CoreDNS logs i/o timeout against the upstream IP when this happens, and pinning forward . 10.0.0.2 (your VPC resolver) or deploying NodeLocal DNSCache to absorb repeat lookups are the usual outs.

Alert on DNS before your apps do

CoreDNS exports Prometheus metrics on :9153 out of the box, and a standard Prometheus + Grafana stack scrapes them automatically. Two alerts cover almost everything above:

# SERVFAIL ratio — upstream/forwarding is failing
sum(rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m]))
  / sum(rate(coredns_dns_responses_total[5m])) > 0.05

# Slow lookups — races, overload, or a dying upstream
histogram_quantile(0.99,
  sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m]))) > 0.5
Enter fullscreen mode Exit fullscreen mode

Pair them with the free one Kubernetes gives you: kube_endpoint_address (or an absent-endpoints check) on kube-dns in kube-system, so an empty-endpoints outage pages you in seconds instead of arriving as a wall of application errors. Watching coredns_dns_requests_total per pod also tells you when to scale — CoreDNS replicas are managed by cluster-proportional-autoscaler on most distros, and a cluster that grew nodes without growing DNS capacity fails exactly like Step 3.3 under load.

A repeatable checklist

  1. Reproduce from a debug pod in the failing namespace: internal name, failing name, external name. Read /etc/resolv.conf. (Step 1)
  2. Scope it: CoreDNS pods healthy, kube-dns endpoints non-empty, failures node-local or cluster-wide? (Step 2)
  3. CoreDNS crash-looping with plugin/loop = systemd-resolved loop; point kubelet's resolvConf at the real file. (Step 3.1)
  4. One namespace broken after a hardening PR = egress NetworkPolicy missing UDP+TCP 53 to kube-dns. (Step 3.2)
  5. Intermittent exact-5s timeouts = conntrack race × ndots:5 — NodeLocal DNSCache, ndots:2, trailing-dot FQDNs; remember musl ignores glibc resolver options. (Step 3.3)
  6. One node can't reach the ClusterIP = kube-proxy or conntrack exhaustion on that node; resync or recycle it. (Step 3.4)
  7. External-only failures = upstream forwarding — node resolv.conf, outbound-53 firewall rules, VPC resolver rate limits. (Step 3.5)
  8. Alert on SERVFAIL ratio, p99 lookup latency, and empty kube-dns endpoints — DNS should page you before your apps translate it into revenue loss.

Related Reading

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.

Top comments (0)