Breaking Kubernetes DNS on Purpose: A Hands-On CoreDNS Tutorial

Zain Ulabidin — Mon, 09 Nov 2026 12:55:23 +0000

Every pod in a Kubernetes cluster can find every Service by name, with zero DNS configuration from you. Type db in your app's connection string and it just works. This post takes that magic apart, one concept at a time, on a live cluster — because DNS is the thing everyone blames during an outage, and the thing surprisingly few people can actually debug.

Everything below was run on a real cluster on my machine. The outputs are real. You should run it too — total time is about an hour, and the cluster is disposable.

Why kind?

Any local Kubernetes works for this — kind, k3d, minikube, microk8s — because every one of them ships CoreDNS as the cluster DNS (it's been the default since Kubernetes 1.13, replacing the older kube-dns). But they're not equally good for learning:

kind (Kubernetes in Docker) uses a vanilla kubeadm-style deployment, so its CoreDNS setup matches the official docs exactly. What you learn transfers 1:1 to any conformant cluster. Runs entirely in Docker — no VM, cluster up in about a minute, and you can destroy and recreate it freely (which matters, because we're going to break DNS on purpose).
k3d/k3s works too, but k3s manages CoreDNS through its own bundled manifests with quirks — direct edits to the CoreDNS ConfigMap get reverted by its manifest reconciler unless you use its customization hooks (the coredns-custom ConfigMap, NodeHosts). Worth knowing if you run k3s, but it's noise while learning fundamentals.
microk8s on a Mac needs a Multipass VM and treats DNS as an addon (microk8s enable dns) — more friction, less transferable.

Setup:

brew install kind kubectl     # or your platform's equivalent
kind create cluster
kubectl cluster-info --context kind-kind

Kubernetes control plane is running at https://127.0.0.1:55842
CoreDNS is running at https://127.0.0.1:55842/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

CoreDNS is already there. Let's go find it.

Concept 1: CoreDNS is just a normal app running in the cluster

DNS isn't baked into Kubernetes itself. It's an ordinary Deployment, scheduled like any other workload:

kubectl -n kube-system get deploy,pods,svc -l k8s-app=kube-dns -o wide

NAME                      READY   UP-TO-DATE   AVAILABLE   CONTAINERS   IMAGES
deployment.apps/coredns   2/2     2            2           coredns      registry.k8s.io/coredns/coredns:v1.14.2

NAME                           READY   STATUS    IP           NODE
pod/coredns-589f44dc88-jn9fw   1/1     Running   10.244.0.2   kind-control-plane
pod/coredns-589f44dc88-nl8ff   1/1     Running   10.244.0.3   kind-control-plane

NAME               TYPE        CLUSTER-IP   PORT(S)
service/kube-dns   ClusterIP   10.96.0.10   53/UDP,53/TCP,9153/TCP

Three objects, all in kube-system:

A Deployment (coredns, 2 replicas). Two replicas for availability — if one pod dies, DNS keeps working while it restarts. Each pod gets a regular pod IP.
A ClusterIP Service (kube-dns) at 10.96.0.10. The single stable IP every pod sends DNS queries to. Note the name quirk: the Service is called kube-dns even though the software is CoreDNS — kept for compatibility with the DNS server it replaced.
The ports tell you the roles: 53/UDP and 53/TCP are DNS itself (UDP is the normal path, TCP the fallback for large responses). 9153/TCP is not DNS at all — it's Prometheus metrics. CoreDNS reports on itself via a plain HTTP /metrics endpoint: query counts, response codes, cache hits/misses. Nothing scrapes it by default; in production you point Prometheus at it and alert on things like SERVFAIL spikes.

The mental model: pod → asks 10.96.0.10:53 → Service load-balances to one of the CoreDNS pods → CoreDNS answers.

Concept 2: How every pod gets wired to CoreDNS

How does a random pod you deploy know to ask 10.96.0.10? Nobody configures your app pods manually. Look inside one:

kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- cat /etc/resolv.conf

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

These three lines are injected by the kubelet into every pod it starts (for the default dnsPolicy: ClusterFirst). Nothing inside the container image put them there.

nameserver 10.96.0.10 — exactly the kube-dns Service ClusterIP. The kubelet is started with that IP as a flag (--cluster-dns) and stamps it into every pod. That one line is the entire wiring.

search + ndots:5 — these two work together and cause ~80% of real-world Kubernetes DNS confusion, so here's the mechanic precisely:

When a program looks up a name, the resolver counts the dots in it. If the name has fewer than 5 dots (ndots:5) and doesn't end in a literal ., it is not tried as-is first — instead the resolver walks the search domains, appending each:

You look up db → it tries db.default.svc.cluster.local, then db.svc.cluster.local, then db.cluster.local, then bare db. The first search domain is namespace-specific — that's what lets your app connect to just db and reach the db Service in its own namespace.
You look up db.payments → the second search domain expands it to db.payments.svc.cluster.local — cross-namespace shorthand.

The dark side: github.com has only 1 dot — fewer than 5. So a pod resolving it first tries github.com.default.svc.cluster.local (NXDOMAIN), github.com.svc.cluster.local (NXDOMAIN), github.com.cluster.local (NXDOMAIN), and only then github.com. Every external lookup from a pod costs up to four round trips. Remember this for the NodeLocal DNSCache discussion at the end.

Watch the walk happen:

kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes

Server:     10.96.0.10
Address:    10.96.0.10:53

** server can't find kubernetes.svc.cluster.local: NXDOMAIN
** server can't find kubernetes.cluster.local: NXDOMAIN

Name:    kubernetes.default.svc.cluster.local
Address: 10.96.0.1

Bare kubernetes (0 dots) → expanded via the first search domain → answered with the API server's Service IP. (Busybox's nslookup fires the queries for all search domains in parallel instead of stopping at the first hit, which is why the NXDOMAINs print in jumbled order — and each name is looked up twice, once for A/IPv4 and once for AAAA/IPv6.)

Concept 3: The Corefile — CoreDNS is a plugin chain

Now for how CoreDNS knew that answer. Its entire configuration lives in one ConfigMap:

kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}'

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
        disable success cluster.local
        disable denial cluster.local
    }
    loop
    reload
    loadbalance
}

That's the whole DNS config for the cluster — about 20 lines.

The first line, .:53 { ... }, is a server block: "listen on port 53 for queries in zone ." — and . is the DNS root, so every domain matches. Inside the braces is a list of plugins. That's the core idea of CoreDNS: it's not a monolithic DNS server, it's a chain of small plugins, and each query passes through the relevant ones until one produces an answer.

The two plugins doing the real work:

kubernetes cluster.local in-addr.arpa ip6.arpa — the plugin that answered the kubernetes.default.svc.cluster.local query. It declares "I am authoritative for cluster.local." It doesn't read zone files — it watches the Kubernetes API for Services and EndpointSlices and synthesizes DNS answers from them in memory. Create a Service and it's resolvable within seconds, no config change ever. The sub-options: ttl 30 stamps answers valid for 30 seconds; pods insecure enables legacy pod-IP names like 10-244-0-2.default.pod.cluster.local; fallthrough lets reverse-DNS queries it can't answer continue to the next plugin instead of dying as NXDOMAIN.

forward . /etc/resolv.conf — the escape hatch for everything else. Queries that don't match cluster.local (like github.com) get forwarded to whatever nameservers are in the CoreDNS pod's own /etc/resolv.conf — which is the node's resolver (in kind, Docker's DNS). Cluster names answered from the API watch, everything else proxied upstream.

The supporting cast:

Plugin	Job
`errors`	Log errors to stdout
`health` / `ready`	HTTP endpoints for liveness/readiness probes; `lameduck 5s` keeps serving 5s during shutdown so in-flight queries finish
`prometheus :9153`	The metrics endpoint from Concept 1
`cache 30`	Cache upstream answers up to 30s — but note the `disable` lines: caching is off for `cluster.local`, because the `kubernetes` plugin's in-memory data is already instant and always fresh; caching it would only delay seeing Service changes
`loop`	Detect forwarding loops and crash on purpose (better than melting down) — a common cause of CrashLoopBackOff on hosts whose `/etc/resolv.conf` points at `127.0.0.1`
`reload`	Watch this very ConfigMap and hot-reload on change — no pod restart needed
`loadbalance`	Shuffle the order of A records in answers — poor man's round-robin

Concept 4: Reading the query log

The log plugin makes CoreDNS print every query it receives — the single most useful debugging tool you have. Enable it by adding one word to the Corefile:

kubectl -n kube-system edit configmap coredns
# add a line containing just:  log  inside the .:53 { } block

Thanks to the reload plugin, the logs soon show the config being hot-swapped — no restart:

[INFO] Reloading
[INFO] plugin/reload: Running configuration SHA512 = 2dd49c56...
[INFO] Reloading complete

Now tail the logs (note -l k8s-app=kube-dns grabs both replicas — the Service load-balances queries between them, so tailing just one pod misses half the traffic):

kubectl -n kube-system logs -f -l k8s-app=kube-dns

...and in another terminal, generate some traffic:

kubectl run dnsgen --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c 'nslookup kubernetes; nslookup example.com; nslookup example.com'

Here's what came back, and it contains everything from Concepts 1–3:

[INFO] 10.244.0.9:55719 - 60987 "A IN kubernetes.cluster.local. udp 42 false 512" NXDOMAIN qr,aa,rd 135 0.000981587s
[INFO] 10.244.0.9:55719 - 61125 "A IN kubernetes.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd 139 0.000780419s
[INFO] 10.244.0.9:55719 - 65472 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 106 0.001500255s
[INFO] 10.244.0.9:34990 - 21746 "A IN example.com. udp 29 false 512" NOERROR qr,rd,ra 83 0.027218625s
[INFO] 10.244.0.9:55840 - 22203 "A IN example.com. udp 29 false 512" NOERROR qr,aa,rd,ra 83 0.000005625s

Anatomy of a line, reading left to right: 10.244.0.9:55719 is the client pod and port, 60987 the query ID, A the record type (A = IPv4, AAAA = IPv6), kubernetes.cluster.local. the queried name, udp the transport (TCP is the fallback for large answers), NXDOMAIN the result code, qr,aa,rd the DNS flags, 135 the answer size in bytes, and the trailing number the duration.

Three stories hide in those five lines:

The search walk, server-side (lines 1–3). The same nslookup kubernetes from Concept 2, seen from CoreDNS's perspective: two NXDOMAINs, then NOERROR. Note the aa flag on all three — authoritative answer. The kubernetes plugin owns cluster.local and answers from its API-watch data. That's also why they're fast: about 1 millisecond.

A forwarded query (line 4). First example.com lookup: no aa flag, and ra (recursion available) instead — not answered locally, it went through forward to the upstream resolver. Cost: 27 milliseconds, ~27× slower than a cluster answer, because it left the building.

The cache earning its keep (line 5). Second example.com lookup, seconds later: same answer, duration 15 microseconds — the cache 30 plugin serving from memory, roughly 1,800× faster than the forwarded query.

One grep shows you who asked, what they asked, which plugin answered (the aa flag is the tell), and what it cost.

Production note: log prints every query in the cluster — fine for a lab, noisy and disk-hungry at scale. Turn it on for debugging, off after.

Concept 5: Service discovery — normal vs headless

Time to close the loop on the kubernetes plugin: create workloads and watch DNS records appear out of thin air.

kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl expose deployment web --port=80
kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup web

Name:    web.default.svc.cluster.local
Address: 10.96.24.66

Zero DNS config written, yet it resolves. Two things to notice:

The answer is one A record — the Service's ClusterIP, not any of the 3 pod IPs. Load-balancing to pods happens later, in the network layer (kube-proxy), invisible to DNS.
In the query log, this answer carries the aa flag — the kubernetes plugin saw the Service appear via its API watch and started answering instantly.

Now the interesting variant. Same pods, second Service, one crucial difference — clusterIP: None:

kubectl create service clusterip web-headless --clusterip=None --tcp=80
kubectl set selector service web-headless app=web
kubectl scale deployment web --replicas=5
kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup web-headless

Name:    web-headless.default.svc.cluster.local
Address: 10.244.0.20
Name:    web-headless.default.svc.cluster.local
Address: 10.244.0.19
Name:    web-headless.default.svc.cluster.local
Address: 10.244.0.12
Name:    web-headless.default.svc.cluster.local
Address: 10.244.0.18
Name:    web-headless.default.svc.cluster.local
Address: 10.244.0.17

No ClusterIP exists, so DNS can't hand out a single VIP — instead you get one A record per pod, the actual pod IPs. The answer is a live view of the EndpointSlice: scale the Deployment and the record set follows within seconds. The client sees every backend and picks one itself — this is how client-side load balancers (gRPC and friends) discover all endpoints, and the foundation of how StatefulSets give each database replica an addressable name.

Two closing details:

Per-pod names. With a headless Service, each backend also gets its own DNS name — the pod IP with dashes, prefixed to the service name:

nslookup 10-244-0-20.web-headless.default.svc.cluster.local   # → 10.244.0.20

StatefulSets improve on this: their pods get stable names like db-0.web-headless.default.svc.cluster.local instead of IP-based ones — that's the actual mechanism behind "connect to the primary at db-0".

The TTL. Busybox's nslookup hides it, but every answer carried ttl 30 from the Corefile — a client may cache the pod list for up to 30 seconds. That's the staleness window: scale down, and a client that cached the answer can keep trying a dead pod IP for up to 30s. It's a tradeoff between DNS load and freshness — now you know where the knob is.

Concept 6: Breaking DNS on purpose

The skill you'll actually use at 2am. Two failures, two very different presentations.

Failure A — CoreDNS is down (total outage)

kubectl -n kube-system scale deployment coredns --replicas=0
kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup web

;; connection timed out; no servers could be reached

Not an NXDOMAIN, not an error answer — a hang, then timeout. Nothing is listening behind 10.96.0.10. Note how slow failure is compared to a crisp NXDOMAIN — this is why DNS outages make everything look "slow" before anyone realizes it's DNS. Also: kubectl itself keeps working (it doesn't use cluster DNS), and running pods with established connections keep working — only new name resolutions die.

Restore:

kubectl -n kube-system scale deployment coredns --replicas=2

Failure B — CoreDNS is up but misconfigured (the subtle one)

Edit the ConfigMap and change forward . /etc/resolv.conf to forward . 10.99.99.99 (a black-hole IP). Wait for the reload, then test one cluster name and one external name:

kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup web
# Name: web.default.svc.cluster.local  Address: 10.96.24.66     ← WORKS

kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- nslookup example.com
# ;; connection timed out; no servers could be reached          ← TIMES OUT

That asymmetry is the diagnosis. Memorize the pattern:

Cluster names resolve, external names don't → CoreDNS itself is fine; the problem is upstream/forwarding. The kubernetes plugin answers cluster.local from memory — the broken forward never gets involved. Both fail → the problem is CoreDNS itself or the network path to it.

Two lookups, and you've localized the fault.

One more lesson hides in the recovery: after fixing the ConfigMap back, the first retry still timed out; the second succeeded. That's config propagation delay — the kubelet syncs ConfigMap volumes into pods periodically (up to ~1 minute), and the reload plugin checks for changes every ~30s on top of that. A Corefile fix can take up to ~90 seconds to bite. Knowing that saves you from "my fix didn't work!" panic-reverts.

The DNS debugging checklist

Covers ~95% of real incidents:

1. Characterize from inside a pod — one cluster name, one external name:

kubectl run dbg --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c 'nslookup kubernetes.default; nslookup example.com'

Both fail → step 2. Only external fails → check the forward target and the upstream resolver. One specific service fails → step 4.

2. Is CoreDNS alive?

kubectl -n kube-system get pods -l k8s-app=kube-dns

CrashLoopBackOff or frequent restarts? Classic cause: the loop plugin detecting a forwarding loop and killing the process on purpose — common on hosts whose /etc/resolv.conf points to 127.0.0.1.

3. Read the logs:

kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50

Reload errors mean a broken Corefile. With log enabled you can see whether queries even arrive, and their rcodes.

4. For one failing service — does it have endpoints?

kubectl get svc web
kubectl get endpointslices -l kubernetes.io/service-name=web

A selector typo gives you a Service with zero endpoints. Gotcha: the normal Service name still resolves (DNS hands out the ClusterIP regardless — connections just go nowhere), but a headless Service with no endpoints returns NXDOMAIN. Same root cause, two different symptoms.

5. Bypass the Service layer — query a CoreDNS pod IP directly:

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide   # grab a pod IP
kubectl run dbg --image=busybox:1.36 --rm -it --restart=Never -- nslookup web 10.244.0.X

Works against the pod IP but not against 10.96.0.10 → CoreDNS is innocent; your problem is kube-proxy / the Service VIP plumbing.

Cleanup

kubectl delete deployment web
kubectl delete service web web-headless
# and remove the `log` line from the coredns ConfigMap, or just:
kind delete cluster

Where to go next

Compare distros: if you have a k3s/k3d cluster, diff its Corefile against kind's (kubectl -n kube-system get cm coredns -o yaml on each) and look up k3s's coredns-custom ConfigMap pattern — k3s's manifest reconciler reverts direct Corefile edits, so customization goes through dedicated hooks.
Customization: add a stub domain (forward corp.example.com to an internal resolver), rewrite rules, or static records via the hosts plugin.
NodeLocal DNSCache: a per-node caching DNS agent that intercepts pod queries before they cross the network — the standard fix at scale for the ndots:5 query-storm problem you saw in Concept 2.
Reference: the CoreDNS plugin list — the Corefile is small precisely because each plugin does one thing; skimming the list tells you what else the chain can do.

Tested on kind with Kubernetes v1.36.1 / CoreDNS v1.14.2.

DEV Community: Zain Ulabidin