Why One DNS Query Becomes Eight: Kubernetes and ndots:5

Step into a Kubernetes pod and run cat /etc/resolv.conf; you get three lines. The first two look familiar: a nameserver, a search list. The third, options ndots:5, is the one most eyes slide over. Yet that line quietly turns a single external name lookup by any application in the cluster into eight, sometimes twelve DNS packets, and it has been doing so since the moment the pod was born.

While preparing this piece I read a GitHub thread from 2016 end to end; why the number is 5, why it could not be 1, and which resolver plays by which rule is all sitting there line by line. My thesis: ndots:5 is not a bug, it is a deliberate trade; in Tim Hockin's own words from that thread, "a tradeoff between automagic and performance". But the cost of the trade is paid not by whoever built the cluster but by the application calling an external API, and until you see where the bill accumulates you can neither defend it nor fix it.

I have covered Kubernetes DNS on this blog before from the angle of cache poisoning; this article is not about security but about search-list mechanics and their cost. If you are curious how the same resolv.conf can lay a trap on a single machine, the nginx DNS trap post is a neighbouring story.

What the search list solves, and what it breaks

The definition in the resolv.conf man page is short: ndots sets the number of dots a name must contain before it is tried "as is", that is, as an absolute name. The default is 1. So if a name contains even a single dot, the resolver asks for it directly first and only falls back to the search list if nothing is found. Below the threshold the order flips: each suffix on the search list is appended and tried in turn, and the absolute name comes last.

Kubernetes raises that threshold to 5 and fills the search list like this:

nameserver 10.32.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The goal is for an application that writes redis to reach redis.default.svc.cluster.local, and one that writes redis.cache to reach redis.cache.svc.cluster.local. The "type the name and it works" feel of service discovery comes from this line. In December 2016 Hockin laid out in ten steps how the number was derived; the short version: same-namespace services are the most frequent lookup, so redis must resolve (threshold at least 1); cross-namespace kubernetes.default must resolve (at least 2); name.namespace.svc (at least 3); StatefulSet pod names such as pod-0.redis.cache.svc (at least 4); and SRV names of the form _http._tcp.redis.cache.svc (5). Every in-cluster shortcut is therefore tried through the search list first and found on the first attempt. In the same comment Hockin admits that SRV on top of StatefulSets would actually require 6: "We did not change ndots to 6 because this is getting out of hand."

The cost shows up on every name that does not belong to the cluster. api.github.com carries two dots; the threshold is 5. glibc tries the name with three suffixes first, then in its absolute form:

api.github.com.default.svc.cluster.local   → NXDOMAIN
api.github.com.svc.cluster.local           → NXDOMAIN
api.github.com.cluster.local               → NXDOMAIN
api.github.com.                            → answer

Four names. Since glibc 2.9 sends the A and AAAA lookups for each name in parallel, those four names mean eight DNS packets. Six of them are lost before they leave. The application made one HTTP request; CoreDNS saw eight queries.

The node's own search list lands on the bill too

Three suffixes are the part that comes from Kubernetes. When kubelet generates the pod's resolv.conf, it layers the node's own search list on top. In the 2016 thread, the file Hockin pasted from his cluster on Google Compute Engine shows this clearly:

search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.thockin-dev.internal
nameserver 10.0.0.10
options ndots:5

Five suffixes. For api.github.com that means six names and twelve packets; ten of them wasted. In the same thread another participant describes a cluster with six search domains where a name outside the cluster, application.internal.my.domain, reached the right answer only after twelve failed queries every single time. Those two or three extra suffixes cloud providers hand to nodes are a tax the developer never sees and pays on every request. And the two kinds of wasted query do not cost the same: the NXDOMAINs with a cluster.local suffix are produced by CoreDNS inside its own authoritative zone, without asking anyone upstream; the ones born from node suffixes such as google.internal leave the cluster and travel upstream. The real latency comes from that second group.

The only measured number in the thread makes the tax concrete. One participant reports measuring all DNS traffic in a small cluster of 7 nodes and 81 pods at 1,246 packets per second with the default settings, and 109 packets per second after lowering ndots to 1 for most pods. More than a tenfold difference, without adding a single service to the cluster.

There is a ceiling, because the list cannot grow forever. For a long time the Kubernetes limit matched glibc's old one: six domains, 256 characters. When the ExpandedDNSConfig feature went stable in 1.28 the ceiling rose to 32 domains and 2048 characters; today a pod is rejected only when it exceeds those. Before glibc 2.26 the limit there was also six domains and 256 characters; older containerd (1.5.5 and earlier) and CRI-O (1.21 and earlier) releases can leave a pod stuck in Pending on long lists. musl silently ignores a search line longer than 256 characters. So the same pod spec can come up with different DNS behaviour depending on the libc underneath it.

Why they did not pick 1: resolvers do not play by the same rule

The most instructive part of the thread is why the "then let's use ndots:1" proposal was rejected. glibc's man page promises this: if a name has at least the threshold number of dots, the absolute query goes first, and if it is not found the search list is still tried. So with ndots:1 an application that writes redis.cache first asks global DNS for redis.cache., gets NXDOMAIN, then finds redis.cache.svc.cluster.local. Slower, but not broken.

The problem is that not everyone is glibc. In the same thread Hockin sets ndots:1 in an Ubuntu pod and shows nslookup kubernetes.default returning NXDOMAIN; another participant, Mac Browning, pastes the tcpdump output of the same query from his own pod:

56391+ A? kubernetes.default. (36)
56391 NXDomain 0/0/0 (36)

One query, NXDOMAIN, without ever falling back to the search list. Because nslookup and dig use BIND's resolver library rather than glibc's, and that library asks a name past the threshold only in absolute form and stops. musl chose the same rule; the reasoning on its wiki is worth reading in my view: falling back to the search list because a name was not found in global DNS means the same name can go somewhere else when a new TLD is registered or a transient failure occurs, and musl counts that as inconsistency. Go's own resolver, on the other hand, behaves like glibc; the nameList function in its source tries a name past the threshold absolute first, then with suffixes, and a name ending in a dot only as is.

Hockin's 2016 conclusion still holds: "resolv.conf behavior is under-specified here and in other ways." A cluster that drops the threshold to 1 works on glibc-based images, breaks the redis.cache shortcut on Alpine, and misleads the engineer debugging with dig. Hockin's second reason is entirely about frequency: in-cluster names are looked up more often than external ones and their TTLs are very short; trying the absolute name first would slow down every in-cluster lookup, "the wrong tradeoff". 5 is the smallest threshold that preserves every in-cluster shortcut including SRV names; that is why it stayed.

Resolver	Name at or above the dot threshold	Name ending in a dot
glibc	Absolute first, search list if not found	Absolute only
musl (1.1.13+)	Absolute only, never falls back to search	Absolute only
Go (pure resolver)	Absolute first, then search list	Absolute only
BIND library (dig, nslookup)	Absolute only	Absolute only

The last column of that table is also the key to the cheapest fix in the rest of this article.

Why eight queries turn into 5 seconds

Eight packets could have been too cheap to measure on their own. One of the longest DNS threads in Kubernetes history, "DNS intermittent delays of 5s", opened in 2017 and collecting 273 comments, explains why those packets occasionally cost five seconds. The short version: UDP DNS queries pass through kube-proxy's DNAT and conntrack; the A and AAAA packets leaving the same socket at the same instant hit a race condition in conntrack and one of them is dropped. The price of the dropped packet is glibc's default timeout: 5 seconds.

ndots:5 is not the cause of that race; it is the multiplier. Every extra name is two more lottery tickets. For the race itself there are two kernel patches, the second released with Linux 5.0 in 2019; but the engineer who wrote them warns plainly in the same thread that in clusters running more than one DNS server instance the timeouts do not disappear completely because of how kube-proxy load-balances, and IPVS mode is even more prone. So even in 2026 the number of tickets still matters.

The NodeLocal DNSCache documentation writes this down as a direct design rationale: the local cache skips DNAT and conntrack, upgrades the connection to CoreDNS to TCP for the cluster domain, and can cache negative answers as well. The default in the sample manifest keeps negative answers for 5 seconds; short, but enough to absorb the six wasted NXDOMAINs inside the node for a service making requests in a tight loop. External names in the .:53 block still go upstream over UDP; the TCP upgrade applies only to cluster.local and the reverse-lookup zones.

Five minutes is enough to see this in your own cluster. Temporarily enable CoreDNS's log plugin, run getent ahosts api.github.com from a pod and count the lines. The ahosts matters: getent hosts asks for A and AAAA one after the other, whereas ahosts exercises the getaddrinfo path that applications use. If the number is eight and your node's own search list is empty, theory and practice agree. If it is higher, look for the difference in the node's resolv.conf.

The security angle: RFC 1535's warning from 1993

Search-list mechanics are not new; neither is their danger. RFC 1535, dated October 1993, is a security note describing how BIND-based resolvers completing partial names with a search list could deliver a name to the wrong machine. The example in the document is chilling even today: a user on a machine under .com trying to reach UnivHost.University.EDU has their resolver ask for UnivHost.University.EDU.COM. first because of the search list; once someone registers the edu.com domain and adds a wildcard CNAME, every connection from .com to .edu terminates on that person's machine. The RFC's recommendation has two parts: the implicit search list derived from the parent components of a name should be disabled by default (BIND 4.9.2 did exactly that), and "in any event where a '.' exists in a specified name it should be assumed to be a FQDN and SHOULD be tried as a rooted name first." ndots:5 is a deliberate violation of that second recommendation; Kubernetes knows the lesson of 1993 and accepts the risk for the sake of automagic.

In Kubernetes the same mechanism runs even more aggressively, because the threshold is 5. Consider a two-label external name: redis.io. One dot, far below the threshold. glibc asks for redis.io.<namespace>.svc.cluster.local first, then redis.io.svc.cluster.local. To CoreDNS the second one is the redis service in the io namespace. If your cluster has a namespace called io with a service called redis in it, a pod that wants redis.io connects to the in-cluster service and sees that not as an error but as a working connection. I have not lived through this as an incident; it is the plain consequence of the mechanics, and it deserves a line in your namespace naming rules: do not name namespaces after TLDs.

Four fixes, four different bills

That was the anatomy of the problem. Now the options; each one's cost is billed to a different team.

1. Add a trailing dot. Writing https://api.github.com./v3 in the application's configuration bypasses the search list on all four resolvers in the table, at zero cost. The catch: some HTTP clients and TLS libraries treat a trailing-dot hostname differently in the Host header or during certificate validation. I would not ship this to production without a library-specific test; but for database connection strings, SMTP servers and gRPC targets, where TLS name matching is more predictable, it is the first thing I would try.

2. Per-pod dnsConfig. The dnsConfig field, stable since Kubernetes 1.14, lets you change the threshold at the pod level:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

This is the right tool for workloads that make many external API calls and address in-cluster services by their full names. The side effect is precisely this: shortcuts whose dot count is equal to or above the new threshold are affected. With ndots:2, redis and redis.cache are untouched, both stay below the threshold and are tried through the search list first; redis.cache.svc and SRV names such as _http._tcp.redis.cache.svc get slower on glibc, which tries the absolute name first, and break on musl, which never falls back to the search list. The other side of the coin: single-dot external names such as redis.io still produce four names and eight packets under ndots:2. When you lower the threshold, make writing every dependency of the pod by its full name the standard; a half-finished migration is the worst outcome.

3. CoreDNS autopath. The search list can also be walked on the server side instead of the client. When the autopath @kubernetes plugin sees a query containing the first suffix, it follows the chain itself and returns the first non-NXDOMAIN answer with a CNAME. The client ends up sending one query. The price has two layers: the plugin needs the pods verified mode to know which pod a query came from, and since that mode makes CoreDNS watch every pod, the documentation explicitly says "substantially more memory". The documentation also notes that when a pod is deleted and its IP is immediately handed to a pod in another namespace, autopath can resolve against the wrong namespace, and that if the server-side search ends negative, the client will still walk the whole list by hand. It is also incompatible with Windows nodes.

4. NodeLocal DNSCache. This lowers not the number of queries but the price of each one. Running as a DaemonSet on the node, the cache listens on a link-local address such as one from 169.254.0.0/16; pods reach it without touching DNAT or conntrack, and it goes out to CoreDNS over TCP for the cluster domain. Stable since 1.18. In return: one more component per node, with its own memory limit and its own upgrade calendar. And one incompatibility: autopath identifies the requesting pod by the source IP of the packet; with NodeLocal in between, the packet reaching CoreDNS comes from the node, not the pod, and autopath is left blind. The third and fourth fixes do not work together.

Beside these there are three smaller knobs; not fixes, mitigations. options no-aaaa, added in glibc 2.36, switches off AAAA queries entirely in a pod that does not use IPv6 and halves the packet count. single-request-reopen is the workaround suggested again and again in the 5-second thread; when one of the A/AAAA pair sent from the same socket goes unanswered, it closes the socket and sends the second request from a new one, restarting the race instead of paying the 5-second price of the dropped packet. dnsPolicy: Default hands a pod that only talks to the outside world the node's resolv.conf as is and frees it from the cluster search list entirely, knowing that if the node's resolver does not know cluster names, that pod cannot resolve any in-cluster name.

The right trade here, in my view, is to combine two of these: NodeLocal DNSCache on the platform side, and on the application side the habit of writing external dependencies by their full names. autopath is a separate decision for large clusters that can absorb the memory cost and the namespace race and do not run NodeLocal; ndots:2 I would apply not cluster-wide but to specific workloads whose external traffic has been measured.

Windows nodes are a different world

The Kubernetes documentation places Windows nodes outside this picture, and rightly so. Windows treats every name containing a dot as an FQDN and skips the search list; there is only one DNS suffix per pod, the namespace's own <namespace>.svc.cluster.local. A Windows pod can resolve kubernetes and kubernetes.default.svc.cluster.local but not partial names such as kubernetes.default. In mixed clusters this means the same Helm chart works on Linux and fails name resolution on Windows. The documentation's advice is to test name resolution in Windows pods not with nslookup but with Resolve-DNSName, because Windows has several resolvers and their behaviour differs in small ways.

Checklist

Look at your own cluster in this order:

1. Read the node's search list. cat /etc/resolv.conf in a pod; if you see more than three suffixes, the extras come from the node and add two packets to every external query.
2. Measure the wasted queries. In CoreDNS, look at the ratio of coredns_dns_responses_total{rcode="NXDOMAIN"} to total responses; NXDOMAINs with a cluster.local suffix are ndots' signature.
3. Write external dependencies by full name. Identify where connection strings can take a trailing-dot name and test TLS name matching with that library.
4. Know your image base. In Alpine/musl-based workloads, before lowering ndots find every shortcut whose dot count reaches the new threshold (redis.cache.svc and SRV names for ndots:2); musl does not fall back to the search list.
5. Do not let namespace names collide with TLDs. Names like io, dev and app pull external names into the cluster once combined with the search list.
6. Evaluate NodeLocal DNSCache. If you see 5-second stalls, the root cause is not ndots but the conntrack race; ndots only increases the number of tickets.
7. Remember the rule when debugging with dig. dig never looks at the search list unless given +search, and even then it does not imitate glibc's fallback behaviour; use getent ahosts to see what the application sees.

The long shadow of a deliberate default

ndots:5 is the smallest threshold that preserves every in-cluster shortcut including SRV names, and the Kubernetes promise of "write the service name and it works" rests on it. It is also a deliberate violation of a 1993 RFC's advice to "try a dotted name as rooted first", and an eight-packet tax that every application calling an external API pays without knowing. Those two sentences do not contradict each other; they are two faces of the same decision.

The lesson I take from this is not about the number. Hockin wrote in 2016 that "good caching means that external lookups are slow the first time but fast subsequently, so that's where we've been focused"; a decade later NodeLocal DNSCache is exactly the product of that sentence. So the counterpart of ndots:5 was never written into the documentation, but it was written into the architecture; the team that reads the three-line example and moves on pays the price because it never built the rest of that architecture. Understanding a default means knowing who chose it, and in exchange for what. The rest is cat /etc/resolv.conf.

Official Sources

• DNS for Services and Pods — Kubernetes documentation (dnsPolicy, dnsConfig, search list limits, Windows notes)
• Using NodeLocal DNSCache in Kubernetes Clusters — Kubernetes documentation
• Kube-dns add-on should accept option ndots — kubernetes/kubernetes #33554 (September 2016)
• DNS intermittent delays of 5s — kubernetes/kubernetes #56903 (December 2017)
• net/dnsclient_unix.go — search list logic of the Go resolver (nameList)
• RFC 1535 — A Security Problem and Proposed Correction With Widely Deployed DNS Software (October 1993)