DEV Community

Hilmi B
Hilmi B

Posted on

Kubernetes DNS was failing 33% of the time. CoreDNS was fine.

#kubernetes #devops #networking #debugging

A WordPress site on our platform started throwing Error establishing a database connection. Not always — maybe one page load in three. Reload, and it worked. Reload again, broken.

The obvious suspect is MySQL. It wasn't MySQL. It took us longer than it should have to stop looking there, so here is the whole path, including the two wrong turns.

The first wrong turn: blaming the database

WordPress talks to MySQL through a hostname, not an IP. In our case the pod's config had:

WORDPRESS_DB_HOST = mysql.cdn.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

That single detail is the whole story, but we didn't see it yet. We went to MySQL first and checked whether we were simply out of connections:

SHOW GLOBAL STATUS LIKE 'Connection_errors_max_connections';
Enter fullscreen mode Exit fullscreen mode

Zero. No saturation, no slow queries, server healthy, direct connections from outside the cluster fine. So the database was answering everyone — just not always this pod.

The canary nobody reads

WordPress does not log the DB connection failure when WP_DEBUG is off. That's why this class of bug feels random: the failure is invisible in the application log.

But the same pod ran Redis for object caching, and Redis does log:

Redis::connect(): getaddrinfo for wp-redis-cache failed: Temporary failure in name resolution
Enter fullscreen mode Exit fullscreen mode

getaddrinfo … Temporary failure in name resolution. That's EAI_AGAIN — a DNS timeout. Redis connects by hostname too, so it was failing for exactly the same reason as MySQL, and it was kind enough to say so out loud.

If you take one thing from this post: when a hostname-based dependency fails intermittently and silently, go find a noisier hostname-based dependency in the same pod and read its log.

The second wrong turn: blaming CoreDNS

So it's DNS. The reflex is "CoreDNS is unhealthy" or "we need NodeLocal DNSCache." We reached for both, and both were wrong.

CoreDNS pods were all Running, all Ready, no restarts, no errors in their logs. And the failure rate had a suspicious shape: roughly one in three.

We had three CoreDNS endpoints.

That ratio is not a coincidence, and it's the clue that changes the question from "is CoreDNS broken?" to "is one specific path to one specific CoreDNS pod broken?" A Service ClusterIP hides which backend answered you, so testing mysql.cdn.svc.cluster.local in a loop just gives you a mushy ~33% failure rate with no information about which backend is bad.

The diagnostic that actually pinned it

Stop querying the Service. Query each CoreDNS pod IP individually, from a pod pinned to the affected node.

Get the endpoints:

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
Enter fullscreen mode Exit fullscreen mode

Then run a debug pod on the node that's having trouble (nodeName pins it — don't leave this to the scheduler, the whole point is which node you're testing from):

kubectl run dnstest --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeName":"<affected-node>"}}' -- sleep 3600
Enter fullscreen mode Exit fullscreen mode

And loop against each CoreDNS pod IP separately (IPs below are anonymised — use whatever the command above printed):

for ip in <coredns-ip-1> <coredns-ip-2> <coredns-ip-3>; do
  ok=0; fail=0
  for i in $(seq 1 30); do
    if kubectl exec dnstest -- nslookup mysql.cdn.svc.cluster.local $ip >/dev/null 2>&1; then
      ok=$((ok+1)); else fail=$((fail+1)); fi
  done
  echo "$ip -> ok=$ok fail=$fail"
done
Enter fullscreen mode Exit fullscreen mode

The output made it obvious:

coredns-1 -> ok=30 fail=0
coredns-2 -> ok=0  fail=30
coredns-3 -> ok=30 fail=0
Enter fullscreen mode Exit fullscreen mode

One endpoint, 30 out of 30 failures. The other two, perfect. kube-dns round-robins across all three, so a third of every pod's lookups on that node went into a black hole.

Then the confirmation step that tells you whether the pod is broken or the link is broken: repeat the same loop from a pod on a different node. From elsewhere, the failing endpoint answered fine. So the CoreDNS pod was healthy. What was broken was the path between two specific nodes.

The actual cause

We run Weave for the pod network. The control plane happily reported established fastdp between the two nodes — the fast datapath was "up" as far as Weave's own status was concerned. But pod-to-pod traffic across that particular link was dropping 100%. A stale kernel datapath flow, reported healthy, forwarding nothing.

The fix was almost insultingly small — delete the weave-net pod on the affected node and let the DaemonSet recreate it, which rebuilds the datapath flows:

kubectl -n kube-system delete pod weave-net-xxxxx
Enter fullscreen mode Exit fullscreen mode

Back in about sixteen seconds. DNS 30/30 from every endpoint. The WordPress errors stopped.

Why NodeLocal DNSCache didn't save us

The instinctive hardening move here is nodelocaldns: cache DNS on every node, stop crossing the overlay for every lookup. We tried it as a canary on one node and rolled it back, because in this topology it makes things worse in two ways:

  1. The node-cache runs on hostNetwork and still has to reach the kube-dns-upstream ClusterIP over the same overlay — so it inherits exactly the network fault we were trying to route around.
  2. It inherits the black-holed upstream endpoint too. Now you've cached your way to a node whose only path to some upstreams is broken.

NodeLocal DNSCache reduces DNS load and latency. It is not a fix for a broken overlay link, and if you deploy it while an overlay link is broken, you can hand yourself a node where DNS is worse than before.

What we changed

Two things, one tactical and one structural.

Tactically, for anything where a hostname buys you nothing, we stopped depending on DNS. A managed database that lives at a stable ClusterIP does not need to be resolved on every connection:

WORDPRESS_DB_HOST = 10.107.207.215
Enter fullscreen mode Exit fullscreen mode

Less elegant, and you must remember it if the Service is ever recreated. But it removes a whole class of intermittent failure from the request path, and for a customer's site that's the right trade.

Structurally, the honest lesson is about our own topology. Our cluster is stretched — nodes in Istanbul and nodes at a European provider, on one overlay. That's a deliberate choice for the platform we run, and it works, but a stretched overlay means the link is a first-class failure domain, not just the node and the pod. We now treat "which node pair" as a thing to test explicitly, and per-endpoint DNS probing is a standing runbook step rather than something we invent under pressure.

The three-line version

  • Intermittent Error establishing a database connection with no MySQL symptoms is usually DNS, and WordPress won't tell you — read the log of any other hostname-based client in the pod.
  • A failure rate that matches 1/n where n is your endpoint count means "one endpoint," not "the service." Probe pod IPs individually, from a pod pinned to the affected node.
  • "Established" in a network overlay's own status output is a claim, not a measurement. Verify it with traffic.

I work on cdn.com.tr, a CDN and managed container platform run out of Istanbul. Most of what I write about is whatever broke that week.

Top comments (0)