DEV Community

Harish
Harish

Posted on

DNS is weird inside k8s on AWS, Part 2: the layer I didn't know existed

#kubernetes #dns #microservices

A follow-up to Part 1, which laid out the three usual suspects for DNS timeouts inside a Kubernetes pod on EKS: ndots amplification, NodeLocal DNS caching, and the per-ENI 1024 pps limit on the VPC resolver. Part 1 ended with the line "proving which link in the chain is actually responsible takes measurement, not assumption." This is what happened when I did the measurement — and found a fourth link that isn't in most write-ups.

Where we left off

The symptom was familiar to anyone who's chased flaky Kubernetes DNS. A worker pod, blasting a modest number of outbound requests, would occasionally log this exact line:

read udp 100.99.139.35:41290->169.254.20.10:53: i/o timeout
Enter fullscreen mode Exit fullscreen mode

It wasn't a firehose. Some pods hit it dozens of times an hour, others none. Restarting the offending pod usually cleared it for a few minutes, then it came back.

Part 1's framework said: figure out which of the three layers is dropping packets. So I did that.

What the measurement said (and what it didn't)

The three-layer diagnostic worked exactly as advertised — for the layers it covers.

Layer 1 — ndots. Confirmed the pod's /etc/resolv.conf had the classic options ndots:5 with four search domains, so each logical lookup for something like example.com was becoming ~10 packets (five names, each with A + AAAA). Setting ndots:1 cut the amplification by 5×. Timeouts still happened.

Layer 2 — NodeLocal DNS. The DaemonSet was running fine on every node. coredns_forward_healthcheck_broken_total was zero. Forward latency to upstream was under 32 ms at the 99th percentile. Cache miss rate was healthy. No restarts, no OOMs, no ratelimit plugin even loaded. If NodeLocal was the culprit, its own metrics would say so.

Layer 3 — EC2 ENI 1024 pps cap. This is the tricky one because AWS drops the packets silently. But ethtool -S <primary ENI> | grep linklocal_allowance_exceeded on the affected nodes returned zero, even during a spike window. No ENI cap was being hit.

So the packet counts I could measure said: none of the three layers Part 1 covered were the actual problem.

And yet — pods were still logging 169.254.20.10:53: i/o timeout at the same rate.

The question a DevOps engineer asked me

I posted the metrics in a Slack channel. Someone I hadn't worked with before replied with a single line:

This seems to be pointing to istio DNS capture I think. Can be a candidate to disable entirely.

I didn't know what Istio ambient DNS capture was. That turned out to be the whole problem.

The layer that was hiding in plain sight

If you're running Istio in ambient mode (as opposed to the older sidecar mode), you have a component called ztunnel running as a DaemonSet on every node. Ambient mode is the "no sidecars" flavor of the service mesh — instead of a proxy container in every pod, the mesh functions live in ztunnel and get applied to pod traffic transparently at the node level.

One of the things ztunnel does — if the feature is on — is intercept DNS syscalls from pods and answer them itself for known cluster services, forwarding the rest onward. This is the "DNS capture" feature, and it's meant to help with things like ServiceEntry resolution and consistent DNS behavior across the mesh.

The catch: it inserts an extra hop between your pod and NodeLocal DNS.

Where Part 1's mental model was:

pod ──► 169.254.20.10 (NodeLocal DNS) ──► kube-dns / VPC resolver
Enter fullscreen mode Exit fullscreen mode

With ambient DNS capture on, the actual path is:

pod ──► ztunnel (DNS capture)  ──► 169.254.20.10 ──► kube-dns / VPC resolver
Enter fullscreen mode Exit fullscreen mode

That extra hop has its own concurrency limits, its own timeouts, its own queue, its own bugs. And crucially: if ztunnel drops a UDP packet, the client-visible symptom is identical to what you'd see if NodeLocal itself dropped it. The pod's syscall returns i/o timeout on 169.254.20.10:53, because from the pod's perspective, that's the address it was talking to. The interception is invisible.

This is why every metric I could pull from NodeLocal or the VPC resolver came back clean. The drop was happening before the packet ever left ztunnel.

The one-line fix

Istio has an annotation for exactly this situation:

metadata:
  annotations:
    ambient.istio.io/dns-capture: "false"
Enter fullscreen mode Exit fullscreen mode

Applied to a workload pod, ztunnel stops intercepting that pod's DNS syscalls. Traffic goes straight from the pod to 169.254.20.10 (NodeLocal), exactly like Part 1's diagram assumed.

We rolled the annotation out chart-wide (helm values under default.pod.annotations so every service inherits it), redeployed, and repushed a batch of 100 domains through the crawler to stress-test.

The i/o timeout log stream went to zero and stayed there.

How to check if this is you

Three quick probes, no permissions you don't already have.

1. Is your namespace enrolled in ambient?

kubectl get ns <your-namespace> -o yaml | grep -iE 'ambient|dataplane-mode'
Enter fullscreen mode Exit fullscreen mode

Look for istio.io/dataplane-mode: ambient on the namespace or the pod. If that label is present, ambient is active on your workload and DNS capture may be intercepting.

2. Is ztunnel running on your nodes?

kubectl -n istio-system get ds ztunnel
Enter fullscreen mode Exit fullscreen mode

If that DaemonSet exists and has running pods on your nodes, DNS capture is a possibility.

3. Compare error rate before and after the annotation.

Put the annotation on one deployment, redeploy just that one, and Loki-count read udp timeouts before-vs-after over a 1-hour window. If the annotation is the difference-maker, you'll see it drop off cleanly.

Why this is worth writing down

The three layers Part 1 covered — ndots, NodeLocal, ENI cap — are load-bearing knowledge for anyone running Kubernetes DNS on AWS. They're in every good post about the topic.

But if your cluster runs a service mesh with a DNS interception feature, that mesh becomes a fourth layer in the chain, and it can produce the exact same client-visible symptom as the other three. Its metrics live in a different namespace, its logs are formatted differently, and the pod itself has no idea it's being intercepted.

The lesson isn't "Istio ambient DNS capture is bad." It's not — it exists for good reasons around ServiceEntry resolution and mesh-consistent behavior. The lesson is that the DNS path from your pod to the upstream resolver may not be the path you think it is, and the way to find out is to enumerate every DaemonSet-scoped component in the request path, not just the ones in Part 1's diagram.

If I was writing the diagnostic framework from scratch today, I'd add a Layer 0 to the list before you look at ndots:

Before assuming your pod talks directly to 169.254.20.10, confirm that assumption.

That's it. That's the whole finding. Everything else — the metrics you inspected, the packet-count math, the ethtool counters — is downstream of that one question.

What silent drops teach you

Every layer in the DNS chain that I care about has the same underlying failure mode: it drops a packet without telling the client. The pod's UDP socket has no way to distinguish "the packet reached the resolver and got no reply" from "the packet was dropped one hop away." Both surface as i/o timeout.

That means diagnostic technique matters more than any single fix. When a timeout can come from four different layers and none of them log an error, you have to work by process of elimination:

  1. Prove Layer 1 (ndots) by counting search-domain queries. Reduce and re-measure.
  2. Prove Layer 2 (NodeLocal) via its own /metrics. Watch for spikes in forward_request_duration_seconds at the 99th percentile.
  3. Prove Layer 3 (ENI cap) via linklocal_allowance_exceeded. Nonzero = you hit it, zero = you didn't.
  4. Prove Layer 0 (mesh interception) exists at all. Check for ztunnel on your nodes and the ambient label on your namespace before any of the above.

For a good six weeks I skipped step 4 because it wasn't in my mental model. Adding it saved me the next six.

Part 1 covered the three DNS layers everyone talks about. This one is the one nobody mentioned, because for anyone not running Istio ambient it doesn't exist. If you are running it, this annotation may be the one-line change you didn't know you needed. If you're not, the meta-lesson still applies: when a chain of components each fails silently, the answer isn't more metrics — it's an accurate diagram.

Top comments (0)