DEV Community

david
david

Posted on Originally published at woitzik.dev

The DNS Drift That Wasn't DHCP: How Proxmox Cloud-Init Silently Injects Search Domains

#kubernetes #dns #proxmox #debugging

Originally published at woitzik.dev

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Every DNS lookup from every pod in my k3s cluster was appending home.lan to the query. External lookups like registry-1.docker.io became registry-1.docker.io.home.lan, failed locally, recursed all the way out, and added ~341ms of latency to every image pull, every API call, every webhook — traffic that should have gone straight to Unbound for recursive resolution, not round-tripped through a phantom search domain. I assumed DHCP was the source, wrote a Terraform fix, and watched the plan show zero diff against live state.

The real source was Proxmox's cloud-init generator — three layers above the DHCP server.

The Symptom

Prometheus showed p99 DNS latency at 341ms for external queries. Local queries (services within the cluster) resolved in under 1ms. The gap was consistent across all three k3s nodes.

# PromQL showing the split
quantile_over_time(0.99, dns_request_duration_seconds{domain!~".*cluster.local"}[5m])
# → ~341ms

quantile_over_time(0.99, dns_request_duration_seconds{domain=~".*cluster.local"}[5m])
# → ~0.8ms
Enter fullscreen mode Exit fullscreen mode

The kubelet's DNS policy appends the node's search domain to every pod's /etc/resolv.conf. If the node has search home.lan, every pod inherits it. With the default ndots:5, any name with fewer than 5 dots gets the search domain appended first — meaning almost everything.

Wrong Hypothesis: DHCP

My first guess was the RouterOS DHCP server. The network was configured via Terraform, but I'd never explicitly set the domain field:

# Before fix — no domain attribute at all
resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
}
Enter fullscreen mode Exit fullscreen mode

Adding domain = "" should force Terraform to detect and correct any drift:

resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
  domain     = ""  # explicitly empty — should fix drift
}
Enter fullscreen mode Exit fullscreen mode

The plan showed zero changes. Live state already matched domain = "". DHCP was never the source.

Live Verification

Confirmed via the RouterOS REST API:

# Check all DHCP server config paths
curl -s "https://router.lan/rest/ip/dhcp-server" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dhcp-server/network" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dns" | jq '.domain'
# → null
Enter fullscreen mode Exit fullscreen mode

The RouterOS DHCP server had no domain/search config anywhere. domain = "" matched live state because the live state genuinely was empty. DHCP was never handing out home.lan.

The Real Source: Proxmox Cloud-Init

On the k3s nodes themselves:

networkctl status --json | jq '.DNS'
# → {"SearchDomains": ["home.lan"], "ConfigSource": "static"}
Enter fullscreen mode Exit fullscreen mode

ConfigSource: static — not DHCP. The search domain was baked into the node's netplan config, generated by Proxmox's cloud-init writer.

When you create a VM in Proxmox with Terraform and leave initialization.dns.domain unset, Proxmox falls back to the PVE node's own configured search domain:

# On the Proxmox host
pvesh get /nodes/pve-mgmt-01/dns
# → search: home.lan
Enter fullscreen mode Exit fullscreen mode

Proxmox's cloud-init generator reads the PVE node's DNS config and writes it into the VM's /etc/netplan/50-cloud-init.yaml. Every k3s VM got home.lan as its search domain — not from DHCP, but from the hypervisor itself.

Why This Is a Trap

The trap is the layering:

Proxmox PVE node DNS config
    ↓ (cloud-init fallback)
VM netplan search domain
    ↓ (kubelet DNS policy)
Pod /etc/resolv.conf search domain
    ↓ (ndots:5)
Every DNS lookup appends .home.lan
    ↓
~341ms latency on every external query
Enter fullscreen mode Exit fullscreen mode

And the debugging trap: terraform plan showed zero diff on the DHCP resource because DHCP wasn't the source. A Terraform-native investigation would have concluded "no drift detected" and moved on. The real fix required:

  1. Clearing the PVE node's default: pvesh set /nodes/pve-mgmt-01/dns -search ""
  2. Regenerating cloud-init on each k3s VM
  3. Declaring domain = "" in the Proxmox VM's initialization.dns block

Steps 1 and 2 are live changes to etcd-member VMs — no Terraform resource models this. Step 3 declares intent for future VMs but has no effect on already-provisioned ones (the dns block is in lifecycle.ignore_changes).

The Terraform Fix (Future VMs) 

# terraform/stacks/proxmox/vm.tf
initialization {
  dns {
    domain  = ""  # explicitly unset — prevents PVE node fallback
    servers = ["10.0.20.5"]
  }
}
Enter fullscreen mode Exit fullscreen mode

This ensures new VMs don't inherit the PVE node's search domain. Existing VMs need the manual cloud-init regen.

Lessons

1. DNS config has multiple sources. DHCP is the obvious one, but cloud-init, node-level config, and kubelet policy all contribute. Check all layers.

2. terraform plan zero-diff doesn't mean no drift. It means the specific resource you checked matches state. The drift might be in a different layer entirely.

3. networkctl status --json is your friend. The ConfigSource field tells you whether the search domain came from DHCP, static config, or another source. If it says "static", stop looking at DHCP.

4. Proxmox cloud-init is a hidden config source. If you leave initialization.dns.domain unset, Proxmox fills it from the PVE node. Always declare it explicitly. It's the same category of hidden-layer surprise as Proxmox's onboot attribute silently not round-tripping through Terraform — the provider abstraction has gaps that only show up once you go looking for drift that isn't there.

Further reading: DNS debugging keeps coming back to the same fundamentals — search domains, ndots, resolution order. Netzwerke: Verstehen, Einrichten, Administrieren* covers those fundamentals in more depth than this Proxmox-specific drift case gets into.

DNS latency from inherited search domains is invisible in most monitoring setups. You'll see high p99 latency on external queries but no obvious cause. If your pods have ndots:5 (the default) and your nodes have a search domain you didn't explicitly set, you're paying the tax on every lookup.

Top comments (0)