Context
Home labs rarely look like the tidy diagrams in a vendor whitepaper. Real setups grow organically, and this one is no exception: a small home Kubernetes deployment nicknamed after its domain, kim.tec.br, running across two separate physical notebooks rather than a single host, a control plane named controlplane 82md running Linux Mint at 10.42.10.11 and a worker named node01 82nm running Manjaro at 10.42.10.12. The Manjaro notebook, node01, reaches the rest of the network purely through its wireless interface, wlp0s20f3, and that is the interface referenced throughout the ARP troubleshooting later in this article. There is no wired uplink anywhere in this setup, so every packet that leaves or enters either node passes through a WiFi radio first.
The goal was simple in appearance: expose a handful of internal services behind LoadBalancer virtual IP addresses, resolve friendly hostnames like internal.kim.tec.br through an internal DNS server, and reach those services securely from a phone using Tailscale, whether on the same home WiFi or out in the world on mobile data. Everything worked from inside the local network, but the moment the phone left the house and relied purely on mobile data, every connection attempt timed out. The container networking layer is Cilium version 1.18.0, running with the kube proxy replacement enabled and implementing almost all of its packet handling through eBPF programs attached directly to network interfaces, rather than the kernel's traditional netfilter rules, a distinction that matters a great deal later in the story. On top of Cilium sits MetalLB, which owns an IPAddressPool covering 10.42.30.0/24 and hands out the internal gateway address 10.42.30.2, answering for every hostname under the wildcard *.internal.kim.tec.br.
Finally, a Tailscale Connector pod runs inside the cluster in the vpn namespace. Unlike a typical VPN gateway, this Connector runs as an ordinary pod rather than with host networking, and it acts as a subnet router, advertising two ranges into the Tailscale mesh network: the home local area network 10.42.0.0/16, and the LoadBalancer virtual IP range 10.42.30.0/24. The Connector also has snatSubnetRoutes enabled, meaning traffic it forwards on behalf of remote Tailscale peers is rewritten to appear as if it originated from the Connector pod's own address before it continues onward. What follows is the full investigation behind the failure above: two separate and unrelated root causes stacked on top of each other, one at the physical network layer involving how WiFi access points handle address resolution, and one deep inside the Cilium eBPF data path involving how tunnel devices interact with network address translation for services. By the end, you will understand not just the fix, but the reasoning that makes it obvious in hindsight, and you will have a set of diagnostic commands you can reuse whenever a virtual IP address becomes mysteriously unreachable from part of a network but not from another part.
Figure 1: Two different entry points into Cilium for the same destination service.
This diagram captures the two very different journeys a packet can take to reach the same service. A device already on the home network reaches the access point directly and Cilium picks the packet up on its physical interface hook. A phone connected through Tailscale takes a much longer road: WireGuard encryption, the public internet, the home router, the WiFi radio, and finally the Connector pod inside the cluster, which then has to hand the packet to Cilium from the inside of a pod rather than from the outside of the cluster. That difference in entry point turns out to be the seed of the entire problem.
Problem
After configuring Tailscale's split DNS feature to forward internal.kim.tec.br queries to the internal DNS server, name resolution worked everywhere. Every device correctly resolved anything.internal.kim.tec.br to 10.42.30.2. The trouble began once actual connections were attempted.
Opening https://anything.internal.kim.tec.br worked without any issue from any device already on the home WiFi network, whether or not Tailscale happened to be installed on it. The TLS handshake completed instantly and the page loaded, which confirmed the gateway itself, its certificates, and the backend pods were all healthy. The failure appeared only on a phone connected through Tailscale while relying purely on mobile data, with WiFi turned off: the browser sat waiting and eventually reported ERR_CONNECTION_TIMED_OUT. Reconnecting that same phone to the home WiFi network directly, even with Tailscale still active, made the connection succeed again. Only the combination of Tailscale plus a remote network, meaning traffic that had to be forwarded by the Connector pod across the mesh rather than entering through the physical WiFi interface, produced the timeout.
This immediately ruled out several suspects. DNS was clearly not at fault, since resolution succeeded in every case. The gateway, its TLS certificates, and the backend application pods were also clearly fine, since local access worked perfectly. What remained was something specific to how traffic reached the cluster when it arrived through the Tailscale Connector rather than through the physical WiFi interface directly, and it turned out there were two distinct issues layered on top of each other.
Problem 1: Address Resolution Protocol Filtering on WiFi
Every device on a local network needs a way to learn which physical hardware address corresponds to a given IP address before it can deliver a frame at layer two. This is the job of the Address Resolution Protocol, commonly called ARP. When MetalLB operates in its layer two mode, it makes a virtual IP address reachable by sending what is called a gratuitous ARP, an unsolicited broadcast that essentially announces to the entire network segment: this IP address now lives at this hardware address, please update your records accordingly.
On a traditional wired switch this works without complaint, because switches are largely indifferent to which IP address claims to live behind a given port. WiFi access points can behave very differently, although it is worth being precise here: this is not a universal property of the 802.11 standard itself, it is a feature some access point firmware implements on top of it, often under names like proxy ARP, client isolation, or anti spoofing protection, and it varies from vendor to vendor and even from firmware release to firmware release. In this particular home network, the access point built into the TP-Link router actively tracked which IP address each connected client registered through DHCP, and it filtered or dropped ARP traffic that did not match that registration. The node's WiFi hardware address was registered for 10.42.10.12, the address it received over DHCP. The moment MetalLB tried to claim 10.42.30.2 using that same hardware address, the access point saw a mismatch and quietly discarded the frame. Anyone reproducing this on different hardware should confirm the behavior of their own access point rather than assume every WiFi router filters ARP the same way.
It is also worth noting that MetalLB does not rely solely on unsolicited gratuitous ARP. One speaker pod is elected leader for each virtual IP through a memberlist based election process, and that leader both answers ordinary ARP requests for the address and sends gratuitous ARP proactively whenever leadership changes, so that neighboring devices refresh their cache quickly after a failover. The gratuitous broadcast described above is the part that a filtering access point tends to reject, but MetalLB's normal request and reply behavior can suffer from the same filtering if the access point's registration table simply has no entry for the virtual IP at all.
Figure 2: Two independent ways the same address resolution fails, both traced back to the access point's DHCP registration table.
Two concrete symptoms confirmed this diagnosis. First, running arping 10.42.30.2 from another machine on the same physical network timed out completely, with no reply of any kind. Second, inspecting the node's own ARP table showed the virtual IP marked as incomplete, meaning the node itself had broadcast the address but never received confirmation that anyone accepted the claim. Capturing traffic directly on the wireless interface made the drop visible in real time.
tcpdump -eni wlp0s20f3 arp and host 10.42.30.2
# node's gratuitous ARP broadcast appears once
# no corresponding reply or forwarded copy ever appears on the wire
The Fix: Replace MetalLB Layer Two Announcement With Cilium Layer Two Announcements
Rather than relying on MetalLB's gratuitous ARP broadcasts, which WiFi access points tend to filter, Cilium offers its own layer two announcement feature that behaves differently. Instead of broadcasting unsolicited claims, it responds reactively whenever another device asks who owns a given address, the same way any ordinary host on the network would answer an ARP request for its own address. Because this is a genuine reply to a genuine request rather than an unsolicited broadcast, access points forward it without suspicion.
Figure 4: A reactive reply passes through the same access point that blocked the earlier unsolicited broadcast in Figure 2.
The steps involved were: removing the existing MetalLB L2Advertisement resource while keeping the IPAddressPool intact for address allocation, enabling l2announcements and externalIPs in the Cilium Helm values, applying a CiliumL2AnnouncementPolicy scoped to the wlp0s20f3 interface, and restarting the Cilium daemon set so the new configuration actually took effect, since a Helm upgrade updates the configuration map but does not itself trigger a pod restart.
kubectl delete l2advertisement -n metallb-system <name>
helm upgrade cilium cilium/cilium \
-n kube-system \
--reuse-values \
--set l2announcements.enabled=true \
--set externalIPs.enabled=true
kubectl apply -f l2-policy.yaml
kubectl rollout restart daemonset -n kube-system cilium
kubectl rollout status daemonset -n kube-system cilium --timeout=180s
Verification came from running arping again and finally receiving a unicast reply from the node's hardware address, something that had never happened before this change. It is also worth confirming which pod is currently the elected leader for a given virtual IP, since only that pod answers and refreshes the address on this network.
arping -I wlp0s20f3 10.42.30.2
kubectl get ciliuml2announcementpolicy
kubectl -n kube-system get lease | grep l2announce
cilium-dbg config --all | grep EnableL2Announcements
Problem 2: Cilium Does Not Rewrite LoadBalancer Addresses for Every Path
Once the first problem was solved, a second and far subtler issue appeared, this time only affecting traffic that arrived through the Tailscale Connector's tunnel device rather than through an ordinary socket connection. To understand it, it helps to understand how the Tailscale Connector actually delivers packets into the pod network.
The Tailscale daemon running inside the Connector pod receives encrypted WireGuard packets from remote peers, decrypts them, and then writes the resulting raw IP packet directly into a virtual network interface known as a tun device. A tun device operates at layer three. It does not involve a program calling connect() on a socket the way a typical application does. Instead, the Tailscale daemon hands the kernel a fully formed packet, header and all, and the kernel simply routes it as if it had arrived from anywhere else.
This distinction matters because Cilium implements its service address translation in two separate places. One mechanism, commonly called the socket level load balancer, attaches to cgroup hooks for the connect4 and connect6 system calls, intercepting the moment a process asks the kernel to connect somewhere and rewriting the destination before the packet is ever built. This is why a command like curl running inside a pod and connecting to a virtual IP works flawlessly, the rewrite happens before the first byte leaves the process. The other mechanism attaches to the pod's virtual ethernet interface itself, on the host side of the veth pair, and inspects packets as they arrive from the container regardless of how they were created.
By default, this second mechanism only rewrites addresses belonging to internal cluster IP services for pod originated traffic, and does not automatically extend the same rewrite to LoadBalancer external addresses, on the assumption that those addresses are only ever approached from outside the cluster through the physical network interface, never from inside a pod.
Tailscale's tun device breaks the assumption behind the default behavior completely, at least when the Connector runs in its ordinary kernel tun mode rather than in an alternative userspace networking mode sometimes called netstack, where Tailscale implements its own network stack entirely in userspace without ever touching a kernel tun device at all. In kernel tun mode, which is what this cluster used, the packet the daemon hands to the kernel already carries the LoadBalancer virtual IP as its destination, was never built through a connect() call, and therefore never benefits from the socket level rewrite mechanism. It reaches the pod's virtual ethernet hook still carrying the untouched virtual IP as its destination, and because that hook does not rewrite LoadBalancer addresses by default, the packet leaves the pod exactly as it arrived, destined for an address the kernel has no route for except back out over the WiFi interface, where it promptly fails to resolve, the very same address resolution problem from Problem 1, except now happening from inside the node rather than from an external client.
Figure 3: The gap between the socket level rewrite and the pod egress rewrite, and where a tun originated packet falls through it.
Cilium's connection tracking table turns this theory into direct evidence. While sending traffic from the phone through Tailscale, the connection tracking entry for that flow showed a backend identifier of zero, meaning no rewrite ever took place, while an entry generated by an ordinary local area network client hitting the same virtual IP showed a proper nonzero backend identifier, the same rewrite already confirmed above for the curl case. Same destination address, same backend service, two different outcomes depending purely on how the packet entered the system.
Inspecting Cilium's BPF maps directly is a reliable way to confirm which code path actually handled a given flow, rather than reasoning about it from behavior alone. The service definitions themselves live in a map commonly exposed through cilium-dbg service list, the backend selection lives in a companion backends map, and the reverse translation needed to rewrite return traffic lives in a reverse network address translation map, while the actual per flow state that decides whether a rewrite already happened is recorded in the connection tracking map queried below.
cilium-dbg service list | grep 10.42.30.2
cilium-dbg bpf lb list
cilium-dbg bpf ct list global | grep "172.20.5.22"
The Fix: Enable LoadBalancer Address Rewriting on the Pod Egress Hook
The second fix targets the Cilium configuration flag that governs whether the pod egress hook rewrites LoadBalancer addresses the same way it already rewrites internal cluster addresses. Setting bpf-lb-external-clusterip to true tells Cilium to add LoadBalancer external addresses into the same internal service table used for cluster addresses, so the egress hook rewrites them for any pod originated packet, whether that packet came from a socket connection or from a tun device like the one Tailscale uses.
Figure 5: The same tun originated packet from Figure 3, now completing the full round trip once the egress hook gains the missing rewrite.
The immediate, no downtime way to apply this was patching the configuration map directly and restarting Cilium, followed by making the change permanent through Helm values so it survives future upgrades.
kubectl patch configmap -n kube-system cilium-config \
--type merge \
-p '{"data":{"bpf-lb-external-clusterip":"true"}}'
kubectl rollout restart ds/cilium -n kube-system
kubectl rollout status ds/cilium -n kube-system --timeout=180s
bpf:
lbExternalClusterIP: true
helm upgrade cilium cilium/cilium \
-n kube-system \
--reuse-values \
--set bpf.lbExternalClusterIP=true
Alternatives That Were Considered
Several other approaches were evaluated before settling on the combination above. Each addressed at most one of the two problems rather than both.
| Approach | Why it was rejected |
|---|---|
| Move the cluster node onto a wired Ethernet uplink | Fixes Problem 1 only; switches do not filter ARP, but the pod egress rewrite gap in Problem 2 has nothing to do with the physical medium |
| Add a static route on the home router for the virtual IP range | Helps ordinary LAN devices bypass the ARP filtering, but does not help mobile Tailscale traffic, which returns through the Connector pod regardless |
| A privileged DaemonSet injecting address translation rules into the Connector pod's network namespace | Worked, but the rules vanished on every pod restart and it needed privileged access the native Cilium flag makes unnecessary |
| Run the Tailscale subnet router on the Proxmox host instead of as a pod | Would sidestep Problem 2 entirely, since host traffic enters through the physical interface hook, but moves Tailscale outside the cluster's declarative configuration |
Validation
With both fixes in place, the full packet journey shown in Figure 1 was confirmed end to end, from the phone's WireGuard connection all the way to the backend pod and back.
Each layer of that journey has a corresponding command that turns the claim into evidence rather than assumption.
# confirm the access point now forwards the reply
arping -I wlp0s20f3 10.42.30.2
# confirm the virtual IP is present in the service map with a real backend
cilium-dbg service list | grep 10.42.30.2
# confirm the flow from the connector pod now shows a nonzero backend id
cilium-dbg bpf ct list global | grep "172.20.5.22"
# confirm the flag itself is active on every node
cilium-dbg config --all | grep ExternalClusterIP
The final configuration state compared against the starting point looked like this.
| Component | Before | After |
|---|---|---|
| MetalLB L2Advertisement | Present | Removed |
| MetalLB IPAddressPool | Present | Kept for address allocation only |
| CiliumL2AnnouncementPolicy | Absent | Added, scoped to wlp0s20f3 |
| bpf lb external clusterip flag | False, the default | True |
| Hubble observability | Enabled | Disabled as a temporary workaround pending upgrade |
| Local network access to virtual IPs | Already working | Still working |
| Tailscale mobile access to virtual IPs | Broken | Fixed |
Take Aways
The most useful lesson from this whole investigation is not really about MetalLB or Cilium specifically. It is that a single symptom, a connection timing out from one particular network path, can hide two entirely unrelated causes stacked on top of each other, one at the physical layer and one deep inside the container networking data path. Fixing only one of them would have looked like progress, since local network access already worked before either change, but it would never have solved the actual complaint. Whenever a virtual IP address behaves differently depending on where the client sits, it is worth asking two separate questions rather than one: can this network segment even resolve the address at layer two, and does the packet path the client is using actually pass through the rewrite mechanism that makes the address usable at all. Answering both questions independently, with real diagnostic commands rather than assumptions, is what turned this confusion into a permanent fix.
The other lesson sits one level below the application entirely. Swapping the physical interface a node relies on, wired Ethernet against WiFi, is not a cosmetic change confined to cabling. It reaches into how the lowest layers of the OSI model behave: how address resolution is broadcast and filtered, whether an access point trusts an unsolicited claim or only a reply to its own request, and ultimately whether a virtual IP address can even be announced at all. A design that has only ever been tested on wired switches can carry hidden assumptions about that layer, assumptions that stay invisible until the same setup runs over WiFi and something that used to just work quietly stops working.
References
Cilium official documentation on layer two announcements, available under the networking section covering load balancer IP address management.
Cilium official documentation on Kubernetes without kube-proxy, covering the bpf.lbExternalClusterIP flag and why external access to ClusterIP and LoadBalancer addresses is disabled by default.
MetalLB official documentation on layer two mode and its gratuitous ARP based announcement mechanism.
Tailscale official documentation on subnet routers and the Connector deployment pattern for Kubernetes.
The Linux kernel documentation on tun and tap virtual network devices, describing how raw packets are injected at layer three without a socket connect call.
Top comments (0)