DEV Community: Kim

A DevOps Perspective on Turning AI Skepticism Into the Right Kind of Productivity

Kim — Tue, 28 Jul 2026 17:42:41 +0000

Introduction

A few years ago I would not have believed that an assistant reading from a chat window could become a real working partner for someone in infrastructure and platform engineering. My first contact with these tools left me unimpressed. They invented configuration properties that simply did not exist, they answered as if every tool only ever had one version, and they could not hold enough context to be trusted with anything beyond a quick question. By 2026 that picture changed completely for me. I now work every day alongside an assistant that reads full repositories, recognizes the patterns I already use, and produces work that feels like it came from an actual member of the team.

This article is a personal account of that journey, told from the perspective of someone working in DevOps, infrastructure, and platform engineering. I am not going to claim to know the best way to build enterprise applications with AI, since that is not the kind of work I focus on. What I do know, in detail, is how I reshaped my own way of working to get consistent, high quality outcomes without losing time to bad advice. That includes the mistakes that cost me real hours, the guardrails I built afterward, and the toolkit of skills and instructions that now sits between me and every task I touch. By the end of this article you will understand how that toolkit evolved, why I still refuse to trust a single answer at face value, and where I plan to take this setup next.

From Skepticism to a Daily Working Buddy

The earliest experiments felt closer to a novelty than a tool. Asking a chat based assistant for help with a configuration file often produced a property that sounded correct but did not exist in the actual software, or an answer clearly written for the wrong release of a service. Those failures were not subtle. They were the kind of mistake that anyone with a bit of experience would catch immediately, which made it hard to imagine handing over anything more serious than a quick lookup.

What changed by 2026 was not a single breakthrough but a combination of better underlying models and a much more deliberate way of working on my side. The assistant became able to read an entire repository, recognize the conventions already in place, and produce output that matched them instead of imposing a generic style. That shift let me start treating it as a working buddy rather than a search engine with a chat interface.

Figure 1: How trust in daily AI use moved from early skepticism, through a cautious experimentation phase, through a moment of overconfidence that led to a costly mistake and a real dip in trust, and finally to a working buddy relationship built on structure and verification.

There is something bittersweet in this change that I want to be honest about. Developing used to feel like a craft, something closer to an art than a mechanical process. Now it looks a lot more like what most companies always wanted it to be, a reliable means to an end. I am not criticizing that shift, and I am not complaining about it either. It is a natural evolution, and I made peace with it. What matters more to me now is that the assistant works in parallel with me. I can hold more than one issue or task active at the same time, which means I can chip away at technical debt without ever losing focus on my main assignment, and I can keep a running memory of my troubleshooting sessions that turns into precise postmortem reports and improvement notes almost for free.

Working this way, in parallel and with far less manual reconstruction, sits at odds with a belief that gets repeated across the industry: that shipping more code, faster, is what productivity actually looks like. That gap between the belief and what I actually experience day to day is worth pulling apart properly.

Defining What Productivity Means

Productivity tends to get measured badly across software work in general, not only in infrastructure and platform engineering. Counting lines of code produced, pull requests opened, or resources created through an assistant are faulty measures wherever they get applied, since none of them say much about whether the underlying work was actually good, for a developer or for someone in my position. For me, productivity means something else entirely, and it has two parts. The first is reaching a state where every delivery holds real quality without sacrificing the time it takes to get there. The second is that a meaningful share of what I deliver is not for myself at all. It is productivity for other people, achieved through platform work and automation that removes friction from their day long before they would ever notice it was there.

No more technical debts left behind

Before I built a structured way of working, a technical debt item like cleaning up an old script or hardening a manifest would sit untouched for weeks because it always competed with whatever incident or deployment was the priority of the day. Now I can hand that kind of task to the assistant as a side task, review its result between the higher priority items, and merge it once I am satisfied. The debt gets paid down without ever displacing the main task from my attention.

Automated postmortems with instant memory

When an incident happens, I used to spend real effort afterward reconstructing the timeline for a postmortem: what was tried, in what order, and why. Now the assistant keeps that memory as we investigate together, so producing a precise postmortem or an improvement report afterward takes minutes instead of an afternoon of reconstructing my own notes from memory.

Platform automation that multiplies other people's time

Some of what I build never touches my own task list at all. A standardized Grafana dashboard template, or an automated onboarding step for a new host joining monitoring, saves time for whoever uses it next, not for me. That compounding effect, work that quietly removes friction for other engineers long after I finish it, is as real a productivity gain as anything I deliver on my own tasks, even though no pull request count or delivery metric would ever capture it.

None of that quality came automatically, though. Trusting the assistant enough to let it carry technical debt, write a postmortem, or automate something other people rely on only became possible after a few investigations went badly wrong first.

The Wild Goose Chase, and What It Taught Me

The clearest of those bad investigations came from trusting an assistant's evaluation of a problem a little too much. On one occasion I accepted a diagnosis at face value instead of checking the primary source myself, and what should have been an hour of investigation stretched into two full days chasing a conclusion that was simply wrong. That kind of experience is uncomfortable, but it was also the moment that pushed me to build real discipline around how I use these tools rather than treating every answer as final.

Figure 2: The two paths an investigation can take. Accepting a finding without verification risks turning a short problem into a multi day detour, while checking primary sources and asking a second model to criticize the finding keeps the investigation on a short, predictable path.

Two days spent chasing a one hour problem

The two day investigation I mentioned above is the clearest example. The assistant proposed a plausible sounding root cause early on, and because it sounded reasonable and matched a story I already half expected, I moved straight to trying fixes based on it instead of confirming it against logs and configuration first. Two days later, after the fix repeatedly failed to hold, going back to the primary source revealed the real cause in under an hour.

A second opinion catches the gap before it costs time

Since then I set up a habit, and eventually a standing instruction, that turns any non trivial investigation into a three way job instead of a conversation with a single assistant. I validate the primary source of information myself, in parallel with a first model working through the same investigation on its own. Once both of those are done, I bring in a second, different model whose only task is to criticize the first model's output and cross check it against my own observations so far. In a later, similar situation, that second model, prompted specifically to look for holes in the reasoning, flagged that the first model's evidence was incomplete before I ever acted on it. That single check redirected the investigation before any time was lost, and it is now a standard part of how I validate anything non trivial.

That change, splitting the work into my own verification, a first model's independent investigation, and a second model's critique of both, improved my success rate enough that I no longer dread being sent on a wild goose chase by an overconfident answer. I still remain the one accountable for the outcome, the assistant simply gives me a faster way to get there when I use it with the right amount of skepticism.

That single habit did not stay a personal quirk for long. It became the seed for a much larger set of rules I now rely on with every task.

Building a Layered Instruction System

Once I understood how much the quality of results depended on how I set things up rather than on the model alone, I built a layered structure of instructions that now underlies everything I do.

At the top sits a set of global instructions that apply across every project I touch. These capture rules I never want to repeat by hand, such as always using a different model to criticize a piece of work before accepting it, or, whenever I ask for a new skill to be created, using one model to draft the prompt and a different model to actually author and execute it. Below that sit project specific instructions, and below those, repository specific instructions for the parts of a project that have their own conventions. One rule sits above all of them as a kind of meta rule: whenever I use a strong statement like always do this or do this always in conversation, that statement itself needs to be captured back into the instructions so it is never lost to a single session.

Figure 3: The instruction hierarchy that shapes every task. Global rules apply everywhere, project rules narrow that down, repository rules handle local specifics, and any strong directive I state in conversation gets written back into the global rules so it persists.

A global rule born from the wild goose chase

The global rule about always using a different model to criticize a finding is the one that came directly out of the wild goose chase experience described above. It now applies automatically to every project I work in, professional or personal, without me needing to remember to ask for it each time.

A narrow rule that stays local to one repository

A repository specific instruction might record something as narrow as which command validates changes in a particular codebase, or a safety check that must run before any mutating command against a specific cluster. These live only where they are relevant, so the global instructions stay focused on things that genuinely apply everywhere.

Instructions only set the rules, though. Where those rules actually turn into repeatable work is a smaller, more concrete layer built from skills and, to a lesser extent, agents.

The Daily Toolkit: Skills Over Agents

Most of the practical value I get day to day comes from a set of custom skills rather than from autonomous agents, and that distinction matters enough that I want to spell it out clearly.

On the professional side I have built skills to create standardized reports, to keep a running memory of notable events and mistakes, to investigate specific Kubernetes issues with a consistent method, to create Grafana dashboards that follow a fixed standard instead of one off layouts, and to develop small scripts, since I still believe not everything should be produced by an assistant. On the personal side I keep a smaller set: a planner for study topics and home server improvements, a diary that records changes to my own infrastructure, and an article writer that turns notes like the ones behind this piece into a finished, structured article.

Figure 4: The skill and agent ecosystem I currently run. Skills carry most of the daily workload on both the professional and personal side, while agents remain a smaller, more experimental layer on top.

A consistent method for every Kubernetes investigation

The Kubernetes investigation skill is a good example of a professional skill earning its place. It enforces the same method every time an issue comes up: gather the right context, confirm the working cluster, collect the mandatory evidence, and only then reason about root cause. That consistency is worth more to me than any single clever answer.

A searchable history of infrastructure changes

The infrastructure diary skill is the personal equivalent. Every completed change to my home network or home server gets recorded in the same narrative format, which means I can grep years of history later and actually find what I am looking for, instead of relying on memory alone.

Agents have had a smaller role so far. I keep exactly one on my personal setup, a memory keeper that supports the diary and report skills, and I run seven on my professional setup. Honestly, the professional agents have not yet shown a clear productivity improvement over well built skills with clear instructions. I keep experimenting with them, but for now the bulk of my real gains come from the more constrained, more predictable skill layer.

Looking ahead, I have a few concrete plans building on this foundation. I want an agent that automatically drafts troubleshooting reports the moment an incident starts, so that whoever is on call gets a solid starting point instead of a blank page, which should meaningfully cut down the time it takes to reach a fix. I am also planning a skill to generate documentation in one fixed format across onboarding guides, knowledge base entries, and simple help pages, so that anyone reading them knows exactly what to expect. Alongside that, I want a skill that scaffolds new workloads from a small set of predefined developer inputs, as a stopgap until proper platform automation exists to do that job on its own.

Conclusion

Looking back at where I started, deeply skeptical of a tool that could not get a configuration property right, the distance to where I am now is significant. The real turning point was never the model alone. It was building the right structure around it: a layered set of instructions that captures decisions once and applies them everywhere, a habit of checking primary sources myself and asking a second model to criticize any important finding, and a set of narrowly scoped skills that do one job well instead of a handful of agents that promise everything and deliver an unclear amount. I do not think I have reached the ideal setup yet, and I expect the instructions, skills, and even the balance between skills and agents to keep shifting as I learn more. What I do know is that I can now work alongside this assistant without dreading being sent on another wild goose chase, and that alone has changed how much I get done in a day without giving up any of the quality I care about.

Why Tailscale Couldn't Reach a Kubernetes LoadBalancer VIP

Kim — Mon, 20 Jul 2026 20:54:52 +0000

Context

Home labs rarely look like the tidy diagrams in a vendor whitepaper. Real setups grow organically, and this one is no exception: a small home Kubernetes deployment nicknamed after its domain, kim.tec.br, running across two separate physical notebooks rather than a single host, a control plane named controlplane 82md running Linux Mint at 10.42.10.11 and a worker named node01 82nm running Manjaro at 10.42.10.12. The Manjaro notebook, node01, reaches the rest of the network purely through its wireless interface, wlp0s20f3, and that is the interface referenced throughout the ARP troubleshooting later in this article. There is no wired uplink anywhere in this setup, so every packet that leaves or enters either node passes through a WiFi radio first.

The goal was simple in appearance: expose a handful of internal services behind LoadBalancer virtual IP addresses, resolve friendly hostnames like internal.kim.tec.br through an internal DNS server, and reach those services securely from a phone using Tailscale, whether on the same home WiFi or out in the world on mobile data. Everything worked from inside the local network, but the moment the phone left the house and relied purely on mobile data, every connection attempt timed out. The container networking layer is Cilium version 1.18.0, running with the kube proxy replacement enabled and implementing almost all of its packet handling through eBPF programs attached directly to network interfaces, rather than the kernel's traditional netfilter rules, a distinction that matters a great deal later in the story. On top of Cilium sits MetalLB, which owns an IPAddressPool covering 10.42.30.0/24 and hands out the internal gateway address 10.42.30.2, answering for every hostname under the wildcard *.internal.kim.tec.br.

Finally, a Tailscale Connector pod runs inside the cluster in the vpn namespace. Unlike a typical VPN gateway, this Connector runs as an ordinary pod rather than with host networking, and it acts as a subnet router, advertising two ranges into the Tailscale mesh network: the home local area network 10.42.0.0/16, and the LoadBalancer virtual IP range 10.42.30.0/24. The Connector also has snatSubnetRoutes enabled, meaning traffic it forwards on behalf of remote Tailscale peers is rewritten to appear as if it originated from the Connector pod's own address before it continues onward. What follows is the full investigation behind the failure above: two separate and unrelated root causes stacked on top of each other, one at the physical network layer involving how WiFi access points handle address resolution, and one deep inside the Cilium eBPF data path involving how tunnel devices interact with network address translation for services. By the end, you will understand not just the fix, but the reasoning that makes it obvious in hindsight, and you will have a set of diagnostic commands you can reuse whenever a virtual IP address becomes mysteriously unreachable from part of a network but not from another part.

Figure 1: Two different entry points into Cilium for the same destination service.

This diagram captures the two very different journeys a packet can take to reach the same service. A device already on the home network reaches the access point directly and Cilium picks the packet up on its physical interface hook. A phone connected through Tailscale takes a much longer road: WireGuard encryption, the public internet, the home router, the WiFi radio, and finally the Connector pod inside the cluster, which then has to hand the packet to Cilium from the inside of a pod rather than from the outside of the cluster. That difference in entry point turns out to be the seed of the entire problem.

Problem

After configuring Tailscale's split DNS feature to forward internal.kim.tec.br queries to the internal DNS server, name resolution worked everywhere. Every device correctly resolved anything.internal.kim.tec.br to 10.42.30.2. The trouble began once actual connections were attempted.

Opening https://anything.internal.kim.tec.br worked without any issue from any device already on the home WiFi network, whether or not Tailscale happened to be installed on it. The TLS handshake completed instantly and the page loaded, which confirmed the gateway itself, its certificates, and the backend pods were all healthy. The failure appeared only on a phone connected through Tailscale while relying purely on mobile data, with WiFi turned off: the browser sat waiting and eventually reported ERR_CONNECTION_TIMED_OUT. Reconnecting that same phone to the home WiFi network directly, even with Tailscale still active, made the connection succeed again. Only the combination of Tailscale plus a remote network, meaning traffic that had to be forwarded by the Connector pod across the mesh rather than entering through the physical WiFi interface, produced the timeout.

This immediately ruled out several suspects. DNS was clearly not at fault, since resolution succeeded in every case. The gateway, its TLS certificates, and the backend application pods were also clearly fine, since local access worked perfectly. What remained was something specific to how traffic reached the cluster when it arrived through the Tailscale Connector rather than through the physical WiFi interface directly, and it turned out there were two distinct issues layered on top of each other.

Problem 1: Address Resolution Protocol Filtering on WiFi

Every device on a local network needs a way to learn which physical hardware address corresponds to a given IP address before it can deliver a frame at layer two. This is the job of the Address Resolution Protocol, commonly called ARP. When MetalLB operates in its layer two mode, it makes a virtual IP address reachable by sending what is called a gratuitous ARP, an unsolicited broadcast that essentially announces to the entire network segment: this IP address now lives at this hardware address, please update your records accordingly.

On a traditional wired switch this works without complaint, because switches are largely indifferent to which IP address claims to live behind a given port. WiFi access points can behave very differently, although it is worth being precise here: this is not a universal property of the 802.11 standard itself, it is a feature some access point firmware implements on top of it, often under names like proxy ARP, client isolation, or anti spoofing protection, and it varies from vendor to vendor and even from firmware release to firmware release. In this particular home network, the access point built into the TP-Link router actively tracked which IP address each connected client registered through DHCP, and it filtered or dropped ARP traffic that did not match that registration. The node's WiFi hardware address was registered for 10.42.10.12, the address it received over DHCP. The moment MetalLB tried to claim 10.42.30.2 using that same hardware address, the access point saw a mismatch and quietly discarded the frame. Anyone reproducing this on different hardware should confirm the behavior of their own access point rather than assume every WiFi router filters ARP the same way.

It is also worth noting that MetalLB does not rely solely on unsolicited gratuitous ARP. One speaker pod is elected leader for each virtual IP through a memberlist based election process, and that leader both answers ordinary ARP requests for the address and sends gratuitous ARP proactively whenever leadership changes, so that neighboring devices refresh their cache quickly after a failover. The gratuitous broadcast described above is the part that a filtering access point tends to reject, but MetalLB's normal request and reply behavior can suffer from the same filtering if the access point's registration table simply has no entry for the virtual IP at all.

Figure 2: Two independent ways the same address resolution fails, both traced back to the access point's DHCP registration table.

Two concrete symptoms confirmed this diagnosis. First, running arping 10.42.30.2 from another machine on the same physical network timed out completely, with no reply of any kind. Second, inspecting the node's own ARP table showed the virtual IP marked as incomplete, meaning the node itself had broadcast the address but never received confirmation that anyone accepted the claim. Capturing traffic directly on the wireless interface made the drop visible in real time.

tcpdump -eni wlp0s20f3 arp and host 10.42.30.2
# node's gratuitous ARP broadcast appears once
# no corresponding reply or forwarded copy ever appears on the wire

The Fix: Replace MetalLB Layer Two Announcement With Cilium Layer Two Announcements

Rather than relying on MetalLB's gratuitous ARP broadcasts, which WiFi access points tend to filter, Cilium offers its own layer two announcement feature that behaves differently. Instead of broadcasting unsolicited claims, it responds reactively whenever another device asks who owns a given address, the same way any ordinary host on the network would answer an ARP request for its own address. Because this is a genuine reply to a genuine request rather than an unsolicited broadcast, access points forward it without suspicion.

Figure 4: A reactive reply passes through the same access point that blocked the earlier unsolicited broadcast in Figure 2.

The steps involved were: removing the existing MetalLB L2Advertisement resource while keeping the IPAddressPool intact for address allocation, enabling l2announcements and externalIPs in the Cilium Helm values, applying a CiliumL2AnnouncementPolicy scoped to the wlp0s20f3 interface, and restarting the Cilium daemon set so the new configuration actually took effect, since a Helm upgrade updates the configuration map but does not itself trigger a pod restart.

kubectl delete l2advertisement -n metallb-system <name>

helm upgrade cilium cilium/cilium \
  -n kube-system \
  --reuse-values \
  --set l2announcements.enabled=true \
  --set externalIPs.enabled=true

kubectl apply -f l2-policy.yaml

kubectl rollout restart daemonset -n kube-system cilium
kubectl rollout status daemonset -n kube-system cilium --timeout=180s

Verification came from running arping again and finally receiving a unicast reply from the node's hardware address, something that had never happened before this change. It is also worth confirming which pod is currently the elected leader for a given virtual IP, since only that pod answers and refreshes the address on this network.

arping -I wlp0s20f3 10.42.30.2

kubectl get ciliuml2announcementpolicy

kubectl -n kube-system get lease | grep l2announce

cilium-dbg config --all | grep EnableL2Announcements

Problem 2: Cilium Does Not Rewrite LoadBalancer Addresses for Every Path

Once the first problem was solved, a second and far subtler issue appeared, this time only affecting traffic that arrived through the Tailscale Connector's tunnel device rather than through an ordinary socket connection. To understand it, it helps to understand how the Tailscale Connector actually delivers packets into the pod network.

The Tailscale daemon running inside the Connector pod receives encrypted WireGuard packets from remote peers, decrypts them, and then writes the resulting raw IP packet directly into a virtual network interface known as a tun device. A tun device operates at layer three. It does not involve a program calling connect() on a socket the way a typical application does. Instead, the Tailscale daemon hands the kernel a fully formed packet, header and all, and the kernel simply routes it as if it had arrived from anywhere else.

This distinction matters because Cilium implements its service address translation in two separate places. One mechanism, commonly called the socket level load balancer, attaches to cgroup hooks for the connect4 and connect6 system calls, intercepting the moment a process asks the kernel to connect somewhere and rewriting the destination before the packet is ever built. This is why a command like curl running inside a pod and connecting to a virtual IP works flawlessly, the rewrite happens before the first byte leaves the process. The other mechanism attaches to the pod's virtual ethernet interface itself, on the host side of the veth pair, and inspects packets as they arrive from the container regardless of how they were created.

By default, this second mechanism only rewrites addresses belonging to internal cluster IP services for pod originated traffic, and does not automatically extend the same rewrite to LoadBalancer external addresses, on the assumption that those addresses are only ever approached from outside the cluster through the physical network interface, never from inside a pod.

Tailscale's tun device breaks the assumption behind the default behavior completely, at least when the Connector runs in its ordinary kernel tun mode rather than in an alternative userspace networking mode sometimes called netstack, where Tailscale implements its own network stack entirely in userspace without ever touching a kernel tun device at all. In kernel tun mode, which is what this cluster used, the packet the daemon hands to the kernel already carries the LoadBalancer virtual IP as its destination, was never built through a connect() call, and therefore never benefits from the socket level rewrite mechanism. It reaches the pod's virtual ethernet hook still carrying the untouched virtual IP as its destination, and because that hook does not rewrite LoadBalancer addresses by default, the packet leaves the pod exactly as it arrived, destined for an address the kernel has no route for except back out over the WiFi interface, where it promptly fails to resolve, the very same address resolution problem from Problem 1, except now happening from inside the node rather than from an external client.

Figure 3: The gap between the socket level rewrite and the pod egress rewrite, and where a tun originated packet falls through it.

Cilium's connection tracking table turns this theory into direct evidence. While sending traffic from the phone through Tailscale, the connection tracking entry for that flow showed a backend identifier of zero, meaning no rewrite ever took place, while an entry generated by an ordinary local area network client hitting the same virtual IP showed a proper nonzero backend identifier, the same rewrite already confirmed above for the curl case. Same destination address, same backend service, two different outcomes depending purely on how the packet entered the system.

Inspecting Cilium's BPF maps directly is a reliable way to confirm which code path actually handled a given flow, rather than reasoning about it from behavior alone. The service definitions themselves live in a map commonly exposed through cilium-dbg service list, the backend selection lives in a companion backends map, and the reverse translation needed to rewrite return traffic lives in a reverse network address translation map, while the actual per flow state that decides whether a rewrite already happened is recorded in the connection tracking map queried below.

cilium-dbg service list | grep 10.42.30.2
cilium-dbg bpf lb list
cilium-dbg bpf ct list global | grep "172.20.5.22"

The Fix: Enable LoadBalancer Address Rewriting on the Pod Egress Hook

The second fix targets the Cilium configuration flag that governs whether the pod egress hook rewrites LoadBalancer addresses the same way it already rewrites internal cluster addresses. Setting bpf-lb-external-clusterip to true tells Cilium to add LoadBalancer external addresses into the same internal service table used for cluster addresses, so the egress hook rewrites them for any pod originated packet, whether that packet came from a socket connection or from a tun device like the one Tailscale uses.

Figure 5: The same tun originated packet from Figure 3, now completing the full round trip once the egress hook gains the missing rewrite.

The immediate, no downtime way to apply this was patching the configuration map directly and restarting Cilium, followed by making the change permanent through Helm values so it survives future upgrades.

kubectl patch configmap -n kube-system cilium-config \
  --type merge \
  -p '{"data":{"bpf-lb-external-clusterip":"true"}}'

kubectl rollout restart ds/cilium -n kube-system
kubectl rollout status ds/cilium -n kube-system --timeout=180s

bpf:
  lbExternalClusterIP: true

helm upgrade cilium cilium/cilium \
  -n kube-system \
  --reuse-values \
  --set bpf.lbExternalClusterIP=true

Alternatives That Were Considered

Several other approaches were evaluated before settling on the combination above. Each addressed at most one of the two problems rather than both.

Approach	Why it was rejected
Move the cluster node onto a wired Ethernet uplink	Fixes Problem 1 only; switches do not filter ARP, but the pod egress rewrite gap in Problem 2 has nothing to do with the physical medium
Add a static route on the home router for the virtual IP range	Helps ordinary LAN devices bypass the ARP filtering, but does not help mobile Tailscale traffic, which returns through the Connector pod regardless
A privileged DaemonSet injecting address translation rules into the Connector pod's network namespace	Worked, but the rules vanished on every pod restart and it needed privileged access the native Cilium flag makes unnecessary
Run the Tailscale subnet router on the Proxmox host instead of as a pod	Would sidestep Problem 2 entirely, since host traffic enters through the physical interface hook, but moves Tailscale outside the cluster's declarative configuration

Validation

With both fixes in place, the full packet journey shown in Figure 1 was confirmed end to end, from the phone's WireGuard connection all the way to the backend pod and back.

Each layer of that journey has a corresponding command that turns the claim into evidence rather than assumption.

# confirm the access point now forwards the reply
arping -I wlp0s20f3 10.42.30.2

# confirm the virtual IP is present in the service map with a real backend
cilium-dbg service list | grep 10.42.30.2

# confirm the flow from the connector pod now shows a nonzero backend id
cilium-dbg bpf ct list global | grep "172.20.5.22"

# confirm the flag itself is active on every node
cilium-dbg config --all | grep ExternalClusterIP

The final configuration state compared against the starting point looked like this.

Component	Before	After
MetalLB L2Advertisement	Present	Removed
MetalLB IPAddressPool	Present	Kept for address allocation only
CiliumL2AnnouncementPolicy	Absent	Added, scoped to wlp0s20f3
bpf lb external clusterip flag	False, the default	True
Hubble observability	Enabled	Disabled as a temporary workaround pending upgrade
Local network access to virtual IPs	Already working	Still working
Tailscale mobile access to virtual IPs	Broken	Fixed

Take Aways

The most useful lesson from this whole investigation is not really about MetalLB or Cilium specifically. It is that a single symptom, a connection timing out from one particular network path, can hide two entirely unrelated causes stacked on top of each other, one at the physical layer and one deep inside the container networking data path. Fixing only one of them would have looked like progress, since local network access already worked before either change, but it would never have solved the actual complaint. Whenever a virtual IP address behaves differently depending on where the client sits, it is worth asking two separate questions rather than one: can this network segment even resolve the address at layer two, and does the packet path the client is using actually pass through the rewrite mechanism that makes the address usable at all. Answering both questions independently, with real diagnostic commands rather than assumptions, is what turned this confusion into a permanent fix.

The other lesson sits one level below the application entirely. Swapping the physical interface a node relies on, wired Ethernet against WiFi, is not a cosmetic change confined to cabling. It reaches into how the lowest layers of the OSI model behave: how address resolution is broadcast and filtered, whether an access point trusts an unsolicited claim or only a reply to its own request, and ultimately whether a virtual IP address can even be announced at all. A design that has only ever been tested on wired switches can carry hidden assumptions about that layer, assumptions that stay invisible until the same setup runs over WiFi and something that used to just work quietly stops working.

References

Cilium official documentation on layer two announcements, available under the networking section covering load balancer IP address management.

Cilium official documentation on Kubernetes without kube-proxy, covering the bpf.lbExternalClusterIP flag and why external access to ClusterIP and LoadBalancer addresses is disabled by default.

MetalLB official documentation on layer two mode and its gratuitous ARP based announcement mechanism.

Tailscale official documentation on subnet routers and the Connector deployment pattern for Kubernetes.

The Linux kernel documentation on tun and tap virtual network devices, describing how raw packets are injected at layer three without a socket connect call.