Nine hours into a live networking bug on my k3s cluster, Claude Code asked me a question with three options. The first was labeled (Recommended). It was to give up.
The task behind it was ordinary by this point: a fourteen-step plan to put PR preview environments behind an Istio Ambient mesh, run task by task through superpowers' subagent-driven-development — one implementer subagent per task, one reviewer subagent to check its work, me approving the boundary. Thirteen tasks had gone this way already, each with its own small fight (an ArgoCD self-heal loop fighting istiod's webhook config, a sync retry limit exhausted, a waypoint proxy quietly asking for ten times the CPU its namespace quota allowed) and each one resolved cleanly enough that I stopped expecting the fourteenth to be different. Task 14 was verification: open a real pull request, send it a request carrying x-pr-lane: 1, confirm it landed on the preview pod instead of the baseline one.
It never did. Every request, header or no header, went straight to baseline. No error, on either side of that route.
Ruling out the obvious
What followed was the kind of debugging that doesn't look like debugging from outside — it looks like a long list of things that turned out not to be the problem. istio-cni's iptables REDIRECT rule inside the pod's network namespace: confirmed catching outbound packets, nsenter and all. ztunnel's xDS config: correct, the waypoint binding was right there in the config dump at the Service level. The waypoint itself: reachable, routes programmed, ready — and in three hours of its own logs, holding zero connections. A namespace-level versus Service-level use-waypoint label swap: no change. A GitHub issue on Istio's tracker with the same symptom: different root cause, cross-namespace enrollment, didn't apply here. A host-side tcpdump that caught traffic arriving on ztunnel's own tunnel port looked, for a while, like proof the fault lived inside ztunnel's routing logic rather than upstream of it — a read that turned out to be wrong, but not obviously wrong, not at 2 a.m.
Somewhere in the middle of that list, at 03:08, Claude Code tried a fix, watched it fail, and said so plainly: the new packet capture didn't just fail to confirm the diagnosis, it contradicted it. The situation was worse than it had looked an hour earlier. It asked whether to keep going. I didn't answer for forty-four minutes — I was asleep — and when I did, I just wrote: continue.
The honest report
Fifty minutes after that, at 04:01, it stopped asking and started reporting. The message was direct about what had happened: it had gone through, in its words, nearly every known and documented possible cause, systematically, and ruled each one out. Time to say so honestly. Then the question, three options:
1. Wrap up here, mark waypoint L7 routing as a known limitation (Recommended)
2. File an issue with Istio upstream — unlikely to get an answer quickly
3. Keep digging — success rate is already falling, may just be unsolvable
I asked what, specifically, was blocking it. It answered plainly: the feature's actual selling point — routing a live PR preview by request header through the mesh — wasn't something it could deliver. Did I want it to close this out as a known limitation, or look at the other options first?
Nothing about that exchange was performative. Every hypothesis it had checked really had checked out negative or inconclusive; the elimination had been careful, not lazy. What made the "Recommended" default worth noticing wasn't that it was reached in bad faith — it's that a careful, honest elimination of every path you can currently see still isn't the same thing as having eliminated every path there is. Nine hours of good work had converged on the wrong conclusion anyway, for the ordinary reason that whatever had anchored the investigation at hour one was still anchoring it at hour nine.
A subagent that owed the last one nothing
I didn't argue the evidence, and I didn't tell it to keep grinding on the same thread either. I told it to open an Opus subagent and have that subagent investigate independently.
The instruction I gave it mattered as much as the decision to give it: don't inherit my read of this, and don't inherit yours either — if you think a verification step upstream was done wrong, don't take it on faith, redo it. The dispatch handed the new subagent the entire trail — every hypothesis raised, tested, and marked ruled-out or inconclusive over the previous nine hours — with explicit permission to distrust every one of them, plus careful, reversible access to the live cluster to go check for itself.
Twenty-eight minutes later it came back with an answer, and the answer was uncomfortable in a specific way: one of the hypotheses from hour one had been right all along. It had just never actually been tested.
The datapath nobody restarted
The candidate was Cilium's own socket-level load balancing. With kube-proxy replacement on, Cilium's eBPF datapath resolves a Service's ClusterIP to a specific pod IP before istio-cni's netns-scoped REDIRECT rule gets a chance to preserve the original ClusterIP as the connection's SO_ORIGINAL_DST. By the time ztunnel inspects the redirected connection, it's already looking at a pod IP, not the Service VIP — and a waypoint binding is keyed to the Service, so it can never apply. Cilium's own chart documents the fix for exactly this case: socketLB.hostNamespaceOnly: true, restricting that early resolution to the host network namespace and leaving pod-namespace traffic alone.
That value had been set, hours earlier, at hypothesis two of nine. helm upgrade had gone out, the cilium-config ConfigMap had updated, and the ConfigMap was what every verification step after that checked — because a ConfigMap is where Kubernetes config normally lives, and reading it back is normally proof enough. It wasn't, here. cilium-agent reads that config exactly once, at its own startup, and compiles it directly into the cgroup BPF program it runs — not something a controller loop re-reads on a timer. And the Helm chart puts no config-hash annotation on the DaemonSet's pod template, so a values-only change leaves that template byte-for-byte identical and triggers no rollout at all. The ConfigMap said true. The compiled datapath, still running the same process it booted with, was still full — silently declining every waypoint binding it was handed, with nothing anywhere logging so much as a warning.
The subagent's proof wasn't a guess dressed up as confidence — it was three independent lines converging. cilium-dbg status --verbose read Socket LB Coverage: Full, not the Hostns-only the ConfigMap promised. The DaemonSet's own history read the wrong way round: the Helm revision that introduced the setting landed nineteen hours after the running cilium-agent pod's own start time, with a restart count still at zero and a controller-revision-hash still pointing at a template from a week earlier. And a live trace, cilium-dbg monitor -t trace-sock, caught the DNAT happening — Service VIP rewritten to pod IP, inside the calling pod's own cgroup, before ztunnel had any chance to see the original destination.
The fix, once that was visible, was one line:
kubectl -n kube-system rollout restart daemonset/cilium
Socket LB Coverage read Hostns-only afterward. The header test that had failed all night routed correctly on the first try. I wrote back one word — do it — and a few minutes later: it finally worked.
Not stuck, anchored
What the first nine hours got wrong wasn't the hypothesis. Hypothesis two was correct from the start. What they got wrong was treating a ConfigMap read as proof that a change had taken effect, and then never going back to re-examine that specific piece of evidence once it had been filed away as "applied." Every hour after that built on the same foundation, and no amount of additional hours was going to notice a crack in a foundation nobody was looking at anymore. That's a different failure than "not smart enough" or "gave up too easily" — the elimination really was careful, right up until the point where it wasn't checking anything new.
An independent subagent fixed it not by being a better engineer, but by not knowing yet which piece of evidence was supposed to be settled. Told explicitly not to trust the read, it went and checked the one thing nobody had checked twice: whether the config change had reached the process actually running the datapath, or only the object Kubernetes stores it in. The gap between those two things is exactly where this kind of bug lives, and it's invisible from inside an investigation that's already decided where the line between "verified" and "still open" sits.
The smaller fix landed the same night, in the values file, as a comment for whoever changes this setting next: after touching anything under socketLB, restart the DaemonSet and check the live datapath, not the ConfigMap. A second one landed in the repo's own CLAUDE.md a couple of hours later, general enough to outlast this specific bug — a rule against testing changes directly against a live GitOps-managed resource before they're committed, because ArgoCD's self-heal will revert an uncommitted edit with nothing pointing at the cause. Both are the same lesson in different clothes: know exactly what you're actually checking, and don't let an early confirmation retire the question that led you to check it in the first place.
Top comments (0)