How Two Processes Fighting Over One WireGuard Interface Broke Our NetBird Mesh in Kubernetes

Jagoum — Wed, 15 Jul 2026 13:25:18 +0000

How Two Processes Fighting Over One WireGuard Interface Broke Our NetBird Mesh in Kubernetes

We run NetBird as a WireGuard mesh overlay across our OpenStack Kubernetes cluster. Three router pods (DaemonSet) advertise K8s service and pod CIDRs to remote agents over the mesh. One proxy pod forwards agent traffic to K8s ClusterIPs.

Last week, all three routers went unstable — relays disconnected, peers dropped to 0/0, and the engine entered a restart loop with 256+ restarts. Here's how we found and fixed it.

The Symptoms

Management: Connected
Signal: Connected  
Relays: 0/1 Available    ← only 1 relay, none connected
Networks: -              ← CIDRs not advertised
Peers: 0/0 Connected     ← nobody can reach us

Meanwhile, the proxy pod on the same cluster was rock-solid:

Management: Connected
Relays: 3/3 Available
Peers: 7/14 Connected

Both were running the same NetBird version (0.73.2), same setup key, same management URL. But the routers were dying and the proxy wasn't.

Investigation: Layer by Layer

Layer 1: Version Mismatch (Red Herring)

The original symptom was failed to handle new offer: relay connection is not established and relay WebSocket connections dying with EOF every ~1.6 seconds. We'd recently upgraded the routers from 0.73.2 to 0.74.2, but the self-hosted relay server was still 0.73.x.

Fix: Downgraded routers back to 0.73.2. The relay connection storms stopped. But the instability remained.

Layer 2: Stale WireGuard Interfaces

After the downgrade, two routers couldn't bind port 51820 — the old WireGuard interface (wt0) persisted on the node from the previous pod. We deleted the stale interfaces:

kubectl exec -n netbird-system <pod> -- ip link delete wt0
kubectl delete pod -n netbird-system <pod>

Workers 2 and 3 recovered. Worker-1 didn't.

Layer 3: The Dual-Process Bug

Worker-1's logs told the real story:

2026-07-14T14:50:14.631Z netbird WireGuard interface monitor: interface wt0 deleted, restarting engine
2026-07-14T14:50:15.354Z netbird stopped Netbird Engine
2026-07-14T14:50:16.098Z netbird new WireGuard interface wt0 has been created
2026-07-14T14:50:18.307Z netbird WireGuard interface monitor: interface wt0 deleted, restarting engine

The WGIfaceMonitor was detecting wt0 deletions every few seconds, triggering engine restarts in a loop.

The root cause: The DaemonSet had no explicit command, so it used the image's default entrypoint. That entrypoint does this:

main() {
  "${NETBIRD_BIN}" service run &    # Process 1: daemon mode
  wait_for_daemon_startup 30
  connect                           # Process 2: "netbird up" (foreground)
  wait "${service_pids[@]}"
}

Two independent NetBird engines were running in the same container, both managing separate WireGuard interfaces, both fighting for the same management gRPC connection. The service run daemon would create wt0, the foreground netbird up process would see the conflict and interfere, the monitor would detect the interference as a deletion, and the engine would restart.

The proxy pod was stable because it explicitly sets command: ["netbird", "service", "run"] — a single process.

Layer 4: Cilium WireGuard Port Conflict

Compounding the problem: Cilium has enable-wireguard: true, which creates cilium_wg0 on every node. The kernel WireGuard module binds UDP port 51820 with no visible userspace process:

$ ss -ulnp | grep 51820
UNCONN  0  0  0.0.0.0:51820  ← kernel socket, no process owner

NetBird's default WireGuard port is also 51820. With the kernel socket holding the port, NetBird falls back to random ports — which works but adds instability when combined with the dual-process fight.

Layer 5: Missing Hardening

The proxy had these env vars set but the routers didn't:

NB_DISABLE_NAT_MAPPER=true — disables UPnP discovery (useless on cloud VMs, clutters logs)
NB_DISABLE_EBPF_WG_PROXY=true — disables eBPF proxy that can hold ports across restarts

The Fix

Match the proxy's configuration. One command change + two env vars:

containers:
- name: netbird
  command: ["netbird", "service", "run"]   # ← explicit single process
  env:
  - name: NB_SETUP_KEY
    valueFrom:
      secretKeyRef:
        key: setup_key
        name: netbird-router-setup-key
  - name: NB_WIREGUARD_PORT
    value: "51820"
  - name: NB_MANAGEMENT_URL
    value: "https://netbird.vpn.skyengpro.app:443"
  - name: NB_HOSTNAME
    value: "$(NODE_NAME)"
  - name: NB_DISABLE_NAT_MAPPER           # ← new
    value: "true"
  - name: NB_DISABLE_EBPF_WG_PROXY        # ← new
    value: "true"

The liveness probe also needed updating to only check for the single process:

livenessProbe:
  exec:
    command: ["sh", "-c", "pgrep -f 'netbird service' >/dev/null || kill -0 1"]

After the Fix

All three routers went from broken to fully stable:

	Worker-1	Worker-2	Worker-3
Before	0/0 peers, 256 restarts	0/0 peers, 264 restarts	3-5/13 peers, flapping
After	7/13 peers, 0 restarts	7/13 peers, 0 restarts	7/13 peers, 0 restarts

Management ✅ Signal ✅ Relays: 3/3 ✅ Networks: 10.20.96.0/19 ✅

Lessons Learned

1. The proxy was your best reference implementation. It was the only stable component all along. When debugging, compare working vs broken configs line by line.

2. Image entrypoints can surprise you. We didn't set a command in the DaemonSet, so we got the image's default entrypoint — which runs two processes. Always check docker inspect <image> or read the Dockerfile before deploying.

3. Kernel WireGuard modules don't show in ss -ulnp. The cilium_wg0 socket was invisible because the kernel module owns it, not a userspace process. This caused confusing "port 51820 is in use" messages with no visible process.

4. HostNetwork pods share the host's WireGuard module state. Stale wt0 interfaces survive pod deletion. When rolling out new pods, you may need to clean up old interfaces first.

5. Don't mix service run and foreground netbird up. They're two independent engines that fight over the same interface name and management connection. Pick one.

TL;DR

The DaemonSet's default entrypoint started two NetBird engines in the same container. They fought over the wt0 WireGuard interface, causing an infinite restart loop. The fix: explicitly set command: ["netbird", "service", "run"] and add NB_DISABLE_NAT_MAPPER=true + NB_DISABLE_EBPF_WG_PROXY=true.

If you're running NetBird in Kubernetes, check your DaemonSet entrypoint. If it doesn't explicitly set command, you might have the same dual-process bug hiding in plain sight.

My First Ever Global Hackathan, With MIT Sloan was an amazing experience. Do i did not win, but i got experience.

Jagoum — Mon, 11 Aug 2025 06:40:40 +0000

One of My first Programming language I learned.

Jagoum — Thu, 19 Dec 2024 00:00:38 +0000

On the first language I learned in my life was HTML while I was still in secondary school and never even had the idea of becoming a software engineer. It was very fun to learn and to see how website are actually being developed.
I learned C programming language when I was writing my exams to leave secondary school. And go to high school. There I fell in love with it .

I loved it because I could actually write a program that can solve a problem like calculating numbers when I wrote my first calculator program in C.
After my advanced level I obtained a scholarship into a preemployment training program by adorsys in Cameroon I am currently working.

I obtained my LPIC-1 there with 100% in both 101 & 102

So right now I am currently studying rust and java programming languages.

DEV Community: Jagoum

How Two Processes Fighting Over One WireGuard Interface Broke Our NetBird Mesh in Kubernetes