DEV Community

Ryan
Ryan

Posted on Edited on Originally published at zenn.dev

Restarting Tailscale silently isolated a k3s node: fixing a flannel VXLAN bug upstream

This is an English version of my original Japanese article on Zenn, translated with the help of AI.

TL;DR

  • On k3s with the flannel VXLAN backend running over a VPN interface (Tailscale here), restarting or upgrading tailscaled left one node unable to reach Pods on any other node. The node stayed Ready and nothing alerted.
  • flannel recreated flannel.1, but the kernel had already dropped the routes, ARP and FDB entries that went through it, and flannel only rewrites those when a remote lease changes.
  • The fix replays the known remote leases after the device is recreated: flannel-io/flannel#2618, released in v0.28.11 (a pre-release as of 2026-11-20).
  • Until your k3s ships it, restart k3s / k3s-agent after a VPN restart and alert on the number of flannel.1 routes.

Issue: vxlan: routes/ARP/FDB not restored after the external interface is recreated (#2617)

Who this affects: anyone running the flannel VXLAN backend (including k3s's embedded flannel) with --iface (flannel-iface in k3s) set to a VPN interface such as Tailscale, NetBird or WireGuard, on flannel v0.28.10 or earlier.

Environment

  • k3s v1.35.10+k3s1 (embedded flannel v0.28.10, VXLAN backend, IPv4 only)
  • 4 nodes, Oracle Linux 9.7 (aarch64)
  • Node-to-node traffic over Tailscale 1.104.1 (flannel-iface: tailscale0)

k3s also has a Tailscale integration (--vpn-auth, experimental); flannel runs on tailscale0 in that case as well.

Symptoms

After restarting tailscaled (or a Tailscale package upgrade), the k3s log showed:

Interface flannel.1 deleted
vxlan device missing, attempting to recreate...
external interface tailscale0 not found, retrying in 1s
VXLAN device flannel.1 recreated successfully
Enter fullscreen mode Exit fullscreen mode

flannel.1 came back, so it looked fine. But on that node:

  • ip route | grep -c 'dev flannel.1' dropped from 3 (the number of other nodes) to 0
  • ip neigh show dev flannel.1 and bridge fdb show dev flannel.1 were empty
  • Traffic to Pods on other nodes timed out (the ingress proxy on the node logged upstream connect error ... connection timeout)
  • The node stayed Ready, and nothing more was logged after recreated successfully

It did not recover after about 2.5 minutes. Restarting k3s-agent brought the routes back within about a second.

Root cause

Following pkg/backend/vxlan/vxlan_network.go, two things combine:

  1. When the external interface (tailscale0) disappears, the kernel removes flannel.1 together with the routes, ARP and FDB entries that use it.
  2. flannel's reCreateVxlan() recreates the device itself, but routes, ARP and FDB entries are written only by handleSubnetEvents(), which runs only when a lease (another node's subnet) changes.

Recreating the interface does not change any remote lease, so the entries are never written again and the node stays isolated until flannel restarts.

Older releases (v0.28.4, for example) panicked on this path in IPv4-only setups (#2433), which restarted k3s and, as a side effect, restored the routes. Once that panic was fixed, the recreate path completed "successfully" and the node quietly stayed isolated instead.

The fix

The PR makes three changes:

  • Keep the remote leases passed to handleSubnetEvents() in a lease.LeaseWatcher owned by the Run loop.
  • Make reCreateVxlan() return the new device instead of overwriting nw.dev / nw.v6Dev from a goroutine.
  • In the Run loop, take the new device and replay the known leases into handleSubnetEvents() as EventAdded, which rewrites routes, ARP and FDB entries.
// knownLeaseEvents returns an EventAdded for every known remote lease.
func (nw *network) knownLeaseEvents() []lease.Event {
    batch := make([]lease.Event, 0, len(nw.knownLeases.Leases))
    for _, l := range nw.knownLeases.Leases {
        batch = append(batch, lease.Event{Type: lease.EventAdded, Lease: l})
    }
    return batch
}
Enter fullscreen mode Exit fullscreen mode

The second change also removes an unsynchronized write to nw.dev from another goroutine.

Test

Using ns.SetUpNetlinkTest, as pkg/backend/route_network_test.go does, the new test TestRecreatedDeviceGetsKnownLeases creates a dummy interface in a network namespace and:

  1. creates the VXLAN device and processes one remote lease,
  2. deletes and recreates the external interface,
  3. checks that the route to the remote subnet comes back via flannel.1.

Against the old code it fails with no route to 198.18.2.0/24 via flannel.1 after flannel.1 was recreated; with the fix it passes, both shown in CI.

pkg/backend/vxlan was not in make test's TEST_PACKAGES at the time, so following review feedback I added it in the same PR.

Detecting it until the fix reaches you

The node stays Ready, so you need another signal. node_exporter's network_route collector (listed under "Disabled by default"; enable it with --collector.network_route) exports the number of routes per interface as node_network_routes{device="..."}.

With the VXLAN backend, a healthy node has as many flannel.1 routes as there are other nodes. When all routes disappear the series disappears too, so the rule also catches a missing series:

- alert: FlannelVxlanRoutesMissing
  expr: |
    (
      (
        max by (node) (node_network_routes{job="node-exporter", device="flannel.1"})
          < scalar(count(kube_node_info) - 1)
      )
      or
      (
        max by (node) (node_uname_info{job="node-exporter"})
          unless on (node) node_network_routes{job="node-exporter", device="flannel.1"}
      )
    )
    and on () (count(kube_node_info) > 1)
  for: 3m
Enter fullscreen mode Exit fullscreen mode
  • This assumes a node label on node-exporter series (commonly added by relabeling in kube-prometheus-stack) and kube_node_info from kube-state-metrics.
  • The same collector also exports one node_network_route_info series per route. With Tailscale that grows with the number of peers, so I dropped it with metricRelabelings and kept only the counts.

What to do

  • Upgrade to flannel v0.28.11 or later. k3s embeds flannel, so you need a k3s release that includes v0.28.11. k3s picks up stable flannel releases automatically with Updatecli, but as of 2026-11-20 v0.28.11 is still a pre-release and k3s main go.mod pins flannel v0.28.10.
  • Until then, restart k3s / k3s-agent after restarting or upgrading tailscaled, and check that the number of flannel.1 routes equals the number of other nodes.
  • Consider pausing Tailscale auto-updates until the fix is in.

Notes on sending it upstream

  • An issue with the same symptom (reproduced with NetBird, #2617) already existed, so instead of opening a new one I added reproduction steps and logs from a different setup (Tailscale) as a comment, and linked the related k3s issue (k3s-io/k3s#12436).
  • The PR's e2e tests failed, but the failure was unrelated: it also happened on master with the new ubuntu-latest runner. I wrote up the cause (the test CA certificate was not accepted as a CA) in a separate issue (#2620), sent the fix as a separate PR (#2621), got that merged first, and then rebased the main PR.
  • I showed in CI that the new test fails on the old code, so reviewers could see it really catches the bug.

How I used generative AI

I used generative AI (Claude Code and Codex) for the investigation and the implementation. Roughly:

  • What I did myself: noticed the symptom in my own cluster, reproduced it by restarting tailscaled, checked the logs, routes and FDB, matched it to the existing issue, judged whether the fix approach (replaying known leases) was sound, and followed the CI results and the conversation with the maintainers.
  • What I delegated to AI: reading the flannel source and locating the relevant code, drafting the fix and the test, and writing the English issue comment and PR description.

I did not ship the AI's code or explanations as they were: I confirmed in CI that the test fails before the fix and passes after it before opening the PR. Checking a root-cause hypothesis against tests and the kernel's actual state, rather than against an AI's explanation, is an order I intend to keep.

Summary

  • Up to v0.28.10, when flannel VXLAN's external interface was recreated, the device came back but routes, ARP and FDB entries did not.
  • The node stays Ready, so you won't notice unless you watch the route count.
  • v0.28.11 replays the known leases onto the recreated device.
  • I used generative AI for reading code and drafting, and verified correctness with tests and the real environment.

Reproducing a problem from your own environment, finding the cause, and sending it upstream with a test helps everyone running the same setup, and lets you remove the workaround from your own operations. I plan to keep sending what I hit upstream.

Top comments (0)