<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ryan</title>
    <description>The latest articles on DEV Community by Ryan (@ryan0215).</description>
    <link>https://dev.to/ryan0215</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254830%2F361ba136-1272-4213-98e2-b2232d2ffec0.jpg</url>
      <title>DEV Community: Ryan</title>
      <link>https://dev.to/ryan0215</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ryan0215"/>
    <language>en</language>
    <item>
      <title>Restarting Tailscale silently isolated a k3s node: fixing a flannel VXLAN bug upstream</title>
      <dc:creator>Ryan</dc:creator>
      <pubDate>Fri, 20 Nov 2026 04:33:06 +0000</pubDate>
      <link>https://dev.to/ryan0215/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug-upstream-1p0n</link>
      <guid>https://dev.to/ryan0215/restarting-tailscale-silently-isolated-a-k3s-node-fixing-a-flannel-vxlan-bug-upstream-1p0n</guid>
      <description>&lt;p&gt;&lt;em&gt;This is an English version of &lt;a href="https://zenn.dev/ryan0215/articles/flannel-vxlan-lease-replay" rel="noopener noreferrer"&gt;my original Japanese article on Zenn&lt;/a&gt;, translated with the help of AI.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On k3s with the flannel VXLAN backend running over a VPN interface (Tailscale here), restarting or upgrading &lt;code&gt;tailscaled&lt;/code&gt; left one node unable to reach Pods on any other node. The node stayed &lt;code&gt;Ready&lt;/code&gt; and nothing alerted.&lt;/li&gt;
&lt;li&gt;flannel recreated &lt;code&gt;flannel.1&lt;/code&gt;, but the kernel had already dropped the routes, ARP and FDB entries that went through it, and flannel only rewrites those when a remote lease &lt;em&gt;changes&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;The fix replays the known remote leases after the device is recreated: &lt;a href="https://github.com/flannel-io/flannel/pull/2618" rel="noopener noreferrer"&gt;flannel-io/flannel#2618&lt;/a&gt;, released in &lt;a href="https://github.com/flannel-io/flannel/releases/tag/v0.28.11" rel="noopener noreferrer"&gt;v0.28.11&lt;/a&gt; (a pre-release as of 2026-11-20).&lt;/li&gt;
&lt;li&gt;Until your k3s ships it, restart &lt;code&gt;k3s&lt;/code&gt; / &lt;code&gt;k3s-agent&lt;/code&gt; after a VPN restart and alert on the number of &lt;code&gt;flannel.1&lt;/code&gt; routes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Issue: &lt;a href="https://github.com/flannel-io/flannel/issues/2617" rel="noopener noreferrer"&gt;vxlan: routes/ARP/FDB not restored after the external interface is recreated (#2617)&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Who this affects:&lt;/strong&gt; anyone running the flannel VXLAN backend (including k3s's embedded flannel) with &lt;code&gt;--iface&lt;/code&gt; (&lt;code&gt;flannel-iface&lt;/code&gt; in k3s) set to a VPN interface such as Tailscale, NetBird or WireGuard, on flannel v0.28.10 or earlier.&lt;/p&gt;

&lt;h2&gt;
  
  
  Environment
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;k3s v1.35.10+k3s1 (embedded flannel v0.28.10, VXLAN backend, IPv4 only)&lt;/li&gt;
&lt;li&gt;4 nodes, Oracle Linux 9.7 (aarch64)&lt;/li&gt;
&lt;li&gt;Node-to-node traffic over Tailscale 1.104.1 (&lt;code&gt;flannel-iface: tailscale0&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;k3s also has a &lt;a href="https://docs.k3s.io/networking/distributed-multicloud#integration-with-the-tailscale-vpn-provider-experimental" rel="noopener noreferrer"&gt;Tailscale integration&lt;/a&gt; (&lt;code&gt;--vpn-auth&lt;/code&gt;, experimental); flannel runs on &lt;code&gt;tailscale0&lt;/code&gt; in that case as well.&lt;/p&gt;

&lt;h2&gt;
  
  
  Symptoms
&lt;/h2&gt;

&lt;p&gt;After restarting &lt;code&gt;tailscaled&lt;/code&gt; (or a Tailscale package upgrade), the k3s log showed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Interface flannel.1 deleted
vxlan device missing, attempting to recreate...
external interface tailscale0 not found, retrying in 1s
VXLAN device flannel.1 recreated successfully
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;flannel.1&lt;/code&gt; came back, so it looked fine. But on that node:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;ip route | grep -c 'dev flannel.1'&lt;/code&gt; dropped from 3 (the number of other nodes) to &lt;strong&gt;0&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ip neigh show dev flannel.1&lt;/code&gt; and &lt;code&gt;bridge fdb show dev flannel.1&lt;/code&gt; were empty&lt;/li&gt;
&lt;li&gt;Traffic to Pods on other nodes timed out (the ingress proxy on the node logged &lt;code&gt;upstream connect error ... connection timeout&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;The node stayed &lt;code&gt;Ready&lt;/code&gt;, and nothing more was logged after &lt;code&gt;recreated successfully&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It did not recover after about 2.5 minutes. Restarting &lt;code&gt;k3s-agent&lt;/code&gt; brought the routes back within about a second.&lt;/p&gt;

&lt;h2&gt;
  
  
  Root cause
&lt;/h2&gt;

&lt;p&gt;Following &lt;code&gt;pkg/backend/vxlan/vxlan_network.go&lt;/code&gt;, two things combine:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;When the external interface (&lt;code&gt;tailscale0&lt;/code&gt;) disappears, the kernel removes &lt;code&gt;flannel.1&lt;/code&gt; together with the routes, ARP and FDB entries that use it.&lt;/li&gt;
&lt;li&gt;flannel's &lt;code&gt;reCreateVxlan()&lt;/code&gt; recreates the device itself, but routes, ARP and FDB entries are written only by &lt;code&gt;handleSubnetEvents()&lt;/code&gt;, which runs &lt;strong&gt;only when a lease (another node's subnet) changes&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Recreating the interface does not change any remote lease, so the entries are never written again and the node stays isolated until flannel restarts.&lt;/p&gt;

&lt;p&gt;Older releases (v0.28.4, for example) panicked on this path in IPv4-only setups (&lt;a href="https://github.com/flannel-io/flannel/issues/2433" rel="noopener noreferrer"&gt;#2433&lt;/a&gt;), which restarted k3s and, as a side effect, restored the routes. Once that panic was fixed, the recreate path completed "successfully" and the node quietly stayed isolated instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;The PR makes three changes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Keep the remote leases passed to &lt;code&gt;handleSubnetEvents()&lt;/code&gt; in a &lt;code&gt;lease.LeaseWatcher&lt;/code&gt; owned by the Run loop.&lt;/li&gt;
&lt;li&gt;Make &lt;code&gt;reCreateVxlan()&lt;/code&gt; return the new device instead of overwriting &lt;code&gt;nw.dev&lt;/code&gt; / &lt;code&gt;nw.v6Dev&lt;/code&gt; from a goroutine.&lt;/li&gt;
&lt;li&gt;In the Run loop, take the new device and replay the known leases into &lt;code&gt;handleSubnetEvents()&lt;/code&gt; as &lt;code&gt;EventAdded&lt;/code&gt;, which rewrites routes, ARP and FDB entries.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c1"&gt;// knownLeaseEvents returns an EventAdded for every known remote lease.&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nw&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;knownLeaseEvents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="n"&gt;lease&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Event&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="n"&gt;lease&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;knownLeases&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Leases&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;nw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;knownLeases&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Leases&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lease&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;Type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;lease&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;EventAdded&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Lease&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second change also removes an unsynchronized write to &lt;code&gt;nw.dev&lt;/code&gt; from another goroutine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test
&lt;/h2&gt;

&lt;p&gt;Using &lt;code&gt;ns.SetUpNetlinkTest&lt;/code&gt;, as &lt;code&gt;pkg/backend/route_network_test.go&lt;/code&gt; does, the new test &lt;code&gt;TestRecreatedDeviceGetsKnownLeases&lt;/code&gt; creates a dummy interface in a network namespace and:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;creates the VXLAN device and processes one remote lease,&lt;/li&gt;
&lt;li&gt;deletes and recreates the external interface,&lt;/li&gt;
&lt;li&gt;checks that the route to the remote subnet comes back via &lt;code&gt;flannel.1&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Against the old code it fails with &lt;code&gt;no route to 198.18.2.0/24 via flannel.1 after flannel.1 was recreated&lt;/code&gt;; with the fix it passes, both shown in CI.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pkg/backend/vxlan&lt;/code&gt; was not in &lt;code&gt;make test&lt;/code&gt;'s &lt;code&gt;TEST_PACKAGES&lt;/code&gt; at the time, so following review feedback I added it in the same PR.&lt;/p&gt;

&lt;h2&gt;
  
  
  Detecting it until the fix reaches you
&lt;/h2&gt;

&lt;p&gt;The node stays &lt;code&gt;Ready&lt;/code&gt;, so you need another signal. node_exporter's &lt;code&gt;network_route&lt;/code&gt; collector (listed under "&lt;a href="https://github.com/prometheus/node_exporter#disabled-by-default" rel="noopener noreferrer"&gt;Disabled by default&lt;/a&gt;"; enable it with &lt;code&gt;--collector.network_route&lt;/code&gt;) exports the number of routes per interface as &lt;code&gt;node_network_routes{device="..."}&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;With the VXLAN backend, a healthy node has as many &lt;code&gt;flannel.1&lt;/code&gt; routes as there are other nodes. When all routes disappear the series disappears too, so the rule also catches a missing series:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;FlannelVxlanRoutesMissing&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;(&lt;/span&gt;
      &lt;span class="s"&gt;(&lt;/span&gt;
        &lt;span class="s"&gt;max by (node) (node_network_routes{job="node-exporter", device="flannel.1"})&lt;/span&gt;
          &lt;span class="s"&gt;&amp;lt; scalar(count(kube_node_info) - 1)&lt;/span&gt;
      &lt;span class="s"&gt;)&lt;/span&gt;
      &lt;span class="s"&gt;or&lt;/span&gt;
      &lt;span class="s"&gt;(&lt;/span&gt;
        &lt;span class="s"&gt;max by (node) (node_uname_info{job="node-exporter"})&lt;/span&gt;
          &lt;span class="s"&gt;unless on (node) node_network_routes{job="node-exporter", device="flannel.1"}&lt;/span&gt;
      &lt;span class="s"&gt;)&lt;/span&gt;
    &lt;span class="s"&gt;)&lt;/span&gt;
    &lt;span class="s"&gt;and on () (count(kube_node_info) &amp;gt; 1)&lt;/span&gt;
  &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;3m&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;This assumes a &lt;code&gt;node&lt;/code&gt; label on node-exporter series (commonly added by relabeling in kube-prometheus-stack) and &lt;code&gt;kube_node_info&lt;/code&gt; from kube-state-metrics.&lt;/li&gt;
&lt;li&gt;The same collector also exports one &lt;code&gt;node_network_route_info&lt;/code&gt; series per route. With Tailscale that grows with the number of peers, so I dropped it with &lt;code&gt;metricRelabelings&lt;/code&gt; and kept only the counts.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What to do
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Upgrade to flannel v0.28.11 or later. k3s embeds flannel, so you need a k3s release that includes v0.28.11. k3s &lt;a href="https://github.com/k3s-io/k3s/blob/main/updatecli/updatecli.d/flannel.yaml" rel="noopener noreferrer"&gt;picks up stable flannel releases automatically with Updatecli&lt;/a&gt;, but as of 2026-11-20 v0.28.11 is still a pre-release and k3s &lt;code&gt;main&lt;/code&gt; &lt;a href="https://github.com/k3s-io/k3s/blob/main/go.mod" rel="noopener noreferrer"&gt;&lt;code&gt;go.mod&lt;/code&gt;&lt;/a&gt; pins flannel v0.28.10.&lt;/li&gt;
&lt;li&gt;Until then, restart &lt;code&gt;k3s&lt;/code&gt; / &lt;code&gt;k3s-agent&lt;/code&gt; after restarting or upgrading &lt;code&gt;tailscaled&lt;/code&gt;, and check that the number of &lt;code&gt;flannel.1&lt;/code&gt; routes equals the number of other nodes.&lt;/li&gt;
&lt;li&gt;Consider pausing Tailscale auto-updates until the fix is in.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Notes on sending it upstream
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;An issue with the same symptom (reproduced with NetBird, #2617) already existed, so instead of opening a new one I added &lt;strong&gt;reproduction steps and logs from a different setup (Tailscale)&lt;/strong&gt; as a comment, and linked the related k3s issue (&lt;a href="https://github.com/k3s-io/k3s/issues/12436" rel="noopener noreferrer"&gt;k3s-io/k3s#12436&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;The PR's e2e tests failed, but the failure was unrelated: it also happened on &lt;code&gt;master&lt;/code&gt; with the new &lt;code&gt;ubuntu-latest&lt;/code&gt; runner. I wrote up the cause (the test CA certificate was not accepted as a CA) in a separate issue (&lt;a href="https://github.com/flannel-io/flannel/issues/2620" rel="noopener noreferrer"&gt;#2620&lt;/a&gt;), sent the fix as a separate PR (&lt;a href="https://github.com/flannel-io/flannel/pull/2621" rel="noopener noreferrer"&gt;#2621&lt;/a&gt;), got that merged first, and then rebased the main PR.&lt;/li&gt;
&lt;li&gt;I showed in CI that the new test fails on the old code, so reviewers could see it really catches the bug.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How I used generative AI
&lt;/h2&gt;

&lt;p&gt;I used generative AI (Claude Code and Codex) for the investigation and the implementation. Roughly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;What I did myself:&lt;/strong&gt; noticed the symptom in my own cluster, reproduced it by restarting &lt;code&gt;tailscaled&lt;/code&gt;, checked the logs, routes and FDB, matched it to the existing issue, judged whether the fix approach (replaying known leases) was sound, and followed the CI results and the conversation with the maintainers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What I delegated to AI:&lt;/strong&gt; reading the flannel source and locating the relevant code, drafting the fix and the test, and writing the English issue comment and PR description.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I did not ship the AI's code or explanations as they were: I confirmed in CI that the test fails before the fix and passes after it before opening the PR. Checking a root-cause hypothesis against tests and the kernel's actual state, rather than against an AI's explanation, is an order I intend to keep.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Up to v0.28.10, when flannel VXLAN's external interface was recreated, the device came back but routes, ARP and FDB entries did not.&lt;/li&gt;
&lt;li&gt;The node stays &lt;code&gt;Ready&lt;/code&gt;, so you won't notice unless you watch the route count.&lt;/li&gt;
&lt;li&gt;v0.28.11 replays the known leases onto the recreated device.&lt;/li&gt;
&lt;li&gt;I used generative AI for reading code and drafting, and verified correctness with tests and the real environment.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Reproducing a problem from your own environment, finding the cause, and sending it upstream with a test helps everyone running the same setup, and lets you remove the workaround from your own operations. I plan to keep sending what I hit upstream.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>k3s</category>
      <category>networking</category>
      <category>sre</category>
    </item>
  </channel>
</rss>
