DEV Community

Osmium
Osmium

Posted on Originally published at osmiumsilver.github.io

How Tailscale's Route Table Gets Wiped by mwan3 on OpenWrt

#openwrt #tailscale #networking #mwan3

My trusty Netcore N60 Pro runs on OpenWrt 25.12 with both mwan3 (2.12.0) and Tailscale (v1.98.3) installed. Tailscale would work fine, then randomly stop being reachable. Every time I touched the mwan3 config, tailscale broke immediately after.

Background: routing tables, ip rule, and firewall marks

Before the investigation, it's worth untangling three concepts that are easy to conflate:

Routing table — Linux allows multiple routing tables at once, each with a numeric id. The default main table is 254. Tailscale uses its own table, id 52, to hold peer routes. mwan3 allocates one table per WAN interface.

IP rule — decides which table a packet's lookup goes to. It sits above the routing tables and matches top to bottom by priority, e.g. from all lookup 52 means "send all packets to table 52 for the lookup."

Firewall mark (fwmark) — mwan3's mechanism is to have iptables/nftables mark a packet first (e.g. tag which WAN it came in on), then have ip rule route based on that mark to the matching table. This is why mwan3 depends so heavily on the firewall layer.

The investigation: a few wrong turns first

First guess: mwan3 must be deleting something

mwan3 restart was rebuilding its own policy routing rules and clobbering tailscale's rule pointing at table 52. Testing didn't confirm it:

root@N60Pro:~# ip rule | grep 52
1370: from all lookup 52
root@N60Pro:~# ip route show table 52
root@N60Pro:~#
Enter fullscreen mode Exit fullscreen mode

The rule was still there, but table 52 itself was empty. This narrowed things down: the problem wasn't at the ip rule layer, it was the table itself getting wiped.

Second guess: This nft seems to be doing weird things

mwan3 restart was throwing a screen full of errors:

iptables v1.8.10 (nf_tables): table `mangle' is incompatible, use 'nft' tool.
Error: cmp sreg undef
Error: meta sreg is not an immediate
Enter fullscreen mode Exit fullscreen mode

OpenWrt has used nftables/firewall4 since 22.03, but the official mwan3 package is still iptables-based and relies on the iptables-nft compatibility shim, which frequently fails to translate rules. The suspicion was that this broken compat layer was causing mwan3 to malfunction outright, which led to considering mwan3-nft (a community nftables port) or even downgrading OpenWrt entirely.

But doing a nft list ruleset showed mwan3's own chains and packet counters actively incrementing (packets 3956602 bytes 456220071, etc.), proving mwan3's core policy routing was actually working fine, those errors were log noise, unrelated to table 52 being wiped. So that leads to one precise question: who is clearing table 52, and why?

Finding the code: mwan3's flush loop

With a little help from Claude, grepping mwan3's init script for anything that flushes routing tables:

grep -rn "ip route flush\|route flush" /lib/mwan3/ /etc/init.d/mwan3
Enter fullscreen mode Exit fullscreen mode

That landed on line 77 of /etc/init.d/mwan3:

for tid in $($IP route list table all | sed -ne 's/.*table \([0-9]\+\).*/\1/p' | sort -u); do
    [ $tid -gt $MWAN3_INTERFACE_MAX ] && continue
    $IP route flush table $tid &> /dev/null
done
Enter fullscreen mode Exit fullscreen mode

So, line by line:

  1. $IP route list table all, lists every routing table currently present on the system, not just mwan3's own.
  2. sed extracts the table X number from each route, deduplicates and sorts, giving a list of every table id currently in use.
  3. For each table id: if it's greater than $MWAN3_INTERFACE_MAX, skip it. Otherwise, flush it.

Plain and simple for mwan3, but bad news for Tailscale. The code assumes "any table id below the threshold must be mine". Tailscale's table 52 just happens to sit below that threshold, so every mwan3 restart sweeps it up along with mwan3's own tables.

Why mmdefault - 3: how the threshold is computed

But what exactly is $MWAN3_INTERFACE_MAX? How it's being used and where to configure it?

Digging into mwan3's common.sh, here's where $MWAN3_INTERFACE_MAX comes from:

config_get MMX_MASK globals mmx_mask '0x3F00'

bitcnt=$(mwan3_count_one_bits MMX_MASK)
mmdefault=$(((1<<bitcnt)-1))
MWAN3_INTERFACE_MAX=$((mmdefault-3))
uci_toggle_state mwan3 globals iface_max "$MWAN3_INTERFACE_MAX"
Enter fullscreen mode Exit fullscreen mode

Breaking it down step by step:

Step 1: bitcnt. mmx_mask is the mask mwan3 uses for its fwmark bits, defaulting to 0x3F00. In binary, 0011 1111 0000 0000, which has 6 set bits. This can be verified directly from the device's own nft list ruleset output, where mwan3's rules consistently show meta mark & 0x00003f00.

Step 2: mmdefault. mmdefault = (1 << bitcnt) - 1 is simply the largest value representable with those 6 bits:

mmdefault = (1 << 6) - 1 = 64 - 1 = 63
Enter fullscreen mode Exit fullscreen mode

Step 3: subtract 3 to get the final threshold. The "-3" in mmdefault - 3 isn't arbitrary: mwan3 reserves the top few values in the mark range for special states (blackhole, unreachable, etc., visible in the ip rule output earlier as entries like 5250: from all fwmark 0x80000/0xff0000 unreachable), which can't be assigned to ordinary WAN interfaces and so get carved out of the total. Final result:

MWAN3_INTERFACE_MAX = 63 - 3 = 60
Enter fullscreen mode Exit fullscreen mode

Found it, this is the core of the problem: tailscale uses table 52, which is less than 60, so it falls squarely inside the range mwan3 manages and gets flushed on every mwan3 restart.

Here's a piquant thing, tailscale using table 52, only because the digits 5 and 2 sit above the letters T and S on a QWERTY keyboard, and there is no environment variable or startup flag to change this table number. Which means this conflict can only be resolved from mwan3's side.

The Solution

Since table 52 can't move, the fix is to push the threshold below 52 instead. Tracing back the formula: the threshold is set by the bit count of mmx_mask, and mmx_mask is an ordinary UCI config option under config globals in /etc/config/mwan3, not hardcoded. Lower the bit count, and the threshold drops with it. The default is 0x3f00, so how about a 0x1f00? 28 < 52. Should get the job done.

One small wrinkle

Changing the mask should solve the problem, but it didn't. table 52 was still being wiped, after some more scratching heads and some more Claude-please-help-me-how-it-happened, common.sh's logic for reading MWAN3_INTERFACE_MAX turned up a caching gotcha:

if [ -e "${MWAN3_STATUS_DIR}/mmx_mask" ]; then
    readfile MMX_MASK "${MWAN3_STATUS_DIR}/mmx_mask"
    MWAN3_INTERFACE_MAX=$(uci_get_state mwan3 globals iface_max)
else
    config_get MMX_MASK globals mmx_mask '0x3F00'
    # recompute bitcnt / mmdefault / MWAN3_INTERFACE_MAX ...
fi
Enter fullscreen mode Exit fullscreen mode

This recompute only happens if the cache file doesn't exist. The cache file lives at:

/var/run/mwan3/mmx_mask
Enter fullscreen mode Exit fullscreen mode

Deleting it forces the else branch to run and recompute:

uci set mwan3.globals.mmx_mask='0x1F00'
uci commit mwan3
rm -f /var/run/mwan3/mmx_mask
/etc/init.d/mwan3 restart
Enter fullscreen mode Exit fullscreen mode

Verification

After the restart, both the cache and the threshold had updated:

root@N60Pro:~# cat /var/run/mwan3/mmx_mask
0x1f00
root@N60Pro:~# cat /var/state/mwan3 | grep iface_max
mwan3.globals.iface_max='28'
Enter fullscreen mode Exit fullscreen mode

Then the full test: restart tailscale so table 52 has content, then restart mwan3 and check whether it survives.

root@N60Pro:~# /etc/init.d/tailscale restart
root@N60Pro:~# sleep 8
root@N60Pro:~# ip route show table 52
100.87.123.123 dev tailscale0
100.100.100.100 dev tailscale0
100.107.123.123 dev tailscale0

root@N60Pro:~# /etc/init.d/mwan3 restart
root@N60Pro:~# sleep 5
root@N60Pro:~# ip route show table 52
100.87.123.123 dev tailscale0
100.100.100.100 dev tailscale0
100.107.123.123 dev tailscale0
Enter fullscreen mode Exit fullscreen mode

Finally it fixed! And turns out no hack required at all.

Sources

Top comments (0)