ip route says the box has a clean default route through the LAN gateway. Then you run ip route get 10.0.0.50 and the kernel picks a path that isn't in that output at all. That gap exists because ip route only shows you the main table. The kernel checks the policy rule list first, and a VPN that died without cleaning up can leave rules there that nobody is looking at.
This post is for anyone running Tailscale, WireGuard (wg-quick), OpenVPN with policy routing, or any homegrown split-tunnel setup on Linux hosts, LXCs, or routers. If you've ever had a host where "some traffic works and some doesn't" after a VPN restart, the policy routing database is where I'd look first.
How the kernel actually picks a route
A Linux routing decision has two layers, and most tutorials only teach the second one.
- The rule list (RPDB). An ordered list of rules, checked from lowest priority number to highest. Each rule has a selector (source, fwmark, input interface, etc.) and an action (usually "look up table N").
-
The routing tables. Each table is a separate set of routes.
mainis the oneip routeshows. There can be up to 2^32 of them.
Rules are evaluated in order. If a rule says lookup 52 and table 52 has a matching route, that route wins and evaluation stops. If table 52 has no match, the kernel falls through to the next rule. If the rule's action is unreachable, blackhole, or prohibit, evaluation stops right there and the packet dies.
A freshly booted box looks like this:
$ ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
Three rules. Everything else on your system was added by something: a VPN client, a CNI plugin, NetworkManager, a script you forgot about.
What VPN clients add (and what they leave behind)
Here's what a host running Tailscale and a wg-quick tunnel looks like:
$ ip rule show
0: from all lookup local
5210: from all fwmark 0x80000/0xff0000 lookup main
5230: from all fwmark 0x80000/0xff0000 lookup default
5250: from all fwmark 0x80000/0xff0000 unreachable
5270: from all lookup 52
32764: from all lookup main suppress_prefixlength 0
32765: not from all fwmark 0xca6c lookup 51820
32766: from all lookup main
32767: from all lookup default
The 52xx block is Tailscale: its own traffic (marked with 0x80000) goes via the normal tables, and everything else consults table 52, where Tailscale puts routes for tailnet IPs and advertised subnets. The 3276x block is wg-quick with AllowedIPs = 0.0.0.0/0: unmarked traffic goes to table 51820 (the tunnel), and the suppress_prefixlength 0 rule lets more-specific routes in main still win.
When these tools shut down cleanly, they remove their rules. When they don't (SIGKILL, OOM kill, a container stopped hard, a binary run by hand in a tmux session that got nuked, an uninstall while the daemon was still running), the rules stay. The kernel doesn't know or care that the process that created them is gone.
Not every orphan is equally dangerous
This is the part the docs don't really spell out. When a network interface is deleted, the kernel removes every route whose nexthop device was that interface. So if tailscale0 disappears, table 52 typically empties itself out. A stale 5270: from all lookup 52 pointing at an empty table just falls through to the next rule. It's clutter, but it isn't actively breaking anything.
The orphans that actually hurt fall into a few categories:
-
Terminal action rules. A leftover
unreachable,blackhole, orprohibitrule kills matching packets outright. Tailscale's5250rule only matches its own fwmark, so it's mostly inert once the daemon is gone. But a custom kill-switch rule likefrom 10.0.0.0/24 unreachablefrom an OpenVPN up-script will black-hole that whole source range forever. -
Tables with gateway routes, not device routes. A route like
default via 10.0.0.1 table 100isn't bound to a VPN interface, so the kernel has no reason to remove it. The rule still points at it, and traffic quietly takes a path you thought you'd torn down. -
Blackhole/throw routes inside the table. Some kill-switch setups put
blackhole defaultin the VPN table so traffic fails closed when the tunnel drops. That's a great design until the VPN is uninstalled and the fail-closed route stays. -
Duplicates from restart loops. Scripts that run
ip rule addwithout checking first stack up identical rules on every restart. The behavior doesn't change, but cleanup stops being "delete one rule" and starts being "delete it until it stops existing". -
Interface-bound rules on a recreated interface. A rule like
iif wg0 lookup 200stays around afterwg0is deleted and shows[detached]. If something later creates a newwg0for a different purpose, the old rule reattaches to it.
That last one is a good canary, so it gets its own grep:
# Rules referencing an interface that no longer exists show [detached]
ip rule show | grep detached
ip -6 rule show | grep detached
Detecting stale rules
My forensic approach: check every rule's table against the actual route state, flag empty tables, and separately list anything with a terminal action. Here's a script I'd drop on any box that has run a VPN:
#!/usr/bin/env bash
# audit-policy-routing.sh: flag policy rules that look orphaned
set -uo pipefail
for fam in -4 -6; do
echo "=== IPv${fam#-} ==="
# Rules that do a table lookup; skip the three built-ins
ip "$fam" -j rule show | jq -r '.[] | select(.table != null)
| "\(.priority) \(.table)"' |
while read -r prio table; do
case "$table" in local|main|default) continue ;; esac
count=$(ip "$fam" route show table "$table" 2>/dev/null | wc -l)
[ "$count" -eq 0 ] && echo "prio $prio -> table $table is EMPTY"
done
# Rules that drop packets without consulting any table
ip "$fam" rule show | grep -E 'unreachable|blackhole|prohibit' \
| sed 's/^/terminal: /'
# Rules bound to interfaces that no longer exist
ip "$fam" rule show | grep detached | sed 's/^/detached: /'
done
An empty table doesn't automatically mean "stale" (Tailscale's table 52 is briefly empty during startup, for example), so treat the output as a list of suspects, not a list of things to delete.
The inverse check is just as useful: tables that hold routes but have no rule pointing at them. They're inert, but they tell you which tool was here.
# Every table that currently holds routes, with a route count
ip route show table all | grep -oE 'table [^ ]+' | sort | uniq -c
Once you have a suspect, ip route get answers the only question that matters: what does the kernel actually do with this packet right now?
# Which path does traffic to this destination take?
ip route get 10.0.0.50
# Simulate traffic carrying a specific fwmark
ip route get 10.0.0.50 mark 0x80000
# Simulate traffic arriving from a specific source
ip route get 1.1.1.1 from 10.0.0.20 iif eth0
If the output mentions table 51820 or a dev that's down and you aren't expecting it, you've found your culprit.
Removing them surgically
ip rule del deletes the first rule that matches whatever selectors you give it. Give it too few and it'll happily delete something you wanted to keep. My habit is to always pass the priority plus the full selector, copied straight out of ip rule show:
# Remove the stale wg-quick pair (copy selectors exactly from ip rule show)
ip rule del priority 32765 not fwmark 0xca6c table 51820
ip rule del priority 32764 table main suppress_prefixlength 0
# Then flush the table they pointed at
ip route flush table 51820
# Don't forget IPv6: it has a completely separate rule list
ip -6 rule del priority 32765 not fwmark 0xca6c table 51820
ip -6 route flush table 51820
The IPv6 side is the one people miss. ip rule without -6 only shows IPv4. wg-quick with ::/0 in AllowedIPs installs both, and if you only clean up v4 you get a host where IPv4 works and anything preferring IPv6 hangs. That's a fun one to debug from the application layer.
For Tailscale's set, the priorities are stable, so this is mechanical:
# Remove all four Tailscale rules (v4 and v6), then empty table 52
for fam in -4 -6; do
for prio in 5210 5230 5250 5270; do
ip "$fam" rule del priority "$prio" 2>/dev/null
done
ip "$fam" route flush table 52 2>/dev/null
done
Before doing this by hand, though: tailscaled --cleanup exists and does exactly this, plus its netfilter chains. Prefer the tool's own cleanup when it has one.
For duplicates, loop until the delete fails:
# Delete every rule at priority 100 (only if you're sure all of them are stale)
while ip rule del priority 100 2>/dev/null; do :; done
Afterwards, verify with the same commands you used to diagnose: ip rule show, ip -6 rule show, and an ip route get against the destination that was misbehaving.
Stopping them from coming back
Manual cleanup is fine once. If a VPN service crashes regularly, you want the unit itself to guarantee a clean state on both ends. The official tailscaled.service already does this with ExecStartPre and ExecStopPost calling --cleanup, and it's a pattern worth copying for anything that adds its own rules:
# /etc/systemd/system/my-vpn.service.d/routing-cleanup.conf
[Service]
# "-" prefix: don't fail the unit if the rule is already gone
ExecStartPre=-/usr/sbin/ip rule del priority 5000
ExecStartPre=-/usr/sbin/ip route flush table 100
ExecStopPost=-/usr/sbin/ip rule del priority 5000
ExecStopPost=-/usr/sbin/ip route flush table 100
ExecStopPost=-/usr/sbin/ip -6 rule del priority 5000
ExecStopPost=-/usr/sbin/ip -6 route flush table 100
ExecStopPost runs after the main process exits for any reason, including a crash or an OOM kill, which is exactly the case where the client's own cleanup never ran. ExecStartPre covers the remaining gap: whatever state the last run left behind, the next start begins clean. Run systemctl daemon-reload after adding the drop-in.
If you write your own up-scripts, make rule creation idempotent instead of relying on cleanup alone:
# Add the rule only if an identical one isn't already present
ip rule show | grep -q '^5000:.*lookup 100' \
|| ip rule add priority 5000 from 10.0.0.0/24 lookup 100
Gotchas
You're probably looking in the wrong namespace. Rules live in a network namespace. A VPN running inside an unprivileged LXC (like the setup in my Tailscale kernel TUN in unprivileged LXC post) writes rules into the container's namespace, not the host's. Running ip rule on the Proxmox host shows you nothing useful. Either exec into the container or enter its namespace from the host:
# Inspect the namespace of a container's init process from the host
nsenter -t <container-init-pid> -n ip rule show
The flip side is Docker containers with network_mode: host and NET_ADMIN. Those write straight into the host's rule list, and they never show up when you go looking at "the VPN config."
Rules that survive a reboot aren't orphans. Kernel rules are not persistent. A reboot clears them. If a stale-looking rule is back after a reboot, something is reinstalling it: a post-up line in /etc/network/interfaces, routing-policy in a netplan file, NetworkManager ipv4.routing-rules on a connection profile, or a forgotten systemd unit. Deleting it at runtime only fixes it until the next boot. grep -r for the table number across /etc is the fastest way to find the source.
Don't flush the wrong table. ip route flush table main on a remote box ends your SSH session along with everything else. Same with deleting priority 32766. If you're cleaning up over SSH, schedule a rollback before you start (systemd-run --on-active=5m systemctl restart networking, or the equivalent for your stack) and cancel it once you've confirmed you still have a shell.
Numeric vs named tables. If /etc/iproute2/rt_tables maps 51820 to a name, ip rule show prints the name, not the number. The JSON output does the same. My audit script handles both since it passes whatever it reads back to ip route show table, but grep patterns written for numbers will miss named tables.
Subnet routers amplify this. A stale rule on a laptop breaks the laptop. A stale rule on a box acting as a Tailscale subnet router breaks routing for everything behind it, and the symptom shows up on clients that have no VPN installed at all. That's where I'd run the audit script first.
Alternatives I considered. You can run each VPN in its own network namespace (or VRF) so its rules die with the namespace. That's the cleanest isolation and the right answer for anything multi-tenant. For a single host running one or two tunnels, it adds more moving parts than the problem deserves, and the systemd drop-in gets you 95% of the benefit.
Wrap-up
Policy routing is invisible plumbing. It's a lot like the DNS search-path layer I wrote about in Wildcard DNS + ndots:5: it works silently until it doesn't, and the standard diagnostic command (ip route, dig) doesn't show you the layer that's misbehaving. Get in the habit of running ip rule show, ip -6 rule show, and ip route get whenever traffic behaves differently from what the main table says it should.
Reach for this cleanup after any VPN crash or uninstall, when a host has partial connectivity that ip route can't explain, or when you inherit a box with an unclear history. Then fix it properly with a cleanup drop-in on the unit, so the next crash doesn't put you back here. If you're running into this sort of thing across a fleet rather than one box, it's the kind of Day 2 networking work I help with through GuatuLabs.
Top comments (0)