DEV Community

Cover image for Stop Mixing Tenant Routes: Practical Linux VRF with systemd-networkd
Lyra
Lyra

Posted on

Stop Mixing Tenant Routes: Practical Linux VRF with systemd-networkd

#linux #networking #systemd #devops

Stop Mixing Tenant Routes: Practical Linux VRF with systemd-networkd

You already know the failure mode: one Linux box has a management NIC, a lab VLAN, and a production uplink. Someone adds a second default route "just for testing," and suddenly SSH, monitoring, or a backup job exits through the wrong gateway. Or two tenants share overlapping RFC1918 space and your host starts answering both with the same FIB.

Linux VRF (Virtual Routing and Forwarding — specifically VRF-lite in the kernel docs) fixes that at Layer 3. Each VRF is a separate routing domain with its own table and default gateway. Interfaces are enslaved to a VRF device; connected and local routes move with them. L2 tools like LLDP keep working on the real NICs because VRF only reshapes L3 and above.

This post is a practical, declarative setup with systemd-networkd, plus the ip vrf helpers you will use day to day. It is not a bonding guide, not keepalived/VRRP, and not full network namespaces.

What you get (and what you do not)

In scope

  • Create VRF devices with dedicated FIB tables
  • Enslave NICs/VLANs into VRFs with systemd-networkd
  • Per-VRF default routes and lookups
  • Run commands inside a VRF (ip vrf exec)
  • Optional cross-VRF service accept sysctls
  • Verification and a clean rollback

Out of scope

  • NIC redundancy (bonding/LACP) — path resilience, not route isolation
  • Floating VIP HA (keepalived/VRRP) and conntrack state sync
  • Full network namespace / container isolation (you can nest VRF inside netns later)
  • Dynamic routing daemons (FRR/BIRD) beyond noting the unreachable-default metric pattern
  • Policy routing that is not tied to a VRF device (classic ip rule PBR alone)

Mental model 

+------------------+          +------------------+
|   vrf-mgmt       |          |   vrf-tenant-a   |
|   table 10       |          |   table 20       |
+--------+---------+          +--------+---------+
         |                             |
    +----+----+                   +----+----+
    | eth0    |                   | eth1.100|
    | 10.0.0.2|                   | 10.10.1.2|
    +---------+                   +---------+
 default via 10.0.0.1            default via 10.10.1.1
Enter fullscreen mode Exit fullscreen mode

Kernel behavior that matters in production:

  1. Create type vrf table <id> and bring it up.
  2. On modern kernels (4.8+), a single l3mdev FIB rule (preference 1000 by default) steers lookups for all VRFs — you usually do not hand-craft per-VRF iif/oif rules anymore.
  3. Enslave interfaces: ip link set dev eth1 master vrf-tenant-a (or VRF= in .network).
  4. Connected/local routes move into that table automatically. Extra routes that depended on the device are dropped and must be re-added into the VRF table.
  5. Processes are in the default VRF unless they bind to a VRF device (SO_BINDTODEVICE) or you launch them with ip vrf exec.

Prerequisites

  • Linux kernel with VRF / l3mdev support (any current distro kernel qualifies)
  • iproute2 with ip vrf (Debian/Ubuntu: package iproute2)
  • systemd-networkd managing the relevant interfaces
  • Root or equivalent for link/route changes
  • A maintenance window if you are moving a live management path into a VRF

Name your tables in /etc/iproute2/rt_tables.d/ so ip route show table tenant-a is readable:

sudo tee /etc/iproute2/rt_tables.d/vrf.conf >/dev/null <<'EOF'
10 mgmt
20 tenant-a
30 tenant-b
EOF
Enter fullscreen mode Exit fullscreen mode

Table IDs are yours to choose; avoid colliding with well-known ones (local/main/default). Using 10+ keeps the examples clear.

Declarative setup with systemd-networkd

1) Create the VRF devices

/etc/systemd/network/10-vrf-mgmt.netdev:

[NetDev]
Name=vrf-mgmt
Kind=vrf

[VRF]
Table=10
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/network/10-vrf-tenant-a.netdev:

[NetDev]
Name=vrf-tenant-a
Kind=vrf

[VRF]
Table=20
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/network/10-vrf-tenant-b.netdev:

[NetDev]
Name=vrf-tenant-b
Kind=vrf

[VRF]
Table=30
Enter fullscreen mode Exit fullscreen mode

Table= is compulsory for Kind=vrf — that is the FIB table the VRF owns.

Bring the VRF devices themselves up (no IPs on the VRF device in the common pattern):

/etc/systemd/network/15-vrf-mgmt.network:

[Match]
Name=vrf-mgmt

[Network]
# VRF master device: L3 domain only; addresses live on enslaved links.
ConfigureWithoutCarrier=yes
Enter fullscreen mode Exit fullscreen mode

Mirror that for vrf-tenant-a and vrf-tenant-b.

2) Enslave interfaces and configure addressing

Example: management NIC stays on eth0, tenant A rides VLAN 100 on eth1.

/etc/systemd/network/20-eth0-mgmt.network:

[Match]
Name=eth0

[Network]
VRF=vrf-mgmt
Address=10.0.0.2/24
Gateway=10.0.0.1
DNS=10.0.0.53
# Keep wait-online sane if other links are optional:
RequiredForOnline=yes
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/network/20-eth1.network (parent link, no IP — only carries VLANs):

[Match]
Name=eth1

[Network]
VLAN=eth1.100
VLAN=eth1.200
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=nearest-bridge
RequiredForOnline=no
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/network/21-eth1.100.netdev:

[NetDev]
Name=eth1.100
Kind=vlan

[VLAN]
Id=100
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/network/21-eth1.100.network:

[Match]
Name=eth1.100

[Network]
VRF=vrf-tenant-a
Address=10.10.1.2/24
Gateway=10.10.1.1
DNS=10.10.1.53
RequiredForOnline=no
Enter fullscreen mode Exit fullscreen mode

And tenant B on VLAN 200 → VRF=vrf-tenant-b, addresses in that tenant's space (even if it overlaps tenant A's RFC1918).

Key systemd facts:

  • VRF= on a .network file enslaves the matched link to that VRF master (same idea as Bond= / Bridge=).
  • Routes generated for the link (including DHCP/RA routes when used) land in the VRF table when VRF= is set; you do not need a separate RouteTable= unless you are doing something custom.
  • For kernels before 4.8, systemd's own example notes that traffic will not follow the VRF table unless extra ip rule entries exist. On anything you actually run in 2026, you are past that.

3) Optional: unreachable default as a safety net

Kernel VRF documentation recommends a high-metric unreachable default in non-management tables so a missing real default does not silently leak into another table's expectations, and so routing suites can override it:

# Equivalent one-shot (prefer declaring Gateway= on the enslaved .network when you have one)
sudo ip route replace table 20 unreachable default metric 4278198272
sudo ip route replace table 30 unreachable default metric 4278198272
Enter fullscreen mode Exit fullscreen mode

With systemd-networkd, a real Gateway= on the enslaved interface is usually enough for homelab/edge hosts. Keep the unreachable default in mind if a routing daemon owns the table.

4) Apply 

sudo networkctl reload
sudo networkctl reconfigure vrf-mgmt vrf-tenant-a vrf-tenant-b eth0 eth1 eth1.100 eth1.200
# or, if your distro prefers a full bounce for netdev creation:
# sudo systemctl restart systemd-networkd
Enter fullscreen mode Exit fullscreen mode

Verify before you trust it 

# VRF devices and table IDs
ip -d link show type vrf
ip -br link show type vrf

# Which links sit in which domain
ip -br link show vrf vrf-tenant-a
ip -br addr show vrf vrf-tenant-a

# Routes are in the VRF table, not main
ip route show vrf vrf-mgmt
ip route show vrf vrf-tenant-a
ip route show table main | head

# l3mdev rule present (modern kernels)
ip rule show | grep -E 'l3mdev|pref 1000' || ip rule show

# FIB lookup from a specific VRF
ip route get 1.1.1.1 vrf vrf-tenant-a
ip route get 1.1.1.1 vrf vrf-mgmt
Enter fullscreen mode Exit fullscreen mode

You want different egress interfaces/gateways for the two ip route get calls when destinations are reached via different defaults.

Neighbor cache is also VRF-scoped:

ip neigh show vrf vrf-tenant-a
Enter fullscreen mode Exit fullscreen mode

Run apps inside a VRF

Most programs are not VRF-aware. They open sockets in the default VRF and use the main table. That is why "ping works from the host" can still mean "tenant path is broken for the service."

One-shot commands 

# Requires cgroup v2 + appropriate capabilities (normal on systemd hosts as root)
sudo ip vrf exec vrf-tenant-a ping -c 3 10.10.1.1
sudo ip vrf exec vrf-tenant-a curl -4 -sS --max-time 5 https://example.com/ -o /dev/null -w '%{http_code}\n'
sudo ip vrf exec vrf-mgmt ssh admin@10.0.0.10 true
Enter fullscreen mode Exit fullscreen mode

Helpers from ip-vrf(8):

ip vrf show
ip vrf identify $$
sudo ip vrf pids vrf-tenant-a
Enter fullscreen mode Exit fullscreen mode

ip vrf exec associates the process (and children) with the VRF so new IPv4/IPv6 sockets inherit the domain. It only affects network-layer sockets.

systemd service bound to a VRF

For a long-running daemon, wrap the ExecStart (simple pattern) or bind the socket to the device.

Pattern A — exec wrapper (works for most off-the-shelf binaries):

# /etc/systemd/system/demo-exporter.service
[Unit]
Description=Demo exporter pinned to tenant-a VRF
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/sbin/ip vrf exec vrf-tenant-a /usr/local/bin/demo-exporter --listen 0.0.0.0:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Pattern B — socket unit with BindToDevice= (when you socket-activate or the daemon honors the systemd socket):

# /etc/systemd/system/demo-exporter.socket
[Socket]
ListenStream=9100
BindToDevice=vrf-tenant-a

[Install]
WantedBy=sockets.target
Enter fullscreen mode Exit fullscreen mode

BindToDevice= pins the listening socket to that interface/VRF device name. Prefer the pattern your app already supports; do not assume every binary behaves identically.

Cross-VRF listeners (usually leave these off)

By default, a process in the default VRF does not accept TCP/UDP connections that arrive on VRF-enslaved interfaces. That is the safe multi-tenant default: same port can exist independently per VRF.

Kernel knobs (documented in the VRF howto):

# Allow default-VRF TCP/UDP servers to accept connections from all VRFs
# Disabled by default — only enable with a clear reason.
sudo sysctl -w net.ipv4.tcp_l3mdev_accept=0
sudo sysctl -w net.ipv4.udp_l3mdev_accept=0

# RAW sockets historically default to accept=1 for compatibility (e.g. older ping).
# Tighten if you want strict VRF isolation for RAW as well:
sudo sysctl -w net.ipv4.raw_l3mdev_accept=0
Enter fullscreen mode Exit fullscreen mode

Persist under /etc/sysctl.d/90-vrf.conf only after you decide the policy:

net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.udp_l3mdev_accept = 0
net.ipv4.raw_l3mdev_accept = 0
# Optional when enslaving interfaces that already hold global IPv6 addresses:
# net.ipv6.conf.all.keep_addr_on_down = 1
Enter fullscreen mode Exit fullscreen mode

If you set tcp_l3mdev_accept=1, the kernel docs warn that it can be unspecified whether a VRF-bound listener or a default-VRF listener wins for new VRF traffic — ugly if you rely on per-VRF TCP MD5 or different socket options. Prefer explicit ip vrf exec / bind-to-device services over global accept.

Netfilter and capture notes

  • You can attach nftables/iptables and tc rules to the VRF device to match the whole domain.
  • tcpdump -i vrf-tenant-a sees traffic that enters/leaves the VRF domain as a whole; forwarded packets that never traverse the VRF netdev path may not show up (kernel docs call this out).
  • For conntrack-heavy firewalls, remember VRF changes which table owns the route; it does not replace state sync across HA pairs.

Controlled isolation test

On a host with two VRFs and distinct defaults:

# 1) Confirm main table is not holding tenant defaults
ip route show table main

# 2) Lookups differ by VRF
ip route get 8.8.8.8 vrf vrf-mgmt
ip route get 8.8.8.8 vrf vrf-tenant-a

# 3) Data plane from each domain
sudo ip vrf exec vrf-mgmt    ping -c 2 -W 2 8.8.8.8
sudo ip vrf exec vrf-tenant-a ping -c 2 -W 2 8.8.8.8

# 4) Negative test: without vrf exec, traffic uses default VRF only
ping -c 2 -W 2 8.8.8.8
Enter fullscreen mode Exit fullscreen mode

If step 3 works per domain and step 2 shows different dev/via, your isolation is real. If both VRFs somehow share one gateway you did not configure, stop and re-check enslavement (ip link show master vrf-tenant-a).

Operational pitfalls

  1. Moving the only management path into a VRF without ip vrf exec

    Your existing SSH session may survive until restart; new admin tools on the box will not reach mgmt routes unless they run in vrf-mgmt. Keep a console/IPMI plan.

  2. Forgetting to re-add static routes after enslavement

    Non-connected routes that pointed at the device are dropped on master assignment. Reinstall them with table <id> / vrf <name> or declare them in the .network file.

  3. Overlapping tenant IP space without VRF

    Overlap is a reason to use VRF. Without it, the FIB cannot disambiguate. With it, still be careful with services that bind *: in the default VRF and *_l3mdev_accept=1.

  4. DHCP on enslaved links

    Works with networkd; learned routes go to the VRF table when VRF= is set. Verify with ip route show vrf … after lease acquisition.

  5. Mixing VRF with ad-hoc ip rule PBR

    Higher-priority policy rules can override VRF l3mdev steering. That is powerful and easy to make undebuggable — document any extra rules next to the VRF config.

  6. cgroup v2 requirement for ip vrf exec

    The man page requires cgroup v2 (normal on current systemd). If exec fails, check that before blaming the VRF device.

Rollback 

# Temporary: free a NIC from its VRF
sudo ip link set dev eth1.100 nomaster

# Persistent: remove VRF= lines / .netdev files, then
sudo networkctl reload
sudo systemctl restart systemd-networkd

# Remove unused VRF devices (after no slaves remain)
sudo ip link delete dev vrf-tenant-a
Enter fullscreen mode Exit fullscreen mode

Connected routes return to the main/local tables when you detach with nomaster.

Minimal lab (two VRFs, no spare physical NICs)

If you only have one NIC, practice with dummy interfaces or veth pairs before touching production uplinks:

sudo ip link add dummy-a type dummy
sudo ip link add dummy-b type dummy
sudo ip link add vrf-a type vrf table 110
sudo ip link add vrf-b type vrf table 120
sudo ip link set vrf-a up
sudo ip link set vrf-b up
sudo ip link set dummy-a up master vrf-a
sudo ip link set dummy-b up master vrf-b
sudo ip addr add 192.0.2.1/24 dev dummy-a
sudo ip addr add 198.51.100.1/24 dev dummy-b
ip route show vrf vrf-a
ip route show vrf vrf-b
# cleanup
sudo ip link del dummy-a; sudo ip link del dummy-b
sudo ip link del vrf-a; sudo ip link del vrf-b
Enter fullscreen mode Exit fullscreen mode

Then promote the same shape into systemd-networkd units.

When to choose something else

Problem Better tool
Cable/NIC failure on one host Bonding/LACP
Host failure for a shared VIP keepalived/VRRP (+ conntrackd if stateful FW)
Full process + interface isolation network namespaces / containers
Encrypt a path WireGuard/IPsec (can still sit inside a VRF)
L2 segmentation only VLANs/bridges

VRF shines when one kernel must hold multiple independent L3 realities — multi-tenant edge, separate mgmt plane, overlapping lab/prod prefixes — without standing up a VM per table.

References

  • Linux kernel documentation: Virtual Routing and Forwarding (VRF) (also plain-text howto on kernel.org)
  • ip-vrf(8) — show, exec, identify, pids (iproute2)
  • ip-link(8) — type vrf, master / vrf enslavement
  • systemd.netdev(5) — Kind=vrf, [VRF] Table=
  • systemd.network(5) — VRF=, examples for enslaving links into a VRF
  • sysctl policy knobs: net.ipv4.tcp_l3mdev_accept, udp_l3mdev_accept, raw_l3mdev_accept, net.ipv6.conf.all.keep_addr_on_down

Ship the units, verify with ip route get … vrf …, and wrap anything that must speak tenant routes in ip vrf exec or a device-bound socket. Once the FIB stops lying to you about which gateway is "default," the rest of the host gets a lot quieter.

Top comments (0)