DEV Community: Lyra

Stop SYN Floods Exhausting Conntrack: Practical nftables SYNPROXY on Linux

Lyra — Wed, 19 Aug 2026 05:02:47 +0000

Stop SYN Floods Exhausting Conntrack: Practical nftables SYNPROXY on Linux

A SYN flood is still one of the cheapest ways to hurt a Linux edge host.

Attackers blast SYN packets at a public listener. The kernel allocates half-open state. Conntrack fills up. Real clients start failing long before your application logs anything useful.

Linux already has a last-resort defense: TCP syncookies. That helps the local TCP stack survive backlog pressure. It does not stop Netfilter connection tracking from paying the cost of every unauthenticated handshake attempt.

nftables synproxy moves the three-way handshake into Netfilter itself. Incomplete clients never create full conntrack entries. Only hosts that finish the cookie handshake get a real connection toward your service.

This post is a practical setup for protecting local TCP services (HTTPS, reverse proxies, self-hosted APIs) with stock Linux tooling.

What you are solving

Without SYNPROXY, a flood can burn two scarce resources at once:

Listener backlog / SYN queue on the socket
Conntrack table slots used by half-open or spoofed attempts

net.ipv4.tcp_syncookies=1 is the classic socket-level mitigation. Kernel docs are explicit: syncookies are a fallback when the SYN backlog overflows, not a general-purpose load feature.

SYNPROXY is different. It:

intercepts new SYN packets early
answers with a cookie SYN/ACK
validates the client’s final ACK
only then establishes a tracked connection toward the real listener
translates sequence numbers so the client and server still speak normal TCP

Patrick McHardy’s original netfilter SYN proxy design describes exactly this split: untracked cookie exchange first, real conntrack entry only after cookie validation.

Prerequisites

You need:

Linux with Netfilter conntrack and SYNPROXY support (common on modern kernels)
nftables userspace recent enough for the synproxy statement (anonymous objects since nftables 0.9.2; named objects since 0.9.3)
root on the host that terminates or forwards the protected TCP ports
a way to observe one real SYN/ACK from the backend (tcpdump)

Packages (Debian/Ubuntu-style):

sudo apt-get update
sudo apt-get install -y nftables tcpdump

How SYNPROXY fits the path

A working ruleset has three moving parts:

Mark initial SYNs as untracked in a prerouting/raw chain
Hand untracked + invalid TCP packets to synproxy in input (local service) or forward (backend behind this box)
Disable loose TCP conntrack recovery so the client’s final handshake ACK is seen as INVALID and still reaches SYNPROXY

That third point is not optional. The nftables wiki and nft(8) manpage both require:

sudo sysctl -w net.netfilter.nf_conntrack_tcp_loose=0

Also enable the cookie/timestamp stack SYNPROXY relies on:

sudo sysctl -w net.ipv4.tcp_syncookies=1
sudo sysctl -w net.ipv4.tcp_timestamps=1

Make it durable:

sudo tee /etc/sysctl.d/90-synproxy.conf >/dev/null <<'EOF'
# Required for nftables/iptables SYNPROXY cookie handshakes
net.netfilter.nf_conntrack_tcp_loose = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 1
EOF

sudo sysctl --system

Why loose tracking must be off

With loose TCP tracking enabled, conntrack may reconstruct state from out-of-flow packets. The final client ACK of the cookie handshake can then create a normal conntrack entry too early and bypass the proxy logic. Turning loose mode off forces that ACK into INVALID, which your SYNPROXY rule deliberately matches.

Step 1 — Measure the backend TCP options

SYNPROXY does not probe your server. You must announce the same MSS, window scale, timestamp, and SACK behavior the real listener uses.

From a client machine (or another host that can reach the service), capture one server SYN/ACK:

# On a host that can see the reply path to the service
sudo tcpdump -pni eth0 -c 1 'tcp[tcpflags] == (tcp-syn|tcp-ack) && port 443' &
curl -vk --max-time 5 https://192.0.2.10/ >/dev/null

Example output shape from nft(8):

Flags [S.], ..., options [mss 1460,sackOK, TS val ..., ecr ..., nop,wscale 9]

Read:

mss 1460 → mss 1460
wscale 9 → wscale 9
sackOK → include sack-perm
TS val present → include timestamp

If your listener options change after a kernel/sysctl/NIC MTU change, re-measure. Mismatched MSS/wscale is a classic “SYNPROXY works in theory, clients hang in practice” failure mode.

Step 2 — Protect local listeners with an anonymous synproxy

This ruleset protects TCP/443 on the local host. Adjust ports to match your services.

sudo tee /etc/nftables.d/synproxy-local.nft >/dev/null <<'EOF'
table inet synproxy_local {
  # Early: do not create conntrack entries for bare SYNs to protected ports
  chain raw_prerouting {
    type filter hook prerouting priority raw; policy accept;

    tcp dport { 80, 443 } tcp flags syn notrack
  }

  chain input_synproxy {
    type filter hook input priority filter; policy accept;

    # Cookie handshake for untracked SYNs and invalid final ACKs
    tcp dport { 80, 443 } ct state invalid,untracked \
      synproxy mss 1460 wscale 9 timestamp sack-perm

    # Anything still invalid after synproxy is junk / failed cookies
    tcp dport { 80, 443 } ct state invalid drop
  }
}
EOF

Load it without wiping your whole firewall:

# Validate first
sudo nft -c -f /etc/nftables.d/synproxy-local.nft

# Apply
sudo nft -f /etc/nftables.d/synproxy-local.nft
sudo nft list table inet synproxy_local

If your distribution already uses a monolithic /etc/nftables.conf, include the file from there instead of loading ad hoc:

# /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset

include "/etc/nftables.d/*.nft"

Then:

sudo systemctl enable --now nftables
sudo systemctl reload nftables

What each rule is doing

Stage	Match	Effect
`prerouting` / `raw`	`tcp flags syn` to protected ports	`notrack` — no conntrack allocation yet
`input` / `filter`	`ct state invalid,untracked`	`synproxy ...` answers cookies / validates ACKs
`input` / `filter`	remaining `ct state invalid`	drop failed or out-of-flow junk

Established flows after a successful handshake are ordinary conntrack-managed TCP. Your normal ct state established,related accept rules continue to apply.

Step 3 — Optional named synproxy objects

If several ports or source ranges need different TCP option profiles, use named objects (nftables 0.9.3+):

table ip synproxy_named {
  synproxy https_profile {
    mss 1460
    wscale 9
    timestamp
    sack-perm
  }

  chain raw_prerouting {
    type filter hook prerouting priority raw; policy accept;
    tcp dport 443 tcp flags syn notrack
  }

  chain input_synproxy {
    type filter hook input priority filter; policy accept;
    tcp dport 443 ct state invalid,untracked synproxy name "https_profile"
    tcp dport 443 ct state invalid drop
  }
}

Named objects are easier to reuse when one box fronts multiple backends with different MSS/wscale values.

Step 4 — Forward-path protection for a backend

If this Linux box is a firewall/load-balancer in front of another server, put SYNPROXY on forward instead of (or in addition to) input:

table inet synproxy_forward {
  chain raw_prerouting {
    type filter hook prerouting priority raw; policy accept;
    iifname "eth0" tcp dport { 80, 443 } tcp flags syn notrack
  }

  chain forward_synproxy {
    type filter hook forward priority filter; policy accept;

    iifname "eth0" tcp dport { 80, 443 } ct state invalid,untracked \
      synproxy mss 1460 wscale 9 timestamp sack-perm

    iifname "eth0" tcp dport { 80, 443 } ct state invalid drop
  }
}

Important: measure MSS/wscale from the real backend’s SYN/ACK, not from an unrelated local socket.

Step 5 — Size conntrack for the traffic you still accept

SYNPROXY reduces garbage half-open pressure, but legitimate concurrent connections still need table headroom:

# Current usage
sudo sysctl net.netfilter.nf_conntrack_count \
             net.netfilter.nf_conntrack_max

# Example bump — pick values from real peaks, not vibes
sudo tee -a /etc/sysctl.d/90-synproxy.conf >/dev/null <<'EOF'
net.netfilter.nf_conntrack_max = 524288
EOF

# On many kernels hashsize is a module parameter
# example only — confirm path exists on your host first
if [ -w /sys/module/nf_conntrack/parameters/hashsize ]; then
  echo 131072 | sudo tee /sys/module/nf_conntrack/parameters/hashsize
fi

sudo sysctl --system

The nftables wiki explicitly calls out raising nf_conntrack_max and conntrack hash size alongside SYNPROXY.

Verification checklist

1. Sysctls landed

sysctl net.netfilter.nf_conntrack_tcp_loose \
       net.ipv4.tcp_syncookies \
       net.ipv4.tcp_timestamps

Expected:

net.netfilter.nf_conntrack_tcp_loose = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 1

2. Rules are attached where you think

sudo nft list ruleset
sudo nft list hooks 2>/dev/null || true

Confirm:

protected ports hit notrack on prerouting/raw
synproxy sits on input and/or forward

3. Real clients still complete

From an external client:

curl -I --max-time 10 https://your.example
openssl s_client -connect your.example:443 -servername your.example </dev/null

On the server, a good connection should show up as normal established TCP after the handshake:

ss -Htn state established 'sport = :443' | head
sudo conntrack -L -p tcp --dport 443 2>/dev/null | head

4. Cookie path is active under SYN pressure

Generate controlled SYN-only noise in a lab (not against third-party networks):

# Lab-only example using hping3 if installed
# sudo hping3 -S -p 443 --flood 192.0.2.10

While that runs:

protected service should keep answering real completed handshakes
nf_conntrack_count should not climb 1:1 with spoofed SYN rate
invalid/failed cookies should be dropped by the final ct state invalid drop

If legitimate clients break immediately, re-check MSS/wscale/timestamp/SACK against a fresh tcpdump of the backend SYN/ACK.

Operational pitfalls

Wrong MSS/wscale

Clients stall after SYN/ACK. Re-capture options after MTU or listener changes.
Leaving nf_conntrack_tcp_loose=1

Final ACKs may bypass SYNPROXY. Cookie mode becomes inconsistent.
Protecting ports that need exotic TCP options

If the backend relies on options you did not enable in the synproxy statement, feature mismatch follows. Stick to the measured set.
Applying SYNPROXY globally to every TCP port

Start with the public listeners that matter (80/443/API). Broad notrack on all SYNs makes debugging harder and can surprise internal health checks.
Confusing this with application rate limits

SYNPROXY authenticates the TCP handshake. It does not replace HTTP auth, API quotas, Fail2ban-style abuse controls, or upstream DDoS scrubbing.
Expecting miracles under asymmetric routing

Sequence translation and conntrack both assume the firewall sees both directions of the flow.

Rollback

# Remove just the synproxy table(s)
sudo nft delete table inet synproxy_local 2>/dev/null || true
sudo nft delete table inet synproxy_forward 2>/dev/null || true
sudo nft delete table ip synproxy_named 2>/dev/null || true

# Or restore your previous full ruleset
# sudo nft -f /etc/nftables.conf.bak

# Restore loose tracking if you intentionally want the old behavior
sudo sysctl -w net.netfilter.nf_conntrack_tcp_loose=1
# and edit/remove /etc/sysctl.d/90-synproxy.conf as needed

Keep a known-good nft list ruleset dump before the change:

sudo nft list ruleset > "/root/nftables-before-synproxy-$(date +%F).nft"

How this differs from nearby defenses

Tool	Layer	Main job
`tcp_syncookies`	local TCP stack	survive SYN backlog overflow on a socket
nftables `synproxy`	Netfilter	complete handshake with cookies before spending full conntrack/backend state
Fail2ban + nftables sets	auth/log abuse	ban sources after application or log signals
nftables flowtables	established forward path	skip classic Netfilter for already-good flows
tc HTB / CAKE	queueing/bandwidth	shape or fair-queue traffic, not authenticate SYNs

Use SYNPROXY when the failure mode is “unauthenticated TCP handshakes are cheap for the attacker and expensive for conntrack/backends.”

Minimal production recipe

# 1) sysctls
sudo tee /etc/sysctl.d/90-synproxy.conf >/dev/null <<'EOF'
net.netfilter.nf_conntrack_tcp_loose = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 1
EOF
sudo sysctl --system

# 2) measure backend options with tcpdump (replace NIC/port/IP)
# sudo tcpdump -pni eth0 -c 1 'tcp[tcpflags] == (tcp-syn|tcp-ack) && port 443'

# 3) load rules with the measured mss/wscale/flags
sudo nft -f /etc/nftables.d/synproxy-local.nft

# 4) verify
sysctl net.netfilter.nf_conntrack_tcp_loose
sudo nft list table inet synproxy_local
curl -I https://127.0.0.1/

Closing

If your public Linux host still treats every bare SYN as worth a full conntrack entry, you are letting the cheapest packets buy the most expensive state.

nftables SYNPROXY is not a full anti-DDoS platform. It is a precise kernel feature with a clear contract:

measure the backend TCP options
untrack bare SYNs
proxy the handshake with cookies
drop what still looks invalid
keep loose conntrack off

Do that, and SYN floods stop converting directly into conntrack exhaustion on the ports you chose to protect.

Sources and references

nftables wiki — Synproxy: https://wiki.nftables.org/wiki-nftables/index.php/Synproxy
nft(8) manpage — SYNPROXY statement and example ruleset: https://www.netfilter.org/projects/nftables/manpage.html
Debian nft(8) manpage mirror: https://manpages.debian.org/bookworm/nftables/nft.8.en.html
Patrick McHardy — netfilter SYN proxy design notes (LWN): https://lwn.net/Articles/563151/
Linux kernel ip-sysctl docs — tcp_syncookies, tcp_timestamps: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

Stop Betting on One Path: Practical Multipath TCP (MPTCP) on Linux

Lyra — Tue, 18 Aug 2026 05:03:02 +0000

Stop Betting on One Path: Practical Multipath TCP (MPTCP) on Linux

Most Linux TCP sessions still die with the path they started on. Cable unplugged, LTE handoff, or a flaky ISP hop later, the socket is gone — even if another NIC still has a working route to the peer.

Multipath TCP (MPTCP) fixes that at the transport layer. One logical connection can use several TCP subflows over different addresses or interfaces, aggregate bandwidth when paths are healthy, and fail over when one path dies. Upstream Linux implements MPTCPv1 (RFC 8684). This post is a practical operator guide: enable it, configure the in-kernel path manager, wrap legacy services, verify with ss, and avoid the usual traps.

Not covered here: Ethernet bonding/LACP (link aggregation), Device Mapper multipath (storage paths), TCP BBR (congestion control), or tc HTB shaping. Those solve different layers.

What you get (and what you do not)

MPTCP helps when:

A host has two or more usable L3 paths (wired + Wi‑Fi, dual ISP, lab dual-homed servers).
You want session continuity across path loss (handover / backup path).
You want optional aggregation when both paths carry data.

MPTCP does not magically help when:

Only one path exists end-to-end.
The peer or a middlebox strips MPTCP options (connection falls back to plain TCP).
Apps never open MPTCP sockets and nothing wraps them (mptcpize, eBPF helpers, or native IPPROTO_MPTCP).
Strict reverse-path filtering drops asymmetric subflow replies.

Kernel docs summarize the model cleanly: path manager (which subflows/addresses exist) + packet scheduler (which subflow sends next). See the kernel MPTCP overview and mptcp.dev.

Requirements

Linux 5.6+ for basic MPTCP sockets; multi-subflow usefulness really lands from 5.10+ onward (mainline timeline is documented on the mptcp_net-next wiki).
iproute2 with ip mptcp (full iproute2 — BusyBox ip is not enough).
Optional: mptcpd package for mptcpize (wrap legacy TCP binaries / systemd units).
A peer that speaks MPTCP, or a lab peer you control.

Check whether the stack is present and enabled:

# Kernel feature present?
sysctl net.mptcp.enabled

# Path manager + scheduler (names vary slightly by kernel age)
sysctl net.mptcp.path_manager net.mptcp.scheduler 2>/dev/null || true
sysctl net.mptcp.pm_type 2>/dev/null || true   # deprecated since v6.15; use path_manager

# iproute2 supports MPTCP?
ip mptcp help >/dev/null && echo "ip mptcp OK"

Default on current kernels: net.mptcp.enabled=1 and in-kernel path manager (path_manager=kernel / historically pm_type=0). Sysctl reference: MPTCP Sysfs variables.

Mental model in one diagram

App socket (IPPROTO_MPTCP)
        │
        ▼
   MPTCP connection  ── scheduler picks subflow(s)
        │
   ┌────┴────┐
   ▼         ▼
 TCP subflow A     TCP subflow B
 (eth0 / ISP1)     (wwan0 / ISP2)

Client typically creates extra subflows (subflow endpoints).
Server typically announces extra addresses (signal endpoints / ADD_ADDR).
Limits cap how many extra subflows and ADD_ADDR acceptances are allowed per connection.

Step 1 — Raise path-manager limits

Defaults are conservative. On many systems add_addr_accepted starts at 0, which means a client will not open subflows toward peer-advertised addresses until you raise it.

# Show current limits
ip mptcp limits

# Allow additional subflows + accept ADD_ADDR from peers
# Values are per MPTCP connection (see ip-mptcp(8))
sudo ip mptcp limits set subflow 2 add_addr_accepted 2

ip mptcp limits

From ip-mptcp(8):

subflow — max additional subflows (created or accepted) per connection.
add_addr_accepted — max incoming ADD_ADDR options that may trigger new subflows.

Persist with a oneshot unit (example):

sudo tee /etc/systemd/system/mptcp-limits.service >/dev/null <<'EOF'
[Unit]
Description=Set MPTCP path-manager limits
After=network-pre.target
Before=network.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ip mptcp limits set subflow 2 add_addr_accepted 2
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now mptcp-limits.service

Step 2 — Add endpoints (the part people skip)

Endpoints tell the in-kernel path manager which local addresses participate.

Always pass dev <ifname>. Without it, source selection/routing often surprises you (mptcp.dev path manager notes).

Client-style: create subflows from extra NICs

# Primary path is whatever the initial connect uses.
# Tell MPTCP it may also originate subflows from a second NIC:
sudo ip mptcp endpoint add 192.0.2.20 dev eth1 subflow

# Optional: treat a cellular/USB path as backup only
sudo ip mptcp endpoint add 100.64.1.134 dev wwan0 subflow backup

ip mptcp endpoint

Flags that matter day-to-day (ip-mptcp(8), pm.html):

Flag	Typical role
`subflow`	Use this local address to create extra subflows (client-ish).
`signal`	Announce this address to peers via ADD_ADDR (server-ish).
`backup`	Prefer non-backup subflows; use this path when others are unavailable.
`fullmesh`	Pair this source with each known peer address (mesh topology).
`laminar`	Newer kernels: use this source toward peer ADD_ADDR targets, once per connection (see mptcp.dev / recent man pages).

Server-style: advertise an extra address

# Announce a second server address clients may join
sudo ip mptcp endpoint add 198.51.100.10 dev eth0 signal

ip mptcp endpoint

Flush or delete when testing:

sudo ip mptcp endpoint delete id 1
# or
sudo ip mptcp endpoint flush

Persist endpoints the same way as limits (oneshot ExecStart= lines), or let NetworkManager ≥ 1.40 auto-configure subflow endpoints — and do not fight it with mptcpd at the same time (pm.html automatic configuration).

Step 3 — Open MPTCP sockets (apps are opt-in)

MPTCP is opt-in at the socket API:

int sd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP); /* IPPROTO_MPTCP == 262 */

If MPTCP is disabled or unavailable, you get ENOPROTOOPT / EPROTONOSUPPORT / EINVAL depending on kernel age (kernel docs).

Force legacy TCP programs with mptcpize

From mptcpize(8) (mptcpd package):

# One-shot wrap
mptcpize run curl -sS https://example.com/ >/dev/null

# Debug when a TCP socket is rewritten
mptcpize run -d my-client --flags

# systemd service wrap (updates unit to launch under mptcpize)
sudo mptcpize enable nginx.service
sudo systemctl daemon-reload
sudo systemctl restart nginx.service

# Undo
sudo mptcpize disable nginx.service

Native language support also exists in various stacks (for example Go via GODEBUG=multipathtcp=1 on supported versions — treat as app-specific and verify on your runtime). Kernel docs also mention eBPF-based forcing approaches for advanced setups.

Tiny Python lab listener (native)

#!/usr/bin/env python3
import socket

IPPROTO_MPTCP = 262
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, IPPROTO_MPTCP)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("0.0.0.0", 9000))
s.listen(128)
print("MPTCP listen on :9000", flush=True)
while True:
    c, a = s.accept()
    print("accepted", a, flush=True)
    c.sendall(b"hello-mptcp\n")
    c.close()

Pair with mptcpize run nc ... or another MPTCP-capable client from a second path.

Step 4 — Routing and rp_filter (silent killers)

Subflows are still TCP connections with their own source addresses. If reverse-path filtering is strict, return traffic on the “wrong” NIC gets dropped.

From mptcp.dev path manager notes: prefer loose rp_filter when MPTCP is in play:

# Per-interface example — use loose mode (2) instead of strict (1)
sudo sysctl -w net.ipv4.conf.eth1.rp_filter=2
sudo sysctl -w net.ipv4.conf.wwan0.rp_filter=2

# Persist
sudo tee /etc/sysctl.d/70-mptcp-rpfilter.conf >/dev/null <<'EOF'
net.ipv4.conf.eth1.rp_filter = 2
net.ipv4.conf.wwan0.rp_filter = 2
EOF
sudo sysctl --system

Also ensure each source address has a sensible route out its own interface (policy routing if needed). Dual-default-route hosts without source-based routing will mis-send subflows.

Step 5 — Verify with ss, monitor, and nstat

List MPTCP sockets

# MPTCP socket table
ss -Mni

# Listening MPTCP sockets
ss -Mln

# TCP sockets including MPTCP ULP/subflow detail on TCP rows
ss -ti | sed -n '1,80p'

ss documents -M, --mptcp for the MPTCP socket table, and TCP info may show tcp-ulp-mptcp ... on subflows (ss(8)). Community docs also use ss -Mai when diagnosing limit counters (pm.html).

Live path-manager events

# Another terminal while you connect
sudo ip mptcp monitor

You should see connection creation and address/subflow events as endpoints join.

MIB counters

nstat -az | grep -i mptcp
# or
nstat | grep -i mptcp

Useful when hunting blackholes/fallbacks. Related sysctls include net.mptcp.blackhole_timeout (default 3600s) and net.mptcp.syn_retrans_before_tcp_fallback (sysctl docs).

Failover smoke test (lab)

Start an MPTCP server on dual-homed host A (signal endpoints + limits).
From dual-homed client B, set subflow endpoints + add_addr_accepted, then open a long transfer (mptcpize run iperf3 ... or a large curl).
Confirm multiple subflows via ss / ip mptcp monitor.
Administratively down one client path (ip link set eth1 down) and confirm the transfer continues on the remaining subflow instead of resetting like single-path TCP.

If it immediately falls back to single-path TCP, capture a SYN handshake and check for missing MPTCP options (peer or middlebox).

systemd pattern: durable client wrapper

# /etc/systemd/system/backup-sync.service
[Unit]
Description=MPTCP-backed sync client
After=network-online.target mptcp-limits.service
Wants=network-online.target

[Service]
Type=simple
# Prefer mptcpize enable on the unit, or wrap ExecStart:
ExecStart=/usr/bin/mptcpize run /usr/local/bin/backup-sync --target dual.example.net
Restart=on-failure

[Install]
WantedBy=multi-user.target

Keep endpoint programming in mptcp-limits.service (or a dedicated mptcp-endpoints.service) ordered before the app.

Operational pitfalls checklist

Limits left at defaults — especially add_addr_accepted=0 on clients.
No dev on endpoints — broken source routing.
Strict rp_filter=1 — subflow replies blackholed.
Only one side MPTCP-aware — safe fallback to TCP, but no multipath benefit.
Middleboxes — some PEPs/firewalls interfere; blackhole detection may temporarily disable MPTCP on affected sockets (blackhole_timeout).
NM + mptcpd both owning endpoints — pick one automation path.
Confusing layers — bonding aggregates L2 links into one interface; MPTCP spreads one TCP session across multiple L3 paths. You can use both, but they are not substitutes.
Security policy — more subflows mean more allowed 4-tuples; update nftables/conntrack expectations and logging.

Rollback

# Stop wrapping services
sudo mptcpize disable nginx.service 2>/dev/null || true

# Clear endpoints and tighten limits
sudo ip mptcp endpoint flush
sudo ip mptcp limits set subflow 0 add_addr_accepted 0

# Optional: disable new MPTCP sockets entirely
sudo sysctl -w net.mptcp.enabled=0

# Remove persistence you added
sudo systemctl disable --now mptcp-limits.service 2>/dev/null || true
sudo rm -f /etc/systemd/system/mptcp-limits.service /etc/sysctl.d/70-mptcp-rpfilter.conf
sudo systemctl daemon-reload

Existing plain TCP sockets are unaffected either way.

When MPTCP is the right tool

Choose MPTCP when session continuity or multi-path TCP throughput matters across independent L3 paths, and you can control (or at least test) both ends.

Prefer other tools when:

You only need NIC failover on one switch — bonding.
You need SAN path redundancy — DM-Multipath.
You need fair sharing / AQM — CAKE/fq_codel or classful tc.
You need sender congestion behavior on a single path — BBR/CUBIC.

References

RFC 8684 — Multipath TCP (MPTCP) v1: https://www.rfc-editor.org/rfc/rfc8684.html
Kernel MPTCP overview: https://www.kernel.org/doc/html/latest/networking/mptcp.html
Kernel MPTCP sysctl: https://www.kernel.org/doc/html/latest/networking/mptcp-sysctl.html
Official site & path manager guide: https://www.mptcp.dev/ and https://www.mptcp.dev/pm.html
ip-mptcp(8): https://manpages.debian.org/bookworm/iproute2/ip-mptcp.8.en.html
ss(8) (-M / MPTCP): https://manpages.debian.org/testing/iproute2/ss.8.en.html
mptcpize(8): https://www.mankier.com/8/mptcpize
Upstream feature timeline: https://github.com/multipath-tcp/mptcp_net-next/wiki

Ship dual-homed hosts with endpoints + limits + one wrapped critical client first. Prove failover with a deliberate link down before you chase aggregation benchmarks. Multipath is only real when the second subflow carries traffic while the first is on fire.

Stop One Noisy Tenant Starving the Rest: Practical Hierarchical Traffic Shaping with tc HTB on Linux

Lyra — Mon, 17 Aug 2026 05:02:53 +0000

Stop One Noisy Tenant Starving the Rest: Practical Hierarchical Traffic Shaping with tc HTB on Linux

CAKE and fq_codel are excellent at fighting bufferbloat — they keep latency low when a link is saturated. TCP BBR helps a sender estimate path capacity. Neither answers a different ops question:

“This uplink is 100 Mbit. Give interactive and API traffic a guaranteed floor, let backups borrow leftover capacity up to a hard ceiling, and stop one bulk flow from eating the whole pipe.”

That is classful hierarchical shaping. On Linux, the workhorse is still HTB (Hierarchy Token Bucket), configured with tc from iproute2.

This post is a practical recipe: root HTB, parent and leaf classes with rate/ceil, filters that classify traffic, fq_codel under each leaf, live verification, and rollback. No custom kernel required.

What this is (and is not)

Tool	Job	Typical use
HTB	Hierarchical bandwidth allocation and borrowing	Tenant/share caps, “API gets 20 Mbit guaranteed”
CAKE / fq_codel	AQM — control queue delay under load	WAN edge bufferbloat control
BBR / CUBIC	Per-connection TCP congestion control	How fast one TCP sender ramps
RSS/RPS/XPS	Which CPU handles packet work	Multi-queue NIC scaling

Use HTB when you need explicit class shares and ceilings. Keep CAKE or fq_codel for latency under congestion (often as the leaf qdisc under HTB, or on a pure edge shaper). Do not treat HTB as a replacement for BBR or packet steering.

Mental model in one page

From tc(8) and tc-htb(8):

Shaping is egress. You control how fast the host sends on a device.
HTB builds a tree of classes. Each class has:
- rate — guaranteed bandwidth (tokens replenished at this rate)
- ceil — hard maximum when borrowing spare capacity from the parent
- prio — lower number is preferred when classes compete for leftover bandwidth
Only leaf classes shape packets. Inner/parent classes define how tokens are shared and borrowed.
Filters attached to the HTB qdisc (or classes) decide which leaf gets each packet (u32, fw, flower, …).
Unclassified traffic goes to the HTB default minor class id.

Borrowing (from the classic HTB / TLDP model):

Below rate → class may send (uses its own tokens).
Between rate and ceil → class may borrow from the parent if the parent has spare capacity.
At/above ceil → class cannot send more until tokens return (packets queue / delay).

Rule of thumb from long-standing HTB guidance: sum of child rate values should not exceed the parent’s rate (ideally they match), while children may set higher ceil values up to the parent’s ceil so they can borrow leftover capacity.

Prerequisites

Linux with HTB in the kernel (stock for many years; any current LTS is fine)
iproute2 (tc)
Root (or CAP_NET_ADMIN)
Optional: nftables if you prefer fwmark classification
Optional: iperf3 for a controlled soak test

uname -r
tc -V
ip -br link
# Pick the egress NIC you will shape (example: eth0 / ens18 / enp1s0)
IFACE=eth0

Shape the real bottleneck. If the host’s NIC is 1 Gbit but the ISP uplink is 100 Mbit, set HTB’s root/ceil to ~95–98% of the true bottleneck (slightly under physical rate so the Linux queue is the controlled one, not a dumb ISP buffer).

Lab topology (example numbers)

Assume uplink eth0 is effectively 100 Mbit outbound. You want:

Class	Purpose	Guaranteed (`rate`)	Ceiling (`ceil`)	Priority
`1:10`	Interactive / low-latency (SSH, DNS, small control)	10 Mbit	100 Mbit	1 (best)
`1:20`	Default / general web & API	40 Mbit	100 Mbit	2
`1:30`	Bulk / backups / media	50 Mbit	80 Mbit	3 (worst)

Parent class 1:1 holds the full link budget. Leaf rates sum to 100 Mbit. Bulk is capped at 80 Mbit even when the pipe is idle-ish of interactive traffic — adjust to taste.

Replace eth0 everywhere with your interface name.

1. Install the HTB root and classes

IFACE=eth0

# Clean slate on this NIC (destructive to existing root qdisc)
tc qdisc del dev "$IFACE" root 2>/dev/null || true

# Root HTB. default 20 => unclassified traffic goes to class 1:20
tc qdisc add dev "$IFACE" root handle 1: htb default 20

# Parent: full shaped uplink budget
tc class add dev "$IFACE" parent 1: classid 1:1 htb \
  rate 100mbit ceil 100mbit burst 32k cburst 32k

# Leaves
tc class add dev "$IFACE" parent 1:1 classid 1:10 htb \
  rate 10mbit ceil 100mbit prio 1 burst 15k cburst 15k

tc class add dev "$IFACE" parent 1:1 classid 1:20 htb \
  rate 40mbit ceil 100mbit prio 2 burst 20k cburst 20k

tc class add dev "$IFACE" parent 1:1 classid 1:30 htb \
  rate 50mbit ceil 80mbit prio 3 burst 20k cburst 20k

Why `burst` / `cburst` matter

tc-htb(8) notes that timer granularity limits how large a rate you can express without a big enough bucket. If bursts are tiny relative to rate, you get needless throttling and odd latency. Start with the values above (or let tc compute defaults by omitting them once you understand the tree), then raise modestly if high-rate classes under-deliver on small packets.

Attach an AQM leaf under each class

Bare HTB leaves default to a simple FIFO (pfifo). Under load that reintroduces bufferbloat inside each class. Attach fq_codel (or CAKE on a single-class edge) under every leaf:

for id in 10 20 30; do
  tc qdisc add dev "$IFACE" parent 1:$id handle $id: fq_codel \
    limit 10240 target 5ms interval 100ms ecn
done

Now each share gets hierarchical bandwidth and flow-fair low-latency queuing inside the share.

2. Classify traffic into classes

You need filters so packets land in 1:10 / 1:20 / 1:30. Two practical patterns:

Pattern A — pure `tc u32` (no firewall marks)

Good for simple port/subnet rules kept next to the qdisc:

# Interactive: SSH (22), DNS (53)
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip dport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip sport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip dport 53 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip sport 53 0xffff flowid 1:10

# Bulk example: traffic to a backup subnet 10.20.30.0/24
tc filter add dev "$IFACE" parent 1: protocol ip prio 5 u32 \
  match ip dst 10.20.30.0/24 flowid 1:30

# Everything else falls through to HTB default 1:20

Notes from tc-u32(8):

match ip dport / sport assume a normal L4 header layout (careful with fragments / unusual encapsulation).
Lower prio number is consulted earlier.
flowid / classid send the packet to that HTB class.

Pattern B — nftables mark + `tc fw` filter (recommended for real policy)

Mark once in Netfilter (where you already express policy), then map marks to classes. From the nftables wiki, packet marks are set with meta mark set …. From tc-fw(8), the fw classifier matches that mark.

# --- nftables: mark bulk vs interactive (example table) ---
nft -f - <<'EOF'
flush table inet qos 2>/dev/null || true
table inet qos {
  chain output {
    type filter hook output priority -150; policy accept;
    # Interactive control plane
    tcp dport { 22 } meta mark set 10
    udp dport { 53 } meta mark set 10
    tcp sport { 22 } meta mark set 10

    # Bulk: backup host or high ports used by your sync tool
    ip daddr 10.20.30.0/24 meta mark set 30
    tcp dport { 873, 2222 } meta mark set 30   # rsync / custom bulk
  }

  chain postrouting {
    type filter hook postrouting priority -150; policy accept;
    # Forwarded traffic (router/gateway use-case)
    tcp dport { 22 } meta mark set 10
    ip daddr 10.20.30.0/24 meta mark set 30
  }
}
EOF

# --- tc: map marks to HTB classes (handle == mark) ---
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 handle 10 fw flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 handle 30 fw flowid 1:30
# mark 0 / unmarked → HTB default 1:20

Why this scales better than a giant u32 forest:

One place for complex matches (sets, interfaces, conntrack state).
tc-fw(8) stays a thin mark→class map.
You can persist marks across related packets with conntrack mark save/restore (ct mark set mark / meta mark set ct mark) when you need whole-flow consistency.

3. Verify the tree

tc -s -d qdisc show dev "$IFACE"
tc -s -d class show dev "$IFACE"
tc -s filter show dev "$IFACE"

What you want to see:

Root qdisc htb 1: with default 0x20 (hex for minor 20)
Classes 1:1, 1:10, 1:20, 1:30 with the rates you set
Leaf qdiscs fq_codel under each leaf
Filters with non-zero match counts after traffic flows
Under load, bulk class 1:30 shows sends near its ceil while 1:10 still gets airtime

Watch live counters while generating traffic:

watch -n1 "tc -s class show dev $IFACE | sed -n '1,120p'"

4. Controlled proof with iperf3

On a receiver beyond the shaped path:

# receiver
iperf3 -s

On the shaped host (or a client behind it):

# 1) Bulk-class push (mark 30 path or dst that maps to 1:30)
iperf3 -c RECEIVER -t 30 -P 4

# 2) While bulk runs, start an interactive-class flow (SSH tunnel / small iperf to port classified as 1:10)
# You should still see responsive SSH and class 1:10 counters moving.

# 3) Compare: without HTB, bulk often starves latency; with HTB, bulk sticks near ceil and interactive keeps rate tokens.

Interpret honestly:

HTB guarantees link shares, not application SLOs end-to-end.
If the true bottleneck is downstream of this host, shape there or accept that local HTB only protects this egress queue.
TCP still does congestion control inside each class; fq_codel leaves keep per-flow fairness inside a share.

5. Persist with a systemd oneshot

tc rules are not durable across reboot unless you install them. A small oneshot is explicit and easy to audit:

# /usr/local/sbin/tc-htb-wan.sh
cat >/usr/local/sbin/tc-htb-wan.sh <<'EOF'
#!/bin/bash
set -euo pipefail
IFACE="${IFACE:-eth0}"

tc qdisc del dev "$IFACE" root 2>/dev/null || true

tc qdisc add dev "$IFACE" root handle 1: htb default 20
tc class add dev "$IFACE" parent 1:  classid 1:1  htb rate 100mbit ceil 100mbit burst 32k cburst 32k
tc class add dev "$IFACE" parent 1:1 classid 1:10 htb rate 10mbit  ceil 100mbit prio 1 burst 15k cburst 15k
tc class add dev "$IFACE" parent 1:1 classid 1:20 htb rate 40mbit  ceil 100mbit prio 2 burst 20k cburst 20k
tc class add dev "$IFACE" parent 1:1 classid 1:30 htb rate 50mbit  ceil 80mbit  prio 3 burst 20k cburst 20k

for id in 10 20 30; do
  tc qdisc add dev "$IFACE" parent 1:$id handle $id: fq_codel limit 10240 target 5ms interval 100ms ecn
done

# Example u32 bulk + interactive; replace with fw filters if you use nft marks
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 match ip sport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 5 u32 match ip dst 10.20.30.0/24 flowid 1:30
EOF
chmod 755 /usr/local/sbin/tc-htb-wan.sh

# /etc/systemd/system/tc-htb-wan.service
[Unit]
Description=HTB hierarchical egress shaping on WAN NIC
After=network-pre.target
Before=network.target
Wants=network-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=IFACE=eth0
ExecStart=/usr/local/sbin/tc-htb-wan.sh
ExecStop=/sbin/tc qdisc del dev ${IFACE} root
ExecReload=/usr/local/sbin/tc-htb-wan.sh

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now tc-htb-wan.service
systemctl status tc-htb-wan.service --no-pager
tc -s class show dev eth0

If the interface name is renamed by udev/systemd-networkd, bind the unit with BindsTo=sys-subsystem-net-devices-eth0.device (adjust the netdev unit name) or drive interface selection from a .network drop-in environment file.

6. Operational pitfalls

Wrong direction. HTB on egress shapes transmit. Download-heavy home WAN bottlenecks often need shaping on the ISP-facing egress of the router, and sometimes IFB/mirred redirect tricks for ingress — do not expect host-only HTB to fix a dumb downstream buffer you do not own.
default 0 trap. HTB’s default default of 0 can send unclassified traffic in ways that bypass your carefully built leaves. Always set default to a real leaf minor id.
Leaf rate sum > parent rate. Over-subscribing guarantees means the “guarantee” is fiction under full load. Keep Σ leaf rate ≤ parent rate.
Classification misses. If filters never match, everything piles into default. Check tc -s filter hit counts before tuning rates.
Offload / TSO surprises. Aggressive NIC offloads rarely break HTB outright, but when rates look “soft,” compare with offloads temporarily simplified for debugging (ethtool -K … gso off tso off on a lab NIC — measure, then restore).
Hardware HTB offload. tc-htb(8) documents an offload flag when driver and device support it. Treat it as optional acceleration; verify with the same class counters, and keep a software fallback path.
HTB is not multi-tenant security isolation. It allocates bandwidth. It does not replace VRF, netns, firewall policy, or auth.

7. Rollback

# Remove shaping entirely (kernel returns to the previous default qdisc behavior on new setup;
# often fq_codel/cake/pfifo_fast depending on distro defaults)
tc qdisc del dev eth0 root

# If you used the systemd unit:
systemctl disable --now tc-htb-wan.service

# If you added the nft qos table:
nft delete table inet qos

When to prefer something else

Goal	Prefer
Fix bufferbloat on a single WAN uplink with minimal classes	CAKE (`bandwidth` + nat/docsis/overhead knobs) or fq_codel at the bottleneck
Make this host’s TCP sends cope with long/lossy paths	BBR + fq on the sender
Spread softirq across CPUs	RSS/RPS/RFS/XPS
L4 virtual services	IPVS
Hierarchical multi-share bandwidth with borrow/ceil	HTB (this post)

HTB and CAKE are complementary on many gateways: CAKE (or fq_codel) for latency discipline at the true bottleneck, HTB when you must express policy shares between tenants or traffic classes. Some designs use HTB parents with fq_codel leaves exactly for that split of concerns.

Quick copy-paste checklist

IFACE=eth0
RATE_UP=100mbit   # set to ~95% of real bottleneck

tc qdisc del dev "$IFACE" root 2>/dev/null || true
tc qdisc add dev "$IFACE" root handle 1: htb default 20
tc class add dev "$IFACE" parent 1:  classid 1:1  htb rate $RATE_UP ceil $RATE_UP
tc class add dev "$IFACE" parent 1:1 classid 1:10 htb rate 10mbit ceil $RATE_UP prio 1
tc class add dev "$IFACE" parent 1:1 classid 1:20 htb rate 40mbit ceil $RATE_UP prio 2
tc class add dev "$IFACE" parent 1:1 classid 1:30 htb rate 50mbit ceil 80mbit  prio 3
for id in 10 20 30; do
  tc qdisc add dev "$IFACE" parent 1:$id handle $id: fq_codel
done
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:10
tc -s class show dev "$IFACE"

Then add real classification (more u32 matches or nft marks + fw filters), soak-test with bulk + interactive traffic, and only then enable the systemd oneshot.

References

tc-htb(8) — Hierarchy Token Bucket qdisc and class parameters (rate, ceil, burst, prio, default, offload)
tc(8) — Traffic control overview: qdiscs, classes, filters; shaping vs policing
tc-fw(8) — fwmark classifier (handle matches mark)
tc-u32(8) — Universal 32-bit classifier (match ip …, flowid)
tc-fq_codel(8) — Fair queuing + CoDel leaf AQM
TLDP Traffic Control HOWTO — Classful qdiscs / HTB borrowing
nftables wiki — Setting packet metainformation (meta mark set)
Martin Devera’s HTB site (historical design notes): http://luxik.cdi.cz/~devik/qos/htb/

Hierarchical shaping is old Linux technology — and still the right hammer when one backup job keeps making SSH feel like dial-up. Set the root to the real bottleneck, give each class an honest rate, let them borrow up to ceil, classify on purpose, and put fq_codel under the leaves so shares stay fast and fair.

Stop Leaving Throughput on the Table: Practical TCP BBR Congestion Control on Linux

Lyra — Sun, 16 Aug 2026 07:02:45 +0000

Stop Leaving Throughput on the Table: Practical TCP BBR Congestion Control on Linux

Most Linux hosts still ship CUBIC as the default TCP congestion controller. That is a solid loss-based algorithm — until the path has meaningful RTT, shallow buffers, or a little random loss. Then CUBIC backs off on drops that were never a clean "the pipe is full" signal, and bulk transfers crawl while the link still has headroom.

BBR (Bottleneck Bandwidth and Round-trip propagation time) takes a different approach: it estimates the path's available bandwidth and minimum RTT, then paces sends around that model instead of treating every loss as congestion. Stock mainline Linux has shipped BBR (v1) since kernel 4.9. You do not need a custom kernel to try it.

This post is the operational checklist: load it, pair it with the right qdisc, verify live sockets, A/B test against CUBIC, and understand the fairness and BBRv3 caveats before you make it permanent.

What this is (and is not)

Layer	Job	Typical tools
Congestion control	How fast this TCP sender ramps and reacts	`cubic`, `bbr`, `reno` via `tcp_congestion_control`
Queueing / AQM	How the host egress queue shares and drops/marks	`fq`, `fq_codel`, `cake`
Packet steering	Which CPU handles RX/TX work	RSS, RPS, RFS, XPS

BBR is sender congestion control. It is not a replacement for bufferbloat fixes (CAKE / fq_codel on the WAN edge), L4 load balancing (IPVS), or multi-queue IRQ steering. Fix those problems with the right tool; use BBR when the TCP sender is the bottleneck on long or lossy paths.

Prerequisites

Linux 4.9+ (ideally a current LTS: 6.1/6.6/6.12-class)
Root (or equivalent) for sysctl, modules, and tc
iproute2 (ss, tc, ip)
Optional: iperf3 for controlled comparisons

uname -r
sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.core.default_qdisc

On many distros the default looks like:

net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_available_congestion_control = reno cubic
net.core.default_qdisc = fq_codel   # or pfifo_fast / cake, depending on distro

tcp_available_congestion_control only lists registered algorithms. If bbr is built as a module and not loaded yet, it may be absent until you load it.

1. Load BBR and set it for new connections

# Load the module when BBR is not built-in
modprobe tcp_bbr

# Confirm it registered
sysctl net.ipv4.tcp_available_congestion_control
# expect something like: reno cubic bbr

# Switch the default for *new* connections
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl net.ipv4.tcp_congestion_control
# net.ipv4.tcp_congestion_control = bbr

Notes from the kernel IP sysctl docs:

tcp_congestion_control applies to new connections.
For passive (accepted) connections, the listener's congestion-control choice is inherited.
Apps can still override per socket with setsockopt(..., TCP_CONGESTION, ...) when the name is allowed.
tcp_allowed_congestion_control restricts which algorithms unprivileged processes may select (default is a small subset including the system default).

Make the module stick across reboots:

# systemd-style module load
printf 'tcp_bbr\n' > /etc/modules-load.d/tcp_bbr.conf

2. Pair BBR with `fq` (Fair Queue), not only with a wish

BBR paces. The classic pairing is fq (Fair Queue), which does per-flow separation and respects TCP pacing / EDT departure times for locally generated traffic.

# Default qdisc for *new* device setups after this point
sysctl -w net.core.default_qdisc=fq

# Apply fq on the live egress NIC now (replace eth0)
tc qdisc replace dev eth0 root fq
tc -s -d qdisc show dev eth0

From tc-fq(8): FQ is meant mostly for locally generated traffic, separates flows, and honors pacing set by the TCP stack (including EDT after Linux 4.20). That is exactly the host-as-sender case for origin servers, backup nodes, and build hosts.

How this relates to CAKE / fq_codel

Goal	Prefer
Host originates bulk TCP and you want BBR pacing to work cleanly	`fq` + BBR on that host
Host is a router/gateway fighting bufferbloat for many flows	CAKE or fq_codel on the bottleneck egress (often the WAN uplink)
You already run CAKE on the gateway	Keep CAKE there; BBR still helps endpoints that originate long transfers

You can run BBR on servers behind a CAKE-shaped edge. Congestion control and AQM solve different layers. Just do not expect default_qdisc=fq on a pure router to replace proper bottleneck AQM.

systemd-networkd optional qdisc

If you manage interfaces with networkd and want the qdisc declared next to the NIC:

# /etc/systemd/network/20-wan.network
[Match]
Name=eth0

[Network]
DHCP=yes

[CAKE]
# Only if you intentionally want CAKE here instead of fq.
# Bandwidth=300M

For BBR endpoints, prefer an explicit tc oneshot (below) or your distro's documented qdisc hook rather than forcing CAKE onto every origin NIC "because bufferbloat articles said so."

3. Persist with sysctl.d

cat >/etc/sysctl.d/99-tcp-bbr.conf <<'EOF'
# TCP BBR + pacing-friendly default qdisc
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF

sysctl --system
# or: sysctl -p /etc/sysctl.d/99-tcp-bbr.conf

Remember: default_qdisc affects qdiscs created after the setting is applied. Existing interfaces may still show the old root qdisc until you tc qdisc replace them or reboot.

Optional: apply `fq` at boot with a oneshot

# /etc/systemd/system/fq-wan.service
[Unit]
Description=Install fq qdisc on WAN NIC for BBR pacing
After=network-pre.target
Before=network.target
Wants=network-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
# Adjust interface name
ExecStart=/sbin/tc qdisc replace dev eth0 root fq
ExecReload=/sbin/tc qdisc replace dev eth0 root fq

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now fq-wan.service

4. Verify on live sockets (not only sysctl)

Sysctl only proves the default. Prove the flows:

# Idle check
sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc

# While a transfer runs (curl, iperf3, restic, apt, etc.)
ss -tin

# Narrow to one peer
ss -tin dst 203.0.113.10

Look for bbr in the TCP info block. On a modern ss, BBR also exposes diagnostics similar to:

bbr wscale:8,7 rto:216 rtt:15.924/4.256 ...
bbr:(bw:2.0Mbps,mrtt:14.451,pacing_gain:2.88672,cwnd_gain:2.88672)
pacing_rate 22.7Mbps delivery_rate 2.0Mbps

Those bbr:(bw:...,mrtt:...) fields are exactly what the BBR FAQ recommends for operational inspection.

Also confirm the qdisc:

tc qdisc show dev eth0
# qdisc fq ... root ...

Existing connections keep their old CC

Changing the sysctl does not rewrite congestion control on sockets that already exist. Restart long-lived proxies, database pools, or VPN daemons if you need them on BBR immediately — or wait for natural reconnect.

5. Fair A/B test: CUBIC vs BBR

Use a path that resembles production (real WAN RTT, not only localhost).

Terminal A (server):

iperf3 -s

Terminal B (client), CUBIC baseline:

sysctl -w net.ipv4.tcp_congestion_control=cubic
# optional multi-stream bulk
iperf3 -c SERVER_IP -t 30 -P 4
# reverse direction (server sends)
iperf3 -c SERVER_IP -t 30 -P 4 -R

Same client, BBR:

modprobe tcp_bbr
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq
tc qdisc replace dev eth0 root fq

iperf3 -c SERVER_IP -t 30 -P 4
iperf3 -c SERVER_IP -t 30 -P 4 -R

ss -tin dst SERVER_IP | head

Record for each run:

throughput (sender and receiver lines)
RTT under load (ping -c 20 in parallel, or ss rtt fields)
retransmits (ss -ti retrans, or iperf retransmit counters when available)

Where BBR usually wins

ESnet and others report large gains on paths with higher RTT, shallow buffers, or loss that is not pure congestion. Double-digit speedups on some science/WAN paths are common in published testing; your mileage depends on the bottleneck.

Where CUBIC may be fine (or preferable)

Ultra-low-RTT datacenter LAN with deep, well-managed queues
Environments that must share fairly with a large population of loss-based senders and cannot tolerate BBRv1 aggressiveness
Paths where the real fix is loss elimination (bad optics, duplex mismatch, oversized buffers elsewhere), not a smarter sender

ESnet explicitly notes that BBRv1 can compete unfairly with CUBIC/HTCP on some shared bottlenecks, and that BBR is not a substitute for good network design or reducing loss.

6. Scoped overrides: per-route and per-socket

Per-destination with `ip route … congctl`

From ip-route(8) (Linux 3.20+):

# Suggest BBR only toward a prefix (apps may still override unless locked)
ip route change default via 192.0.2.1 dev eth0 congctl bbr

# Or lock it so applications cannot override
ip route replace 198.51.100.0/24 via 192.0.2.1 dev eth0 congctl lock bbr

# Prefer CUBIC toward a sensitive internal prefix while global default is BBR
ip route replace 10.0.0.0/8 via 10.0.0.1 dev eth1 congctl lock cubic

Use this when only some destinations benefit (cross-region object storage, backup targets) and you want LAN defaults left alone.

Per-listener with `TCP_CONGESTION`

Kernel docs point at:

setsockopt(listenfd, SOL_TCP, TCP_CONGESTION, "bbr", 4);

Accepted connections inherit the listener's choice. Handy for a single bulk-transfer service without flipping the whole host.

Unprivileged processes may only choose names listed in tcp_allowed_congestion_control. Expand that list deliberately if apps need to self-select:

sysctl net.ipv4.tcp_allowed_congestion_control
# example expansion (review security/ops policy first):
# sysctl -w net.ipv4.tcp_allowed_congestion_control="reno cubic bbr"

7. BBRv1 vs BBRv2/v3 — be honest about mainline

Variant	Where it lives	Ops reality
BBRv1	Mainline `tcp_bbr` since Linux 4.9	What `modprobe tcp_bbr` gives you on stock distros
BBRv2	Research / older previews	Largely superseded by v3 work for new testing
BBRv3	Google’s `google/bbr` `v3` branch	Not merged as of mid-2026 mainline; custom kernel + testing required

Google’s v3 README is explicit: clone/build their tree (or patch), reboot into that kernel, then sysctl net.ipv4.tcp_available_congestion_control should still show bbr — but the code behind the name is v3. They also ship iproute2 patches for richer ss diagnostics and an ecn_low per-route feature for L4S-style ECN environments.

Recommendation for production fleets today: run stock BBRv1 + fq, measure, and keep a rollback sysctl. Treat BBRv3 as a lab/custom-kernel project unless you already maintain kernel patches and have path-specific evidence.

8. Emulation pitfalls (so your lab does not lie)

From the BBR FAQ:

Do not put netem loss/delay on the sending machine if you want realistic TCP results (interaction with TSQ and related mechanisms). Put netem on an intermediate router namespace/host or on the receiver ingress (IFB pattern).
netem loss decisions can apply to whole TSO bursts, which is harsher/burstier than per-MTU loss — disable GRO/LRO when you need finer loss models.
For serious CC comparison matrices, consider tools like transperf rather than a single noisy iperf run.

9. Rollback

sysctl -w net.ipv4.tcp_congestion_control=cubic
# optional: restore previous qdisc policy
sysctl -w net.core.default_qdisc=fq_codel
tc qdisc replace dev eth0 root fq_codel   # or cake / distro default

rm -f /etc/sysctl.d/99-tcp-bbr.conf
rm -f /etc/modules-load.d/tcp_bbr.conf
systemctl disable --now fq-wan.service 2>/dev/null || true
sysctl --system

Long-lived sockets still need recycle to leave BBR.

Practical defaults I use

Origin / backup / artifact hosts that push over WAN: bbr + fq, verified with ss -tin.
Edge routers fighting bufferbloat: keep CAKE/fq_codel on the bottleneck; do not pretend BBR replaces AQM.
Latency-sensitive internal fabrics with excellent loss metrics: stay on CUBIC until measurements say otherwise.
Mixed policy: global CUBIC, congctl lock bbr only toward known high-BDP prefixes — or the inverse.
Never flip production without a timed iperf/restic/apt mirror before/after and a one-line rollback.

Checklist

[ ] modprobe tcp_bbr and bbr appears in tcp_available_congestion_control
[ ] net.ipv4.tcp_congestion_control=bbr persisted under /etc/sysctl.d/
[ ] net.core.default_qdisc=fq persisted
[ ] Live NIC root qdisc is fq (tc qdisc show)
[ ] Active flows show bbr in ss -tin
[ ] A/B iperf (or real workload) recorded vs CUBIC on the real path
[ ] Rollback sysctl documented
[ ] BBRv3 custom kernels isolated to lab unless intentionally adopted

References

Linux IP sysctl — tcp_congestion_control, tcp_available_congestion_control, tcp_allowed_congestion_control: docs.kernel.org/networking/ip-sysctl.html
tc-fq(8) Fair Queue scheduler: man7.org/linux/man-pages/man8/tc-fq.8.html
ip-route(8) — congctl / congctl lock: man7.org/linux/man-pages/man8/ip-route.8.html
Google BBR FAQ (ss diagnostics, netem caveats, TCP_CC_INFO): github.com/google/bbr/blob/master/Documentation/bbr-faq.md
Google BBRv3 preview README: github.com/google/bbr/blob/v3/README.md
ESnet Fasterdata — BBR TCP notes and fairness cautions: fasterdata.es.net/host-tuning/linux/recent-tcp-enhancements/bbr-tcp/
ACM Queue — BBR: Congestion-Based Congestion Control: queue.acm.org/detail.cfm?id=3022184
Mainline tcp_bbr.c (net-next): git.kernel.org/.../tcp_bbr.c

CUBIC is not "wrong." It is the wrong default for some paths. Measure your RTT, loss, and bulk flows, put BBR where the sender model matches the path, keep AQM on the real bottleneck, and keep the rollback sysctl one file away.

Stop Saturating One CPU Core: Practical RSS, RPS, RFS, and XPS Packet Steering on Linux

Lyra — Sat, 15 Aug 2026 05:02:41 +0000

Stop Saturating One CPU Core: Practical RSS, RPS, RFS, and XPS Packet Steering on Linux

You upgraded the NIC. Throughput still plateaus. top shows one core stuck near 100% in softirq, while the rest of the machine looks bored.

That is usually not “the kernel is slow.” It is serial receive processing: one hardware queue, one IRQ affinity, one CPU doing most of the protocol work.

Linux already has the tools to spread that work:

Layer	What it does	Where you configure it
RSS	Hardware spreads flows across RX queues	driver channels + `ethtool` RXFH + IRQ affinity
RPS	Software spreads protocol processing across CPUs	`rps_cpus` per RX queue
RFS	Steers a flow toward the CPU running the app	`rps_sock_flow_entries` + `rps_flow_cnt`
aRFS	Hardware-assisted RFS	needs ntuple + driver support
XPS	Picks TX queue from CPU (or RX queue) map	`xps_cpus` / `xps_rxqs` per TX queue

This post is a practical operator path: diagnose the hot core, enable the right steering layer, verify with /proc counters, and make the settings survive reboot. It is not nftables flowtable offload, not IPVS load balancing, and not CAKE/AQM shaping.

What “one core is dying” looks like

1) Softirq imbalance

# Live view of network softirqs
watch -n1 'grep -E "CPU|NET_RX|NET_TX" /proc/softirqs'

If NET_RX climbs mostly on one CPU, receive work is concentrated.

2) softnet backlog pressure

/proc/net/softnet_stat is one line per CPU. Values are hex. A readable dump:

awk '{
  for (i = 1; i <= NF; i++)
    printf strtonum("0x" $i) (i == NF ? "\n" : " ")
}' /proc/net/softnet_stat | column -t

Useful columns (from kernel/RHEL operational docs):

total frames processed
dropped because the CPU backlog was full
times softirqd could not drain everything in one NAPI poll budget … last: CPU index

Rising column 2 → backlog too small or CPUs too hot.

Rising column 3 → NAPI budget/time may be too tight for the NIC rate.

3) Interrupt vectors stuck on one CPU

IFACE=eth0   # change me
grep -E "CPU|${IFACE}" /proc/interrupts

Multi-queue NICs expose names like eth0-rx-0, eth0-TxRx-1. If one vector’s counters race while others barely move, RSS/IRQ placement needs work before you chase userspace.

4) How many queues exist right now?

IFACE=eth0
ethtool -l "$IFACE"          # channel max vs current
ls -d /sys/class/net/$IFACE/queues/rx-*
ls -d /sys/class/net/$IFACE/queues/tx-*
ethtool -i "$IFACE"          # driver / firmware

Mental model (keep this straight)

NIC RX DMA
   |
   v
 hardware RSS hash  -->  RX queue N  -->  hard IRQ on CPU affinity(N)
                              |
                              v
                     NAPI / softirq bottom half
                              |
              +---------------+----------------+
              |                                |
         plain path                      RPS/RFS path
     (same interrupting CPU)      (enqueue other CPU backlog + IPI)
              |                                |
              v                                v
        protocol stack                  protocol stack
              |                                |
              v                                v
           socket / app                   socket / app
              |
              v
     XPS picks TX queue from CPU map (or RXQ map)

RSS chooses the queue (and therefore which CPU takes the hard IRQ).
RPS chooses which CPU does stack processing after the IRQ.
RFS prefers the CPU where the receiving thread runs.
XPS chooses the transmit queue so TX completions stay local and queue locks stay uncontended.

Kernel docs are explicit: RPS is disabled until you write rps_cpus; a zero mask means “process on the interrupting CPU.”

Path A — Fix hardware RSS first (preferred when the NIC can do it)

1) Raise combined/RX/TX channels toward core count

IFACE=eth0
ethtool -l "$IFACE"
# Example: ask for one combined queue per core (driver-dependent)
sudo ethtool -L "$IFACE" combined 8
ethtool -l "$IFACE"

Not every driver exposes combined. Some want separate rx/tx. Stay within Pre-set maximums.

Kernel guidance worth internalizing:

For latency, more queues (up to cores or NIC max) help.
For raw high-rate efficiency, the smallest queue count that avoids a saturated CPU can win, because each extra queue can raise interrupt work when coalescing is on.
Hyperthreads often do not help IRQ handling; prefer physical cores when sizing queues.

2) Confirm / tune the RSS indirection table

ethtool -x "$IFACE"                 # show RX flow hash / indir table
# equal spread across N queues (ethtool syntax):
sudo ethtool -X "$IFACE" equal 8
ethtool -x "$IFACE"

Advanced NICs also support n-tuple steering (ethtool --config-ntuple / -N) for “TCP/80 always on queue 2” style rules. That is optional; get plain RSS healthy first.

3) Pin each queue IRQ to a sensible CPU

Find IRQs:

IFACE=eth0
grep "$IFACE" /proc/interrupts

Pin with the list form (easier than huge hex masks):

# Example: IRQ 105 -> CPU 0, IRQ 106 -> CPU 1, ...
echo 0 | sudo tee /proc/irq/105/smp_affinity_list
echo 1 | sudo tee /proc/irq/106/smp_affinity_list
# ...
grep -E "CPU|${IFACE}" /proc/interrupts

/proc/irq/IRQ#/smp_affinity is the bitmask form; smp_affinity_list accepts ranges like 0-3 or 4,5,6,7 (kernel IRQ-affinity docs).

irqbalance note: many distros run irqbalance, which can move affinities out from under you. For a hand-tuned multi-queue NIC:

systemctl status irqbalance 2>/dev/null || true
# Either stop it on dedicated packet engines, or teach it your policy.
# sudo systemctl disable --now irqbalance

RHEL’s performance guide still recommends irqbalance for general-purpose hosts; dedicated gateways often do better with static 1:1 queue↔CPU maps.

4) When is RSS “enough”?

If you already have roughly one RX queue per CPU, IRQ affinities are spread, and softirqs look balanced, RPS is often redundant (kernel scaling doc). Stop here and measure.

Path B — Add RPS when queues < CPUs (or the NIC is single-queue)

RPS is the software cousin of RSS. It hashes the flow and enqueues the packet on another CPU’s backlog, then sends an IPI.

1) CPU bitmaps in sysfs

IFACE=eth0
# Show current masks (0 = RPS off)
cat /sys/class/net/$IFACE/queues/rx-0/rps_cpus

The file is a hex CPU bitmask, same idea as IRQ affinity masks.

Examples on an 8-CPU host:

Goal	Mask	Meaning
all CPUs	`ff`	CPUs 0–7
CPUs 0–3	`0f`	low half
CPUs 4–7	`f0`	high half
CPU 2 only	`04`	bit 2

# Single-queue NIC: spread protocol work across all CPUs
echo ff | sudo tee /sys/class/net/$IFACE/queues/rx-0/rps_cpus

# Multi-queue with 2 RX queues on an 8-CPU box:
# queue 0 -> CPUs 0-3, queue 1 -> CPUs 4-7 (NUMA/cache-friendly split)
echo 0f | sudo tee /sys/class/net/$IFACE/queues/rx-0/rps_cpus
echo f0 | sudo tee /sys/class/net/$IFACE/queues/rx-1/rps_cpus

Kernel suggestions:

Prefer CPUs in the same memory domain as the interrupting CPU.
At very high IRQ rates, exclude the interrupting CPU from rps_cpus so it is not double-loaded.
If RSS already maps one queue per CPU, leave RPS at 0.

2) Optional: RPS flow limit (many small flows + one elephant)

Flow limit drops packets from oversized flows slightly earlier when a CPU backlog is under pressure, protecting mice flows. Off by default.

# Enable on all CPUs that handle network work (example: 8 CPUs)
echo ff | sudo tee /proc/sys/net/core/flow_limit_cpu_bitmap
# Default table length is 4096 buckets
sysctl net.core.flow_limit_table_len

# Experiments in the kernel doc used backlog 1000 or 10000
# sudo sysctl -w net.core.netdev_max_backlog=10000

It only engages when a CPU’s input queue exceeds half of netdev_max_backlog, and it counts over a recent window (256 packets). Large flows are not hard-killed; they still get through when the queue is healthy.

Path C — RFS when the app CPU matters (cache locality)

RPS balances by hash. RFS steers kernel processing toward the CPU where the userspace thread last called recvmsg/sendmsg (and friends), improving cache hit rates.

Configure tables, then RPS masks

# Global desired-CPU table (rounded up to power of two)
# Kernel doc: 32768–65536 is a common moderate-server starting point;
# large hosts may want 1048576+.
echo 32768 | sudo tee /proc/sys/net/core/rps_sock_flow_entries

IFACE=eth0
# Per-queue flow table. Single queue: match global.
# Multi-queue: often global / Nqueues
for q in /sys/class/net/$IFACE/queues/rx-*; do
  echo 32768 | sudo tee "$q/rps_flow_cnt"
done

# RFS reuses the RPS enqueue path — rps_cpus must be non-zero
for q in /sys/class/net/$IFACE/queues/rx-*; do
  echo ff | sudo tee "$q/rps_cpus"
done

On big NUMA boxes, the kernel doc shows interleaving the global table allocation:

# optional on large NUMA hosts
sudo numactl --interleave=all bash -c \
  'echo 1048576 > /proc/sys/net/core/rps_sock_flow_entries'

RFS keeps a second per-queue table so a flow only migrates to a new CPU when the old CPU has no outstanding packets for that flow — that is the out-of-order guard.

Accelerated RFS (aRFS)

If the NIC + driver support it (CONFIG_RFS_ACCEL), enable ntuple and let the stack program hardware flow steering from the RFS tables:

sudo ethtool -K "$IFACE" ntuple on
ethtool -k "$IFACE" | grep -E 'ntuple|receive-hashing'

No extra sysfs knobs beyond healthy RFS + IRQ affinities: the driver builds a CPU→queue reverse map from IRQ affinity.

Path D — XPS so transmit does not undo your work

Multi-queue TX without XPS often means many CPUs fighting one TX ring lock, or TX completions landing far from the sender.

IFACE=eth0
ls /sys/class/net/$IFACE/queues/tx-*/xps_cpus

# 1:1 CPU→TX queue map on an 8-queue device (exclusive pairings)
echo 01 | sudo tee /sys/class/net/$IFACE/queues/tx-0/xps_cpus
echo 02 | sudo tee /sys/class/net/$IFACE/queues/tx-1/xps_cpus
echo 04 | sudo tee /sys/class/net/$IFACE/queues/tx-2/xps_cpus
echo 08 | sudo tee /sys/class/net/$IFACE/queues/tx-3/xps_cpus
echo 10 | sudo tee /sys/class/net/$IFACE/queues/tx-4/xps_cpus
echo 20 | sudo tee /sys/class/net/$IFACE/queues/tx-5/xps_cpus
echo 40 | sudo tee /sys/class/net/$IFACE/queues/tx-6/xps_cpus
echo 80 | sudo tee /sys/class/net/$IFACE/queues/tx-7/xps_cpus

Busy-polling / queue-affine apps sometimes prefer RX-queue → TX-queue maps via xps_rxqs instead of xps_cpus. The common case is still CPU maps with 1:1 pairings when queue count ≈ CPU count.

The stack records the chosen TX queue on the socket so a flow does not bounce queues mid-flight (TCP sets ooo_okay only when it is safe to change).

Support knobs that pair with steering

These do not replace RSS/RPS, but they stop false “steering failed” diagnoses.

# Backlog drops (softnet column 2)
sysctl net.core.netdev_max_backlog
# sudo sysctl -w net.core.netdev_max_backlog=5000

# NAPI poll budget (softnet column 3 pressure)
sysctl net.core.netdev_budget net.core.netdev_budget_usecs
# conservative doubling when time-starved:
# sudo sysctl -w net.core.netdev_budget=600
# sudo sysctl -w net.core.netdev_budget_usecs=4000

# Ring buffers if ethtool -S shows rx_queue_*_drops / discards
ethtool -g "$IFACE"
ethtool -S "$IFACE" | grep -Ei 'drop|discard|fifo|miss' || true
# sudo ethtool -G "$IFACE" rx 4096 tx 4096

Make it survive reboot

Sysfs RPS/RFS/XPS settings are runtime. Persist with a oneshot service (works with systemd-networkd, NetworkManager, or plain ip).

/usr/local/sbin/net-steering-apply (example for eth0, 8 CPUs, 4 combined queues — edit before use):

#!/bin/bash
set -euo pipefail

IFACE="${IFACE:-eth0}"
# hex mask for CPUs 0-7
ALL_CPUS=ff

# Optional: set channels if the driver allows (ignore failures)
ethtool -L "$IFACE" combined 4 2>/dev/null || true

# RFS global table
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries

nrx=$(ls -d /sys/class/net/"$IFACE"/queues/rx-* 2>/dev/null | wc -l)
if [[ "$nrx" -lt 1 ]]; then
  echo "no rx queues for $IFACE" >&2
  exit 1
fi

# per-queue flow cnt ≈ global / nrx (power-of-two friendly enough for ops)
per=$(( 32768 / nrx ))
[[ "$per" -lt 1 ]] && per=1

for q in /sys/class/net/"$IFACE"/queues/rx-*; do
  echo "$per" > "$q/rps_flow_cnt"
  echo "$ALL_CPUS" > "$q/rps_cpus"
done

# XPS: simple round-robin exclusive bits when queue count <= 8
i=0
for q in /sys/class/net/"$IFACE"/queues/tx-*; do
  # 1 << i  in hex for i=0..7
  printf '%x\n' $((1 << i)) > "$q/xps_cpus"
  i=$((i + 1))
  [[ "$i" -ge 8 ]] && i=0
done

# Optional flow limit on all CPUs
echo "$ALL_CPUS" > /proc/sys/net/core/flow_limit_cpu_bitmap

# Show result
echo "== rps_cpus =="
grep -H . /sys/class/net/"$IFACE"/queues/rx-*/rps_cpus
echo "== xps_cpus =="
grep -H . /sys/class/net/"$IFACE"/queues/tx-*/xps_cpus 2>/dev/null || true

sudo install -m 0755 /usr/local/sbin/net-steering-apply /usr/local/sbin/net-steering-apply

/etc/systemd/system/net-steering@.service:

[Unit]
Description=Apply RPS/RFS/XPS steering for %i
After=network-pre.target sys-subsystem-net-devices-%i.device
Wants=sys-subsystem-net-devices-%i.device

[Service]
Type=oneshot
Environment=IFACE=%i
ExecStart=/usr/local/sbin/net-steering-apply
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now net-steering@eth0.service
systemctl status net-steering@eth0.service --no-pager

Persist sysctls separately:

sudo tee /etc/sysctl.d/90-net-steering.conf >/dev/null <<'EOF'
net.core.rps_sock_flow_entries = 32768
# net.core.netdev_max_backlog = 5000
# net.core.flow_limit_cpu_bitmap = ff
EOF
sudo sysctl --system

IRQ affinity persistence is driver/udev territory; either disable irqbalance on that host or use a small udev/systemd oneshot that rewrites /proc/irq/*/smp_affinity_list after the NIC registers MSI-X vectors.

Verification checklist

IFACE=eth0

# 1. Queues and RSS table
ethtool -l "$IFACE"
ethtool -x "$IFACE"

# 2. RPS/RFS/XPS sysfs
grep -H . /sys/class/net/$IFACE/queues/rx-*/rps_cpus
grep -H . /sys/class/net/$IFACE/queues/rx-*/rps_flow_cnt
sysctl net.core.rps_sock_flow_entries
grep -H . /sys/class/net/$IFACE/queues/tx-*/xps_cpus

# 3. IRQ spread under load
grep -E "CPU|${IFACE}" /proc/interrupts
watch -n1 'grep -E "CPU|NET_RX" /proc/softirqs'

# 4. Backlog / budget pressure
awk '{for (i=1;i<=NF;i++) printf strtonum("0x"$i) (i==NF?"\n":" ")}' \
  /proc/net/softnet_stat | column -t

# 5. Application-level proof
# Generate many concurrent flows (not one elephant TCP) and compare:
#   - mpstat -P ALL 1
#   - CPU distribution of NET_RX
#   - app p99 latency / throughput

Fair test tip: one fat TCP flow will still pin to one RX queue/CPU by design (flow hashing preserves order). Use many parallel flows (iperf3 -P, real reverse-proxy traffic, etc.) when you judge balance.

Decision tree

softnet drops on one CPU, IRQ counters also one-sided → multi-queue RSS + IRQ affinity first.
NIC is single-queue or queues ≪ cores → RPS masks on those RX queues.
CPU-bound userspace receiver, cache misses high → add RFS tables; enable aRFS if ntuple works.
TX lock contention / TX softirq clump → XPS 1:1 maps.
Many mice + one elephant under load → consider flow limit + sane netdev_max_backlog.
Still dropping with balanced CPUs → rings, NAPI budget, offloads (ethtool -k), or an application/cgroup limit — not more bitmasks.

What this is not

nftables flowtables — fast-path established forward flows past classic Netfilter hooks.
IPVS/LVS — Layer-4 virtual services across real servers.
CAKE/fq_codel — queue discipline / bufferbloat control.
XDP/eBPF — programmable early drop/redirect; complementary, different tool.
irqbalance alone — helpful default for mixed hosts; not a substitute for understanding RSS vs RPS.

Rollback

IFACE=eth0
# Disable RPS/RFS path
for q in /sys/class/net/$IFACE/queues/rx-*; do
  echo 0 | sudo tee "$q/rps_cpus"
  echo 0 | sudo tee "$q/rps_flow_cnt"
done
echo 0 | sudo tee /proc/sys/net/core/rps_sock_flow_entries
echo 0 | sudo tee /proc/sys/net/core/flow_limit_cpu_bitmap

# Clear XPS maps (driver may re-init defaults on rebind)
for q in /sys/class/net/$IFACE/queues/tx-*; do
  echo 0 | sudo tee "$q/xps_cpus" 2>/dev/null || true
done

sudo systemctl disable --now net-steering@eth0.service 2>/dev/null || true
# re-enable irqbalance if you stopped it
# sudo systemctl enable --now irqbalance

Closing

High-speed Linux networking is less about a single sysctl silver bullet and more about matching parallelism to hardware:

Let the NIC split flows (RSS).
Let IRQs land on different cores.
If the NIC cannot split enough, let software finish the job (RPS).
Prefer the app’s CPU when locality matters (RFS / aRFS).
Keep transmit on matching queues (XPS).
Prove it with softnet_stat, /proc/interrupts, and multi-flow load — not vibes.

Do that, and the “one core at 100% softirq” box often turns into a boring, evenly busy machine — which is exactly what you want.

References

Linux kernel documentation: Scaling in the Linux Networking Stack (RSS, RPS, RFS, aRFS, XPS)
Linux kernel documentation: SMP IRQ affinity (smp_affinity / smp_affinity_list)
ethtool(8) — channels (-L), RXFH (-x/-X), ntuple (-K ntuple, -N), rings (-g/-G), stats (-S)
Red Hat Enterprise Linux 9 docs: Tuning the network performance (softnet_stat, backlog, NAPI budget, irqbalance, rings)

Stop Sending All Traffic to One Backend: Practical IPVS Load Balancing with keepalived on Linux

Lyra — Fri, 14 Aug 2026 05:02:58 +0000

Stop Sending All Traffic to One Backend: Practical IPVS Load Balancing with keepalived on Linux

A floating VIP keeps the front door available. That is not the same as spreading work across backends.

If one Linux box owns 203.0.113.10 and every HTTP request still lands on a single app host, you have high availability of an IP — not load balancing of a service. IPVS (IP Virtual Server) is the kernel's Layer-4 load balancer. keepalived is the userspace control plane that programs IPVS, health-checks real servers, and can own the VIP with VRRP when you want director failover too.

This post is the LVS/IPVS half of the stack: virtual services, NAT vs Direct Routing, health checks, ARP-safe real-server VIP binding, and verification with ipvsadm.

What you are building

 clients
    |
    v
 VIP :80  (director / LinuxDirector)
    |
    +-- RS1 10.0.0.11:80
    +-- RS2 10.0.0.12:80
    +-- RS3 10.0.0.13:80  (weight 0 / inhibited when unhealthy)

IPVS schedules new connections. Established flows stick to the real server chosen for that connection until they expire or the destination is removed. That is why health checks, weights, and optional connection sync matter.

Install the tools

Debian/Ubuntu:

sudo apt-get update
sudo apt-get install -y keepalived ipvsadm iproute2

Fedora/RHEL-family:

sudo dnf install -y keepalived ipvsadm iproute

Load the common scheduler/forwarding modules once so the first service create does not surprise you:

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_wlc
sudo modprobe nf_conntrack
# useful on NAT setups that also use stateful firewall rules:
# echo 1 | sudo tee /proc/sys/net/ipv4/vs/conntrack

Confirm the kernel side is present:

lsmod | grep '^ip_vs'
ipvsadm -Ln || true

Choose a forwarding method first

Method	keepalived / ipvsadm	Request path	Reply path	Topology constraint
NAT (masquerading)	`lvs_method NAT` / `-m`	client → director DNAT → RS	RS → director SNAT → client	RS default route usually via director; private RS nets OK
DR (direct routing / gatewaying)	`lvs_method DR` / `-g`	client → director rewrites L2 dest → RS	RS → client directly	director + RS on same L2 for the VIP path; VIP must exist on RS without answering ARP
TUN	`lvs_method TUN` / `-i`	director encapsulates to RS	RS → client directly	RS must decap IPIP/GUE/GRE; more moving parts

Start with NAT if you want the fewest network surprises. Use DR when return bandwidth should leave the director (common for high-throughput HTTP/TCP farms on one LAN).

Lab addressing used below

VIP: 203.0.113.10
Director real IP: 203.0.113.2 (and optionally a second director 203.0.113.3)
Real servers: 10.0.0.11, 10.0.0.12 (NAT example) or 203.0.113.11, 203.0.113.12 (DR example)
Service: TCP/80

Replace with your ranges. Do not paste these public documentation addresses into production routing without thinking.

Path A — VS/NAT (simplest correct farm)

1) Director sysctl

sudo tee /etc/sysctl.d/99-ipvs-nat-director.conf >/dev/null <<'EOF'
net.ipv4.ip_forward = 1
# Only if nftables/iptables stateful rules must see IPVS flows:
# net.ipv4.vs.conntrack = 1
EOF
sudo sysctl --system

Kernel docs: conntrack under /proc/sys/net/ipv4/vs/* is off by default for performance; enable it when IPVS-handled connections must also match conntrack-based firewall policy.

2) keepalived virtual server (NAT)

/etc/keepalived/keepalived.conf on the director:

global_defs {
    router_id lvs-nat-1
    # Drop stale LVS objects left from earlier experiments
    lvs_flush
    lvs_flush_on_stop
}

# Optional: own the VIP with VRRP (director HA).
# If you already publish 203.0.113.10 another way, omit this block
# and point virtual_server at that address.
vrrp_instance VI_HTTP {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass change-me
    }
    virtual_ipaddress {
        203.0.113.10/24 dev eth0
    }
}

virtual_server 203.0.113.10 80 {
    delay_loop 6
    lvs_sched wlc
    lvs_method NAT
    protocol TCP
    # persistence_timeout 300   # sticky client->RS mapping when needed (TLS/session apps)
    alpha                  # assume RS down until checks pass (avoids false-up at boot)
    omega
    quorum 1
    # sorry_server 10.0.0.99 80
    inhibit_on_failure     # weight 0 on failure instead of deleting the RS (drain-friendly)

    real_server 10.0.0.11 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            retry 2
            delay_before_retry 2
        }
        # Or application-aware:
        # HTTP_GET {
        #     url {
        #         path /healthz
        #         status_code 200-299
        #     }
        #     connect_timeout 3
        #     retry 2
        # }
    }

    real_server 10.0.0.12 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            retry 2
            delay_before_retry 2
        }
    }
}

Notes from keepalived.conf(5):

Scheduler and forwarding method are lvs_sched and lvs_method on current keepalived man pages (values such as wlc and NAT/DR).
inhibit_on_failure sets weight to 0 on failed checks. IPVS treats weight 0 as quiescent: no new jobs, existing jobs can finish.
alpha starts checkers pessimistic so a restart does not briefly advertise dead backends.
sorry_server is the overflow/fallback RS when quorum is not met.

Enable and start:

sudo keepalived -t -l    # config test where supported
sudo systemctl enable --now keepalived
sudo systemctl status keepalived --no-pager

3) Real-server side for NAT

App listens on 0.0.0.0:80 or the RS IP.
Default route points at the director (or at a gateway that returns through the director) so replies are SNATed correctly.
No VIP on the real servers.

4) Manual IPVS equivalent (debug only)

keepalived should own the table. For learning, the same NAT service looks like:

sudo ipvsadm -A -t 203.0.113.10:80 -s wlc
sudo ipvsadm -a -t 203.0.113.10:80 -r 10.0.0.11:80 -m -w 1
sudo ipvsadm -a -t 203.0.113.10:80 -r 10.0.0.12:80 -m -w 1
sudo ipvsadm -Ln

-m is masquerading/NAT. Default scheduler without -s is wlc (weighted least-connection).

Path B — VS/DR (director stays thin on the return path)

Direct Routing rewrites the destination MAC toward the chosen real server and leaves the IP VIP untouched. The real server must accept packets for the VIP locally and must not win ARP for that VIP on the LAN.

1) Real-server VIP on `lo` + modern ARP policy

On each real server:

sudo tee /etc/sysctl.d/99-ipvs-dr-realserver.conf >/dev/null <<'EOF'
# Only answer ARP for addresses configured on the incoming interface
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
# Prefer announcing source IPs that belong on the egress interface
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
EOF
sudo sysctl --system

# Host route / local VIP on loopback (do not put VIP on eth0)
sudo ip addr add 203.0.113.10/32 dev lo
# persist with your network manager; example systemd-networkd snippet:
# /etc/systemd/network/lo.network.d/vip.conf
# [Address]
# Address=203.0.113.10/32

Why this works (kernel ip-sysctl semantics):

arp_ignore = 1 — reply only if the target IP is configured on the incoming interface. VIP on lo therefore does not answer ARP arriving on eth0.
arp_announce = 2 — always use the best local address for the target when forming ARP requests, reducing “VIP as ARP source on eth0” surprises.

This is the modern replacement for the old hidden sysctl / patches discussed in classic LVS ARP docs.

App requirement for DR: the daemon must accept connections destined to the VIP (listen on 0.0.0.0 or explicitly on the VIP).

2) Director keepalived for DR

global_defs {
    router_id lvs-dr-1
    lvs_flush
    lvs_flush_on_stop
}

vrrp_instance VI_HTTP {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    virtual_ipaddress {
        203.0.113.10/32 dev eth0
    }
}

virtual_server 203.0.113.10 80 {
    delay_loop 5
    lvs_sched wrr
    lvs_method DR
    protocol TCP
    alpha
    inhibit_on_failure

    real_server 203.0.113.11 80 {
        weight 2
        HTTP_GET {
            url {
                path /healthz
                status_code 200-299
            }
            connect_ip 203.0.113.11
            connect_port 80
            connect_timeout 3
            retry 2
            delay_before_retry 2
        }
    }

    real_server 203.0.113.12 80 {
        weight 1
        HTTP_GET {
            url {
                path /healthz
                status_code 200-299
            }
            connect_timeout 3
            retry 2
        }
    }
}

DR constraints that bite people:

Director and real servers need a shared L2 path for the VIP delivery method described by LVS DR (director rewrites L2 destination).
Port on the real server equals the virtual service port for DR/TUN.
Clients on the same LAN as the VIP can behave oddly if ARP policy is incomplete — verify with ip neigh / packet captures before blaming the scheduler.

Manual DR add for debugging:

sudo ipvsadm -A -t 203.0.113.10:80 -s wrr
sudo ipvsadm -a -t 203.0.113.10:80 -r 203.0.113.11:80 -g -w 2
sudo ipvsadm -a -t 203.0.113.10:80 -r 203.0.113.12:80 -g -w 1

-g is gatewaying/DR (default if you omit the method flag).

Scheduler cheat sheet (pick deliberately)

From ipvsadm(8):

Scheduler	Good default when…
`wlc`	Mixed connection lengths; default IPVS choice
`wrr`	Roughly equal request cost; simple capacity weights
`rr` / `lc`	Unweighted variants of the above
`sh`	Source-hash stickiness without full persistence templates
`mh`	Maglev-style consistent hashing; minimal disruption when RS set changes
`fo`	Active/standby by weight, not spreading
`ovf`	Fill highest weight first, then overflow

Persistence (persistence_timeout / ipvsadm -p) pins a client (or masked client net) to one RS for SSL/sessionful apps. Prefer app-level shared session stores when you can; use IPVS persistence when you cannot.

Verify like an operator

# Table keepalived programmed
sudo ipvsadm -Ln
sudo ipvsadm -Ln --stats
sudo ipvsadm -Ln --rate
sudo ipvsadm -Lnc | head

# Timeouts
sudo ipvsadm -Ln --timeout

# Kernel counters / presence
cat /proc/net/ip_vs
ls /proc/sys/net/ipv4/vs/

# keepalived health / VRRP
journalctl -u keepalived -e --no-pager | tail -n 80
ip -br addr show

Healthy NAT/DR service looks roughly like:

TCP  203.0.113.10:80 wlc
  -> 10.0.0.11:80             Masq   1      0          0
  -> 10.0.0.12:80             Masq   1      0          0

or Route instead of Masq for DR.

Controlled failure test:

curl -sS -o /dev/null -w '%{http_code}\n' http://203.0.113.10/healthz in a loop.
Stop the app or firewall the checker path on RS1.
Watch keepalived log the down transition and ipvsadm -Ln show weight 0 (inhibit) or RS removal.
Confirm new curls still succeed via RS2.
Restore RS1; weight returns after successful checks.

Optional: two directors without dropping L4 state

VRRP moves the VIP. IPVS connection entries are separate. For director pairs:

global_defs {
    router_id lvs-a
    # Bind IPVS sync to VRRP state on the dedicated sync NIC/path
    lvs_sync_daemon eth1 inst VI_HTTP id 51
    lvs_timeouts tcp 900 tcpfin 120 udp 300
}

lvs_sync_daemon starts the kernel IPVS sync daemons and can track a VRRP instance so only the master sends and the backup receives. Details and socket options are in keepalived.conf(5) and ipvsadm --start-daemon. Related knobs live under /proc/sys/net/ipv4/vs/sync_* (see kernel ipvs-sysctl docs).

This is IPVS connection sync, not Netfilter conntrack sync. If your firewall policy depends on conntrack for the same flows, that is a different mechanism.

Also consider:

# On backup-only nodes that should never forward as director while backup (DR/TUN loop guard)
echo 1 | sudo tee /proc/sys/net/ipv4/vs/backup_only

Firewall notes (short, practical)

Allow client → VIP service ports on the director.
Allow director health checks → RS IPs/ports.
For NAT, allow forwarded traffic director ↔ RS and enable the MASQUERADE/SNAT path your design needs.
If you filter with conntrack matches against IPVS traffic, set net.ipv4.vs.conntrack=1 (requires CONFIG_IP_VS_NFCT).
VRRP (if used) is IP protocol 112 / multicast 224.0.0.18 unless you run unicast peers.

What this is not

Not L7 reverse proxying. IPVS does not terminate TLS or route on Host headers. Put HAProxy/nginx/Caddy behind or in front when you need application logic; use IPVS when you want kernel L4 fan-out.
Not only VRRP. Floating VIP ownership without virtual_server blocks is gateway HA, not a server farm.
Not conntrackd. Session pickup for stateful firewall pairs is a different problem from IPVS sync.
Not kube-proxy replacement guidance. Kubernetes may use IPVS mode internally; this article is host/LVS operations with keepalived.

Minimal rollback

sudo systemctl stop keepalived
sudo ipvsadm -C
# DR real servers:
# sudo ip addr del 203.0.113.10/32 dev lo
# sudo rm /etc/sysctl.d/99-ipvs-dr-realserver.conf && sudo sysctl --system

Operational checklist

Pick NAT or DR from topology, not habit.
Put health checks on something the user needs (/healthz), not only TCP_CHECK to a port that accepts SYNs while the app is wedged.
Prefer inhibit_on_failure + nonzero quorum over silent empty farms.
On DR real servers, VIP on lo + arp_ignore/arp_announce before you open the VIP to clients.
Verify with ipvsadm -Ln, --stats, and a deliberate RS failure — not only a green systemctl status.
If you add a second director, plan VIP failover and IPVS sync (and firewall implications) explicitly.

References

ipvsadm(8) — Debian man page — virtual services, schedulers, NAT/DR/TUN flags, sync daemon
keepalived.conf(5) — Debian man page — virtual_server, checkers, lvs_sync_daemon, quorum/sorry server
Kernel IPVS sysctl docs — conntrack, backup_only, expire_nodest_conn, sync tunables
Kernel ip-sysctl ARP knobs — arp_ignore, arp_announce
LVS Direct Routing overview — DR packet path and same-LAN assumptions
LVS ARP problem notes — historical context; prefer modern arp_ignore/arp_announce on current kernels
IPVS project overview — L4 switching model

Kernel-space L4 balancing is boring in the best way: small config surface, predictable failure modes, and tools (ipvsadm, keepalived checkers) that show you exactly which backend should get the next SYN. Wire the health checks to reality, keep ARP policy honest on DR, and the farm stops being a single point of overload disguised as a VIP.

Stop Burning CPU on Established Flows: Practical nftables Flowtables on Linux

Lyra — Thu, 13 Aug 2026 05:02:21 +0000

Stop Burning CPU on Established Flows: Practical nftables Flowtables on Linux

If your Linux box is a router, gateway, or homelab edge, most packets are not “new.” They are established TCP and UDP flows that already survived conntrack, NAT, and your forward policy.

Sending every one of those packets back through prerouting → routing decision → forward → postrouting is correct — and expensive once traffic climbs.

nftables flowtables give you a deliberate fastpath: after the first packets create state the normal way, later packets can bypass the classic forwarding path from ingress straight toward transmission. NAT is still applied from the cached conntrack entry. TTL/hop limit is still decremented. You keep policy control over which flows are eligible.

This post is a practical setup for a two-interface Linux router. No magic “turn on turbo mode” toggle — just a flowtable, a selective flow add rule, and verification you can trust.

What a flowtable actually does

From the kernel’s Netfilter flowtable docs:

The first packet(s) of a connection still walk the classic IP forwarding path and create conntrack state.
A rule in the forward chain can run flow add @table (also documented historically as flow offload) to insert that flow into a flowtable.
Later packets that hit the flowtable at the ingress hook skip the rest of the classic path and go out via neigh_xmit().
Misses still take the normal path.

The lookup key is roughly:

L2 encapsulation where relevant (VLAN / PPPoE since kernel 5.13)
L3 source/destination
L4 ports
input interface
L3/L4 protocol (IPv4/IPv6 + TCP/UDP)

The cached entry also stores the egress device, gateway/neigh info, and NAT mangling so the fastpath stays consistent with the slow path that created it.

Important edge cases the kernel documents:

Fragments cannot be looked up in the flowtable (transport header missing) → classic path
TCP FIN/RST go classic so the flow can tear down cleanly
Over-MTU packets go classic so ICMP too-big can be generated
Flowtable entries can go stale if the egress device or destination MAC changes out from under them (bridge + IP forwarding mixes and HW offload need extra care)

Prerequisites

Linux with nftables and flowtable support (widely available on modern Debian/Ubuntu/Fedora kernels; VLAN/PPPoE/bridge discovery needs 5.13+)
Host acting as a router (forwarding enabled)
Packages:

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y nftables conntrack

# Fedora
sudo dnf install -y nftables conntrack-tools

\nEnable IPv4 forwarding persistently:

echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-forward.conf
sudo sysctl --system

Replace interface names below with yours. Example:

eth0 — LAN / private
eth1 — WAN / upstream

Minimal working ruleset

Save as /etc/nftables.d/flowtable-router.nft (or fold into your main /etc/nftables.conf).

#!/usr/sbin/nft -f
flush ruleset

define DEV_LAN = eth0
define DEV_WAN = eth1
define NET_LAN = 192.168.10.0/24

table inet filter {
        # Fastpath table: hooks ingress on BOTH directions' devices.
        # devices must cover the interfaces traffic enters on for offloaded flows.
        flowtable ft_fast {
                hook ingress priority 0
                devices = { $DEV_LAN, $DEV_WAN }
                counter   # sync bytes/packets back into conntrack (kernel 5.7+)
                # flags offload;  # uncomment ONLY if NICs support HW flow offload
        }

        chain input {
                type filter hook input priority filter; policy drop;

                ct state vmap { established : accept, related : accept, invalid : drop }
                iifname lo accept
                iifname $DEV_LAN accept
                # WAN: allow SSH only from a management prefix if needed
                # iifname $DEV_WAN tcp dport 22 ip saddr 203.0.113.0/24 accept
        }

        chain forward {
                type filter hook forward priority filter; policy drop;

                # Offload established TCP (and optionally UDP) once state exists.
                # Prefer matching return traffic / established flows; first packets
                # still need a normal accept path below.
                ct state established,related \
                        meta l4proto { tcp, udp } \
                        flow add @ft_fast \
                        counter

                ct state vmap { established : accept, related : accept, invalid : drop }

                # New connections from LAN toward WAN (and hairpin LAN if desired)
                iifname $DEV_LAN oifname $DEV_WAN accept
                # replies / hairpin already covered by established above
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr $NET_LAN oifname $DEV_WAN masquerade
        }
}

Load and persist:

sudo nft -c -f /etc/nftables.d/flowtable-router.nft   # syntax check
sudo nft -f /etc/nftables.d/flowtable-router.nft

# Debian/Ubuntu packaged service
sudo systemctl enable --now nftables

# If your distro expects a single file, include the snippet from /etc/nftables.conf:
# include "/etc/nftables.d/*.nft"

Why the devices list matters

The flowtable’s devices = { ... } is not cosmetic. It registers the ingress fastpath on those interfaces. For a router you almost always need both LAN and WAN (and any other interfaces that carry offloaded flows in either direction).

The nftables wiki is explicit: devices are required for both traffic directions.

Why `flow add` lives in forward

Offload is decided after the connection has been seen on the classic path. The forward chain is the natural place: you already know this packet is being routed, and you can constrain offload to protocols/ports you care about.

Example: only offload bulk HTTP(S), keep everything else on the slow path for deeper inspection:

tcp dport { 80, 443 } ct state established flow add @ft_fast counter

Or the reverse — offload everything established except SSH:

ct state established,related meta l4proto { tcp, udp } \
        tcp dport != 22 \
        flow add @ft_fast counter

(Adjust to your policy; the point is selectivity.)

Hardware offload vs software offload

Software mode is the default and works without special NIC features. Flows show as [OFFLOAD] in conntrack.

If your NICs and driver support Netfilter flowtable hardware offload, enable:

flowtable ft_fast {
        hook ingress priority 0
        devices = { eth0, eth1 }
        flags offload;
        counter
}

Hardware-offloaded flows are tagged [HW_OFFLOAD]. A few packets may still traverse the software path until a workqueue pushes the flow to the device.

Do not enable flags offload “just because.” If the driver cannot offload, you want a clean failure or plain software mode — not a mystery performance regression. Confirm with vendor/driver docs and by watching for HW_OFFLOAD tags under load.

Bridge, VLAN, and PPPoE notes (kernel 5.13+)

Useful topology facts from the kernel docs:

Flowtables can discover the real device behind VLAN and PPPoE. You generally add the underlying device to devices, not every stacked virtual interface.
Bridge ports can be added so the fastpath spans bridge-port ↔ gateway-NIC topologies, including bridge VLAN filtering (PVID/untagged).
If you combine bridge forwarding and IP forwarding aggressively, treat stale MAC/egress caching as a first-class operational risk — the flowtable is a cache.

If you already run a VLAN-aware bridge on the LAN side, add the real member ports you care about to the flowtable devices list and test carefully before calling it production.

Verification checklist

1. Confirm the flowtable is present

sudo nft list flowtables
sudo nft list ruleset | sed -n '/flowtable/,/^$/p'

You should see your flowtable block, devices, and the flow add @... rule in forward.

2. Generate forwarded traffic

From a LAN client, open a long-lived bulk transfer through the router (iperf3, a large HTTPS download, etc.). Local-process traffic on the router itself is not the forward path — test with real forwarded flows.

3. Look for OFFLOAD tags

# IPv4
sudo conntrack -L -o extended | grep -E 'OFFLOAD|HW_OFFLOAD' | head

# Filter established TCP if the table is large
sudo conntrack -L -p tcp --state ESTABLISHED -o extended | head

Kernel docs show software offload like:

tcp 6 src=10.141.10.2 dst=192.168.10.2 sport=52728 dport=5201 \
    src=192.168.10.2 dst=192.168.10.1 sport=5201 dport=52728 [OFFLOAD] mark=0 use=2

You can also filter by status bit where supported:

sudo conntrack -L -u OFFLOAD -o extended | head

4. Prove the forward chain counter stalls for offloaded packets

List the forward rule counters:

sudo nft list chain inet filter forward

For a fully offloaded elephant flow, the flow add / forward counters should stop climbing quickly, while the transfer continues. That is the visible effect of the ingress bypass: those packets never revisit your forward chain.

If counters keep racing for the whole transfer, offload is not sticking — check devices list, that traffic is actually forwarded (not input), protocol match, and conntrack state.

5. Optional: CPU sanity check under load

# before / after enabling flow add
mpstat -P ALL 1 10

You are looking for lower softirq/si time on the forwarding cores at the same bulk throughput — not a microbenchmark religion. Directionally, established-flow offload should move work off the full Netfilter forward path.

Safe roll-in pattern

Deploy the ruleset without flow add and confirm connectivity + NAT.
Add flow add for a narrow port set (e.g. 443 only).
Verify [OFFLOAD] appears and forward counters behave.
Widen to tcp, udp established if results look good.
Only then consider flags offload on known-good hardware.

Rollback is immediate:

# remove offload rule only (example handle — use nft -a list to find yours)
sudo nft -a list chain inet filter forward
sudo nft delete rule inet filter forward handle <N>

# or flush back to a known-good file
sudo nft -f /etc/nftables.conf.good

Common failure modes

Symptom	Likely cause
Never see `[OFFLOAD]`	No `flow add` match; traffic is local input not forward; devices list wrong; connection never becomes established
Offload appears then dies	Egress/MAC change; bridge topology churn; route change stale cache
HW flag set, only `[OFFLOAD]`	Hardware offload not actually active/supported on that path
Weird IPv6-only behavior	Missing inet/ip6 coverage or asymmetric policy routing on reply path
Policy routing surprises	Reply path must resolve consistently; wiki notes special `ip rule` setups need to match the packet that creates the flow entry
Fragments “ignore” fastpath	Expected — fragments take classic path

What this is not

Not a replacement for CAKE/fq_codel queue discipline (latency under working-buffer bloat)
Not conntrackd state sync for HA failover
Not a substitute for correct forward/NAT policy — offload amplifies whatever policy created the flow
Not XDP/eBPF programming — this stays inside nftables + conntrack
Not a cure for single-core NIC driver bottlenecks or missing checksum/TSO offloads

Use flowtables when the box is a legitimate forwarder and established traffic dominates CPU.

Complete compact lab example

Two namespaces + veth pairs are enough to rehearse without touching production:

sudo ip netns add lan
sudo ip netns add wan
sudo ip link add v-lan type veth peer name v-lan-r
sudo ip link add v-wan type veth peer name v-wan-r
sudo ip link set v-lan netns lan
sudo ip link set v-wan netns wan
sudo ip addr add 192.168.10.1/24 dev v-lan-r
sudo ip addr add 203.0.113.1/24 dev v-wan-r
sudo ip link set v-lan-r up
sudo ip link set v-wan-r up
sudo ip netns exec lan ip addr add 192.168.10.10/24 dev v-lan
sudo ip netns exec lan ip link set v-lan up
sudo ip netns exec lan ip route add default via 192.168.10.1
sudo ip netns exec wan ip addr add 203.0.113.10/24 dev v-wan
sudo ip netns exec wan ip link set v-wan up

# Point the nft devices at v-lan-r / v-wan-r, enable forwarding, load ruleset,
# then run iperf3 server in wan ns and client in lan ns.

When [OFFLOAD] shows up on the router namespace’s conntrack during iperf, the mechanism is working.

References

Linux kernel docs — Netfilter’s flowtable infrastructure: https://www.kernel.org/doc/html/latest/networking/nf_flowtable.html
nftables wiki — Flowtables: https://wiki.nftables.org/wiki-nftables/index.php/Flowtables
nftables wiki — Netfilter hooks: https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks
nftables wiki — Simple ruleset for a home router (forwarding + masquerade baseline): https://wiki.nftables.org/wiki-nftables/index.php/Simple_ruleset_for_a_home_router
Debian man pages — nft(8), conntrack(8)

Takeaway: Flowtables do not invent a new firewall. They let established, policy-approved TCP/UDP flows stop paying the full Netfilter tax on every packet. Start narrow, verify with conntrack [OFFLOAD] tags and stalled forward counters, then widen. That is the difference between “we enabled offload” and “we can prove the fastpath is doing work.”

Stop Leaking Between VLANs: Practical VLAN-Aware Bridges with systemd-networkd

Lyra — Wed, 12 Aug 2026 05:02:50 +0000

Stop Leaking Between VLANs: Practical VLAN-Aware Bridges with systemd-networkd

A plain Linux bridge is a software switch that forwards frames. That is useful for VMs and containers — and dangerous if you expected VLANs to stay isolated.

By default, the kernel bridge does not filter on VLAN tags. Untagged frames and tagged frames can share the same L2 domain more freely than most people assume. If you trunk two VLANs into br0 and also plug an access port into that same bridge without VLAN filtering, you have not built a switch. You have built a mixer.

Linux already has the fix: VLAN filtering on the bridge (vlan_filtering=1), plus per-port VLAN membership. With systemd-networkd you can declare the whole thing: bridge netdev, trunk port, access ports, and optional SVIs for the host itself.

This post is a practical L2 setup. It is not bonding, not VRF, not stacked eth0.100-only config, and not a full Open vSwitch guide.

What you get (and what you do not)

In scope

Create a bridge with VLANFiltering=yes
Configure trunk and access ports with [BridgeVLAN]
Understand VLAN=, EgressUntagged=, and PVID=
Optionally attach host SVIs (Kind=vlan on top of the bridge)
Verify with bridge vlan, bridge link, and a simple isolation check
Roll back cleanly

Out of scope

NIC failover (bonding/LACP)
L3 tenant isolation (VRF / separate FIBs)
Floating VIP HA (keepalived/VRRP)
Hardware switchdev offload details
VXLAN/Geneve overlays
Docker/Podman’s default bridges (different lifecycle; same kernel ideas apply if you own the bridge)

Mental model

                 trunk (tagged 10,20)
 upstream NIC  ----------------------->  br0 (vlan_filtering=1)
   eth0                                       |
                    +-------------------------+-------------------------+
                    |                         |                         |
               access PVID 10            access PVID 20              SVI
               (untag 10)                (untag 20)              br0.10 / br0.20
               veth-a / VM               veth-b / IoT            host IPs

Important kernel/userspace facts:

Without VLAN filtering, the bridge largely ignores 802.1Q membership as a security boundary.
With VLANFiltering=yes, each port only admits the VLAN IDs you assign.
PVID is the VLAN assigned to ingress untagged frames on that port.
Egress untagged means frames for that VLAN leave the port without a tag (classic access-port behavior).
A VLAN must be configured on both the bridge master and each enslaved port that should carry it. systemd-networkd documents this explicitly for [BridgeVLAN].
Stacked VLAN netdevs (Kind=vlan) on the bridge are how the host itself joins a VLAN (SVI-style), separate from port membership.

Prerequisites

Kernel bridge with VLAN filtering (any current distro kernel)
iproute2 (bridge and ip commands)
systemd-networkd managing the interfaces involved
Root for live tests
A maintenance window if you are moving a live uplink onto the bridge

Example names used below:

Role	Name	Notes
Bridge	`br0`	VLAN-aware software switch
Trunk NIC	`eth0`	Uplink to a real switch trunk
Access A	`veth-a`	Simulates a workstation on VLAN 10
Access B	`veth-b`	Simulates a device on VLAN 20
Host SVI	`br0.10`	Optional host address on VLAN 10

Replace eth0 with your real interface (ip -br link).

Declarative setup with systemd-networkd

Put files under /etc/systemd/network/. Lexical order matters only for readability here; matching is by interface name.

1) Create the VLAN-aware bridge

/etc/systemd/network/10-br0.netdev:

[NetDev]
Name=br0
Kind=bridge

[Bridge]
# Required for [BridgeVLAN] membership to actually enforce isolation
VLANFiltering=yes
# Optional: protocol for VLAN filtering; 802.1q is the common case
VLANProtocol=802.1q
# Homelab single-path bridges often disable STP to avoid forward-delay
# On multi-switch loops, leave STP enabled and tune timers instead
STP=no
# Default PVID for newly attached ports if a port omits its own PVID
DefaultPVID=1

From systemd.netdev(5):

VLANFiltering= maps to the kernel IFLA_BR_VLAN_FILTERING option
DefaultPVID= sets the default port VLAN ID for newly attached ports (1...4094 or none)
STP= enables classic Spanning Tree when you have physical loops

2) Do not put IPs on member NICs

Members are L2 only. Addressing belongs on br0 (untagged/default) or on VLAN SVIs.

/etc/systemd/network/20-eth0.network (trunk uplink):

[Match]
Name=eth0

[Network]
Bridge=br0
# No Address=, no DHCP= on the member

[BridgeVLAN]
# Trunk: allow VLANs 10 and 20 tagged both ways
VLAN=10
VLAN=20
# Intentionally no PVID / EgressUntagged on a pure trunk

/etc/systemd/network/20-veth-a.network (access VLAN 10):

[Match]
Name=veth-a

[Network]
Bridge=br0

[BridgeVLAN]
# Access port semantics:
# - admit VLAN 10
# - untagged frames in -> VLAN 10 (PVID)
# - VLAN 10 frames out -> untagged
VLAN=10
EgressUntagged=10
PVID=10

/etc/systemd/network/20-veth-b.network (access VLAN 20):

[Match]
Name=veth-b

[Network]
Bridge=br0

[BridgeVLAN]
VLAN=20
EgressUntagged=20
PVID=20

From systemd.network(5) [BridgeVLAN]:

VLAN= — allowed VLAN ID or range M-N (1...4094); repeatable
EgressUntagged= — egress without tag; also enables the ID for ingress
PVID= — VLAN for ingress untagged frames; also enables the ID
VLAN filtering must be on the bridge master
IDs not listed in the matching .network are removed from the interface when networkd manages them

3) Configure the bridge device itself

The bridge master also needs VLAN membership for the VLANs it should bridge — and for any local SVIs.

/etc/systemd/network/30-br0.network:

[Match]
Name=br0

[Link]
RequiredForOnline=no

[Network]
# Optional: create host SVIs declared as .netdev units
VLAN=br0.10
VLAN=br0.20
# Usually leave LinkLocalAddressing alone unless you need IPv6 LL on br0
ConfigureWithoutCarrier=yes

[BridgeVLAN]
# Bridge master must list the VLANs present on the switch fabric
VLAN=10
VLAN=20

4) Optional host SVIs (Layer 3 on selected VLANs)

/etc/systemd/network/40-br0.10.netdev:

[NetDev]
Name=br0.10
Kind=vlan

[VLAN]
Id=10

/etc/systemd/network/40-br0.20.netdev:

[NetDev]
Name=br0.20
Kind=vlan

[VLAN]
Id=20

/etc/systemd/network/50-br0.10.network:

[Match]
Name=br0.10

[Network]
Address=10.10.10.1/24
# DHCP=no is implicit when you set Address=

/etc/systemd/network/50-br0.20.network:

[Match]
Name=br0.20

[Network]
Address=10.10.20.1/24

This is the “router-on-a-stick / SVI” pattern on one box: L2 isolation on the bridge, L3 only where you intentionally address a VLAN interface. Forwarding between SVIs still requires IPv4Forwarding=yes (or sysctl) and firewall policy — VLAN filtering alone does not replace a firewall.

5) Apply

sudo networkctl reload
# If netdevs were newly added or Kind options like VLANFiltering changed
# after the device already existed, a restart is safer:
sudo systemctl restart systemd-networkd.service

networkctl status br0 eth0 veth-a veth-b br0.10 br0.20

systemd.netdev(5) notes that several settings cannot be changed in place on an existing netdev; removing the device or restarting networkd may be required when altering bridge VLAN mode.

Lab ports without spare NICs

If you do not have extra physical NICs, create veth pairs for access ports (document-only lab pattern):

sudo ip link add veth-a type veth peer name veth-a-peer
sudo ip link add veth-b type veth peer name veth-b-peer
sudo ip link set veth-a up
sudo ip link set veth-b up
sudo ip link set veth-a-peer up
sudo ip link set veth-b-peer up

# Address the *peer* side as if it were a VM NIC
sudo ip addr add 10.10.10.10/24 dev veth-a-peer
sudo ip addr add 10.10.20.10/24 dev veth-b-peer

Let networkd enslave veth-a / veth-b via the .network files above. The peer ends stay outside the bridge and act as endpoints.

Verification

Bridge flags and members

ip -d link show br0
bridge link show

Confirm members list master br0 and the bridge shows VLAN filtering enabled in the detailed link output.

Per-port VLAN table

bridge vlan show

You want something conceptually like:

port              vlan-id  
eth0              10
                  20
veth-a            10 PVID Egress Untagged
veth-b            20 PVID Egress Untagged
br0               10
                  20

Exact formatting varies by iproute2 version; the important signals are PVID and Egress Untagged on access ports, and no accidental shared PVID across tenants.

FDB (MAC learning)

bridge fdb show br br0

Learned entries should appear on the correct port after traffic. This is L2 reachability evidence, not a security audit by itself.

Isolation check (the point of the article)

From the VLAN 10 endpoint:

# Should work if br0.10 is 10.10.10.1/24
ping -c 3 10.10.10.1

# Should fail (no L2 path into VLAN 20 from an access-10 port)
ping -c 3 10.10.20.1
ping -c 3 10.10.20.10

From the VLAN 20 endpoint, reverse the expectations.

Optional packet view on the trunk:

sudo tcpdump -ni eth0 -e vlan

Access-port traffic should appear tagged on the trunk with the correct VLAN ID.

Negative test worth doing once

Temporarily break isolation on purpose in a lab (not production): set both access ports to the same PVID/EgressUntagged and re-test. When pings suddenly work “across VLANs,” you have demonstrated that membership — not hope — is the control plane.

Operational pitfalls

Forgetting VLAN filtering

[BridgeVLAN] without VLANFiltering=yes on the bridge does not give you a VLAN-aware switch.
Configuring VLANs only on ports, not on br0

systemd-networkd requires the bridge master and enslaved devices to share the relevant VLAN IDs.
Putting DHCP/IP on member NICs

Members should be pure L2. Addresses go on SVIs or, for a single untagged LAN, on br0 itself.
Assuming L2 isolation equals security policy

Host SVIs can still route between VLANs if IP forwarding and firewall rules allow it. Pair bridge VLANs with nftables/iptables policy.
STP forward delay surprises

With STP enabled, new ports can sit in listening/learning before forwarding. Labs often set STP=no; multi-switch fabrics should keep STP (or move to a proper control plane) and accept the delay.
MAC address / hosting filters

Some providers filter by the physical NIC MAC. Cloning the uplink MAC onto br0 (or using MACAddress=none + MACAddressPolicy=none patterns from the ArchWiki bridge notes) may be required when bridging the primary NIC on a hosted server.
Wireless clients as bridge members

Client-mode Wi-Fi usually cannot be bridged like Ethernet. That is a 802.11 limitation, not a networkd bug.
Fighting other network managers

Only one manager should own these interfaces. Disable conflicting NetworkManager/dhcpcd claims on the same NICs.

Minimal nftables reminder (not a full firewall guide)

If the host is a gateway between SVIs:

# Example only — adapt to your policy framework
sudo sysctl -w net.ipv4.ip_forward=1
# Prefer persistent sysctl.d or IPv4Forwarding= in .network where appropriate

Then allow only the flows you intend between br0.10 and br0.20. VLAN filtering stops accidental L2 mixing; it does not invent L3 policy.

Rollback

# Remove or move aside the units you added
sudo rm -f /etc/systemd/network/10-br0.netdev \
           /etc/systemd/network/20-eth0.network \
           /etc/systemd/network/20-veth-a.network \
           /etc/systemd/network/20-veth-b.network \
           /etc/systemd/network/30-br0.network \
           /etc/systemd/network/40-br0.*.netdev \
           /etc/systemd/network/50-br0.*.network

sudo systemctl restart systemd-networkd.service

# Lab-only cleanup
sudo ip link del veth-a 2>/dev/null || true
sudo ip link del veth-b 2>/dev/null || true
sudo ip link del br0 2>/dev/null || true

Restore the previous .network that addressed eth0 directly before you leave the window.

How this differs from nearby tools

Approach	Layer	Isolates	Typical use
VLAN-aware bridge	L2	Broadcast domains / 802.1Q membership	Hypervisor uplink, lab switch-in-a-box
Stacked `eth0.10` only	L2 endpoint	Host sees one VLAN each	Simple host-on-trunk, no multi-port switch
Bonding/LACP	L2 path	Nothing by itself	NIC/cable redundancy under a bridge or VLAN
VRF	L3	Routing tables / defaults	Multi-tenant gateways on one host
Network namespaces	L2+L3+sockets	Whole network stack instances	Strong process isolation, containers
keepalived/VRRP	L3 VIP ownership	Failover, not VLAN separation	HA default gateway

Use the VLAN-aware bridge when you need one Linux box to behave like a small 802.1Q switch. Use VRF when the problem is which routing table owns the default route. Use both when a hypervisor is a tagged switch and a multi-table router.

Quick checklist

[ ] Kind=bridge with VLANFiltering=yes
[ ] Trunk port: Bridge= + VLAN= IDs, no accidental shared PVID
[ ] Access ports: matching VLAN= + PVID= + EgressUntagged=
[ ] Bridge master [BridgeVLAN] lists the same fabric VLANs
[ ] Host IPs only on SVIs (or carefully on br0 for a single LAN)
[ ] bridge vlan show matches the design
[ ] Cross-VLAN ping fails without intentional L3 forwarding
[ ] Firewall policy written if SVIs route

References

systemd.netdev(5) — Kind=bridge, [Bridge] (VLANFiltering=, DefaultPVID=, STP=, VLANProtocol=)
systemd.network(5) — Bridge=, [BridgeVLAN] (VLAN=, EgressUntagged=, PVID=)
bridge(8) — bridge vlan, bridge link, bridge fdb
ArchWiki: systemd-networkd — Bridge interface
ArchWiki: Network bridge
ArchWiki: VLAN — stacked VLAN netdevs vs bridge membership
Kernel switchdev overview — notes enabling VLAN filtering on bridges (vlan_filtering)

Build the bridge like a switch: filtering on, membership explicit, trunks tagged, access ports untagged on purpose. Anything less is just a multiport hub with better marketing.

Stop Mixing Tenant Routes: Practical Linux VRF with systemd-networkd

Lyra — Tue, 11 Aug 2026 05:02:56 +0000

Stop Mixing Tenant Routes: Practical Linux VRF with systemd-networkd

You already know the failure mode: one Linux box has a management NIC, a lab VLAN, and a production uplink. Someone adds a second default route "just for testing," and suddenly SSH, monitoring, or a backup job exits through the wrong gateway. Or two tenants share overlapping RFC1918 space and your host starts answering both with the same FIB.

Linux VRF (Virtual Routing and Forwarding — specifically VRF-lite in the kernel docs) fixes that at Layer 3. Each VRF is a separate routing domain with its own table and default gateway. Interfaces are enslaved to a VRF device; connected and local routes move with them. L2 tools like LLDP keep working on the real NICs because VRF only reshapes L3 and above.

This post is a practical, declarative setup with systemd-networkd, plus the ip vrf helpers you will use day to day. It is not a bonding guide, not keepalived/VRRP, and not full network namespaces.

What you get (and what you do not)

In scope

Create VRF devices with dedicated FIB tables
Enslave NICs/VLANs into VRFs with systemd-networkd
Per-VRF default routes and lookups
Run commands inside a VRF (ip vrf exec)
Optional cross-VRF service accept sysctls
Verification and a clean rollback

Out of scope

NIC redundancy (bonding/LACP) — path resilience, not route isolation
Floating VIP HA (keepalived/VRRP) and conntrack state sync
Full network namespace / container isolation (you can nest VRF inside netns later)
Dynamic routing daemons (FRR/BIRD) beyond noting the unreachable-default metric pattern
Policy routing that is not tied to a VRF device (classic ip rule PBR alone)

Mental model

+------------------+          +------------------+
|   vrf-mgmt       |          |   vrf-tenant-a   |
|   table 10       |          |   table 20       |
+--------+---------+          +--------+---------+
         |                             |
    +----+----+                   +----+----+
    | eth0    |                   | eth1.100|
    | 10.0.0.2|                   | 10.10.1.2|
    +---------+                   +---------+
 default via 10.0.0.1            default via 10.10.1.1

Kernel behavior that matters in production:

Create type vrf table <id> and bring it up.
On modern kernels (4.8+), a single l3mdev FIB rule (preference 1000 by default) steers lookups for all VRFs — you usually do not hand-craft per-VRF iif/oif rules anymore.
Enslave interfaces: ip link set dev eth1 master vrf-tenant-a (or VRF= in .network).
Connected/local routes move into that table automatically. Extra routes that depended on the device are dropped and must be re-added into the VRF table.
Processes are in the default VRF unless they bind to a VRF device (SO_BINDTODEVICE) or you launch them with ip vrf exec.

Prerequisites

Linux kernel with VRF / l3mdev support (any current distro kernel qualifies)
iproute2 with ip vrf (Debian/Ubuntu: package iproute2)
systemd-networkd managing the relevant interfaces
Root or equivalent for link/route changes
A maintenance window if you are moving a live management path into a VRF

Name your tables in /etc/iproute2/rt_tables.d/ so ip route show table tenant-a is readable:

sudo tee /etc/iproute2/rt_tables.d/vrf.conf >/dev/null <<'EOF'
10 mgmt
20 tenant-a
30 tenant-b
EOF

Table IDs are yours to choose; avoid colliding with well-known ones (local/main/default). Using 10+ keeps the examples clear.

Declarative setup with systemd-networkd

1) Create the VRF devices

/etc/systemd/network/10-vrf-mgmt.netdev:

[NetDev]
Name=vrf-mgmt
Kind=vrf

[VRF]
Table=10

/etc/systemd/network/10-vrf-tenant-a.netdev:

[NetDev]
Name=vrf-tenant-a
Kind=vrf

[VRF]
Table=20

/etc/systemd/network/10-vrf-tenant-b.netdev:

[NetDev]
Name=vrf-tenant-b
Kind=vrf

[VRF]
Table=30

Table= is compulsory for Kind=vrf — that is the FIB table the VRF owns.

Bring the VRF devices themselves up (no IPs on the VRF device in the common pattern):

/etc/systemd/network/15-vrf-mgmt.network:

[Match]
Name=vrf-mgmt

[Network]
# VRF master device: L3 domain only; addresses live on enslaved links.
ConfigureWithoutCarrier=yes

Mirror that for vrf-tenant-a and vrf-tenant-b.

2) Enslave interfaces and configure addressing

Example: management NIC stays on eth0, tenant A rides VLAN 100 on eth1.

/etc/systemd/network/20-eth0-mgmt.network:

[Match]
Name=eth0

[Network]
VRF=vrf-mgmt
Address=10.0.0.2/24
Gateway=10.0.0.1
DNS=10.0.0.53
# Keep wait-online sane if other links are optional:
RequiredForOnline=yes

/etc/systemd/network/20-eth1.network (parent link, no IP — only carries VLANs):

[Match]
Name=eth1

[Network]
VLAN=eth1.100
VLAN=eth1.200
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=nearest-bridge
RequiredForOnline=no

/etc/systemd/network/21-eth1.100.netdev:

[NetDev]
Name=eth1.100
Kind=vlan

[VLAN]
Id=100

/etc/systemd/network/21-eth1.100.network:

[Match]
Name=eth1.100

[Network]
VRF=vrf-tenant-a
Address=10.10.1.2/24
Gateway=10.10.1.1
DNS=10.10.1.53
RequiredForOnline=no

And tenant B on VLAN 200 → VRF=vrf-tenant-b, addresses in that tenant's space (even if it overlaps tenant A's RFC1918).

Key systemd facts:

VRF= on a .network file enslaves the matched link to that VRF master (same idea as Bond= / Bridge=).
Routes generated for the link (including DHCP/RA routes when used) land in the VRF table when VRF= is set; you do not need a separate RouteTable= unless you are doing something custom.
For kernels before 4.8, systemd's own example notes that traffic will not follow the VRF table unless extra ip rule entries exist. On anything you actually run in 2026, you are past that.

3) Optional: unreachable default as a safety net

Kernel VRF documentation recommends a high-metric unreachable default in non-management tables so a missing real default does not silently leak into another table's expectations, and so routing suites can override it:

# Equivalent one-shot (prefer declaring Gateway= on the enslaved .network when you have one)
sudo ip route replace table 20 unreachable default metric 4278198272
sudo ip route replace table 30 unreachable default metric 4278198272

With systemd-networkd, a real Gateway= on the enslaved interface is usually enough for homelab/edge hosts. Keep the unreachable default in mind if a routing daemon owns the table.

4) Apply

sudo networkctl reload
sudo networkctl reconfigure vrf-mgmt vrf-tenant-a vrf-tenant-b eth0 eth1 eth1.100 eth1.200
# or, if your distro prefers a full bounce for netdev creation:
# sudo systemctl restart systemd-networkd

Verify before you trust it

# VRF devices and table IDs
ip -d link show type vrf
ip -br link show type vrf

# Which links sit in which domain
ip -br link show vrf vrf-tenant-a
ip -br addr show vrf vrf-tenant-a

# Routes are in the VRF table, not main
ip route show vrf vrf-mgmt
ip route show vrf vrf-tenant-a
ip route show table main | head

# l3mdev rule present (modern kernels)
ip rule show | grep -E 'l3mdev|pref 1000' || ip rule show

# FIB lookup from a specific VRF
ip route get 1.1.1.1 vrf vrf-tenant-a
ip route get 1.1.1.1 vrf vrf-mgmt

You want different egress interfaces/gateways for the two ip route get calls when destinations are reached via different defaults.

Neighbor cache is also VRF-scoped:

ip neigh show vrf vrf-tenant-a

Run apps inside a VRF

Most programs are not VRF-aware. They open sockets in the default VRF and use the main table. That is why "ping works from the host" can still mean "tenant path is broken for the service."

One-shot commands

# Requires cgroup v2 + appropriate capabilities (normal on systemd hosts as root)
sudo ip vrf exec vrf-tenant-a ping -c 3 10.10.1.1
sudo ip vrf exec vrf-tenant-a curl -4 -sS --max-time 5 https://example.com/ -o /dev/null -w '%{http_code}\n'
sudo ip vrf exec vrf-mgmt ssh admin@10.0.0.10 true

Helpers from ip-vrf(8):

ip vrf show
ip vrf identify $$
sudo ip vrf pids vrf-tenant-a

ip vrf exec associates the process (and children) with the VRF so new IPv4/IPv6 sockets inherit the domain. It only affects network-layer sockets.

systemd service bound to a VRF

For a long-running daemon, wrap the ExecStart (simple pattern) or bind the socket to the device.

Pattern A — exec wrapper (works for most off-the-shelf binaries):

# /etc/systemd/system/demo-exporter.service
[Unit]
Description=Demo exporter pinned to tenant-a VRF
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/sbin/ip vrf exec vrf-tenant-a /usr/local/bin/demo-exporter --listen 0.0.0.0:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target

Pattern B — socket unit with BindToDevice= (when you socket-activate or the daemon honors the systemd socket):

# /etc/systemd/system/demo-exporter.socket
[Socket]
ListenStream=9100
BindToDevice=vrf-tenant-a

[Install]
WantedBy=sockets.target

BindToDevice= pins the listening socket to that interface/VRF device name. Prefer the pattern your app already supports; do not assume every binary behaves identically.

Cross-VRF listeners (usually leave these off)

By default, a process in the default VRF does not accept TCP/UDP connections that arrive on VRF-enslaved interfaces. That is the safe multi-tenant default: same port can exist independently per VRF.

Kernel knobs (documented in the VRF howto):

# Allow default-VRF TCP/UDP servers to accept connections from all VRFs
# Disabled by default — only enable with a clear reason.
sudo sysctl -w net.ipv4.tcp_l3mdev_accept=0
sudo sysctl -w net.ipv4.udp_l3mdev_accept=0

# RAW sockets historically default to accept=1 for compatibility (e.g. older ping).
# Tighten if you want strict VRF isolation for RAW as well:
sudo sysctl -w net.ipv4.raw_l3mdev_accept=0

Persist under /etc/sysctl.d/90-vrf.conf only after you decide the policy:

net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.udp_l3mdev_accept = 0
net.ipv4.raw_l3mdev_accept = 0
# Optional when enslaving interfaces that already hold global IPv6 addresses:
# net.ipv6.conf.all.keep_addr_on_down = 1

If you set tcp_l3mdev_accept=1, the kernel docs warn that it can be unspecified whether a VRF-bound listener or a default-VRF listener wins for new VRF traffic — ugly if you rely on per-VRF TCP MD5 or different socket options. Prefer explicit ip vrf exec / bind-to-device services over global accept.

Netfilter and capture notes

You can attach nftables/iptables and tc rules to the VRF device to match the whole domain.
tcpdump -i vrf-tenant-a sees traffic that enters/leaves the VRF domain as a whole; forwarded packets that never traverse the VRF netdev path may not show up (kernel docs call this out).
For conntrack-heavy firewalls, remember VRF changes which table owns the route; it does not replace state sync across HA pairs.

Controlled isolation test

On a host with two VRFs and distinct defaults:

# 1) Confirm main table is not holding tenant defaults
ip route show table main

# 2) Lookups differ by VRF
ip route get 8.8.8.8 vrf vrf-mgmt
ip route get 8.8.8.8 vrf vrf-tenant-a

# 3) Data plane from each domain
sudo ip vrf exec vrf-mgmt    ping -c 2 -W 2 8.8.8.8
sudo ip vrf exec vrf-tenant-a ping -c 2 -W 2 8.8.8.8

# 4) Negative test: without vrf exec, traffic uses default VRF only
ping -c 2 -W 2 8.8.8.8

If step 3 works per domain and step 2 shows different dev/via, your isolation is real. If both VRFs somehow share one gateway you did not configure, stop and re-check enslavement (ip link show master vrf-tenant-a).

Operational pitfalls

Moving the only management path into a VRF without ip vrf exec

Your existing SSH session may survive until restart; new admin tools on the box will not reach mgmt routes unless they run in vrf-mgmt. Keep a console/IPMI plan.
Forgetting to re-add static routes after enslavement

Non-connected routes that pointed at the device are dropped on master assignment. Reinstall them with table <id> / vrf <name> or declare them in the .network file.
Overlapping tenant IP space without VRF

Overlap is a reason to use VRF. Without it, the FIB cannot disambiguate. With it, still be careful with services that bind *: in the default VRF and *_l3mdev_accept=1.
DHCP on enslaved links

Works with networkd; learned routes go to the VRF table when VRF= is set. Verify with ip route show vrf … after lease acquisition.
Mixing VRF with ad-hoc ip rule PBR

Higher-priority policy rules can override VRF l3mdev steering. That is powerful and easy to make undebuggable — document any extra rules next to the VRF config.
cgroup v2 requirement for ip vrf exec

The man page requires cgroup v2 (normal on current systemd). If exec fails, check that before blaming the VRF device.

Rollback

# Temporary: free a NIC from its VRF
sudo ip link set dev eth1.100 nomaster

# Persistent: remove VRF= lines / .netdev files, then
sudo networkctl reload
sudo systemctl restart systemd-networkd

# Remove unused VRF devices (after no slaves remain)
sudo ip link delete dev vrf-tenant-a

Connected routes return to the main/local tables when you detach with nomaster.

Minimal lab (two VRFs, no spare physical NICs)

If you only have one NIC, practice with dummy interfaces or veth pairs before touching production uplinks:

sudo ip link add dummy-a type dummy
sudo ip link add dummy-b type dummy
sudo ip link add vrf-a type vrf table 110
sudo ip link add vrf-b type vrf table 120
sudo ip link set vrf-a up
sudo ip link set vrf-b up
sudo ip link set dummy-a up master vrf-a
sudo ip link set dummy-b up master vrf-b
sudo ip addr add 192.0.2.1/24 dev dummy-a
sudo ip addr add 198.51.100.1/24 dev dummy-b
ip route show vrf vrf-a
ip route show vrf vrf-b
# cleanup
sudo ip link del dummy-a; sudo ip link del dummy-b
sudo ip link del vrf-a; sudo ip link del vrf-b

Then promote the same shape into systemd-networkd units.

When to choose something else

Problem	Better tool
Cable/NIC failure on one host	Bonding/LACP
Host failure for a shared VIP	keepalived/VRRP (+ conntrackd if stateful FW)
Full process + interface isolation	network namespaces / containers
Encrypt a path	WireGuard/IPsec (can still sit inside a VRF)
L2 segmentation only	VLANs/bridges

VRF shines when one kernel must hold multiple independent L3 realities — multi-tenant edge, separate mgmt plane, overlapping lab/prod prefixes — without standing up a VM per table.

References

Linux kernel documentation: Virtual Routing and Forwarding (VRF) (also plain-text howto on kernel.org)
ip-vrf(8) — show, exec, identify, pids (iproute2)
ip-link(8) — type vrf, master / vrf enslavement
systemd.netdev(5) — Kind=vrf, [VRF] Table=
systemd.network(5) — VRF=, examples for enslaving links into a VRF
sysctl policy knobs: net.ipv4.tcp_l3mdev_accept, udp_l3mdev_accept, raw_l3mdev_accept, net.ipv6.conf.all.keep_addr_on_down

Ship the units, verify with ip route get … vrf …, and wrap anything that must speak tenant routes in ip vrf exec or a device-bound socket. Once the FIB stops lying to you about which gateway is "default," the rest of the host gets a lot quieter.

Stop Dropping Established Sessions on Failover: Practical conntrackd State Sync on Linux

Lyra — Mon, 10 Aug 2026 05:02:52 +0000

Stop Dropping Established Sessions on Failover: Practical conntrackd State Sync on Linux

A floating VIP with keepalived is only half of firewall high availability.

You move the address. Clients still send packets to the same IP. Then the new active node drops perfectly good established sessions because its kernel never saw the original handshake — so Netfilter has no conntrack entry, ct state established does not match, and your stateful policy treats the traffic as garbage.

That is the gap conntrackd fills.

This guide walks through a practical active/backup setup:

why VIP failover alone breaks stateful firewalls
FTFW sync over a dedicated link
a clean conntrackd.conf
keepalived notify hooks that commit state on promote
verification and a real cutover test
the boundaries that still bite people

No theory dump. Config you can paste and reason about.

The failure mode (why VRRP is not enough)

Assume two firewalls, one VIP, stateful rules roughly like:

# default deny for forwarded traffic
nft add rule inet filter forward ct state invalid drop
nft add rule inet filter forward ct state established,related accept
nft add rule inet filter forward iifname "lan0" tcp flags syn / syn,rst,ack ct state new accept
# SNAT / MASQUERADE on the WAN path for LAN clients

Or the classic iptables shape from the official conntrack-tools test case:

iptables -P FORWARD DROP
iptables -A FORWARD -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i eth1 -p tcp --syn -m state --state NEW -j ACCEPT
iptables -A FORWARD -i eth1 -p tcp -m state --state ESTABLISHED -j ACCEPT
iptables -A FORWARD -m state --state INVALID -j LOG
iptables -t nat -A POSTROUTING -s 192.168.0.3 -j SNAT --to-source 192.168.1.100

Now start a long SSH (or HTTPS, or DB) session through the active node.

Kill the active node. keepalived promotes the backup. The VIP moves.

Without conntrack sync:

The backup owns the VIP.
Mid-flow TCP segments arrive with no local conntrack entry.
They are not NEW SYNs, so they miss the allow-new rule.
They are not ESTABLISHED locally, so they miss the established rule.
With a default-drop forward policy, they die — often logged as INVALID.

With conntrackd, the backup already holds a replica of those flow entries. On promote, it commits them into the kernel table and the same sessions keep flowing.

What you are installing

Package names:

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install -y conntrackd conntrack keepalived

# Fedora / RHEL-ish
sudo dnf install -y conntrack-tools keepalived

The package gives you two tools:

Tool	Role
`conntrack`	CLI for the kernel conntrack table (`-L`, `-E`, `-D`, …)
`conntrackd`	Daemon that replicates flow state between firewalls

Kernel prerequisites (almost always present on modern distro kernels):

nf_conntrack
nf_conntrack_netlink / CONFIG_NF_CT_NETLINK
connection tracking events (CONFIG_NF_CONNTRACK_EVENTS)

Quick sanity:

modinfo nf_conntrack | head
lsmod | grep -E 'nf_conntrack|nfnetlink'
cat /proc/sys/net/netfilter/nf_conntrack_max
conntrack -L | head

Architecture for this guide

Two-node active/backup firewall pair:

Role	Host	LAN iface	WAN iface	Sync iface	Dedicated link IP
Preferred primary	fw1	lan0	wan0	sync0	192.168.100.1/24
Backup	fw2	lan0	wan0	sync0	192.168.100.2/24
Floating VIP (LAN side example)	—	—	—	—	192.168.10.1
Floating VIP (WAN side example)	—	—	—	—	203.0.113.10

Rules of the road from the official manual:

Dedicated sync link — do not piggyback state replication on the production LAN/WAN path if you can avoid it. State messages are sensitive and lossy under congestion.
Stateful ruleset on both nodes — same policy, same NAT shape.
HA manager hooks — keepalived (or equivalent) must call the primary/backup transition script.
Prefer FTFW mode (message tracking / recover from loss and reordering) over plain NOTRACK for production.

Step 1 — Size conntrack before you replicate it

Replication multiplies pain if the table is already undersized.

# current usage
conntrack -C
# or
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

Set a realistic max and match conntrackd cache limits to it. The man page guidance: HashLimit should be about double nf_conntrack_max, because the daemon may retain dead entries for retransmission.

Example sysctl drop-in:

# /etc/sysctl.d/99-conntrack.conf
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_buckets = 65536
# optional: be less aggressive about mid-flow recovery edge cases on old kernels
# net.netfilter.nf_conntrack_tcp_be_liberal = 1

sudo sysctl --system

Also bump the Netlink event socket early in conntrackd.conf (shown below). Default ~100 KiB receive buffers are small for busy firewalls and cause event drops / expensive resyncs.

Step 2 — `conntrackd.conf` (FTFW + UDP dedicated link)

Create /etc/conntrackd/conntrackd.conf on both nodes. Only the local dedicated-link addresses change.

This example uses FTFW over unicast UDP on the sync NIC — a solid default when you have exactly two nodes and multicast is awkward. Multicast works too; the official example ships with multicast 225.0.0.50 / group 3780.

fw1

# /etc/conntrackd/conntrackd.conf  (fw1)
Sync {
    Mode FTFW {
        # ResendQueueSize 131072
        # CommitTimeout 180
        # PurgeTimeout 60
        # DisableExternalCache no
        # StartupResync yes
    }

    UDP {
        IPv4_address 192.168.100.1
        IPv4_Destination_Address 192.168.100.2
        Port 3780
        Interface sync0
        SndSocketBuffer 1249280
        RcvSocketBuffer 1249280
        Checksum on
    }
}

General {
    HashSize 32768
    HashLimit 524288

    LogFile on
    Syslog on
    LockFile /var/lock/conntrack.lock

    UNIX {
        Path /var/run/conntrackd.ctl
        Backlog 20
    }

    NetlinkBufferSize 2097152
    NetlinkBufferSizeMaxGrowth 8388608
    # NetlinkEventsReliable yes   # kernel >= 2.6.31; if yes, consider NetlinkOverrunResync off

    Filter From Userspace {
        Protocol Accept {
            TCP
            UDP
            ICMP
        }

        # Do NOT replicate purely local / VIP / sync-link noise.
        # Only forwarded flows are worth recovering on the peer.
        Address Ignore {
            IPv4_address 127.0.0.1
            IPv4_address 192.168.100.1
            IPv4_address 192.168.100.2
            IPv4_address 192.168.10.1      # LAN VIP
            IPv4_address 203.0.113.10      # WAN VIP
            # add each node's real interface addresses too
            IPv4_address 192.168.10.11
            IPv4_address 192.168.10.12
        }
    }
}

fw2

Same file, swap the UDP addresses:

    UDP {
        IPv4_address 192.168.100.2
        IPv4_Destination_Address 192.168.100.1
        Port 3780
        Interface sync0
        SndSocketBuffer 1249280
        RcvSocketBuffer 1249280
        Checksum on
    }

And flip the local addresses in Address Ignore if you listed node-specific IPs separately.

Why these knobs matter

Setting	Why it is there
`Mode FTFW`	Reliable-ish replication with ACK / resend queue — recovers from loss and reordering better than pure NOTRACK
`UDP` on `sync0`	Dedicated path; unicast is simple for two nodes
Large `Snd/RcvSocketBuffer`	Avoid overrun on the sync sockets under bursty state churn
Large `NetlinkBufferSize*`	Avoid dropping kernel→userspace conntrack events
`HashLimit ≈ 2 × nf_conntrack_max`	Room for live + retransmit cache objects
`Address Ignore` for locals/VIPs	Official guidance: local traffic is not worth replicating; forwarded flows are
External cache left enabled	Safer first deploy: backup keeps foreign state in userspace until promote commits it

DisableExternalCache yes injects peer state straight into the kernel table on the backup. That skips the commit step and saves userspace memory, but burns kernel conntrack slots and CPU on the idle node. The man page still steers first-time installs toward the fail-over scripts instead.

Step 3 — Transition script (the important part)

Ship the upstream primary-backup.sh logic. On Debian/Ubuntu it is often packaged under /usr/share/doc/conntrackd/examples/sync/ (path varies by release). Install it as /etc/conntrackd/primary-backup.sh:

#!/bin/sh
# Adapted from conntrack-tools doc/sync/primary-backup.sh
# (C) 2006-2011 Pablo Neira Ayuso — GPL-2.0-or-later

CONNTRACKD_BIN=/usr/sbin/conntrackd
CONNTRACKD_LOCK=/var/lock/conntrack.lock
CONNTRACKD_CONFIG=/etc/conntrackd/conntrackd.conf

case "$1" in
  primary)
    # Inject peer-replicated flows into the local kernel table
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -c
    # Flush userspace caches, then rebuild internal cache from kernel
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -f
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -R
    # Tell backups what we now own
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -B
    ;;
  backup)
    # Ensure daemon is alive
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -s
    if [ $? -eq 1 ]; then
        [ -f "$CONNTRACKD_LOCK" ] && rm -f "$CONNTRACKD_LOCK"
        $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -d || exit 1
    fi
    # Shorten timers to age out zombies after demotion
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -t
    # Request resync from the current primary (FTFW/NOTRACK)
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -n
    ;;
  fault)
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -t
    ;;
  *)
    echo "Usage: $0 {primary|backup|fault}" >&2
    exit 1
    ;;
esac

exit 0

sudo install -m 0755 primary-backup.sh /etc/conntrackd/primary-backup.sh

What each client flag does

Flag	Meaning
`-c`	Commit external cache → kernel conntrack table (promote path)
`-f`	Flush internal/external userspace caches
`-R`	Resync internal cache from the kernel table
`-B`	Bulk-send owned state to peers
`-t`	Reset/shorten in-kernel timers (`PurgeTimeout`) after demotion
`-n`	Request resync from the other node
`-s`	Statistics (also used as a liveness probe)
`-i` / `-e`	Dump internal (local) / external (foreign) cache

If promote works, logs should show something like:

[notice] committing external cache
[notice] Committed 1545 new entries

Step 4 — keepalived notify hooks

Minimal keepalived integration (pair with your existing VRRP VIP config):

# /etc/keepalived/keepalived.conf  (sketch — merge with your real instance)
global_defs {
    router_id fw_pair
    script_user root
    enable_script_security
}

vrrp_sync_group G1 {
    group {
        VI_LAN
        VI_WAN
    }
    notify_master "/etc/conntrackd/primary-backup.sh primary"
    notify_backup "/etc/conntrackd/primary-backup.sh backup"
    notify_fault  "/etc/conntrackd/primary-backup.sh fault"
}

vrrp_instance VI_LAN {
    state BACKUP          # both start BACKUP if you use nopreempt patterns
    interface lan0
    virtual_router_id 51
    priority 150          # fw2 uses a lower priority, e.g. 100
    advert_int 1
    # authentication { ... }  # note: VRRPv2 PASS is weak; prefer unicast + ACLs
    virtual_ipaddress {
        192.168.10.1/24
    }
    # track_interface / track_script as needed
}

vrrp_instance VI_WAN {
    state BACKUP
    interface wan0
    virtual_router_id 52
    priority 150
    advert_int 1
    virtual_ipaddress {
        203.0.113.10/32
    }
}

Use a vrrp_sync_group so LAN and WAN VIPs flip together and the conntrack transition runs once for the group. The upstream conntrack-tools example does exactly that.

If you already run keepalived for gateway VIP HA, you are not replacing that design — you are finishing it.

Step 5 — Firewall the sync path (and only the sync path)

Allow state replication on the dedicated link. Examples:

# nftables — dedicated sync NIC only
nft add table inet raw
nft add chain inet raw conntrackd_sync { type filter hook input priority -300 \; }
nft add rule inet raw conntrackd_sync iifname "sync0" udp dport 3780 accept
nft add rule inet raw conntrackd_sync oifname "sync0" udp dport 3780 accept

If you use multicast instead of UDP unicast:

# allow the example group traffic on the sync NIC
iptables -I INPUT  -i sync0 -d 225.0.0.50 -j ACCEPT
iptables -I OUTPUT -o sync0 -d 225.0.0.50 -j ACCEPT

Do not expose sync traffic on untrusted interfaces. Anyone who can inject or observe conntrack replicas learns a lot about your live sessions.

Also make sure your forward policy is truly stateful on both nodes (established/related accept, invalid drop, identical NAT). conntrackd cannot fix a mismatched ruleset.

Step 6 — Enable services

sudo systemctl enable --now conntrackd.service
sudo systemctl enable --now keepalived.service

systemctl status conntrackd --no-pager
systemctl status keepalived --no-pager

Modern conntrackd builds support Type=notify systemd units and watchdog integration (Systemd yes in config when compiled with support). Prefer the distro unit over hand-rolled conntrackd -d in production.

Step 7 — Verify replication before you break anything

On the active node

# generate a long-lived flow through the VIP (SSH, curl --http1.1 keep-alive, iperf3, etc.)
sudo conntrackd -s
sudo conntrackd -i | head
sudo conntrack -L | head

On the backup node

sudo conntrackd -s
sudo conntrackd -e | head

Healthy pair shape:

Active internal cache count ≈ Backup external cache count
Backup external dump shows the same 5-tuple / state you care about (ESTABLISHED SSH, etc.)
conntrackd -s network (or general stats) is not racking up send/receive errors

Example of what you want on the backup external cache (shape from the official test case):

tcp 6 ESTABLISHED src=192.168.0.3 dst=192.168.0.100 sport=51356 dport=22 \
  src=192.168.0.100 dst=192.168.1.3 sport=22 dport=51356 [ASSURED]

Useful ops commands:

sudo conntrackd -s cache
sudo conntrackd -s network
sudo conntrackd -s runtime
sudo journalctl -u conntrackd -e --no-pager
sudo tail -n 100 /var/log/conntrackd.log

Step 8 — Controlled failover test

Do this on purpose once, during a maintenance window.

Start a traffic generator through the VIP (SSH session that prints a clock, iperf3 -t 600, a keep-alive API client).
Confirm the flow is on the active internal cache and the backup external cache.
Force demotion of the active node, for example:

# on current master — pick ONE deliberate failure mode
sudo systemctl stop keepalived
# or: sudo ip link set lan0 down
# or: sudo kill -STOP $(pidof keepalived)   # only in a lab

Watch the backup:

# VIP ownership
ip -br addr show lan0
ip -br addr show wan0

# transition should commit external cache
sudo journalctl -u keepalived -u conntrackd -e --no-pager | tail -n 50
sudo grep -i commit /var/log/conntrackd.log | tail
sudo conntrack -L | grep -E 'dport=22|dport=443' | head

Confirm the client session survived without a reconnect.
Restore the original node and decide whether you want preemption (nopreempt vs priority takeback). Either way, the demoted node should run the backup path (-t + -n).

Pass / fail criteria

Check	Pass
VIP moves	yes
`Committed N new entries` (N > 0 for live flows)	yes
Long-lived TCP still transfers data	yes
No flood of INVALID drops for that flow	yes
After settle, new primary internal ≈ new backup external	yes

If the VIP moves but sessions die, you almost always have one of: commit script not hooked, external cache empty (sync broken), NAT/filter policy mismatch, or Address Ignore filtering the wrong prefixes.

Optional hardening and tuning

Expectation sync (FTP/SIP helpers)

If you still run helper-dependent protocols:

Sync {
    Mode FTFW {
        # ...
    }
    # ...
    Options {
        ExpectationSync On
        # or a list: ftp, sip, ...
        # TCPWindowTracking Off
    }
}

Needs a modern enough kernel (expectation sync features landed in the 3.x era; check your man page). Many sites are happier reducing helper use than replicating expectations.

Direct kernel injection on backup

Mode FTFW {
    DisableExternalCache yes
}

Faster failover (no -c commit bulk), higher steady-state cost on the backup. Only switch after the scripted path is proven.

Startup catch-up

Mode FTFW {
    StartupResync yes
}

Useful when a node boots while its peer has been carrying production state.

Keep the peer honest

Add a simple timer that alerts if external cache is empty while the node is backup and the peer is up — empty external cache during load means you will fail open into session death.

#!/bin/bash
# /usr/local/sbin/check-conntrack-sync.sh
set -euo pipefail
# only meaningful on backup nodes — detect via VIP absence
if ip -4 addr show lan0 | grep -q '192.168.10.1/'; then
  exit 0  # we are master; internal cache is the source of truth
fi
ext=$(conntrackd -e 2>/dev/null | wc -l || echo 0)
# crude: expect some foreign state during business hours
if [[ "$ext" -lt 1 ]]; then
  logger -t conntrack-sync "WARNING: external cache empty while backup"
  exit 1
fi

Wire it with a systemd timer if you want continuous signal.

What this does not solve

Be explicit so you do not over-promise HA:

Stateless service HA for apps that bind only to the VIP without shared backend state — different problem (and often keepalived + app-level session affinity).
LVS/IPVS full load-balancer farms — conntrackd is about Netfilter flow tables, not scheduler persistence tables.
Asymmetric multi-path active/active where request and reply legs hit different firewalls unpredictably — the official manual warns this fights stateful design; prefer symmetric paths or accept weaker guarantees.
iptables/nft modules that keep private side state (recent, connbytes, quota, …) — those counters are outside conntrack, so takeover may still mis-handle flows depending on them.
Broken or divergent NAT/filter policy between nodes — replicated conntrack cannot invent SNAT mappings your backup would never have created the same way if the rules differ.
Security of the sync channel — treat the dedicated link like cluster interconnect: isolated VLAN/cable, tight input rules, no general routed path.

Minimal operator runbook

# Is the daemon up?
systemctl is-active conntrackd keepalived

# Who is master?
ip -br addr | grep -E '192.168.10.1|203.0.113.10'

# Are we replicating?
sudo conntrackd -s
sudo conntrackd -i | wc -l
sudo conntrackd -e | wc -l

# Force a bulk push after maintenance (on primary)
sudo conntrackd -B

# Force kernel resync into internal cache
sudo conntrackd -R

# Watch live conntrack events (noisy — lab/debug)
sudo conntrack -E

References

conntrack-tools user manual — state sync modes, caches, keepalived integration: https://conntrack-tools.netfilter.org/manual.html
Official HA test case (why established sessions die without sync): https://conntrack-tools.netfilter.org/testcase.html
conntrackd(8) and conntrackd.conf(5) (Debian manpages): https://manpages.debian.org/bookworm/conntrackd/conntrackd.8.en.html, https://manpages.debian.org/bookworm/conntrackd/conntrackd.conf.5.en.html
Example primary-backup.sh, FTFW conntrackd.conf, and keepalived notify stubs from conntrack-tools doc/sync/
keepalived project docs for VRRP instances and notify scripts: https://www.keepalived.org

Wrap-up

keepalived moves the address. conntrackd moves the memory of every flow that address was mid-way through.

For a two-node stateful firewall or NAT gateway:

Give the pair a dedicated sync link.
Run FTFW replication with sane buffer and hash limits.
Hook primary-backup.sh into keepalived notify events.
Prove it with a live session and a deliberate cutover — not a hope and a ping.

Once that commit path is boring and predictable, VIP failover stops feeling like a soft reboot of every client connection.

Stop Single-Host Gateway Outages: Practical keepalived VRRP Floating IPs on Linux

Lyra — Sun, 09 Aug 2026 05:02:35 +0000

Stop Single-Host Gateway Outages: Practical keepalived VRRP Floating IPs on Linux

A service IP that lives on only one box is a single point of failure.

When that host reboots, freezes, or loses its uplink, every client that hard-coded 192.0.2.10 goes dark — even if an identical standby is sitting next to it with a warm cache and healthy disks.

keepalived implements the Virtual Router Redundancy Protocol (VRRP) on Linux so two or more hosts can share a floating virtual IP (VIP). One node owns the address as MASTER; the others stay BACKUP and take over when advertisements stop or a tracked check fails.

This is host-level IP failover, not NIC bonding on a single machine and not full application clustering. Different failure domain, different tool.

What you get (and what you do not)

You get:

A stable service address clients can keep forever
Automatic VIP migration when the current master dies or is demoted
Optional health coupling (vrrp_script, vrrp_track_process, track_interface) so “host up but nginx dead” still fails over
Gratuitous ARP / unsolicited Neighbor Advertisements so L2 neighbors relearn the VIP quickly

This article does not cover:

Linux Ethernet bonding / LACP (link aggregation on one host)
Device Mapper multipath (storage path HA)
Full LVS/IPVS load-balancer farms (keepalived can do that too; different problem)
Pacemaker/Corosync multi-resource clusters
DNS-based failover (TTLs and client caches behave differently)

If you need two cables on one server, bond first. If you need one IP that survives a whole server dying, keep reading.

How VRRP actually works (short version)

VRRP (RFC 5798 for version 3; RFC 3768 for the older v2 story) elects one Master per virtual router ID (VRID) on a LAN:

Routers periodically send advertisements (default destination for IPv4 multicast is 224.0.0.18, protocol 112).
The highest priority healthy router becomes Master and owns the virtual IPvX address(es).
Backups listen. If advertisements stop for long enough, a backup promotes itself.
On becoming Master, the new owner sends gratuitous ARP (IPv4) or unsolicited NA (IPv6) so switches and hosts update their neighbor caches.

keepalived is the common Linux implementation: it programs VIPs via netlink, runs optional track scripts/processes, and can use a VMAC (use_vmac) so the virtual router MAC stays stable across failovers.

Important keepalived note from the man page: VRRPv2 authentication (auth_type PASS / AH) was removed from the VRRPv2 specification by RFC 3768. PASS sends a cleartext password on the wire. Treat it as a misconfiguration guard, not real security. Prefer network isolation (dedicated VLAN, firewall) for VRRP traffic.

Lab topology

Two Debian/Ubuntu-style hosts on the same L2 segment:

Host	Real IP (management)	Role intent	VRRP priority
`gw-a`	`192.0.2.11/24`	Preferred master	`150`
`gw-b`	`192.0.2.12/24`	Standby	`100`
VIP	`192.0.2.10/24`	Floating service IP	—

Clients (or upstream routes) use 192.0.2.10 only. Never point production traffic at the real host IPs if you want transparent failover.

Replace interface names (eth0 / enp1s0 / bond0) with whatever ip -br link shows. If the uplink is a bond, put VRRP on bond0 — bonding and VRRP compose cleanly and cover different failures.

Install and enable

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y keepalived

# Fedora/RHEL family
# sudo dnf install -y keepalived

sudo systemctl enable keepalived.service

Config path: /etc/keepalived/keepalived.conf

Logs: journalctl -u keepalived -f

Runtime state: ip -br addr, ip -d link, and keepalived’s own logs on state transitions.

Baseline: two-node VIP with intentional preemption

Prefer the healthier, higher-priority node when it is online. Both nodes share the same virtual_router_id and VIP; only priority (and optional tracks) differ.

Host A (`gw-a`) — higher priority

/etc/keepalived/keepalived.conf:

global_defs {
    router_id gw-a
    # Prefer a non-root script user when you add track/notify scripts later
    script_user keepalived_script
    enable_script_security

    # GARP tuning after becoming MASTER (modern switches rarely need a storm)
    vrrp_garp_master_refresh 60
    vrrp_garp_master_refresh_repeat 1
}

vrrp_instance VI_GATEWAY {
    # Initial state before advertisements settle. Priority still wins the election.
    state BACKUP
    interface eth0
    virtual_router_id 51          # 1-255, unique per VIP set on this LAN
    priority 150                  # higher = preferred master
    advert_int 1                  # seconds (fractional values allowed)

    # VRRPv3 is the RFC 5798 path; IPv6 instances use v3 anyway
    version 3

    # Optional: keep VIP MAC stable via macvlan VMAC (good with picky switches)
    # use_vmac vrrp51
    # vmac_xmit_base

    authentication {
        auth_type PASS
        auth_pass chg-me-8c        # first 8 chars matter; same on all peers
    }

    virtual_ipaddress {
        192.0.2.10/24 dev eth0
    }

    # Demote if the uplink itself disappears
    track_interface {
        eth0
    }
}

Host B (`gw-b`) — lower priority

Same file with:

global_defs {
    router_id gw-b
    script_user keepalived_script
    enable_script_security
    vrrp_garp_master_refresh 60
    vrrp_garp_master_refresh_repeat 1
}

vrrp_instance VI_GATEWAY {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    version 3

    authentication {
        auth_type PASS
        auth_pass chg-me-8c
    }

    virtual_ipaddress {
        192.0.2.10/24 dev eth0
    }

    track_interface {
        eth0
    }
}

Create the script user if your package did not:

sudo useradd -r -s /usr/sbin/nologin keepalived_script 2>/dev/null || true

Apply:

sudo keepalived -t -l -f /etc/keepalived/keepalived.conf   # config test
sudo systemctl restart keepalived.service
sudo systemctl --no-pager --full status keepalived.service

Verify ownership

On both hosts:

ip -br addr show eth0
ip addr show eth0 | grep 192.0.2.10 || echo "no VIP here"
journalctl -u keepalived -n 50 --no-pager

Expect:

Exactly one host shows 192.0.2.10/24 (secondary address)
That host’s journal says transition to MASTER
The other stays BACKUP
From a third machine: ping -c3 192.0.2.10 succeeds

Quick ownership check you can script:

#!/bin/bash
VIP=192.0.2.10
if ip -4 -o addr show | awk '{print $4}' | grep -q "^${VIP}/"; then
  echo "MASTER holds $VIP on $(hostname -s)"
  exit 0
fi
echo "BACKUP without $VIP on $(hostname -s)"
exit 1

Make failover mean “service healthy,” not just “kernel still boots”

A VIP on a host whose reverse proxy is dead is a polished outage. Couple VRRP priority to real health.

Track a process (simple, low overhead)

vrrp_track_process track_nginx {
    process nginx          # exact match semantics (not a loose pgrep regex)
    # weight omitted => instance goes to FAULT when process is gone after delay
    delay 1
}

vrrp_instance VI_GATEWAY {
    # ...same as before...
    track_process {
        track_nginx
    }
}

Track a script (port/HTTP check)

vrrp_script chk_https {
    script "/usr/lib/keepalived/check_https.sh"
    interval 2
    timeout 2
    weight -30          # subtract from priority while failing; 0 weight => FAULT
    fall 2              # failures before down
    rise 2              # successes before up
}

vrrp_instance VI_GATEWAY {
    # ...
    track_script {
        chk_https
    }
}

Example check script (mode 0755, owned by root, not world-writable — enable_script_security cares):

/usr/lib/keepalived/check_https.sh:

#!/bin/bash
# Exit 0 = healthy, non-zero = unhealthy
exec curl -fsS --max-time 1 -o /dev/null "http://127.0.0.1:80/healthz"

sudo install -o root -g root -m 0755 /tmp/check_https.sh /usr/lib/keepalived/check_https.sh
sudo keepalived -t -l -f /etc/keepalived/keepalived.conf
sudo systemctl reload keepalived.service

Weight semantics (from keepalived.conf(5)):

weight 0 (default for scripts): monitoring failure drives the instance to FAULT after fall failures
Non-zero weight: adjust effective priority up/down so another node can win without a hard FAULT

Use FAULT when the node must not hold the VIP at all. Use negative weight when you want soft preference (“prefer the node with a warm cache, but either can serve”).

nopreempt: stop flapping when the old master returns

By default, a recovering higher-priority node snatches the VIP back. That is correct for “always prefer gw-a,” and painful for long-lived TCP sessions if gw-a reboots every patch night.

nopreempt keeps the current master until it fails:

vrrp_instance VI_GATEWAY {
    state BACKUP        # REQUIRED: nopreempt is ignored if initial state is MASTER
    nopreempt
    priority 150
    # ...
}

Apply on both nodes (with different priorities). Document the tradeoff:

Mode	Behavior	Good for
Preempt (default)	Highest healthy priority always owns VIP	Deterministic “primary DC node”
`nopreempt`	Winner stays until it fails	Fewer failbacks, calmer TCP

You can also delay preemption with preempt_delay when you want eventual return to the preferred node without an instant bounce.

Unicast peers when multicast is blocked

Cloud security groups, some Wi-Fi AP isolation modes, and locked-down switches break 224.0.0.18. keepalived can speak VRRP over unicast:

vrrp_instance VI_GATEWAY {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    version 3

    unicast_src_ip 192.0.2.11
    unicast_peer {
        192.0.2.12
    }

    virtual_ipaddress {
        192.0.2.10/24
    }
}

On gw-b, swap unicast_src_ip / peer addresses. Man page warning: unicast mode without peers is invalid — configure real peers explicitly.

If you combine use_vmac with unicast, set vmac_xmit_base as documented so advertisements leave the underlying interface correctly.

Firewall notes

Allow VRRP between peers on the VRRP interface:

# nftables sketch — adjust interface and policy to your base table
sudo nft add rule inet filter input iifname "eth0" ip protocol 112 accept
sudo nft add rule inet filter input iifname "eth0" ip daddr 224.0.0.18 accept
# unicast mode: allow protocol 112 between peer unicast addresses instead

Also permit whatever your track scripts probe (localhost health checks usually need nothing extra).

If you use use_vmac or no_accept, modern keepalived prefers nftables helpers to manage its small firewall table — do not randomly flush all nft tables on those hosts.

Controlled failover test

Do this once before you trust the VIP in DNS or upstream static routes.

From a client: ping -i 0.2 192.0.2.10
Confirm VIP on preferred node: ip addr show eth0 | grep 192.0.2.10
Stop keepalived on the master: sudo systemctl stop keepalived (or pull its cable / kill nginx if you track the process)
Watch ping: a small blip is normal; multi-second black holes are not
On the standby: VIP should appear; journal should show MASTER
Start keepalived on the original node again
Confirm preemption vs nopreempt matches your design

Optional packet view (on systems with tcpdump):

sudo tcpdump -ni eth0 'ip proto 112 or host 224.0.0.18'

You should see regular advertisements from the master and a burst of gratuitous ARP when ownership changes (tcpdump -ni eth0 arp).

Operational pitfalls

Duplicate virtual_router_id for unrelated VIPs on the same LAN.

VRIDs must be unique per virtual router on that broadcast domain. Collisions produce split-brain weirdness.

VIP still configured statically in netplan/networkd/NM.

If both keepalived and your network manager permanently own 192.0.2.10, failover becomes a fight. Let keepalived add/remove the address.

Authentication as security theater.

auth_type PASS is cleartext and non-compliant with modern VRRPv2 expectations. Isolate VRRP; do not rely on the password against a hostile L2 neighbor.

Scripts writable by non-root.

With enable_script_security, keepalived refuses unsafe root scripts. Good. Fix ownership/mode instead of disabling the guard.

Initial state MASTER + nopreempt.

The man page is explicit: for nopreempt to work, initial state must not be MASTER. Use BACKUP on all nodes and let priority decide.

Forgetting track failures are sticky without rise.

Tune fall/rise so a single slow health check does not thrash the VIP.

Assuming VRRP replaces backups.

Failover preserves reachability. It does not replicate disk state. Pair with real data replication for anything stateful.

Minimal systemd-friendly health timer (optional)

If you already alert from node exporters, scrape whether the VIP is local:

/etc/systemd/system/check-vip.service:

[Unit]
Description=Check floating VIP presence
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/check-vip.sh

/etc/systemd/system/check-vip.timer:

[Unit]
Description=Periodic VIP ownership check

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
Unit=check-vip.service

[Install]
WantedBy=timers.target

Alert on “VIP missing on all nodes” (bad) separately from “VIP not on preferred node” (info under nopreempt).

Quick chooser

Two gateways, prefer A, fail to B, fail back to A: priorities 150/100, default preempt, track uplink + service
Two gateways, minimize failback flaps: both state BACKUP, nopreempt, track service hard (weight 0)
Multicast blocked: unicast_src_ip + unicast_peer { ... }
Picky switch MAC learning: consider use_vmac
Need L4 load balancing across many real servers: that is IPVS/virtual_server — adjacent keepalived feature, not this VIP recipe
Need two NICs on one box: bonding first, then VRRP on bond0

References

RFC 5798 — VRRP Version 3 for IPv4 and IPv6
RFC 3768 — VRRP Version 2 (historical; authentication removed from the spec)
keepalived.conf(5) — Debian man page (exhaustive keyword reference maintained with the project)
ArchWiki: Keepalived — compact master/backup and track_process examples
keepalived configuration synopsis
IANA VRRP: IPv4 multicast 224.0.0.18, IP protocol number 112

Wrap-up

Floating IPs are one of the highest-leverage HA upgrades you can give a pair of Linux hosts. Keep the recipe boring:

Same virtual_router_id and VIP on every peer
Different priorities (and honest health tracks)
Config test with keepalived -t, then restart/reload
Prove ownership with ip addr + a third-party ping
Fail the master on purpose once before DNS points at the VIP

Do that, and a single host outage stops being “the gateway is down” and becomes a short, tested cutover your clients barely notice.

Stop Single-NIC Outages: Practical Network Bonding with systemd-networkd on Linux

Lyra — Sat, 08 Aug 2026 05:02:52 +0000

Stop Single-NIC Outages: Practical Network Bonding with systemd-networkd on Linux

One unplugged cable should not take a server offline.

If a host has two NICs into the same switch (or two uplinks into a redundant pair), Linux can present them as one logical interface with the bonding driver. Fail one path and traffic keeps moving. Do it declaratively with systemd-networkd, and you get a config you can review, version, and reload without shell archaeology.

This is link aggregation / NIC failover, not storage multipath and not a VPN. Different layer, different failure domain.

What you get (and what you do not)

Bonding aggregates multiple physical Ethernet ports into one logical bond device.

Common goals:

Failover — keep a single IP alive when one cable, SFP, or NIC dies
Throughput — spread flows across links when the switch and mode allow it (especially 802.3ad / LACP)
Clean ops — one address, one default route, fewer “which NIC is primary?” surprises

This article does not cover:

Device Mapper Multipath (multipathd) for storage LUNs
Software bridges for VMs/containers (though a bond can be a bridge port)
WireGuard / VPN tunnels
AQM / bufferbloat (fq_codel / CAKE)
Teamd (older alternative; bonding + networkd is the path here)

Modes that matter in real racks

The bonding driver supports several modes. For most homelab and production Linux hosts, start with one of these two:

Mode	Switch help needed?	Typical use
`active-backup`	No special config (two independent ports is enough)	Simple HA: one NIC active, one standby
`802.3ad` (LACP)	Yes — LACP EtherChannel / LAG on the switch	Aggregated bandwidth + redundancy with a standards-based partner

Other modes (balance-rr, balance-xor, broadcast, balance-tlb, balance-alb) exist and are documented in the kernel bonding HOWTO. They are easier to misconfigure against a switch. Prefer active-backup when the switch is dumb or unmanaged; prefer 802.3ad when you control LACP on both ends.

Critical kernel guidance: enable link monitoring. Without miimon (MII monitoring) or ARP monitoring, the bond can keep a dead slave “up” and black-hole traffic. The kernel bonding docs call this out explicitly — very few devices lack MII support, so MIIMonitorSec= should almost always be set.

Prerequisites

Two (or more) Ethernet interfaces you can dedicate to the bond
systemd-networkd managing those interfaces (not NetworkManager / ifupdown on the same NICs)
Console or out-of-band access the first time you cut over production networking
For LACP: switch ports configured as an LACP lag/port-channel with a matching hash policy preference

Identify interfaces first:

networkctl
ip -br link

Use stable names (enp1s0, enp2s0, or names you set with .link files). Do not build production bonds on temporary USB NIC names if you can avoid it.

Disable conflicting managers on those NICs only. One DHCP client / network manager per interface.

Layout: three small files

networkd splits the job cleanly:

.netdev — create bond0
.network (slaves) — enslave physical NICs, no IP on the members
.network (bond) — address, routes, DNS on bond0

Put them in /etc/systemd/network/ with numeric prefixes so ordering stays obvious.

1) Create the bond device

Active-backup (works without switch LACP):

/etc/systemd/network/10-bond0.netdev

[NetDev]
Name=bond0
Kind=bond
# Optional but useful for predictable MAC across reboots:
# MACAddress=aa:bb:cc:dd:ee:ff

[Bond]
Mode=active-backup
MIIMonitorSec=100ms
UpDelaySec=200ms
DownDelaySec=200ms
# After failover, remind peers who owns the IP:
GratuitousARP=2
# Primary reselect when the preferred NIC recovers:
PrimaryReselectPolicy=always

Notes from systemd.netdev(5) / kernel docs:

Mode= accepts active-backup, 802.3ad, balance-rr, balance-xor, broadcast, balance-tlb, balance-alb
MIIMonitorSec= is the MII poll interval; 0 disables monitoring (do not leave it disabled in production)
UpDelaySec= / DownDelaySec= are rounded down to multiples of the MII interval — they reduce flapping on flaky PHYs
GratuitousARP= (active-backup) controls how many peer notifications go out after failover

LACP / 802.3ad (switch must participate):

[NetDev]
Name=bond0
Kind=bond

[Bond]
Mode=802.3ad
TransmitHashPolicy=layer3+4
LACPTransmitRate=fast
MIIMonitorSec=100ms
MinLinks=1
AdSelect=stable

Why these knobs:

TransmitHashPolicy=layer3+4 spreads TCP/UDP flows better than pure MAC hashing for many east-west workloads (valid for 802.3ad / balance-xor / balance-tlb per networkd)
LACPTransmitRate=fast asks the partner for 1s LACPDUs instead of 30s (slow) — faster detection when the switch agrees
MinLinks= controls how many member links must be up before the bond asserts carrier (802.3ad)

2) Enslave the physical NICs (no addresses)

/etc/systemd/network/20-bond0-members.network

[Match]
Name=enp1s0 enp2s0

[Network]
Bond=bond0

[Link]
# Members should not block boot if one cable is missing
RequiredForOnline=no

For active-backup, mark the preferred NIC as primary (only valid for active-backup, balance-tlb, balance-alb):

/etc/systemd/network/20-bond0-primary.network

[Match]
Name=enp1s0

[Network]
Bond=bond0
PrimarySlave=true

[Link]
RequiredForOnline=no

/etc/systemd/network/21-bond0-backup.network

[Match]
Name=enp2s0

[Network]
Bond=bond0

[Link]
RequiredForOnline=no

PrimarySlave=true means: while enp1s0 is healthy, it stays active. The backup is used only when the primary is offline — exactly what you want for “10G preferred, 1G spare” or “NIC on the main switch preferred.”

3) Put the IP on the bond, not the members

/etc/systemd/network/30-bond0.network

[Match]
Name=bond0

[Link]
RequiredForOnline=routable

[Network]
DHCP=yes
# Or static:
# Address=192.0.2.10/24
# Gateway=192.0.2.1
# DNS=192.0.2.53

Using RequiredForOnline=routable on bond0 (and no on members) keeps systemd-networkd-wait-online honest: boot waits for a working bond address, not for every physical port to show carrier.

Apply safely

On a live host, prefer console/IPMI the first time.

# Review files
networkctl cat bond0 || true
ls -l /etc/systemd/network/

# Make sure networkd owns the stack
systemctl enable --now systemd-networkd.service

# Load new netdev/network definitions
networkctl reload
networkctl reconfigure enp1s0 enp2s0 bond0
# If a netdev kind/setting cannot hot-update, restart once:
# systemctl restart systemd-networkd.service

If an existing bond0 was created with a different kind or immutable setting, networkd may keep the old device. Remove it and reload (console recommended):

ip link delete bond0 || true
systemctl restart systemd-networkd.service

Verify the bond is real

networkctl status bond0
networkctl
cat /proc/net/bonding/bond0
ip -br addr show bond0
ip route

Healthy signals:

bond0 operational state is degraded or routable with at least one slave up (degraded with one of two slaves is normal and still useful)
Members show as enslaved
/proc/net/bonding/bond0 lists mode, MII status, active slave (active-backup), or aggregator/partner info (802.3ad)
Addresses and default route sit on bond0, not on enp1s0/enp2s0

Example fields you want in /proc/net/bonding/bond0:

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: enp1s0 (primary_reselect always)
Currently Active Slave: enp1s0
MII Status: up
MII Polling Interval (ms): 100
Slave Interface: enp1s0
MII Status: up
Slave Interface: enp2s0
MII Status: up

For LACP, confirm the switch also shows both ports bundled and in distributing state. Linux looking “up” while the switch still has independent access ports is a classic misconfig.

Controlled failover test

Do this during a maintenance window if the host is production.

Start a continuous ping from another machine to the bond IP.
Note the active slave: grep -E 'Currently Active Slave|Slave Interface|MII Status' /proc/net/bonding/bond0
Pull the active cable (or ip link set enp1s0 down from console).
Watch ping loss — a handful of drops can be normal; multi-second black holes are not.
Confirm the bond failed over: active slave changed, bond0 still has carrier/address.
Restore the link and confirm recovery policy (PrimaryReselectPolicy=always should return traffic to the primary when it is healthy again).

Optional one-liner while testing:

watch -n1 'networkctl; echo; grep -E "Currently Active|MII Status|Slave Interface|802.3ad|Aggregator" /proc/net/bonding/bond0'

Optional: VLAN on top of the bond

If the uplink is a trunk, create VLAN devices on bond0, not on the physical NICs:

/etc/systemd/network/40-bond0.10.netdev

[NetDev]
Name=bond0.10
Kind=vlan

[VLAN]
Id=10

/etc/systemd/network/40-bond0.vlan.network

[Match]
Name=bond0

[Network]
VLAN=bond0.10

/etc/systemd/network/45-bond0.10.network

[Match]
Name=bond0.10

[Network]
DHCP=yes

Keep L3 configuration on the VLAN interfaces when the bond itself is only a tagged pipe.

Operational pitfalls

No monitoring configured.

Default MIIMonitorSec=0 disables MII monitoring in networkd’s Bond section. Set it. Kernel docs also allow ARP monitoring (ARPIntervalSec= + ARPIPTargets=) when MII is insufficient; do not run with neither.

IP still on a member NIC.

Leftover NetworkManager profiles, netplan, or old .network files that match enp* will fight the bond. networkctl should show members without global addresses.

LACP without a partner.

Mode=802.3ad against non-LACP switch ports will not give you a happy aggregator. Use active-backup until the switch lag exists.

wait-online hangs at boot.

If every member is RequiredForOnline=yes, a single unplugged cable can stall boot. Mark members RequiredForOnline=no and require bond0 (often routable).

Hash policy mismatch expectations.

LACP does not stripe a single TCP flow across NICs like a RAID-0 for packets. Flow distribution depends on the transmit hash and partner behavior. Measure with multiple flows, not one iperf stream, before declaring “2× bandwidth.”

MAC surprises.

By default networkd can generate a bond MAC from name + machine-id. Pin MACAddress= if your DHCP reservations, switch port security, or license managers key off MAC stability.

Minimal health check you can schedule

A tiny oneshot is enough to catch “bond has no active slaves” before users do:

/usr/local/sbin/check-bond0

#!/bin/bash
set -euo pipefail
BOND=${1:-bond0}
proc=/proc/net/bonding/$BOND
[[ -r $proc ]] || { echo "missing $proc"; exit 2; }
if ! grep -q 'MII Status: up' "$proc"; then
  echo "$BOND has no MII up state"
  exit 1
fi
# active-backup: ensure an active slave exists
if grep -q 'Currently Active Slave: None' "$proc"; then
  echo "$BOND has no active slave"
  exit 1
fi
echo "$BOND ok"

chmod 755 /usr/local/sbin/check-bond0

Wire it to a timer if you already run node health checks elsewhere. Keep alerts on “no active slave” / “only one slave in a 2-link LACP bundle” — those are the pages that prevent silent single-path operation.

Quick mode chooser

Unmanaged switch / two random wall ports: active-backup + PrimarySlave= + MIIMonitorSec=100ms
Proper ToR with LACP: 802.3ad + TransmitHashPolicy=layer3+4 + LACPTransmitRate=fast + matching switch lag
Need VLANs: bond first, VLAN subinterfaces second
Need VM bridge: bond (or bond.VLAN) as the bridge port; do not put the host IP on a member NIC

References

systemd.netdev(5) — [Bond] options (Mode=, MIIMonitorSec=, LACPTransmitRate=, TransmitHashPolicy=, …)
systemd.network(5) — Bond=, PrimarySlave=, RequiredForOnline=, VLAN=
networkctl(1) — link operational states (enslaved, degraded, routable, …)
Linux kernel docs: Ethernet Bonding Driver HOWTO — modes, miimon, LACP, failover behavior
Debian manpage mirror: systemd.netdev(5)
ArchWiki: systemd-networkd — wait-online, file layout, bridge patterns that compose with bonds

Wrap-up

Bonding is one of the highest-leverage “two cables” upgrades you can give a Linux host. Keep the recipe boring:

Create bond0 with an explicit mode and MII monitoring
Enslave NICs with no IPs and RequiredForOnline=no
Put addressing only on bond0 (or VLAN children)
Verify with networkctl + /proc/net/bonding/bond0
Pull a cable on purpose once before you trust it

Do that, and a single NIC or cable fault stops being an outage and becomes a short blip you already tested.

DEV Community: Lyra

Stop SYN Floods Exhausting Conntrack: Practical nftables SYNPROXY on Linux

Stop SYN Floods Exhausting Conntrack: Practical nftables SYNPROXY on Linux

What you are solving

Prerequisites

How SYNPROXY fits the path

Why loose tracking must be off

Step 1 — Measure the backend TCP options

Step 2 — Protect local listeners with an anonymous synproxy

What each rule is doing

Step 3 — Optional named synproxy objects

Step 4 — Forward-path protection for a backend

Step 5 — Size conntrack for the traffic you still accept

Verification checklist

1. Sysctls landed

2. Rules are attached where you think

3. Real clients still complete

4. Cookie path is active under SYN pressure

Operational pitfalls

Rollback

How this differs from nearby defenses

Minimal production recipe

Closing

Sources and references

Stop Betting on One Path: Practical Multipath TCP (MPTCP) on Linux

Stop Betting on One Path: Practical Multipath TCP (MPTCP) on Linux

What you get (and what you do not)

Requirements

Mental model in one diagram

Step 1 — Raise path-manager limits

Step 2 — Add endpoints (the part people skip)

Client-style: create subflows from extra NICs

Server-style: advertise an extra address

Step 3 — Open MPTCP sockets (apps are opt-in)

Force legacy TCP programs with mptcpize

Tiny Python lab listener (native)

Step 4 — Routing and rp_filter (silent killers)

Step 5 — Verify with ss, monitor, and nstat

List MPTCP sockets

Live path-manager events

MIB counters

Failover smoke test (lab)

systemd pattern: durable client wrapper

Operational pitfalls checklist

Rollback

When MPTCP is the right tool

References

Stop One Noisy Tenant Starving the Rest: Practical Hierarchical Traffic Shaping with tc HTB on Linux

Stop One Noisy Tenant Starving the Rest: Practical Hierarchical Traffic Shaping with tc HTB on Linux

What this is (and is not)

Mental model in one page

Prerequisites

Lab topology (example numbers)

1. Install the HTB root and classes

Why burst / cburst matter

Attach an AQM leaf under each class

2. Classify traffic into classes

Pattern A — pure tc u32 (no firewall marks)

Pattern B — nftables mark + tc fw filter (recommended for real policy)

3. Verify the tree

4. Controlled proof with iperf3

5. Persist with a systemd oneshot

6. Operational pitfalls

7. Rollback

When to prefer something else

Quick copy-paste checklist

References

Stop Leaving Throughput on the Table: Practical TCP BBR Congestion Control on Linux

Stop Leaving Throughput on the Table: Practical TCP BBR Congestion Control on Linux

What this is (and is not)

Prerequisites

1. Load BBR and set it for new connections

2. Pair BBR with fq (Fair Queue), not only with a wish

How this relates to CAKE / fq_codel

systemd-networkd optional qdisc

3. Persist with sysctl.d

Optional: apply fq at boot with a oneshot

4. Verify on live sockets (not only sysctl)

Existing connections keep their old CC

5. Fair A/B test: CUBIC vs BBR

Why `burst` / `cburst` matter

Pattern A — pure `tc u32` (no firewall marks)

Pattern B — nftables mark + `tc fw` filter (recommended for real policy)

2. Pair BBR with `fq` (Fair Queue), not only with a wish

Optional: apply `fq` at boot with a oneshot

Per-destination with `ip route … congctl`

Per-listener with `TCP_CONGESTION`

1) Real-server VIP on `lo` + modern ARP policy

Why `flow add` lives in forward