DEV Community

Cover image for Stop Burning CPU on Established Flows: Practical nftables Flowtables on Linux
Lyra
Lyra

Posted on

Stop Burning CPU on Established Flows: Practical nftables Flowtables on Linux

#opensource #linux #devops #networking

Stop Burning CPU on Established Flows: Practical nftables Flowtables on Linux

If your Linux box is a router, gateway, or homelab edge, most packets are not “new.” They are established TCP and UDP flows that already survived conntrack, NAT, and your forward policy.

Sending every one of those packets back through prerouting → routing decision → forward → postrouting is correct — and expensive once traffic climbs.

nftables flowtables give you a deliberate fastpath: after the first packets create state the normal way, later packets can bypass the classic forwarding path from ingress straight toward transmission. NAT is still applied from the cached conntrack entry. TTL/hop limit is still decremented. You keep policy control over which flows are eligible.

This post is a practical setup for a two-interface Linux router. No magic “turn on turbo mode” toggle — just a flowtable, a selective flow add rule, and verification you can trust.

What a flowtable actually does

From the kernel’s Netfilter flowtable docs:

  1. The first packet(s) of a connection still walk the classic IP forwarding path and create conntrack state.
  2. A rule in the forward chain can run flow add @table (also documented historically as flow offload) to insert that flow into a flowtable.
  3. Later packets that hit the flowtable at the ingress hook skip the rest of the classic path and go out via neigh_xmit().
  4. Misses still take the normal path.

The lookup key is roughly:

  • L2 encapsulation where relevant (VLAN / PPPoE since kernel 5.13)
  • L3 source/destination
  • L4 ports
  • input interface
  • L3/L4 protocol (IPv4/IPv6 + TCP/UDP)

The cached entry also stores the egress device, gateway/neigh info, and NAT mangling so the fastpath stays consistent with the slow path that created it.

Important edge cases the kernel documents:

  • Fragments cannot be looked up in the flowtable (transport header missing) → classic path
  • TCP FIN/RST go classic so the flow can tear down cleanly
  • Over-MTU packets go classic so ICMP too-big can be generated
  • Flowtable entries can go stale if the egress device or destination MAC changes out from under them (bridge + IP forwarding mixes and HW offload need extra care)

Prerequisites

  • Linux with nftables and flowtable support (widely available on modern Debian/Ubuntu/Fedora kernels; VLAN/PPPoE/bridge discovery needs 5.13+)
  • Host acting as a router (forwarding enabled)
  • Packages: 
# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y nftables conntrack

# Fedora
sudo dnf install -y nftables conntrack-tools
Enter fullscreen mode Exit fullscreen mode

\nEnable IPv4 forwarding persistently:

echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-forward.conf
sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

Replace interface names below with yours. Example:

  • eth0 — LAN / private
  • eth1 — WAN / upstream

Minimal working ruleset

Save as /etc/nftables.d/flowtable-router.nft (or fold into your main /etc/nftables.conf).

#!/usr/sbin/nft -f
flush ruleset

define DEV_LAN = eth0
define DEV_WAN = eth1
define NET_LAN = 192.168.10.0/24

table inet filter {
        # Fastpath table: hooks ingress on BOTH directions' devices.
        # devices must cover the interfaces traffic enters on for offloaded flows.
        flowtable ft_fast {
                hook ingress priority 0
                devices = { $DEV_LAN, $DEV_WAN }
                counter   # sync bytes/packets back into conntrack (kernel 5.7+)
                # flags offload;  # uncomment ONLY if NICs support HW flow offload
        }

        chain input {
                type filter hook input priority filter; policy drop;

                ct state vmap { established : accept, related : accept, invalid : drop }
                iifname lo accept
                iifname $DEV_LAN accept
                # WAN: allow SSH only from a management prefix if needed
                # iifname $DEV_WAN tcp dport 22 ip saddr 203.0.113.0/24 accept
        }

        chain forward {
                type filter hook forward priority filter; policy drop;

                # Offload established TCP (and optionally UDP) once state exists.
                # Prefer matching return traffic / established flows; first packets
                # still need a normal accept path below.
                ct state established,related \
                        meta l4proto { tcp, udp } \
                        flow add @ft_fast \
                        counter

                ct state vmap { established : accept, related : accept, invalid : drop }

                # New connections from LAN toward WAN (and hairpin LAN if desired)
                iifname $DEV_LAN oifname $DEV_WAN accept
                # replies / hairpin already covered by established above
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr $NET_LAN oifname $DEV_WAN masquerade
        }
}
Enter fullscreen mode Exit fullscreen mode

Load and persist:

sudo nft -c -f /etc/nftables.d/flowtable-router.nft   # syntax check
sudo nft -f /etc/nftables.d/flowtable-router.nft

# Debian/Ubuntu packaged service
sudo systemctl enable --now nftables

# If your distro expects a single file, include the snippet from /etc/nftables.conf:
# include "/etc/nftables.d/*.nft"
Enter fullscreen mode Exit fullscreen mode

Why the devices list matters

The flowtable’s devices = { ... } is not cosmetic. It registers the ingress fastpath on those interfaces. For a router you almost always need both LAN and WAN (and any other interfaces that carry offloaded flows in either direction).

The nftables wiki is explicit: devices are required for both traffic directions.

Why flow add lives in forward

Offload is decided after the connection has been seen on the classic path. The forward chain is the natural place: you already know this packet is being routed, and you can constrain offload to protocols/ports you care about.

Example: only offload bulk HTTP(S), keep everything else on the slow path for deeper inspection:

tcp dport { 80, 443 } ct state established flow add @ft_fast counter
Enter fullscreen mode Exit fullscreen mode

Or the reverse — offload everything established except SSH:

ct state established,related meta l4proto { tcp, udp } \
        tcp dport != 22 \
        flow add @ft_fast counter
Enter fullscreen mode Exit fullscreen mode

(Adjust to your policy; the point is selectivity.)

Hardware offload vs software offload

Software mode is the default and works without special NIC features. Flows show as [OFFLOAD] in conntrack.

If your NICs and driver support Netfilter flowtable hardware offload, enable:

flowtable ft_fast {
        hook ingress priority 0
        devices = { eth0, eth1 }
        flags offload;
        counter
}
Enter fullscreen mode Exit fullscreen mode

Hardware-offloaded flows are tagged [HW_OFFLOAD]. A few packets may still traverse the software path until a workqueue pushes the flow to the device.

Do not enable flags offload “just because.” If the driver cannot offload, you want a clean failure or plain software mode — not a mystery performance regression. Confirm with vendor/driver docs and by watching for HW_OFFLOAD tags under load.

Bridge, VLAN, and PPPoE notes (kernel 5.13+)

Useful topology facts from the kernel docs:

  • Flowtables can discover the real device behind VLAN and PPPoE. You generally add the underlying device to devices, not every stacked virtual interface.
  • Bridge ports can be added so the fastpath spans bridge-port ↔ gateway-NIC topologies, including bridge VLAN filtering (PVID/untagged).
  • If you combine bridge forwarding and IP forwarding aggressively, treat stale MAC/egress caching as a first-class operational risk — the flowtable is a cache.

If you already run a VLAN-aware bridge on the LAN side, add the real member ports you care about to the flowtable devices list and test carefully before calling it production.

Verification checklist

1. Confirm the flowtable is present 

sudo nft list flowtables
sudo nft list ruleset | sed -n '/flowtable/,/^$/p'
Enter fullscreen mode Exit fullscreen mode

You should see your flowtable block, devices, and the flow add @... rule in forward.

2. Generate forwarded traffic

From a LAN client, open a long-lived bulk transfer through the router (iperf3, a large HTTPS download, etc.). Local-process traffic on the router itself is not the forward path — test with real forwarded flows.

3. Look for OFFLOAD tags 

# IPv4
sudo conntrack -L -o extended | grep -E 'OFFLOAD|HW_OFFLOAD' | head

# Filter established TCP if the table is large
sudo conntrack -L -p tcp --state ESTABLISHED -o extended | head
Enter fullscreen mode Exit fullscreen mode

Kernel docs show software offload like:

tcp 6 src=10.141.10.2 dst=192.168.10.2 sport=52728 dport=5201 \
    src=192.168.10.2 dst=192.168.10.1 sport=5201 dport=52728 [OFFLOAD] mark=0 use=2
Enter fullscreen mode Exit fullscreen mode

You can also filter by status bit where supported:

sudo conntrack -L -u OFFLOAD -o extended | head
Enter fullscreen mode Exit fullscreen mode

4. Prove the forward chain counter stalls for offloaded packets

List the forward rule counters:

sudo nft list chain inet filter forward
Enter fullscreen mode Exit fullscreen mode

For a fully offloaded elephant flow, the flow add / forward counters should stop climbing quickly, while the transfer continues. That is the visible effect of the ingress bypass: those packets never revisit your forward chain.

If counters keep racing for the whole transfer, offload is not sticking — check devices list, that traffic is actually forwarded (not input), protocol match, and conntrack state.

5. Optional: CPU sanity check under load 

# before / after enabling flow add
mpstat -P ALL 1 10
Enter fullscreen mode Exit fullscreen mode

You are looking for lower softirq/si time on the forwarding cores at the same bulk throughput — not a microbenchmark religion. Directionally, established-flow offload should move work off the full Netfilter forward path.

Safe roll-in pattern

  1. Deploy the ruleset without flow add and confirm connectivity + NAT.
  2. Add flow add for a narrow port set (e.g. 443 only).
  3. Verify [OFFLOAD] appears and forward counters behave.
  4. Widen to tcp, udp established if results look good.
  5. Only then consider flags offload on known-good hardware.

Rollback is immediate:

# remove offload rule only (example handle — use nft -a list to find yours)
sudo nft -a list chain inet filter forward
sudo nft delete rule inet filter forward handle <N>

# or flush back to a known-good file
sudo nft -f /etc/nftables.conf.good
Enter fullscreen mode Exit fullscreen mode

Common failure modes

Symptom Likely cause
Never see [OFFLOAD] No flow add match; traffic is local input not forward; devices list wrong; connection never becomes established
Offload appears then dies Egress/MAC change; bridge topology churn; route change stale cache
HW flag set, only [OFFLOAD] Hardware offload not actually active/supported on that path
Weird IPv6-only behavior Missing inet/ip6 coverage or asymmetric policy routing on reply path
Policy routing surprises Reply path must resolve consistently; wiki notes special ip rule setups need to match the packet that creates the flow entry
Fragments “ignore” fastpath Expected — fragments take classic path

What this is not

  • Not a replacement for CAKE/fq_codel queue discipline (latency under working-buffer bloat)
  • Not conntrackd state sync for HA failover
  • Not a substitute for correct forward/NAT policy — offload amplifies whatever policy created the flow
  • Not XDP/eBPF programming — this stays inside nftables + conntrack
  • Not a cure for single-core NIC driver bottlenecks or missing checksum/TSO offloads

Use flowtables when the box is a legitimate forwarder and established traffic dominates CPU.

Complete compact lab example

Two namespaces + veth pairs are enough to rehearse without touching production:

sudo ip netns add lan
sudo ip netns add wan
sudo ip link add v-lan type veth peer name v-lan-r
sudo ip link add v-wan type veth peer name v-wan-r
sudo ip link set v-lan netns lan
sudo ip link set v-wan netns wan
sudo ip addr add 192.168.10.1/24 dev v-lan-r
sudo ip addr add 203.0.113.1/24 dev v-wan-r
sudo ip link set v-lan-r up
sudo ip link set v-wan-r up
sudo ip netns exec lan ip addr add 192.168.10.10/24 dev v-lan
sudo ip netns exec lan ip link set v-lan up
sudo ip netns exec lan ip route add default via 192.168.10.1
sudo ip netns exec wan ip addr add 203.0.113.10/24 dev v-wan
sudo ip netns exec wan ip link set v-wan up

# Point the nft devices at v-lan-r / v-wan-r, enable forwarding, load ruleset,
# then run iperf3 server in wan ns and client in lan ns.
Enter fullscreen mode Exit fullscreen mode

When [OFFLOAD] shows up on the router namespace’s conntrack during iperf, the mechanism is working.

References

Takeaway: Flowtables do not invent a new firewall. They let established, policy-approved TCP/UDP flows stop paying the full Netfilter tax on every packet. Start narrow, verify with conntrack [OFFLOAD] tags and stalled forward counters, then widen. That is the difference between “we enabled offload” and “we can prove the fastpath is doing work.”

Top comments (0)