When you use a Linux box as a router, every packet takes the same journey: connection tracking, rule chain, NAT, rule chain again, egress. For a single connection that journey is negligible. On a box forwarding hundreds of thousands of packets per second, most of the CPU bill accumulates exactly there — and the interesting part is that the decision about the vast majority of those packets has already been made.
Netfilter's flowtable infrastructure is built on that observation. The first packets of a flow go through the normal path and get the decision; after that the flow is written into a table, and packets matching it reach the egress without ever seeing the classic path.
The real price of flowtable, paid in exchange for the CPU it saves, is visibility. The speed you gain is paid for by losing the answer to "which rule saw this packet?" Set it up knowing that and it's an excellent tool; set it up without knowing and months later you'll be chasing a counter that doesn't move or a rule that never fires.
What exactly the fast path skips
The kernel documentation says it in one sentence: a packet that finds a matching entry in the flowtable is transmitted straight to the output netdevice, so packets bypass the classic IP forwarding path — and then the critical note in parentheses: the visible effect is that you do not see these packets from any of the Netfilter hooks coming after ingress.
If there's no match, the packet continues on the classic path. So flowtable isn't an all-or-nothing switch, it's a cache.
An update is needed here, because this document hasn't been meaningfully revised since 2021. The text describes the transmit path via neigh_xmit(); current kernels consolidated that path and call dev_queue_xmit(). The difference isn't academic: on the dev_queue_xmit() path, nftables' netdev egress hook and the tc/qdisc layer are still in play. So what you lose on the software fast path isn't "all visibility" but the rules in the forward and postrouting hooks, and their counters. Your traffic shaping (HTB, fq_codel, CAKE) keeps working; tc ingress and XDP run before the flowtable anyway, so they're unaffected too.
With hardware offload it really is "you see nothing": the packet never reaches the CPU, so no rule, no qdisc and no egress hook is involved. The meaningful line is software versus hardware.
The lookup uses this n-tuple: layer 2 encapsulation (VLAN and PPPoE), layer 3 source and destination, layer 4 source and destination ports, and the input interface. That last one matters for setups with several conntrack zones.
Setup: one table, one rule
The documentation's example is as plain as it gets — you define a flowtable and add a single rule to the forward chain:
table inet x {
flowtable f {
hook ingress priority 0; devices = { eth0, eth1 };
}
chain y {
type filter hook forward priority 0; policy accept;
ip protocol tcp flow add @f
counter packets 0 bytes 0
}
}
The flow add @f statement means you choose which flows take the fast path. The documentation is explicit about this: packets follow the classic path unless the user explicitly instructs otherwise. Flowtable doesn't kick in by default; it is enabled by policy.
If you need it, you can create several flowtables for resource partitioning. The flowtable priority defines the order in which hooks run in the pipeline. If you already have an nftables ingress chain, the documentation's rule is explicit: keep the flowtable priority smaller than that chain's, so the flowtable runs first.
What falls off the fast path
Not every packet qualifies, and the rules make sense:
- Fragmented traffic goes to the classic path, because with the transport header missing a flowtable lookup isn't possible.
-
TCP RST and FIN packets are described by the documentation as passed up, but that has gone stale: since the change landed in 6.14 these packets mark the flow as
CLOSINGand stay on the fast path; the flow is torn down only when a new SYN arrives on a closing connection. - Packets exceeding the MTU take the classic path too, so a packet-too-big ICMP error can be reported to the sender.
The list needs additions that aren't in the documentation but sit in the source, because "why aren't some of my flows accelerated" is usually answered here: flows with a conntrack helper attached — FTP, SIP, TFTP and friends — are never offloaded; packets carrying IPsec, IPv4 packets with options, and connections marked with sequence adjustment or NAT clash stay out too. On the TCP side the flow must be established. And there's a configuration constraint: flow add is only valid in the forward chain.
On the NAT side: the flowtable stores the NAT configuration, and packets are mangled according to the NAT policy specified from the classic path. The TTL is decremented before neigh_xmit() is called. So the fast path doesn't skip the router's duties, only the decision chain.
To see which flows took the fast path, look at the connection tracking table; offloaded flows carry the [OFFLOAD] tag:
# conntrack -L
tcp 6 src=10.141.10.2 dst=192.168.10.2 sport=52728 dport=5201 ... [OFFLOAD] mark=0 use=2
Visibility: what exactly do you lose?
In the example above the rule ends with a counter. The documentation states plainly that once the flow is offloaded, that counter no longer gets updated for packets forwarded through the bypass.
This holds for everything rule-based: counters, log statements, filter rules further down, marking. You wrote the rule, but the packet never visits it.
Don't assume you lose all accounting, though. You can add a counter statement to the flowtable itself, in which case it synchronises packet and byte counters with the existing connection tracking entry:
table inet x {
flowtable f {
hook ingress priority 0; devices = { eth0, eth1 }; counter
}
}
That support has been available since kernel 5.7. On its own, though, it isn't enough: the counters are written into the conntrack entry, so conntrack accounting has to be on too — and it's off by default.
sysctl -w net.netfilter.nf_conntrack_acct=1 # make it permanent in /etc/sysctl.d/
One caveat: the accounting extension is attached when the conntrack entry is created, so enabling this later affects new connections only. Do both and flow-level accounting stays with you; what you lose is rule-level detail. Clarify that distinction before deployment: if your monitoring reports "how many packets did each rule see", that report starts lying quietly the moment flowtable comes online.
There's a second cost next to visibility, and it gets even less attention: when a flow is offloaded, the kernel disables TCP window validation for that conntrack entry. The comment in the source states the reason plainly — conntrack will no longer see all the packets, so sequence and window checking becomes meaningless. In other words, state validation is relaxed for the flows you put on the fast path; whether that's acceptable depends on how trusted the flow is.
The same distinction applies to security. If you built detailed egress control like the one in my article on an nftables egress policy layer, you have to narrow your flow add condition knowing that those rules won't inspect the offloaded flows. Opening the fast path only to already-decided, trusted traffic is mandatory here.
How long does a flow stay on the fast path?
The behaviour that confuses people most after setup: a flow shows up on the fast path, disappears, then comes back. The reason is timeouts, and they have their own sysctls:
$ sysctl net.netfilter.nf_flowtable_tcp_timeout net.netfilter.nf_flowtable_udp_timeout
net.netfilter.nf_flowtable_tcp_timeout = 30
net.netfilter.nf_flowtable_udp_timeout = 30
In the kernel documentation's words these control the offload timeout: TCP and UDP connections may be offloaded from conntrack to the flow table, and once aged, the connection is returned to conntrack.
The important detail: this is an idle timer. Every packet taking the fast path refreshes the timeout, so a flow with traffic doesn't drop off every 30 seconds. What drops off are flows that genuinely carry no packet for 30 seconds — sparse heartbeat channels, idle sessions. When those come back, your rule counters tick by a packet or two per flow; that's why you see small increments where you expected "the counter never moves".
That's why you shouldn't judge the fast path from a single snapshot; watch the number of [OFFLOAD]-tagged flows in conntrack -L over time.
Hardware offload and bridges
If your network device supports hardware offload, adding flags offload; to the flowtable definition is enough. A workqueue adds flows to the hardware; the documentation warns that a few packets may still run over the software fast path until the workqueue gets a chance. Flows that made it to hardware appear with the [HW_OFFLOAD] tag in conntrack. The infrastructure also supports DSA (Distributed Switch Architecture).
At layer 2 things got easier in 5.13: the flowtable discovers the real netdevice behind VLAN and PPPoE devices, parses those headers, and uses the ethertype and VLAN/PPPoE session id for lookups. So you don't need to add the VLAN and PPPoE devices to your flowtable definition; the real device is sufficient.
For bridges the rule is the opposite, and the difference is easy to miss: if you want a fast path between your bridge ports and the IP forwarding path, you have to add the bridge ports (as represented by the real netdevice) to your flowtable definition. With VLAN/PPPoE you don't need to add them; with a bridge you must. The infrastructure then discovers the topology behind the bridge device and establishes a fast path between bridge ports and the gateway device; bridge VLAN filtering (PVID, untagged) is supported as well. For people building home or branch routers, that's usually where the real gain is.
The price of being a cache: stale entries
The documentation's "limitations" section is short but important: the flowtable behaves like a cache and entries might get stale if either the destination MAC address or the egress netdevice used for transmission changes. Two situations are called out: running the flowtable in software mode while combining bridge and IP forwarding, and having hardware offload enabled.
In practice that means care in setups with moving topology: a router switching between redundant uplinks, virtual MACs carried by VRRP, bridge ports relearned after a cable change. In such an environment, test the failover scenario deliberately before enabling the fast path: the real question is what happens the moment the link changes.
Do you need it on your box?
Looking at my own VPS, the answer is clearly no:
$ nft --version
nftables v1.0.9 (Old Doc Yak #3)
$ ls /lib/modules/$(uname -r)/kernel/net/netfilter/ | grep flow
nf_flow_table.ko.zst
nf_flow_table_inet.ko.zst
nft_flow_offload.ko.zst
$ lsmod | grep -c nf_flow
0
$ cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
745
262144
The modules exist, they aren't loaded, and they don't need to be: this machine is an application server, forwarded traffic is negligible, and the conntrack table hovers around 745 entries out of a 262k capacity. The CPU flowtable would save here disappears into measurement noise.
Where it does add value is clear: edge boxes forwarding at high packet rates, home and branch routers, VPN gateways, heavy east-west traffic on container hosts. What they share is millions of packets from the same flow passing through the same decision over and over.
Where it doesn't is equally clear: traffic you want to inspect deeply, environments doing detailed rule-level accounting, and setups whose topology keeps changing. On the IDS/IPS side the concrete trap is NFQUEUE: if you run Suricata or Snort inline in the forward chain, offloaded packets never enter that queue — silently. Your inspection rule is still there, your engine is running, but part of the traffic no longer passes through it.
Let's also correct a conntrack expectation: offloading doesn't reduce conntrack entries. The entry stays, it just gets the [OFFLOAD] tag. Flowtable is not a cure for a filling conntrack table.
If you want to measure the gain, conntrack -L | grep -c OFFLOAD alone won't do; compare forwarded packet rate and softirq CPU (mpstat -P ALL, /proc/net/softnet_stat) before and after. I didn't run that measurement on my own box for this article — there's no load to measure; do it before you enable it.
Checklist before you enable it
- Keep the
flow addcondition narrow: only decided, trusted flows (specific interface pairs and protocols, say). - Add
counterto the flowtable definition so you keep flow-level accounting. - Review monitoring dashboards that rely on rule counters or logs; offloaded flows won't appear there.
- Check for the
[OFFLOAD]and[HW_OFFLOAD]tags inconntrack -L: are the flows you expected really on the fast path? - Test failover, VLAN changes and bridge topology scenarios; the stale-entry risk lives there.
- With hardware offload, verify your NIC driver support and account for the first packets still taking the software path.
In my article about moving from iptables to nftables we talked about the readability of a rule set; flowtable changes how much of that rule set actually runs. The two belong in the same conversation.
Speed always comes from the same place
What flowtable does is computing's oldest trick: instead of making the same decision over and over, store the result. And like every stored decision, a gap can open between it and reality — an entry going stale when a MAC address changes, a counter that stops moving, a packet that never reaches the queue.
So the question for your own setup: how many times is the same decision made on this machine? If the answer is millions, flowtable has a place; if it's a few thousand, what you get for the lost visibility is speed you won't be able to measure.
Top comments (0)