The Kernel Underneath Kubernetes: nftables, netfilter, and Why svclb-traefik Kept Crash-Looping

A few weeks ago, I hit a bug that sent me further down the Linux networking stack than I'd gone in years: an NVIDIA Jetson device, running k3s at the edge, with svclb-traefik stuck in a CrashLoopBackOff that made no sense from the Kubernetes side. Pods scheduled fine. Images pulled fine. The manifests were correct. The problem was three layers below anything kubectl describe could tell me, in the kernel's packet-classification engine.

The fix ended up being: blacklist nf_tables. But understanding why that line worked meant actually learning what nftables is, how it relates to the iptables tools everyone's used for twenty years, and where eBPF fits into all of this — because the honest answer is that most of us run this stack daily without ever needing to know what's underneath it, until a vendor kernel breaks the abstraction.

This is that deep dive.

Netfilter: the actual foundation

Before nftables, before iptables, there's netfilter — the kernel subsystem that makes any of this possible. Netfilter itself doesn't know what a "rule," "chain," or "table" is. It's simpler and older than that: it's a set of five hook points wired into the network stack where kernel code can intercept a packet and hand back a verdict.

PREROUTING → [routing decision] → INPUT → (local socket)
                                 → FORWARD → POSTROUTING → (out the interface)
              OUTPUT (locally-generated packets) → POSTROUTING

• PREROUTING — packet has just arrived, before the kernel has decided whether it's for this host or being forwarded
• INPUT — packet is headed to a local process
• FORWARD — packet is being routed through this host
• OUTPUT — packet originated locally
• POSTROUTING — packet is about to leave an interface

At each hook, netfilter calls whatever's registered there, in priority order, and each callback returns a verdict — ACCEPT, DROP, QUEUE, and a few others. That's the entire contract. Everything else — iptables, nftables, connection tracking, NAT — is a tenant of this hook system, not part of it.

Packet classification, defined properly

Once you have hooks, you need something to actually decide what happens to a packet at each one. That's packet classification: given a packet's header fields — source/destination address, protocol, ports, TCP flags, interface — find which rule(s) apply and execute the associated action.

There are, broadly, two ways to implement this:

1. Linear evaluation — walk an ordered list of rules top to bottom, stop at first match. Simple to reason about, but the cost is O(n) per packet. A table with 10,000 rules can mean 10,000 comparisons for one packet in the worst case.
2. Indexed evaluation — use hash tables, sets, or interval trees so a packet jumps close to O(1) or O(log n) to the relevant rule instead of scanning linearly.

This distinction is the entire reason nftables exists. Classic iptables is a linear-evaluation engine implemented as a chain of kernel modules. nftables is an indexed-evaluation engine implemented as a small in-kernel virtual machine that interprets a compact bytecode program against structured data — sets, maps, concatenated key lookups — instead of walking modules one at a time.

Why there used to be four separate tools

If netfilter's hooks are protocol-agnostic in principle, why did we end up with iptables, ip6tables, arptables, and ebtables as four completely separate tools? Because each protocol family has a different header shape, and historically each got its own userspace binary, its own kernel module family, and its own rule storage — with zero sharing between them.

Tool	Operates on	Header fields classified
iptables	IPv4	source/dest IP, protocol, TTL
ip6tables	IPv6	128-bit addresses, extension headers, no in-header fragmentation
arptables	ARP	opcode, hardware/protocol addresses
ebtables	Ethernet bridging (L2)	MAC addresses, EtherType, VLAN tags — inside the bridge code path, not the routed IP path

Each tool has its own binary, its own kernel modules (ip_tables, ip6_tables, arp_tables, ebtables), its own match/target extension modules, and — critically — no shared rule-set with the others. A single bridged-and-routed packet in a Kubernetes CNI setup might conceptually need to be evaluated by both the IP-layer rules and the bridge-layer rules, with no shared engine, syntax, or coordination between them. That fragmentation, multiplied across dual-stack IPv4/IPv6 clusters with bridge-mode CNIs, is exactly what nftables was built to collapse into one engine with multiple table families (ip, ip6, inet, arp, bridge, netdev) instead of four unrelated tools.

Anatomy of a rule, and a packet walking through one

The classic model — shared by iptables, ip6tables, and arptables — is table → chain → rule.

• Tables group rules by purpose: filter (accept/drop decisions), nat (SNAT/DNAT/MASQUERADE), mangle (header rewriting), raw (conntrack exemptions).
• Chains are ordered rule lists. Built-in chains map directly onto netfilter's five hooks. User-defined chains are only reached via a jump from a built-in one.
• Rules are match conditions (ANDed together) plus a target — a verdict that either terminates evaluation (ACCEPT, DROP) or lets it continue (LOG, MARK).

Here's a concrete trace. A client sends a TCP SYN to port 22:

IPv4: src=203.0.113.50 dst=198.51.100.10 proto=6(TCP) ttl=54
TCP:  sport=51422 dport=22 flags=SYN

Against this INPUT chain:

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP
iptables -A INPUT -j DROP

1. Conntrack, running at PREROUTING, tags this as a NEW connection (no prior entry for this 4-tuple).
2. Rule 1: -i lo checks the packet's incoming-interface metadata. Arrived on eth0, not lo → no match.
3. Rule 2: -m state ESTABLISHED,RELATED queries conntrack. State is NEW → no match.
4. Rule 3: -p tcp matches (protocol field is 6), but -s 10.0.0.0/8 fails — the source isn't in that range. One failed AND-condition fails the whole rule.
5. Rule 4: -p tcp --dport 22 — both match. DROP fires. Evaluation stops. The packet is silently discarded.

Notice what determined the outcome: rule order, not rule specificity. If rules 3 and 4 were swapped, an authorized 10.x.x.x client would hit the blanket DROP before ever reaching its allow rule. There's no automatic "most specific wins" resolution — whichever rule the packet reaches first and fully matches, wins. This is the sharpest edge of linear packet classification, and it's the source of a large fraction of "why isn't my rule working" debugging sessions.

Where nftables changes the mechanics

The same rule-set in nftables doesn't have to be a flat list. -s 10.0.0.0/8 and a destination port can be expressed as set-membership tests and verdict maps, so the kernel's bytecode VM does a hash or interval lookup instead of repeating a linear CIDR comparison per rule. For five rules, this makes no practical difference. For five thousand rules covering thousands of allowed subnets — which is not a hypothetical in any reasonably-sized cluster, iptables degrades linearly while nftables' set-backed lookup stays close to constant time.

This is also, not coincidentally, why most modern distros quietly turned the iptables binary itself into a compatibility shim: iptables-nft translates classic iptables syntax into nftables rules under the hood, so most users get nftables' engine without ever typing nft directly. The legacy path — iptables-legacy — still exists, talking straight to the old ip_tables kernel module, bypassing nftables entirely.

Where the Jetson actually broke

My first instinct was that this was an update-alternatives problem — that iptables was somehow resolving to the wrong binary. It wasn't:

$ update-alternatives --display iptables
iptables - auto mode
  link currently points to /usr/sbin/iptables-legacy

Already pinned to legacy at the host level. Dead end, but an instructive one, because it meant the failure was happening somewhere the host's alternatives system doesn't reach.

The actual answer was in the crash logs of the svclb-traefik pod itself:

+ lsmod
+ grep -qF nf_tables
+ '[' 0 '=' 0 ]
+ mode=nft
+ ln -sf xtables-nft-multi /usr/sbin/iptables
+ iptables -t filter -I FORWARD -s 0.0.0.0/0 -p TCP --dport 80 -j ACCEPT
Warning: Extension tcp revision 0 not supported, missing kernel module?
iptables v1.8.11 (nf_tables): RULE_INSERT failed (No such file or directory): rule in chain FORWARD

The non-obvious part — and the reason pinning the host's alternative wasn't enough, is that the svclb-traefik container's own entrypoint script does an independent lsmod | grep nf_tables check at container startup, entirely separate from whatever the host has configured. If the module is loaded anywhere on the system, the container self-selects nft mode and symlinks its own iptables to xtables-nft-multi internally, silently overriding the host-level legacy setting. It then tries to insert a FORWARD rule through that nft path, and NVIDIA's L4T/tegra kernel, which doesn't track mainline closely and ships an incomplete nf_tables implementation, rejects the insert outright.

So the fix isn't at the alternatives layer at all. It's at the kernel module layer, applied and verified live:

sudo modprobe -r nf_tables
lsmod | grep nf_tables            # confirm empty, and check nothing else (e.g. a WireGuard mesh) depends on it first
sudo systemctl restart k3s
kubectl get pods -n kube-system -w | grep svclb-traefik

With the module gone, the container's startup check finds nothing, stays in legacy mode, and the FORWARD rule inserts cleanly — pod goes to 2/2 Running. Made permanent with:

echo "blacklist nf_tables" | sudo tee /etc/modprobe.d/nf_tables-blacklist.conf
sudo update-initramfs -u

The lesson generalizes past this one bug: a host-level setting (the alternatives symlink) can be quietly overridden by a container that does its own environment detection at startup. Fixing the layer you can see isn't the same as fixing the layer the failure is actually in.

ebtables and eBPF are not related — but eBPF matters more

It's worth naming a coincidence explicitly, because the letters invite the confusion: ebtables stands for "Ethernet Bridge tables" and is a netfilter tenant, same family as iptables, just hooked into the bridge code path instead of the routed IP path. eBPF stands for "extended Berkeley Packet Filter" and is architecturally unrelated — a general-purpose, register-based virtual machine that lets verified programs attach to a wide range of kernel hook points, not just networking ones (syscalls, tracepoints, kprobes are all fair game too). eBPF doesn't extend netfilter. It's a different subsystem the kernel exposes as an alternative to it.

eBPF's networking relevance runs through a few specific attachment points:

• XDP — programs attached at the NIC driver level, before the kernel even allocates an sk_buff. This is earlier than netfilter's first hook.
• TC (traffic control) — ingress/egress hooks at the queuing layer, earlier than or parallel to netfilter.
• Socket-level hooks (cgroup/connect, sockops) — used to redirect traffic transparently at connect() time, without ever touching a packet header.

This is exactly the layer Cilium operates at as a kube-proxy replacement. Traditional kube-proxy installs DNAT rules in the nat table — a ClusterIP gets rewritten to a pod IP through the same rule-chain-target model traced above. Cilium's eBPF datapath instead attaches at TC/XDP and does that same translation via an eBPF map lookup, or — further still — rewrites the destination at connect()/sendmsg() time through a cgroup/connect4 program, so the packet is correctly addressed before it's even built. No DNAT rule-walk happens at all.

The performance argument is the same shape as iptables-vs-nftables — linear scan versus indexed lookup — just taken one step further: eBPF doesn't only index better, it can skip the netfilter hook traversal entirely.

The mental model worth keeping

Three layers, cleanly separated:

1. netfilter — the hook architecture. Stable, unchanged in essence for two decades.
2. nf_tables / ip_tables — competing in-kernel rule-evaluation engines that plug into those hooks.
3. nft / iptables(-nft / -legacy) — userspace tools that talk to whichever engine is active.

A fix like blacklisting nf_tables operates entirely at layer 3→2 — it steers which engine gets used without touching netfilter's actual architecture. eBPF, by contrast, is a genuinely different building next door, one that's increasingly capable of replacing the whole netfilter-based stack for Kubernetes networking rather than just choosing a working half of it.

Kernel-level networking is one of those areas I've barely scratched the surface of and i'm really interested in. this nf_tables bug pulled me from a Kubernetes crash loop all the way down to the netfilter hook architecture, and it's clear there's a lot more underneath worth digging into. I plan on coming back to this space and going deeper.