Memory was tight on the box, so I stopped another project by hand to make room for a k3s install. The install pulled in br_netfilter, which flipped a kernel switch host-wide. The project I'd stopped came back up — and its containers couldn't reach each other anymore.
My first move was restarting the docker daemon, on the theory that bringing everything back up fresh would shake the problem loose. It didn't touch this one. It did reassign every container IP on a second, unrelated network, which left me with an allow-list somewhere no longer admitting the source it had been written to admit. That's its own story. This one is about the first failure, and it turned out to have nothing to do with the daemon restart at all.
The symptom held up under testing. No container on that bridge could reach any other — I checked several pairs, and they all failed the same way. Every attempt just ran until it timed out; nothing came back refused. A refused connection means a socket answered and said no. A timeout means the packet went somewhere and nothing came back, which puts the fault in the network layer, not in whatever was supposed to be listening on the other end. A different stack on the same host, on a different bridge, kept working the whole time. Whatever this was, it wasn't host-wide.
The only change
Only one thing had actually changed on the host that day: the k3s install, and specifically what it did to the kernel underneath it. k3s's own systemd unit runs ExecStartPre=-/sbin/modprobe br_netfilter — from /etc/systemd/system/k3s.service — before the service itself starts. That's k3s loading the module as shipped, not a step I added.
Loading the module isn't what matters on its own. What matters is what comes with it: br_netfilter brings net.bridge.bridge-nf-call-iptables into play, and somewhere in that install, the switch flipped — from 0 to 1, host-wide. That sysctl decides whether traffic that never leaves a bridge gets shown to iptables at all. Two containers on the same bridge talking to each other is bridge-internal traffic in the strict sense — packets forwarded between ports of the same bridge, no routing involved. At 0, the kernel never submits that traffic to iptables; nothing evaluates it. At 1, it does — the same chains that judge traffic arriving on a real interface now get asked to judge this too.
Nothing about the containers changed, and nothing about the project I'd stopped and brought back up changed either. What changed is which packets iptables now gets a say over. k3s did that on purpose: loading the module and turning on that sysctl is baked into how it starts, on any host it's installed to. It isn't a step aimed at this box specifically, and it isn't a step k3s skipped or fumbled. The install did exactly what it was built to do.
Flipping the switch, and what it doesn't prove
I tested the sysctl directly that day. net.bridge.bridge-nf-call-iptables to 0, and the containers could reach each other again — immediately, no restart, no wait. Back to 1, and they stopped again, just as fast. I flipped it a second time to be sure. Same result both directions.
That's confirmation, not a diagnosis. All it shows is that whatever's dropping this traffic sits downstream of that switch — not what that thing is, and not why it's dropping these particular packets between these particular containers instead of every packet crossing the bridge. This identifies the mechanism, not the rule. And leaving the sysctl at 0 was never a real fix — the switch is host-wide, and k3s's unit sets it unconditionally every time it starts, so leaving it at 0 only holds until the next restart flips it back. It isn't a fix scoped to whatever's actually dropping these packets. From here the question splits in two: something inside iptables is dropping this traffic, and nothing I'd tested so far said which table, which chain, or which rule.
Ruling out everything else
Bridge port STP state first — a port stuck outside forwarding would produce exactly this symptom on its own, no iptables involved. bridge link show on both ports: forwarding, forwarding. Not that.
tc had no filters attached to either interface, so there was nothing to even check a counter on; XDP had no program loaded on either side either. nftables' bridge family — the modern interface to the same hooks ebtables covers — had no tables defined at all, and ebtables itself came back with an empty ruleset. Last, the ARP cache, because a stale entry would produce this exact kind of silent black hole without touching anything above it: the destination MAC each sending container had cached matched the target container's actual, current MAC.
Five places checked, five negatives. Whatever was eating this traffic wasn't sitting in any of the spots that usually catch a problem at the bridge layer.
What conntrack didn't have
conntrack wasn't on this box to begin with — I installed it mid-investigation, specifically to watch this flow live.
It had nothing to show me. Not a dropped entry, not a half-open one — no entry, for a flow I could reproduce on demand just by opening a connection between the two containers. A conntrack entry gets created the moment a new flow is first evaluated, win or lose. If there's no entry at all, connection tracking never got a look at this traffic.
That points at exactly one place. In the IPv4 PREROUTING hook, netfilter runs raw at priority -300 and connection tracking at -200, and hooks fire in increasing priority order — raw always goes first. A packet dropped in raw never reaches conntrack at all, which is exactly the shape of what I wasn't seeing. One table left to check.
The raw table
iptables -t raw -L PREROUTING -n -v -x --line-numbers turned up two matched sets of rules, seven lines each, keyed to the same seven destination addresses — 172.18.0.2 through 172.18.0.8. Each set named a different bridge interface. One bridge was live; the other was stale, already deleted.
The live set's shape, verbatim:
-A PREROUTING -d 172.18.0.3/32 ! -i br-b951f3fb0958 -j DROP
DROP, gated on a /32 destination match and a negated input-interface match. dockerd puts one of these on every container IP itself, and what it's enforcing is anti-spoofing: drop traffic for this address unless it arrived on the bridge that's supposed to own it. ! -i br-b951f3fb0958 reads as "drop unless it came in on br-b951f3fb0958" — a real exception, as long as br-b951f3fb0958 exists. The stale set carried the identical logic, just naming br-66885a1f7aad in place of br-b951f3fb0958 — and br-66885a1f7aad no longer existed. So there was no interface left for a packet to arrive on that could ever satisfy that stale set's exception. The condition could never be true. A rule that reads as conditional is, in practice, unconditional — every packet for those seven addresses, arriving from anywhere, got dropped by the stale set regardless of where it actually came from.
The counters said it before I'd finished reasoning through it. The rules pinned to the deleted bridge were already past 4,967 hits that day, and still climbing. The rules pinned to the live bridge — the correct rules, the ones actually describing the network as it existed — sat at zero. That's not reassuring. iptables walks a chain top to bottom and stops at the first match, and the stale rules came first in the chain. Zero hits on the correct rules didn't mean they were fine. It meant they'd never been reached at all.
Whose bug this is
k3s loaded br_netfilter and flipped the sysctl because that's what it does on every host it installs to — already established, not up for reargument. The seven rules keyed to a bridge that no longer existed are a separate question, and the answer isn't k3s.
dockerd wrote those rules. This host runs Docker Engine 29.6.0, and since Docker Engine 28.0 (2025) — a deliberate hardening change, not a bug — dockerd drops unsolicited inbound traffic to a container's internal IP by default unless the port is explicitly published. The raw-table rules from the previous section are that change's implementation. The blog post doesn't get me that far on its own — it frames the change around unpublished ports and never names a table. What names it is moby#49621, which added DOCKER_INSECURE_NO_IPTABLES_RAW=1 as an opt-out for kernels built without CONFIG_IP_NF_RAW: an escape hatch that only makes sense if the rules were going into raw in the first place.
At some point before this incident, the network these seven rules belonged to went down and came back up, and dockerd built a new bridge to replace the old one — the live br-b951f3fb0958 standing in for the dead br-66885a1f7aad. The new bridge got its own seven rules. The old bridge's seven didn't go anywhere. On this host, nothing removed those rules when the interface they named stopped existing.
I looked for whether that specific gap is a known, filed defect, and I didn't find one. The closest matches are moby/libnetwork#570 — stale FORWARD-chain rules left behind after a bridge is removed — and a docker/cli report about network rm corrupting a different network's rules. Different chain in one case, different trigger in the other; neither is this. What I can say is what happened on this host on 2026-08-06. I can't call it a tracked bug, and I'm not hanging a version range on one.
And it sat there doing nothing for the same reason established at the start: at bridge-nf-call-iptables=0, bridge-internal traffic never reaches iptables at all, live rule or stale one. The seven orphaned rules weren't evaluated, weren't skipped, weren't anything — they just weren't asked. k3s didn't write this bug. It was the first thing on this box to turn the light on in the room where it had been sitting the whole time.
Seven deletions, and the check that would have caught it earlier
The fix is narrower than the last six sections might suggest: delete the seven rules pinned to br-66885a1f7aad, and leave everything else in the raw table exactly as it is — including bridge-nf-call-iptables, which stays at 1. k3s and its CNI need that sysctl on. Turning it back off doesn't fix this; it just re-hides it, and takes bridge-internal traffic on every other network on the box off iptables' radar along with it. That's not a fix, that's the same outage wearing a different cause.
iptables -t raw -D PREROUTING <line>, seven times, one call per stale rule, in descending order of line number. Delete low-to-high and every deletion renumbers the lines below it — the seventh line you meant to hit has already moved by the time you get to it. High-to-low, each deletion only touches numbers you've already dealt with, so the number you read off --line-numbers is still the number that's true when you use it.
Checked again on 2026-08-06, the fix held: iptables -t raw -S PREROUTING on this host shows exactly seven DROP rules for 172.18.0.2 through 172.18.0.8 — all seven addresses, none missing — every one of them naming the live br-b951f3fb0958. Nothing in the table names br-66885a1f7aad anymore.
That closes the one incident. It doesn't close the bug, because the bug isn't scoped to that bridge or that day — the same orphaned rule can turn up on any other network on this host, for as long as nothing on this host cleans up a stale rule when a bridge gets recreated. The check that catches it is the fix run in reverse: take every interface name the raw table's PREROUTING chain references, and confirm each one still exists.
comm -23 \
<(iptables -t raw -S PREROUTING | grep -oP '(?<=-i )\S+' | sort -u) \
<(ip link show type bridge | grep -oP '^\d+: \K[^:@]+' | sort -u)
Anything that prints is a rule pointed at an interface that isn't there — the same bug, on some other bridge, waiting for whatever eventually submits its traffic to iptables. Run against this host on 2026-08-06, it prints nothing: four bridge names in the raw table, all four listed in ip link show type bridge, none orphaned.
Top comments (0)