Everyone in our team was obsessed with the network layer. We had Calico policies locked down tight. Every Kubernetes namespace tagged. VPC flow logs streaming into our SIEM. We were proud of it. Then a pentester showed us we'd missed something embarrassingly basic: the host firewall.
I'm talking about the local firewall running on the machine itself. iptables, nftables, Windows Firewall, whatever flavor your OS ships. We'd been so focused on east-west traffic controls and overlay networks that we'd ignored the fact that every server still has its own network stack. And that stack doesn't know about your fancy Kubernetes network policies.
Here's what happened. We have a monitoring agent that runs on every host. It sends metrics to a central collector. The agent binds to a local port for health checks — standard stuff. Someone had left that port open to 0.0.0.0/0 in the host firewall during testing and never locked it down. The overlay network policies wouldn't catch this because the traffic didn't traverse the overlay. It was direct host-to-host on the underlay network.
The pentester scanned our internal ranges, found that open port on a dozen hosts, and had a path straight into our monitoring pipeline. From there it was a short hop to lateral movement.
The fix was simple: audit every host firewall rule, lock everything down to specific source ranges, and add host firewall compliance to our deployment pipeline. But the real lesson was about layered defense. You can't rely on one mechanism to enforce Zero Trust. The network overlay covers pod-to-pod traffic, the host firewall covers host-to-host, and both need to be maintained.
We now validate host firewall rules as part of every deployment. It takes an extra 30 seconds in the CI pipeline. Worth every millisecond.
As an Amazon Associate I earn from qualifying purchases.
Top comments (0)