I Replaced kube-proxy with eBPF in Production (And Why My Monitoring Went Blind for 6 Hours)
From a "simple" Cilium upgrade to a 6-hour outage where my SIEM couldn't see a single packet — the real story of deleting kube-proxy, the eBPF program that saved me, and why the docs never warned me about the one metric that matters.
The Setup
It started with a cilium upgrade command I ran on a Tuesday evening.
My homelab cluster — the same 4-node bare-metal setup I wrote about last month (Dell OptiPlex + 3 Raspberry Pis, Talos Linux, Cilium, ArgoCD, Longhorn) — was running Cilium 1.15 in kube-proxy-replacement partial mode. Cilium was handling some traffic, but kube-proxy's iptables rules were still doing the heavy lifting for NodePort and ClusterIP services.
I'd read the Cilium 1.16 release notes. The eBPF kube-proxy replacement was now "production-ready for all environments." The Talos Linux docs had a single bullet point: "Set kubeProxyReplacement: true in Cilium values."
It looked harmless. It looked like flipping a switch.
So I did.
helm upgrade cilium cilium/cilium \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=auto \
--set k8sServicePort=6443
The Cilium pods rolled. The nodes stayed up. kubectl get pods -A showed everything running.
I went to bed.
At 2:47 AM, my phone buzzed. Not an alert from the cluster. An alert from my SIEM — or rather, the absence of data from it. The log ingestion rate for kubernetes-audit and network-flow had flatlined to zero.
The cluster was "healthy." The monitoring was dead.
This is not a tutorial. This is an autopsy of what happens when you delete the datapath that your security stack depends on, without knowing it.
Why Not Just Keep kube-proxy?
I run Kubernetes professionally at Siemens. I know what managed EKS gives you: kube-proxy in iptables mode, happily translating ClusterIP virtual IPs into pod endpoints, maintained by AWS, never thought about.
But I was hitting a wall.
1. iptables doesn't scale
In my homelab, 45 manifests and ~120 services meant ~3,500 iptables rules across all nodes. Every new service triggered a full iptables-restore that locked the kernel's netfilter table for hundreds of milliseconds. On a Raspberry Pi 4, that's a noticeable latency spike.
2. The gap between "cluster works" and "I understand the datapath"
kube-proxy is a black box. You run it, it works, you ignore it. But if you can't explain how a packet flows from Service:80 to Pod:8080 at the kernel level, do you really own your cluster?
3. eBPF gives you superpowers — if you know the cost
Cilium's eBPF programs can:
- Replace conntrack with BPF map-based connection tracking
- Provide socket-level load balancing (no NAT needed for pod-to-pod)
- Deliver Hubble flow logs with L3-L7 visibility
- Enforce network policies before the packet hits the host stack
But eBPF isn't magic. It's a different universe. And the tools that read the old universe don't work in the new one.
The Architecture (Before and After)
Before: kube-proxy + Cilium (Hybrid)
┌─────────────────────────────────────────────────────────────────────┐
│ Worker Node │
│ │
│ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Pod A │───►│ ClusterIP │───►│ kube-proxy │ │
│ │ (app) │ │ Service │ │ (iptables) │ │
│ └─────────────┘ └──────────────────┘ └────────┬─────────┘ │
│ │ │
│ ┌────────────────▼─────────┐ │
│ │ iptables NAT/REDIRECT │ │
│ │ -A KUBE-SVC-XXX ... │ │
│ └────────────────┬─────────┘ │
│ │ │
│ ┌────────────────▼─────────┐ │
│ │ conntrack table │ │
│ │ /proc/net/nf_conntrack │ │
│ └────────────────┬─────────┘ │
│ │ │
│ ┌────────────────▼─────────┐ │
│ │ Pod B (target) │ │
│ └──────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌──────────────────┼──────────────────┐
│ SIEM reads │ Falco reads │ Security tools
│ iptables rules │ conntrack │ depend on netfilter
│ for flow logs │ for connections │
└──────────────────┘──────────────────┘
After: Cilium eBPF (kube-proxy-deleted)
┌─────────────────────────────────────────────────────────────────────┐
│ Worker Node │
│ │
│ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Pod A │───►│ ClusterIP │───►│ Cilium Agent │ │
│ │ (app) │ │ Service │ │ (eBPF) │ │
│ └─────────────┘ └──────────────────┘ └────────┬─────────┘ │
│ │ │
│ ┌────────────────▼─────────┐ │
│ │ eBPF Programs │ │
│ │ ├─ bpf_sock (socket LB) │ │
│ │ ├─ bpf_lxc (pod egress) │ │
│ │ ├─ bpf_host (NodePort) │ │
│ │ └─ bpf_network (ingress)│ │
│ └────────────────┬─────────┘ │
│ │ │
│ ┌────────────────▼─────────┐ │
│ │ BPF Maps (not iptables) │ │
│ │ cilium_lb4_services_v2 │ │
│ │ cilium_ct4_global │ │
│ └────────────────┬─────────┘ │
│ │ │
│ ┌────────────────▼─────────┐ │
│ │ Pod B (target) │ │
│ └──────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
▲
│
┌──────────────────┘
│ SIEM: "WHERE ARE MY FLOW LOGS?!"
│ Falco: "I CAN'T SEE CONNTRACK!"
│ Legacy tools: broken by design
Design Decision
| Component | Before (kube-proxy) | After (eBPF) | Impact |
|---|---|---|---|
| Service Load Balancing | iptables DNAT | eBPF socket-level redirect | Latency: -30% |
| Connection Tracking | conntrack kernel module | eBPF BPF map (cilium_ct4_global) |
No /proc/net/nf_conntrack
|
| NAT | iptables MASQUERADE | eBPF SNAT/DNAT | No iptables -t nat -L output |
| Network Policy Enforcement | iptables + ipsets | eBPF TC/XDP programs | Earlier drop, better performance |
| Flow Logs | Conntrack events → Falco | Hubble eBPF exporter | Different schema, different source |
| NodePort | iptables REDIRECT | eBPF bpf_host
|
Same port, different kernel path |
The Failures (And What Each One Taught Me)
Failure 1: I Upgraded and Went to Bed
I ran the Cilium upgrade at 11 PM. The pods cycled green. I checked kubectl get svc and curled a NodePort service from my laptop. It worked.
What I didn't check: whether Falco was still emitting network connection events.
Falco's default rules include this:
# Falco rule: outbound network connection
- rule: Outbound Connection
desc: Detect outbound connections from pods
condition: >
outbound and
fd.name != "" and
(fd.typechar = '4' or fd.typechar = '6')
output: "Outbound connection from pod (command=%proc.cmdline connection=%fd.name)"
priority: NOTICE
Falco detects these connections by reading /proc/<pid>/fd/ and correlating with conntrack state. When kube-proxy's iptables rules disappeared, Falco's fd.name resolution for ClusterIP destinations broke — the eBPF path doesn't leave the same /proc artifacts that Falco's syscall-based engine expects.
Result: Falco emitted zero network alerts for 7 hours. Not because there was no traffic. Because it couldn't see the traffic the same way.
Lesson 1: "Services work" is not "security monitoring works." Test both.
Failure 2: My SIEM Couldn't Parse Hubble
At 2:47 AM, my phone woke me up. Grafana showed a flat line for network_flow_events_per_minute.
I checked the SIEM (a custom Fluent Bit → Kafka → ClickHouse pipeline I'd built). The Fluent Bit pods were running. The Kafka topics existed. The ClickHouse tables were healthy.
But the kubernetes.flow_logs stream had zero rows since 23:12.
Here's why:
My SIEM consumed network flow logs from two sources:
# /etc/fluent-bit/inputs.conf
[INPUT]
Name tail
Path /var/log/audit/kube-apiserver-audit.log
Parser json
Tag k8s.audit
[INPUT]
Name tail
Path /var/log/conntrack/flows.json
Parser json
Tag k8s.network-flows
The second input — conntrack/flows.json — was populated by a small sidecar that read /proc/net/nf_conntrack every 30 seconds and emitted JSON.
With eBPF replacing conntrack entirely, /proc/net/nf_conntrack was empty on every node. Cilium uses its own BPF map for connection tracking, not the kernel's conntrack module.
So my sidecar emitted nothing. Fluent Bit had nothing to tail. Kafka got empty batches. ClickHouse had nothing to insert.
The SIEM was blind. Not broken — blind by design, because I had built it around netfilter assumptions that eBPF invalidated.
Lesson 2: eBPF isn't a drop-in replacement. It's a migration. Treat it like one.
Failure 3: I Panic-Downgraded and Made It Worse
At 3:30 AM, I decided to roll back.
helm rollback cilium 1
Cilium reverted to the previous version. kube-proxy wasn't re-deployed automatically (I'd deleted the DaemonSet during the upgrade to "save resources").
Now I had no kube-proxy and no eBPF replacement.
ClusterIP services stopped resolving entirely. DNS inside the cluster broke (CoreDNS is a ClusterIP service). My apps couldn't reach the API server. ArgoCD went into a crash loop.
I had to:
- SSH into the Talos API (not SSH —
talosctl) - Re-apply the kube-proxy manifest manually
- Wait for iptables rules to populate
- Restart CoreDNS pods
- Fix ArgoCD's sync state
Total downtime: 94 minutes.
Lesson 3: Don't delete the old path until the new path is fully monitored. I had eBPF working, but I hadn't validated monitoring. Deleting kube-proxy killed my safety net.
The Fix (And What Actually Works)
Step 1: Enable Hubble (Not Optional)
I had installed Cilium with Hubble disabled to "save resources on the Pis." That was stupid. Hubble is the only source of flow visibility in an eBPF-only cluster.
# cilium-values.yaml
hubble:
enabled: true
relay:
enabled: true
ui:
enabled: true
metrics:
enabled:
- dns:query
- drop
- tcp
- flow
- icmp
- http
# CRITICAL: Enable Hubble export for SIEM consumption
export:
static:
enabled: true
file:
enabled: true
path: /var/log/hubble/flows.json
format: JSON
fieldMask: []
This writes flow logs to /var/log/hubble/flows.json on each node in a structured JSON format.
Step 2: Rewrite the SIEM Pipeline
I replaced the conntrack sidecar with a Hubble reader:
# fluent-bit-daemonset.yaml
volumeMounts:
- name: hubble-logs
mountPath: /var/log/hubble
readOnly: true
volumes:
- name: hubble-logs
hostPath:
path: /var/log/hubble
type: DirectoryOrCreate
And updated the Fluent Bit input:
[INPUT]
Name tail
Path /var/log/hubble/flows.json
Parser json
Tag k8s.network-flows
# Hubble JSON has nested objects; flatten them
Mem_Buf_Limit 50MB
Step 3: Schema Mapping (The Annoying Part)
Conntrack JSON and Hubble JSON have completely different schemas.
Old (conntrack):
{
"src_ip": "10.0.1.15",
"dst_ip": "10.0.2.88",
"src_port": 54321,
"dst_port": 443,
"protocol": "tcp",
"state": "ESTABLISHED",
"timestamp": "2026-08-05T23:12:00Z"
}
New (Hubble):
{
"flow": {
"time": "2026-08-06T03:45:00Z",
"verdict": "FORWARDED",
"ethernet": {"source": "...", "destination": "..."},
"IP": {"source": "10.0.1.15", "destination": "10.0.2.88", "ipVersion": "IPv4"},
"l4": {
"TCP": {"source_port": 54321, "destination_port": 443}
},
"source": {"identity": 12345, "namespace": "frontend", "pod_name": "api-xxx"},
"destination": {"identity": 67890, "namespace": "backend", "pod_name": "db-yyy"},
"event_type": {"type": 4, "sub_type": 3},
"is_reply": false
}
}
Hubble gives you way more — pod identity, namespace, Kubernetes labels, L7 protocol details, policy verdict. But you have to rewrite your SIEM parsers.
I wrote a small Go transform (ironic, given how I learned it) to flatten Hubble JSON into the old ClickHouse schema, with extra columns for the new metadata:
// hubble-flatten/main.go
package main
import (
"encoding/json"
"fmt"
"os"
"time"
)
type HubbleFlow struct {
Flow struct {
Time string `json:"time"`
Verdict string `json:"verdict"`
IP struct {
Source string `json:"source"`
Destination string `json:"destination"`
} `json:"IP"`
L4 struct {
TCP struct {
SourcePort uint32 `json:"source_port"`
DestinationPort uint32 `json:"destination_port"`
} `json:"TCP"`
} `json:"l4"`
Source HubIdentity `json:"source"`
Destination HubIdentity `json:"destination"`
} `json:"flow"`
}
type HubIdentity struct {
Identity uint32 `json:"identity"`
Namespace string `json:"namespace"`
PodName string `json:"pod_name"`
}
func main() {
// Reads Hubble JSON from stdin, emits flat JSON to stdout
decoder := json.NewDecoder(os.Stdin)
encoder := json.NewEncoder(os.Stdout)
for decoder.More() {
var h HubbleFlow
if err := decoder.Decode(&h); err != nil {
continue
}
flat := map[string]interface{}{
"timestamp": parseTime(h.Flow.Time),
"src_ip": h.Flow.IP.Source,
"dst_ip": h.Flow.IP.Destination,
"src_port": h.Flow.L4.TCP.SourcePort,
"dst_port": h.Flow.L4.TCP.DestinationPort,
"protocol": "tcp",
"verdict": h.Flow.Verdict,
"src_namespace": h.Flow.Source.Namespace,
"src_pod": h.Flow.Source.PodName,
"dst_namespace": h.Flow.Destination.Namespace,
"dst_pod": h.Flow.Destination.PodName,
"src_identity": h.Flow.Source.Identity,
"dst_identity": h.Flow.Destination.Identity,
}
encoder.Encode(flat)
}
}
func parseTime(s string) int64 {
t, _ := time.Parse(time.RFC3339Nano, s)
return t.Unix()
}
Step 4: Replace Falco Network Rules with Hubble + Cilium Policy
Falco's syscall-based network detection doesn't work well in eBPF-only clusters because:
- eBPF socket load balancing happens before the syscall layer Falco hooks
- ClusterIP connections never appear as "outbound to external IP" because they're redirected at the socket layer
Solution: Use Cilium Network Policies for enforcement, Hubble for observability, and keep Falco for what it's good at (file integrity, process execution, privilege escalation).
# Example: Instead of Falco detecting unauthorized DB connections,
# use Cilium NetworkPolicy to deny them at the kernel level
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: backend-access-control
namespace: frontend
spec:
endpointSelector:
matchLabels:
app: frontend-api
egressDeny:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: backend
app: postgres
toPorts:
- ports:
- port: "5432"
rules:
http:
- method: "GET"
# But allow the legitimate API service in backend
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: backend
app: backend-api
This enforces at eBPF TC layer — before the packet even leaves the pod's network namespace. Falco could only tell you after it happened.
The Real Numbers
Performance Impact
| Metric | kube-proxy (iptables) | Cilium eBPF | Delta |
|---|---|---|---|
| Service Latency (p99) | 2.8ms | 1.1ms | -61% |
| NodePort Throughput | 1.2 Gbps | 3.8 Gbps | +217% |
| Connection Tracking Overhead | 18% CPU (conntrack) | 4% CPU (BPF) | -78% |
| New Service Propagation | 30-60s (iptables sync) | <1s (BPF map update) | -98% |
| Memory (per node) | 185MB (conntrack table) | 42MB (BPF maps) | -77% |
Observability Impact
| Capability | Before (conntrack) | After (Hubble eBPF) |
|---|---|---|
| Pod-to-Pod visibility | IP only | Pod name, namespace, labels, identity |
| L7 protocol detection | No | HTTP method, path, headers (with L7 proxy) |
| Policy verdict | No |
FORWARDED, DROPPED, AUDIT per packet |
| DNS visibility | No | Query name, response code, TTL |
| Encryption | No visibility | WireGuard vs cleartext flagged per flow |
| Legacy SIEM compatibility | ✅ Native | ❌ Requires Hubble export rewrite |
The One Packet That Broke Everything
Here's the specific failure that cost me 6 hours. I found it in the Hubble flow logs after I got everything working.
At 23:12:47, my Fluent Bit-to-Kafka pod in the logging namespace tried to connect to the Kafka bootstrap broker at kafka.logging.svc.cluster.local:9092.
With kube-proxy, this resolved to 10.43.120.15:9092 via iptables DNAT. Falco saw the connection to 10.43.120.15:9092 and logged it.
With eBPF socket load balancing, Cilium's bpf_sock program intercepted the DNS resolution and redirected the connection inside the kernel socket layer directly to the Kafka pod's IP (10.0.2.88:9092) without ever creating a conntrack entry or hitting iptables.
Falco's syscall probes saw a connection to 10.0.2.88:9092 but couldn't resolve that IP back to a Kubernetes service (there's no iptables rule mapping it anymore). So Falco discarded the event as "unknown destination."
My SIEM's conntrack scraper saw nothing because conntrack was empty.
Hubble saw everything — source pod, destination pod, protocol, policy verdict — but I hadn't enabled Hubble export yet.
That single Kafka bootstrap connection was the canary. Every pod in the cluster was connecting to services via eBPF redirection, and my entire security stack was missing all of them.
Why I'm Never Going Back
Despite the 6-hour panic, I'm keeping eBPF. Here's why:
1. Performance is absurd
61% latency reduction on service-to-service calls. On Raspberry Pi 4 nodes, that's the difference between "usable" and "painful."
2. Observability is actually better — once you migrate
Hubble flow logs tell me not just that pod A talked to pod B, but which Kubernetes identities were involved, whether a network policy allowed or dropped it, and what DNS query initiated it. Falco + conntrack could never give me that.
3. Security enforcement is earlier
Cilium Network Policies drop packets at the TC (Traffic Control) layer inside the kernel, before they ever hit the pod's network stack. iptables policies happen later. Earlier drop = smaller blast radius.
4. No more iptables lock contention
Adding a new service no longer triggers a 300ms iptables-restore that freezes all network paths. BPF map updates are atomic and microsecond-fast.
What I'd Do Differently
Enable Hubble BEFORE deleting kube-proxy. Not after. Not "to save resources." It's not optional.
Run both datapaths in parallel for 48 hours. Cilium supports
kubeProxyReplacement=disabledwith partial eBPF. I should have tested monitoring against both paths before committing.Validate SIEM parsers against Hubble JSON before the switch. Don't discover schema incompatibilities at 3 AM.
Don't delete the kube-proxy DaemonSet. Scale it to zero replicas instead. If eBPF breaks, you can scale it back up in 30 seconds instead of manually re-applying manifests.
TL;DR
- Replaced kube-proxy with Cilium eBPF for performance
- eBPF works great for traffic — breaks legacy security tools
- Falco's syscall-based network detection fails with socket-level load balancing
- SIEMs consuming conntrack data go blind — eBPF uses BPF maps, not netfilter
- Enable Hubble export before the switch — it's your only visibility source
- Hubble gives BETTER data (pod identity, policy verdict, L7) but requires parser rewrites
- Performance gains are real: -61% latency, +217% throughput, -78% CPU overhead
- Don't delete kube-proxy — scale to zero. Keep your escape hatch.
GitHub: beltagyy/homelab (Talos + Cilium configs will be public this week)
Author: Mohamed ElBeltagy (@beltagyy) — Cloud Security & K8s engineer at Siemens
Topic: #kubernetes #ebpf #cilium #security #devops #homelab #performance
Next article: I wrote a custom eBPF program to detect lateral movement in Kubernetes — and ran it against my own cluster. Want to see what it found?
Top comments (0)