Replacing tc netem with eBPF: 10x Latency Precision, 80% CPU Cut

At 3 AM, I was woken up by an alert: "Service 504 timeout." Checking the logs, I found that a microservice call timed out after 2 seconds, even though the code explicitly set a 500ms timeout. After a long investigation, I pinpointed the cause: a colleague doing chaos engineering had used tc netem to inject network latency into a container, which ended up increasing jitter across the entire host by 30% and taking down my service with it. Right then I decided: I had to find a fault-injection solution that is non-intrusive to containers, more accurate, and won’t drag down the host.

Problem Breakdown: Why Container Network Fault Injection Is So Tricky

When stress-testing service high availability, we often need to simulate network anomalies—latency, packet loss, reordering, bandwidth limits. Traditionally, we used tc netem to configure queueing disciplines on container NICs. That worked well enough in the VM era, but in Kubernetes container environments, three fatal flaws emerge:

1. Host-wide Jitter: The container veth pair shares the netns with the host. Manipulating ingress with tc requires the ifb module. Any misconfiguration can “leak” latency to other containers on the same host.
2. Terrible Precision: netem depends on kernel softirq scheduling, leading to latency errors of over 10ms under high concurrency, while we need to simulate precise 1ms-level jitter.
3. Intrusive Operations: Every injection requires entering the container netns and running commands. The scripts are fragile and cannot be trusted in CI/CD pipelines.

We also considered using service mesh sidecars for fault injection, like Istio’s Fault Injection. But it can only inject application-layer HTTP delays/aborts—it cannot simulate packet loss during TCP handshakes or do physical-layer faults targeting a single Pod. Moreover, the sidecar itself demands injecting a proxy container, which completely violates the zero-intrusion obsession.

Solution Design: A Transparent Millisecond-Level Network Chaos Architecture with eBPF

The weapon I chose is eBPF + Go. In short: write an eBPF program that runs in the kernel’s tc subsystem, precisely injecting faults when packets enter or leave a container’s NIC, and build a Go-based control plane that manages “which Pod, what type of fault, for how long.”

Why I discarded the other options:

• XDP elimination: XDP hooks at the NIC driver level, primarily for physical NICs. Its support for per-container virtual devices like veth is poor, and it only handles ingress—not flexible enough.
• User-space transparent proxying: It changes the network topology, turning direct container IP connections into proxy hops. Troubleshooting becomes a nightmare.
• CNI chain plugins: Too invasive, requiring Pod restart to take effect, unable to dynamically inject at runtime.

The final architecture has just three layers: a Go manager uses the cilium/ebpf library to load eBPF programs onto the tc clsact hook of the target container’s veth; the eBPF program reads a shared map, matches packets by 5-tuple + fault type, and performs delay or drop; the Go side exposes a gRPC interface for CI or stress-test scripts. The entire process never enters the container network namespace, modifies no container files, and requires no privileged container (only host CAP_BPF).

Core Implementation: Three Tricks for Millisecond-Precision Latency Injection

I’ve put the full code on GitHub in the netchaos repo. Here, I’ll dissect the three most impressive pieces.

1. eBPF Program: “Hijacking” Packets at the tc Hook and Delaying Them Precisely

We use the clsact (classifier-action) hook of tc. It doesn’t rely on the netem queue; instead, it directly manipulates packets with eBPF helper functions. The following C file is the soul of the solution. Notice the delay_pkt function—we use bpf_timer_start instead of a traditional busy loop, avoiding CPU hogging.

// ebpf/tc_delay.c
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

// 故障规则：五元组掩码 + 故障类型 + 延迟毫秒数
struct fault_rule {
    __u32 saddr;
    __u32 daddr;
    __u16 sport;
    __u16 dport;
    __u8  protocol;
    __u8  fault_type;   // 0:丢包 1:延迟
    __u32 delay_ms;
};

// BPF map：Go 端通过它下发故障指令
struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, struct fault_rule);
} fault_map SEC(".maps");

// 定时器 map：保存待延时触发的包
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, struct sk_buff *);
    __type(value, struct bpf_timer);
} timer_map SEC(".maps");

// 延时到期后回调：重新注入包到内核协议栈
static int delay_timeout(void *map, int *key, struct bpf_timer *timer) {
    struct __sk_buff *skb = (struct __sk_buff *)*key; // 夺回 skb 指针
    bpf_timer_cancel(timer);
    bpf_map_delete_elem(&timer_map, key);
    // TC_ACT_UNSPEC 让包继续正常流程，等效于“延时后放行”
    bpf_skb_change_tail(skb, 0, 0);  // 轻微重置，确保校验和正确
    return TC_ACT_UNSPEC;
}

SEC("tc")
int chaos_tc(struct __sk_buff *skb) {
    struct fault_rule rule = {};
    __u32 key = 0;
    bpf_map_lookup_elem(&fault_map, &key, &rule);
    if (rule.fault_type == 0) return TC_ACT_OK; // 无故障

    // 简易的五元组匹配（可扩展为 LPM）
    void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;
    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end) return TC_ACT_OK;
    if (eth->h_proto != bpf_htons(ETH_P_IP)) return TC_ACT_OK;

    struct iphd