---
title: "eBPF Observability: Zero-Code Latency Tracing in Kubernetes"
published: true
description: "Learn how eBPF kernel programs capture inter-service latency, TCP retransmits, and syscall profiles inside Kubernetes pods — no sidecars, no code changes, production-ready on a budget."
tags: [devops, kubernetes, performance, cloud]
canonical_url: https://blog.mvp-factory.dev/ebpf-observability-zero-code-latency-tracing-kubernetes
---
## What We Will Build
By the end of this tutorial, you will have a working mental model — and enough working code — to run a lightweight eBPF DaemonSet that captures inter-service TCP latency, retransmit counts, and syscall profiles from inside your Kubernetes pods. No application changes. No Envoy sidecars. No budget explosion.
Let me show you a pattern I use in every project where someone is about to reach for Istio before they have actually counted the cost.
---
## Prerequisites
- Linux kernel 5.4+ on your nodes (CO-RE requires BTF support)
- `kubectl` access to a running cluster
- Go 1.21+ for the userspace daemon
- Basic familiarity with Kubernetes DaemonSets
- `clang` and `libbpf` installed in your build environment
---
## The Problem with Sidecar-First Observability
Most teams reach for Envoy or Istio before understanding the cost. A service mesh sidecar adds **50–200ms cold-start latency**, **~50MB of memory per pod**, and CPU overhead that compounds as you scale horizontally. For a startup running 40 microservices on a mid-tier Kubernetes cluster, that overhead is not trivial — it is a budget line item.
eBPF changes the calculus entirely.
---
## Step 1: Understand What eBPF Actually Does
eBPF (Extended Berkeley Packet Filter) programs run inside the Linux kernel in a sandboxed JIT-compiled VM. They attach to tracepoints, kprobes, and uprobes — giving you direct access to network events, scheduler decisions, and syscall execution without any userspace instrumentation.
For inter-service latency tracing, you attach to `tcp_sendmsg` and `tcp_recvmsg` kernel functions and correlate timestamps using BPF maps — shared memory structures readable from both kernel and userspace.
Here is the minimal setup to get this working:
c
// Simplified BPF program: capture TCP send timestamp
SEC("kprobe/tcp_sendmsg")
int trace_tcp_send(struct pt_regs *ctx) {
u64 pid = bpf_get_current_pid_tgid();
u64 ts = bpf_ktime_get_ns();
bpf_map_update_elem(&send_start, &pid, &ts, BPF_ANY);
return 0;
}
SEC("kretprobe/tcp_recvmsg")
int trace_tcp_recv(struct pt_regs *ctx) {
u64 pid = bpf_get_current_pid_tgid();
u64 *tsp = bpf_map_lookup_elem(&send_start, &pid);
if (tsp) {
u64 latency_ns = bpf_ktime_get_ns() - *tsp;
bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
&latency_ns, sizeof(latency_ns));
}
return 0;
}
The kernel emits events into a perf ring buffer. A Go or Rust userspace daemon reads them and exports metrics. No application change required.
---
## Step 2: Design Your BPF Map Architecture
BPF maps are typed, bounded, and kernel-managed. For multi-service Kubernetes environments, a hash map keyed by `(pid, container_id)` lets you correlate latency to a specific pod without cross-contamination.
| Map type | Use case | Lookup complexity |
|---|---|---|
| `BPF_MAP_TYPE_HASH` | PID → timestamp correlation | O(1) average |
| `BPF_MAP_TYPE_PERF_EVENT_ARRAY` | Streaming events to userspace | O(1) per CPU |
| `BPF_MAP_TYPE_RINGBUF` | High-throughput trace export | Lock-free, lower overhead than perf |
| `BPF_MAP_TYPE_LRU_HASH` | Connection tracking at scale | Auto-evicts stale entries |
For TCP retransmit detection, attach to the `tcp_retransmit_skb` tracepoint and aggregate counts per `(src_ip, dst_ip, dport)` in an LRU hash. You get network-layer signal with zero application awareness.
---
## Step 3: Use CO-RE for Kernel Portability
The historical barrier to eBPF adoption was kernel version fragmentation. CO-RE (Compile Once, Run Everywhere), enabled by BTF (BPF Type Format) and `libbpf`, eliminates this. Your eBPF object file ships with type information and the loader relocates field offsets at load time against the running kernel's BTF data.
The docs do not mention this, but the practical workflow is: compile against kernel headers once in CI, ship the `.o` file as a container image layer, and load it on kernels from 5.4 through 6.x without recompilation.
This makes eBPF viable in heterogeneous cloud environments where node kernel versions drift across node pools. It sounds like a minor toolchain detail until you have actually managed a fleet with mixed kernel versions — then it becomes the whole reason the approach is practical.
---
## Step 4: Wire Into Grafana and Tempo
On tight infrastructure budgets, here is the pipeline I use:
1. eBPF daemon (Go + `cilium/ebpf` library) reads ring buffer events
2. OpenTelemetry Collector receives spans via OTLP, batches and compresses them
3. Grafana Tempo stores traces in object storage (S3/GCS) — roughly **$0.02/GB/month** vs managed APM at $0.10–$0.30/GB
4. Grafana renders service maps from trace data, no Jaeger or Zipkin dependency
Compared to a full Istio mesh, this stack runs in one DaemonSet pod per node with a typical footprint of **~30MB RAM and <0.5% CPU per core**. You get 90% of the observability value at roughly 15% of the infrastructure cost.
That gap is large enough that it is worth treating the eBPF path as the default and only reaching for a service mesh when you genuinely need its traffic management features.
---
## Observability Stack Comparison
| Approach | Code changes | Memory overhead | Kernel visibility | Setup complexity |
|---|---|---|---|---|
| Envoy sidecar (Istio) | None | ~50MB/pod | L7 only | High |
| Manual SDK (OTel) | Yes | Minimal | App layer only | Medium |
| eBPF DaemonSet | None | ~30MB/node | L3–L7 + syscalls | Medium |
| No instrumentation | None | None | None | N/A |
---
## Gotchas
**Key your BPF maps correctly.** This is the gotcha that will save you hours. Use `(netns_ino, pid)` as your map key — not just PID. In a shared-kernel Kubernetes node, PIDs can collide across containers. Namespacing by network namespace inode prevents cross-pod data leakage entirely.
**Route through OTel Collector before Tempo.** Do not send eBPF events directly to Tempo. The Collector gives you batching, retry logic, and the ability to swap backends later without touching your BPF code.
**BTF must be enabled on your nodes.** Verify with `ls /sys/kernel/btf/vmlinux`. If that file does not exist, CO-RE will not work and you will hit confusing load errors at runtime.
**Start with `cilium/ebpf` in Go.** It has the most mature CO-RE support, active maintenance, and clean APIs for map management and ring buffer consumption. You can ship your first latency histogram in under a day.
---
## Conclusion
eBPF gives you kernel-level observability — TCP latency, retransmits, syscall profiles — at a fraction of the memory and CPU cost of a service mesh. The combination of CO-RE portability, BPF ring buffers, and a lean OTel → Tempo → Grafana pipeline is production-ready today and runs comfortably on startup-scale infrastructure.
Reserve Istio for when you genuinely need its traffic management. For pure observability, eBPF is the better default.
**Resources:**
- [`cilium/ebpf` Go library](https://github.com/cilium/ebpf)
- [libbpf and CO-RE documentation](https://libbpf.readthedocs.io)
- [Grafana Tempo docs](https://grafana.com/docs/tempo/latest/)
- [BPF map types reference](https://docs.kernel.org/bpf/maps.html)
Top comments (0)