---
title: "eBPF Observability for Mobile Backends: Zero-Instrumentation Tracing Without the APM Tax"
published: true
description: "Skip the APM tax. eBPF kernel probes deliver automatic HTTP/gRPC tracing, syscall latency, and CPU flame graphs for your mobile backend — zero instrumentation, zero SDK coupling."
tags: devops, api, performance, cloud
canonical_url: https://mvpfactory.co/blog/ebpf-observability-mobile-backends
---
## What we are building
Let me show you a pattern I use in every production mobile backend that processes serious request volume: kernel-level observability with **zero application code changes**.
By the end of this walkthrough you will have eBPF probes attached to your backend processes capturing HTTP/gRPC spans, syscall latency, and per-container CPU flame graphs — all without an APM SDK, without per-span billing, and without adding a single import to your Kotlin or Go service.
---
## Prerequisites
- **Linux kernel ≥ 5.8** — eBPF CO-RE and BTF (BPF Type Format) support are required. Verify with:
bash
uname -r
ls /sys/kernel/btf/vmlinux
Most modern managed node pools ship compliant kernels by default (EKS 1.27+, GKE 1.26+).
- Kubernetes cluster with DaemonSet deployment rights
- `CAP_BPF` or privileged pod access (more on why this matters in Gotchas)
---
## Why this exists: the hidden cost of SDK-based APM
The financial cost of Datadog, New Relic, or Dynatrace at scale is visible — $3,000–$6,000/month for 10 billion spans is a real line-item conversation. The hidden cost rarely makes it into the postmortem: library coupling, SDK version drift, and misconfigured spans generating alert fatigue.
Here is the comparison that drives the decision:
| Approach | Instrumentation Effort | Avg Overhead | Monthly Cost (10B spans/month) |
|---|---|---|---|
| Datadog APM (SDK) | High | 4–8ms/req | $3,000–$6,000 |
| OpenTelemetry (manual) | Medium-High | 2–5ms/req | Infrastructure only |
| eBPF (kernel probes) | Near-zero | <0.5ms/req | Infrastructure only |
*Datadog pricing also scales with host count — actual bills vary significantly by deployment size.*
---
## How eBPF span capture works
eBPF programs run inside the Linux kernel in a sandboxed VM. The relevant probe points for HTTP/gRPC observability are:
- `tcp_sendmsg` / `tcp_recvmsg` — capture request/response bytes at the socket layer
- `sys_enter_read` / `sys_exit_read` — attribute syscall latency per process/container
- `sched_switch` — build CPU flame graphs without `perf` overhead
Here is the minimal kprobe to tag spans at the TCP layer:
c
SEC("kprobe/tcp_sendmsg")
int trace_tcp_send(struct pt_regs *ctx) {
struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);
u64 pid_tgid = bpf_get_current_pid_tgid();
u64 cgroup_id = bpf_get_current_cgroup_id();
bpf_ringbuf_output(&events, &event, sizeof(event), 0);
return 0;
}
A lightweight userspace collector reads the ring buffer, reconstructs spans, and exports to Jaeger, Grafana Tempo, or any OTLP endpoint.
---
## Wiring it to your mobile backend
Here is the minimal setup to get this working for a Kotlin/Spring or Go gRPC backend:
Mobile Client
│
▼
API Gateway (nginx/envoy)
│
▼
Backend Service ← no APM SDK
│ ← eBPF probes attach here at kernel level
▼
eBPF Collector (node-level DaemonSet)
│
▼
OTLP Exporter → Grafana Tempo / Jaeger
The collector runs as a **DaemonSet — one per node, not per pod**. You are not paying the per-container memory overhead of a full APM agent on every replica.
For gRPC specifically, HTTP/2 binary framing means you need protocol-aware parsing. **Grafana Beyla** and **Pixie** handle this out of the box. Beyla has first-class JVM support via bytecode-level uprobes when kernel probes alone cannot reconstruct application-layer context.
For continuous CPU profiling, `perf_event` + `bpf_get_stackid` adds roughly **0.1–1% CPU overhead** versus 5–15% for traditional samplers. For JVM backends, combine kernel stacks with async-profiler output for full mixed-mode flame graphs — kernel frames and JVM frames in a single view.
---
## Gotchas
Here is the gotcha that will save you hours.
**eBPF collectors require privileged access.** They need `CAP_BPF` or a privileged pod. Treat node-level access with the same scrutiny you apply to any privileged workload — audit your RBAC policies before rolling to production.
**Kernel version compliance bites you in staging.** Your local dev machine or a legacy staging node may not have BTF enabled. Always confirm with `/sys/kernel/btf/vmlinux` before assuming the collector will load its programs.
**gRPC without a tool that understands HTTP/2 produces garbage spans.** Raw socket-layer probes alone will not reconstruct application context from binary-framed gRPC. Start with Beyla or Pixie — both handle this without manual protocol parsing.
---
## Conclusion
Start with Grafana Beyla or Pixie before writing any custom eBPF. Both support automatic HTTP/gRPC span capture for JVM and Go backends with a single DaemonSet deploy. Export via OTLP and you stay completely vendor-neutral — your data goes to Tempo or Jaeger today, and you never pay per-span ingestion fees regardless of how your traffic grows.
The docs do not mention this, but the operational complexity here lives entirely at the cluster level, not in your application code. For mobile backends at scale, that is exactly the tradeoff worth making.
---
*Resources: [Grafana Beyla](https://grafana.com/oss/beyla/) · [Pixie](https://px.dev/) · [Cilium eBPF library](https://github.com/cilium/ebpf)*
Top comments (0)