Syed Mannan Saood

Posted on Jul 28

OpenTelemetry eBPF Instrumentation: Zero-Code Observability Becomes a Standard

#kubernetes #devops #cloudnative #sre

TL;DR: For twenty years, getting distributed traces out of an application meant adding an SDK, a language agent, or a sidecar code changes, restarts, dependency upgrades, and per-language maintenance. OpenTelemetry eBPF Instrumentation (OBI), which reached alpha in November 2025 and beta at KubeCon EU 2026, captures HTTP/gRPC/SQL/Redis/Kafka traces directly from the kernel with zero code changes, by attaching to a running process from outside it entirely. It's the OpenTelemetry project's formal successor to Grafana Beyla, now co-developed with Splunk, Coralogix, and Odigos. The catch: doing this from outside the process requires kernel privileges most security teams have spent years trying to eliminate.

The Instrumentation Problem OBI Is Solving

Every observability approach before eBPF required getting into the process somehow.

Manual instrumentation: developers add SDK calls around every operation they want traced. Accurate, but it's ongoing engineering work multiplied across every service, every language, every internal library.

Auto-instrumentation agents: a language-specific agent (Java agent, Python auto-instrumentor, Node.js require-in-the-middle hook) attaches at startup and monkey-patches known libraries. Zero code changes, but it's still in-process; it adds a runtime dependency, requires a restart to attach, and if that dependency has a vulnerability or a memory leak, it's now your application's problem too.

Service mesh sidecars: move instrumentation to a proxy sitting next to the container. Solves the in-process dependency problem, but you're now running an extra container per pod, and the proxy still only sees traffic at the network boundary, not internal function calls.

Every one of these approaches has the same shape: something has to be added to, or deployed alongside, the thing you want to observe.

eBPF-based instrumentation breaks that pattern. It attaches from outside the process, at the kernel level, without touching the application's code, dependencies, or container image at all.

What OBI Actually Is

OBI's official description: it "runs out-of-process and instruments at the protocol level, rather than at the library level." That phrasing matters; it's not patching a specific HTTP client library's send function. It's watching the actual bytes on the wire and in relevant kernel/library data structures, and reconstructing HTTP/gRPC/SQL/Redis/Kafka semantics from that.

The practical claims from the OpenTelemetry project's own announcement:

No restarts, no code changes, no configuration changes; telemetry capture starts on an already-running process
No new application dependencies, so no new dependency-introduced vulnerabilities
Minimal CPU/memory footprint even at high request rates, because the heavy lifting happens in the kernel
Automatic W3C trace context propagation across every supported language, without each language needing its own context-propagation implementation
Protocol coverage: HTTP/HTTPS, HTTP/2, gRPC, SQL, Redis, MongoDB, Kafka, GraphQL, Elasticsearch/OpenSearch, AWS S3

It detects when an application is already instrumented with a proper OpenTelemetry SDK and avoids duplicating those signals, meaning it's explicitly designed to coexist with, not replace, existing instrumentation.

Lineage: From Beyla to OBI

OBI didn't start from nothing. It's the direct continuation of Grafana Beyla, first released by Grafana Labs in September 2023, built specifically to auto-instrument compiled languages like Go and Rust where you can't easily inject a runtime agent the way you can with Java or Python.

Earlier in 2025, Grafana Labs donated Beyla to the OpenTelemetry project. The first alpha release under the new name OpenTelemetry eBPF Instrumentation landed November 3, 2025, co-authored by maintainers from Grafana Labs and Splunk, with Coralogix and Odigos also contributing. Splunk then announced the beta at KubeCon + CloudNativeCon Europe 2026 in Amsterdam, alongside general availability of the Splunk Operator for Kubernetes.

Why does the donation-and-rename matter architecturally, not just organizationally? Before OBI, the eBPF observability space was fragmented by vendor: Pixie was tied to New Relic, Beyla skewed toward Grafana Cloud, and Cilium Hubble covered network flows (L3/L4) but stopped before application-level tracing. OBI is explicitly positioned as the vendor-neutral convergence point; a CNCF Observability TAG survey found 67% of production Kubernetes clusters were already running at least one eBPF-based observability tool by early 2026, which is the adoption curve that made a standard layer worth building.

Architecture: Two Spaces, One Pipeline

OBI's system architecture (per the project's own DeepWiki-documented internals) splits cleanly into kernel space and user space.

Kernel space: eBPF programs attached via kprobes, uprobes, and TC (traffic control) hooks capture network and application events, storing intermediate state in BPF maps and the shared memory structures that let kernel-space eBPF programs and user-space processes exchange data safely.

User space: a discovery subsystem identifies which running processes should be instrumented (based on configurable criteria process name, port, Kubernetes labels), a tracer component controls the eBPF program lifecycle (loading, attaching, unloading), reader components consume events out of the BPF maps, processing layers enrich and normalize that raw data into OpenTelemetry's semantic conventions, and exporters ship the final metrics/traces/spans to a backend.

The mechanism for capturing HTTP/gRPC data specifically:

1. OBI inspects the target binary to identify language/framework
2. Attaches uprobes at entry/return points of relevant functions
   (e.g., Go's net/http handler functions, gRPC stream methods)
3. Attaches to TLS library functions (e.g., OpenSSL) to read
   plaintext BEFORE encryption / AFTER decryption
4. Captures request/response metadata directly from kernel
   and userspace data structures at those hook points
5. Correlates request/response pairs, computes duration,
   status code, route assembles a span
6. Injects/reads W3C traceparent headers for cross-service correlation

Step 3 is worth pausing on: because a uprobe sits at the function boundary inside the process's own address space, OBI can read HTTP payloads before TLS encrypts them on the way out, and after TLS decrypts them on the way in without ever touching the TLS handshake or terminating encryption anywhere. This is a meaningfully different approach from a network proxy, which only ever sees encrypted bytes unless it's doing active TLS termination.

The uprobe-only Pivot

Here's a detail that didn't make it into the launch announcements but is architecturally significant: OBI removed kprobe-based collection in favour of a uprobe-only approach (tracked in PR #752 on the project's GitHub).

Why this matters: kprobes attach to kernel functions and syscalls, meaning a kprobe-based collector sits in the network datapath, observing every packet that crosses a syscall boundary, regardless of which process it belongs to. Uprobes attach to specific user-space functions inside a specific process, meaning the instrumentation only fires when that particular application code path executes.

The trade-off is explicit, and it's the same one that shows up in Cilium Hubble vs. Tetragon vs. Beyla comparisons generally:

kprobe/network-datapath approach: broader visibility (every packet, every process), but adds per-packet latency risk and a larger blast radius if something in the hot path misbehaves
uprobe-only approach: narrower, application-scoped visibility, lower risk to network connectivity and packet forwarding, but you lose the ability to see traffic OBI wasn't specifically told to watch for

OBI's maintainers chose the narrower, safer default. If you're building or evaluating a network-observability tool that does sit in the datapath packet inspection, flow tracking, that category, this is precisely the design fork you're navigating, and OBI's choice tells you which side of the safety/coverage trade-off the OpenTelemetry community landed on for application tracing specifically. It doesn't mean the datapath approach is wrong; Cilium Hubble and Tetragon exist specifically because network-layer and security-enforcement use cases need that broader vantage point. It means OBI scoped itself deliberately to stay out of that territory.

What Zero-Code Actually Costs: Kernel Privileges

Here's the part vendor announcements gloss over. To load eBPF programs, attach uprobes to arbitrary processes, and resolve symbols for accurate function-boundary hooking, OBI needs real kernel privileges:

CAP_BPF load and manage eBPF programs
CAP_PERFMON open perf event buffers, attach kprobes/uprobes
CAP_SYS_PTRACE read process memory for symbol resolution
CAP_NET_ADMIN required specifically for TC eBPF socket-level hooks (kernel 5.8+)

The practical result, as one security-focused writeup on Beyla's model puts it plainly: this is "a process that can read arbitrary kernel structures, load unrestricted eBPF programs, and attach to any process anywhere on the node." That's not a criticism of implementation quality; it's the structural cost of a tool whose entire value proposition is "see everything without being told about it in advance."

The data-scope consequence: because OBI captures full HTTP request paths, query strings, and response codes directly from kernel-level interception, any PII embedded in a URL user IDs, session tokens, search parameters, record identifiers flows straight into trace output. This isn't a bug; it's the same category of exposure any full-request-capture tool has, but the "zero code, drop it in and go" pitch makes it easy to deploy before anyone's reviewed what's actually landing in your trace backend.

For production deployment, the security-hardening path is to replace the common privileged: true DaemonSet config with the specific minimal capability set above, plus allowPrivilegeEscalation: false and a read-only root filesystem, none of which is the default in most getting-started guides.

Where OBI Genuinely Struggles

The OpenTelemetry project's own release notes are unusually candid about current limitations, which is worth taking at face value rather than reading as false modesty:

Distributed tracing quality varies sharply by language/runtime. OBI works well for Go (HTTP and gRPC), Node.js (HTTP), Python (HTTP), NGINX (HTTP), and PHP (HTTP/FPM). It currently does not handle distributed tracing well for:

Reactive programming frameworks
Java virtual threads
Complex thread pools

The underlying reason is structural, not a missing feature: uprobe-based correlation relies on being able to associate an incoming request with the specific execution context that handles it. Reactive frameworks and virtual threads deliberately decouple "the thread that received the request" from "the thread that ultimately processes it", exactly the assumption uprobe-based request/response pairing depends on.

It's additive, not a replacement. The project's own guidance is direct: if you've successfully instrumented a service with a proper OpenTelemetry SDK or agent, there's rarely a reason to rip that out for OBI, unless you're hitting specific performance or cost problems the SDK approach caused. OBI's stated best use cases are (a) getting any telemetry out of currently-uninstrumented services, especially compiled binaries where SDK instrumentation is awkward, and (b) covering libraries that don't have official OpenTelemetry support at all: legacy versions, unmaintained packages, anything nobody's written an instrumentation for.

The 2026 Roadmap

The OBI SIG's stated 2026 priorities, per their published goals:

Stable 1.0 release the flagship goal, requiring documentation completeness, configuration standardisation, and production-readiness validation
Aligning network attributes with OpenTelemetry semantic conventions, and updating all semantic convention usage to current versions
An OpenTelemetry Collector distribution with OBI as a receiver, meaning OBI becomes a pluggable data source inside the standard Collector pipeline, rather than a standalone agent with its own export path
Integration with the OpenTelemetry eBPF profiler for unified observability (metrics, traces, and continuous profiling from the same kernel-level vantage point)
Runtime metrics directly from OBI, without needing a separate metrics exporter

The Collector-receiver integration is the one worth watching most closely; it's the difference between OBI as a standalone tool you deploy and manage separately, versus OBI as one interchangeable input into whatever OpenTelemetry pipeline you're already running.

The Architectural Question This Raises

If you're building or evaluating any eBPF-based tooling, whether that's application tracing, network flow visibility, or packet-level inspection, OBI's trajectory says something concrete about where the ecosystem is converging:

Vendor-specific eBPF tools are consolidating into CNCF-governed standards. The Beyla → OBI donation, following Cilium's earlier CNCF graduation, suggests this is the pattern going forward rather than a one-off.
The uprobe-vs-kprobe scoping decision is a real architectural fork, not an implementation detail, and it's one every eBPF observability or security tool has to make explicitly, trading breadth of visibility against blast radius and privilege requirements.
"Zero-code" doesn't mean "zero operational cost." It relocates the cost from application-code maintenance to kernel-privilege management and PII-in-traces governance a different problem, not a solved one.
Coexistence, not replacement, is the honest framing. OBI fills gaps around existing SDK instrumentation rather than obsoleting it, and the project says so explicitly rather than overselling.

Key Takeaways

Zero-code instrumentation is real and works well for a specific slice of the problem HTTP/gRPC/SQL tracing for mainstream frameworks in Go, Python, Node.js, PHP, and NGINX, especially for services you can't or don't want to modify.

The uprobe-only architecture is a deliberate safety trade-off, not a limitation; it trades broader kprobe-level visibility for lower blast radius in the network datapath, which is the right call for application tracing even though it means OBI can't see traffic patterns a network-layer tool would.

Kernel privilege requirements are substantial and often under-hardened by default CAP_BPF, CAP_PERFMON, CAP_SYS_PTRACE, and CAP_NET_ADMIN together grant broad node-level access that most getting-started guides deploy with privileged: true rather than the minimal capability set.

Distributed tracing correlation breaks down exactly where execution context decouples from the receiving thread reactive frameworks, virtual threads, complex pools because uprobe-based pairing fundamentally assumes a stable request-to-thread relationship.

This is consolidation, not novelty. OBI's significance isn't a new technical capability; Beyla already did most of this it's that the OpenTelemetry Governance Committee now owns it, which is what turns a good vendor tool into infrastructure other tools build on top of.

DEV Community