DEV Community

Cover image for The OpenTelemetry Ecosystem Is Strong — DataBuff Can Plug Into Its eBPF Trace Path
databufflabs
databufflabs

Posted on Originally published at databuff.ai

The OpenTelemetry Ecosystem Is Strong — DataBuff Can Plug Into Its eBPF Trace Path

#opentelemetry #ebpf #kubernetes #devops

The OpenTelemetry community packaged eBPF trace collection as OBI (OpenTelemetry eBPF Instrumentation). On Kubernetes you run a DaemonSet per node; eBPF watches HTTP calls between local services, assembles traces, and exports them — app Pods stay unchanged. Image: otel/ebpf-instrument. Project: opentelemetry-ebpf-instrumentation.

Where do traces land? DataBuff is an open-source APM platform for services, topology, and call chains. Point OBI at DataBuff and you're done.

One DaemonSet, one collector Pod per node — ns obi, app Pods untouched, eBPF on local HTTP → export to DataBuff.

Step 1 · Install DataBuff 

curl -fsSL https://databuff.ai/databuff/ai-apm-k8s-install.sh | bash
kubectl -n databuff get pods
Enter fullscreen mode Exit fullscreen mode

All Pods Running and the UI opens — you're good.

Step 2 · Check eBPF readiness on app nodes 

uname -r
ls /sys/kernel/btf/vmlinux
Enter fullscreen mode Exit fullscreen mode

Kernel 5.8+ recommended; the second command must list a file. Without BTF, collector Pods won't capture.

Step 3 · Image

Use otel/ebpf-instrument:latest (worked in our test; pin a version in production). Skip if the cluster can pull; offline clusters need docker load on nodes.

Step 4 · Apply the DaemonSet

Replace YOUR_APP_NAMESPACE and YOUR_DATABUFF_HOST (ai-apm-ingest.databuff.svc). hostPID + privileged are required.

Key config:

  • discovery.instrument.k8s_namespace
  • ebpf.context_propagation: headers
  • otel_traces_export.endpoint: http://YOUR_DATABUFF_HOST:4318 
kubectl apply -f obi.yaml
kubectl -n obi get ds,pods -o wide
Enter fullscreen mode Exit fullscreen mode

DESIRED / READY should match node count.

DaemonSet READY 5/5

Step 5 · Logs 

kubectl -n obi logs -l app=obi --tail=80 | grep -iE "instrumenting|process|error" | head -30
Enter fullscreen mode Exit fullscreen mode

Look for instrumenting process.

Step 6 · Traffic 

for i in $(seq 1 80); do
  curl -sS -m 2 "http://your-app-url/" >/dev/null || true
  sleep 0.2
done
Enter fullscreen mode Exit fullscreen mode

Step 7 · Verify in DataBuff

Open APM → Services, then topology and trace detail.

Service list

Topology

Trace list

Trace waterfall

eBPF vs language Agent

Approach Better when
eBPF + DaemonSet No injection/restart; HTTP-first; polyglot quick layer
Language Agent Dubbo, slow SQL, method stacks; kernel < 5.8 / no BTF

Limitations: no Dubbo RPC yet; no method stacks or custom business spans; privileged Pod + BTF required.

How multi-hop traces connect

With context_propagation: headers and no app code changes:

  1. Ingress — read Traceparent: on incoming HTTP; attach or create trace context.
  2. Local correlate — match outbound HTTP to the inbound request.
  3. Egress — sockmap sk_msg inserts this hop's Traceparent after the request line.

HTTPS uses a separate TCP Option path upstream; this walkthrough is plain HTTP + headers.

Full bilingual post (with more YAML detail): databuff.ai/blog/en/databuff-obi-ebpf-traces

Top comments (2)

Collapse
 
topstar_ai profile image
Luis Cruz •

I found the comparison between eBPF and language agents particularly insightful, highlighting the trade-offs between the two approaches. The table outlining when each approach is better suited is a great resource, emphasizing the importance of considering factors like injection, restarts, and protocol support when choosing between eBPF and language agents. The mention of limitations, such as the current lack of support for Dubbo RPC and method stacks, is also appreciated, as it sets clear expectations for the current capabilities of the OpenTelemetry eBPF Instrumentation. How do you envision the eBPF approach evolving to address some of these limitations, especially in scenarios where kernel version or BTF support is a concern?

Collapse
 
databufflabs profile image
databufflabs •

Thanks — that table is the part we hoped people would keep. Those limitations aren’t footnotes; they’re the selection criteria.

I’d split the gaps into two buckets, because they don’t evolve the same way.

Protocol / depth (Dubbo, method stacks, custom spans) can move. OBI sits at HTTP / gRPC / SQL boundaries today. Upstream’s 2026 work is more protocols (messaging, NoSQL, cloud SDKs) plus hybrid with language SDKs, so one hop can be eBPF and the next an agent without breaking the trace. Dubbo isn’t on that published list, and method stacks probably never become an eBPF strength — you’re outside the process. So the picture we use is: eBPF as the no-restart HTTP layer, agents where you need RPC / SQL / method detail. DataBuff just takes OTLP from either side.

Kernel / BTF is a kernel contract, not a knob we can relax. OBI needs BTF (/sys/kernel/btf/vmlinux). Older than 5.8 without the RHEL-family 4.18+ backports is out of scope; “eBPF without BTF” isn’t a realistic path. What actually changes is nodes moving to kernels that already ship BTF (most distros from 5.14+). Until a node has that, the language-agent row in the table is the answer — not a fallback we’re embarrassed about.

On a mixed cluster we’d run OBI only on nodes that pass those two checks, and put agents on the rest / on Dubbo services. Same backend, different collectors.