Wiring eBPF-Based Profiling to Your Kubernetes Backend: Continuous CPU and Latency Flame Graphs Without Instrumentation Overhead

#webdev #programming

---
title: "eBPF-Based Continuous Profiling in Kubernetes: Flame Graphs Without the Overhead"
published: true
description: "Deploy Parca or Pyroscope as a Kubernetes DaemonSet for always-on CPU flame graphs. Correlate hot paths with HPA events — zero code changes, zero APM tax."
tags: kubernetes, devops, architecture, performance
canonical_url: https://blog.mvpfactory.co/ebpf-continuous-profiling-kubernetes
---

What You Will Build

By the end of this tutorial you will have Parca running as a Kubernetes DaemonSet, sampling CPU stack traces across every node at kernel level, with flame graphs you can timestamp-align to HPA scaling events. No SDK. No code changes. No 15–20% APM overhead.

Let me show you a pattern I use in every production Kubernetes cluster.

Prerequisites

Kubernetes cluster running kernel ≥ 4.9 (≥ 5.8 recommended for DWARF unwinding)
kubectl access with ability to create privileged workloads
An observability namespace (we will use observability)
Go or JVM microservices under real traffic load — this tool earns its keep there

Why Not Just Use an APM Agent?

Traditional APM agents extract a 15–20% CPU and memory tax just to tell you something is slow. eBPF programs run inside the Linux kernel's sandboxed VM, using perf_event_open syscalls to sample stack traces at a fixed interval (typically 19–99 Hz). Here is what that looks like in practice:

Approach	Overhead	Code Changes	Always-On
Agent-based APM (Datadog, New Relic)	15–20%	Yes (SDK)	Yes
async-profiler / pprof manual	Negligible	Yes (endpoint)	No
eBPF (Parca / Pyroscope)	1–3%	None	Yes
eBPF with DWARF unwinding	2–5%	None	Yes

The docs do not mention this, but for Go services this works out of the box — Go has preserved frame pointers by default since 1.12. JVM services need one extra flag, which we cover in the Gotchas section.

Step 1 — Deploy Parca Agent as a DaemonSet

One pod per node. It needs hostPID, hostNetwork, and a privileged security context to load its eBPF programs. Here is the minimal setup to get this working:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: parca-agent
  namespace: observability
spec:
  selector:
    matchLabels:
      app: parca-agent
  template:
    spec:
      hostPID: true
      hostNetwork: true
      containers:
      - name: parca-agent
        image: ghcr.io/parca-dev/parca-agent:latest
        securityContext:
          privileged: true
        args:
        - --node=$(NODE_NAME)
        - --remote-store-address=parca.observability.svc:7070
        - --remote-store-insecure
        volumeMounts:
        - mountPath: /sys/fs/cgroup
          name: cgroup
        - mountPath: /sys/kernel/debug
          name: kerneldebug
      volumes:
      - name: cgroup
        hostPath: { path: /sys/fs/cgroup }
      - name: kerneldebug
        hostPath: { path: /sys/kernel/debug }

The Parca server stores profiles in a time-series-aware format backed by object storage. Its query UI is Prometheus-compatible.

Step 2 — Correlate Flame Graphs With HPA Scaling Events

This is the part that actually earns its keep. When your HPA scales from 4 to 12 replicas under a mobile traffic surge, you want to know why CPU spiked, not just that it did.

Parca exposes a gRPC Query API. Annotate your HPA events into your observability pipeline, then timestamp-align them against flame graph deltas:

parcactl query \
  --start 2026-11-09T14:30:00Z \
  --end   2026-11-09T14:35:00Z \
  --label-selector 'app=payment-service' \
  --profile-type process_cpu:cpu:nanoseconds:cpu:nanoseconds

In my experience, the most expensive surprises are rarely in application logic. They show up in Jackson serialization, Protobuf reflection, connection pool contention, and G1GC humongous allocations — things that only appear past a certain concurrency threshold. Flame graphs timed to HPA events expose exactly this.

Gotchas

JVMs omit frame pointers by default. Without them, eBPF unwinding falls back to DWARF — slower and less accurate. Add this flag to every JVM service:

-XX:+PreserveFramePointer

This carries a measured 1–3% throughput cost. That is well below what you recover by dropping APM agents entirely, so it is not really a trade-off.

Hot paths under real mobile traffic do not appear locally. A JVM service under sustained load at 2,000 RPS will exhibit lock contention and GC pressure that simply does not appear at 50 RPS on a developer laptop. Always-on production profiling is the only way to see this. Do not profile in development and ship the fix without validating under production traffic patterns.

Privileged DaemonSet is a one-time ops conversation. Front-load it. What you get back is permanent visibility into production behavior without touching application code.

Conclusion

Three changes get you there: deploy Parca or Pyroscope as a DaemonSet, annotate HPA events into your observability pipeline, and add -XX:+PreserveFramePointer to all JVM services. That combination turns autoscaling from a reactive measure into a diagnostic tool — you see the root cause of load-induced CPU growth, not just the symptom.

Resources:

DEV Community