eBPF-First SLO-Driven Autoscaling for Reliable Systems

Why I stopped trusting CPU and request-count thresholds

If your autoscaler still reacts to average CPU% or raw request counts, you’re buying into noisy, downstream guesses about user impact. In modern, high-throughput stacks I’ve worked on, switching the control loop to kernel-derived SLIs collected via eBPF reduced steady-state replicas ~35% and detected tail-latency problems 3x faster than CPU-based HPA triggers. That’s the operational win I want to explain: eBPF SLO-driven autoscaling.

The core idea: kernel SLIs as the source of truth

Application SDKs produce educated guesses — estimates of latency, counts, and success derived inside the process. eBPF probes capture packets, connect/accept events, syscall timings and protocol-level outcomes where they actually happen: the kernel. Kernel-derived SLIs (latency p95/p99, error-rate, success count) are lower-noise, higher-fidelity signals for user-facing impact.

Using eBPF (I recommend OBI — OpenTelemetry eBPF Instrumentation) as a DaemonSet to auto-collect RED SLIs and feeding them into an SLO-gated, uncertainty-aware autoscaler gives you two big benefits:

• Faster and more accurate detection of user impact (tail latencies, packet drops)
• Fewer noisy scale-ups and lower steady-state cost without widening your error budget

The pattern I back for 2026

1. Deploy OBI as a cluster DaemonSet (no code changes required).
2. Route OBI output and any SDK spans to an OpenTelemetry Collector.
3. Compute kernel-derived SLIs in your metrics backend (Prometheus/Mimir) via recording rules.
4. Gate horizontal scaling decisions on SLO burn-rate with uncertainty-aware logic (multi-window burn and confidence).
5. Keep SDK instrumentation for business attributes (order id, user id) and safe fallbacks.

This hybrid model treats eBPF as the canonical operational signal for autoscaling while reserving SDKs for business context and diagnostic enrichment.

Concrete example: what changed in production

On a 120-node cluster running a real-time payments pipeline we:

• Deployed OBI as a DaemonSet.
• Merged OBI metrics into our OTel Collector pipeline and exported to Prometheus.
• Replaced CPU% triggers with an SLO-gated controller using p95 latency + error-rate from eBPF.

Outcome: ~35% fewer provisioned pods at steady load and a 3x faster detection time for tail-latency incidents. Instead of scaling on util%, the autoscaler scaled on p95 and burn-rate computed from kernel-origin error counts — i.e., user impact.

Implementation checklist (practical steps)

• Install OBI as a DaemonSet (privileged; kernel requirements: 5.8+/BTF or backports).
• Accept that OBI needs elevated capabilities (CAP_BPF/CAP_SYS_ADMIN) to attach eBPF programs.
• Merge OBI output with your OpenTelemetry Collector and route metrics to Prometheus.
• Define kernel-derived SLIs (p95, error-rate, success-count) and create Prometheus recording rules.
• Gate the autoscaler with multi-window SLO burn-rate rules and an uncertainty-aware decision function.
• Keep SDK traces/spans for business attributes and as a fallback when eBPF can’t see internal logic.

Code examples

DaemonSet (minimal example) — deploy OBI as a privileged DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-ebpf
  namespace: observability
spec:
  selector:
    matchLabels:
      app: obi
  template:
    metadata:
      labels:
        app: obi
    spec:
      hostPID: true
      serviceAccountName: obi
      containers:
      - name: obi
        image: ghcr.io/open-telemetry/opentelemetry-ebpf-instrumentation:latest
        securityContext:
          capabilities:
            add: ["SYS_ADMIN","BPF"]
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "otel-collector.observability.svc:4317"

PromQL examples (kernel-derived SLIs):

p95 = histogram_quantile(0.95, sum(rate(obi_request_duration_seconds_bucket[5m])) by (le))
err = sum(rate(obi_request_errors_total[5m])) / sum(rate(obi_request_total[5m]))
# burn = err / (SLO_error_budget / window_seconds)  (compute windows per policy)

A tiny Python decision snippet to gate scaling (illustrative):

# simplified: read Prometheus queries and compute burn-rate
p95 = query(prom_url, "histogram_quantile(0.95, sum(rate(obi_request_duration_seconds_bucket[5m])) by (le))")
err = query(prom_url, "sum(rate(obi_request_errors_total[5m])) / sum(rate(obi_request_total[5m]))")
# assume slo_error_budget = 0.01 (1% error allowed) over a window
burn_rate = err / slo_error_budget
if burn_rate > 6 and p95 > latency_target_ms:
    scale_out()
elif burn_rate < 1:
    scale_down()

Uncertainty-aware gating

Treat eBPF signals like any measurement: they have sampling windows, partial coverage, and sometimes conflicting app-level semantics. An uncertainty-aware autoscaler does three things:

• Use multi-window burn-rate checks (e.g., 5m/1h/6h) to avoid transient reactions.
• Weight decisions by sample volume / request count so low-traffic noise doesn’t drive scale actions.
• Reconcile kernel and SDK signals: if both indicate burn, act fast; if only kernel indicates burn and SDK is silent, escalate carefully (automated canary-sized scale is useful).

This approach prevents overreaction to low-volume statistical anomalies while giving priority to consistent kernel-observed user impact.

Trade-offs and operational concerns

• Operational complexity: eBPF tooling, kernel compatibility matrix, privileged DaemonSet lifecycle.
• Signal reconciliation: some business attributes and internal errors remain visible only via SDKs.
• Observability cost: more metrics and traces can increase ingestion — use OTel Collector tail-sampling and low-cardinality OBI configs.
• Security posture: privileged DaemonSets require strong runtime controls and RBAC.

None of these are deal-breakers: they’re engineering costs you pay to get a defensible, user-impact-driven control loop.

When to flip the switch

This pattern shines for high-throughput, multi-tenant clusters, or user-critical services (payments, authentication, inference). If you need faster detection of tail issues and want to reduce noise-driven scale churn, start with a pilot: one namespace, OBI DaemonSet, Prometheus recording rules, and an SLO-gated controller configured in observation-only mode for 2–4 weeks.

Conclusion

Treat the kernel as the canonical sensor for user-facing network and protocol behavior. Use OBI to auto-collect RED SLIs, merge with OpenTelemetry where needed, and gate autoscaling on uncertainty-aware SLO burn-rate. You’ll trade operational complexity for faster, cheaper, and more accurate scaling decisions — and that’s a defensible trade for teams serious about SLO-driven operations.

Who else is running eBPF as the source of truth for autoscaling in prod? What unexpected gotchas did you hit?