Stop Guessing Production Latency: Practical bpftrace One-Liners for Linux Debugging

Production boxes rarely fail politely. CPU is "fine," disk is "fine," and the app is still slow. When that happens, you do not need another dashboard panel—you need answers from the kernel, on the live host, with low overhead.

That is where bpftrace shines.

bpftrace is a high-level tracing language for Linux eBPF. It compiles short scripts into BPF programs, attaches them to kernel/user events, and summarizes results safely. Think of it as the middle path between strace (too heavy for production) and writing custom BCC tools in C/Python (too slow when the pager is already loud).

This guide is a practical field kit: install, verify, run safe one-liners, use the packaged tools Debian already ships, and know when not to attach a probe.

What bpftrace is (and is not)

bpftrace is great for:

• Ad-hoc production questions ("which process is opening /etc/passwd?")
• Latency histograms (run-queue wait, I/O sizes, syscall durations)
• Short-lived process visibility (fork/exec storms)
• Kernel-level events without restarting the app

bpftrace is not:

• A full APM product
• A replacement for continuous metrics (Prometheus still matters)
• A free pass to attach unstable kprobes to every kernel function forever

Prefer tracepoints when they exist. They are a more stable ABI than raw kprobes. Use kprobes only when you need a function that has no suitable tracepoint—and treat those scripts as kernel-version sensitive.

Prerequisites

You need:

• Root (or equivalent capability) to attach most probes
• A recent Linux kernel with BPF support
• BTF is a big quality-of-life win (/sys/kernel/btf/vmlinux)

On Debian 13 (Trixie):

sudo apt-get update
sudo apt-get install -y bpftrace
bpftrace -V
# bpftrace v0.23.2 (package version may vary by distro)

Quick health check:

# Feature matrix for this kernel + bpftrace build
sudo bpftrace --info | sed -n '1,80p'

# Confirm BTF is present (recommended)
ls -l /sys/kernel/btf/vmlinux

# Hello world
sudo bpftrace -e 'BEGIN { printf("bpftrace is alive\n"); exit(); }'

If bpftrace --info shows btf: yes and tracepoint: yes, you are in good shape.

Mental model in 30 seconds

A bpftrace program is one or more action blocks:

probe[,probe2]
/optional_filter/ {
  action;
}

Important pieces:

• Probe: when to run (tracepoint:…, kprobe:…, profile:hz:99, interval:s:5, BEGIN/END)
• Filter/predicate: only run the action when true (/pid == 1234/)
• Maps (@name): in-kernel aggregation (count(), hist(), sum(), …)
• Builtins: pid, tid, comm, nsecs, kstack, ustack, …

Maps print automatically when the program exits (Ctrl-C or exit()).

Safety rules before you attach anything

1. Start narrow. Filter by PID, cgroup, or process name before tracing everything.
2. Prefer summaries over printf floods. Histograms and counts are cheaper than per-event logs.
3. Time-box noisy probes. Use interval:s:N { exit(); } for short captures.
4. Prefer tracepoints over kprobes.
5. Do not leave experimental scripts attached overnight unless you measured overhead.
6. Avoid --unsafe unless you fully understand the side effects (system() etc.).

Recipe 1: "Who is opening files right now?"

Classic missing-config / permission / noisy-scanner question.

sudo bpftrace -e '
BEGIN {
  printf("Tracing openat... Ctrl-C to stop\n");
  printf("%-8s %-16s %s\n", "PID", "COMM", "PATH");
}
tracepoint:syscalls:sys_enter_openat
{
  printf("%-8d %-16s %s\n", pid, comm, str(args.filename));
}'

Inspect probe arguments first if you forget field names:

sudo bpftrace -vl 'tracepoint:syscalls:sys_enter_openat'

On modern glibc, userspace open() usually becomes openat(2), so this single probe covers most open activity.

For less spam and more structure, use the packaged tool:

sudo opensnoop.bt

On Debian, many tools land in /usr/sbin/*.bt (for example opensnoop.bt, execsnoop.bt, runqlat.bt, biolatency.bt).

Recipe 2: Short-lived process storms

If load spikes and ps never catches the culprit, trace exec:

sudo bpftrace -e '
BEGIN {
  printf("%-10s %-7s %-7s %s\n", "TIME", "PID", "PPID", "COMM");
}
tracepoint:syscalls:sys_enter_execve,
tracepoint:syscalls:sys_enter_execveat
{
  printf("%-10s %-7d %-7d %s\n", strftime("%H:%M:%S", nsecs), pid, curtask->real_parent->pid, comm);
}'

Or just run:

sudo execsnoop.bt

execsnoop is one of the highest-ROI tools in production: cron loops, shell wrappers, failing health checks, and deploy scripts show up immediately.

Recipe 3: Syscall hot spots by process

When CPU is elevated but userspace profilers look boring, check syscall pressure:

sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter
{
  @[comm] = count();
}
interval:s:10
{
  exit();
}'

After 10 seconds you get a frequency table of which process names entered the kernel most often. That often points at chatty logging, over-polling, or "tiny reads in a tight loop" patterns.

Filter to one service once you have a suspect:

sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter
/comm == "myapp"/
{
  @[probe] = count();
}
interval:s:10 { exit(); }'

Recipe 4: Read-size histogram for one PID

"Is this app doing thousands of tiny reads?"

# Replace 1234 with the target PID
sudo bpftrace -e '
tracepoint:syscalls:sys_exit_read
/pid == 1234 && args.ret > 0/
{
  @bytes = hist(args.ret);
}
interval:s:15 { exit(); }'

hist() builds a power-of-two histogram in-kernel. You get distribution shape without shipping every event to userspace.

Recipe 5: Latency of a kernel path (timed entry/exit)

To time a function, store a start timestamp keyed by thread ID, then histogram the delta on return:

sudo bpftrace -e '
kprobe:vfs_read
{
  @start[tid] = nsecs;
}
kretprobe:vfs_read
/@start[tid]/
{
  @ns[comm] = hist(nsecs - @start[tid]);
  delete(@start, tid);
}
interval:s:20 { exit(); }'

Notes that matter in production:

• The /@start[tid]/ filter avoids bogus durations when you attach mid-call.
• delete(@start, tid) prevents map growth.
• This uses kprobes, so it can break across kernel versions if the symbol changes. Prefer a tracepoint-based approach when available.

Recipe 6: CPU scheduler run-queue latency

If the app is "slow" while CPU looks free, you may be waiting on the run queue (noisy neighbor, too many threads, cgroup CPU throttling side effects).

Debian ships a ready tool:

sudo runqlat.bt
# Ctrl-C after 30-60s and read the histogram

The core idea: timestamp wakeups, then measure delay until the task is switched in. Large tails in the histogram are your smoking gun.

Recipe 7: Block I/O request sizes

Before you blame "the disk," check what sizes you are actually issuing:

sudo bpftrace -e '
tracepoint:block:block_rq_issue
{
  @bytes = hist(args.bytes);
}
interval:s:20 { exit(); }'

Or use:

sudo bitesize.bt
sudo biolatency.bt

Interpretation tips:

• Lots of 4K requests can mean unaligned/random I/O or chatty fsync patterns.
• Wide latency tails with healthy sizes often point at contention, queueing, or the storage path—not just "needs a bigger instance."

Recipe 8: On-CPU kernel stacks (quick profile)

When you need a cheap kernel-side sample without building a full perf flame-graph pipeline:

sudo bpftrace -e '
profile:hz:99
{
  @[kstack] = count();
}
interval:s:30 { exit(); }'

Why 99 Hz instead of 100? To reduce lockstep sampling with other 100 Hz timers. This is complementary to perf record + flame graphs: bpftrace is often faster to attach for a one-off answer; perf is still excellent for richer offline analysis.

Turn one-liners into reusable scripts

Once a one-liner works twice, save it.

/usr/local/sbin/openat-by-comm.bt:

#!/usr/bin/env bpftrace
BEGIN
{
  printf("Counting openat by process for 30s...\n");
}

tracepoint:syscalls:sys_enter_openat
{
  @opens[comm] = count();
}

interval:s:30
{
  exit();
}

sudo chmod +x /usr/local/sbin/openat-by-comm.bt
sudo /usr/local/sbin/openat-by-comm.bt

Add a comment header with:

• kernel versions tested
• expected overhead class (low/med)
• whether it uses kprobes (unstable) or tracepoints (preferred)

Production workflow that actually works

When paged, use this order:

1. Establish the symptom class
   • CPU bound? I/O bound? latency while idle?
2. Catch short-lived work
   • execsnoop.bt
3. Identify chatty syscalls / file access
   • syscall counts by comm, opensnoop.bt
4. Quantify latency distributions
   • runqlat.bt, read/IO histograms, biolatency.bt
5. Only then go deep
   • kprobe timing, stack histograms, targeted perf flame graphs

This sequence avoids the classic failure mode: attaching the noisiest probe first and drowning in events.

Common pitfalls

• Printing every event on a busy host can become the incident.
• Forgetting filters (/pid == .../, /comm == "..."/) makes output useless.
• Map leaks from missing delete() on timed entry/exit scripts.
• Misreading probe context: some block I/O completions run in kernel context, so comm may not be the app you expect.
• Assuming kprobe scripts are portable across kernels and distros.
• Confusing bpftrace with continuous monitoring: keep these as diagnostic scalpels.

How this differs from perf and "generic eBPF posts"

• perf is outstanding for sampling profiles and flame graphs.
• bpftrace is outstanding for questions: counts, histograms, and event predicates you can write in one minute.
• You do not need to author C BPF programs to get production signal.

Use both. Start with bpftrace when you need a fast answer; switch to perf when you need deep multi-minute profiles or richer offline workflows.

References

• bpftrace one-liner tutorial (Brendan Gregg / bpftrace project): https://bpftrace.org/tutorial-one-liners
• bpftrace project documentation and language reference: https://bpftrace.org/docs
• Debian man page for bpftrace: https://manpages.debian.org/trixie/bpftrace/bpftrace.8.en.html
• Brendan Gregg, "Learn eBPF Tracing: Tutorial and Examples": https://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
• bpftrace packaged tools (source examples): https://github.com/bpftrace/bpftrace/tree/master/tools
• Linux openat(2) man page: https://manpages.debian.org/trixie/manpages-dev/openat.2.en.html

Wrap-up

You do not need a custom observability platform to answer most Linux production mysteries. With bpftrace installed, you can:

• watch file opens and exec storms live
• quantify syscall and scheduler behavior with histograms
• inspect block I/O sizes and latency tails
• keep overhead under control with filters and short timed captures

Install it on your admin image before the next incident. The worst time to learn probe syntax is while customers are already refreshing the status page.