Hi everyone, I want to show you how eBPF can help us understand what is happening inside the Linux kernel. In practice, it is one of the most effective ways to build observability into a system, because it lets us attach small programs to kernel events and turn low-level behavior into useful signals.
Observability is the ability to understand the internal state of a system by looking at the signals it emits. It matters because modern systems are too complex to reason about only from their external behavior. In Linux, observability means collecting traces, logs, metrics, and other events so we can answer questions such as: what happened, why did it happen, and what is happening now?
That is where eBPF fits so well. It gives us a safe and practical way to observe kernel behavior directly, at the point where many of those events actually happen. If you are new to this topic, think of eBPF as a way to place tiny programs at important points in the Linux kernel and inspect what happens there. The examples in this post are practical, and the full repository is available here:
Note: The step-by-step instructions for running the examples are also included in the README files inside the repository folders.
What you will learn
- What eBPF is
- How system calls are traced
- How eBPF programs are loaded
- Python and Go examples
- Containers, Falco, and Tetragon
- Why eBPF matters for observability
Environment and setup
For these examples, we will assume a Linux machine with root access, kernel headers, and the required eBPF toolchain. The goal is to keep things practical and runnable. If you are coming from Python, BCC is a friendly place to start because it lets you write and load eBPF programs from Python without needing to dive into the lower-level tooling first.
In this repository, the examples are organized around three main areas:
- Python examples with BCC.
- Go examples with Cilium eBPF.
- Runtime examples with Falco and Tetragon.
The repository already includes the necessary structure for experimenting locally in a Linux environment.
Before running the examples, it is important to prepare the host properly. The repository includes dependency instructions for both Ubuntu and Alpine, so you can choose the distribution that fits your environment.
Ubuntu
Install the required packages for BCC, Python bindings, and LLVM support. LLVM is the toolchain that helps turn your eBPF code into the low-level bytecode the kernel can verify and run, so it plays an important role in the workflow even if you do not write it directly:
sudo apt-get update
sudo apt-get install -y bpfcc-tools linux-headers-$(uname -r) python3-bpfcc
sudo apt-get install -y llvm clang
sudo apt-get install -y libbpf-dev libbpf
Alpine
Install the packages required for BCC, Python, and eBPF development:
apk update
apk add build-base linux-headers git unzip nano vim elfutils-dev
apk add bcc-tools python3 py3-pip py3-bcc
apk add linux-virt-dev
apk add llvm clang
apk add libbpf-dev libbpf
If you need Go support on Alpine, also install:
apk add go
You can verify that BCC is available with:
sudo python3 -c "from bcc import BPF; print('BCC installed correctly')"
What eBPF is
eBPF stands for Extended Berkeley Packet Filter. Originally designed for packet filtering, it evolved into one of the most important technologies in modern Linux systems.
At a high level, eBPF allows you to run sandboxed programs inside the Linux kernel. The developer writes code in a restricted language, the toolchain compiles it into eBPF bytecode, the kernel verifies that bytecode, and only then is it executed by the eBPF virtual machine when the relevant hook is triggered. That is why eBPF is often described as a safe and programmable extension point inside the kernel.
The flow is roughly:
Source code
│
▼
eBPF bytecode
│
▼
Kernel verification
│
▼
Attach to hook
│
▼
System call / kernel event
│
▼
Linux kernel
│
▼
eBPF virtual machine executes it
│
├─ inspect context
├─ read data from maps
└─ emit trace / alert / metric
The important detail is that eBPF is not just code that gets attached. It is a verified, sandboxed program that the kernel runs only when a relevant event happens.
The following hooks are commonly used:
- kprobes, which attach to kernel functions and are one of the simplest ways to observe them.
- tracepoints, which attach to stable instrumentation points.
- networking hooks, which are used to inspect traffic-related events.
- syscall entry and exit points, which let you observe how user-space programs interact with the kernel.
The hook you attach is the specific kernel event you want to observe. In the repository examples, the hook is the execve syscall entry path, so the eBPF program runs whenever a new process is created. In other words, the hook is the place where the kernel says “this event happened,” and the eBPF program is attached there to inspect or react to it.
The main difference is safety: a kernel module can do almost anything, but it runs with full kernel privileges and can easily destabilize the system if it is not written carefully. An eBPF program is verified before it is accepted by the kernel, which makes it much safer for instrumentation and observability.
In other words, eBPF lets you add behavior to the kernel without replacing the kernel itself.
Tracing system calls
One of the most common uses of eBPF is tracing system calls. A system call is the boundary between user space and kernel space. When a process calls something like execve, open, read, or write, the kernel executes the corresponding handler.
The relationship is simple to picture:
User space program
│
▼
System call
│
▼
Linux kernel
│
▼
eBPF hook / kprobe
│
▼
eBPF program runs
│
├─ inspect context
├─ read data from maps
└─ emit trace / alert / metric
In practice, eBPF attaches to the flow of execution at the moment the kernel is handling a syscall or a kernel function call. That is why it can observe behavior so precisely: it runs exactly where the event happens.
With eBPF, we can attach a program to that moment and inspect what is happening. For example, we can observe when a process starts a new program, when a file is written, or when network traffic is emitted.
Note: If you want to experiment with other system calls in your own code, the Linux syscall reference is a good place to start: https://man7.org/linux/man-pages/man2/syscalls.2.html. That page lists many common syscalls such as open, read, write, connect, and bind, which you can swap into the same pattern used in the repository example.
The repository includes a Python example in code/python/exec.py that demonstrates this idea. If you run it, it attaches a probe to the execve path and prints a message every time a process starts a new program. That is a simple but powerful example of how eBPF turns a kernel event into something visible and useful.
from bcc import BPF
program = r"""
int hello(void *ctx) {
bpf_trace_printk("Hello World from eBPF!\n");
return 0;
}
"""
b = BPF(text=program)
syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")
print("Tracing execve()... Press Ctrl+C to stop.")
b.trace_print()
This example is simple, but it shows the core idea clearly: it compiles a small C-like program into eBPF bytecode, loads it into the kernel, and attaches it to the execve syscall path. In this case, the hook is the execve syscall entry point, which means the program runs whenever a new executable is launched. Every time that event occurs, the eBPF code runs in kernel context and emits a trace message.
A typical output looks like this:
Tracing execve()... Press Ctrl+C to stop.
Hello World from eBPF!
Hello World from eBPF!
Hello World from eBPF!
That is the essence of low-level observability: we can watch the kernel at the point where behavior crosses the boundary between user space and kernel space.
In this context, a trace is simply a record of an event as it happens inside the kernel. It can be as simple as a log line such as "Hello World from eBPF!" or as rich as a structured stream of syscalls, process names, file paths, and timing data. Tracing is a practical way to collect those records, and it is closely related to observability because it gives us the detailed evidence needed to understand what the system is doing. The important idea is that the trace turns an internal kernel event into something observable and useful to a developer or operator.
Loading eBPF programs into the kernel
The flow is usually the following:
- A developer writes a small program in C, usually targeting a specific hook.
- The toolchain compiles that program into eBPF bytecode.
- The kernel verifies the bytecode.
- The program is attached to a hook such as a kprobe or tracepoint.
- When the hook is triggered, the eBPF logic executes.
The important part is that the program is not running as a regular user-space process. It is executed by the kernel when the relevant event happens. That is why eBPF is so useful for security, tracing, and performance analysis.
This is one of the reasons why eBPF is often described as a way to add functionality to the kernel without writing a full kernel module: it is more dynamic, safer, and easier to load and unload.
A kprobe is one of the most common kinds of hook. It lets us attach an eBPF program to the entry point of a kernel function, almost like a small breakpoint for kernel code. In practical terms, this means we can observe when a function such as sys_execve is called, inspect the surrounding context, and react to it without modifying the kernel source itself. That is exactly what the examples in this repository use to trace process execution and other low-level events.
Python and Go examples
Python example
The Python examples in this repository show how easy it is to start with eBPF in a scripting-friendly environment. The BCC toolkit makes this approachable by allowing you to write small programs and attach them to kernel events from Python.
To run the examples, make sure the environment is prepared as described above and then execute a script from the repository folder. For example, to run the exec example:
cd code/python
sudo python3 exec.py
You can similarly run other examples such as:
sudo python3 chmod.py
sudo python3 delete_file.py
sudo python3 ping.py
sudo python3 write_file.py
Another example, code/python/ping.py, hooks a network function and filters traffic by process name:
from bcc import BPF
program = r"""
#include <uapi/linux/ptrace.h>
#include <linux/skbuff.h>
#define TASK_COMM_LEN 16
int kprobe____dev_queue_xmit(struct pt_regs *ctx, struct sk_buff *skb) {
char comm[TASK_COMM_LEN];
bpf_get_current_comm(&comm, sizeof(comm));
if (comm[0] != 'p' || comm[1] != 'i' || comm[2] != 'n' || comm[3] != 'g' || comm[4] != '\0') {
return 0;
}
u32 len = 0;
bpf_probe_read_kernel(&len, sizeof(len), &skb->len);
bpf_trace_printk("ping traffic pid=%d comm=%s len=%d\n",
bpf_get_current_pid_tgid() >> 32,
comm,
len);
return 0;
}
"""
BPF(text=program).trace_print()
This example shows how eBPF can observe network behavior at a very low level. The program runs in the kernel and inspects the skb structure for outgoing packets, but it only prints output for the ping process. In other words, you can use it to answer a very concrete question: which processes are generating network traffic, and what kind of packets are leaving the host? A sample output may look like this:
ping traffic pid=1234 comm=ping len=64
ping traffic pid=1234 comm=ping len=64
ping traffic pid=1234 comm=ping len=64
The same repository also includes other examples such as file permission monitoring, file deletion detection, and write tracking, so you can mix and match them depending on what you want to observe.
Go example
The Go example in this repository uses the Cilium eBPF libraries and gives a slightly more production-oriented experience. The C program in code/go/kprobe.c attaches to sys_execve and increments a counter stored in a BPF map:
To build and run it, use the Go example directory:
cd code/go
go generate
go build -o kprobe-example .
sudo ./kprobe-example
In another terminal, trigger some activity such as:
ls
whoami
You should then see the counter increase as the kernel function is invoked.
SEC("kprobe/sys_execve")
int kprobe_execve() {
u32 key = 0;
u64 initval = 1, *valp;
valp = bpf_map_lookup_elem(&kprobe_map, &key);
if (!valp) {
bpf_map_update_elem(&kprobe_map, &key, &initval, BPF_ANY);
return 0;
}
__sync_fetch_and_add(valp, 1);
return 0;
}
The Go loader in code/go/main.go loads the object, attaches the program to the kernel function, and reads the map periodically:
kp, err := link.Kprobe(fn, objs.KprobeExecve, nil)
if err != nil {
log.Fatalf("opening kprobe: %s", err)
}
This is a great example of how eBPF can be used from a modern systems language. Instead of writing a full kernel module, the program attaches to an existing kernel symbol and collects information in a high-performance kernel map that the user-space process can read. A typical output from the running example looks like this:
Waiting for events..
sys_execve called 1 times
sys_execve called 2 times
sys_execve called 3 times
The same pattern can be extended to other kernel events, which is exactly why this repository is useful: it gives you a starting point for building your own observability logic.
Containers, Falco, and Tetragon: eBPF in real runtime scenarios
The runtime examples in this repository show how this technology becomes important in container and security environments. Once you have seen the smaller examples, you can move to the higher-level tools and ask a broader question: how do I turn low-level kernel events into runtime detection and alerting for real workloads?
Falco
Falco is an open-source runtime security tool that uses kernel-level telemetry to detect suspicious activity in real time. It is often used to monitor syscalls, file access, and container behavior, and it can raise alerts when an application does something unexpected. The example in runtime/falco/README.md starts Falco with privileged access so it can observe the host and correlate events with container metadata.
A simple example is to read a sensitive file such as /etc/shadow from the host. Falco can alert on that behavior because it is watching events at a very low level.
The repository also includes concrete commands for both runtime examples. For Falco, you can start it with:
docker run --rm -it \
--name falco \
--privileged \
-v /sys/kernel/tracing:/sys/kernel/tracing:ro \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /proc:/host/proc:ro \
-v /etc:/host/etc:ro \
falcosecurity/falco:0.44.1
After it is running, you can simulate suspicious activity with:
sudo cat /etc/shadow
An expected Falco output (simplified) looks like this:
15:22:41 Warning Sensitive file opened for reading
15:22:41 file=/etc/shadow proc=cat user=root container=host
15:22:41 rule=Read sensitive file (trusted dirs)
The exact fields and wording may vary depending on the Falco ruleset and version in use.
Tetragon
Tetragon is a security and observability tool from the Cilium ecosystem that uses eBPF to inspect Linux events in a policy-driven way. It can observe processes, files, and network activity and help enforce runtime rules without requiring a traditional kernel module. The example in runtime/tetragon/README.md mounts a policy file and uses Tetragon to capture relevant events. This is a clear example of how eBPF supports security and observability without requiring a traditional kernel extension.
For Tetragon, the example starts the runtime with a policy file mounted into the container:
docker run -d --name tetragon --rm --pull always \
--pid=host --cgroupns=host --privileged \
-v ${PWD}/file_monitoring.yaml:/etc/tetragon/tetragon.tp.d/file_monitoring.yaml \
-v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
quay.io/cilium/tetragon:v1.7.0
You can then inspect the events it collects with:
docker exec -ti tetragon tetra getevents -o compact
An expected Tetragon output (compact view, simplified) may look like this:
process_exec: binary=/usr/bin/cat args="cat /etc/shadow" pid=12345
process_kprobe: policy=file_monitoring event=security_file_open file=/etc/shadow
The exact event names and fields can vary depending on your mounted policy and Tetragon version.
What these tools have in common is that they do not need to reimplement the kernel. They rely on the kernel’s own instrumentation points and use eBPF to turn those events into structured signals that developers and operators can react to.
Why eBPF matters for observability
eBPF is powerful because it gives us visibility into the kernel at the same level where the system actually behaves. It helps us answer questions like:
- Which processes are starting?
- Which files are being accessed?
- Which network traffic is leaving the host?
- Which syscalls are involved in a suspicious sequence?
- How can we understand a runtime issue without modifying application code first?
This is why eBPF is so attractive for observability, security, and runtime tracing. It bridges the gap between user-space applications and the underlying kernel behavior that often determines performance and correctness.
Next steps with Kubernetes
Once you are comfortable with the basics, the next step is to explore how eBPF is used inside Kubernetes. In modern clusters, it is a key technology for networking, observability, and runtime security.
Projects such as Cilium, Tetragon, Falco, and Inspektor Gadget build on eBPF to provide deeper visibility into containers and services, improve traffic handling, and detect suspicious behavior at the kernel level.
The main benefit is simple: you can understand what is happening in the cluster more precisely, without relying on heavier or more intrusive approaches.
Conclusion
eBPF has become one of the most practical ways to observe and influence what happens inside the Linux kernel. It allows you to attach small programs to kernel events, collect detailed information, and build observability and security workflows without creating traditional kernel modules.
Its appeal is also practical: it works from different languages such as Python and Go, it is relatively simple to start with, and it scales well to larger and more complex environments such as Kubernetes-based systems. In practice, eBPF is especially valuable for tracing, security, and observability, including network monitoring, where understanding kernel-level behavior is essential for both performance and protection.
Top comments (0)