What’s Happening Inside Your Linux Kernel?

Have you ever wondered how exactly your Linux system knows when a program is running, a file system is being mounted, or a new module is being loaded? It’s all happening deep inside the Linux kernel, and most of the time, we’re completely blind to it.

But what if I told you there’s a way to peek inside, without rebooting the system, installing special software, or breaking anything?

Welcome to the world of "kprobes", where you can trace important kernel events in real-time, like a system detective 🕵️‍♂️.

Let’s dive in.

Why Should You Care? 🤔

Let’s say you’re running containers in production. One day, something feels off—a container might be doing something it shouldn’t.

• Is it spawning weird processes?
• Is it trying to mount a new filesystem?
• Is it trying to gain extra privileges?
• Is someone loading a sketchy kernel module?

These are early warning signs of something going wrong—maybe a misconfigured app, maybe an attack. Kprobes let you catch these signs early.

What kernel functions we can trace?

Linux is full of functions, but we’ll focus on four powerful ones that reveal a lot about what’s going on:

1. do_execve: Process Launch Tracker 🚀

Whenever you run a program like ls, python, or a shell script, the Linux kernel calls a function named do_execve. This function is the gateway to launching any new executable in the system. Why is this important? Because if you're monitoring for suspicious activity, like an unexpected script suddenly running on your server:do_execve is your best friend. It's essentially a signal that something new is being executed. Think of it like someone opening a door and entering a new room in a secure building. Naturally, you’d want to know who just walked in and whether they belong there.

2. security_capable: Permission Checkpoint 🛡️

The security_capable function is called whenever a process tries to perform an action that requires special privileges like changing network configurations, modifying system time, or accessing restricted resources. It's a key part of Linux's internal permission checking system. In essence, this function asks: Does this process have the right capabilities? Monitoring this can reveal when processes are trying to act like an admin or escalate privileges. Imagine someone attempting to unlock the admin panel in a web app. Wouldn’t you want to know who they are and whether they should be allowed?

3. security_sb_mount: Filesystem Mount Auditor 🗂️

Every time a filesystem is mounted whether it’s an external drive, a virtual filesystem, or a container volume, the security_sb_mount function is called. Mount operations are usually routine, but they can also be exploited to access unauthorized data or escape from container environments. From a security perspective, this function lets you keep tabs on what’s being attached to your system and by whom. Think of it like plugging a USB stick into a laptop. What’s on it? Is it safe? Who’s doing it? eBPF can help you find out.

4. load_module: Kernel Code Gatekeeper 🧩

Linux is modular, and the kernel allows dynamic loading of new code modules, like drivers or extensions at runtime. This is managed by the load_module function. While many modules are harmless or necessary, some can introduce vulnerabilities or even backdoors. By tracing load_module, you can detect whenever new kernel code is being added. It’s like someone installing an app on your phone. Is it coming from a trusted source? Should it be there at all? Keeping an eye on this function gives you insight into what’s being added to the core of your OS.

How Do We Trace These Functions?

This is where kprobes come in. Think of kprobes as little spy cameras inside the kernel. You can place them at any function and they’ll quietly record what’s happening.

Here’s how the flow works:

1. You choose a kernel function, like do_execve.
2. You set a kprobe on it, which says: “Whenever this function is called, tell me.”
3. The system logs each call, including key details like PID, UID, and arguments.
4. You read the trace from a special file: /sys/kernel/debug/tracing/trace_pipe.

💬 Final Thoughts

Think of kernel tracing like having X-ray vision into your Linux system. You don’t need to guess what your containers or processes are doing, you will be able to see it.

Kprobes give you power, visibility, and control especially when something feels off but you don’t know where to start looking.

Ready to try it out? Start small, trace one function, and see what insights pop up. The Linux kernel is full of secrets it’s time to uncover them.

FAQs

1. What is do_execve, and why trace it?

The do_execve function is the kernel’s internal handler for the execve system call, responsible for loading a new program into a process’s address space and starting its execution. Tracing do_execve reveals exactly when and how processes are launched, making it invaluable for detecting unauthorized or unexpected binaries running on a system.

2. What does security_capable check?

The security_capable function is part of the Linux Security Modules (LSM) framework and is invoked whenever a process requests a privileged capability (e.g., CAP_SYS_ADMIN). Tracing this hook shows when processes attempt actions that require elevated privileges, helping to catch privilege escalations or policy violations.

3. What is security_sb_mount, and why is it important?

security_sb_mount is the LSM hook called whenever a filesystem is mounted (including container volumes). Monitoring this function can detect unauthorized mounts—which may indicate container escapes or illicit data access—by reporting details such as device path and filesystem type.

4. How do kprobes attach hooks to kernel functions?

Kprobes dynamically insert breakpoints into almost any kernel routine. You define a probe by writing a line like:

echo 'p:myprobe do_execve' > /sys/kernel/debug/tracing/kprobe_events

This tells the kernel to invoke your handler whenever do_execve runs. The recorded data is then read from /sys/kernel/debug/tracing/trace_pipe for live monitoring.