Tainting the kernel

Introduction

In this post, I continue documenting my journey as I pivot into Linux kernel development. This week I spent time understanding how to instrument the kernel using tracepoints and kprobes. My goal was to observe process lifecycle events in a running kernel and eventually build a small out-of-tree module that hooks into these events.

I ran into a few unexpected issues, including missing symbols, build environment mismatches, and understanding how tracepoint definitions propagate through the kernel build system. This post walks through the concepts, what I tried, what worked, and what I learned.

What I Was Trying To Do

Access a specific tracepoint (like sched_process_exit) from an out-of-tree module.

Experiment with registering a probe that fires when processes exit.

Validate the behavior in a QEMU guest running a kernel built from mainline sources.

The bigger purpose being to learn kernel instrumentation, understanding how trace events are plumbed, exploring debugging tooling, etc.

Understanding How Tracepoints Work

Tracepoints are static instrumentation sites built directly into the kernel code.

They expose a small, stable ABI for things like scheduler events, block I/O, filesystem operations, etc.

They are declared with macros that generate a struct named _tracepoint.

Accessing a tracepoint from a loadable module requires that the tracepoint symbol exists in the kernel’s symbol table.

Tracepoint symbols do not always exist in Module.symvers for out-of-tree modules. In fact it seems that most don't as they're not exported.

Some tracepoints depend on config options.

If the kernel was not built with tracing support enabled (like CONFIG_TRACEPOINTS, CONFIG_SCHEDSTATS, or CONFIG_EVENT_TRACING), the symbols may not be exported.

While this was interesting to learn, it seems that more often than not these are not exported for external modules. This makes sense for safety, while it was edifying to go down this route ultimate it did not end up meeting my needs.

Kprobes as an Alternative

Kprobes allow attaching dynamic probes to almost any kernel function.

Unlike tracepoints, kprobes do not require kernel config flags or static declarations. In fact if you are building external modules you might run into compilation issues if certain flags are set for instrumentation.

They work even when the kernel was not built with full tracing options.

They can be registered from an out-of-tree module with minimal prerequisites.

This module does the following:

Defined a struct kprobe

Set the symbol name (for example, "do_exit" or "release_task")

Registered pre-handlers and post-handlers

Added pr_info logging for validation

Kprobes are incredibly flexible but less stable than tracepoints.

They depend on internal function names which can change across kernel versions.

The Build Environment Problem I Hit

I'm building the kernel on a host machine.

I'm loading the module into a QEMU VM running a kernel built from a checkout of the mainline source tree.

I found that most tracepoint symbols are not exported. Leaving my module to have issue in linking.

Running grep __tracepoint_sched_process_exit Module.symvers returned nothing, which was the clue.

This was were I decided to use kprobes, which avoid the symbol export issue entirely.

Results

The kprobe fired reliably during process teardown.

I print small debugging messages, found in dmesg.

So I learned:

Getting the tracepoint symbol exported in an out-of-tree module is not a thing that, while possible with kernel source modification, should be done. Perhaps this whole idea of an out-of-tree module that can instrument internals is doomed from the beginning. But hey, this is how we learn. I'd love to hear a more experienced perspective on this.

Exported symbols appear in Module.symvers. This is what something called modpost seems to use.

Key Lessons Learned

Tracepoints depend heavily on kernel config and build environment.

Kprobes are powerful when you just need visibility.

Further understanding the kernel build system (Module.symvers, CONFIG options, modules_prepare).

Instrumentation is a good entry point for learning kernel internals.

What’s Next

I think I might go back toward doing a basic character driver, syscall, or /proc creation. I'm not sure, but if you have a suggestion I'd love to hear it. Stay tuned.