When it comes to Linux observability and security, traditional tools like top, ps, and standard audit daemons have a fundamental flaw: they lie. Or more accurately, they only tell you what the system wants you to see. Advanced rootkits, fleeting container processes, and malicious scripts can easily evade user-space monitoring.
The solution? We have to go deeper. We have to go to the Kernel itself.
In this post, I'll walk you through why I built Procscope—a blazing-fast, process-scoped runtime investigator written in Go and powered by the absolute superpower of modern Linux: eBPF.
The Problem with User-Space Observability
Standard monitoring tools periodically poll the /proc filesystem to see what's running. This introduces two massive problems:
Performance Overhead: Polling wastes CPU cycles.
Blind Spots: If a malicious script starts, executes its payload, and dies between your polling intervals, you will never even know it happened.
To solve this, modern security agents use eBPF (Extended Berkeley Packet Filter).
Enter eBPF: The Kernel's Superpower
eBPF allows us to run sandboxed programs directly inside the Linux kernel without changing kernel source code or loading unstable kernel modules. It is event-driven. Instead of asking the kernel "what's happening?" every second, we attach scripts to specific kernel events (like a file being opened or a network connection being made). When the event happens, our script fires instantly.
It is the technology powering the next generation of cloud-native tools like Cilium, Tracee, Falco, and now, Procscope.
How Procscope Works
I wanted to build an open-source tool that DevOps and Platform Engineers could drop onto a Linux box to instantly trace application behavior with near-zero overhead.
Procscope intercepts syscalls at the lowest possible level and bridges them up to a beautifully structured Go user-space application. Here is the architecture:
eBPF C Code: Tiny, highly optimized C programs attach to tracepoints (like sys_enter_execve).
Ring Buffers: When an event fires, the C program pushes the kernel data (Process ID, Comm name, arguments) into a high-performance eBPF Ring Buffer.
Go User-Space: The main application, written in Golang using the excellent Cilium eBPF library, consumes these ring buffers in real-time and formats them for the user.
A Look at the Code
Writing eBPF requires bridging two worlds. Here is a simplified look at how Procscope catches a process execution.
First, the eBPF Kernel probe (written in C):
c
SEC("tracepoint/syscalls/sys_enter_execve")
int tracepoint_syscalls_sys_enter_execve(struct trace_event_raw_sys_enter* ctx) {
struct event_t event = {};
// Get the Process ID and Thread ID
u64 id = bpf_get_current_pid_tgid();
event.pid = id >> 32;
// Get the process command name (e.g. "bash", "curl")
bpf_get_current_comm(&event.comm, sizeof(event.comm));
// Submit the event to our Go application
bpf_ringbuf_submit(&events, &event, 0);
return 0;
}
Then, the Go application seamlessly consumes it:
go
// Start reading from the eBPF ring buffer
rd, err := ringbuf.NewReader(bpfObjs.Events)
if err != nil {
log.Fatalf("failed to create ringbuf reader: %v", err)
}
for {
record, err := rd.Read()
if err != nil {
log.Printf("ringbuf read error: %v", err)
continue
}
// Unmarshal the byte array directly into our Go struct!
var event bpfEvent
binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event)
fmt.Printf("🚨 New Process Detected | PID: %d | Comm: %s\n", event.Pid, string(event.Comm[:]))
}
Because we use Go's generate tools, compiling this application automatically compiles the C code into eBPF bytecode and embeds it directly into the final Go binary. The result is a single, static binary that you can drop onto any modern Linux server. No dependencies needed.
Why Build This?
While heavyweight enterprise agents exist, I wanted to create an open-source, lightweight alternative that is easy to compile, easy to read, and laser-focused on process scoping. Right now, it's perfect for incident response, malware analysis on a honeypot, or simply figuring out why a rogue container keeps crashing.
What's Next (and How You Can Help)
The project is entirely open-source, and I'd love to invite the community to help shape it! We are currently working on adding Kubernetes Pod Context Resolution so that intercepted PIDs can be automatically mapped to their respective Pods and Namespaces.
I've opened up several Good First Issues specifically designed for developers wanting to dip their toes into Go and eBPF observability.
If this sounds interesting to you: 🌟 Check out Procscope on GitHub and drop a Star! 🌟
Have you worked with eBPF before? Let me know your thoughts or questions about kernel tracing in the comments below!
Top comments (0)