How I Built a Syscall Tracer That Runs at <1% Overhead (Using eBPF)

strace is the first tool most people reach for when a Linux process is misbehaving. Attach it to a PID, watch the syscalls scroll by, find the problem. It works.

Until you try to use it in production. strace uses ptrace() to intercept every syscall, which context-switches the target process twice per syscall — once to stop it, once to resume it. The measured overhead is roughly 40%. On a busy web server doing thousands of syscalls per second, that's not debugging. That's creating a new problem.

I wanted something that could answer the same questions — what syscalls are happening, how long they take, which ones are failing — but designed to run continuously, system-wide, without measurable impact.

So I built KernelLens, an eBPF-based syscall tracer. This article walks through the key technical challenges I hit and how I solved them.

The core idea: tracepoints, not ptrace

Linux has static instrumentation points called tracepoints baked into the kernel source. Two of them are interesting:

• raw_syscalls:sys_enter — fires before every syscall executes
• raw_syscalls:sys_exit — fires after every syscall returns

Unlike ptrace, tracepoints don't stop the target process. The kernel runs your handler inline, in the same context, and continues. There's no context switch, no signal, no scheduling delay.

eBPF lets you attach small programs to these tracepoints. The programs run inside the kernel, in a sandboxed VM with a verifier that guarantees they can't crash the system, loop forever, or access invalid memory.

Here's the stripped-down sys_enter handler:

SEC("tracepoint/raw_syscalls/sys_enter")
int handle_sys_enter(struct trace_event_raw_sys_enter *ctx)
{
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    __s32 sc_id = ctx->id;

    struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e) return 0;

    e->pid = pid;
    e->syscall_id = sc_id;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    bpf_ringbuf_submit(e, 0);
    return 0;
}

This runs inside the kernel. User space reads events from the ring buffer using Go and cilium/ebpf. That's the entire data path.

Challenge 1: The BPF verifier vs. context registers

This was the first real "what is happening" moment.

The BPF verifier tracks the register that holds the ctx pointer. After any branch, map lookup, or helper call, the verifier may consider that register modified. If you try to read ctx->args[0] after a bpf_map_lookup_elem() call, the verifier rejects your program with a cryptic error about invalid pointer access.

The fix is to read all ctx fields into local variables at the very top of the function, before doing anything else:

// Read ALL ctx fields FIRST, before any branches or helper calls.
__u32 pid = bpf_get_current_pid_tgid() >> 32;
__s32 sc_id = ctx->id;
__u64 arg0 = ctx->args[0];
__u64 arg1 = ctx->args[1];
__u64 arg2 = ctx->args[2];

// Now it's safe to do map lookups, branches, etc.
// Use arg0/arg1/arg2 instead of ctx->args from here on.

This isn't documented anywhere obvious. I found it by reading verifier source and other eBPF projects that had the same problem.

Challenge 2: Filtering without wasting CPU

If you're monitoring a specific process, you don't want the overhead of processing every syscall on the system. The naive approach is to filter in user space — read every event, check the PID, discard the ones you don't care about. This still burns CPU on the ring buffer read and decode.

The better approach is kernel-side filtering. I use BPF maps as configuration:

// PID filter: user space writes the target PID into this map before attaching
struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u32);
} config_pid SEC(".maps");

// In the handler:
__u32 *target_pid = bpf_map_lookup_elem(&config_pid, &key);
if (target_pid && *target_pid != 0 && pid != *target_pid)
    return 0;  // filtered out — never touches the ring buffer

Same pattern for syscall allowlists (hash map) and event sampling (counter % N). Filtered events return immediately from the kernel handler. They never touch the ring buffer, never wake up user space, never consume any resources beyond the map lookup.

Challenge 3: Measuring syscall latency without ptrace

This requires hooking both sys_enter and sys_exit. On enter, store a timestamp. On exit, compute the difference.

The tricky part is the key. You need to correlate the enter and exit for the same syscall on the same thread. bpf_get_current_pid_tgid() returns a 64-bit value: upper 32 bits are the process ID, lower 32 bits are the thread ID. Using the full 64-bit value as the key prevents threads within the same process from overwriting each other's timestamps.

// Hash map: pid_tgid -> entry timestamp
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, __u64);
    __type(value, __u64);
} entry_timestamps SEC(".maps");

// On sys_enter:
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u64 ts = bpf_ktime_get_ns();
bpf_map_update_elem(&entry_timestamps, &pid_tgid, &ts, BPF_ANY);

// On sys_exit:
__u64 *entry_ts = bpf_map_lookup_elem(&entry_timestamps, &pid_tgid);
if (!entry_ts) return 0;
__u64 duration = bpf_ktime_get_ns() - *entry_ts;
bpf_map_delete_elem(&entry_timestamps, &pid_tgid);

The bpf_map_delete_elem on exit is important. Without it, short-lived processes (think execve followed by exit) leak entries in the hash map. With max_entries=10240, this is bounded but still wastes memory.

bpf_ktime_get_ns() returns monotonic nanosecond time. The precision is real — I've measured syscalls completing in under 1 microsecond. strace can't even come close to this resolution because the ptrace overhead dwarfs the actual syscall duration.

Challenge 4: Knowing when you're losing data

Ring buffers have a fixed size (1 MB in my case). Under heavy syscall load, the buffer fills up and bpf_ringbuf_reserve() returns NULL. When this happens, you've silently lost an event.

Silent data loss is unacceptable in a monitoring tool. The fix is a per-CPU drop counter:

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u64);
} drop_counter SEC(".maps");

struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
if (!e) {
    __u64 *drop_cnt = bpf_map_lookup_elem(&drop_counter, &zero);
    if (drop_cnt)
        (*drop_cnt)++;  // per-CPU, no atomics needed
    return 0;
}

The key detail is BPF_MAP_TYPE_PERCPU_ARRAY. A regular array map would require atomic increments since multiple CPUs hit this path concurrently. Per-CPU maps give each CPU its own counter. User space reads all CPUs and sums them. No locks, no contention, no overhead on the hot path.

On my test system under heavy load, the drop rate hit 29%. The fix for that was kernel-side event sampling — emit every Nth event using a per-CPU counter with modulo check. This runs after filters but before the ring buffer reserve, so you're only sampling the events you actually care about.

Challenge 5: Reading user-space strings from the kernel

Syscalls like openat() take a filename as a pointer argument. From the kernel handler, ctx->args[1] contains a user-space pointer. You can't dereference it directly — it's in a different address space.

bpf_probe_read_user_str() copies a null-terminated string from user-space into your buffer. But the verifier requires that you prove the argument index is bounded. A simple if (idx < 3) isn't enough — the verifier needs a switch statement to track each case:

switch (*str_idx) {
case 0: str_ptr = (void *)arg0; break;
case 1: str_ptr = (void *)arg1; break;
case 2: str_ptr = (void *)arg2; break;
}
if (str_ptr)
    bpf_probe_read_user_str(e->str_arg, sizeof(e->str_arg), str_ptr);

Which syscall arguments to capture is configured via a BPF hash map, populated by user space at startup. For example, openat's filename is arg index 1, so user space writes {257: 1} into the map (257 is the openat syscall number on x86_64).

The user-space side

The Go binary uses cilium/ebpf with bpf2go for code generation. go generate compiles the C code into eBPF bytecode and generates Go structs that match the BPF maps and event struct. The bytecode is embedded in the binary — the final artifact is a single file with no runtime dependencies.

The event loop is straightforward: read from the ring buffer, decode the binary event, resolve the syscall number to a name (parsed from the system's kernel headers at startup), run it through anomaly detection, format, and print.

Anomaly detection works by observing all syscalls during a configurable baseline window. After the window, any syscall that wasn't seen during the baseline triggers an alert. It's simple but effective — a web server suddenly calling ptrace is worth knowing about.

What I learned

• eBPF's power comes with a steep learning curve, mostly due to the verifier. The verifier is right — you just have to understand what it's checking.
• Per-CPU maps are the answer to "how do I count things in BPF without locks." Use them for any counter on the hot path.
• Kernel-side filtering is not optional for a production tracing tool. If you're filtering in user space, you're wasting CPU proportional to total system syscall rate, not your target's syscall rate.
• Ring buffer drops are inevitable under load. Make them visible, not silent.
• The gap between a working prototype and a production tool is about 10x the code. The 79-line version worked. Making it reliable, observable, and deployable took 2000+ lines.

Try it

brew tap ojas-2003/kernellens && brew install kernellens
sudo kernellens --latency --slow 10ms

Source: github.com/ojas-2003/kernellens
Linkedin: https://www.linkedin.com/in/ojasgupta2003/