eBPF on Linux — kprobe vs fentry: Hooking Internals & What Production Observability Misses

There are two common ways to attach an eBPF program to a kernel function: kprobe and fentry. Most tutorials treat them as interchangeable — pick one, attach, read your data. They are not interchangeable. They install into the kernel by different mechanisms, they hand you the function's arguments in different forms, and on a hot path they cost different amounts of CPU per call. Get the choice wrong on a function that fires a million times a second and you have added measurable overhead to every request on the box.

This post walks the difference at the source level. The companion video disassembles both attachment paths so you can see exactly where the hook lands in machine code; the code here is from SentinelEdge, an eBPF project of mine that hooks the kernel with kprobe across 13 attach points, which makes it a useful place to talk honestly about when I'd reach for fentry instead.

The short version

• kprobe attaches by patching the target instruction. Classically that is a breakpoint (int3 on x86) that traps into the kprobe machinery, runs your program, then resumes the displaced instruction. It works almost anywhere and needs nothing special from the kernel build. You receive a raw struct pt_regs * and dig the arguments out of registers yourself.
• fentry attaches to the function's __fentry__ site — the call slot the compiler already left at the top of every traceable function for ftrace — through a generated BPF trampoline. No exception, closer to the cost of a plain call. You receive the arguments already typed, resolved from BTF. It needs a modern kernel (≥ 5.5) built with BTF, and the function has to be ftrace-attachable.

Same observability. Different install, different per-call cost, different failure modes.

The kprobe way, in real code

Here is a live kprobe from SentinelEdge, hooking do_mmap to watch memory-mapping activity per process:

SEC("kprobe/do_mmap")
int trace_mmap_detailed(struct pt_regs *ctx) {
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    __u64 addr  = PT_REGS_PARM1(ctx);
    __u64 len   = PT_REGS_PARM2(ctx);
    __u32 prot  = PT_REGS_PARM3(ctx);
    __u32 flags = PT_REGS_PARM4(ctx);

    struct fs_event *event = bpf_ringbuf_reserve(&fs_events, sizeof(*event), 0);
    if (!event)
        return 0;

    event->timestamp = bpf_ktime_get_ns();
    event->pid       = pid;
    event->operation = 10;  // MMAP
    event->size      = len;
    event->mode      = prot;
    bpf_ringbuf_submit(event, 0);
    return 0;
}

Look at what the context gives you: struct pt_regs *ctx, the raw register file at the moment of the trap. There is no do_mmap prototype in sight. You recover the arguments with PT_REGS_PARM1..N, which expand to the architecture's calling-convention registers (rdi, rsi, rdx, rcx, … on x86-64). Nothing checks that PARM3 is really prot. If the kernel's do_mmap signature shifts between versions — and it has — your offsets keep compiling and keep returning a number, just the wrong one. The breakage is silent.

That rawness is also the strength. A kprobe does not care about the function's type information and, in its classic form, does not even need the function to be specially compiled. It patches bytes. That is why it attaches to almost anything and runs on kernels that predate the typed tracing infrastructure.

The fentry way

The same hook as an fentry program looks like this:

SEC("fentry/do_mmap")
int BPF_PROG(trace_mmap, struct file *file, unsigned long addr,
             unsigned long len, unsigned long prot /* … per the kernel prototype */) {
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    // addr, len, prot arrive already typed — no PT_REGS_PARMn
    ...
}

BPF_PROG is a libbpf macro that unpacks the trampoline's argument array into named, typed parameters that mirror the kernel's real prototype, with the types resolved from BTF. You stop reading registers by position and start reading arguments by name. If the signature you wrote does not match the kernel's, it fails at load time against BTF, instead of silently handing you a wrong register at run time. That single property — a mismatch becomes a load error rather than corrupt data — is most of why fentry is the safer default on any kernel new enough to offer it.

fexit is the same idea for function return: it fires on the way out and gives you the arguments and the return value in one program, which a kretprobe cannot do without pairing it to an entry probe and stashing state.

Where the cost comes from

The overhead difference is not a tuning constant; it falls out of the two mechanisms.

A classic kprobe fires through an exception. The CPU hits the patched breakpoint, traps, the kernel walks its int3 path into the kprobe handler, runs your program, single-steps or emulates the instruction it displaced, and returns. An exception round trip is one of the more expensive things you can put on a hot path.

An fentry program fires through a call. The trampoline saves the registers it needs, calls your program, restores, and continues into the real function. There is no trap, no single-step. It is close to the cost of an ordinary indirect call.

One honest nuance the tutorials skip: on a modern kernel, a kprobe placed at a function's entry can be promoted onto the same ftrace call site fentry uses (KPROBES_ON_FTRACE) or jump-optimized, which narrows the gap considerably. So "kprobe always means an int3 trap" is not strictly true anymore. But the ceiling still favors fentry: it is purpose-built for the entry hook, it is typed, and it never falls back to an exception. When you need the lowest, most predictable per-call cost, it is the one to reach for. The video takes both attachment paths apart in a disassembler if you want to see the instruction-level difference rather than take my word for the shape of it.

What you need for fentry (and what breaks)

fentry buys you type safety and low overhead by leaning on infrastructure that older or stripped-down kernels do not have:

• Kernel ≥ 5.5 for the BPF tracing (fentry/fexit) program types.
• BTF in the kernel (CONFIG_DEBUG_INFO_BTF=y, surfaced at /sys/kernel/btf/vmlinux). No BTF, no typed attach — this is the usual reason fentry "mysteriously" refuses to load on a machine where kprobe is fine.
• A function the tracer can reach. Anything inlined, marked notrace, or otherwise missing its __fentry__ site is not attachable. Static/inlined helpers fall into this gap; kprobe on an address can sometimes still reach them.

That list is the real decision tree. kprobe is the tool that works when the target or the kernel does not meet those conditions. fentry is the tool that wins when they do.

What production observability tutorials leave out

The standard eBPF walkthrough shows you the attach and the read. It almost never shows the three things that decide whether your tracer survives contact with production:

1. How the probe installs, because that is what sets the per-call cost.
2. What each call actually costs, because you are about to multiply it by the call frequency.
3. The BTF and kernel-version constraints, because they decide whether your program loads at all on the fleet you are targeting.

The multiplication is the part that bites. Hook something cold — module load, process exit — and the install mechanism barely matters; SentinelEdge uses kprobe on exactly those kinds of events (do_exit, init_module, do_mount) and the overhead is noise. Hook something on the request path — recvmsg on a service doing 100K requests a second — and the same per-call gap you could ignore becomes a percentage of the machine. That is the whole game: not "is eBPF fast," but "is this hook cheap enough at this frequency."

It shows up sharply in AI infrastructure, where eBPF is often the only way to see real syscall latency during inference without changing the serving code — network and file I/O, driver entry points, the per-token streaming path, container-boundary cost in multi-tenant serving. Those are hot paths by definition. Put a trap-based hook on one and your tail latency tells the story at 3am; put a trampoline-based one and you keep the visibility for a cost you can round off.

How I actually hook it, and when I'd switch

I'll be straight about the code you'll find in SentinelEdge: it hooks with kprobe and tracepoints, not fentry. Thirteen kprobe attach points — vfs_open, do_mmap, tcp_connect, do_exit, kmem_cache_alloc, and so on — plus tracepoints for the syscall entries. That was the right call for what the project is: a breadth-first map of kernel activity across a wide set of functions, much of it cold or medium-frequency, aimed at running on a range of kernels without assuming BTF. kprobe attaches everywhere and asks nothing of the build.

The moment I'd move a specific hook to fentry is when it lands on a genuine hot path and I control the kernel it runs on — a modern BTF-enabled build. At that point the typed arguments remove a class of silent version-drift bugs, and the trampoline removes the per-call tax that a request-path frequency would otherwise amplify. It is a per-hook decision, not a project-wide religion.

(The eBPF programs and the ring-buffer data path in SentinelEdge are real and runnable; the distributed pieces are design sketches. The kernel-hooking code is the part worth reading.)

Choosing, quickly

	kprobe	fentry
Installs by	patching the instruction (classically int3)	BPF trampoline on the __fentry__ site
Per-call cost	higher (exception path; less on optimized kernels)	lower (call, no trap)
Arguments	raw pt_regs, PT_REGS_PARMn, untyped	typed via BTF (BPF_PROG)
Needs BTF	no	yes
Min kernel	old	5.5+
Reaches inlined/notrace	sometimes	no
Return values	kretprobe (paired)	fexit (args + retval in one)
Best for	old kernels, broad coverage, cold hooks	hot paths, typed access, controlled modern kernels

The one-line version: fentry is the default on any kernel new enough to give it to you, and kprobe is what you use when the kernel or the target won't. The mistake isn't picking one — it's picking without knowing that a hook on a hot function just made the choice a line item on your CPU budget.

---

Code above is from SentinelEdge. The companion video disassembles both attachment paths at the instruction level.