Originally published on Andrax Pentester by Syed Zada Abrar.

Linux Kernel LSM & eBPF Syscall Hooking Blueprint: From Kernel Memory Architecture to Production Security Enforcement

BLUF: Traditional Linux security auditing tools (such as auditd, syslog, or basic Kprobe-based EDR agents) suffer from a fundamental architectural flaw: Time-of-Check to Time-of-Use (TOCTOU) race conditions. Because Kprobes and Tracepoints execute asynchronously or inspect process state after a system call has already been dispatched to the virtual filesystem (VFS), an attacker can swap file descriptors, modify memory pages via process_vm_writev, or unshare namespaces before an alert is raised.

Linux Security Modules (LSM) combined with eBPF (BPF_PROG_TYPE_LSM, introduced in Linux Kernel 5.7) solve this vulnerability by offering synchronous, inline access control. BPF LSM hooks directly into the Linux kernel's Mandatory Access Control (MAC) decision points (security_file_open, security_bprm_check, security_socket_connect). When a BPF LSM program returns a negative error code (such as -EPERM or -EACCES), the kernel halts the syscall immediately, unwinds stack frames, and returns "Permission denied" to the caller without ever touching the target resource.

---

Step-0 First-Principles Intuition: Why Hooking the Kernel Is Hard

+-----------------------------------------------------------------------------------+
| USER SPACE: Process invokes execve("/bin/malware", argv, envp)                   |
+-----------------------------------------------------------------------------------+
                                         |
                                         v [Syscall Entry via INT 0x80 / SYSCALL]
+-----------------------------------------------------------------------------------+
| KERNEL SPACE: sys_execve() -> do_execveat_common()                                |
+-----------------------------------------------------------------------------------+
                                         |
                                         v
+-----------------------------------------------------------------------------------+
| LSM Security Check: security_bprm_check(bprm)                                     |
|                                                                                   |
|  +-----------------------------------------------------------------------------+  |
|  | eBPF LSM Probe (BPF_PROG_TYPE_LSM) Attached via CO-RE                       |  |
|  | - Inspects bprm->filename, bprm->cred, process cgroup                       |  |
|  | - Evaluates security policy in kernel JIT memory space                     |  |
|  |                                                                             |  |
|  | DECISION:                                                                   |  |
|  |  +--> Allow (Return 0)       --> Continue binary loading to VFS             |  |
|  |  +--> Deny  (Return -EPERM)  --> Abort syscall, return EPERM to User Space |  |
|  +-----------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

The Legacy Hooking Matrix & Architectural Failures

1. Syscall Table Hooking: Modern kernels neutralized sys_call_table modification by marking kernel code pages read-only (CR0.WP bit) and introducing Kernel Page Table Isolation (KPTI).
2. Kprobes and Kretprobes: Execute after critical parameters have been parsed and cannot safely reject system calls or mutate return values in standard kernels.
3. Static LSM Modules (SELinux, AppArmor): Compiled into kernel binary or loaded early at boot. Rely on rigid policy binaries.
4. eBPF LSM (BPF_PROG_TYPE_LSM): Combines the safety and performance of the eBPF JIT compiler with the synchronous blocking authority of LSM hooks.

---

Production eBPF C Hook Snippet

// lsm_monitor.bpf.c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char _license[] SEC("license") = "GPL";

#define EPERM 1
#define MAX_PATH_LEN 256

// eBPF LSM Hook: Intercept Process Execution
SEC("lsm/bprm_check_security")
int BPF_PROG(restrict_exec, struct linux_binprm *bprm)
{
    char filename[MAX_PATH_LEN] = {0};
    int len = bpf_d_path(&bprm->file->f_path, filename, sizeof(filename));
    if (len < 0) return 0;

    char target_prefix[] = "/tmp/malware";
    bool is_blocked = true;

    #pragma unroll
    for (int i = 0; i < 12; i++) {
        if (filename[i] != target_prefix[i]) {
            is_blocked = false;
            break;
        }
    }

    if (is_blocked) {
        return -EPERM; // Synchronous kernel block
    }

    return 0;
}

---

E-E-A-T Technical Matrix

Metric / Dimension	Kprobes	Tracepoints	XDP	eBPF LSM
Execution Domain	Dynamic instruction	Static tracepoint	NIC ingress	Kernel Mandatory Access Control
Hook Timing	Arbitrary function	Static kernel trace	Packet arrival	Inline before resource access
Synchronous Blocking?	❌ No	❌ No	✅ Yes	✅ Yes (Returns -EPERM)
TOCTOU Race Resistance	❌ Vulnerable	❌ Vulnerable	N/A	✅ Immune

---

Read the full 4,000-word implementation guide with full libbpf C loader harness and BTF verifier workarounds at Andrax Pentester.