Mustafa ERBAY

Posted on Sep 19 Originally published at mustafaerbay.com.tr

Locked on My Laptop: the One-Way 1 of unprivileged_bpf_disabled, and bpf_jit_harden

#linux #kernel #ebpf #security

This evening I wrote one line, as root, into the Docker Desktop virtual machine on my laptop:

# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled
# echo 0 > /proc/sys/kernel/unprivileged_bpf_disabled
bash: echo: write error: Operation not permitted
# sysctl -w kernel.unprivileged_bpf_disabled=0
sysctl: setting key "kernel.unprivileged_bpf_disabled": Operation not permitted

I am root, the file is 0644, the valid range is 0-2, and I cannot write anything but 1. Six containers were running in that VM at the time (three Postgres, a ClickHouse, NATS, MinIO), all in the middle of other work; I didn't want to restart the machine. As I write these lines, the lock is still in place. This is one of a handful of "write once" knobs in the kernel's sysctl interface, and I turned it on purpose, because that one-way direction is exactly what this post is about.

The curiosity started on the real server. On the VPS (Ubuntu 24.04, 6.8.0-139-generic) the same file shows 2. There are 16 files under /etc/sysctl.d/, some mine, some Ubuntu's; not one line in them mentions bpf. Same for /usr/lib/sysctl.d/. Nobody wrote it, yet the value is 2. In the previous post I discovered, four months late, that I had set kptr_restrict to 2 with my own hands; this time it's the reverse, a 2 nobody wrote. I'll look at two knobs: kernel.unprivileged_bpf_disabled and net.core.bpf_jit_harden. Both appear one under the other in "eBPF hardening" lists, both are aimed at unprivileged users, and neither list says who pays for them.

Who wrote the 2

The answer is in /boot/config-6.8.0-139-generic: CONFIG_BPF_UNPRIV_DEFAULT_OFF=y. With that option on, the kernel sets the knob to 2 at boot; no sysctl file needed. bpftool feature probe kernel says as much in its first line: "bpf() syscall restricted to privileged users (admin can change)".

The option's history is short and has two steps. Daniel Borkmann added the knob in May 2021 (commit 08389d888287, 5.13); in that form it defaulted to off, and distributions turned it on if they wanted. Five months later Pawan Gupta of Intel made it default y (commit 8a03e56b253e, 5.16). The commit message doesn't hide the reasoning: "Disabling unprivileged BPF would help prevent unprivileged users from creating certain conditions required for potential speculative execution side-channel attacks", and "Sync with what many distros are currently applying already". So the distributions were already closing it, and upstream followed. Ubuntu's 6.8 has it on; the linuxkit kernel in Docker Desktop on my laptop (6.10.14-linuxkit, arm64) has # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set and the value is 0. Two kernels on one laptop, two different defaults. Hardening lists say "set this sysctl to 1" without asking which kernel you're on; the kernel doesn't tell you either.

Three values, two directions

The documentation (admin-guide/sysctl/kernel.rst) separates the three values like this: at 0, unprivileged bpf() calls are allowed. Writing 1 makes unprivileged calls return -EPERM, and "once set to 1, this can't be cleared from the running kernel anymore". 2 does the same shutdown, but an admin can later write 0 or 1. With BPF_UNPRIV_DEFAULT_OFF on, the default is 2.

The lock itself is a twenty-line sysctl handler in kernel/bpf/syscall.c, bpf_unpriv_handler. Before writing, it looks at the current value: locked_state = unpriv_enable == 1. After the new value passes the range check, if we're in the locked state and the new value isn't 1, -EPERM. That's all. The irreversibility is not a hardware register or a secure-boot trick, it's an if statement; but since changing that if means changing the kernel, in practice it means a reboot. Writing 1 after 1 is allowed (rc=0); 0 and 2 are closed. I tried them in order in the VM: 2 → 0 → 2 went back and forth without complaint; after 1 the door shut, and bpftool feature probe changed its sentence: "restricted to privileged users (without recovery)".

The logic of the design is this: 2 is the distribution's safe default; if root changes their mind, they open it. 1 is for the installation that doesn't trust even root, the decision "this door will not be opened on this machine" bound to the running kernel. What root cannot open with sysctl -w, whoever takes over root cannot open either. In the kptr_restrict post I wrote that there is no defending against root with a sysctl; this knob is the one exception to that rule, and it was deliberately made one-way so it could be one.

Which calls the gate actually blocks

"Unprivileged eBPF disabled" brings tcpdump, seccomp and containers to mind at first hearing. Looking at the source, the gate is narrower than it sounds.

The check lives inside the bpf() system call and in only two commands: BPF_MAP_CREATE and BPF_PROG_LOAD. The condition is the same in both: if sysctl_unprivileged_bpf_disabled && !bpf_capable(), then -EPERM; and bpf_capable() means CAP_BPF or CAP_SYS_ADMIN. The comment above the code states the intent plainly: the sysctl is meant to stop unprivileged users from creating objects, because the other commands (attach, pin, lookup) already require holding a file descriptor. A user who can't create objects has no descriptors to hold. This narrowing came with 5.19 (commit c8644cd0efe7, May 2022); before that, the check sat at the entry of bpf() and closed every command. Ubuntu 22.04's 5.15 and RHEL 9's 5.14 still have the old behaviour: an unprivileged process can't even do a lookup on a map descriptor it inherited. The sysctl itself has existed since 4.4; Alexei Starovoitov brought the off switch in the same commit that enabled unprivileged eBPF (1be7f75d1668, October 2015).

So what can an unprivileged user load at 0? The same file says: if the program type is not BPF_PROG_TYPE_SOCKET_FILTER or BPF_PROG_TYPE_CGROUP_SKB and there's no CAP_BPF, -EPERM. Network types such as tc, XDP and sk_skb additionally require CAP_NET_ADMIN; tracing types such as kprobe, tracepoint and LSM require CAP_PERFMON. So even at 0, the unprivileged process's world is two program types and a handful of map types, and the verifier is stricter with it (pointer arithmetic, speculation barriers, the instruction count capped at BPF_MAXINSNS, 4096).

Classic BPF is outside this gate. Attaching a filter to a socket with SO_ATTACH_FILTER, or installing a seccomp filter, doesn't go through the bpf() call. I tried this in the locked VM as the nobody user: a "drop everything" filter attached to a socket through setsockopt(SO_ATTACH_FILTER), while the same user's bpf(BPF_MAP_CREATE) got EPERM. tcpdump's filter, Docker's seccomp profile, the browsers' sandbox: all classic BPF, none of them sees this sysctl. The answer to "does writing 1 break seccomp" is no.

I ran the same test on the VPS in Python, raw syscall(321, BPF_MAP_CREATE, ...) through ctypes:

uid=0     bpf(BPF_MAP_CREATE) -> 3 (ok)
uid=65534 bpf(BPF_MAP_CREATE) -> -1 errno=1 (Operation not permitted)

The container side is one layer more interesting, because there the first wall isn't this sysctl. Docker's default capability list (daemon/pkg/oci/caps/defaults.go) has no CAP_BPF, and the default seccomp profile allows the bpf call only if CAP_SYS_ADMIN or CAP_BPF is present. An ordinary container's root cannot call bpf() even with the sysctl at 0; the EPERM comes from seccomp. With --cap-add BPF, both seccomp and the sysctl are passed, because bpf_capable() is now true: even with the lock at 1 in the VM, --cap-add BPF loaded a map and a program. I tried the same flag with --user 65534; it didn't work: the capability is in the container's bounding set (CapBnd), not in the process's effective set (CapEff: 0). In short, the question in a container isn't "what's the sysctl", it's "who holds the capability".

Why it was closed: not code execution, speculation

When I first saw this knob I put the threat in the wrong place: "unprivileged users shouldn't run code in the kernel". The verifier already exists for that. The real reasoning changed after 2018: Spectre. Even a legitimate eBPF program that passed the verifier gives an unprivileged user a way to run, inside the kernel address space, the exact instruction sequences that trigger speculative execution (an array access right after a bounds check, an indirect branch): an attacker writing their own favourite gadget. The verifier adds barriers against this, but closing every variant for every microarchitecture is an ongoing job; closing the door is cheaper.

The trace of this is concrete in x86's arch/x86/kernel/cpu/bugs.c. unpriv_ebpf_notify(), called on every write to the sysctl but speaking only when the new value is 0, prints this to the kernel log at pr_err level if the processor is protected by eIBRS: "WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!". More visible is /sys/devices/system/cpu/vulnerabilities/spectre_v2: on a machine that normally reads "Mitigation: Enhanced / Automatic IBRS" (Intel eIBRS or AMD's AutoIBRS; the kernel puts both in the same mode), with unprivileged eBPF enabled the line becomes "Vulnerable: eIBRS with unprivileged eBPF"; with SMT on and in eIBRS+LFENCE mode, "Vulnerable: eIBRS+LFENCE with unprivileged eBPF and SMT". So setting this sysctl to 0 flips that server's vulnerability report from a "Mitigation" line to a "Vulnerable" line; if you run a scanner, you see red the next morning.

My VPS doesn't show this, because the AMD EPYC that the underlying KVM presents is protected by retpolines rather than eIBRS: the line reads "Mitigation: Retpolines; IBPB: conditional; STIBP: disabled; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected". Writing 0 on this machine wouldn't change the line. Not changing doesn't mean safe; it only means the kernel has nothing to report for this particular combination.

bpf_jit_harden: blinding the constants

The second knob belongs to the JIT and defends against an entirely different attack: JIT spraying. The idea is old (2010, the JITs of browser plug-ins): the attacker puts innocent-looking large constants into the code handed to the compiler; the JIT embeds those constants verbatim in machine code; the bytes of the constants were chosen so that, jumped into mid-instruction, they form a meaningful instruction sequence. The program is legal, the verifier passed it, but bytes the attacker chose now sit in an executable page. The eBPF JIT's answer is "constant blinding": Borkmann's commit from May 2016 (4f3446bb809f, 4.7), masking constants with XOR before they reach the JIT.

The mechanism is bpf_jit_blind_insn in kernel/bpf/core.c. For each instruction with an immediate, a random imm_rnd is drawn and the instruction is split into three: write imm ^ rnd into the auxiliary register AX, XOR AX with rnd, then perform the original operation using AX. User space cannot see AX; bpftool displays it as r11. The constant 0 is special: xor r, r instead of mov r, 0. The result is that the attacker's chosen constant never enters the machine code; two random numbers do.

I looked at it in the lab with the smallest possible program: r0 = 0x1badcafe; exit, loaded as a socket_filter. At bpf_jit_harden = 0, bpftool prog dump xlated shows two instructions; at 1, when nobody loads the same program, four:

harden=1, uid=65534 (nob_h1b):        harden=1, uid=0 (root_h1b):
   0: (b7) r11 = 1614785567              0: (b7) r0 = 464374526
   1: (a7) r11 ^= 2073192161             1: (95) exit
   2: (bf) r0 = r11
   3: (95) exit

1614785567 ^ 2073192161 = 0x1badcafe; the constant is there, but in two pieces. The same load as root is untouched; 1 affects only the unprivileged loader. Writing 2 made root's program four instructions too, and two consecutive loads produced two different pairs (454489470 ^ 12271488 and 1739738543 ^ 2082441041); the randomness is per load. On arm64 the JIT output went from 120 bytes to 136. In a program this small the difference is nothing; in a constant-heavy program it's two extra instructions per constant, register pressure and I-cache. The documentation says "trades off performance" and gives no number; I won't either, I didn't measure it.

The document also defines what "privileged" means here: CAP_BPF or CAP_SYS_ADMIN in the root user namespace. Root inside a container, without --cap-add BPF, is on the blinded side even at 1.

There is a side effect the lists don't mention: if bpf_jit_harden is non-zero, bpf_jit_kallsyms is disabled. bpf_jit_kallsyms_enabled() in filter.h says if (bpf_jit_harden) return false on its first line. In the lab, at 0, root's program landed in /proc/kallsyms as bpf_prog_ed6bbaf49c88a2a1_root_h0 [bpf]; the root program loaded after switching to 1 did not. So the 1 you enabled "for unprivileged users" also erases the BPF symbols from root's perf output; from that day on, every program's address in traces is a bare number. bpftool prog show keeps working, because it reads from bpf(), not from kallsyms. And an unprivileged loader can't even see the JIT size of its own program: BPF_OBJ_GET_INFO_BY_FD returns jited_prog_len = 0 to an unprivileged caller, explicitly zeroed in the code. My program, my descriptor, answer: 0 bytes.

Who on this machine would care

I did the same accounting here that I did in the kptr_restrict post: the ones who pay are the unprivileged processes, so I counted them.

In a bpftool cgroup tree snapshot on the VPS, 135 programs are loaded: 69 are sd_devices (the cgroup_device filter systemd attaches for DevicePolicy/DeviceAllow), 59 are unnamed cgroup_device (the device filter runc attaches for every Docker container; there were 59 containers at that moment), 3 pairs of sd_fw_ingress/sd_fw_egress (systemd-udevd, systemd-journald and systemd-logind, which set IPAddressDeny=any), and the HID driver's hid_tail_call tracing program. In bpftool prog show, the uid column is 0 on all 135 lines. Process side: 700 processes, 459 root, 241 non-root; none of the 241 non-root processes has CAP_BPF, 416 of the root ones do. In /proc/kallsyms there are 136 symbols tagged [bpf] (135 programs + 1 trampoline), every one with the address 0000000000000000, because kptr_restrict = 2 on the same machine. The sysctls of two posts met on one line.

That table translates the knobs for this server as follows. unprivileged_bpf_disabled = 2 costs me zero; there is not a single process loading unprivileged eBPF. Pulling it to 1 would also cost zero and buy one thing: root (or whoever becomes root) not being able to open the door. On a server whose spectre_v2 line says "Enhanced / Automatic IBRS" I would do it; on this retpoline box I leave it at 2, because being able to open the door without a reboot, should I one day need it for a tracing tool, is worth more than a risk I cannot measure. bpf_jit_harden stays at 0: on this machine 1 blinds nobody (no unprivileged loaders) but takes away the BPF symbols in perf; 2 makes every root program loaded from now on (blinding happens at JIT time; it doesn't touch the 135 already loaded, and applies to all of them after a reboot) symbol-less and slightly slower, not for free. On a multi-user build server, on a box where unprivileged users can load socket_filters (if the sysctl must be kept at 0), harden = 1 is a sensible middle ground; 2 only where JIT spraying is genuinely in your threat model and you can sacrifice perf.

The Docker Desktop VM on the laptop is a separate story: there the default was 0; that is, the sysctl-side precondition for a container with a relaxed seccomp profile to load a socket_filter without --cap-add BPF was in place. Now it's 1, and it will stay that way until Docker Desktop restarts. Postgres, NATS and ClickHouse don't care; none of them calls bpf().

The 2026 way to give a container BPF

The real design question here is: if a container (or an unprivileged service) needs to load eBPF, what do you do? There are three options and two are bad. Pulling the sysctl to 0 opens the whole machine. --cap-add BPF (usually with PERFMON and NET_ADMIN alongside) gives the container a capability for the entire host; capabilities know nothing about namespaces. The third is the BPF token that arrived with 6.9: the BPF_TOKEN_CREATE command lets the host administrator delegate specific program and map types to a container's user namespace through a bpffs mount point; what is delegated is written in the mount options (mount -t bpf -o delegate_cmds=...,delegate_maps=...,delegate_progs=...,delegate_attachs=..., kernel/bpf/inode.c). This is why the checks in syscall.c today read bpf_token_capable(token, CAP_BPF) instead of bpf_capable(); the token also passes the sysctl gate for that container. I didn't test it for this post; my 6.8 server doesn't have it, and I read the principles from the source. But the direction is clear: the future of unprivileged eBPF isn't "open the sysctl", it's "write down what you delegated to whom".

The last box in the diagram is the part most often skipped: at harden = 1, root's constants stay in the clear, but kallsyms is off for everyone.

Questions to ask your own server

What does cat /proc/sys/kernel/unprivileged_bpf_disabled say, and why? grep BPF_UNPRIV /boot/config-$(uname -r) has the answer; searching your sysctl files may be a wasted trip.
Is bpftool prog show | grep -v "uid 0" empty? If so, 2 costs you nothing; pulling to 1 is free too, but one-way.
Does the cat /sys/devices/system/cpu/vulnerabilities/spectre_v2 line start with "Mitigation: Enhanced / Automatic IBRS" (without "+ Retpolines" next to it)? If so, writing 0 turns that line into "Vulnerable"; this is the strongest reason to consider 1. (The "PBRSB-eIBRS" suffix that appears on every x86 machine is not it; my retpoline server has it too.)
Before enabling bpf_jit_harden, do you need BPF symbols in perf? Once it's on, programs loaded from then on don't enter /proc/kallsyms.
If a container needs eBPF, will you give it --cap-add BPF, or delegate into its namespace with a BPF token on a 6.9+ kernel? The first is quick, the second is narrow.
If you're going to write 1, write it in a sysctl file and let it arrive at reboot; don't try it by hand on a live machine. I did; on a laptop, so the lesson would be cheap.

What a one-way door is worth

Almost all sysctls are symmetric: you write, and if something breaks, you write it back. That symmetry comforts the administrator, and the attacker too. The 1 of unprivileged_bpf_disabled breaks that symmetry on purpose: a decision that cannot be reversed even by someone who takes the decider's authority. The kernel has few like it (kernel.modules_disabled and kernel.kexec_load_disabled are from the same family), and all of them ask the same question: are you running a system in which "root doesn't trust itself"? If yes, 1 is cheap and clean. If no, 2 is the default the kernel developers picked for you and didn't even bother writing to a file. bpf_jit_harden is a different story; that knob asks whether the group of users you want to protect actually exists, and takes the symbols out of your perf in return. On my server that group doesn't exist; the knob is at 0. On my laptop, a door closed this evening, and the one to open it won't be me; it will be Docker Desktop's next start.

DEV Community