Mustafa ERBAY

Posted on Sep 19 Originally published at mustafaerbay.com.tr

Sysctl Says Zero, the Door Is Open: userfaultfd's Three Entrances

#linux #kernel #userfaultfd #security

After yesterday's kptr_restrict piece I went through the server's sysctl list once more, asking whether I actually know what every security-flavoured line does. I stopped at vm.unprivileged_userfaultfd = 0. I hadn't written that one; it's the kernel's own default. The definition I had in my head was: "if zero, unprivileged users can't open a userfaultfd." Fine, closed, I said. Then, to be sure, I ran a small C program as nobody. It called userfaultfd(2) with one flag, and the kernel handed over the fd.

The sysctl was zero. The door was open. Both were true; my definition was incomplete.

This post chases that incomplete definition: the three separate ways into userfaultfd, the fact that the sysctl cuts only one of them, EPERM arriving in the same shape from two different sources, and how Docker's default seccomp profile mixes into the picture. The lab machine is VPS3 as usual: Ubuntu 24.04, kernel 6.8.0-139-generic, Docker 29.4.3. Every setting I changed (flipping the sysctl to 1 for a second, adding an ACL on /dev/userfaultfd) was reverted; the machine is at the values it started with.

userfaultfd in thirty seconds

Normally, the first time you touch a page the kernel resolves the fault itself: find a zeroed page, map it, carry on. userfaultfd hands that job to user space. You open an fd, register an address range with UFFDIO_REGISTER; a thread that touches a missing page in that range goes to sleep, a message can be read from the fd, you fill the page with UFFDIO_COPY, the sleeper wakes up. The kernel documentation describes this for QEMU/KVM postcopy live migration: the VM starts running on the destination first, and the pages it touches are pulled from the source on demand. CRIU does "lazy pages" restore with the same mechanism; on Android, ART's Concurrent Mark-Compact garbage collector compacts its heap with it; on by default first from T upwards, since December 2023 from S upwards.

My lab program is the barest form of that loop. A handler thread polls the fd and, for every fault, copies in a page full of 'A'. As root, with the plain call:

uid=0 mode=syscall
userfaultfd: OK fd=3
UFFDIO_API: features=0x1ffff
1) user-mode touch 0x7ec46f734000 ...
  [handler] fault addr=0x7ec46f734000 flags=0 tid=3997850
   read byte 'A', served=1
2) kernel-mode read(/dev/zero -> 0x7ec46f733000) ...
  [handler] fault addr=0x7ec46f733000 flags=0x1 tid=3997850
   read: 4096 bytes, served=2

There are two kinds of fault here, and the rest of the post rests on the distinction. The first is user-mode: I read p[0], the CPU itself took the page fault. The second is kernel-mode: I called read(/dev/zero, p, 4096), and the kernel faulted while trying to write into my page with copy_to_user. That's why the handler sees flags=0x1: UFFD_PAGEFAULT_FLAG_WRITE, because this time the writer was the kernel. Both faults came through the same fd and were resolved with the same UFFDIO_COPY. From outside there's no difference. For security the difference is everything.

Three entrances, three lines of code

In fs/userfaultfd.c, the syscall's permission check in 6.8 is three decisions (the file was merged into mm/userfaultfd.c in May 2026; the same three lines sit there on today's master):

static inline bool userfaultfd_syscall_allowed(int flags)
{
    if (flags & UFFD_USER_MODE_ONLY)
        return true;
    if (capable(CAP_SYS_PTRACE))
        return true;
    return sysctl_unprivileged_userfaultfd;
}

The sysctl comes last. Two doors stand before it, and neither looks at it. Anyone passing the UFFD_USER_MODE_ONLY flag gets an fd; anyone carrying CAP_SYS_PTRACE gets an fd. The sysctl only cuts the "no flag, no capability" call. That's why nobody's flagged call succeeded; the sysctl's definition already says so, only my mental shorthand didn't. The official text: zero "restrict[s] unprivileged users to handle page faults in user mode only." It restricts; it doesn't close.

There's a third entrance too, outside the syscall: /dev/userfaultfd. A misc device is registered in the same file; the USERFAULTFD_IOC_NEW ioctl calls new_userfaultfd(flags) directly, and userfaultfd_syscall_allowed never runs. Whoever can open the device gets an fd that also traps kernel-mode faults, whatever the sysctl says. Who can open the device is decided by file permissions; on this machine, crw------- root root 10,257.

These three doors weren't designed together; each came to fix a problem the previous one created, and the history carries the reasoning.

4.3: userfaultfd is born. Anyone can open one.
5.2 (May 2019, Peter Xu): the vm.unprivileged_userfaultfd sysctl is added, default 1. The commit message states the reason plainly: userfaultfd can be used "to stall a kernel thread" and is "one of the few that never needs privilege."
5.11 (December 2020, Lokesh Gidra / Google): the UFFD_USER_MODE_ONLY flag arrives and the default drops to 0. Half the reasoning is security, half is Android: the commit message says the Android userland "will behave as with the sysctl set to zero"; the flag leaves user-mode faults to unprivileged apps while closing kernel-mode faults, giving consumers like ART's garbage collector a path independent of the sysctl.
6.1 (August 2022, Axel Rasmussen / Google): /dev/userfaultfd is added. The complaint in the commit message is on behalf of hypervisors doing live migration: toggling the sysctl "increases attack surface by allowing any unprivileged user to do it," while granting CAP_SYS_PTRACE also brings the ability to "examine and change [another process's] memory and registers." All that's wanted is userfaultfd; the device file grants exactly that.

Two of the three doors came out of Google: one for the phone's garbage collector, the other for VM migration. The sysctl sits between them, having handed part of its 2019 meaning to the flag in 2020 and another part to the device in 2022.

UFFD_USER_MODE_ONLY is a property of the fd, not the caller

To measure what the flag means I made the flagged call as root. The fd came, the user-mode fault was resolved; then:

uid=0 mode=umo
2) kernel-mode read(/dev/zero -> 0x701890d27000) ...
   read: Bad address (errno=14), served=1
3) kernel-mode write(0x701890d26000 -> pipe) ...
   write: Bad address (errno=14), served=1

Being root changed nothing. A single line in handle_userfault does this: if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY)) goto out; — if the fault didn't come from user mode and the fd is flagged, VM_FAULT_SIGBUS. The decision is frozen when the fd is created; the caller's identity isn't consulted later. Hand a flagged fd to a more privileged process and it still can't trap kernel-mode faults; I think that's the right design.

Two observations worth recording. First, the userfaultfd(2) man page says that in this case "a SIGBUS signal will be delivered"; what I saw wasn't a signal but EFAULT returned from read(). When the kernel faults inside its own copy_to_user, it turns that into -EFAULT for the syscall via exception fixup; the process doesn't die. The commit that added the flag already has it right: kernel-mode faults are handled "as if SIGBUS were always raised, causing the kernel code to fail with EFAULT." And the same man page's EPERM entry never mentions the flag exception at all; my incomplete morning definition may well have come from there. Second, in the first version my third test did write(fd, p, 4096) to /dev/null and produced no fault at all: /dev/null's write doesn't even read the buffer, it just returns count. To provoke a kernel-mode fault the data has to actually be copied; switching to a pipe brought the EFAULT. The assumption "the kernel touched my page" falls apart when tested against a driver that doesn't touch it.

Why kernel-mode faults are a category of their own

The commit messages explain this better than I could. The 5.11 series opens with: "It has been demonstrated on various occasions that suspending kernel code execution for an arbitrary amount of time at any access to userspace memory (copy_from_user()/copy_to_user()/...) can be exploited to change the intended behavior of the kernel." The mechanism is simple: while the kernel reads your buffer inside a syscall, if the page is missing it stops and waits for your handler. Keep the handler deliberately slow and the kernel waits at that line as long as you like. In another thread, during that wait, you change something that wasn't supposed to change. The race window grows from a nanosecond to an hour. That's the sysctl documentation's sentence about "certain vulnerabilities" being harder to exploit.

Why aren't user-mode faults equally dangerous? Because the one waiting is your own thread; the kernel holds no lock, there's no half-finished structure. That's exactly what ART wants: to move its own heap's pages at its own pace. So the flag doesn't say "close userfaultfd entirely for unprivileged users"; it says "leave unprivileged users their own faults, not the kernel's." I think that's the right trade-off; the alternative would be handing CAP_SYS_PTRACE to every app on Android.

/dev/userfaultfd: device permission, not capability

I tried the third door as nobody. The device is 0600 root, so open returned EACCES; expected. Then I gave nobody capabilities one at a time with setpriv:

nobody +	plain `userfaultfd(2)`	`/dev/userfaultfd`
nothing	EPERM	EACCES
`CAP_SYS_PTRACE`	OK, kernel-mode included	EACCES
`CAP_DAC_OVERRIDE`	EPERM	OK, kernel-mode included
ACL `u:nobody:rw`	EPERM	OK, kernel-mode included

What the table says: the two doors have different keys. The syscall's key is CAP_SYS_PTRACE, the device's key is file permission; CAP_DAC_OVERRIDE opens the device because it punches through file permissions, but says nothing to the syscall. I granted the ACL with setfacl -m u:nobody:rw /dev/userfaultfd, nobody resolved both user- and kernel-mode faults, and I removed it with setfacl -b. Since the device belongs to udev, a persistent rule would go under /etc/udev/rules.d; I left nothing persistent behind.

The answer to who uses this door in real life is in QEMU's source: uffd_open in util/userfaultfd.c tries /dev/userfaultfd first, uses USERFAULTFD_IOC_NEW if it opens, and falls back to the syscall otherwise; the comment says it has "better permission controls" and "allows kernel faults without any privilege requirement." On the libvirt side, the 10.1.0 (March 2024) NEWS entry reads: on kernels that support the device, "libvirt will now automatically grant QEMU access to this device. It's no longer needed to set vm.unprivileged_userfaultfd sysctl." qemuProcessAllowPostCopyMigration in qemu_process.c checks whether the device exists and is in the cgroup_device_acl list, then labels it in the mount namespace; the default list in the qemu.conf template includes /dev/userfaultfd. So on a modern KVM host, postcopy needs neither the sysctl nor CAP_SYS_PTRACE; the device file and libvirt's ACL suffice. The 5.11 commit's note — "this will fail postcopy live migration ... set 'vm.userfault = 1'" — became unnecessary two years later thanks to the device.

Two EPERMs in a container, one errno

Docker's default seccomp profile (moby/profiles, default.json) opens with defaultAction: SCMP_ACT_ERRNO, defaultErrnoRet: 1; every syscall not on the list gets EPERM. userfaultfd isn't on the list. ptrace is; on kernels newer than 4.8 it's allowed with no capability requirement. So in a container, ptrace is free and userfaultfd isn't. The matrix:

`docker run ...`	plain syscall	`UFFD_USER_MODE_ONLY`	`/dev/userfaultfd`
default	EPERM	EPERM	ENOENT (no device)
`--cap-add SYS_PTRACE`	EPERM	EPERM	—
`--security-opt seccomp=unconfined`	EPERM	OK, user-mode only	—
`unconfined --cap-add SYS_PTRACE`	OK, kernel-mode included	—	—
`--device /dev/userfaultfd`	EPERM	EPERM	OK, kernel-mode included
`--device /dev/userfaultfd --user 65534`	—	—	EACCES

The two bold cells are the container-side lesson. First: under the default profile, the flagged call gets EPERM too; what nobody managed on the host, root can't manage in the container. Same errno 1, completely different source: on the host it's the kernel's userfaultfd_syscall_allowed, in the container it's the seccomp filter. A little sibling of the "four ceilings, one message" situation from the fork post. --cap-add SYS_PTRACE changes nothing, because we never reach the kernel; the profile cuts first. The practical way to tell them apart is to try once with --security-opt seccomp=unconfined: if the result changes, the wall was seccomp.

The second is more interesting: with --device /dev/userfaultfd, seccomp profile still in place, the container's root got a full userfaultfd that traps kernel-mode faults. Because the profile blocks the userfaultfd syscall but not ioctl; the open + ioctl(USERFAULTFD_IOC_NEW) path looks like ordinary file I/O to the profile. That's the design the kernel chose deliberately: the device path means "the permission lives in the file, not in the syscall," and the documentation says outright that "vm.unprivileged_userfaultfd is not considered." In May 2026 someone reported this to the stable list as an "ioctl bypass"; Greg Kroah-Hartman's only reply was to resend it properly to the right maintainers and lists, and on today's master userfaultfd_dev_ioctl still calls new_userfaultfd directly. Not a bug, a design. But when you see a devices: [/dev/userfaultfd] line in a compose file, you need to know that, as far as userfaultfd is concerned, it opens the same door as the --cap-add SYS_PTRACE + seccomp=unconfined pair; it only looks narrower and less alarming.

Two more notes. Podman's profile (containers/common, seccomp.json) reaches the same result by a different route: its default errno is ENOSYS, but userfaultfd is explicitly listed with SCMP_ACT_ERRNO/EPERM; so in Docker it's "not on the list," in Podman it's "on the list, forbidden." In rootless setups the CAP_SYS_PTRACE row of the table disappears entirely: capable() checks the initial user namespace, and the man page says so, "in the initial user namespace"; a "root" inside a user namespace with CAP_SYS_PTRACE doesn't open the syscall door. The flag and the device are what's left.

The cost: waking the handler

Then there's the question "what do I lose if I turn this on in production." I measured the first touch of 4096 pages two ways: ordinary anonymous memory and memory registered with userfaultfd. The machine was carrying a load of 12 on 18 cores at the time, so the numbers are noisy; that's why I give ranges, not averages. An anonymous page fault: 4–29 µs. The same fault through userfaultfd: 83–625 µs unpinned; 27–490 pinned to two cores, 22–74 µs pinned to a single core. The copy itself is a 4 KiB memcpy; the entire difference is waking the handler waiting on the fd and carrying the answer back. In postcopy migration a network round trip sits on top of that cost; it's why QEMU's fault thread is a thread of its own.

What's happening while waiting is shown by /proc/PID/fdinfo/<fd>:

pending:    1
total:  1
API:    aa:80000100:80000000000001ff

pending is the number of faults currently awaiting an answer; if a userfaultfd consumer is stuck, this is the first place to look. The waiting thread itself shows a plain S in ps (I deliberately left a registration without a handler hanging: both the user-mode fault and the kernel-mode fault inside read() were S, with wchan at handle_userfault); so if you search with the "process stuck in D" reflex you won't find it, you have to look at wchan.

No trace in dmesg

Between 5.11 and 6.0 the kernel printed, once per boot, a printk_once warning: "uffd: Set unprivileged_userfaultfd sysctl knob to 1 if kernel faults must be handled without obtaining CAP_SYS_PTRACE capability". In two places: on the first syscall refused because of the sysctl, and on a flagged fd's first kernel-mode fault. Both lines were removed in 6.1 when the device arrived. I produced dozens of EPERMs on this machine; dmesg | grep -ci uffd → 0. For diagnosis, strace is all you have:

userfaultfd(O_NONBLOCK|O_CLOEXEC)       = -1 EPERM (Operation not permitted)

At fleet scale, the answer to "who is holding a userfaultfd" is the anon_inode:[userfaultfd] links under /proc/*/fd (zero on this machine) and an auditd rule with -S userfaultfd; for the device path you need an open watch on /dev/userfaultfd, because that path never shows up to a syscall watcher.

Three questions to ask when you see that line: is the flag there (if not, add it; it's enough for most user-mode consumers), is CAP_SYS_PTRACE in CapEff, and if you're in a container, what does the seccomp profile say. Setting the sysctl to 1 should be last on the list; on a 6.1+ kernel, an ACL on the device file does the same job narrowed to a single user.

Questions to ask your own server

If sysctl vm.unprivileged_userfaultfd is zero, don't say "off"; say "unprivileged users trap only their own faults." An application wanting that is normal; ART wants it.
If ls -l /dev/userfaultfd isn't 0600 root, who changed it, which udev rule? Anything that can open that file can stall kernel-mode faults.
On a virtualization host with libvirt ≥ 10.1 and postcopy, you don't need to touch the sysctl; check whether cgroup_device_acl in qemu.conf includes the device.
When you see EPERM in a container, try once with seccomp=unconfined first; if the result changes, the wall is seccomp, not the kernel.
Review devices: [/dev/userfaultfd] in compose files with the same seriousness as cap_add: [SYS_PTRACE].
If a consumer is stuck, look at pending in /proc/PID/fdinfo/<fd>; non-zero means the handler isn't answering.

Fixing the definition

In the morning I said "zero, closed." By evening I have this: the sysctl cuts only one of three doors, and only the unprivileged-and-unflagged use of that one. The other two doors don't even know it exists. That's not a bug; it's where the design arrived in three steps from 2019 to 2022. The sysctl was born as a blunt switch, the flag split it by "whose fault is it," and the device moved the permission into the filesystem. In today's picture, the only thing the sysctl still owns is "a process with nothing at all being able to stall the kernel," and a default of zero for that is right.

What actually bothers me is looking at a number in a list and saying "I know." Yesterday I tripped over a 2 I'd written myself in kptr_restrict; today over a 0 the kernel wrote. Both times the fix was the same: try the value instead of reading it.

DEV Community