Mustafa ERBAY

Posted on Sep 13 Originally published at mustafaerbay.com.tr

Root and Still Unable to Write to /tmp: fs.protected_regular

#linux #kernel #security #sysctl

This afternoon, on my own server, as root, I could not append a single line to an empty file in /tmp. The file belonged to my CI runner's user, its mode was rw-rw-r--, the directory was the wide-open /tmp, and I was root. The kernel shut the door anyway:

$ sudo -u github-runner touch /tmp/protreg-lab
$ ls -l /tmp/protreg-lab
-rw-rw-r-- 1 github-runner github-runner 0 Sep 13 13:33 /tmp/protreg-lab

$ sudo sh -c 'echo root-yazdi >> /tmp/protreg-lab'
sh: 1: cannot create /tmp/protreg-lab: Permission denied

$ sudo dd if=/dev/null of=/tmp/protreg-lab
dd: failed to open '/tmp/protreg-lab': Permission denied

$ sudo tee -a /tmp/protreg-lab </dev/null
tee: /tmp/protreg-lab: Permission denied

Then I wrote to the same file, as the same root, from Python, and nothing happened:

$ sudo python3 -c 'open("/tmp/protreg-lab","r+").write("python-r+-yazdi\n")'
$ sudo cat /tmp/protreg-lab
python-r+-yazdi

Shell redirection, dd and tee are refused; Python's r+ mode goes through. Same user, same file, same second. This is not a permissions problem, because permissions are meaningless for root anyway. It is a rule that entered the kernel in 2018, reached servers through systemd in 2019 and then through the distributions' own packages: fs.protected_regular. The point of this post is simple: the first time you meet this rule without knowing it, you will most likely "fix" it by switching it off, because even the official error text of a large project tells you to. Today I measured instead what the rule says, how it got onto my server, and where it actually breaks.

The difference is a single flag: O_CREAT

strace shows what the shell redirection does:

$ sudo strace -f -e trace=openat -o /tmp/protreg-strace.txt sh -c 'echo x >> /tmp/protreg-lab'
$ sudo grep protreg-lab /tmp/protreg-strace.txt
436194 openat(AT_FDCWD, "/tmp/protreg-lab", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 EACCES (Permission denied)

>> means "create if missing, otherwise append", so it carries O_CREAT. Python's r+ means "open an existing file for read and write"; no O_CREAT. The question the kernel asks is not about the file's permissions but about the opener's intent: "Did you mean to create this file, or are you knowingly opening an existing one?" A call that arrives with O_CREAT, when the file is already there and belongs neither to the caller nor to the directory owner, is turned away with EACCES. For a call without O_CREAT the check never runs; classic permissions speak there, and for root classic permissions always say yes.

Rather than guessing which everyday tool arrives with which flag, I counted with strace. As root I ran cp over the existing runner-owned file: openat(..., O_WRONLY|O_TRUNC), no O_CREAT, it went through, and the file stayed the runner's. install first deleted it with unlinkat and then opened a new file with O_CREAT|O_EXCL; the new file was root's, so it went through. rsync opened a temporary file next to it, something like .protreg-lab2.BvKtda, and renamed it over the top; sed -i took the same road. So >>, >, dd of= and tee trip; cp, install, rsync and sed -i do not. Same job, four different system-call patterns, and the rule looks at only one of them.

The rule itself is the may_create_in_sticky() function in fs/namei.c. The comment block above it lists the conditions one by one: the sysctl must be on, the file must already exist, it must live in a sticky directory, we must not own the file, the directory owner must not own it either, and the directory must be world-writable. With the value 2 the last condition relaxes: group-writable is enough. The line that stopped me in the code is one that does not exist: nowhere in the function is there a capable() or any similar capability check. No exception was written for root. The check looks only at identities (file owner, directory owner, the caller's fsuid) and the directory mode.

Why such a rule? The 2018 commit message sums it up as making "data spoofing attacks harder" and lists eight old CVE numbers. The shape of the attack is simple: a privileged program assumes it will "create" a predictable file such as /tmp/report.txt, opens it with O_CREAT but without O_EXCL. If an attacker creates that file beforehand under their own user, the program writes into the attacker's file without a hint of suspicion; the content belongs to the attacker, the read permission is in the attacker's hands. The rule steps in at exactly that moment: "You meant to create this, but it is someone else's file." It is the hardening Openwall had done for years, carried into the mainline kernel.

Four siblings, two waves

fs.protected_regular did not arrive alone. The kernel documentation describes four settings side by side; I tested all four on my server.

The first wave was 2012, Linux 3.6: protected_symlinks and protected_hardlinks. The symlink rule goes like this: a symlink in a sticky, world-writable directory may be followed only when the link's owner and the follower's owner are the same, or when the link's owner is the directory's owner. Again, no exception for root:

$ sudo -u github-runner ln -s /etc/hostname /tmp/protreg-link
$ sudo cat /tmp/protreg-link
cat: /tmp/protreg-link: Permission denied

$ sudo -u github-runner cat /tmp/protreg-link
mustafaerbay-vps3

The user who created the link can read it; root cannot. The hardlink rule is written a little differently: may_linkat() contains a call to inode_owner_or_capable(), so root, who does not own the file but carries CAP_FOWNER, passes this check. An ordinary user, on the other hand, cannot hardlink a file they neither own nor have read-write access to; and the error code here is EPERM, not EACCES:

$ sudo -u github-runner ln /etc/hostname /tmp/protreg-hard
ln: failed to create hard link '/tmp/protreg-hard' => '/etc/hostname': Operation not permitted

The second wave was 2018, Linux 4.19: protected_fifos and protected_regular. Same logic, this time for named pipes and regular files. Root is refused on the pipe as well:

$ sudo -u github-runner mkfifo /tmp/protreg-fifo
$ sudo sh -c 'echo x > /tmp/protreg-fifo'
sh: 1: cannot create /tmp/protreg-fifo: Permission denied

The four settings do not even accept the same values. The sysctl table in namei.c defines the upper bound as 1 for symlinks and hardlinks and 2 for FIFOs and regular files. I tested what 2 means too: I set up a group-writable, sticky directory (1770), the runner user dropped a file in it, and root again could not append. With the value 2 on my server that was the expected outcome; with 1 this directory would have been out of scope.

The scope is not just /tmp

The rule looks at directory mode, not directory name; every place with the sticky bit and write permission is in scope. My server has three:

$ ls -ld /tmp /var/tmp /dev/shm
drwxrwxrwt 56 root root 262144 Sep 13 13:33 /tmp
drwxrwxrwt 17 root root   4096 Sep 13 13:33 /var/tmp
drwxrwxrwt  2 root root     40 Sep 10 21:30 /dev/shm

/var/tmp is not wiped on reboot, so old files there live longer and their owner may be a user who left long ago. /dev/shm is home to shared-memory files; a segment two services open by the same name with O_CREAT is closed to the second arrival if they run as different users. With the value 2 the list grows: team-shared build and hand-off directories set up with sticky modes such as 1770 or 3770 fall in scope too (a setgid-only 2770 is out of scope; the function's first line returns without looking if there is no sticky bit). Where classic permissions let group members append to each other's files, in these directories >> now works only for the file owner and the directory owner. The experiment in the 1770 directory I set up showed exactly that. I have not lived this on a build farm, I am combining the conditions on paper: if the value ever moves from 1 to 2, these directories are what sits behind the complaint "the script that worked yesterday gives Permission denied today".

How this setting reached your server

The kernel's own default is zero for all four. Whoever raised it is someone who dropped a file on your server. On mine, that file comes from the procps package:

$ uname -r
6.8.0-139-generic

$ sysctl fs.protected_symlinks fs.protected_hardlinks fs.protected_regular fs.protected_fifos
fs.protected_symlinks = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_fifos = 1

$ dpkg -S /usr/lib/sysctl.d/99-protect-links.conf
procps: /usr/lib/sysctl.d/99-protect-links.conf

My own /etc/sysctl.d/99-security.conf contains only the hardlinks and symlinks lines; regular = 2 and fifos = 1 were not my doing, they came with Ubuntu 24.04's procps package. Seeing that, I opened three distribution images side by side and looked:

ubuntu:24.04      procps 2:4.0.4-4ubuntu3.3   /usr/lib/sysctl.d/99-protect-links.conf  regular=2 fifos=1
debian:bookworm   procps 2:4.0.2-3            /usr/lib/sysctl.d/99-protect-links.conf  regular=2 fifos=1
debian:trixie     linux-sysctl-defaults 4.12.1 /usr/lib/sysctl.d/50-default.conf      regular=2 fifos=1

In Debian 13 the file has moved out of procps into the linux-sysctl-defaults package built from linux-base, its name became 50-default.conf, and the value stayed the same. systemd's own sysctl.d/50-default.conf, by contrast, says regular = 1 and fifos = 1; the NEWS file for v241 (February 2019) openly admits this is "technically a backwards incompatible change" and hands anyone who wants to disable it a recipe for /etc/sysctl.d/60-protected.conf. Debian and Ubuntu do not even package that systemd file; under /usr/lib/sysctl.d/ on my server there are three files (10-apparmor.conf, 50-pid-max.conf, 99-protect-links.conf) and no 50-default.conf. The gap is filled by procps on bookworm and noble and by linux-sysctl-defaults on trixie. So upstream says 1, Debian and Ubuntu go a step further to 2. The only way to know which one a given server has is to look.

And then a surprise from a place I had not looked: the Linux virtual machine inside Docker Desktop on my Mac. Opening the same debian:trixie-slim image there, sysctl said this:

6.10.14-linuxkit
fs.protected_regular = 0
fs.protected_fifos = 0
fs.protected_symlinks = 1
fs.protected_hardlinks = 1

The container had installed the linux-sysctl-defaults package, the file was in place, and the value was zero. Because fs.protected_* is a per-kernel setting, not a per-namespace one; the value you see inside a container is the host kernel's value, not the value in the file under the container's /usr/lib/sysctl.d. I confirmed that on my server too: an alpine container sees 2, sysctl -w from inside gets "Read-only file system", and docker run --sysctl fs.protected_regular=0 says "is not allowed" before it even starts. Result: 0 on the development machine, 2 in production. Same image, same code, two different /tmp behaviours. One of the cheapest ways to manufacture a "works on my machine".

Where it breaks in real life: minikube and a lock file

One victim of this rule that made it into an official error catalogue is minikube. In an issue opened in March 2020 a user runs sudo minikube --vm-driver=none and gets this error: unable to open /tmp/juju-mk72a1...: permission denied. The same error had been filed two months earlier, in January 2020, as issue #6391; minikube's code links to that number today. The lock file comes from the juju/mutex library; mutex_flock.go opens the lock under os.TempDir() as juju-<name>, and the open call is the same today:

fd, err := syscall.Open(flockName, syscall.O_CREAT|syscall.O_RDONLY|syscall.O_CLOEXEC, 0600)

Right below it is a well-meaning addition: if the program runs under sudo, it chowns the file to SUDO_UID/SUDO_GID so that the user running it later without sudo can use the lock too. Two good intentions collide here. The lock file is handed over to the user; on the next sudo minikube call root tries to open a /tmp file owned by someone else with O_CREAT; protected_regular kicks in. The user in the issue finds this themselves, writes that the problem disappears when opening with O_RDWR instead of O_CREAT, and opens an issue in the juju/mutex repository. That issue is still open; the repository's last push was September 2023, the last commit on master is from February 2022, and the line there is the line above.

minikube's own fix deserves more discussion. The file pkg/minikube/reason/known_issues.go has an entry for this error; the ID is HOST_JUJU_LOCK_PERMISSION, and the advice text reads, verbatim:

Run 'sudo sysctl fs.protected_regular=0', or try a driver which does not require root, such as '--driver=docker'

Having the user switch off a kernel-wide protection because of the path of a single lock file. A bad trade in my view, but I understand why it is tempting: the sysctl is one line, moving the lock is another repository's responsibility. That is why I said at the top that you will switch it off without knowing; the road there has been paved for you.

My own /tmp is fertile ground for the same collision. I counted:

$ ls -la /tmp | awk '{print $3}' | sort | uniq -c
     11 501
    129 github-runner
     44 root

501 owns the vpsman build directories that arrive from my Mac via rsync; there is no such user on the machine, the number was carried over as is. Of github-runner's 129 entries, 115 are regular files, SQLite leftovers from CI tests. Of root's 44 entries only seven are regular files, the rest are directories like .X11-unix. Three owners, one directory. If a CI step appended to a fixed-name /tmp file with sudo, today's experiment would replay itself. I also checked the audit records to see whether it had happened in the past; apart from my own experiments there is not a single record. But let me be honest: my log files only reach back sixteen hours. I cannot say "it never happened"; I can say "it did not happen in the last sixteen hours".

The invisible refusal: no trace in dmesg

The most insidious part of this rule is its silence. The refused call gets EACCES, and not a single line lands in dmesg. The commit message even records the decision: Kees Cook dropped the rate-limited pr_warn "in favor of audit changes in the future". The code today calls audit_log_path_denied(), and that function in kernel/audit.c first of all returns silently unless audit_enabled. So no auditd, no trace. My server has it, and my experiments looked like this:

type=ANOM_CREAT msg=audit(1789295611.382:338283): op=sticky_create_regular ... uid=0 ... comm="dd" exe="/usr/bin/dd" ... res=0
type=ANOM_LINK  msg=audit(1789295624.590:338351): op=follow_link ... uid=0 ... comm="cat" exe="/usr/bin/cat" ... res=0
type=ANOM_LINK  msg=audit(1789295624.655:338364): op=linkat ... uid=1001 ... comm="ln" exe="/usr/bin/ln" ... res=0
type=ANOM_CREAT msg=audit(1789295624.732:338377): op=sticky_create_fifo ... uid=0 ... comm="sh" exe="/usr/bin/dash" ... res=0

The op field tells you which rule fired: sticky_create_regular, sticky_create_fifo, follow_link, linkat. A small trap: ausearch -m ANOM_CREAT came back empty for me, and adding --input-logs found the seven ANOM_CREAT records from my experiments. The reason is that when ausearch's standard input is not a terminal (a single command over ssh, a script, cron) it reads records from standard input rather than from the log file; grep sticky_create /var/log/audit/audit.log* is the shortest route, and since it matches on the prefix it is version-independent. That last note matters, because the op string changes with the kernel version: my 6.8 writes the refusal in a world-writable directory as sticky_create_regular, and after the may_create_in_sticky() reflow that landed in 6.11 the same refusal becomes sticky_create, with the _regular/_fifo suffixes left only in the group-writable branch that the value 2 opens. Even a single one of these lines showing up in production is proof that some tool is wandering around /tmp with a fixed name and O_CREAT; that is a signal, not noise.

What to do: move the file, not the rule

When you run into it there are four roads ahead, and this is my ordering.

First, take the file out of the shared directory. For services that is one line: PrivateTmp=true. The service sees its own /tmp and never meets anyone else's file. In the systemd sandbox post I listed it as one of the three low-risk settings of the first phase, without giving the reason; this experiment supplies the reason. For non-persistent working files, RuntimeDirectory= gives you your own directory under /run/<service>, owned by you, no sticky bit, the rule never engages.

Second, do not leave files with predictable names. mktemp and O_EXCL exist for this: if the file exists, do not open it, fail. The attack the rule tries to block is precisely "writing into a pre-created file"; O_EXCL closes that attack even without the rule.

Third, if you are knowingly opening an existing file, do not say O_CREAT. In Python r+, in Go os.OpenFile(path, os.O_RDWR, 0). The suggestion in the juju/mutex issue is exactly this. If the lock file must pre-exist, create it in a setup step as the directory owner (root); the directory owner's files are exempt from the rule; I measured that too: the runner user appended to root's 666 file with O_CREAT without trouble.

Fourth, and deliberately last on the list: lowering the sysctl. Going from 2 to 1 takes group-writable sticky directories out of scope; 0 turns everything off. If you do it, mind the file name: sysctl.d files are sorted by name regardless of which directory they live in, and the last one to set a key wins. The 60-protected.conf that systemd's NEWS suggests sorts before 99-protect-links.conf, so on Ubuntu and Debian 12 it is overridden at boot; sysctl --system shows you the order. My own 99-security.conf only works by coincidence, because 99-s sorts after 99-p. The name must sort after 99-protect-links.conf, the reason belongs in a comment line, and remember that the setting is machine-wide; for one tool's lock file you are lowering the protection of all of /tmp, /var/tmp and /dev/shm.

There is also a road zero, for emergencies: delete the blocked file, or hand it to the directory owner with chown root:. That is what the people in the minikube issue did first, and they were right; it rescues the minute without touching the sysctl, but on the next sudo call the file belongs to someone else again.

My checklist:

Read sysctl fs.protected_regular separately in production and on the development machine; if they differ, write the difference down.
Find which file the value comes from with grep -rn protected_ /usr/lib/sysctl.d /etc/sysctl.d; mine came from procps, yours may come from linux-sysctl-defaults or systemd.
If you have auditd, look at grep -h sticky_create /var/log/audit/audit.log* weekly; a single record names the tool.
Count how many owners your /tmp has with ls -la /tmp | awk '{print $3}' | sort | uniq -c; more than two and a collision is a matter of time.
Look for fixed-name /tmp files in CI steps that run with sudo; move what you find to mktemp or RuntimeDirectory.
Do not try sysctl -w fs.protected_* inside a container; it is read-only and it is the host's value anyway.

The flag of intent

This morning I believed root opened every door. This afternoon the kernel pointed at one door and said "you meant to create this, but it belonged to someone else". The decision was made in 2012 for links and in 2018 for regular files; I learned it eight years later by experimenting on my own server. Opening a file with O_CREAT is not a technical detail, it is a declaration of intent: "This file will be mine." The kernel now compares that declaration with the file's real owner, and if they do not match it does not care that you are root. When a tool's error text tells you to "disable the protection", the real question to ask is why that tool was written to mistake someone else's file for its own.

DEV Community