An Unrelated Rename Broke One in Five of My Hardened Opens

Opening a path that came from a user is one of the oldest traps in software. If you check the path first and open it afterwards, a window opens between the two steps; in that window the file can be replaced with a symbolic link, and you end up opening not the thing you checked but the thing the attacker pointed at. Linux has been able to push that job into the kernel since 5.6: give openat2(2) the RESOLVE_BENEATH flag and every step of path resolution is forced to stay below your dirfd. The check leaves your code and moves inside the resolution itself.

So I did that, and got slapped from a direction I did not expect: a share of my opens never happened at all. The error was not EXDEV — it was not "I caught an escape". It was EAGAIN. And its cause had nothing to do with the file I was opening, the directory it lived in, or even its filesystem: somewhere else on the machine, something was renaming files.

In a 200,000-attempt run, 42,489 of them failed. One in five. To make sure it was not the luck of a single run I repeated it three times; the rate came out at 20.6% / 21.3% / 21.2%.

What follows is where that rate comes from and which single change takes it to zero. Along the way openat2's promise to reject unknown flags got tested too; it failed on one bit, and that bit recently became a real flag.

The setup

Everything was measured inside the linuxkit VM of Docker Desktop on my own Mac: kernel 6.10.14-linuxkit, aarch64. The container is alpine:3.20, compiler gcc 13.2.1, libc musl 1.2.5. It is all one C program, built with gcc -O2 -pthread.

An admission is due up front: 6.10 is not today's kernel. On the day I wrote this, kernel.org's stable release was 7.2.11 and mainline was 7.3-rc7. My Docker Desktop VM is old. So I kept two kinds of statements apart: everything I measured was measured on 6.10.14, while cross-version differences were read from the kernel source at the relevant release tag. I say which is which everywhere.

First practical detail: there is no libc wrapper for openat2. The manual page says so in its opening lines and tells you to use syscall(2). That holds for musl and for glibc alike, so your code starts like this:

#include <linux/openat2.h>
#include <sys/syscall.h>

static int sys_openat2(int dfd, const char *path, struct open_how *how, size_t size) {
    return (int)syscall(SYS_openat2, dfd, path, how, size);
}

The second detail is sneakier: always zero struct open_how. The struct was designed to be extensible; when a field is added and your old code gets recompiled against new headers, any garbage in that field blows the call up. A designated initializer (struct open_how h = { .flags = O_RDONLY };) does the zeroing for you, and that is the form the lab program sticks to throughout.

The test tree looks like this: under /lab/root there is file, sub/inner, link_abs pointing at /etc/hostname, link_rel pointing at ../secret, and link_in pointing at sub/inner. Out of scope there is also /lab/secret, whose contents start with SECRET-DATA — that first byte is how I answer the did-it-leak question. The noise-producing renames happen under /lab/noise and /dev/shm, directories that share no component with the path I am opening.

First, I tested the promise: does it really reject unknown flags?

This is the crispest line in openat2's pitch. openat(2) silently swallows bits it does not recognise in flags; openat2 rejects them. It sounds small, but the consequence is not: a call that rejects an unrecognised bit is also a feature probe. It answers "does this kernel know that flag?" with EINVAL.

I tested the claim bit by bit rather than taking it as rhetoric, feeding every bit from 20 to 40 to both calls:

Bit	Value	openat	openat2
20	0x100000 (__O_SYNC)	OK	OK
21	0x200000 (O_PATH)	OK	OK
22	0x400000 (__O_TMPFILE)	EINVAL	EINVAL
23, 24, 25	—	OK	EINVAL
26	0x4000000	OK	OK
27 – 31	—	OK	EINVAL
32 – 40	—	never reaches the call	EINVAL

The last row needs an explanation, because my first probe script misled me here. openat's flags argument is an int; you cannot put bit 32 or above into it. My first version narrowed the bit with an (int) cast, so in the 32–40 range the value reaching the kernel was plainly zero and the "OK" that came back proved nothing. I deleted the finding my own script had manufactured and left the table as it stands: there is no question to ask openat in that range.

That narrowing is not an accident, it is one of openat2's design reasons. The OPENAT2_REGULAR flag added in 7.2 sits exactly at bit 32 (0x100000000), and the kernel source carries a build-time assertion next to it: the bit must live in the upper 32 bits so that open()/openat() cannot express it at all.

The remaining count goes like this: openat2 was asked about 18 unknown bits and rejected 17 of them with EINVAL. openat could only be asked about the 9 bits from 23 to 31, and it quietly accepted all nine. The promise holds there. The one bit where it did not hold is 26.

Worth noting that the table belongs to aarch64: the kernel header says bits 23, 24 and 25 "must not be used" because they collide with real flags on alpha, parisc and sparc. The same probe can answer differently on another architecture, which only sharpens the lesson coming up.

I did not want to walk past that single row without explaining it, because if you cannot explain a measurement you do not have a finding, you have noise. I went to the 6.10 source and found a line in fs/open.c that runs before validation:

u64 strip = __FMODE_NONOTIFY | O_CLOEXEC;
...
flags &= ~strip;
if (flags & ~VALID_OPEN_FLAGS)
    return -EINVAL;

__FMODE_NONOTIFY is a kernel-internal bit and it is exactly 0x4000000. If userspace raises that bit, the kernel strips it, the stripped value goes into validation, and validation quite rightly sees a clean set of flags. In other words the strictness has a hole in precisely the one bit the kernel had reserved for itself.

The rest of the story is more fun. I followed that same line across release tags: the strip is still there in 6.11, 6.12 and 6.13, and in 6.14 strip shrinks to O_CLOEXEC alone. So from 6.14 onwards bit 26 gets EINVAL too. And in 7.2 the bit does not stay empty: it becomes a real flag named O_EMPTYPATH.

Put the three windows side by side:

Kernel	openat2 with bit 26	What a feature probe concludes
6.10 – 6.13	succeeds	"supported" — wrong
6.14 – 7.1	EINVAL	"not supported" — correct
7.2 and later	succeeds	"supported" — correct

The same measurement was repeated with 7.2's new flag: sending OPENAT2_REGULAR (that is, bit 32) returned EINVAL on 6.10 — the right answer, since that kernel does not know the flag. On 7.2 the flag does its job and produces a new error code when the path is not a regular file: EFTYPE. That second half is not measured — there is no 7.2 here; it comes from the manual page. So when writing a probe it is not enough to ask whether EINVAL came back; you also need to know which error you expect.

My VM sits exactly inside the false-positive window. Code that probes a feature by "raise the bit, and if EINVAL comes back it is missing" concludes on 6.10 that O_EMPTYPATH exists, and then gets an error it does not understand when it calls with an empty path. The lesson I am keeping for the next probe I write: a feature probe must verify not just the error, but the expected side effect of success.

RESOLVE_* flags: the measured behaviour

Back to the main subject. The flags in the resolve field constrain the whole of path resolution. The lab walks through 23 cases one by one; these are the instructive ones (dirfd = /lab/root unless noted):

Path	resolve	Result
file	BENEATH	OK
sub/../file	none	OK
sub/../file	BENEATH	OK
..	BENEATH	EXDEV
/etc/hostname	BENEATH	EXDEV
/etc/hostname	IN_ROOT	ENOENT
link_abs → /etc/hostname	BENEATH	EXDEV
link_abs → /etc/hostname	IN_ROOT	ENOENT
link_rel → ../secret	BENEATH	EXDEV
link_rel → ../secret	NO_SYMLINKS	ELOOP
link_in → sub/inner	NO_SYMLINKS	ELOOP
link_in → sub/inner	NO_SYMLINKS + `O_PATH\	O_NOFOLLOW`
proc/self/environ (dirfd = /)	NO_XDEV	EXDEV
proc/self/cwd/... (dirfd = /)	NO_MAGICLINKS	ELOOP
file	BENEATH + IN_ROOT	EINVAL

Three rows deserve a mark.

sub/../file + BENEATH → OK. A .. that stays within scope passes; only a .. that leaves scope gets EXDEV. The reason for measuring this is that the one-line comment in the kernel header is easy to misread: it says the flag blocks "lexical trickery like .., symlinks, and absolute paths" and then closes with "which escape the dirfd". That trailing clause qualifies the whole list, so the header and the manual agree — but someone reading the short comment on its own could conclude that .. is forbidden outright. The manual page is plainer: do not let resolution succeed if any component is not a descendant of dirfd. That distinction is the reason for the entire next section, because .. passing does not mean .. is free.

link_in + NO_SYMLINKS → ELOOP. link_in is an innocent link pointing at sub/inner, inside the scope. RESOLVE_NO_SYMLINKS rejects it too; the flag's job is not to block escapes but to switch symlink resolution off entirely. That is why the manual warns against using it indiscriminately for non-security purposes, as it will break working systems. Verifying that on my own infrastructure is easy: all it takes is a distribution upgrade putting a symlink in the middle of a system path. My own view is that the right use is to enable this flag only in the narrow context of "I am opening a user-supplied path inside my own data directory", not to sprinkle it over every open in the application.

NO_SYMLINKS + O_PATH|O_NOFOLLOW → OK. If you want to inspect the link itself, the door is open: that combination hands you an O_PATH descriptor referring to the link. So "reject links" and "examine the link" can be done in the same call.

Note too that the scope flags cannot be combined: BENEATH | IN_ROOT is a flat EINVAL. The code says so as well, with a comment that scope flags are mutually exclusive.

The real finding: the price of a '..' step is a global lock

Opening sub/../file with RESOLVE_BENEATH works. But what if someone else on the machine is renaming files at the same time?

The setup is simple. One thread calls rename() in a loop — not inside /lab/root, but in a completely different directory. The main loop performs the same open 200,000 times and counts the outcomes. There are four control runs as well: one without noise, one with .. in the path but resolve = 0, and two BENEATH runs with no .. in the path.

First with the noise in /lab/noise, i.e. on the same filesystem as the target (the container's overlayfs):

Run	Opens	EAGAIN	Rate	Renames
1	200,000	3,895	1.95%	45,420
2	200,000	4,338	2.17%	45,498
3	200,000	4,213	2.11%	45,162

Then I moved the noise to /dev/shm — tmpfs, i.e. a different filesystem. My intent was to separate whether the coupling lives inside one filesystem or across the whole machine:

Run	resolve	Opens	EAGAIN	Rate	Renames
1	BENEATH	200,000	41,229	20.6%	208,662
2	BENEATH	200,000	42,682	21.3%	202,176
3	BENEATH	200,000	42,489	21.2%	192,480
1	IN_ROOT	200,000	25,910	13.0%	199,558
2	IN_ROOT	200,000	41,601	20.8%	202,524
3	IN_ROOT	200,000	28,744	14.4%	190,626

The rate went up because renaming is faster on tmpfs: in the same stretch of time overlayfs does roughly 45,000 operations while tmpfs does roughly 200,000, four times as many. That alone does not account for it, though; the rate rose tenfold, not fourfold. Per rename the gap is still there: 0.091 EAGAIN per rename on overlayfs against 0.210 on tmpfs, a further factor of 2.3. That residual is something I neither measured nor can explain; my guess involves the speed of the swap itself, but a guess does not get written up as a measurement. Care is needed here — the gap between BENEATH and IN_ROOT is smaller than the swing in the amount of noise, and the two share the same mechanism; I am not concluding that "IN_ROOT is more robust", because the data does not say that.

The controls carry the thesis on their own:

Run	Opens	EAGAIN
sub/../file, BENEATH, no noise	200,000	0 (14 in one repeat)
sub/../file, resolve = 0, noise	200,000	0
file (no ..), BENEATH, noise	200,000	0
sub/inner (no ..), BENEATH, noise	200,000	0

All three conditions have to hold at once: a scope flag, a .. in the path, and a concurrent rename on the machine. Drop any one of them and the rate is zero.

One honesty note: in one of the "no noise" runs, 14 of 200,000 opens returned EAGAIN (0.007%). My own threads had stopped; those renames came from something else in the VM. So the floor is not zero but near zero — and even that shows the mechanism is machine-wide.

The source explains why. In fs/namei.c, handle_dots() does this after every .. step in a scoped resolution:

if (unlikely(nd->flags & LOOKUP_IS_SCOPED)) {
    smp_rmb();
    if (__read_seqcount_retry(&mount_lock.seqcount, nd->m_seq))
        return ERR_PTR(-EAGAIN);
    if (__read_seqcount_retry(&rename_lock.seqcount, nd->r_seq))
        return ERR_PTR(-EAGAIN);
}

The comment next to it gives away the intent: if there was a racing rename or mount along the path, the kernel cannot be sure .. has not jumped above the root, so it tells userspace to retry. The logic is right — passing the lookup where it cannot be sure would throw the scope guarantee away.

One inference is being made here: the code checks mount_lock first, and both checks return an indistinguishable EAGAIN, so the measurement says "a sequence counter moved", not "rename_lock moved". Since the noise consists only of rename() calls, and the rename path advances rename_lock by writing to it inside __d_move, the attribution comes from there.

The catch is that rename_lock is a single, global seqlock. It does not care whether the rename had anything to do with my directory; any rename on the system advances the counter. So my hardening flag carries the concurrency of unrelated workloads into my error path.

The error codes do not mean the same thing

The most practical consequence of this measurement is that lumping openat2's error codes into one bucket is wrong. Each has a different owner:

errno	Meaning	Who fixes it
EXDEV	An escape from the scope was caught	Security event — log it, alert
ELOOP	A symbolic or magic link was rejected	Usually configuration, sometimes an attack
EAGAIN (scope)	Concurrency uncertainty	Retry as is; do not raise an alarm
EAGAIN (RESOLVE_CACHED)	The path is not in the lookup cache	Retry without RESOLVE_CACHED
EINVAL	Invalid flag or field	A code bug or a version mismatch
E2BIG	A field the kernel does not know is non-zero	Old kernel, new headers
EFTYPE	OPENAT2_REGULAR set and the path is not a regular file	The caller's expectation (7.2 and later)

The second EAGAIN row is split out deliberately, because the same error code says two different things and the treatments are opposites: under scope uncertainty you repeat the identical call, while under RESOLVE_CACHED repeating the identical call can fail forever — you have to drop the flag.

Log EAGAIN as "I caught an attack" and you will generate false alarms on a busy machine — one open in five, in my setup. The opposite is worse: never retry, and your work fails for no reason whenever your neighbours are busy.

My own preference is to clean .. out of the user-supplied path before the call; the class of EAGAIN measured above disappears at the root with that alone. For the ones that still arrive I keep a bounded retry budget — three to five attempts — and if the budget runs out I reject the request and record the event under a separate counter as "kernel retry budget exhausted". Letting it through would mean admitting an open whose scope guarantee was never verified; rejecting it has to be counted separately so it does not masquerade as an attack. The manual only says the caller may choose to retry, and leaves the decision to you.

So what will the rate be on your machine? My 21% comes from an artificial thread producing thousands of renames a second — not a ceiling, but not typical either. The portable number is this one: in my setup every rename cost between 0.09 and 0.21 EAGAIN. Measure the rename rate on your own server, put your count of ..-bearing opens next to it, and that factor gives you a rough estimate. I did not measure a real rename rate on my own VPS, so I am not claiming a production figure.

Normalising the path also filters out classic path traversal attempts. It is worth adding that it is not free: simplifying .. textually changes meaning in the presence of symbolic links. But together with RESOLVE_BENEATH that worry shrinks, because the kernel is already catching escapes; here normalisation is not a security measure but noise reduction.

So does the hardening actually work?

Do not let the EAGAIN discussion devalue openat2. The headline question got its own run: is the difference between the classic "check first, then open" pattern and openat2 real?

This is a different class from path traversal: a symlink attack that turns on the race between check and use. The setup is an attacker thread swapping /lab/root/upload back and forth between a regular file and a symbolic link to /lab/secret. There are two victims. The first is the classic pattern — fstatat(AT_SYMLINK_NOFOLLOW), then "if it is not a link it is safe", then openat. The second uses openat2 with RESOLVE_NO_SYMLINKS | RESOLVE_BENEATH. I detect a leak from the first byte I read.

I made the window a parameter too, because real code does work between the check and the open: it writes a log line, checks permissions, counts a quota. Runs of 20,000 attempts:

Victim	Window	Successful opens	Leaked	ELOOP	Skipped / ENOENT
check-then-open	0 µs	1,157	0	—	18,806 / 37
openat2	0 µs	2,229	0	3,877	— / 13,894
check-then-open	50 µs	394	121	—	18,961 / 645
openat2	50 µs	5,078	0	3,036	— / 11,886
check-then-open	200 µs	328	90	—	19,128 / 544
openat2	200 µs	5,009	0	3,106	— / 11,885

The last two columns are there so the reader can add up to 20,000. "Skipped" counts the attempts where the classic pattern looked at the fstatat result, saw a link and never tried to open at all; openat2 has no such step, so the same attempts show up there as ENOENT the moment the attacker unlinks the file. That makes the two victims' "successful opens" figures not comparable with each other; the column to read is the leak column.

With a zero window the classic pattern was never caught in 20,000 attempts; if you put nothing between the two calls, winning the race is hard. Insert 50 µs of work and the table turns: of 394 successful opens, 121 — 30.7% — were the attacker's file. On the openat2 side the leak was zero in all three configurations.

The real lesson here is less about the numbers: the safety of the classic pattern depends on the width of the window, and the width of the window depends on what your code happens to be doing. That is not an auditable property. With openat2 there is no window between check and use, because there is no separate check step.

In my own blog infrastructure the same distinction has a counterpart: every path that writes an uploaded file to a location derived from a user-supplied name belongs to this class. I wrote earlier about not being able to write to /tmp even as root; the fs.protected_* settings there reduce the same attack class system-wide in sticky directories. The two are not substitutes: protected_regular puts a blunt guard over the whole system, while RESOLVE_* gives one open full control.

Checklist

If you are taking these flags to production, this is my order of operations:

• There is no wrapper; you will be writing syscall(SYS_openat2, ...), and you will zero struct open_how with a designated initializer.
• Pick one scope flag: BENEATH (stay below dirfd) or IN_ROOT (treat dirfd as the root). Both together is EINVAL.
• Switch magic links off explicitly: BENEATH and IN_ROOT currently disable magic-link resolution too, but the manual says that may change. If that is your intent, write NO_MAGICLINKS.
• Enable NO_SYMLINKS only for a security reason and in a narrow context; it rejects innocent in-scope links as well.
• Clean .. out of user-supplied paths before the call; the class of EAGAIN in the tables above comes from there.
• Put a bounded retry behind EAGAIN and count it separately; do not drop it in the same bucket as EXDEV.
• When probing a new flag, do not look only at EINVAL; verify the side effect you expect on success.
• If you run on older kernels, treat E2BIG separately: it does not mean "I called it wrong", it means "this kernel does not know that field".

Conclusion

openat2 is the modern answer to opening files safely, and the measurements here do not refute it; in the symlink race the classic pattern leaked 121 times out of 20,000 attempts while openat2 leaked nothing. But it does not come for free.

When you move the check out of your own code and into the kernel's path resolution, you also import the uncertainties the kernel has during that resolution. Because rename_lock is global, that uncertainty scales with your neighbours' workload: the busier the machine, the more frequent the EAGAIN. The bill for hardening is not a security hole, it is a more complicated error path.

The right reading, I think, is this: a hardening flag cannot be judged only by what it blocks. It is judged together with the questions "what new error does it produce, who produces that error, and can I tell it apart from the others?" In my setup the answer was a seven-line lookup table; in your code it will be the shape of a catch block.

Official Sources

• openat2(2) manual page — man-pages repository
• include/uapi/linux/openat2.h — RESOLVE_* and OPENAT2_REGULAR definitions
• fs/namei.c — handle_dots() and the seqcount check in scoped resolution
• fs/open.c — flag validation and the OPENAT2_REGULAR translation
• kernel.org release list