DEV Community

Cover image for I Moved the Binary Twelve Ways, Six Kept the File Capability
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

I Moved the Binary Twelve Ways, Six Kept the File Capability

#linux #capabilities #security #docker

"Don't use setuid, grant a capability" is good advice. Giving a program the whole of root just so it can open a raw socket, when cap_net_raw alone would do, trades the forty-one capabilities root carries (the kernel's cap_last_cap is 40, i.e. 0-40) for one. I agree with the advice and I follow it myself.

The problem isn't the advice — it's what the advice leaves out. The setuid bit is part of the file mode: ls -l shows it, cp carries it, tar packs it, everyone knows about it. A file capability lives in an extended attribute called security.capability — and extended attributes are quietly dropped by default in most of the tools that move a file from one place to another. So granting the capability correctly isn't the end of the job. The job ends when the file actually arrives at its destination still carrying it.

That is what today's lab run shows. The same binary went through twelve different transfer routes; six arrived with the capability, six silently left it behind. Along the way the capability model showed four more sharp edges: a case where getcap tells the truth and the kernel still hands out nothing, a one-letter typo that makes a binary unrunnable even for root, the same typo turning a loud failure into silent nothing when you change that letter, and the fact that the capability cannot be written at all into a directory mounted from my Mac.

Everything here comes from commands run today. The rig: Docker Desktop 4.37.2 (Docker Engine 27.4.0, Buildx v0.19.2) on my Mac, linuxkit kernel 6.10.14, debian:trixie-slim inside it (Debian 13.7), libcap2-bin 1:2.75-10+deb13u1+b3, tcpdump 4.99.5-2, coreutils 9.7-3, rsync 3.5.0, git 2.47.3.

Source verification was done against security/commoncap.c in Linux 7.3, while the measurements ran on 6.10.14. The two were compared directly: the functions quoted below (bprm_caps_from_vfs_caps and get_file_caps) are byte-for-byte identical in v6.10 and v7.3, so the code quotes apply to the kernel under test as well.

First, an inventory: nobody is using this model

Curiosity first: in a clean Debian image, how many files are setuid and how many carry a capability?

find / -xdev -type f -perm -4000 2>/dev/null | wc -l
getcap -r / 2>/dev/null | wc -l
Enter fullscreen mode Exit fullscreen mode

The answer is lopsided. Eight setuid files:

/usr/bin/chsh
/usr/bin/chfn
/usr/bin/umount
/usr/bin/mount
/usr/bin/newgrp
/usr/bin/passwd
/usr/bin/gpasswd
/usr/bin/su
Enter fullscreen mode Exit fullscreen mode

Files carrying a capability: zero.

That is surprising, because neither mount nor passwd genuinely needs full root, and the model is hardly new — the attribute has been around long enough to go through three revisions (capability.h defines VFS_CAP_REVISION_1, _2 and _3). But the more interesting half of the story is ping. For years ping was the showcase example: once setuid, later migrated to cap_net_raw=ep. Looking at it now:

-rwxr-xr-x 1 root root 213448 Apr 22  2025 /usr/bin/ping
Enter fullscreen mode Exit fullscreen mode

No setuid bit, and getcap prints nothing. Debian has taken ping out of the capability model too. The package changelog says why; the iputils 3:20240905-1 entry (24 September 2024) reads: "ping: Add Recommends on linux-sysctl-defaults and stop setting additional capabilities". Two releases earlier, in 3:20240117-1, the setuid path was closed: "Stop falling back to setuid binaries for raw socket access".

The same changelog, in the same release, also contains the opposite move: "Set cap_sys_nice in iputils-clockdiff". So Debian isn't abandoning capabilities wholesale — it is taking one away from ping and giving one to clockdiff. Capabilities aren't the wrong tool; they became an unnecessary tool for ping.

What replaced it is decided by a sysctl:

ping_group_range: 0 2147483647
Enter fullscreen mode Exit fullscreen mode

One caveat is owed here: that value was read inside a container, and as such it is Docker's contribution rather than Debian's — the debian:trixie-slim image does not install linux-sysctl-defaults, and the range is written into the container by Docker's own default. On a real Debian 13 install the same result is reached through the linux-sysctl-defaults package the changelog recommends. Different route, same destination: ping no longer opens a raw socket at all — it opens an unprivileged ICMP datagram socket, and that range decides who may.

The showcase example has been taken out of the showcase, because the best capability is the one you never need. That's a useful frame for the rest of this article: there are ways to carry a capability correctly, but not carrying one is always cheaper.

So the experiment here rests not on ping but on a program that genuinely needs a raw socket: tcpdump. I wrote about the tool itself in the production packet-capture runbook; here I'm dealing with its privileges.

Granting the capability: three lines, three different outcomes

Copy tcpdump, leave its mode at 755, run it as an ordinary user:

cp /usr/bin/tcpdump /opt/td
su deneme -c "/opt/td -i lo -c 1 -w /dev/null"
Enter fullscreen mode Exit fullscreen mode 
td: lo: You don't have permission to perform this capture on that device
(socket: Operation not permitted)
Enter fullscreen mode Exit fullscreen mode

Exit code 1. As expected. Now the capability. Since -c 1 waits for the first packet, and lo sits idle in a fresh container, traffic has to be generated first:

setcap cap_net_raw+ep /opt/td
(ping -c 3 -i 0.3 127.0.0.1 >/dev/null 2>&1 &) ; sleep 0.2
su deneme -c "/opt/td -i lo -c 1 -w /dev/null"
Enter fullscreen mode Exit fullscreen mode 
td: listening on lo, link-type EN10MB (Ethernet), snapshot length 262144 bytes
1 packet captured
8 packets received by filter
0 packets dropped by kernel
Enter fullscreen mode Exit fullscreen mode

The file is still owned by root, its mode is still -rwxr-xr-x, there is no setuid bit — and an ordinary user is capturing packets. The model worked exactly as advertised.

Then +p instead of +ep. One letter:

setcap cap_net_raw+p /opt/td
Enter fullscreen mode Exit fullscreen mode 
td: lo: You don't have permission to perform this capture on that device
(socket: Operation not permitted)
Enter fullscreen mode Exit fullscreen mode

Back to square one. Meanwhile getcap happily confirms there is a capability on the file: cap_net_raw=p. That e is the "effective" bit, and it means: should the capability go into the process's effective set, or will the program raise it itself? tcpdump is not capability-aware — it doesn't call capset to promote something from its permitted set into its effective set. So with +p it receives the capability and cannot use it.

To see this from the process side, three copies of capsh marked three different ways, each printing its own sets:

marking on the file process's Current: line
no capability =
cap_net_raw=p cap_net_raw=p
cap_net_raw=ep cap_net_raw=ep

The middle row is the trap itself: the capability is there, in the process's hands, and not effective. My rule became this — if the program you're marking doesn't call capset, write +ep, and don't go hunting for why +p didn't work.

Where the capability actually lives

To see where the difference between +ep and +p is stored, look straight at the attribute:

getfattr -d -m - /opt/td
Enter fullscreen mode Exit fullscreen mode 
# file: opt/td
security.capability=0sAQAAAgAgAAAAAAAAAAAAAAAAAAA=
Enter fullscreen mode Exit fullscreen mode

Twenty bytes. Decoded, the structure is entirely legible:

raw hex:   0100000200200000000000000000000000000000
magic_etc = 0x02000001
  revision = 0x02000000   effective bit = 1
permitted = 0x00002000 -> bit set: 13
Enter fullscreen mode Exit fullscreen mode

Twenty bytes matches the kernel's version-2 layout. The capability.h definition reads:

#define VFS_CAP_U32_2           2
#define XATTR_CAPS_SZ_2         (sizeof(__le32)*(1 + 2*VFS_CAP_U32_2))
Enter fullscreen mode Exit fullscreen mode

With VFS_CAP_U32_2 equal to 2, that is 4 × (1 + 4) = 20. Version 3 would be 24 bytes, because XATTR_CAPS_SZ_3 counts one more __le32: the rootid field. That field isn't decoration — inside a user namespace it records which mapped root the capability belongs to, which is what rootless containers need. If you run rootless Docker or Podman, you'll meet v3 rather than v2.

The effective bit, meanwhile, is not a separate set but the lowest bit of the magic_etc field; the entire difference between +ep and +p is that one bit. Only bit 13 is set in the permitted set, and capability.h says #define CAP_NET_RAW 13. The capability granted is where it was expected.

Worth pausing on: a twenty-byte attribute shows up neither in the file size nor in ls -l output. That invisibility is the reason this whole article exists.

Twelve ways to move it, six survivors

The capability is granted and working. Now the real-world question: if the file moves somewhere, does the capability come along? The same binary went through twelve different transfer routes, with getcap checked at the destination each time.

transfer method capability arrived
cp no
cp -p no
cp -a yes — but silently (see below)
cp --preserve=xattr yes
install -m755 no
tar cf + tar xf no
tar --xattrs --xattrs-include='*' (both ends) yes
rsync -a no
rsync -aX yes
mv (across filesystems) yes
git add + git checkout no
Dockerfile COPY --from yes

Six of twelve kept it. Five rows stand out.

The cp -p row is the one that stings. We read -p as "preserve", but cp --help defines it precisely: -p = --preserve=mode,ownership,timestamps. Extended attributes aren't on that list. Assuming -p is enough is the shortest path to shipping the mistake to production without noticing.

The cp -a row says "yes" but deserves an asterisk, because the coreutils manual describes -a as trying to preserve extended attributes but "ignore any failure to do that and print no corresponding diagnostic". --preserve=xattr is the opposite: "Preserve extended attributes of the file, or fail with full diagnostics." On this rig both carried the capability. But in an environment where CAP_SETFCAP has been dropped, or onto a destination filesystem that can't hold the attribute, cp -a drops the capability with exit code 0 and without a word. It would be odd for an article whose thesis is "the system doesn't tell you" to recommend the flag that is documented not to tell you: in automation, write --preserve=xattr.

The install -m755 row is sneakier, because install is the final step of nearly every Makefile. Here the measurement simply matches the documentation, which states it in one line: "install never preserves extended attributes (xattr)." This isn't a side effect, it's by design. A flow that runs setcap during the build and then puts the file in /usr/local/bin with make install leaves the capability behind every single time. The correct order is the reverse: install first, then setcap the installed file.

The rsync -a row will bite hardest in practice, because rsync is the deployment transport in most places and -a has become a reflex. -a isn't enough here either; -X (--xattrs) is required. The same reflex breaks with tar, and there the measured detail is the one that made me think hardest. Comparing the two archives:

plain.tar size: 1331200
xattr.tar size: 1331200
Enter fullscreen mode Exit fullscreen mode

Identical. The twenty-byte attribute disappears into the archive's block padding. So you cannot answer "does my archive carry the attribute" by looking at file size; you have to unpack it and check with getcap. It also matters that --xattrs is required on both ends: specify it only when packing and the archive will contain the attribute but extraction will drop it.

The git row isn't a shortcoming but a structural fact: git stores no extended attributes. A file that is cap_net_raw=ep before the commit comes back bare after git checkout. The practical consequence is that a capability can never live in a repository. That makes the "install first, then setcap" rule a requirement rather than a suggestion. In production the place this happens is the packaging step: on the .deb side setcap is called from postinst (exactly what Debian does for clockdiff), and on the RPM side the %caps directive does it.

The mv row stands slightly apart as the only true move: within one filesystem mv is a rename, so the attribute simply stays with the file. Across filesystems it genuinely copies, and the documentation reads: "mv always tries to copy extended attributes (xattr)… Upon failure all warnings are emitted, but do not modify the exit status." So mv warns but still exits 0. That yields a three-point spectrum: --preserve=xattr errors on failure, mv warns but doesn't change the exit status, cp -a says nothing at all.

The last row refuted my expectation. COPY --from inside a Dockerfile preserves the capability — I assumed it dropped it, and measuring showed the opposite:

#5 3.439 YAPICI KATMANDA:
#5 3.440 /td cap_net_raw=ep
#8 0.087 COPY --from SONRASI:
#8 0.088 /td-copy cap_net_raw=ep
Enter fullscreen mode Exit fullscreen mode

(The labels in that log are the Turkish echo strings from my Dockerfile: "in the builder layer" and "after COPY --from". The output is quoted verbatim.)

In a multi-stage build, running setcap in the builder stage and carrying the binary into the final image with COPY --from works. I went looking for this in Docker's documentation and couldn't find it: COPY is documented as preserving "file permissions" and says nothing about extended attributes. So the basis for that row isn't documentation, it's the build log above — and undocumented behaviour is behaviour that can change with a version. Measure it once on your own version; this one came out on Buildx v0.19.2.

I can't say the same for an ordinary COPY from the build context, though, because on my Mac a capability-carrying file cannot be put into the context at all. The reason comes a little further down.

getcap is telling the truth, and the kernel grants nothing

Now the measurement with the strongest smell of a trap. Two tmpfs filesystems — one normal, one nosuid — with the same binary and the same capability on both.

mount -t tmpfs -o nosuid tmpfs /mnt/nosuid
mount -t tmpfs tmpfs /mnt/suid
Enter fullscreen mode Exit fullscreen mode 
--- /mnt/suid    on file: cap_net_raw=ep   -> Current: cap_net_raw=ep
--- /mnt/nosuid  on file: cap_net_raw=ep   -> Current: =
Enter fullscreen mode Exit fullscreen mode

Note that on the nosuid side getcap still reports cap_net_raw=ep on the file. The attribute really is there; getcap isn't lying. But the process receives nothing. Before reading the attribute the kernel looks at the mount, and if it's nosuid it skips file capabilities entirely. The code is in get_file_caps, two lines:

if (!mnt_may_suid(file->f_path.mnt))
    return 0;
Enter fullscreen mode Exit fullscreen mode

The auditing consequence is this: getcap alone is not sufficient as an audit tool. It answers "which files carry a capability"; it does not answer "which files can actually use one". For the second question you also have to look at mount options — and /tmp, /dev/shm, many /home mounts and most mounts inside containers are already nosuid. Just above those two lines there's also a file_caps_enabled check; file capabilities can be switched off wholesale at kernel build or boot time.

The place this rule bit me directly was the bind-mounted directory from my Mac. Docker Desktop shows the mount like this:

fakeowner rw,nosuid,nodev,relatime,fakeowner
Enter fullscreen mode Exit fullscreen mode

Trying setcap there simply didn't work:

Failed to set capabilities on file 'setcap': Operation not supported
Enter fullscreen mode Exit fullscreen mode

Exit code 1. So anyone bind-mounting a directory from their Mac and trying to grant a capability to a binary inside it hits two walls at once: the filesystem can't hold the security.capability attribute at all, and the mount is nosuid anyway. The same limit applies to NFS and to many read-only image layers; the issue isn't macOS, it's whether the filesystem can hold the attribute. The practical upshot is clear — a capability-carrying binary has to live inside the container image, not on a mounted volume. (The error message naming the file 'setcap' is a quirk of libcap 2.75's own wording; it doesn't change the measurement, but it did sidetrack me for a minute the first time I saw it.)

Writing the capability needs its own permission, too. In a container with CAP_SETFCAP dropped, setcap fails:

unable to set CAP_SETFCAP effective capability: Operation not permitted
Enter fullscreen mode Exit fullscreen mode

And that came back as uid 0 — being root inside the container is not enough to write a file capability. On the kernel side the corresponding check is capable_wrt_inode_uidgid(idmap, inode, CAP_SETFCAP) inside cap_convert_nscap.

The bounding-set ceiling: a wrong capability locks the binary

This is the finding with the most potential to hurt. Granting the same binary a capability the container does not have:

setcap cap_sys_time+ep /opt/t
Enter fullscreen mode Exit fullscreen mode

The result isn't the program failing — it's the program never starting:

bash: line 1: /opt/t: Operation not permitted
Enter fullscreen mode Exit fullscreen mode

The exit code the shell reports is 126. Tried as root too: same error, same 126. So a single wrong setcap call renders a working binary unrunnable for everyone in that container. You can't even get --version out of it.

A note on that number: 126 is the shell's language, not the kernel's. The execve() call returns EPERM; it's bash that renders "found but not executable" as 126. The same failure appears as 203/EXEC under systemd, and as plain errno == EPERM in a program that calls execve() directly. When you write a monitoring rule, the thing to match is not a single number.

The cause is the container's capability bounding set. The CapBnd value in /proc/self/status is 00000000a80425fb, which decodes to 14 capabilities:

cap_chown, cap_dac_override, cap_fowner, cap_fsetid, cap_kill,
cap_setgid, cap_setuid, cap_setpcap, cap_net_bind_service,
cap_net_raw, cap_sys_chroot, cap_mknod, cap_audit_write, cap_setfcap
Enter fullscreen mode Exit fullscreen mode

That set of 14 is Docker's default; it matches the DefaultCapabilities list in the moby source exactly, and cap_sys_time is not in it. (A note on naming: "bounding set" is my term — Docker's documentation only speaks of capabilities being "kept" or "dropped" and never says which set it means; the 14 were read from CapBnd directly.)

The kernel states plainly what it does here. The formula in bprm_caps_from_vfs_caps and the check right after it:

/*
 * pP' = (X & fP) | (pI & fI)
 * The addition of pA' is handled later.
 */
new->cap_permitted.val =
    (new->cap_bset.val & caps->permitted.val) |
    (new->cap_inheritable.val & caps->inheritable.val);

if (caps->permitted.val & ~new->cap_permitted.val)
    /* insufficient to execute correctly */
    ret = -EPERM;
Enter fullscreen mode Exit fullscreen mode

X in the formula is the bounding set. The file wants cap_sys_time, the bounding set doesn't have it, the intersection comes out empty — and when a bit the file asked for is missing from the result, the kernel returns -EPERM. The comment states the intent: a program running with a capability missing would do the wrong thing, so the kernel chooses not to start it at all.

The next line of that code predicted something that hadn't been measured yet:

return *effective ? ret : 0;
Enter fullscreen mode Exit fullscreen mode

EPERM is returned only if the effective bit is set. So granting the same wrong capability with +p should let the program start. Setting up that experiment:

marking on the file exit code process's Current: line
cap_sys_time+ep 126, Operation not permitted (never ran)
cap_sys_time+p 0 =

The prediction held exactly. With +p the program starts fine — and receives no capability at all. The choice between them is a choice between a loud failure and silent nothing. Which is worse depends on your setup: your monitoring will catch the 126 from +ep immediately, whereas the silence of +p only surfaces when the program tries to open a raw socket and fails, probably in the middle of the night.

The same experiment again with one change — giving the container the capability:

docker run --cap-add SYS_TIME ...
Enter fullscreen mode Exit fullscreen mode

The bounding set went from 14 to 15, cap_sys_time joined it, and the binary marked cap_sys_time+ep ran with exit code 0. Same file, same setcap, same kernel; the only variable was the bounding set. That's the cause.

How the decision flows at execve

Every behaviour recorded above sits on a single decision chain:

Diagram

The order in that diagram is itself a claim: the nosuid check comes before the bounding-set arithmetic. That's why granting a wrong capability on a nosuid mount produces no EPERM — the kernel skipped the attribute before it ever got there.

Secure-exec: you get the side effect too

The last measurement is a behaviour a file capability inherits whether you want it or not. Two binaries, one with a capability and one bare, both asked to preload a nonexistent library:

su deneme -c "LD_PRELOAD=/yok/kitaplik.so /opt/ciplak --version"
su deneme -c "LD_PRELOAD=/yok/kitaplik.so /opt/td --version"
Enter fullscreen mode Exit fullscreen mode

The bare binary complained:

ERROR: ld.so: object '/yok/kitaplik.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Enter fullscreen mode Exit fullscreen mode

The capability-carrying binary said nothing at all; it printed its version line straight away. Because an exec that gains a capability triggers the kernel's secure-execution mode, and in that mode the dynamic loader silently ignores variables like LD_PRELOAD and LD_LIBRARY_PATH.

The su deneme part is essential, because the condition is not merely "a capability was gained". The assignment in commoncap.c reads:

(!__is_real(root_uid, new) &&
 (effective ||
  __cap_grew(permitted, ambient, new))))
    bprm->secureexec = 1;
Enter fullscreen mode Exit fullscreen mode

!__is_real(root_uid, new) — a caller whose real uid is not root. Run the same command as root and secureexec stays 0, LD_PRELOAD is honoured, and the capability-carrying binary prints the very same ld.so error. A file capability does not put root into secure-exec. (The capability-related part of this condition is identical in v6.10 and v7.3; the setuid checks around it changed.)

Security-wise this is what we want: a privileged binary should not accept injected code from outside. But it has a side effect that can bite you in practice: if you have a wrapper script that wraps the binary with LD_PRELOAD or a custom LD_LIBRARY_PATH, the day you apply setcap that wrapper quietly stops mattering. It doesn't error. It just no longer does anything.

Two ways to avoid granting it at all

Before the checklist, a reminder of the frame from the opening: the best capability is the one you don't grant. If you're writing a service you may not need setcap at all, because systemd can hand the capability to the process rather than the file — via AmbientCapabilities= and CapabilityBoundingSet=. The binary then stays bare, the attribute-transport problem disappears entirely, and the twelve-row table above stops being your concern. I covered the neighbours of those settings on the systemd service hardening side.

There's a warning in the same place: setting NoNewPrivileges=yes silently disables file capabilities — the container equivalents being --security-opt no-new-privileges and Kubernetes' allowPrivilegeEscalation: false. So your hardening setting can cancel your setcap solution without telling you. That is the last of the silent failures on this list.

The second route is replacing the capability with a different mechanism altogether. ping moving to ping_group_range is the example. Similarly, binding to a privileged port can use net.ipv4.ip_unprivileged_port_start or socket activation instead of cap_net_bind_service. In the article on ptrace_scope I noted the reverse direction: some capabilities (cap_sys_ptrace among them) are in practice aliases for root, and granting them is no safer than setuid.

Implementation checklist

The order these measurements produced, and what I apply in my own setups:

  1. First ask whether you need the capability at all. If systemd's AmbientCapabilities=, a suitable sysctl or socket activation will do, don't touch the file.
  2. Always grant the capability after installation. install never carries the attribute by definition and git never stores it, so a capability can only be granted at the packaging/installation step (.deb postinst, RPM %caps).
  3. In automation, write --preserve=xattr rather than cp -a, and rsync -aX rather than rsync -a. The first errors on failure; the second adds the attribute -a skips.
  4. Choose +ep or +p by looking at the program. If it doesn't call capset, use +ep. +p is meaningful for capability-aware programs; otherwise it hands you silent nothing.
  5. Verify the capability is in the bounding set before you grant it. Check with capsh --decode=$(grep CapBnd /proc/self/status | awk '{print $2}'); writing one that isn't there with +ep makes the binary unrunnable.
  6. Check that the mount the file lives on isn't nosuid. findmnt -no OPTIONS <path>. A positive answer from getcap does not replace this check. Scan for NoNewPrivileges / no-new-privileges with the same eye.
  7. In containers, keep the capability-carrying binary inside the image. Mounted volumes are nosuid, and macOS mounts and NFS can't hold the attribute at all. Running rootless, expect the v3 attribute layout.
  8. Audit with two questions: which files have the attribute (getcap -r /) and whether those files sit on a non-nosuid mount. The first question alone misleads.
  9. Measure your backup and restore path once. Is --xattrs on both ends of your tar, does your rsync get -X, does your image build carry the attribute — file size won't tell you, getcap will.

Conclusion

File capabilities are better than setuid; that still holds after measuring. But the difference between the two models isn't only the breadth of privilege granted — it's visibility. The setuid bit lives in the file mode, and every tool knows and carries the mode. A file capability lives in an extended attribute, and extended attributes are a data type that falls off by default all along the plumbing: dropped by cp, dropped by install by design, dropped by tar and rsync unless you ask for a flag, never held by git, present but inert on a nosuid mount, unwritable on a directory mounted from a Mac.

Today's real lesson: in the capability model, the mistake rarely arrives as "I granted the wrong capability". It arrives as "I granted the right capability, then moved the file". And outside two edge cases — the binary that refuses to run and the silence of ld.so — the system doesn't tell you. Six of twelve transfer routes quietly leaving the capability behind shows that the discipline isn't in granting it, it's in everything after.

So the question to ask about your own setup: how many transfer steps did your capability-carrying binary take to reach where it sits in production today, and which of those steps was verified with getcap?

Official Sources

Top comments (0)