DEV Community

Cover image for I Made /usr Writable, systemd-analyze Said 1.5 OK
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

I Made /usr Writable, systemd-analyze Said 1.5 OK

#systemd #security #hardening #linux

systemd-analyze security is the most satisfying toy you can hand a sysadmin. You type the command, your server gets a report card, and the report card ends in little faces. You see :-) and feel good. You see :-{ and your evening is gone.

This piece is about whether to take that report card seriously. Short answer: yes, but only once you know what it measures.

For the long answer I spent an afternoon on VPS3. I took a bare unit file from 9.4 down to 1.1 with 25 directives, measuring the score at every step. Then I added a single line to the hardened unit:

ReadWritePaths=/usr /etc /var
Enter fullscreen mode Exit fullscreen mode

The operating system's binaries, the system configuration and the entire state directory — all three writable. The score did not move. A green check mark was still on screen, next to this sentence:

✓ ProtectSystem=   Service has strict read-only access to the OS file hierarchy
Enter fullscreen mode Exit fullscreen mode

It did not. It could create and write files under /usr, which I went on to verify for real.

First, a report card from the field

All measurements were taken on VPS3, on systemd 255.4-1ubuntu8.17 (Ubuntu 24.04). The other machines in the fleet were unreachable during this session — IP restricted — so the field data below comes from a single server, one carrying real production load. I am not generalising; I am writing down what I saw.

Run with no arguments, systemd-analyze security scans loaded service units. On VPS3, 71 units were scored, and the distribution is this:

48  UNSAFE
10  OK
 8  EXPOSED
 5  MEDIUM
Enter fullscreen mode Exit fullscreen mode

Two thirds "UNSAFE". My first reaction was probably yours too. Then I looked at the list:

containerd.service        9.6 UNSAFE
cron.service              9.6 UNSAFE
docker.service            9.6 UNSAFE
fail2ban.service          9.6 UNSAFE
nginx.service             9.6 UNSAFE
user@0.service            9.8 UNSAFE
kopru-gateway.service     1.7 OK
kopru-signer.service      1.5 OK
kopru-web.service         1.5 OK
systemd-hostnamed.service 1.3 OK
Enter fullscreen mode Exit fullscreen mode

docker.service scores 9.6. The Docker daemon genuinely does need access to everything in order to run containers; give it PrivateDevices=yes and it cannot do its job. user@0.service scores 9.8 — that is root's user session. Root's session having root privileges is not a finding, it is a definition.

nginx.service also scores 9.6. I paused on that one for a second, then looked at the process table:

ps -eo user,comm --no-headers | grep nginx | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode 
     19 nginx    nginx
      1 root     nginx
Enter fullscreen mode Exit fullscreen mode

One root master, nineteen unprivileged workers. The side that faces the internet, that meets the first malicious byte, is those nineteen workers, and none of them is root. nginx has made this split since its very first public release: the 0.1.0 tarball (5 October 2004) ships user nobody; in conf/nginx.conf, and in src/os/unix/ngx_process_cycle.c the worker calls setgid()/setuid() on lines 586 and 595. systemd has no idea, because systemd reads the unit file, not the code. The man page says as much: the analysis covers "only the per-service security features systemd itself implements", and any measures inside the service's own code are not accounted for.

So the first reading of the report card is this: UNSAFE does not mean "this service is dangerous", it means "this unit file has no sandboxing directives". Those are not the same thing, and the gap between them is the whole of this article.

The other end of the list was instructive too. Most of the ten units scoring OK are either systemd's own services (systemd-hostnamed 1.3, systemd-resolved 2.2, systemd-timesyncd 2.1) or my own services that I hardened by hand. The good grades go to the category "somebody sat down and wrote sandboxing directives in this unit file". The score measures attention, not security.

mustafaerbay.service is in that list too, at 8.4 EXPOSED — this blog's old host service. It has been unused since I moved the blog into a container in June:

mustafaerbay.service loaded inactive dead  mustafaerbay.com.tr Astro SSR server
LoadState=loaded
ActiveState=inactive
UnitFileState=disabled
Enter fullscreen mode Exit fullscreen mode

disabled and inactive, but loaded. Because systemd still keeps the unit file in memory, it keeps showing up on the report card. One of the worst lines in your score table turning out to be a service that has not run for months is another side effect of reading the card on its own.

How the number is produced

Reading the source here is more than curiosity, it is necessary. Inside src/analyze/analyze-security.c the arithmetic is one line:

exposure = DIV_ROUND_UP(badness_sum * 100U, weight_sum);
Enter fullscreen mode Exit fullscreen mode

Every directive has a weight, and a "badness" value when it is not satisfied. Total badness is divided by total weight, producing an integer between 0 and 100. On the way to the screen it is divided by ten — 15 inside, 1.5 outside.

That branch on the right is the source of the fourth trap. I will get to it.

What matters is this: the number is a weighted average, not a threat model. It tells you how many of the checks in the table you passed. It counts how many questions you ticked on the exam; it does not look at whether the answers were right.

The ladder: from 9.4 to 1.1

To measure without touching services in production, I used --offline=true. That mode never asks PID 1; it reads the file directly, so you can experiment without starting a service, stopping one, or touching /etc. It has existed since systemd 250 and is made for exactly this kind of measurement.

I started with a bare unit:

[Unit]
Description=lab
[Service]
Type=simple
ExecStart=/usr/bin/node /opt/lab/server.js
Enter fullscreen mode Exit fullscreen mode

Score: 9.4 UNSAFE. Then I added directives one at a time, re-measuring at every step. If you want to repeat the measurement yourself, the whole story is in these two functions:

base=$'[Unit]\nDescription=lab\n[Service]\nType=simple\nExecStart=/usr/bin/node /opt/lab/server.js'

score() {
  printf '%s\n%s\n' "$base" "$1" > /tmp/lab.service
  systemd-analyze security --offline=true --no-pager /tmp/lab.service |
    grep 'Overall exposure' | sed 's/.*: //'
}

score 'NoNewPrivileges=yes'
Enter fullscreen mode Exit fullscreen mode

The abbreviated ladder — each line measured with everything above it already applied:

(bare)                                     9.4 UNSAFE
+ NoNewPrivileges=yes                      9.2 UNSAFE
+ User=lab                                 8.8 EXPOSED
+ PrivateTmp=yes                           8.7 EXPOSED
+ ProtectSystem=strict                     8.5 EXPOSED
+ ProtectHome=yes                          8.3 EXPOSED
+ PrivateDevices=yes                       7.7 EXPOSED
+ ProtectKernelTunables=yes                7.5 EXPOSED
+ ProtectKernelModules=yes                 7.2 MEDIUM
+ ProtectControlGroups=yes                 7.0 MEDIUM
+ RestrictAddressFamilies=AF_INET AF_INET6 6.5 MEDIUM
+ RestrictNamespaces=yes                   5.8 MEDIUM
+ LockPersonality=yes                      5.8 MEDIUM
+ MemoryDenyWriteExecute=yes               5.8 MEDIUM
+ RestrictRealtime=yes                     5.7 MEDIUM
+ RestrictSUIDSGID=yes                     5.5 MEDIUM
+ SystemCallArchitectures=native           5.3 MEDIUM
+ SystemCallFilter=@system-service         4.0 OK
+ CapabilityBoundingSet=                   1.9 OK
+ UMask=0077                               1.9 OK
+ ProtectClock=yes                         1.8 OK
+ ProtectHostname=yes                      1.8 OK
+ ProtectKernelLogs=yes                    1.7 OK
+ PrivateUsers=yes                         1.5 OK
+ ProtectProc=invisible                    1.3 OK
+ IPAddressDeny=any                        1.1 OK
Enter fullscreen mode Exit fullscreen mode

Twenty-five lines, 8.3 points. Sounds like work. It is not — and that is the first trap.

Trap 1: two lines buy most of the score

Because the ladder is cumulative, it hides what each directive is actually worth. I added each one to the bare unit on its own and re-measured:

Directive Score Gain
CapabilityBoundingSet= 6.9 2.5
SystemCallFilter=@system-service 8.0 1.4
RestrictNamespaces=yes 8.7 0.7
PrivateDevices=yes 8.8 0.6
User=lab 9.0 0.4
NoNewPrivileges=yes 9.2 0.2
ProtectSystem=strict 9.2 0.2
PrivateUsers=yes 9.2 0.2
IPAddressDeny=any 9.2 0.2
MemoryDenyWriteExecute=yes 9.4 0.0
LockPersonality=yes 9.4 0.0
UMask=0077 9.4 0.0

One empty CapabilityBoundingSet= line takes a quarter of the scale. That makes sense: the table in the source has 26 separate capability checks, and that single line turns all of them green at once. It is also a good line — most network services need no capabilities at all.

I measured the top two together:

bare                                       9.4 UNSAFE
bare  + CapabilityBoundingSet=
      + SystemCallFilter=@system-service   5.5 MEDIUM
Enter fullscreen mode Exit fullscreen mode

Two lines, 3.9 points. But note: 5.5 is still MEDIUM. You can take almost half the scale in a coffee break, and it does not even move you up one band. The score comes down cheaply, the band does not follow it, and hardening is a different job altogether.

Trap 2: a line worth zero points can kill your service

The last three rows of that table caught my eye. MemoryDenyWriteExecute, LockPersonality, UMask=0077 — none of them moves the number. My first thought was "are they not recognised?" They are, perfectly well:

✓ MemoryDenyWriteExecute=   Service cannot create writable executable memory mappings
✓ LockPersonality=          Service cannot change ABI personality
✓ UMask=                    Files created by service are accessible only by service's own user by default
Enter fullscreen mode Exit fullscreen mode

Three green check marks, zero points. Their weights are small enough that they vanish in the rounding.

Now the real part. MemoryDenyWriteExecute=yes forbids writable-executable memory mappings. That is exactly what every runtime with a JIT compiler needs — the systemd.exec man page says plainly that the directive is incompatible with some runtimes, "including JIT execution engines". I tried it with a transient unit:

systemd-run --collect --wait --pipe --quiet \
  -p MemoryDenyWriteExecute=yes /usr/bin/node -e 'console.log(1+1)'
Enter fullscreen mode Exit fullscreen mode

The output:

#
# Fatal error in , line 0
# Check failed: 12 == (*__errno_location ()).
#
Enter fullscreen mode Exit fullscreen mode

Followed by a twenty-line stack trace from V8's baseline compiler. 2 was never printed. The same command without the directive prints 2 and exits.

The same man page adds one more thing: the protection can be circumvented if the service can write to a filesystem not mounted noexec (such as /dev/shm), or if it can use memfd_create(). So the balance sheet for that line reads: contribution to the score zero, chance of breaking your Node service certain, and the protection it provides bypassable. Copy a "recommended hardening directives" list wholesale out of some document and your report card will not change at all while your service refuses to start.

The reverse holds too: CapabilityBoundingSet=, the line that lowers the score the most, breaks almost nothing in a web service already running without root. There is a relationship between score and risk, but not the one you expect.

An aside: what I expected to break, and didn't

Part of the experiment was where my expectation failed, and skipping that would be dishonest.

The classic warning when writing RestrictAddressFamilies= is this: forget AF_UNIX and AF_NETLINK and name resolution collapses, because the NSS layer talks to a local socket. I was ready to demonstrate it:

systemd-run --collect --wait --pipe --quiet \
  -p 'RestrictAddressFamilies=AF_INET AF_INET6' \
  /usr/bin/getent hosts github.com
Enter fullscreen mode Exit fullscreen mode

I expected empty output. What I got:

140.82.121.3    github.com
Enter fullscreen mode Exit fullscreen mode

It resolved fine. The reason is specific to this machine: on VPS3, resolution goes to systemd-resolved's stub at 127.0.0.53, which is an ordinary UDP AF_INET query. The path that needs a local socket never comes into play. The same directive could well blow up on a machine using nscd, or resolving via LDAP/SSSD.

That was the lesson in itself: the effect of hardening directives depends on the machine. Memorised lists of "this directive breaks that" may not hold on your setup — in either direction.

Trap 3: the place the score never looks

Everything so far is uncomfortable. This part is bad.

I took the ladder at its twentieth step — 1.5 OK, a nice report card — and played with ReadWritePaths=:

A) hardened base (20 directives)     : 1.5 OK :-)
B) + ReadWritePaths=/etc             : 1.5 OK :-)
C) + ReadWritePaths=/usr /etc /var   : 1.5 OK :-)
D) + SupplementaryGroups=docker      : 1.6 OK :-)
Enter fullscreen mode Exit fullscreen mode

Look at row C. I gave the service write access to /usr, /etc and /var — the operating system's binaries, passwd/shadow/sudoers, and the whole state directory — and the score did not change. I repeated it with the 25-directive unit at the bottom of the ladder (1.1); that stayed at 1.1. In both cases the detail table said:

✓ ProtectSystem=   Service has strict read-only access to the OS file hierarchy
Enter fullscreen mode Exit fullscreen mode

"Strict read-only access." With /usr writable. To check this was not a matter of interpretation, I actually tried it:

systemd-run --collect --wait --pipe --quiet \
  -p ProtectSystem=strict -p ReadWritePaths=/usr \
  /bin/sh -c 'echo x > /usr/sdlab-probe && echo WRITABLE && rm -f /usr/sdlab-probe'
Enter fullscreen mode Exit fullscreen mode

WRITABLE. A file was created under /usr, written, then cleaned up. ProtectSystem=strict was in place the whole time. Same for /etc and /var.

Why? Back to the source: in the assessment table inside analyze-security.c there is no row for ReadWritePaths= — not in systemd 255, and not in today's main either. It is never examined. The assessment asks "is ProtectSystem set?"; it does not ask "it is set, but how many holes were punched in it?"

And ReadWritePaths= exists in every hardened unit in real life. It has to — a service needs to write somewhere. My own service on VPS3 has it too:

ReadWritePaths=/opt/kopru/data
StateDirectory=kopru-audit
Enter fullscreen mode Exit fullscreen mode

That is a reasonable hole: a data directory. But the score sees no difference whatsoever between /opt/kopru/data and /usr. Those two units get the same grade. The directive that draws the actual boundary of the sandbox is invisible to the tool that grades the sandbox.

Row D is the same story in different clothing — but here you have to speak carefully, because I got it wrong on my first pass too. Write access to /run/docker.sock is, in practice, root on the host; Docker's own security documentation warns that "only trusted users should be allowed to control your Docker daemon". But ReadWritePaths=/run/docker.sock on its own does not grant that access: the socket is root:docker and 0660, while the service runs as User=lab, so ordinary file permissions still hold the door. The line that actually opens it is SupplementaryGroups=docker. And that is the only place the score moves at all: by 0.1. So the tool does notice a handover of root-equivalent privilege — at one thousandth of the scale.

The man page is honest about one thing, to be fair: a high score "does not mean the service is actually vulnerable". That the converse also holds — that a low score does not mean safe — is what nobody says quite as loudly.

Side note: ReadWritePaths=/ does nothing

On my first attempt I tried opening the hole across the entire root — ReadWritePaths=/. The score stayed at 1.5, which was expected. But testing it at runtime turned up something surprising:

ReadWritePaths=/      /etc  -> READ-ONLY
ReadWritePaths=/      /usr  -> READ-ONLY
ReadWritePaths=/etc   /etc  -> WRITABLE
ReadWritePaths=/usr   /usr  -> WRITABLE
Enter fullscreen mode Exit fullscreen mode

Giving the root on its own quietly achieves nothing; ProtectSystem=strict keeps holding the root mount point read-only. But list the directories individually and every one of them opens. So the line you write meaning "I opened everything" may open nothing, while the line you write meaning "I only opened this one" opens exactly that. Both look identical in the score.

Trap 4: --threshold is not on the scale you think

There is a ready-made gate for anyone wanting this tool in CI: --threshold=N returns an error if the score exceeds the threshold. The man page gives the default as 100.

kopru-web.service scores 1.5. I tried --threshold=5, reasoning that 1.5 is below 5 and should pass. Exit code 1. I raised the threshold; still 1. So I went looking for where it flips:

kopru-web (displayed 1.5)   threshold=14 -> rc=1
kopru-web (displayed 1.5)   threshold=15 -> rc=0
nginx     (displayed 9.6)   threshold=95 -> rc=1
nginx     (displayed 9.6)   threshold=96 -> rc=0
Enter fullscreen mode Exit fullscreen mode

15 and 96. --threshold compares against the internal 0–100 scale, not the 0–10 one on your screen. The code says exposure > threshold, and exposure is already 0–100.

What is genuinely interesting here is that this is not merely an unstated detail. Defining the exposure level, the man page calls it "an estimation in the range 0.0…10.0"; for --threshold it says a custom value "to compare the overall exposure level with". So the documentation states that the threshold is compared against a value on the 0.0–10.0 scale it just defined. The code uses 0–100. That is a documentation-versus-code mismatch.

The practical result: a CI gate written as --threshold=5 meaning "reject anything above 5.0" actually sets a threshold of 0.5, which no service on the planet will pass. You want --threshold=50. And the default of 100 means 10.0, so it never fails — which also explains why the gate stays green when you leave --threshold out.

So what is this tool good for

Do not close the tab thinking "rubbish, then". I think the opposite: it is a good tool, just not for the job it seems to advertise.

For taking inventory it is excellent. In under a minute, systemd-analyze security lists which units have never been sandboxed. Not all 48 UNSAFE units on VPS3 are dangerous, but every one of them means "nobody has touched this unit file". The difference between the nginx.service your distribution packages and the app.service you wrote yourself shows up right here: one has twenty years of security architecture behind it, the other has six lines written in a hurry last Wednesday. The score marks both 9.6; the one you should care about is the second.

It also teaches well. The per-unit detail table explains every sandboxing directive systemd offers, in plain language, on one screen, far faster than reading systemd.exec end to end. And with --offline=true you can review packaged unit files, images, and services that are not installed yet, without running anything. Every score in this article was measured that way; not a single service on VPS3 was restarted.

If you want it as a CI gate, --threshold=50 is the crude version. There is a better one, and it is barely known: --security-policy=PATH lets you supply your own JSON policy — you set a weight and range per test, and weight: 0 disables a test entirely. So you can zero out MDWX, weight User= and CapabilityBoundingSet= heavily, and turn the score into your own threat model. If you want machine-readable output there is --json=pretty as well. This is the official answer to the tool's "it measures the wrong things" problem, and almost nobody uses it.

Here is what it is not good for: telling you whether a service is secure. For that you have to read the unit file — especially the lines the score never looks at.

An order for actually reviewing a unit

This is the order I settled on. It does not start with the score, and it does not end with it.

  1. Who does the service run as? If there is no User=, the rest is detail. Getting off root is worth 0.4 points, and its real effect is larger than everything else on the list.
  2. What does ReadWritePaths= say? It is the one line the score cannot see, which makes it the line that most needs a human eye. To sweep a fleet: systemctl show --property=Id,ReadWritePaths,BindPaths '*.service' | grep -B1 'ReadWritePaths=.\+'. Can you justify why each path is there? If you cannot, that path is surplus.
  3. Let systemd punch the hole instead of doing it by hand. Rather than writing ReadWritePaths=/var/lib/app, use StateDirectory=app; systemd creates the directory with the right ownership, adds the path itself, and cleans up when the service is removed. LogsDirectory= and CacheDirectory= do the same job in their own areas. The hole stays narrow and stops being hand-managed.
  4. Which sockets can the service reach? The Docker socket, the D-Bus system bus, some management API. The man page states outright that privileges obtained over IPC fall outside the scope of the analysis. Count SupplementaryGroups= in here as well.
  5. Add the cheap, harmless ones: NoNewPrivileges=yes, CapabilityBoundingSet=, PrivateTmp=yes, ProtectSystem=strict, ProtectHome=yes. These break nothing in most network services, and they already bring in the bulk of the score.
  6. Try the runtime-dependent ones one at a time: MemoryDenyWriteExecute, SystemCallFilter, RestrictAddressFamilies, PrivateUsers. Here systemd-run --collect --wait --pipe is your best friend: it gives you a one-shot unit that leaves no trace behind. To find out whether a directive breaks your service you do not have to edit the unit file, run daemon-reload, restart the service and then undo it all — you write one line and get the answer. That is how I learned MDWX kills Node, not in production.
  7. Read the score last, and read it only as "compared to last month". Every number here was measured on systemd 255; the assessment table changes between versions, new checks get added, and when weights move, scores move. The same unit scoring differently on different distributions is normal, so make sure the two numbers you are comparing came from the same systemd version.

I have written about the sandboxing directives themselves before, as a phased model, in Hardening systemd Services with Sandboxing; that piece answers "which directive, in what order". This one questions the report card laid on top of that model. How to get secrets out of the unit file is a separate subject and lives over in systemd Credentials — another thing the score does not measure.

Closing

Things get strange when measuring takes the place of understanding. Any tool that can produce a number turns, over time, into a game of improving that number; and the game is usually winnable, because the formula behind the number is public.

Here the formula really is public: badness_sum * 100 / weight_sum. Someone who knows it can take 3.9 points with two lines, leave /usr writable and stay at 1.5, watch the screen say "strict read-only", and not have lied at any point. The tool is not lying either; it simply is not answering a question nobody asked it.

Use systemd-analyze security as an index, not as an exam. It tells you where to look.

Official Sources

Top comments (0)