Most server break-ins don't start as root. They start with one ordinary account: a web app user, a CI deploy user, a leaked SSH key for deploy. What happens next depends on one question: how far is it from that account to root?
On a lot of servers the answer is "one command". Not because of a clever kernel exploit, but because of ordinary setup choices that made sense at the time.
Here are the ones I see most, how to check each by hand, and what to do about it.
1. A sudo rule without a password
deploy ALL=(ALL) NOPASSWD: ALL
Many cloud images add exactly this line for their default user (ubuntu, ec2-user, admin). Anyone who gets that account, through a stolen key or a hole in an app running as it, is root with sudo -i. No password needed.
Check:
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d/
Fix: remove NOPASSWD for accounts that log in remotely, or limit the rule to the few commands that really need it.
2. A sudo rule for "just one program"
deploy ALL=(root) NOPASSWD: /usr/bin/vim /etc/nginx/nginx.conf
This looks narrow. It isn't. vim can run a shell (:!sh), and that shell is root. The same goes for less, find, python3, tar, awk, cp and many others: if the program can start a shell or write any file, the rule gives full root.
Check: read the rules for each account and ask, for every program listed, "can this run a command or write anywhere?"
sudo -l -U deploy # run as root, or as an admin with sudo
Fix: use sudoedit for editing files. It edits a copy as the user and writes it back, so no editor runs as root.
3. The docker, lxd or disk group
getent group docker lxd disk
Being in the docker group means you can start a container that mounts / from the host, which is root in all but name. lxd is the same. disk gives raw read and write access to the disks.
Fix: treat these groups like sudo. Only admins belong in them. For rootless containers, look at rootless Docker or Podman.
4. A container socket other accounts can use
ls -l /var/run/docker.sock /run/podman/podman.sock /run/containerd/containerd.sock 2>/dev/null
If the socket is writable by everyone, or by a group many accounts are in, every one of those accounts can do what root can, through the container engine.
Fix: the socket should be root:docker and srw-rw----, and see point 3 for who is in docker.
5. A file root runs that someone else can change
Root runs scripts from cron (/etc/crontab, /etc/cron.d, /etc/cron.daily) and services from systemd. If another account can edit one of those scripts, or the folder it's in, it can add a line and wait.
Check: for each script root runs, look at the file and its folders:
ls -l /etc/cron.daily/
namei -l /opt/backup/run.sh
namei -l shows the permissions of every folder along the path. A writable /opt/backup is enough, even if run.sh itself is locked down: the account can replace the file.
Fix: scripts root runs, and every folder above them, should be owned by root and not writable by anyone else.
6. A writable folder in root's PATH
If root's PATH includes a folder another account can write to, that account can put a program named ls or tar there. The next time root, or a root cron job, runs that command, it runs theirs.
Check:
sudo grep secure_path /etc/sudoers
for d in $(echo "$PATH" | tr : ' '); do ls -ld "$d"; done
(Run the second line as root, so you see root's PATH.)
Fix: remove such folders from PATH, or make them root-only.
7. A setuid program that opens a shell
sudo, su and passwd are setuid root on purpose. find, python3 or bash should never be, but sometimes are, left over from a quick fix years ago.
find /usr/bin /usr/sbin /bin /sbin -perm -4000 -user root 2>/dev/null
Fix: chmod u-s on anything that isn't meant to be setuid.
8. File capabilities that give root
Capabilities split root into pieces. Some pieces are root again: cap_setuid lets a program become any user, cap_dac_override lets it read and write any file.
getcap -r /usr/bin /usr/sbin /usr/local/bin 2>/dev/null
Fix: remove capabilities that a program doesn't need (setcap -r), especially on interpreters such as python3 or perl.
Bonus: secrets the wrong people can read
Becoming root isn't the only goal. A readable ~/.ssh/id_ed25519, ~/.aws/credentials, ~/.kube/config or an app's .env file can be worth more than root on that one machine, because it opens other machines.
sudo find /home /root /srv /opt /var/www -maxdepth 4 \
\( -name 'id_*' ! -name '*.pub' -o -name credentials -o -name .env -o -name config.json -o -name .git-credentials \) \
-perm /o+r 2>/dev/null
And one more: if the same SSH key is trusted in authorized_keys on many machines, one laptop, CI secret or old backup with that key opens all of them at once.
Doing this on every server, every week
Checking one server by hand takes ten minutes. Checking twenty, again after every change, doesn't happen. That's why I added an "attack paths" check to DeaconGuard, a free, MIT-licensed security scanner you host yourself.
It runs all of the checks above on each machine and shows each finding as a chain, for example:
deploy > sudo rule without a password > root
every account > reads /home/deploy/.ssh/id_rsa > logs in where deploy's key is trusted
Each finding has the evidence (the sudo line, the file and its permissions) and what to do about it. A few things it is careful about:
- It only reads. It never runs an exploit or tries a password. For secrets it reads only the path, owner and permissions, never the contents.
- SSH keys are compared by fingerprint. The agent sends the server each key's SHA256 fingerprint, never the key, and the server spots keys trusted on three or more machines.
- It's light. The check is three short shell scripts at the lowest CPU priority; on a test machine it took about a tenth of a second.
Try it on a spare Linux box:
curl -fsSL https://get.deaconguard.io | sudo sh -
It needs root, verifies the release's signature before installing, and scans its own server straight away. Docs for this check: docs.deaconguard.io/usage/attack-paths.
If you know a path to root that's common on real servers and isn't on this list, tell me in the comments. I'd like to add it.
Top comments (0)