DEV Community

Vainamoinen | Pulsed Media
Vainamoinen | Pulsed Media

Posted on

Why systemctl --user daemon-reload silently does nothing

#linux #systemd #devops #sysadmin

You drop a .service file into ~/.config/systemd/user/, run systemctl --user daemon-reload, and nothing happens. No error, no unit, no clue. On a shared Linux host this is one of the most confusing failure modes there is, because the command that should surface the problem is exactly the command that stays quiet.

We run shared Linux hosts for a living (seedbox and storage hosting out of our own datacenter in Finland), so we see this one often. The symptom is always the same: a user swears their unit file is correct, and it is, but systemctl --user acts like the file does not exist. The cause is almost never the file.

The symptom: a reload that reloads nothing

Here is what it looks like from the user's seat:

$ ls ~/.config/systemd/user/
myapp.service

$ systemctl --user daemon-reload
$ systemctl --user start myapp
Failed to start myapp.service: Unit myapp.service not found.
Enter fullscreen mode Exit fullscreen mode

The reload returned 0. It printed nothing. And the unit still is not there. If you have ever lost an hour to this, you already know the trap: you keep re-reading the unit file, because the tool is telling you, by saying nothing, that the file is the problem. It usually is not.

Why it happens: the per-user manager can die

Every logged-in (or lingering) user on a systemd host gets their own private systemd instance. It runs as a system-level unit named user@<uid>.service, and it is the thing that owns systemctl --user. When you run systemctl --user daemon-reload, you are really talking to your user@<uid>.service and asking it to rescan your unit files.

If that per-user manager is in a failed state, the reload has nothing to talk to. Rather than erroring loudly, the client side quietly no-ops. No manager, no rescan, no units. The file on disk is fine — there is simply nothing running to read it.

So the real question is never "what is wrong with my unit file?" It is "is my per-user manager actually alive?"

Diagnosing it, step by step

Run these from a root or sudo-capable shell, because a normal user cannot inspect or restart a root-owned user@ unit for another account.

1. Get the real UID. Do not guess it, and do not read it off /run/user/<n> — that directory only shows UIDs with a live session, which may not be the account you care about.

# id alice
uid=1033(alice) gid=1033(alice) groups=1033(alice)
Enter fullscreen mode Exit fullscreen mode

2. Check the manager's state. Active is healthy; failed is your answer.

# systemctl status user@1033.service
● user@1033.service - User Manager for UID 1033
     Loaded: loaded (/lib/systemd/system/user@.service; static)
     Active: failed (Result: signal) since Tue ...
Enter fullscreen mode Exit fullscreen mode

3. List every failed per-user manager at once. Useful for spotting whether this is one account or several.

# systemctl --failed
  UNIT                LOAD   ACTIVE SUB    DESCRIPTION
● user@1033.service   loaded failed failed User Manager for UID 1033
Enter fullscreen mode Exit fullscreen mode

4. Confirm the unit files are even in the right place. This is the one case where the file is the problem. User units must live in ~/.config/systemd/user/ — the user/ subdirectory is mandatory. Files dropped straight into ~/.config/systemd/ are ignored, no warning given.

# ls -la ~alice/.config/systemd/user/
Enter fullscreen mode Exit fullscreen mode

If that directory does not exist, the reload was never going to find anything even with a healthy manager. Create it and move the units in.

Reading the death: killed, not stopped

Once you know the manager failed, resist the urge to guess why. systemctl status and the journal tell you the difference between a clean shutdown and a kill:

# systemctl status user@1033.service
   Main PID: 21847 (code=killed, signal=KILL)
Enter fullscreen mode Exit fullscreen mode

code=killed, signal=KILL with no preceding Stopping User Manager... line in the journal means something sent a SIGKILL from outside — this was not a graceful systemctl stop. That is a real signal, and it is worth ruling causes in or out with evidence instead of a hunch.

The usual suspect is the OOM killer, so check it directly rather than assuming. On a cgroup v1 host:

# cat /sys/fs/cgroup/memory/user.slice/user-1033.slice/memory.oom_control
oom_kill 0
Enter fullscreen mode Exit fullscreen mode

oom_kill 0 means the cgroup OOM killer never fired for this slice. (On a cgroup v2 host the same signal lives in memory.events — look for the oom_kill counter under /sys/fs/cgroup/user.slice/user-<uid>.slice/.) Two more cheap checks keep you honest:

  • cat .../memory.stat — compare total_rss against total_cache. A slice sitting near its limit almost entirely on page cache is not leaking; page cache is reclaimable, and a high cache figure is not evidence of memory exhaustion. This is the classic cgroup-v1 "it looks full but it is fine" reading.
  • free -m — if the host as a whole has gigabytes free, a host-wide OOM did not do this either.
  • uptime — a long uptime rules out "the box just rebooted."

If every one of those comes back clean, be honest in your notes: the kill happened, but the killer is undetermined. "Undetermined, ruled out OOM and reboot with evidence" is a better answer than a confident guess that sends the next person down the wrong path.

The fix: reset, restart, verify

Once you have confirmed a failed manager, the recovery is two commands:

# systemctl reset-failed user@1033.service
# systemctl restart user@1033.service
Enter fullscreen mode Exit fullscreen mode

reset-failed clears the failed state so systemd will act on the unit again; restart brings the manager back up. Then verify — do not assume:

# systemctl is-active user@1033.service
active

# ls /run/user/1033/bus
/run/user/1033/bus
Enter fullscreen mode Exit fullscreen mode

active plus a present /run/user/<uid>/bus socket means the manager is back and systemctl --user will work again. Now the user's daemon-reload actually reloads, and their units appear.

The nuance worth setting expectations on: linger and auto-restart

Here is the part that bites people twice. Many setups do not automatically resurrect a per-user manager after it dies. If the account has no active login session and lingering is off, the manager stays down until something starts it again — a fresh login, or a manual restart like the one above.

loginctl enable-linger <user> keeps a user manager running without an active session, which is what you want for a user meant to run background services around the clock. But enabling linger does not make a failed manager self-heal — it governs whether the manager runs without a login, not whether it recovers from a SIGKILL. If your platform relies on per-user units staying up unattended, the honest expectation to set is: if the manager dies, someone or something has to bring it back. Design for that, and don't promise auto-recovery you have not actually wired up.

The one-line takeaway

When systemctl --user ignores a unit file that is clearly correct, stop staring at the unit file. Check whether user@<uid>.service is alive first. A failed per-user manager makes daemon-reload a silent no-op, and no amount of editing the .service file will fix a manager that is not running.

If you run shared Linux hosts and want fewer of these mysteries in your day, that is most of what we do: seedboxes and storage boxes on our own hardware, in our own datacenter in Finland. Open-source platform (PMSS, GPL v3), EU jurisdiction, 14-day money-back. pulsedmedia.com

Top comments (0)