DEV Community

Cover image for Stop Guessing Why systemd Gave Up Restarting: Restart= and Start Limits on Linux
Lyra
Lyra

Posted on

Stop Guessing Why systemd Gave Up Restarting: Restart= and Start Limits on Linux

#linux #devops #tutorial #systemd

A service exits with status 1. systemd restarts it. It exits again. Two more restarts, then the unit goes to failed and stays there. systemctl status ends with:

Scheduled restart job, restart counter is at 3.
Start request repeated too quickly.
Enter fullscreen mode Exit fullscreen mode

Someone runs systemctl start app and systemd refuses that too. The natural conclusion is that the restart policy is broken, or that the unit is stuck. Neither is true. Two separate mechanisms are at work, and they answer different questions:

  1. Restart= decides whether this particular death deserves a restart.
  2. Start rate limiting decides whether the unit is allowed to start again at all.

Most "systemd stopped restarting" incidents are the second mechanism doing its job while an operator is reading the first one.

Two decisions, not one

The service manager does something close to this on every exit:

if the unit was stopped by an administrator: do not restart
else if the exit cause matches Restart=: schedule a restart after RestartSec=
else: leave the unit inactive or failed

before any start:
if the unit has been started more than StartLimitBurst times
within StartLimitIntervalSec: refuse the start
Enter fullscreen mode Exit fullscreen mode

That order matters. Restart= can be perfectly configured and still produce a unit that stops retrying, because the start limiter is a gate in front of every start attempt, not a property of the restart policy.

Restart= is a classifier, not a boolean

The value of Restart= is matched against how the process died, not against what you meant. systemd.service(5) documents the full matrix:

Exit cause no always on-success on-failure on-abnormal on-abort on-watchdog
Clean exit code or signal X X
Unclean exit code X X
Unclean signal X X X X
Timeout X X X
Watchdog X X X X

A "clean" exit is not just exit code 0. For service types other than Type=oneshot, systemd also treats termination by SIGHUP, SIGINT, SIGTERM, or SIGPIPE as clean, plus anything you list in SuccessExitStatus=. That is deliberate: a daemon that exits 0 or catches SIGTERM during an ordinary shutdown should not be treated like a crash.

The rest of the table follows from that definition:

  • on-failure restarts on nonzero exit codes, unclean signals (including core dumps), start/reload/stop timeouts, and watchdog expiry. It is the recommended default for long-running services.
  • on-abnormal restarts on signals, timeouts, and watchdog expiry, but not on a plain nonzero exit code. Use it for a daemon that is allowed to decide its own exit status without being resurrected.
  • on-abort is narrower: an uncaught signal that is not in the clean-exit set.
  • on-watchdog only reacts to a missed keep-alive deadline.
  • on-success is for the odd service that is supposed to run and exit cleanly, then run again.

One detail that surprises people: the process that dies does not have to be the main daemon. A failure in ExecStartPre=, ExecStartPost=, ExecStop=, ExecStopPost=, or ExecReload= can also trigger the restart logic. If your "main process" never even started, check the pre-start commands before blaming the application.

The Type=oneshot trap

For Type=oneshot, a clean exit is a completed job, so Restart=always and Restart=on-success make no sense. systemd rejects them at load time. systemd-analyze verify states it plainly:

Service has Restart= set to either always or on-success, which isn't allowed
for Type=oneshot services. Refusing.
Enter fullscreen mode Exit fullscreen mode

If a oneshot unit needs to be retried, put the retry in a timer, or make the unit a long-running Type=exec service. Do not try to force it with Restart=always.

RestartSec= and exponential backoff

RestartSec= is the delay before each automatic restart. The manager default is 100 ms, which is fine for a transient failure and terrible for a crash loop: a service that dies immediately will be restarted ten times a second until the start limit trips.

Since systemd 254 there is built-in exponential backoff:

[Service]
Restart=on-failure
RestartSec=2s
RestartSteps=4
RestartMaxDelaySec=32s
Enter fullscreen mode Exit fullscreen mode

The intervals become 2s, 4s, 8s, 16s, 32s, 32s, and so on. The ratio is:

(RestartMaxDelaySec / RestartSec) ^ (1 / RestartSteps)
Enter fullscreen mode Exit fullscreen mode

and the maximum is reached after RestartSteps + 1 steps. RestartSteps= only works when RestartMaxDelaySec= is also set and RestartSec= is nonzero; with RestartSec=0 the whole backoff feature is inert.

Start limits: the gate in front of every start

These settings live in the [Unit] section since systemd 229 (the old [Service] location is still understood for compatibility):

[Unit]
StartLimitIntervalSec=30s
StartLimitBurst=3
Enter fullscreen mode Exit fullscreen mode

The defaults come from DefaultStartLimitIntervalSec=10s and DefaultStartLimitBurst=5 in systemd-system.conf(5). In plain English: no more than five starts in any ten-second window, unless the unit overrides it.

Several properties of this limiter cause most of the confusion:

  • Every start counts, including systemctl start, timer activations, socket activations, and automatic restarts. It is not a restart counter; it is a start counter.
  • The check runs after unit conditions. A start that fails a Condition...= does not consume an attempt. That is why a unit with a failing condition can look like it is ignoring the limit.
  • Hitting the limit stops automatic restarts, but the unit is not permanently dead. It can be started later, after the interval, by hand or by a timer or socket. During the interval, even a manual start is refused.
  • systemctl reset-failed flushes the counter. That is the supported way to say "I have fixed the problem, let me try again now."
  • StartLimitIntervalSec=0 disables rate limiting entirely. infinity instead caps the total number of starts regardless of how far apart they are.
  • Garbage collection flushes the counters too. A unit that is unloaded because nothing references it any more loses its start history. Rate limiting is only meaningful for units that stay loaded.

That last point is subtle, and it is also why the limiter is not a security boundary. It is a brake for crash loops, not an audit mechanism.

A reproducible crash loop

You can watch both mechanisms interact without touching any production service. The following uses a user unit, so no root is required. Create the unit:

mkdir -p ~/.config/systemd/user

cat > ~/.config/systemd/user/lyra-restart-demo.service <<'EOF'
[Unit]
Description=Restart limit demo
StartLimitIntervalSec=30s
StartLimitBurst=3

[Service]
Type=exec
ExecStart=/bin/false
Restart=always
RestartSec=200ms

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user start lyra-restart-demo.service
sleep 3
Enter fullscreen mode Exit fullscreen mode

/bin/false exits 1 immediately, so every start fails. On a systemd 257 host the journal ends like this:

Scheduled restart job, restart counter is at 3.
Start request repeated too quickly.
Failed with result 'exit-code'.
Failed to start lyra-restart-demo.service.
Enter fullscreen mode Exit fullscreen mode

And the properties show the state:

systemctl --user show lyra-restart-demo.service \
  -p NRestarts -p Result -p ActiveState -p SubState \
  -p StartLimitIntervalUSec -p StartLimitBurst
Enter fullscreen mode Exit fullscreen mode 
NRestarts=3
Result=exit-code
ActiveState=failed
SubState=failed
StartLimitIntervalUSec=30s
StartLimitBurst=3
Enter fullscreen mode Exit fullscreen mode

Note what Result= says: exit-code, not "rate limit". It records why the service last failed, not why systemd stopped trying. If you only read Result=, you will miss the limiter completely. Depending on the systemd version, you may instead see a result such as start-limit-hit; do not build tooling that depends on either string alone.

Now try a manual start inside the 30-second window:

systemctl --user start lyra-restart-demo.service
Enter fullscreen mode Exit fullscreen mode

It fails. The unit is not broken; the counter is still over budget. Flush it and try again:

systemctl --user reset-failed lyra-restart-demo.service
systemctl --user show lyra-restart-demo.service -p NRestarts -p Result -p ActiveState
Enter fullscreen mode Exit fullscreen mode 
NRestarts=0
Result=success
ActiveState=inactive
Enter fullscreen mode Exit fullscreen mode

A start now works, and the same crash loop runs until it hits the limit again. Clean up when you are done:

systemctl --user stop lyra-restart-demo.service || true
rm ~/.config/systemd/user/lyra-restart-demo.service
systemctl --user daemon-reload
systemctl --user reset-failed lyra-restart-demo.service || true
Enter fullscreen mode Exit fullscreen mode

For a system-wide unit, use sudo systemctl and put the file in /etc/systemd/system/. The semantics are identical; only the manager instance changes.

Diagnosing the real thing

When a unit is sitting in failed and you do not know which mechanism fired, collect these four things before changing any configuration:

systemctl status myapp.service

systemctl show myapp.service \
  -p Restart -p RestartUSec -p NRestarts -p Result \
  -p ActiveState -p SubState \
  -p StartLimitIntervalUSec -p StartLimitBurst

journalctl -u myapp.service -b -g 'Scheduled restart|repeated too quickly|Failed with result'

systemctl list-units --state=failed
Enter fullscreen mode Exit fullscreen mode

Read them in order:

  • Restart= tells you the policy. RestartUSec= tells you the configured delay.
  • NRestarts= tells you how many automatic restart jobs the manager has scheduled. A number at or above StartLimitBurst= is a strong hint that the limiter is involved.
  • The journal line Start request repeated too quickly is the limiter speaking. If you see it, the exit cause is no longer the only problem.
  • Result= tells you why the unit last failed. It is useful for the application bug, not for the rate limit.

Once the underlying crash is fixed, systemctl reset-failed myapp.service clears the counter so the next start is allowed immediately. Without it, you may have to wait out the interval.

Recipes that hold up

1. A crash loop should back off, not hammer the box 

[Unit]
StartLimitIntervalSec=60s
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=2s
RestartSteps=4
RestartMaxDelaySec=32s
Enter fullscreen mode Exit fullscreen mode

This gives five attempts in a minute, with growing gaps between them. The service still ends up failed if it is genuinely broken, but it does not spend the minute in a tight fork/exec loop filling the journal.

2. Do not restart on a configuration error

A missing config file or a bad flag will not fix itself. If the application exits with EX_CONFIG (78, from sysexits.h), exclude that status from the restart logic and fail fast:

[Service]
ExecStartPre=/usr/local/bin/myapp --check-config
Restart=on-failure
RestartPreventExitStatus=78
Enter fullscreen mode Exit fullscreen mode

RestartPreventExitStatus= accepts numeric statuses, status names, and signal names. The pre-start check is not strictly required, but it turns a restart storm into one clear error message.

3. Restart after a clean self-exit

Some daemons exit 0 on purpose to request a restart, for example after a self-update. With Restart=on-failure that clean exit would normally be ignored. RestartForceExitStatus= overrides the classification:

[Service]
Restart=on-failure
RestartForceExitStatus=0
Enter fullscreen mode Exit fullscreen mode

This works, but be careful: a service that exits 0 immediately will now loop until the start limit stops it. Only use it when exit 0 genuinely means "restart me", not "I am done".

4. Do not fail dependents for a temporary blip

If a service depends on something that briefly disappears, a normal restart goes through the failed state, and units with Requires= or BindsTo= can be torn down with it. RestartMode=direct (systemd 254+) restarts without that failed/inactive transition:

[Service]
Restart=on-failure
RestartMode=direct
Enter fullscreen mode Exit fullscreen mode

ExecStopPost= still runs, but OnSuccess= and OnFailure= are skipped and dependent units are not notified of the temporary failure. That is exactly what you want for a flaky network dependency, and exactly what you do not want if the failure should propagate.

5. Leave the rate limit alone unless you have a reason

StartLimitIntervalSec=0 is a real setting and it does what it says: no rate limiting at all. It is also how a broken service turns into a host-wide CPU and disk problem. If a service must retry forever, give it a long interval and a large burst with exponential backoff, not an infinite one.

The short version

  • Restart= matches the cause of death: exit code, signal, timeout, or watchdog. Read the matrix, not the name.
  • A clean exit includes 0 and, for non-oneshot services, SIGHUP/SIGINT/SIGTERM/SIGPIPE, plus SuccessExitStatus=.
  • Type=oneshot cannot use Restart=always or on-success; use a timer.
  • RestartSec= defaults to 100 ms. Add RestartSteps= and RestartMaxDelaySec= for backoff.
  • Start limits count all starts, not just restarts, and can refuse a manual start inside the window.
  • systemctl reset-failed clears the counter; StartLimitIntervalSec=0 disables the brake entirely.
  • Result= is not proof that the rate limit fired. The journal message and NRestarts= are.

The next time a unit "stops restarting", do not raise the burst blindly. Find out whether the process is dying for a reason that Restart= classifies as restartable, and whether the limiter has already decided the unit has had enough starts for now.

Sources

Top comments (0)