Stop Frozen Linux Hosts Silently Dying: Practical systemd Hardware and Service Watchdogs

A hard lockup is worse than a crash. Crashes leave evidence. Freezes just sit there — SSH dead, timers stuck, monitoring silent — until a human power-cycles the box.

Linux already has two complementary watchdog layers you can turn on without a third-party daemon:

1. Hardware (or softdog) watchdog — PID 1 pings /dev/watchdog*. If userspace or the kernel stops making progress, the board resets.
2. Service watchdog — a daemon must send WATCHDOG=1 keep-alives. Miss the deadline and systemd restarts that unit, not the whole machine.

This guide wires both up on Debian/Ubuntu-style systems with drop-ins you can copy, verify, and reverse.

What this is (and is not)

Layer	Protects against	Typical action
RuntimeWatchdogSec= in system.conf	Host hang / PID 1 stuck / total userspace freeze	Hardware (or softdog) reset
WatchdogSec= on a .service	One daemon wedged while the rest of the host is fine	Kill + optional restart of that unit
systemd-coredump / coredumpctl	Userspace process crash evidence	Capture core, not hang recovery
systemd-pstore	Kernel panic/oops persistence across reboot	Archive dmesg-like panic logs
full kdump / vmcore	Deep kernel post-mortem	Separate crash kernel workflow

Use watchdogs for recovery. Use coredump/pstore/kdump for forensics. They stack well; they are not substitutes.

Prerequisites

• systemd as PID 1 (default on modern Debian/Ubuntu/Fedora/Arch).
• Root (or equivalent) to write drop-ins and load modules.
• Either a real watchdog device or willingness to use the kernel softdog module for labs/VMs.

Step 1 — Discover watchdog devices

# List kernel watchdog class devices (empty if none registered yet)
ls -l /sys/class/watchdog/ 2>/dev/null || echo "no watchdog class entries"

# Common device nodes once a driver is bound
ls -l /dev/watchdog* 2>/dev/null || echo "no /dev/watchdog* yet"

# On systems where pstore/sysfs is populated, identity and timeout show up here:
for d in /sys/class/watchdog/watchdog[0-9]*; do
  [ -d "$d" ] || continue
  echo "== $(basename "$d") =="
  for f in identity state timeout timeleft nowayout; do
    [ -r "$d/$f" ] && printf '  %-10s %s\n' "$f" "$(cat "$d/$f")"
  done
done

Many servers expose Intel TCO (iTCO_wdt), IPMI, or platform watchdogs. Laptops and cloud VMs often expose nothing until you load a driver — that is expected.

Lab fallback: softdog

The software watchdog is a kernel timer, not a separate chip. It still reboots the machine if it is not pinged, but it cannot save you from a fully wedged kernel the way independent hardware can.

# Load for testing (on systems where modules are allowed)
sudo modprobe softdog soft_margin=60

# Confirm it appeared
ls -l /dev/watchdog*
cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || true

Kernel module parameters (from the upstream watchdog parameters doc):

• soft_margin — timeout in seconds (default 60; range roughly 1..65535)
• nowayout — if set, closing the device does not disarm the watchdog
• soft_noboot — set to 1 to log only instead of rebooting (useful while validating)

To make softdog persistent across boots on Debian/Ubuntu:

echo 'softdog' | sudo tee /etc/modules-load.d/softdog.conf
echo 'options softdog soft_margin=60' | sudo tee /etc/modprobe.d/softdog.conf

Prefer real hardware on production hosts. Use softdog for CI images, nested VMs, and learning the systemd side safely.

Step 2 — Enable the system runtime watchdog (PID 1)

systemd opens and pets the hardware watchdog when you set RuntimeWatchdogSec= in manager config.

Create a drop-in (do not hand-edit the whole vendor system.conf if you can avoid it):

sudo mkdir -p /etc/systemd/system.conf.d
sudo tee /etc/systemd/system.conf.d/10-watchdog.conf >/dev/null <<'EOF'
[Manager]
# Pet /dev/watchdog0 (or WatchdogDevice=) at least twice per timeout.
# 20s is a reasonable starting point for servers; raise if boot/IO stalls are noisy.
RuntimeWatchdogSec=20s

# During the late reboot phase (after PID 1 is replaced by systemd-shutdown),
# keep a longer safety net so a stuck umount cannot hang forever.
RebootWatchdogSec=10min

# Optional: pin the device if you have more than one
# WatchdogDevice=/dev/watchdog0

# Optional (hardware that supports pretimeout): fire an action before hard reset
# RuntimeWatchdogPreSec=5s
# RuntimeWatchdogPreGovernor=panic
EOF

Apply manager config:

sudo systemctl daemon-reexec
# or: sudo systemctl daemon-reload && sudo systemctl reboot   # if you prefer a clean cycle

What those knobs actually mean

From systemd-system.conf(5):

• RuntimeWatchdogSec= — if non-zero, systemd programs the watchdog and contacts it at least once every half of the configured timeout. Miss that window → hardware reset. off/0 disables. default opens/pings without forcing a new timeout value.
• RebootWatchdogSec= — applies in the second reboot phase (after regular services are gone and systemd-shutdown is running). Default is typically 10min. First-phase shutdown stalls are a different lever (JobTimeoutSec= / JobTimeoutAction= on shutdown.target).
• KExecWatchdogSec= — only if you kexec; enable together with RuntimeWatchdogSec= so the timer is not left armed incorrectly across kexec.
• WatchdogDevice= — defaults to /dev/watchdog0.
• RuntimeWatchdogPreSec= / RuntimeWatchdogPreGovernor= — optional pre-timeout (for example panic) so you may still get a kernel message before the hard reset. Requires driver support; check /sys/class/watchdog/watchdogX/pretimeout_available_governors.

Verify the system watchdog is armed

# Manager should report the configured runtime watchdog
systemctl show -p RuntimeWatchdogUSec -p RebootWatchdogUSec -p WatchdogDevice

# Device node should be held open by PID 1 once armed
sudo lsof /dev/watchdog0 2>/dev/null || sudo fuser /dev/watchdog0 2>/dev/null || true

# timeleft should bounce as the hardware is serviced (when the sysfs file exists)
watch -n1 'cat /sys/class/watchdog/watchdog0/timeleft 2>/dev/null || echo n/a'

If RuntimeWatchdogUSec=0 after daemon-reexec, the manager did not take the device (missing driver, wrong path, or permissions). Fix discovery first — do not assume the drop-in “silently protects” you.

Safety notes before you arm production

• Start with a generous timeout (30–60s) on storage-heavy hosts so a long fsck or peak IO does not false-trigger.
• Never test by kill -9 1. Validate in a lab VM with softdog + console access.
• If you open /dev/watchdog yourself while systemd also owns it, you can fight over the same device. Let PID 1 be the sole petter for the system watchdog.
• Magic-close / nowayout semantics matter for manual daemons; systemd’s manager path is designed to keep the device serviced for the life of the boot.

Step 3 — Service-level watchdogs for critical daemons

Hardware watchdogs are a blunt instrument. Most of the time you want one stuck worker restarted, not a rack bounce.

systemd’s per-service watchdog:

1. Set WatchdogSec= on the unit.
2. Run the service with notification access (Type=notify is the clean model).
3. Have the process send WATCHDOG=1 via sd_notify often enough (recommended: every half of WatchdogSec).

From systemd.service(5) and sd_watchdog_enabled(3):

• Watchdog arms after start-up completes.
• Missed pings → unit fails; default kill signal path uses SIGABRT (or WatchdogSignal=).
• Restart=on-watchdog restarts only on watchdog expiry; on-failure / always also cover it among other failure modes.
• systemd exports $WATCHDOG_USEC (and $WATCHDOG_PID) so the daemon can self-configure.

Minimal notify + watchdog unit

sudo tee /etc/systemd/system/watchdog-demo.service >/dev/null <<'EOF'
[Unit]
Description=Demo daemon with systemd service watchdog
After=network-online.target

[Service]
Type=notify
ExecStart=/usr/local/bin/watchdog-demo.py
WatchdogSec=30s
Restart=on-watchdog
RestartSec=2s
# NotifyAccess= is implied as main when WatchdogSec= is set; being explicit is fine:
NotifyAccess=main
# Optional: capture a core when the watchdog aborts the process
# LimitCORE=infinity

[Install]
WantedBy=multi-user.target
EOF

Reference daemon (Python)

This uses the systemd notification socket the same way sd_notify(3) does. No extra Python packages required.

sudo tee /usr/local/bin/watchdog-demo.py >/dev/null <<'EOF'
#!/usr/bin/env python3
"""Minimal Type=notify service that pets systemd's WatchdogSec timer."""
import os
import socket
import time

def notify(state: str) -> None:
    addr = os.environ.get("NOTIFY_SOCKET")
    if not addr:
        return
    # Abstract namespace sockets are exposed as "@..." in the environment.
    if addr.startswith("@"):
        addr = "\0" + addr[1:]
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    try:
        sock.connect(addr)
        sock.sendall(state.encode())
    finally:
        sock.close()

def main() -> None:
    # Advertise readiness for Type=notify
    notify("READY=1\nSTATUS=watchdog-demo running\n")

    # Prefer the manager-provided timeout; fall back to 30s.
    usec = int(os.environ.get("WATCHDOG_USEC", "30000000"))
    # Official guidance: ping about twice per watchdog window.
    interval = max(usec / 2_000_000.0, 1.0)

    while True:
        # Do real health work here (DB ping, queue depth, event-loop heartbeat).
        # Only send WATCHDOG=1 if the process is actually healthy.
        notify("WATCHDOG=1\nSTATUS=healthy\n")
        time.sleep(interval)

if __name__ == "__main__":
    main()
EOF
sudo chmod 755 /usr/local/bin/watchdog-demo.py

Enable and confirm:

sudo systemctl daemon-reload
sudo systemctl enable --now watchdog-demo.service
systemctl status watchdog-demo.service --no-pager
systemctl show watchdog-demo.service -p WatchdogUSec -p WatchdogTimestamp -p NotifyAccess

Prove the service watchdog fires (safe)

Temporarily break keep-alives without rebooting the host:

# 1) Lower the timeout for a quick lab test
sudo mkdir -p /etc/systemd/system/watchdog-demo.service.d
sudo tee /etc/systemd/system/watchdog-demo.service.d/10-short.conf >/dev/null <<'EOF'
[Service]
WatchdogSec=5s
EOF
sudo systemctl daemon-reload
sudo systemctl restart watchdog-demo.service

# 2) Freeze the main process so WATCHDOG=1 stops
MAINPID=$(systemctl show -p MainPID --value watchdog-demo.service)
sudo kill -STOP "$MAINPID"

# 3) Wait > WatchdogSec and inspect
sleep 8
systemctl status watchdog-demo.service --no-pager || true
journalctl -u watchdog-demo.service -n 30 --no-pager

You should see the unit enter a failed/restarting state tied to the watchdog, not a machine reboot. Clean up:

sudo kill -CONT "$MAINPID" 2>/dev/null || true
sudo rm -f /etc/systemd/system/watchdog-demo.service.d/10-short.conf
sudo systemctl daemon-reload
sudo systemctl reset-failed watchdog-demo.service
sudo systemctl restart watchdog-demo.service

C / Go one-liners (production style)

If you already link libsystemd:

#include <systemd/sd-daemon.h>

/* after successful init */
sd_notify(0, "READY=1\nSTATUS=listening\n");

/* in your heartbeat path */
sd_notify(0, "WATCHDOG=1\n");

/* optional: force the configured watchdog action immediately */
sd_notify(0, "WATCHDOG=trigger\n");

Check whether the manager expects pings:

uint64_t usec;
if (sd_watchdog_enabled(0, &usec) > 0) {
    /* schedule keep-alives at usec/2 */
}

Shell services can pet with:

systemd-notify WATCHDOG=1
# or, equivalently when NOTIFY_SOCKET is set:
printf 'WATCHDOG=1\n' | socat - UNIX-DGRAM:"$NOTIFY_SOCKET"

Only do that from the main process (or configure NotifyAccess= carefully). Random helper scripts with NotifyAccess=all are a footgun.

Step 4 — Production patterns that age well

Pick timeouts from the workload, not vibes

• System runtime watchdog: long enough to survive intentional heavy stalls (package upgrades, RAID rebuild spikes), short enough that a dead host is painful for less than a minute or two. Many operators land in the 15–60s band.
• Service watchdog: slightly above worst-case legitimate blocking work. If a request can block for 20s, a 10s WatchdogSec= will lie to you all day.
• Ping interval ≈ half of WatchdogSec / RuntimeWatchdogSec (this matches both systemd manager behavior and sd_watchdog_enabled(3) guidance).

Health-gated pings beat blind timers

A timer thread that always sends WATCHDOG=1 while the accept loop is dead buys you nothing. Gate the ping on a real liveness signal:

• last successful event-loop iteration
• worker pool not deadlocked
• critical dependency probe (local only — do not block the heartbeat on flaky remote APIs without a timeout budget)

WATCHDOG=trigger is available when the app itself detects a fatal internal condition and wants the unit’s watchdog failure path immediately (sd_notify(3), since v243).

Restart policy

[Service]
WatchdogSec=30s
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=60s
StartLimitBurst=5

on-watchdog is narrower if you only want restarts for missed keep-alives. Pair with start limits so a permanently broken binary does not thrash.

Journal breadcrumbs

journalctl -b -u your-service.service --grep='watchdog'
journalctl -b -p err --grep='watchdog'

After a hardware watchdog reset, look for reboot reason clues from the firmware/BMC and correlate with last -x, BMC event logs, or journalctl --list-boots. Not every platform stamps “watchdog” clearly in Linux userspace.

Step 5 — Optional hardening around the edges

# /etc/systemd/system/critical-app.service.d/20-watchdog.conf
[Service]
Type=notify
WatchdogSec=45s
Restart=on-failure
# Keep notification socket limited to the main process
NotifyAccess=main

For shutdown hangs (different failure mode than runtime freezes), remember:

• RuntimeWatchdogSec= still applies while PID 1 is alive during early shutdown.
• RebootWatchdogSec= covers the late systemd-shutdown phase.
• First-phase job stalls → shutdown.target job timeouts, not the runtime watchdog alone.

Troubleshooting checklist

Symptom	Likely cause	Fix
RuntimeWatchdogUSec=0 after config	No driver / wrong device	Load module, set WatchdogDevice=, confirm /dev/watchdog*
Service kills itself immediately	Never sends WATCHDOG=1	Implement notify pings; check $NOTIFY_SOCKET
Pings ignored	Wrong process / NotifyAccess=	Ping from main PID; use NotifyAccess=main
False restarts under load	Timeout too aggressive	Raise WatchdogSec=; health-check with budget
Lab VM never reboots on purpose	soft_noboot=1 or no softdog	Check module params; use a disposable VM
Fight over /dev/watchdog	Second petter opened the device	One owner: systemd manager or a dedicated daemon, not both

A sane rollout plan

1. Inventory hosts with real watchdog hardware (/sys/class/watchdog).
2. Enable service watchdogs first on 1–2 critical units (low blast radius).
3. Add RuntimeWatchdogSec= in a staging environment; soak through patch windows.
4. Document expected reboot signatures for on-call (so a watchdog reset is not treated like a mystery power event).
5. Keep pstore/coredump/kdump configured for the cases where you still need artifacts after the fact.

Copy-paste summary

# --- A) System watchdog (hardware or softdog) ---
sudo mkdir -p /etc/systemd/system.conf.d
sudo tee /etc/systemd/system.conf.d/10-watchdog.conf >/dev/null <<'EOF'
[Manager]
RuntimeWatchdogSec=20s
RebootWatchdogSec=10min
EOF
sudo systemctl daemon-reexec
systemctl show -p RuntimeWatchdogUSec -p WatchdogDevice

# --- B) Service watchdog sketch ---
# Type=notify
# WatchdogSec=30s
# Restart=on-failure
# + periodic sd_notify("WATCHDOG=1")

References

• systemd-system.conf(5) — RuntimeWatchdogSec / RebootWatchdogSec / WatchdogDevice
• systemd.service(5) — WatchdogSec= and Restart=
• sd_notify(3) — WATCHDOG=1 / WATCHDOG=trigger
• sd_watchdog_enabled(3) — $WATCHDOG_USEC half-interval guidance
• Linux Watchdog driver API
• Kernel watchdog module parameters (incl. softdog)
• Debian manpage mirrors: systemd-system.conf(5), systemd.service(5)

---

Frozen hosts do not page you politely. A petted watchdog turns “silent forever” into “back in under a minute with a boot entry you can alert on.” Start with service keep-alives where you can, arm the hardware watchdog where the platform provides one, and keep your crash forensics stack for the cases that still need a post-mortem.