DEV Community

Cover image for Your Service Is Ready When It Says So: Practical sd_notify Readiness on Linux
Lyra
Lyra

Posted on

Your Service Is Ready When It Says So: Practical sd_notify Readiness on Linux

#linux #systemd #devops #python

A service that loads a 14 GB model takes longer to become useful than it takes to start. For years the workaround has been the same everywhere:

[Service]
Type=simple
ExecStartPre=/bin/sleep 10
ExecStart=/usr/local/bin/inference-server --config /etc/inference.toml
Enter fullscreen mode Exit fullscreen mode

It works until the day the disk is slower, the model is bigger, or the host is busy. Then the reverse proxy starts routing traffic ten seconds after systemctl start returned, and every request during that window gets a 502. The opposite failure is just as common: a ten-second sleep on a service that needed two, which is eight seconds of boot time you will never get back.

The sleep is a guess about readiness. systemd has a real mechanism for this, and most services never use it.

What systemd actually considers "started"

The service Type= decides when a unit becomes active and when units ordered with After= are allowed to start.

  • Type=simple considers the service started immediately after it forks the process. The binary could be halfway through dlopen() and systemd would not know.
  • Type=exec waits until execve() succeeds. That catches a missing binary or bad interpreter, but says nothing about whether the application is listening.
  • Type=notify waits for the service itself to send a READY=1 datagram. Only then does the unit become active and the rest of the boot transaction continue.
  • Type=notify-reload adds a second contract: reloads are also completed by notification instead of by guessing.
  • Type=dbus uses acquisition of a bus name as the readiness signal.

Type=notify is the one worth reaching for when a service has real initialization to do: opening a database, replaying a journal, loading weights, warming a cache, binding a privileged port.

The protocol is smaller than you think

systemd sets $NOTIFY_SOCKET in the service environment. It is the address of a Unix datagram socket, either a filesystem path like /run/systemd/notify or an abstract socket whose name starts with @. A service writes newline-separated KEY=VALUE assignments to that socket. Unknown keys are ignored, which is what makes the protocol extensible.

There is no handshake and no response body. Two details matter:

  1. If $NOTIFY_SOCKET is unset, the service is not running under systemd. A correct client treats this as a no-op and keeps working, so the same binary runs fine in a terminal or a container.
  2. A leading @ in the environment variable means the abstract namespace. In the kernel's sockaddr, that first character becomes a NUL byte.

You do not need libsystemd to speak this protocol. Here is a complete Python client:

import os
import socket


def sd_notify(state: str) -> None:
    addr = os.environ.get("NOTIFY_SOCKET")
    if not addr:
        return
    if addr.startswith("@"):
        addr = "\0" + addr[1:]
    with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as sock:
        sock.sendto(state.encode("utf-8"), addr)
Enter fullscreen mode Exit fullscreen mode

That is the whole dependency.

A lab you can run in five minutes

This daemon does three things in order: binds a socket, pretends to load something for two seconds, then announces readiness. The order is the point. Sending READY=1 before listen() is how you build a service that is "ready" and still refuses connections.

#!/usr/bin/env python3
import os
import signal
import socket
import sys
import time

SOCKET_PATH = "/run/readiness-demo/demo.sock"


def sd_notify(state: str) -> None:
    addr = os.environ.get("NOTIFY_SOCKET")
    if not addr:
        return
    if addr.startswith("@"):
        addr = "\0" + addr[1:]
    with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as sock:
        sock.sendto(state.encode("utf-8"), addr)


def main() -> int:
    sd_notify("STATUS=Binding socket")

    srv = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    srv.bind(SOCKET_PATH)
    os.chmod(SOCKET_PATH, 0o660)
    srv.listen(16)

    sd_notify("STATUS=Loading model weights")
    time.sleep(2)  # replace with the real initialization work

    sd_notify("STATUS=Serving on " + SOCKET_PATH)
    sd_notify("READY=1")

    def shutdown(signum, frame):
        sd_notify("STOPPING=1")
        srv.close()
        try:
            os.unlink(SOCKET_PATH)
        except FileNotFoundError:
            pass
        sys.exit(0)

    signal.signal(signal.SIGTERM, shutdown)

    while True:
        conn, _ = srv.accept()
        conn.sendall(b"ok\n")
        conn.close()


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Install it and give it a unit:

# /etc/systemd/system/readiness-demo.service
[Unit]
Description=Readiness demo daemon
After=network.target

[Service]
Type=notify
NotifyAccess=main
DynamicUser=yes
RuntimeDirectory=readiness-demo
RuntimeDirectoryMode=0750
ExecStart=/usr/local/bin/readiness-demo.py
TimeoutStartSec=30s
Restart=on-failure
RestartSec=2s

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode 
sudo install -m 0755 readiness-demo.py /usr/local/bin/readiness-demo.py
sudo systemctl daemon-reload
sudo systemctl start readiness-demo.service
Enter fullscreen mode Exit fullscreen mode

The start command returns only after READY=1 arrives. Verify what systemd recorded:

systemctl show -p Type -p NotifyAccess -p SubState -p StatusText \
  readiness-demo.service
Enter fullscreen mode Exit fullscreen mode 
Type=notify
NotifyAccess=main
SubState=running
StatusText=Serving on /run/readiness-demo/demo.sock
Enter fullscreen mode Exit fullscreen mode

And the human-readable view:

systemctl status readiness-demo.service --no-pager
Enter fullscreen mode Exit fullscreen mode

The Status: line comes straight from the last STATUS= assignment. No custom logging, no parsing, no extra metric endpoint.

STATUS is the cheapest observability you will add today

STATUS= accepts a single-line UTF-8 string. Use it for the thing an operator would otherwise have to guess:

sd_notify(f"STATUS=Loading shard {i}/{total}")
sd_notify("STATUS=Compacting WAL")
sd_notify("STATUS=Waiting for quorum (3/5)")
Enter fullscreen mode Exit fullscreen mode

systemd keeps the latest value and exposes it as the StatusText property. It survives into systemctl status, which means the first command anyone runs during an incident already contains your application's own description of what it is doing.

Two related fields are worth knowing. ERRNO= takes an errno number as a string, and BUSERROR= takes a D-Bus error name. Both are recorded when a service is about to fail, and both show up in unit status. If your startup code knows why it is giving up, say so in the notification instead of only writing it to the journal.

One limit: STATUS= is single-line by construction. A newline starts the next assignment, so multi-line messages must be flattened or logged separately.

The same thing in C

If the service is already C or C++, link against libsystemd and let sd_notify(3) handle the socket details:

#include <systemd/sd-daemon.h>
#include <unistd.h>

int main(void) {
    sd_notify(0, "STATUS=Binding listening socket");

    /* bind(), listen(), and real initialization go here */

    sd_notify(0, "STATUS=Serving");
    if (sd_notify(0, "READY=1") < 0)
        return 1;

    for (;;)
        pause();
}
Enter fullscreen mode Exit fullscreen mode 
cc readiness.c -o readiness $(pkg-config --cflags --libs libsystemd)
Enter fullscreen mode Exit fullscreen mode

The return value is easy to misread. sd_notify() returns 0 when $NOTIFY_SOCKET is unset, a positive value when the message was queued, and a negative errno on failure. Treat only negative values as errors, otherwise the binary fails when you run it by hand.

Shell wrappers and the attribution trap

Sometimes the long-running process is a third-party binary that cannot notify, and you wrap it in a shell script. systemd-notify exists for exactly this:

#!/bin/sh
set -eu

/usr/local/bin/worker --config /etc/worker.conf &
worker_pid=$!

# Wait for the actual condition you care about, not a fixed sleep.
until ss -ltn 'sport = :8080' | grep -q ':8080'; do
    sleep 0.2
done

systemd-notify --ready --status="Serving on :8080"
wait "$worker_pid"
Enter fullscreen mode Exit fullscreen mode

The unit needs NotifyAccess=all here. The reason is subtle: systemd-notify is a separate process. It first tries to send the message as its parent (the shell, which is the unit's main process) and falls back to its own PID when it lacks the privilege to do so. With the default NotifyAccess=main, that fallback is rejected. all accepts messages from any process in the unit's cgroup.

Do not add --no-block unless you understand the race it reintroduces. By default systemd-notify waits until the manager has processed the message; with --no-block, the helper can exit before PID 1 looks up which unit the sender belonged to, and the notification is dropped.

The shell also has to stay alive. If the script execs the worker, the process that systemd is tracking becomes the worker, and there is nobody left to send the notification.

Reloads that finish when the reload is finished

Type=notify-reload is the newer half of the contract, available since systemd 253. It removes the need for an ExecReload= command that returns immediately while the daemon is still re-reading its config.

On systemctl reload, systemd sends SIGHUP to the main process (change it with ReloadSignal=). The service must reply with RELOADING=1 and a MONOTONIC_USEC= timestamp, do the work, then send READY=1:

def reload_config() -> None:
    now_us = time.clock_gettime_ns(time.CLOCK_MONOTONIC) // 1000
    sd_notify(f"RELOADING=1\nMONOTONIC_USEC={now_us}")
    load_config()
    sd_notify("READY=1")


signal.signal(signal.SIGHUP, lambda signum, frame: reload_config())
Enter fullscreen mode Exit fullscreen mode

MONOTONIC_USEC is what lets systemd correlate this reload cycle with the one it initiated, so a notify-reload service should always include it alongside RELOADING=1. If the service never sends the final READY=1, systemctl reload fails after the start timeout, the old configuration stays in effect, and the running service is not killed. That is a much better failure mode than a reload command that exits 0 while the daemon is half-configured.

Timeouts, and initialization that legitimately takes minutes

TimeoutStartSec= defaults to DefaultTimeoutStartSec=, which is 90 seconds on a stock systemd. When it expires before READY=1, the start is considered failed and the service is shut down again.

For a long migration or a cold model load, do not raise the timeout to an hour and hope. Send EXTEND_TIMEOUT_USEC= while you are making progress:

deadline_budget_us = 120_000_000  # ask for two more minutes
sd_notify(f"EXTEND_TIMEOUT_USEC={deadline_budget_us}")
Enter fullscreen mode Exit fullscreen mode

The first extension must arrive before the original timeout expires. After that, systemd keeps allowing the start as long as the service repeats the message within the interval it granted. This turns the timeout into a progress deadline: a hung load still fails, while a slow but working one is allowed to finish.

When it does not work

Start with the unit's own view of itself:

systemctl show -p Type -p NotifyAccess -p SubState -p StatusText -p MainPID \
  readiness-demo.service
systemctl status readiness-demo.service --no-pager
journalctl -u readiness-demo.service -b --no-pager
systemd-analyze verify /etc/systemd/system/readiness-demo.service
Enter fullscreen mode Exit fullscreen mode

Then match the symptom:

  • Stuck in activating (start) until the timeout. Type=notify is set, but no READY=1 ever arrives. Either the code path did not run, or the message went to the wrong socket. Check that the process actually has $NOTIFY_SOCKET in its environment.
  • Start returns instantly and clients still race. The unit is still Type=simple, or READY=1 is sent before listen(). Type=simple ignores readiness entirely.
  • StatusText is empty although the code sends STATUS=. NotifyAccess=none. This is the default for non-notify service types; a Type=simple unit silently discards status updates unless NotifyAccess= is set explicitly.
  • Works as root, not as a user service. systemd-notify could not impersonate its parent and fell back to its own PID. Set NotifyAccess=all for wrapper scripts.
  • Notification works before a fork, not after. The sender exited before PID 1 processed the datagram. Send from the process systemd tracks, send MAINPID= for the new main process, or use sd_notify_barrier() to synchronize.
  • systemctl reload times out. The unit is Type=notify-reload but the service sends neither RELOADING=1 with MONOTONIC_USEC= nor the final READY=1.

If you need to see the datagram itself, trace the main process:

sudo strace -f -p "$(systemctl show -p MainPID --value readiness-demo.service)" \
  -e trace=sendmsg
Enter fullscreen mode Exit fullscreen mode

What this is not

sd_notify is a startup and lifecycle protocol, not a health check. Once READY=1 has been sent, systemd will not notice that the event loop is wedged unless you also configure WatchdogSec= and send WATCHDOG=1 keep-alives. Readiness and liveness are separate contracts, and mixing them up produces services that restart for the wrong reason.

It is also not a replacement for socket activation. If systemd owns the listening socket with a .socket unit, the service can be started on the first connection and there is no window for a client to hit a closed port. Use notify when the service must do work before it can serve at all; use socket activation when the kernel can hold the connection while it starts.

The point is simple: replace the sleep with a statement. Your service knows when it is ready. Let it say so.

References

Cover photo: Unsplash.

Top comments (0)