<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Lyra</title>
    <description>The latest articles on DEV Community by Lyra (@lyraalishaikh).</description>
    <link>https://dev.to/lyraalishaikh</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3755481%2F7174207e-67eb-4a72-9c1a-6fdad7505b9c.png</url>
      <title>DEV Community: Lyra</title>
      <link>https://dev.to/lyraalishaikh</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/lyraalishaikh"/>
    <language>en</language>
    <item>
      <title>Too Many Open Files Is Not a Leak: Practical File Descriptor Limits on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Mon, 16 Nov 2026 05:04:48 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/too-many-open-files-is-not-a-leak-practical-file-descriptor-limits-on-linux-2007</link>
      <guid>https://dev.to/lyraalishaikh/too-many-open-files-is-not-a-leak-practical-file-descriptor-limits-on-linux-2007</guid>
      <description>&lt;p&gt;A service runs for three weeks and then starts returning 500s. The log has one line worth reading:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OSError: [Errno 24] Too many open files
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The reflex is to hunt for a leak. Sometimes that is the right answer. Often it is not. The process is simply holding as many descriptors as it is allowed to hold, and nobody ever checked what that number was.&lt;/p&gt;

&lt;p&gt;The error is precise about &lt;em&gt;which&lt;/em&gt; limit was hit, but vague about &lt;em&gt;why&lt;/em&gt;. Linux has at least four different ceilings that produce the same family of symptoms: the per-process soft limit, the per-process hard limit, the kernel's ceiling on that hard limit, and the system-wide file limit. Add &lt;code&gt;select()&lt;/code&gt; and inotify and you have two more traps that produce equally confusing failures.&lt;/p&gt;

&lt;p&gt;Here is how the numbers actually fit together, how to reproduce the failure on purpose, and how to raise the right one without papering over a real leak.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two errors, two scopes
&lt;/h2&gt;

&lt;p&gt;The first useful split is the errno:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;errno&lt;/th&gt;
&lt;th&gt;Name&lt;/th&gt;
&lt;th&gt;Scope&lt;/th&gt;
&lt;th&gt;Counter to inspect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;24&lt;/td&gt;
&lt;td&gt;&lt;code&gt;EMFILE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Per process&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/proc/PID/limits&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;23&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ENFILE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;System-wide&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/proc/sys/fs/file-nr&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;EMFILE&lt;/code&gt; means this process tried to allocate a descriptor number at or above its &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt;. &lt;code&gt;ENFILE&lt;/code&gt; means the kernel ran out of file handles globally, regardless of what any single process is allowed.&lt;/p&gt;

&lt;p&gt;Most "too many open files" incidents are &lt;code&gt;EMFILE&lt;/code&gt;. &lt;code&gt;ENFILE&lt;/code&gt; is rarer and usually points at a system-wide flood or a very low &lt;code&gt;fs.file-max&lt;/code&gt;, not at one badly behaved service.&lt;/p&gt;

&lt;h2&gt;
  
  
  A descriptor is a handle, not a file
&lt;/h2&gt;

&lt;p&gt;It is tempting to read "open files" as "files on disk". That is wrong, and it makes the counts confusing.&lt;/p&gt;

&lt;p&gt;A file descriptor is an index into a per-process table. Each entry points at an open file description in the kernel. The same description can be referenced by several descriptors after &lt;code&gt;dup()&lt;/code&gt;, and a descriptor can refer to almost anything:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;regular files and directories&lt;/li&gt;
&lt;li&gt;TCP and Unix sockets&lt;/li&gt;
&lt;li&gt;pipes and FIFOs&lt;/li&gt;
&lt;li&gt;character and block devices&lt;/li&gt;
&lt;li&gt;epoll instances, eventfd, signalfd, timerfd&lt;/li&gt;
&lt;li&gt;inotify instances, pidfd, memfd&lt;/li&gt;
&lt;li&gt;files passed in from another process over a Unix socket
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ls -l /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/fd
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The listing is the actual table. A healthy web server might show a few log files, a listening socket, hundreds of &lt;code&gt;socket:[...]&lt;/code&gt; entries, and a couple of &lt;code&gt;anon_inode:[eventpoll]&lt;/code&gt; entries. All of them compete for the same numeric limit.&lt;/p&gt;

&lt;p&gt;Two consequences follow.&lt;/p&gt;

&lt;p&gt;First, &lt;code&gt;fork()&lt;/code&gt; copies the descriptor table but shares the underlying open file descriptions. Closing a descriptor in the child does not close the parent's copy. A forking service can therefore have many processes each holding a descriptor for the same socket.&lt;/p&gt;

&lt;p&gt;Second, since Linux 4.5, &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt; also caps how many descriptors an unprivileged process may have "in flight" to other processes via &lt;code&gt;SCM_RIGHTS&lt;/code&gt; on Unix sockets. That is an edge case, but it explains why a descriptor-passing daemon can hit &lt;code&gt;EMFILE&lt;/code&gt; without a table that looks full.&lt;/p&gt;

&lt;h2&gt;
  
  
  The soft limit is the one that bites
&lt;/h2&gt;

&lt;p&gt;Every process has two values for &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;soft limit&lt;/strong&gt; — the value enforced right now&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;hard limit&lt;/strong&gt; — the ceiling the soft limit can be raised to
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;ulimit&lt;/span&gt; -Sn      &lt;span class="c"&gt;# soft&lt;/span&gt;
&lt;span class="nb"&gt;ulimit&lt;/span&gt; -Hn      &lt;span class="c"&gt;# hard&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An unprivileged process can lower either limit and can raise the soft limit up to the hard limit. Raising the hard limit requires &lt;code&gt;CAP_SYS_RESOURCE&lt;/code&gt;. The kernel also refuses to set the hard limit above &lt;code&gt;/proc/sys/fs/nr_open&lt;/code&gt;, which defaults to 1048576 and is enforced for privileged processes too (&lt;code&gt;setrlimit(2)&lt;/code&gt;, &lt;code&gt;proc_sys_fs(5)&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;This is where systemd's default explains a lot of surprise:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="na"&gt;DefaultLimitNOFILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;524288&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the documented manager default in &lt;code&gt;systemd-system.conf(5)&lt;/code&gt;: a soft limit of 1024 and a hard limit of 524288 for processes started by units. A service that never overrides it starts with 1024 usable descriptors, even though &lt;code&gt;ulimit -Hn&lt;/code&gt; in a login shell might show half a million.&lt;/p&gt;

&lt;p&gt;The application is allowed to raise its own soft limit to 524288. Some applications do that at startup; many do not, and they are the ones that fall over at roughly a thousand connections.&lt;/p&gt;

&lt;p&gt;One more detail from the same man page: PID 1 bumps &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt; for itself, then reverts the limit for its children. So this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /proc/1/limits &lt;span class="o"&gt;|&lt;/span&gt; grep &lt;span class="s1"&gt;'open files'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;tells you what the manager allows itself, not what your service gets. Always inspect the service process.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;PID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;systemctl show -p MainPID --value myapp.service&lt;span class="o"&gt;)&lt;/span&gt;
cat /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/limits &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="s1"&gt;'open files'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output has two numbers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Max open files            1024                 524288               files
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first is the soft limit that is actually enforced. The second is the hard ceiling.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproduce EMFILE in ten lines
&lt;/h2&gt;

&lt;p&gt;Guessing is unnecessary. Lower the limit in a subshell and open descriptors until the kernel says no.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /tmp/fd_lab.py&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;

&lt;span class="n"&gt;fds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;fds&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;devnull&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;O_RDONLY&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;OSError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"opened {len(fds)} descriptors; errno {e.errno} {errno.errorcode[e.errno]}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"last fd:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fds&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;fds&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;fd&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;fds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;bash -c &lt;span class="s1"&gt;'ulimit -n 64; python3 /tmp/fd_lab.py'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Debian 13 with systemd 257, the run printed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;opened 59 descriptors; errno 24 EMFILE
last fd: 63
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact number of successful opens varies with interpreter startup and inherited descriptors. The last descriptor is the interesting part: 63 is one less than the limit of 64. The kernel will not hand out descriptor number 64.&lt;/p&gt;

&lt;p&gt;That is the precise meaning of &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt;: it is one greater than the highest descriptor number the process may hold. It is not a byte budget, a disk quota, or a count of files on disk.&lt;/p&gt;

&lt;p&gt;Because Linux allocates the lowest free descriptor number, a process fails when every slot below the limit is occupied, regardless of what those slots point at. Holes left by &lt;code&gt;close()&lt;/code&gt; are reused immediately, so the count of open descriptors is normally close to the limit when &lt;code&gt;EMFILE&lt;/code&gt; appears. A leak is one reason the table fills up; a low limit is another.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read the process before changing anything
&lt;/h2&gt;

&lt;p&gt;Four commands answer most of the question.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;PID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;systemctl show -p MainPID --value myapp.service&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# 1. What is the process actually allowed?&lt;/span&gt;
cat /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/limits &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="s1"&gt;'open files'&lt;/span&gt;

&lt;span class="c"&gt;# 2. How many descriptors are open?&lt;/span&gt;
ls /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/fd &lt;span class="o"&gt;|&lt;/span&gt; wc -l

&lt;span class="c"&gt;# 3. What kinds of objects are they?&lt;/span&gt;
ls -l /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/fd 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; awk &lt;span class="s1"&gt;'{print $NF}'&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; sed &lt;span class="s1"&gt;'s/\[.*//'&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c &lt;span class="o"&gt;|&lt;/span&gt; sort -rn &lt;span class="o"&gt;|&lt;/span&gt; head

&lt;span class="c"&gt;# 4. Is it still growing?&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; i &lt;span class="k"&gt;in&lt;/span&gt; 1 2 3&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; ls /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/fd &lt;span class="o"&gt;|&lt;/span&gt; wc -l&lt;span class="o"&gt;;&lt;/span&gt; sleep 10&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The third command collapses &lt;code&gt;socket:[12345]&lt;/code&gt; to &lt;code&gt;socket:&lt;/code&gt; and leaves paths intact, which is usually enough to distinguish a connection pile-up from a file-handle leak:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    412 socket:
     18 /var/lib/myapp/data.db
      4 anon_inode:[eventpoll]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the count is flat across the three samples and the process is at its soft limit, you have a sizing problem. If it climbs monotonically and never comes down, you have a leak — raise the limit if you need breathing room, but fix the leak.&lt;/p&gt;

&lt;p&gt;Also check the system-wide counter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /proc/sys/fs/file-nr
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It returns three numbers: allocated file handles, free handles, and the maximum. Since Linux 2.6 the middle number is always zero because the kernel deallocates handles instead of caching them. If the first number is close to the third, you are in &lt;code&gt;ENFILE&lt;/code&gt; territory and the per-process limit is not your problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Size by connections, not by superstition
&lt;/h2&gt;

&lt;p&gt;Copying &lt;code&gt;LimitNOFILE=65535&lt;/code&gt; from a blog post is not sizing. A useful starting point is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;soft limit &amp;gt;= idle descriptors + (peak concurrent connections x descriptors per connection) + headroom
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a network daemon, one accepted connection is normally one descriptor. A reverse proxy usually needs two per proxied request: one for the client and one for the upstream. HTTP/2 multiplexes many requests over one connection, so streams do not multiply descriptors the way requests do.&lt;/p&gt;

&lt;p&gt;nginx makes the relationship explicit. &lt;code&gt;worker_connections&lt;/code&gt; is the maximum number of simultaneous connections per worker, and the documentation notes that it includes connections to proxied servers, not just clients. It also notes that the real number cannot exceed the process's open-file limit. That is why the two directives belong together:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="n"&gt;worker_processes&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;events&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;worker_connections&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;worker_rlimit_nofile&lt;/span&gt; &lt;span class="mi"&gt;16384&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;worker_rlimit_nofile&lt;/code&gt; changes &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt; for worker processes without restarting the master. If it is lower than &lt;code&gt;worker_connections&lt;/code&gt;, the descriptor limit wins.&lt;/p&gt;

&lt;p&gt;Databases deserve a separate note. PostgreSQL uses one backend process per client connection. Each backend has its own descriptor table and its own inherited limit, so raising &lt;code&gt;LimitNOFILE&lt;/code&gt; on the unit raises the ceiling for every backend, but the limit is not a single pool shared by the service. The same per-process arithmetic applies to a pool of application workers: ten workers at 1024 each can collectively hold far more than 1024 descriptors.&lt;/p&gt;

&lt;p&gt;That is also why there is no per-cgroup descriptor limit to reach for. cgroup v2 has controllers for CPU, memory, I/O, PIDs, and a few other resources, but no file-descriptor controller. systemd's own documentation points out that file descriptors are accounted like any other form of memory and suggests &lt;code&gt;MemoryMax=&lt;/code&gt; for overall service memory, rather than lowering the hard descriptor limit. A runaway descriptor count can therefore count against the service's memory budget as well as producing &lt;code&gt;EMFILE&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix, in a drop-in
&lt;/h2&gt;

&lt;p&gt;For a system service, do not edit the vendor unit. Add a drop-in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo systemctl edit myapp.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the editor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;LimitNOFILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;65536&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;524288&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Apply and restart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo systemctl daemon-reload
sudo systemctl restart myapp.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;soft:hard&lt;/code&gt; form matters. A single value sets both limits to the same number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="na"&gt;LimitNOFILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;65536&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That works, but it removes the application's ability to raise its own soft limit later. Keeping the hard limit at 524288 preserves that option, and matches systemd's own default shape.&lt;/p&gt;

&lt;p&gt;Verify against the running process, not the unit file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;PID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;systemctl show -p MainPID --value myapp.service&lt;span class="o"&gt;)&lt;/span&gt;
grep -i &lt;span class="s1"&gt;'open files'&lt;/span&gt; /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/limits
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a temporary change without a restart, &lt;code&gt;prlimit&lt;/code&gt; can adjust a live process (same UID or &lt;code&gt;CAP_SYS_RESOURCE&lt;/code&gt; required):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo prlimit --pid &lt;span class="nv"&gt;$PID&lt;/span&gt; --nofile=65536:524288
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is a tourniquet, not a fix. It disappears on restart, and an application that has already entered an error path may not recover just because the limit changed.&lt;/p&gt;

&lt;p&gt;For user services, put the override in the user unit with &lt;code&gt;systemctl --user edit&lt;/code&gt;. Remember that a user unit cannot raise its limit above the hard limit the user manager started with. To change that ceiling, set &lt;code&gt;DefaultLimitNOFILE=&lt;/code&gt; in &lt;code&gt;/etc/systemd/user.conf&lt;/code&gt; and restart the user manager or session; the manager reads that file at startup.&lt;/p&gt;

&lt;p&gt;In containers, the OCI runtime sets the limit for the container process. Docker and Podman both accept the same flag:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;podman run --ulimit &lt;span class="nv"&gt;nofile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;65536:524288 myimage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the runtime does not set one, the container inherits whatever the runtime process had. Check with &lt;code&gt;ulimit -n&lt;/code&gt; inside the container instead of assuming the host value applies.&lt;/p&gt;

&lt;h2&gt;
  
  
  The select() ceiling
&lt;/h2&gt;

&lt;p&gt;Raising the limit above 1024 can make an old application worse, not better.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;select()&lt;/code&gt; stores descriptors in &lt;code&gt;fd_set&lt;/code&gt;, a fixed-size bitmask. On glibc, &lt;code&gt;FD_SETSIZE&lt;/code&gt; is 1024. The man page is blunt about it: &lt;code&gt;select()&lt;/code&gt; can monitor only descriptor numbers below &lt;code&gt;FD_SETSIZE&lt;/code&gt;, calls that "an unreasonably low limit for many modern applications", and says modern applications should use &lt;code&gt;poll(2)&lt;/code&gt; or &lt;code&gt;epoll(7)&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The failure mode is nasty. An application that raises its soft limit and then passes descriptor 1500 to &lt;code&gt;FD_SET()&lt;/code&gt; is writing outside the bitmask. You do not get a clean &lt;code&gt;EMFILE&lt;/code&gt;; you get memory corruption, a missed event, or a crash that looks unrelated.&lt;/p&gt;

&lt;p&gt;If you are about to raise a limit past 1024, check what the process actually uses while it is handling traffic:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo strace -f -e &lt;span class="nv"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;select,pselect6,poll,ppoll,epoll_wait -p &lt;span class="nv"&gt;$PID&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;select&lt;/code&gt; or &lt;code&gt;pselect6&lt;/code&gt; is in the output, the safe move is to keep the descriptor limit at or below 1024 or fix the application to use &lt;code&gt;poll&lt;/code&gt;/&lt;code&gt;epoll&lt;/code&gt;. &lt;code&gt;poll()&lt;/code&gt; and &lt;code&gt;epoll&lt;/code&gt; do not have the &lt;code&gt;FD_SETSIZE&lt;/code&gt; limitation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The other "too many files": inotify
&lt;/h2&gt;

&lt;p&gt;File-watching tools — editors, sync clients, dev servers, backup agents — can fail with a message that sounds like a descriptor problem but is not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ENOSPC: No space left on device, watch ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inotify has its own per-user limits (&lt;code&gt;inotify(7)&lt;/code&gt;, &lt;code&gt;inotify_add_watch(2)&lt;/code&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;fs.inotify.max_user_watches&lt;/code&gt; — total watches per real UID&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;fs.inotify.max_user_instances&lt;/code&gt; — inotify instances per real UID&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;fs.inotify.max_queued_events&lt;/code&gt; — events queued per instance&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;code&gt;inotify_add_watch()&lt;/code&gt; returns &lt;code&gt;ENOSPC&lt;/code&gt; when the watch limit is reached. Each inotify &lt;em&gt;instance&lt;/em&gt; also consumes one file descriptor, so a process can hit both limits, but they are separate counters.&lt;/p&gt;

&lt;p&gt;Inspect them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances

&lt;span class="c"&gt;# Count inotify watches held by one process&lt;/span&gt;
awk &lt;span class="s1"&gt;'/^inotify / {n++} END {print n+0}'&lt;/span&gt; /proc/&lt;span class="nv"&gt;$PID&lt;/span&gt;/fdinfo/* 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Raise the watch limit temporarily:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo sysctl -w fs.inotify.max_user_watches=524288
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Persist it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'fs.inotify.max_user_watches = 524288'&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; sudo tee /etc/sysctl.d/99-inotify.conf
sudo sysctl --system
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The per-UID detail is the trap. Two services running as the same user share one watch budget. Giving a chatty file-watching service its own user is often better than raising the global number again.&lt;/p&gt;

&lt;h2&gt;
  
  
  A short checklist
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Read the errno: 24 is per-process, 23 is system-wide.&lt;/li&gt;
&lt;li&gt;Read &lt;code&gt;/proc/PID/limits&lt;/code&gt;, not &lt;code&gt;/proc/1/limits&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Count &lt;code&gt;/proc/PID/fd&lt;/code&gt; and break it down by type.&lt;/li&gt;
&lt;li&gt;Sample the count over time to separate a leak from a sizing problem.&lt;/li&gt;
&lt;li&gt;Check whether the app uses &lt;code&gt;select()&lt;/code&gt; before raising past 1024.&lt;/li&gt;
&lt;li&gt;Set &lt;code&gt;LimitNOFILE=soft:hard&lt;/code&gt;, not a single value, when you want headroom.&lt;/li&gt;
&lt;li&gt;Restart and verify the running process, not just the unit file.&lt;/li&gt;
&lt;li&gt;If the error is &lt;code&gt;ENOSPC&lt;/code&gt; from a file watcher, look at inotify, not descriptors.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The one-line summary: "too many open files" is a statement about a number, and the first job is to find out which number. Once you know whether it is the soft limit, the hard limit, the system-wide maximum, or a completely different subsystem wearing the same error message, the fix is usually boring — which is exactly what you want from an infrastructure problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;getrlimit(2)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man2/getrlimit.2.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;setrlimit(2)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man2/setrlimit.2.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;proc_sys_fs(5)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man5/proc_sys_fs.5.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd.exec(5)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man5/systemd.exec.5.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd-system.conf(5)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man5/systemd-system.conf.5.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd-user.conf(5)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man5/systemd-user.conf.5.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;inotify(7)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man7/inotify.7.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;inotify_add_watch(2)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man2/inotify_add_watch.2.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;select(2)&lt;/code&gt; — &lt;a href="https://man7.org/linux/man-pages/man2/select.2.html" rel="noopener noreferrer"&gt;man7.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;nginx core module, &lt;code&gt;worker_connections&lt;/code&gt; and &lt;code&gt;worker_rlimit_nofile&lt;/code&gt; — &lt;a href="https://nginx.org/en/docs/ngx_core_module.html" rel="noopener noreferrer"&gt;nginx.org&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Podman &lt;code&gt;--ulimit&lt;/code&gt; — &lt;a href="https://docs.podman.io/en/latest/markdown/podman-run.1.html" rel="noopener noreferrer"&gt;docs.podman.io&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>linux</category>
      <category>systemd</category>
      <category>devops</category>
      <category>performance</category>
    </item>
    <item>
      <title>Stop Writing Firewall Rules Twice: Practical nftables on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Sun, 15 Nov 2026 05:03:42 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-writing-firewall-rules-twice-practical-nftables-on-linux-183n</link>
      <guid>https://dev.to/lyraalishaikh/stop-writing-firewall-rules-twice-practical-nftables-on-linux-183n</guid>
      <description>&lt;p&gt;A dual-stack host running iptables has two firewalls that are supposed to be identical. One lives in &lt;code&gt;iptables&lt;/code&gt;, one in &lt;code&gt;ip6tables&lt;/code&gt;, each with its own save file, its own restore command, and its own chance to drift. Add a port to one and forget the other, and the service is reachable over IPv4 while IPv6 quietly rejects it.&lt;/p&gt;

&lt;p&gt;nftables was designed to remove that duplication. One tool, one ruleset, one transaction, both protocols. But the syntax is different enough that most people copy a snippet, get it working, and never learn why it works — until the day a rule silently stops matching.&lt;/p&gt;

&lt;p&gt;This is the mental model, a complete server ruleset, and the failure modes worth knowing before you deploy it.&lt;/p&gt;

&lt;h2&gt;
  
  
  One ruleset, one transaction
&lt;/h2&gt;

&lt;p&gt;The most important behavioural difference is not syntax. It is how changes are applied.&lt;/p&gt;

&lt;p&gt;Each &lt;code&gt;iptables&lt;/code&gt; invocation is its own kernel operation. A script that adds twenty rules is twenty separate moments in time, and a mistake halfway through leaves you with half a policy. &lt;code&gt;iptables-restore&lt;/code&gt; improves this by applying a table in one go, but you still need separate IPv4 and IPv6 files.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;nft -f file&lt;/code&gt; parses and validates the entire file, then commits it as a single transaction. If one line is wrong, nothing is applied:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo nft -c -f /etc/nftables.conf   &lt;span class="c"&gt;# parse and validate only, change nothing&lt;/span&gt;
sudo nft -f /etc/nftables.conf      &lt;span class="c"&gt;# apply atomically&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That &lt;code&gt;-c&lt;/code&gt; flag is the cheapest insurance in this article. Run it before every apply, especially over SSH.&lt;/p&gt;

&lt;h2&gt;
  
  
  The mental model: tables, chains, rules
&lt;/h2&gt;

&lt;p&gt;nftables has three levels, and confusing them is why copied rulesets feel arbitrary.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;table&lt;/strong&gt; belongs to an address family and holds everything else. A &lt;strong&gt;chain&lt;/strong&gt; is attached to a hook in the packet path (or called from another chain). A &lt;strong&gt;rule&lt;/strong&gt; is a list of expressions followed by statements: match something, then decide.&lt;/p&gt;

&lt;p&gt;The families you will actually use:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Family&lt;/th&gt;
&lt;th&gt;Handles&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ip&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;IPv4 only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ip6&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;IPv6 only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;inet&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;IPv4 and IPv6 together&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;bridge&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ethernet frames crossing a bridge&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;netdev&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Per-interface ingress/egress, before layer 3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;arp&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ARP&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For a normal server, &lt;code&gt;inet&lt;/code&gt; is the whole point. You write the rule once and it is evaluated for both protocols; expressions that only make sense for one family simply do not match the other.&lt;/p&gt;

&lt;p&gt;Within &lt;code&gt;ip&lt;/code&gt;, &lt;code&gt;ip6&lt;/code&gt;, and &lt;code&gt;inet&lt;/code&gt;, the five hooks are &lt;code&gt;prerouting&lt;/code&gt;, &lt;code&gt;input&lt;/code&gt;, &lt;code&gt;forward&lt;/code&gt;, &lt;code&gt;output&lt;/code&gt;, and &lt;code&gt;postrouting&lt;/code&gt;. A base chain declares which hook it belongs to and at what priority. Lower numbers run first, and the standard names are worth memorising because they tell you where you are relative to NAT:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Priority name&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Hook&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;raw&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;-300&lt;/td&gt;
&lt;td&gt;before conntrack&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;mangle&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;-150&lt;/td&gt;
&lt;td&gt;all&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dstnat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;-100&lt;/td&gt;
&lt;td&gt;prerouting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;filter&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;all&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;security&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;all&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;srcnat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;postrouting&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A base chain also has a type. &lt;code&gt;filter&lt;/code&gt; is the default choice. &lt;code&gt;nat&lt;/code&gt; chains are special: only the first packet of a connection traverses them, because their job is to create the conntrack entry that later packets reuse. &lt;code&gt;route&lt;/code&gt; chains re-run the route lookup when a rule changed the destination.&lt;/p&gt;

&lt;p&gt;And a base chain has a policy: &lt;code&gt;accept&lt;/code&gt; or &lt;code&gt;drop&lt;/code&gt;. The policy only applies to packets that reach the end of the chain without a verdict.&lt;/p&gt;

&lt;h2&gt;
  
  
  A complete dual-stack server ruleset
&lt;/h2&gt;

&lt;p&gt;Here is a full &lt;code&gt;/etc/nftables.conf&lt;/code&gt; for a host exposing SSH and a web server. It is deliberately boring, because boring firewalls are the ones you can reason about at 3 a.m.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;#!/usr/sbin/nft -f
# /etc/nftables.conf — one ruleset for IPv4 and IPv6

flush ruleset

table inet filter {
    set admin_nets {
        type ipv4_addr
        flags interval
        elements = { 203.0.113.0/24, 198.51.100.10 }
    }

    chain input {
        type filter hook input priority filter; policy drop;

        # Drop packets conntrack already considers broken
        ct state invalid counter drop

        # Replies to connections we opened, plus related ICMP errors
        ct state established,related accept

        # Never filter loopback: local resolvers, databases, and sockets rely on it
        iifname "lo" accept

        # Trusted management networks get full access to this host
        ip saddr @admin_nets accept

        # IPv6 neighbour discovery. Drop this and IPv6 fails in confusing ways.
        icmpv6 type { nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept

        # Rate-limited ping, still useful for diagnostics
        ip protocol icmp icmp type echo-request limit rate 5/second accept

        # New SSH connections, capped per source burst
        tcp dport 22 ct state new limit rate 10/minute burst 5 packets accept

        tcp dport { 80, 443 } accept

        # Log what is about to be dropped, but do not let logging flood the disk
        limit rate 5/minute log prefix "nft input drop: " level info
        counter drop
    }

    chain forward {
        # This host is not a router
        type filter hook forward priority filter; policy drop;
    }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few details that matter more than they look:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;ct state established,related accept&lt;/code&gt; must come before the port rules. Without it, reply traffic from your own outbound connections is dropped by the policy.&lt;/li&gt;
&lt;li&gt;The ICMPv6 rule is not optional. Neighbour solicitation and router advertisements are how IPv6 hosts find each other and their gateway. Filter them and the network appears to work, then stalls.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;iifname&lt;/code&gt; matches the interface &lt;em&gt;name&lt;/em&gt;. &lt;code&gt;iif&lt;/code&gt; matches the interface &lt;em&gt;index&lt;/em&gt;, which is unstable for interfaces that are created and destroyed (veth pairs, tunnels). If a rule works at boot and stops working after a container restart, this is usually why.&lt;/li&gt;
&lt;li&gt;The final &lt;code&gt;log&lt;/code&gt; rule has no verdict, so evaluation continues to &lt;code&gt;counter drop&lt;/code&gt;. The &lt;code&gt;limit&lt;/code&gt; in front means only the first five packets per minute are logged; the rest are still dropped, just not printed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No &lt;code&gt;output&lt;/code&gt; chain is defined. With no chain on that hook, outbound traffic is accepted — which is what you want on a server unless you are building an egress filter.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sets: stop writing one rule per address
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;admin_nets&lt;/code&gt; block above is a named set. Referencing it with &lt;code&gt;@admin_nets&lt;/code&gt; turns a potential list of forty rules into one lookup against a data structure.&lt;/p&gt;

&lt;p&gt;The practical payoff is runtime updates. You do not have to rewrite the ruleset to add an address:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo nft add element inet filter admin_nets &lt;span class="o"&gt;{&lt;/span&gt; 192.0.2.0/24 &lt;span class="o"&gt;}&lt;/span&gt;
sudo nft delete element inet filter admin_nets &lt;span class="o"&gt;{&lt;/span&gt; 198.51.100.10 &lt;span class="o"&gt;}&lt;/span&gt;
sudo nft list &lt;span class="nb"&gt;set&lt;/span&gt; inet filter admin_nets
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;flags interval&lt;/code&gt; is what allows CIDR ranges in the set and lets the kernel merge overlapping intervals. Without it, the set only accepts single addresses.&lt;/p&gt;

&lt;p&gt;Sets also support timeouts, which is how you build temporary bans without a cron job:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;set ssh_guard {
    type ipv4_addr
    flags timeout
    timeout 1h
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Elements added to that set expire on their own after an hour. Firewall tooling such as fail2ban and CrowdSec can use the same mechanism when it is configured for the nftables backend.&lt;/p&gt;

&lt;h2&gt;
  
  
  NAT is a chain type, not a separate tool
&lt;/h2&gt;

&lt;p&gt;Source NAT and port forwarding live in &lt;code&gt;nat&lt;/code&gt; chains. A minimal IPv4 router or container host looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;table ip nat {
    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        iifname "eth0" tcp dport 8080 dnat to 10.0.0.10:80
    }

    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        oifname "eth0" masquerade
    }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things surprise people here. First, &lt;code&gt;masquerade&lt;/code&gt; is dynamic source NAT: it uses the address currently on the outgoing interface, so it survives DHCP changes. &lt;code&gt;snat to&lt;/code&gt; pins a specific address. Second, because NAT chains only see the first packet of a connection, a &lt;code&gt;counter&lt;/code&gt; on a NAT rule counts connections, not packets. Do not use it to measure bandwidth.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making it survive a reboot
&lt;/h2&gt;

&lt;p&gt;A ruleset applied with &lt;code&gt;nft -f&lt;/code&gt; is gone after a reboot unless something loads it again. On Debian and Ubuntu, the &lt;code&gt;nftables&lt;/code&gt; package ships &lt;code&gt;/etc/nftables.conf&lt;/code&gt; and a &lt;code&gt;nftables.service&lt;/code&gt; that is &lt;strong&gt;not enabled by default&lt;/strong&gt;. Installing the package does not activate your file.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo systemctl &lt;span class="nb"&gt;enable&lt;/span&gt; --now nftables
sudo systemctl reload nftables     &lt;span class="c"&gt;# re-reads /etc/nftables.conf&lt;/span&gt;
systemctl status nftables --no-pager
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The service is &lt;code&gt;Type=oneshot&lt;/code&gt; with &lt;code&gt;RemainAfterExit=yes&lt;/code&gt;: it runs once at boot and stays "active". A syntax error means the unit fails and the host boots with &lt;strong&gt;no rules at all&lt;/strong&gt;, which is fail-open. That is the strongest argument for running &lt;code&gt;nft -c -f&lt;/code&gt; in a pre-deploy check, and for keeping a tested copy of the file in version control.&lt;/p&gt;

&lt;p&gt;If you use cloud-init, configuration management, or a container runtime that manages netfilter, pick one owner. Two systems writing the same tables will overwrite each other, and the symptom is rules that exist until the next reboot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Debugging what the kernel actually has
&lt;/h2&gt;

&lt;p&gt;Reading the live ruleset is the first step in every firewall bug:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo nft list ruleset                 &lt;span class="c"&gt;# everything, all families&lt;/span&gt;
sudo nft -a list ruleset              &lt;span class="c"&gt;# include handles for precise edits&lt;/span&gt;
sudo nft -s list chain inet filter input   &lt;span class="c"&gt;# omit counters and other state&lt;/span&gt;
sudo nft -j list ruleset &lt;span class="o"&gt;|&lt;/span&gt; jq &lt;span class="s1"&gt;'.nftables[].table'&lt;/span&gt;   &lt;span class="c"&gt;# machine-readable&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Counters only exist on rules that explicitly contain the &lt;code&gt;counter&lt;/code&gt; statement. If a rule shows no packet count, it is not being evaluated — usually because an earlier &lt;code&gt;accept&lt;/code&gt; or &lt;code&gt;drop&lt;/code&gt; already matched, or because the chain is attached to a different hook than you think.&lt;/p&gt;

&lt;p&gt;For the harder cases, nftables can trace a packet through the ruleset:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo nft add rule inet filter input meta nftrace &lt;span class="nb"&gt;set&lt;/span&gt; 1
sudo nft monitor trace
&lt;span class="c"&gt;# ... reproduce the traffic in another terminal, then Ctrl-C ...&lt;/span&gt;
sudo nft delete rule inet filter input handle &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;handle&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The trace output shows every rule the packet visited and the verdict it received. Remove the tracing rule afterwards; it is expensive on a busy interface.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;nft --debug=netlink&lt;/code&gt; is the other useful lever when a rule is rejected or translated into something unexpected: it prints the netlink messages nft is sending to the kernel, which is the ground truth.&lt;/p&gt;

&lt;h2&gt;
  
  
  The traps that actually bite
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Flushing the wrong ruleset.&lt;/strong&gt; &lt;code&gt;flush ruleset&lt;/code&gt; removes &lt;em&gt;every&lt;/em&gt; table in every family, not just yours. On a host running Docker, libvirt, or Kubernetes, that deletes their NAT and filter chains too. Container networking breaks until the runtime re-creates them. If you need to coexist, scope the flush to your own table (&lt;code&gt;delete table inet filter&lt;/code&gt;) instead of wiping the world.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Mixing nftables with ufw or firewalld.&lt;/strong&gt; Both write their own tables. Enabling nftables.service on top of them produces rules that appear and disappear depending on boot order. Choose one front end.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The iptables compatibility layer is still nftables.&lt;/strong&gt; On current distributions, &lt;code&gt;iptables&lt;/code&gt; is usually the nft-backed variant, so its rules appear in &lt;code&gt;nft list ruleset&lt;/code&gt; as compat tables. This is fine, but it means &lt;code&gt;nft list ruleset&lt;/code&gt; can look duplicated or unfamiliar. &lt;code&gt;update-alternatives --config iptables&lt;/code&gt; shows which backend is active; do not switch backends on a live firewall host.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Locking yourself out.&lt;/strong&gt; Applying a &lt;code&gt;policy drop&lt;/code&gt; ruleset over SSH with no allow rule ends the session immediately, and the fix requires console access. Validate first, then apply with an automatic rollback armed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Snapshot the working ruleset, including the flush, as one atomic file&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'flush ruleset'&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; sudo nft list ruleset&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sudo tee /root/nft-backup.nft &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null

sudo nft -c -f /etc/nftables.conf

&lt;span class="c"&gt;# If you cannot cancel this within 10 minutes, the old ruleset comes back&lt;/span&gt;
sudo systemd-run --on-active=10min --unit=nft-rollback \
  /usr/sbin/nft -f /root/nft-backup.nft

sudo nft -f /etc/nftables.conf
&lt;span class="c"&gt;# SSH still working? Cancel the rollback:&lt;/span&gt;
sudo systemctl stop nft-rollback.timer
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The backup file starts with &lt;code&gt;flush ruleset&lt;/code&gt;, so restoring it is a single atomic transaction rather than a merge into a half-broken policy.&lt;/p&gt;

&lt;h2&gt;
  
  
  A short migration path
&lt;/h2&gt;

&lt;p&gt;You do not have to rewrite everything at once. Translate one table at a time, compare the live ruleset against the old one between changes, and keep the old ruleset saved. The nftables wiki's server example is a good reference for the canonical input chain, and &lt;code&gt;nft list ruleset&lt;/code&gt; output is valid input to &lt;code&gt;nft -f&lt;/code&gt;, which makes backups and diffs straightforward.&lt;/p&gt;

&lt;p&gt;The end state is worth it: one file, one command, one transaction, and no second copy of your policy for the other address family.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist before you deploy
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;nft -c -f /etc/nftables.conf&lt;/code&gt; passes.&lt;/li&gt;
&lt;li&gt;Loopback is accepted before the policy drop.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ct state established,related&lt;/code&gt; is accepted.&lt;/li&gt;
&lt;li&gt;ICMPv6 neighbour discovery is accepted if IPv6 is enabled.&lt;/li&gt;
&lt;li&gt;The ruleset is loaded by a service or config management at boot, not just typed into a shell.&lt;/li&gt;
&lt;li&gt;A rollback path exists and has been tested.&lt;/li&gt;
&lt;li&gt;Nothing else (Docker, ufw, firewalld) owns the same tables.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;nft(8), the nftables administration tool manual: &lt;a href="https://manpages.debian.org/trixie/nftables/nft.8.en.html" rel="noopener noreferrer"&gt;https://manpages.debian.org/trixie/nftables/nft.8.en.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;nftables wiki, "Simple ruleset for a server": &lt;a href="https://wiki.nftables.org/wiki-nftables/index.php/Simple_ruleset_for_a_server" rel="noopener noreferrer"&gt;https://wiki.nftables.org/wiki-nftables/index.php/Simple_ruleset_for_a_server&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;nftables project documentation and release notes: &lt;a href="https://www.netfilter.org/projects/nftables/" rel="noopener noreferrer"&gt;https://www.netfilter.org/projects/nftables/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Debian package &lt;code&gt;nftables&lt;/code&gt; 1.1.3-1 (trixie), including &lt;code&gt;/etc/nftables.conf&lt;/code&gt; and &lt;code&gt;nftables.service&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>linux</category>
      <category>networking</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>Stop Re-Adding IP Addresses by Hand: Practical systemd-networkd on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Sat, 14 Nov 2026 05:02:58 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-re-adding-ip-addresses-by-hand-practical-systemd-networkd-on-linux-3b6a</link>
      <guid>https://dev.to/lyraalishaikh/stop-re-adding-ip-addresses-by-hand-practical-systemd-networkd-on-linux-3b6a</guid>
      <description>&lt;p&gt;You add a second address to an interface, everything works, and then you reboot. The address is gone.&lt;/p&gt;

&lt;p&gt;So you add it again. Then a package upgrade restarts &lt;code&gt;systemd-networkd&lt;/code&gt;, and it is gone again. Eventually you put the &lt;code&gt;ip addr add&lt;/code&gt; in a script, the script runs before the interface exists, and now you have a race condition on top of a configuration problem.&lt;/p&gt;

&lt;p&gt;The reason is not that Linux forgot. It is that something else owns that interface. &lt;code&gt;systemd-networkd&lt;/code&gt; manages links for which it finds a matching &lt;code&gt;.network&lt;/code&gt; file, and when it brings such a link up it &lt;strong&gt;flushes the existing addresses and routes&lt;/strong&gt; first. Your manual &lt;code&gt;ip&lt;/code&gt; command is not configuration to networkd. It is debris that gets cleaned up on the next reconfigure.&lt;/p&gt;

&lt;p&gt;This post is about replacing that debris with files networkd actually owns: which file does what, why the wrong file sometimes wins, and the handful of settings whose defaults surprise people.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three file types, three different jobs
&lt;/h2&gt;

&lt;p&gt;The names look interchangeable. They are not.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;.link&lt;/code&gt;&lt;/strong&gt; files are processed by &lt;code&gt;systemd-udevd&lt;/code&gt;, before networkd gets involved. They set device naming, MAC address policy, MTU, and similar low-level properties.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;.netdev&lt;/code&gt;&lt;/strong&gt; files tell networkd to &lt;em&gt;create&lt;/em&gt; a virtual device: a bridge, bond, VLAN, VXLAN, veth, and so on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;.network&lt;/code&gt;&lt;/strong&gt; files match an existing link and assign the IP layer to it: addresses, routes, DNS, DHCP behaviour.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A physical NIC needs a &lt;code&gt;.network&lt;/code&gt; file. A bridge needs both a &lt;code&gt;.netdev&lt;/code&gt; file (create it) and a &lt;code&gt;.network&lt;/code&gt; file (give it an address). The bridge &lt;em&gt;ports&lt;/em&gt; need &lt;code&gt;.network&lt;/code&gt; files that attach them to the bridge.&lt;/p&gt;

&lt;p&gt;If you only remember one thing: your problem is almost always in the matching step, not the address syntax.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the files live, and why that matters
&lt;/h2&gt;

&lt;p&gt;networkd reads from four directories:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;/usr/lib/systemd/network&lt;/code&gt; — packages&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/usr/local/lib/systemd/network&lt;/code&gt; — local installs&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/run/systemd/network&lt;/code&gt; — volatile, often generated at boot&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/etc/systemd/network&lt;/code&gt; — local administration&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All files are collected and sorted by filename across all four directories. Files with the &lt;strong&gt;same filename&lt;/strong&gt; replace each other, with &lt;code&gt;/etc&lt;/code&gt; winning over &lt;code&gt;/run&lt;/code&gt;, and &lt;code&gt;/run&lt;/code&gt; winning over &lt;code&gt;/usr&lt;/code&gt;. A zero-byte file, or a symlink to &lt;code&gt;/dev/null&lt;/code&gt;, &lt;strong&gt;masks&lt;/strong&gt; a file of the same name completely.&lt;/p&gt;

&lt;p&gt;That masking rule is how you neutralize generated configuration without editing it. If &lt;code&gt;networkctl status eth0&lt;/code&gt; says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight systemd"&gt;&lt;code&gt;&lt;span class="nc"&gt;Network&lt;/span&gt; &lt;span class="nc"&gt;File&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;systemd&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;netplan&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;then a drop-in is not enough if you want to replace the whole file. Put your own file at &lt;code&gt;/etc/systemd/network/10-netplan-eth0.network&lt;/code&gt; with the same name, or mask it with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo ln -s /dev/null /etc/systemd/network/10-netplan-eth0.network
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and write a differently named file that matches the same interface.&lt;/p&gt;

&lt;p&gt;The upstream man page recommends a filename prefix &lt;strong&gt;below 70&lt;/strong&gt;. That is not decoration. Files are applied in filename order, and the first matching &lt;code&gt;.network&lt;/code&gt; file wins. Later matches are ignored, even if they also match.&lt;/p&gt;

&lt;h3&gt;
  
  
  The matching contest
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;systemd.network(5)&lt;/code&gt; is explicit: "The first (in alphanumeric order) of the network files that matches a given interface is applied, all later files are ignored, even if they match as well."&lt;/p&gt;

&lt;p&gt;So this pair is a bug:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/50-all.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=*&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;DHCP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/90-eth0.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;Address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.10&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;50-all.network&lt;/code&gt; matches &lt;code&gt;eth0&lt;/code&gt; first, so &lt;code&gt;90-eth0.network&lt;/code&gt; never applies. The static address in the "more specific" file loses to the generic file purely because of the number.&lt;/p&gt;

&lt;p&gt;Two habits prevent this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Give specific files lower numbers than generic fallbacks.&lt;/li&gt;
&lt;li&gt;Verify with &lt;code&gt;networkctl status &amp;lt;if&amp;gt;&lt;/code&gt;, which prints the &lt;code&gt;Network File:&lt;/code&gt; that actually won.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Also note that a &lt;code&gt;[Match]&lt;/code&gt; section with no valid settings matches &lt;strong&gt;every&lt;/strong&gt; interface and produces a warning. If you really mean everything, write &lt;code&gt;Name=*&lt;/code&gt; so the intent is explicit.&lt;/p&gt;

&lt;p&gt;One more modern escape hatch: if a udev property named &lt;code&gt;ID_NET_MANAGED_BY&lt;/code&gt; is set on a device, networkd will not match it at all unless the value is &lt;code&gt;io.systemd.Network&lt;/code&gt;. That is a deliberate ownership marker for systems running more than one network manager.&lt;/p&gt;

&lt;h2&gt;
  
  
  A static configuration that survives reboots
&lt;/h2&gt;

&lt;p&gt;Here is the smallest complete example for a server with a fixed address.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/10-eth0.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;Address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.10&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;
&lt;span class="na"&gt;Gateway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.1&lt;/span&gt;
&lt;span class="na"&gt;DNS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.53&lt;/span&gt;
&lt;span class="na"&gt;Domains&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;internal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;Address=&lt;/code&gt; in &lt;code&gt;[Network]&lt;/code&gt; is shorthand for an &lt;code&gt;[Address]&lt;/code&gt; section containing only &lt;code&gt;Address=&lt;/code&gt;. &lt;code&gt;Gateway=&lt;/code&gt; is shorthand for a &lt;code&gt;[Route]&lt;/code&gt; section containing only &lt;code&gt;Gateway=&lt;/code&gt;. The long forms are useful when you need more control:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/10-eth0.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;DNS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.53&lt;/span&gt;
&lt;span class="na"&gt;Domains&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;internal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;

&lt;span class="k"&gt;[Address]&lt;/span&gt;
&lt;span class="na"&gt;Address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.10&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;

&lt;span class="k"&gt;[Route]&lt;/span&gt;
&lt;span class="na"&gt;Destination&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="na"&gt;Gateway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.1&lt;/span&gt;
&lt;span class="na"&gt;Metric&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details worth knowing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;DNS=&lt;/code&gt; is &lt;strong&gt;read by systemd-resolved&lt;/strong&gt;, not by networkd's own resolver. If resolved is not running, setting &lt;code&gt;DNS=&lt;/code&gt; here will not change what glibc uses.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Domains=&lt;/code&gt; entries are search domains. A leading tilde makes one routing-only: &lt;code&gt;Domains=~internal.example.com&lt;/code&gt; sends queries for that suffix to this link's DNS servers without appending it as a search suffix. &lt;code&gt;Domains=~.&lt;/code&gt; is the catch-all routing domain.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Metric=&lt;/code&gt; on a route is how you choose between two default routes. Lower wins.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Apply it with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo networkctl reload
sudo networkctl reconfigure eth0
networkctl status eth0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;networkctl reload&lt;/code&gt; re-reads &lt;code&gt;.netdev&lt;/code&gt; and &lt;code&gt;.network&lt;/code&gt; files and reconfigures interfaces that matched a changed file. &lt;code&gt;networkctl reconfigure&lt;/code&gt; re-applies the &lt;em&gt;already loaded&lt;/em&gt; configuration; it does not re-read the files. If you edited a file, you need &lt;code&gt;reload&lt;/code&gt; first (or just &lt;code&gt;reload&lt;/code&gt;, which reconfigures matched links itself).&lt;/p&gt;

&lt;h3&gt;
  
  
  The part that bites remote admins
&lt;/h3&gt;

&lt;p&gt;Reconfiguring the interface carrying your SSH session can drop the session. Do this on a console, a VM snapshot, or a host where you have a second management path. A typo in a default route is not something you can fix by reconnecting.&lt;/p&gt;

&lt;h2&gt;
  
  
  DHCP, but with opinions
&lt;/h2&gt;

&lt;p&gt;The usual client config is three lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/20-eth0.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;DHCP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;DHCP=&lt;/code&gt; accepts &lt;code&gt;yes&lt;/code&gt;, &lt;code&gt;no&lt;/code&gt;, &lt;code&gt;ipv4&lt;/code&gt;, or &lt;code&gt;ipv6&lt;/code&gt;, and defaults to &lt;code&gt;no&lt;/code&gt;. &lt;code&gt;yes&lt;/code&gt; means both families. DHCPv6 can also be triggered by a Router Advertisement even when you did not enable it here, so "I only enabled IPv4" is not always what happens on a dual-stack LAN.&lt;/p&gt;

&lt;p&gt;The defaults that matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;[DHCPv4] RouteMetric=&lt;/code&gt; defaults to 1024.&lt;/strong&gt; Routes learned from DHCP, including the default route, get that metric. If a VPN or a second NIC also installs a default route, this number decides the winner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;[DHCPv4] UseRoutes=&lt;/code&gt; defaults to true&lt;/strong&gt;, so the server's static routes are installed too, also at metric 1024 unless you change it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;[DHCPv4] UseDomains=&lt;/code&gt; defaults to no&lt;/strong&gt;, so the DHCP-provided domain is &lt;em&gt;not&lt;/em&gt; used as a search domain. Set &lt;code&gt;UseDomains=route&lt;/code&gt; to use it for query routing only, which is usually what you want on a network you do not fully trust.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;[IPv6AcceptRA] RouteMetric=&lt;/code&gt; defaults to &lt;code&gt;512:1024:2048&lt;/code&gt;&lt;/strong&gt;, mapped to high, medium, and low router preference. Raising it above your wired DHCP metric makes IPv6 politely lose to the wired default route.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A wired-primary, Wi-Fi-backup client looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/20-wired.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;DHCP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;

&lt;span class="k"&gt;[DHCPv4]&lt;/span&gt;
&lt;span class="na"&gt;RouteMetric&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;
&lt;span class="na"&gt;UseDomains&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;route&lt;/span&gt;

&lt;span class="k"&gt;[IPv6AcceptRA]&lt;/span&gt;
&lt;span class="na"&gt;RouteMetric&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/30-wifi.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;wlan0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;DHCP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;

&lt;span class="k"&gt;[DHCPv4]&lt;/span&gt;
&lt;span class="na"&gt;RouteMetric&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;600&lt;/span&gt;
&lt;span class="na"&gt;UseDomains&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;route&lt;/span&gt;

&lt;span class="k"&gt;[IPv6AcceptRA]&lt;/span&gt;
&lt;span class="na"&gt;RouteMetric&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;600&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the wired default route wins while it exists, and Wi-Fi takes over when the cable is pulled. networkd handles the IP layer; association itself is still the job of &lt;code&gt;wpa_supplicant&lt;/code&gt; or your distribution's Wi-Fi stack.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;LinkLocalAddressing=&lt;/code&gt; is worth a look too. It defaults to &lt;code&gt;ipv6&lt;/code&gt; in the normal case, but to &lt;code&gt;no&lt;/code&gt; when the link is a bridge port or has &lt;code&gt;KeepMaster=&lt;/code&gt; set. That default explains why a bridge port sometimes has no link-local address even though IPv6 is "enabled."&lt;/p&gt;

&lt;h2&gt;
  
  
  Creating a bridge and handing out addresses
&lt;/h2&gt;

&lt;p&gt;A bridge needs a &lt;code&gt;.netdev&lt;/code&gt; file to exist, then a &lt;code&gt;.network&lt;/code&gt; file to get an address. Ports get their own &lt;code&gt;.network&lt;/code&gt; files that name the bridge.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/10-br0.netdev
&lt;/span&gt;&lt;span class="k"&gt;[NetDev]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;br0&lt;/span&gt;
&lt;span class="na"&gt;Kind&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;bridge&lt;/span&gt;

&lt;span class="k"&gt;[Bridge]&lt;/span&gt;
&lt;span class="na"&gt;STP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/20-br0.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;br0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;Address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.1&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;
&lt;span class="na"&gt;DHCPServer&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;

&lt;span class="k"&gt;[DHCPServer]&lt;/span&gt;
&lt;span class="na"&gt;PoolOffset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;
&lt;span class="na"&gt;PoolSize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;
&lt;span class="na"&gt;EmitDNS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;
&lt;span class="na"&gt;DNS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/30-eth1.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth1&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;Bridge&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;br0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;DHCPServer=yes&lt;/code&gt; requires an IPv4 address on the interface; networkd uses it as the server address. &lt;code&gt;PoolOffset=&lt;/code&gt; and &lt;code&gt;PoolSize=&lt;/code&gt; carve a contiguous pool out of that subnet, and the server address is reserved automatically. Default lease time is one hour, maximum is twelve.&lt;/p&gt;

&lt;p&gt;If this bridge is your LAN and you want those clients to reach the internet, the &lt;em&gt;uplink&lt;/em&gt; interface needs forwarding and NAT, not the bridge:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/10-wan.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;eth0&lt;/span&gt;

&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;DHCP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ipv4&lt;/span&gt;
&lt;span class="na"&gt;IPMasquerade&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ipv4&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;IPMasquerade=ipv4&lt;/code&gt; implies the per-interface forwarding setting and the global one unless you have explicitly set them elsewhere. For IPv6, read the man page carefully: per-interface &lt;code&gt;IPv6Forwarding=&lt;/code&gt; does &lt;strong&gt;not&lt;/strong&gt; by itself make packets forward between interfaces; the global setting has to be on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading the state instead of guessing
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;networkctl list&lt;/code&gt; gives you the operational state and, more usefully, the setup state:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;IDX LINK    TYPE     OPERATIONAL SETUP
  1 lo      loopback carrier     unmanaged
  2 eth0    ether    routable    configured
  3 virbr0  ether    no-carrier  unmanaged
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;configured&lt;/code&gt; — networkd applied a matching file successfully.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;configuring&lt;/code&gt; — still waiting on DHCP, RA, duplicate address detection, or carrier.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;unmanaged&lt;/code&gt; — no matching &lt;code&gt;.network&lt;/code&gt; file, &lt;code&gt;Unmanaged=yes&lt;/code&gt;, or an ownership property told networkd to stay away.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;failed&lt;/code&gt; — configuration was attempted and failed; the journal has the reason.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then drill in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;networkctl status eth0
journalctl -u systemd-networkd -b --no-pager
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A trimmed status view, with documentation addresses substituted:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;● 2: eth0
       State: routable (configured)
       Address: 192.0.2.10
       Gateway: 192.0.2.1
       Network File: /etc/systemd/network/10-eth0.network
       Activation Policy: up
       Required For Online: yes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That &lt;code&gt;Network File:&lt;/code&gt; line answers the "which file won?" question directly.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;networkctl cat @eth0&lt;/code&gt; shows the configuration associated with the interface, including drop-ins; &lt;code&gt;networkctl edit @eth0&lt;/code&gt; opens it for editing, creating a file under &lt;code&gt;/etc&lt;/code&gt; if none exists. With &lt;code&gt;--drop-in=&lt;/code&gt;, it edits a drop-in instead. Note that link-level changes from &lt;code&gt;.link&lt;/code&gt; files need a udev event to take effect; reloading alone does not re-apply them.&lt;/p&gt;

&lt;h3&gt;
  
  
  Boot ordering
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;systemd-networkd-wait-online&lt;/code&gt; waits for all managed links to be configured or failed, and for at least one link to be online. "Online" means the operational state is at least &lt;code&gt;degraded&lt;/code&gt; by default. Useful options:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Wait for one specific interface, not the whole world&lt;/span&gt;
systemd-networkd-wait-online --interface=eth0 --timeout=60

&lt;span class="c"&gt;# Succeed as soon as any managed interface is online&lt;/span&gt;
systemd-networkd-wait-online --any --timeout=120
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The default timeout is 120 seconds. On a laptop with an unplugged dock, the default "wait for all" behaviour is exactly how you get a two-minute boot delay. &lt;code&gt;RequiredForOnline=no&lt;/code&gt; on the irrelevant link is the declarative fix, and it can also take an operational-state range such as &lt;code&gt;degraded:routable&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure modes and what they usually mean
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Likely cause&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SETUP=unmanaged&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No matching &lt;code&gt;.network&lt;/code&gt;; an earlier file masks yours; &lt;code&gt;Unmanaged=yes&lt;/code&gt;; or &lt;code&gt;ID_NET_MANAGED_BY&lt;/code&gt; is set&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Address vanishes after restart&lt;/td&gt;
&lt;td&gt;networkd flushed the link; put the address in a &lt;code&gt;.network&lt;/code&gt; file, or set &lt;code&gt;KeepConfiguration=&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;configuring&lt;/code&gt; forever&lt;/td&gt;
&lt;td&gt;DHCP/RA has no answer, DAD is stuck, or there is no carrier&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Static config ignored on a dead link&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ConfigureWithoutCarrier=&lt;/code&gt; is false by default; enabling it requires disabling DHCP/RA and DAD to reach &lt;code&gt;configured&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wrong file applied&lt;/td&gt;
&lt;td&gt;A lower-sorted &lt;code&gt;.network&lt;/code&gt; matches first; check &lt;code&gt;Network File:&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DNS setting has no effect&lt;/td&gt;
&lt;td&gt;systemd-resolved is not running, or &lt;code&gt;UseDNS=&lt;/code&gt;/&lt;code&gt;UseDomains=&lt;/code&gt; is disabled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Default route is on the wrong NIC&lt;/td&gt;
&lt;td&gt;DHCPv4 metric 1024 vs RA 512:1024:2048 vs another link's metric&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;networkctl reload&lt;/code&gt; did not apply an edit&lt;/td&gt;
&lt;td&gt;Run &lt;code&gt;networkctl reconfigure &amp;lt;if&amp;gt;&lt;/code&gt;; some &lt;code&gt;.netdev&lt;/code&gt; properties (for example VLAN ID) need the interface recreated&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;KeepConfiguration=&lt;/code&gt; is newer than most of this article's other settings: it arrived in systemd 257. It accepts &lt;code&gt;static&lt;/code&gt;, &lt;code&gt;dynamic-on-stop&lt;/code&gt;, and &lt;code&gt;dynamic&lt;/code&gt;. The default is &lt;code&gt;no&lt;/code&gt; on a normal system, &lt;code&gt;dynamic-on-stop&lt;/code&gt; in the initrd, and &lt;code&gt;yes&lt;/code&gt; when the root filesystem is on the network. It is a deliberate escape hatch for the case where losing the address is worse than ignoring DHCP lease expiry, not a general fix for "my config does not apply."&lt;/p&gt;

&lt;h2&gt;
  
  
  Coexistence, not combat
&lt;/h2&gt;

&lt;p&gt;Two network managers on one link produce symptoms that look like driver bugs: addresses appear and disappear, routes flip, and each tool blames the other. Pick an owner per link.&lt;/p&gt;

&lt;p&gt;With networkd, the per-link opt-out is in the &lt;code&gt;[Link]&lt;/code&gt; section:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/network/10-tailscale0.network
&lt;/span&gt;&lt;span class="k"&gt;[Match]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tailscale0&lt;/span&gt;

&lt;span class="k"&gt;[Link]&lt;/span&gt;
&lt;span class="na"&gt;Unmanaged&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That prevents networkd (and later matching files) from bringing the link up or configuring it. It is the right pattern for interfaces owned by Tailscale, libvirt, container runtimes, or anything else that creates and configures its own device.&lt;/p&gt;

&lt;h2&gt;
  
  
  A rollout checklist
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Read the current state: &lt;code&gt;networkctl list&lt;/code&gt;, then &lt;code&gt;networkctl status &amp;lt;if&amp;gt;&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Find the file that currently wins: the &lt;code&gt;Network File:&lt;/code&gt; line.&lt;/li&gt;
&lt;li&gt;Write new files in &lt;code&gt;/etc/systemd/network&lt;/code&gt; with a prefix below 70, specific files first.&lt;/li&gt;
&lt;li&gt;Keep the old file until the new one is proven; mask generated files by name rather than editing them.&lt;/li&gt;
&lt;li&gt;Run &lt;code&gt;networkctl reload&lt;/code&gt;, then &lt;code&gt;networkctl status &amp;lt;if&amp;gt;&lt;/code&gt; and &lt;code&gt;ip route show&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Test a reboot in a VM or on a console. A network change that has not survived a reboot is not finished.&lt;/li&gt;
&lt;li&gt;Remove the old file only after the new path works.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The appeal of &lt;code&gt;ip addr add&lt;/code&gt; is that it is immediate. The catch is that it is not a configuration system, and something with a configuration system is going to notice and undo it. Once the address lives in a &lt;code&gt;.network&lt;/code&gt; file, reboots stop being an event.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.network.html" rel="noopener noreferrer"&gt;systemd.network(5) — network configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.netdev.html" rel="noopener noreferrer"&gt;systemd.netdev(5) — virtual network device configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.link.html" rel="noopener noreferrer"&gt;systemd.link(5) — link-level device configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-networkd.html" rel="noopener noreferrer"&gt;systemd-networkd(8) — network manager service&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/networkctl.html" rel="noopener noreferrer"&gt;networkctl(1) — network status and control&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-networkd-wait-online.html" rel="noopener noreferrer"&gt;systemd-networkd-wait-online(8)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/networkd.conf.html" rel="noopener noreferrer"&gt;networkd.conf(5) — global networkd configuration&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Commands and defaults in this article were checked against systemd 257.13 on Debian 13. The example addresses use the documentation ranges from RFC 5737 and RFC 3849; substitute your own.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>networking</category>
      <category>systemd</category>
      <category>devops</category>
    </item>
    <item>
      <title>Your Service Gets 90 Seconds: Practical systemd Timeout Semantics on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Fri, 13 Nov 2026 05:04:45 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/your-service-gets-90-seconds-practical-systemd-timeout-semantics-on-linux-5ggp</link>
      <guid>https://dev.to/lyraalishaikh/your-service-gets-90-seconds-practical-systemd-timeout-semantics-on-linux-5ggp</guid>
      <description>&lt;p&gt;A deploy stalls. You run &lt;code&gt;systemctl restart payments.service&lt;/code&gt;, wait, and then get a line about the start operation timing out. The process did not crash. It ran out of a clock you may not have known was ticking.&lt;/p&gt;

&lt;p&gt;systemd's 90-second default is not one timeout. It is a family of clocks that bound different phases of a unit's life, and they compose in ways that surprise people. This post maps them, shows what each one actually measures, and ends with drop-ins you can copy.&lt;/p&gt;

&lt;p&gt;The values below were checked on Debian 13 with systemd 257.13. The core semantics have been stable for years; options added later are called out.&lt;/p&gt;

&lt;h2&gt;
  
  
  The default really is 90 seconds, twice
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;DefaultTimeoutStartSec=&lt;/code&gt; and &lt;code&gt;DefaultTimeoutStopSec=&lt;/code&gt; default to 90 seconds in both the system manager and the per-user manager, per &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html" rel="noopener noreferrer"&gt;systemd-system.conf(5)&lt;/a&gt;. A unit inherits those unless it sets &lt;code&gt;TimeoutStartSec=&lt;/code&gt;, &lt;code&gt;TimeoutStopSec=&lt;/code&gt;, or the shorthand &lt;code&gt;TimeoutSec=&lt;/code&gt;, which sets both.&lt;/p&gt;

&lt;p&gt;You can read the effective values for any unit without guessing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ systemctl show -p TimeoutStartUSec -p TimeoutStopUSec -p TimeoutAbortUSec ssh.service
&lt;span class="nv"&gt;TimeoutStartUSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1min 30s
&lt;span class="nv"&gt;TimeoutStopUSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1min 30s
&lt;span class="nv"&gt;TimeoutAbortUSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1min 30s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The property names end in &lt;code&gt;USec&lt;/code&gt; because systemd stores these as microseconds, but the values are the ordinary 90-second defaults. &lt;code&gt;TimeoutAbortUSec&lt;/code&gt; is also 90 seconds here because &lt;code&gt;DefaultTimeoutAbortSec=&lt;/code&gt; is unset by default and falls back to &lt;code&gt;TimeoutStopSec=&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Disabled timeouts print as &lt;code&gt;infinity&lt;/code&gt;, which is a useful thing to check when a stop seems to hang forever.&lt;/p&gt;

&lt;h2&gt;
  
  
  The start timeout does not mean what you think
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;TimeoutStartSec=&lt;/code&gt; bounds the interval during which systemd waits for a service to signal that start-up completed. What counts as "completed" depends entirely on &lt;code&gt;Type=&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Start-up is considered complete when&lt;/th&gt;
&lt;th&gt;What the timeout really bounds&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;simple&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The program has begun executing&lt;/td&gt;
&lt;td&gt;Almost nothing after fork/exec&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;exec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The binary was executed successfully&lt;/td&gt;
&lt;td&gt;Missing binary, bad user, permission errors&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;forking&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The parent process exits&lt;/td&gt;
&lt;td&gt;Daemonization, not application readiness&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;notify&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The service sends &lt;code&gt;READY=1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Actual initialization&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;oneshot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ExecStart=&lt;/code&gt; exits&lt;/td&gt;
&lt;td&gt;The whole job; disabled by default&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That first row is the trap. For &lt;code&gt;Type=simple&lt;/code&gt;, systemd assumes the unit is started immediately after the program begins executing. A Python service that then spends ten minutes loading a model is already "started" as far as the manager is concerned. Raising &lt;code&gt;TimeoutStartSec=&lt;/code&gt; on a &lt;code&gt;simple&lt;/code&gt; unit will not make a slow initialization succeed, because the clock stopped long before the slow part began.&lt;/p&gt;

&lt;p&gt;If initialization matters, make it observable. Use &lt;code&gt;Type=notify&lt;/code&gt; and have the application send &lt;code&gt;READY=1&lt;/code&gt; through &lt;code&gt;sd_notify(3)&lt;/code&gt; when it can actually serve traffic. A notify service can also request more time with &lt;code&gt;EXTEND_TIMEOUT_USEC=&lt;/code&gt;, but the first extension has to arrive before the original deadline; it is an extension, not a reset.&lt;/p&gt;

&lt;p&gt;For &lt;code&gt;Type=oneshot&lt;/code&gt;, the start timeout is disabled by default. That is deliberate: batch jobs can legitimately run for hours. If you want a ceiling, set one explicitly.&lt;/p&gt;

&lt;p&gt;One more detail that catches people: the start timeout also applies to reloads, whether the reload runs through &lt;code&gt;ExecReload=&lt;/code&gt; or through the &lt;code&gt;Type=notify-reload&lt;/code&gt; logic. A slow reload can fail with the same timeout as a slow start.&lt;/p&gt;

&lt;h2&gt;
  
  
  What happens at the deadline
&lt;/h2&gt;

&lt;p&gt;When the start timeout expires, &lt;code&gt;TimeoutStartFailureMode=&lt;/code&gt; decides the response. It defaults to &lt;code&gt;terminate&lt;/code&gt;, and the alternatives are &lt;code&gt;abort&lt;/code&gt; and &lt;code&gt;kill&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;terminate&lt;/code&gt; sends &lt;code&gt;KillSignal=&lt;/code&gt; (SIGTERM by default). If the process still has not exited after &lt;code&gt;TimeoutStopSec=&lt;/code&gt;, systemd sends &lt;code&gt;FinalKillSignal=&lt;/code&gt; (SIGKILL by default).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;abort&lt;/code&gt; sends &lt;code&gt;WatchdogSignal=&lt;/code&gt; (SIGABRT by default) instead, waits for &lt;code&gt;TimeoutAbortSec=&lt;/code&gt;, then sends the final kill signal. It is useful when you want a core dump from a service that hangs during startup.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;kill&lt;/code&gt; sends the final kill signal immediately, with no grace period. It is for services you already know will not clean up.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is why a "90-second timeout" can occupy closer to three minutes: 90 seconds waiting for readiness, then up to another 90 seconds of grace before SIGKILL.&lt;/p&gt;

&lt;p&gt;The failure modes were added in systemd 246, so very old units will not have them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The stop timeout is two clocks in one setting
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;TimeoutStopSec=&lt;/code&gt; does double duty, and the documentation is explicit about both roles:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;It bounds &lt;strong&gt;each&lt;/strong&gt; &lt;code&gt;ExecStop=&lt;/code&gt; command. If one times out, the remaining &lt;code&gt;ExecStop=&lt;/code&gt; commands are skipped and the service is terminated with SIGTERM.&lt;/li&gt;
&lt;li&gt;It bounds &lt;strong&gt;the service's own shutdown&lt;/strong&gt; after that signal. If the process is still alive at the end, systemd sends SIGKILL, subject to &lt;code&gt;KillMode=&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So a unit with an &lt;code&gt;ExecStop=&lt;/code&gt; script and a process that ignores SIGTERM can consume up to two full stop timeouts: one for the script, one for the process. With the defaults, that is up to 180 seconds.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;KillMode=&lt;/code&gt; controls who receives the signals. The default, &lt;code&gt;control-group&lt;/code&gt;, sends SIGTERM to every process in the unit's cgroup and SIGKILL to whatever remains. &lt;code&gt;mixed&lt;/code&gt; sends SIGTERM only to the main process while still SIGKILLing the rest after the deadline. That is often the right choice for a supervisor that needs to shut its children down in order.&lt;/p&gt;

&lt;h2&gt;
  
  
  RuntimeMaxSec is a different limit
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;RuntimeMaxSec=&lt;/code&gt; caps how long a service may remain active, not how long it may take to start. When the limit is reached, the service is terminated and the unit enters a failure state. The default is &lt;code&gt;infinity&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It does not apply to &lt;code&gt;Type=oneshot&lt;/code&gt; services, because those are considered done as soon as &lt;code&gt;ExecStart=&lt;/code&gt; exits. For a batch job, use &lt;code&gt;TimeoutStartSec=&lt;/code&gt;; for a daemon that must recycle after a fixed interval, use &lt;code&gt;RuntimeMaxSec=&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fourth clock: job timeouts
&lt;/h2&gt;

&lt;p&gt;There is a separate layer above services. &lt;code&gt;JobTimeoutSec=&lt;/code&gt; starts when a job is queued; &lt;code&gt;JobRunningTimeoutSec=&lt;/code&gt; starts when the queued job actually begins running. Both default to &lt;code&gt;infinity&lt;/code&gt; and live in the &lt;code&gt;[Unit]&lt;/code&gt; section.&lt;/p&gt;

&lt;p&gt;If either limit is reached, systemd cancels the job, but the unit does not change state and does not enter &lt;code&gt;failed&lt;/code&gt; mode. That explains the confusing case where &lt;code&gt;systemctl status&lt;/code&gt; looks normal while the start command reported a job timeout. The usual cause is ordering: the unit is waiting behind a mount, a device, or another unit that never becomes ready.&lt;/p&gt;

&lt;p&gt;Useful commands when that happens:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl list-jobs
systemctl show -p JobTimeoutUSec -p JobRunningTimeoutUSec payments.service
systemd-analyze critical-chain payments.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For mounts, &lt;code&gt;x-systemd.device-timeout=&lt;/code&gt; in &lt;code&gt;/etc/fstab&lt;/code&gt; bounds the device wait separately, and &lt;code&gt;DefaultDeviceTimeoutSec=&lt;/code&gt; defaults to 90 seconds.&lt;/p&gt;

&lt;h2&gt;
  
  
  Drop-ins that match reality
&lt;/h2&gt;

&lt;p&gt;A slow-starting API that can signal readiness:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/payments.service.d/timeouts.conf
&lt;/span&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;notify&lt;/span&gt;
&lt;span class="na"&gt;TimeoutStartSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5min&lt;/span&gt;
&lt;span class="na"&gt;TimeoutStopSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2min&lt;/span&gt;
&lt;span class="na"&gt;TimeoutStartFailureMode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;terminate&lt;/span&gt;
&lt;span class="na"&gt;TimeoutStopFailureMode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;abort&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A nightly import that must be allowed to finish:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/nightly-import.service.d/timeouts.conf
&lt;/span&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;oneshot&lt;/span&gt;
&lt;span class="na"&gt;TimeoutStartSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2h&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A service with a slow, well-behaved shutdown:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/search.service.d/timeouts.conf
&lt;/span&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;TimeoutStopSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3min&lt;/span&gt;
&lt;span class="na"&gt;KillMode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;mixed&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Apply and verify:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo systemctl daemon-reload
systemctl show -p TimeoutStartUSec -p TimeoutStopUSec -p TimeoutAbortUSec payments.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For the first example, expect &lt;code&gt;5min&lt;/code&gt;, &lt;code&gt;2min&lt;/code&gt;, and &lt;code&gt;2min&lt;/code&gt;: the abort timeout falls back to the stop timeout because &lt;code&gt;DefaultTimeoutAbortSec=&lt;/code&gt; is unset. That is a quick way to confirm the drop-in was actually read.&lt;/p&gt;

&lt;h2&gt;
  
  
  A short diagnostic sequence
&lt;/h2&gt;

&lt;p&gt;When a unit times out and you are not sure which clock fired:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl status payments.service
systemctl show -p Result -p ActiveState -p SubState \
  -p TimeoutStartUSec -p TimeoutStopUSec payments.service
journalctl -u payments.service -b --no-pager &lt;span class="o"&gt;|&lt;/span&gt; tail -40
systemctl list-jobs
systemd-analyze verify /etc/systemd/system/payments.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;Result=timeout&lt;/code&gt; points at a start or stop deadline. A normal unit state plus a failed start command points at a job timeout. A dead unit with no useful journal lines often means the process died before it could log, which is a different problem.&lt;/p&gt;

&lt;p&gt;After fixing the unit, clear the failed state with &lt;code&gt;systemctl reset-failed payments.service&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rules of thumb
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Do not raise &lt;code&gt;TimeoutStartSec=&lt;/code&gt; on a &lt;code&gt;Type=simple&lt;/code&gt; service and call it fixed. The clock is not measuring initialization.&lt;/li&gt;
&lt;li&gt;Prefer &lt;code&gt;Type=notify&lt;/code&gt; when the application can say when it is ready. Prefer &lt;code&gt;Type=exec&lt;/code&gt; over &lt;code&gt;simple&lt;/code&gt; even when it cannot, because at least exec failures are detected.&lt;/li&gt;
&lt;li&gt;Set the start timeout from the slowest documented startup step, plus a margin. A timeout that is too generous hides hangs; one that is too tight causes avoidable restarts.&lt;/li&gt;
&lt;li&gt;Keep the stop timeout below whatever your deployment tool waits for before it gives up. A service that takes 180 seconds to stop while the deploy waits 60 is a service that gets killed by the deploy anyway.&lt;/li&gt;
&lt;li&gt;Remember that job timeouts are independent. If &lt;code&gt;systemctl status&lt;/code&gt; looks healthy but the start never returns, look at ordering and &lt;code&gt;systemctl list-jobs&lt;/code&gt;, not at &lt;code&gt;TimeoutStartSec=&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The 90-second default is reasonable for a simple daemon. It is a bad fit for a database, a model server, or a migration job. The fix is not to disable timeouts globally; it is to choose the clock that matches the phase you actually care about.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html" rel="noopener noreferrer"&gt;systemd.service(5)&lt;/a&gt; — &lt;code&gt;Type=&lt;/code&gt;, &lt;code&gt;TimeoutStartSec=&lt;/code&gt;, &lt;code&gt;TimeoutStopSec=&lt;/code&gt;, failure modes, &lt;code&gt;RuntimeMaxSec=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html" rel="noopener noreferrer"&gt;systemd-system.conf(5)&lt;/a&gt; — &lt;code&gt;DefaultTimeoutStartSec=&lt;/code&gt;, &lt;code&gt;DefaultTimeoutStopSec=&lt;/code&gt;, &lt;code&gt;DefaultTimeoutAbortSec=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html" rel="noopener noreferrer"&gt;systemd.unit(5)&lt;/a&gt; — &lt;code&gt;JobTimeoutSec=&lt;/code&gt;, &lt;code&gt;JobRunningTimeoutSec=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html" rel="noopener noreferrer"&gt;systemd.kill(5)&lt;/a&gt; — &lt;code&gt;KillMode=&lt;/code&gt;, &lt;code&gt;KillSignal=&lt;/code&gt;, &lt;code&gt;FinalKillSignal=&lt;/code&gt;, &lt;code&gt;WatchdogSignal=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemctl.html" rel="noopener noreferrer"&gt;systemctl(1)&lt;/a&gt; and &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/journalctl.html" rel="noopener noreferrer"&gt;journalctl(1)&lt;/a&gt; — inspecting effective values and logs&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>linux</category>
      <category>systemd</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Stop Editing /etc/resolv.conf: Practical systemd-resolved on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Thu, 12 Nov 2026 05:04:13 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-editing-etcresolvconf-practical-systemd-resolved-on-linux-dbf</link>
      <guid>https://dev.to/lyraalishaikh/stop-editing-etcresolvconf-practical-systemd-resolved-on-linux-dbf</guid>
      <description>&lt;p&gt;You SSH into a box, DNS is broken, and you do what everyone does: add a &lt;code&gt;nameserver&lt;/code&gt; line to &lt;code&gt;/etc/resolv.conf&lt;/code&gt;. It works. Then the machine reboots, the DHCP lease renews, or a VPN connects, and the file is back to this:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nameserver 127.0.0.53
options edns0 trust-ad
search corp.example.com
~~~

Nothing you wrote survived. On a system running `systemd-resolved`, `/etc/resolv.conf` is not configuration. It is output. The real configuration lives in `/etc/systemd/resolved.conf`, in per-link `.network` files, in DHCP leases, and in the resolver's runtime state. Editing the generated file is like editing a log to change a program's behavior.

That is the small version of the problem. The larger version is that `systemd-resolved` is not just a caching DNS forwarder. It is a per-link query router, and most "DNS is broken" incidents come from misunderstanding which link and which server the resolver chose.

Here is the model, the failure modes, and a debugging sequence that tells you what actually happened.

## The daemon in the middle

`systemd-resolved` provides a caching, DNSSEC-validating stub resolver, plus LLMNR and MulticastDNS support. Local programs can reach it through three different interfaces:

1. **The native D-Bus or Varlink API**, which returns rich metadata such as which link answered and whether the data was DNSSEC-authenticated.
2. **glibc's NSS resolver**, through the `nss-resolve` module. This is how ordinary `getaddrinfo()` calls reach the resolver.
3. **A DNS stub listener** on `127.0.0.53` and `127.0.0.54`, port 53. This exists for programs that speak raw DNS and bypass NSS.

The two stub addresses are not identical. `127.0.0.53` is the full resolver, including LLMNR and mDNS. `127.0.0.54` is a proxy-only listener: it forwards messages to upstream servers largely unmodified, does not validate DNSSEC, and does not answer LLMNR or mDNS. It exists for compatibility with software that needs a plain forwarding DNS server.

Check what is listening before you debug anything else:

~~~bash
systemctl status systemd-resolved --no-pager
sudo ss -lntup 'sport = :53'
~~~

On a working host you should see `127.0.0.53:53` and `127.0.0.54:53`. If another process owns port 53, read on. That is one of the classic failures.

## Four ways /etc/resolv.conf can exist

The `systemd-resolved.service` manual documents four supported modes. The mode is detected automatically, mostly by what `/etc/resolv.conf` points to.

- **Symlink to `/run/systemd/resolve/stub-resolv.conf`.** The file contains `nameserver 127.0.0.53` plus the search domains currently in use. This is the recommended mode. It is generated and updated by the resolver.
- **Symlink to `/usr/lib/systemd/resolv.conf`.** A static file containing only the stub address. No search domains. Useful in minimal images, less useful on a laptop that moves between networks.
- **Symlink to `/run/systemd/resolve/resolv.conf`.** This file lists all known upstream DNS servers. Programs that read it talk directly to those servers and bypass `systemd-resolved` entirely. You lose the cache, DNSSEC validation, and per-link routing.
- **A regular file managed by something else.** In this mode `systemd-resolved` is a consumer, not the provider. It reads the file to discover DNS servers, but it does not manage it.

That last point is the one people miss. When `/etc/resolv.conf` is a symlink to the stub file, `systemd-resolved` ignores its contents as input. Adding a `nameserver` there does not tell the resolver to use that server. It only changes what other programs read, and the file will be regenerated.

Find out which mode you are in:

~~~bash
readlink -f /etc/resolv.conf
resolvectl status | head -5
~~~

The status output includes a line like `resolv.conf mode: stub` or `resolv.conf mode: foreign`. Believe that line over whatever you remember doing to the machine six months ago.

## Where the settings actually live

There are four layers, and they are not equal:

**Global configuration** lives in `/etc/systemd/resolved.conf` and in drop-ins under `/etc/systemd/resolved.conf.d/*.conf`. This is where you set system-wide DNS servers, DNSSEC, DNS-over-TLS, and cache behavior.

**Per-link static configuration** lives in `/etc/systemd/network/*.network` files when `systemd-networkd` manages the interface. The relevant settings are in the `[Network]` section: `DNS=`, `Domains=`, `DNSDefaultRoute=`, `DNSSEC=`, `DNSOverTLS=`, and `DNSSECNegativeTrustAnchors=`. In the `[DHCP]` section, `UseDNS=no` tells networkd to ignore DNS servers offered by DHCP.

**Per-link dynamic configuration** comes from DHCP, IPv6 router advertisements, NetworkManager, or other services. This is why a VPN can quietly change your DNS behavior without touching a file you edited.

**Runtime configuration** comes from `resolvectl dns`, `resolvectl domain`, and friends. It is not written to disk. It disappears on reboot or when the link disappears. It is excellent for testing and terrible as a permanent fix.

A practical primary-link configuration looks like this:

~~~ini
# /etc/systemd/network/10-lan.network
[Match]
Name=eth0

[Network]
DHCP=yes
DNS=192.0.2.53
Domains=~corp.example.com
DNSDefaultRoute=yes

[DHCP]
UseDNS=no
~~~

`UseDNS=no` is doing real work here. Without it, the DHCP-provided DNS servers are added alongside the static one, and the resolver may prefer a server you did not intend. With it, the link uses `192.0.2.53` and nothing else.

A VPN link that should only receive corporate queries looks different:

~~~ini
# /etc/systemd/network/90-wg0.network
[Match]
Name=wg0

[Network]
DNS=10.20.0.53
Domains=~corp.example.com ~20.10.in-addr.arpa
DNSDefaultRoute=no
~~~

Apply networkd changes with:

~~~bash
sudo networkctl reload
sudo networkctl reconfigure wg0
~~~

For a quick experiment, the runtime equivalent is:

~~~bash
sudo resolvectl dns wg0 10.20.0.53
sudo resolvectl domain wg0 '~corp.example.com' '~20.10.in-addr.arpa'
sudo resolvectl default-route wg0 no
~~~

Then `resolvectl revert wg0` puts the link back to its discovered settings. Do not stop at this step and call it a fix; it will not survive the next link reset.

## How a query picks a server

This is the part worth learning properly. The rules are documented, deterministic, and simpler than the folklore.

For a unicast DNS query, `systemd-resolved` does this:

1. If the name matches a configured routing domain, pick the **best match** — the domain with the most labels. Send the query to all DNS servers on every link that has that best-matching domain.
2. If no domain matches, send the query to all links marked as a default route, plus any globally configured DNS servers.
3. If there are no default-route links and no global servers, use the compiled-in fallback servers.
4. Otherwise, fail. There is no suitable server.

Search domains and route-only domains both participate in routing. The tilde is what changes the behavior:

- `Domains=corp.example.com` is a **search domain**. It is appended to single-label names, and it also routes matching queries to this link.
- `Domains=~corp.example.com` is a **route-only domain**. It routes matching queries to this link but is not used as a search suffix.
- `Domains=~.` is the root route-only domain. It claims every name that is not matched by a more specific domain. Use it when you want one link to be the preferred resolver for everything else.

`DNSDefaultRoute=` controls the fallback. If you do not set it, the automatic rule is: a link with a route-only domain other than `~.` defaults to `no`; a link without such a domain defaults to `yes`. That automatic rule is responsible for a lot of accidental VPN DNS leaks. A link configured with the search domain `corp.example.com` and no route-only domain becomes a default route, so it receives queries for the public internet too. Add the tilde, or set `DNSDefaultRoute=no` explicitly, and the leak stops.

When multiple links carry the same best-matching domain, the query is sent to all of them in parallel and the first successful answer wins. That is why two DNS servers in the same lookup scope must serve equivalent data. The resolver is not doing a vote; it is doing a race.

## The search-domain trap that breaks muscle memory

The `systemd-resolved` manual has a compatibility section, and one paragraph deserves to be quoted:

&amp;gt; Search domains are not used for suffixing of multi-label names. Any name with at least one dot is always interpreted as a FQDN.

And:

&amp;gt; This stub resolver does not implement this at all: multi-label names are only resolved as FQDNs.

If your search domain is `corp.example.com`, then:

~~~bash
getent hosts db          # may try db.corp.example.com
getent hosts db.prod     # queries db.prod. as-is, no suffix
~~~

The second command does not try `db.prod.corp.example.com`. Traditional glibc behavior, with `ndots` and a search list, often would. Here, `ndots` is not implemented. If you rely on `ssh db.prod` resolving through a search suffix, it will fail even though `ssh db` works.

Single-label names have a separate rule. By default, A and AAAA queries for a single-label name are not sent to unicast DNS at all. They are tried through LLMNR, and then through search domains if any are configured. `ResolveUnicastSingleLabel=yes` changes that and forwards them upstream, but the manual warns that this is non-standard and can leak internal names to servers you do not control. Prefer a search domain or an FQDN.

The `.local` suffix has its own rule: it is reserved for MulticastDNS by RFC 6762. Queries for `.local` names are not routed to unicast DNS unless you explicitly configure that domain as a search or routing domain. Do not put your company's internal namespace under `.local` and then wonder why the resolver refuses to ask a normal DNS server about it.

## DNSSEC and DNS-over-TLS without fairy tales

Two defaults matter:

- `DNSSEC=` defaults to `allow-downgrade`. Validation is attempted, but if the server does not support DNSSEC properly, the resolver turns validation off. This is pragmatic and deliberately vulnerable to downgrade attacks.
- `DNSOverTLS=` defaults to `no`. DNS is sent over UDP.

You can see the effective state per link in `resolvectl status`, in a line like `DNSSEC=no/unsupported` or `DNSSEC=allow-downgrade/supported`. The first half is the mode; the second half is whether the current server appears to support it.

If you set `DNSSEC=yes`, every lookup must validate. That is the point, but it has consequences:

- A private zone that is not signed will fail validation. The resolver returns a failure, not an unsigned answer.
- A broken or outdated trust anchor can make the whole internet fail validation.
- A server that does not support DNSSEC will produce failures for every name, not a quiet fallback.

For a private, unsigned zone, do not disable DNSSEC globally. Add a negative trust anchor for that subtree, either per link:

~~~ini
# inside the [Network] section of the corporate link
DNSSECNegativeTrustAnchors=corp.example.com
~~~

or globally in a file:

~~~text
# /etc/dnssec-trust-anchors.d/corp.negative
corp.example.com
~~~

Then restart the resolver. Negative trust anchors tell the validator "do not require authentication below this name." They are a scalpel, not a global off switch.

Strict DNS-over-TLS has a similar shape. This configuration is intentionally strict:

~~~ini
# /etc/systemd/resolved.conf.d/dot.conf
[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net
FallbackDNS=
DNSOverTLS=yes
DNSSEC=yes
~~~

The `#hostname` part is not decoration. It sets the TLS Server Name Indication and gives the resolver a name to validate the certificate against. Without it, the certificate is checked against the server's IP address, which usually fails for public resolvers.

`DNSOverTLS=yes` is strict: if a server does not support TLS, queries to it fail. `opportunistic` tries TLS and falls back to plaintext when the server does not support it, which is exactly the downgrade you were trying to avoid. Use `opportunistic` only when you value compatibility over authentication.

The global `DNSOverTLS=` also applies to per-link servers unless the link overrides it. If DHCP hands you a resolver that does not speak TLS, strict mode will fail on that link. Either stop accepting DHCP DNS (`UseDNS=no`) or set `DNSOverTLS=no` on that specific link.

## The failures you will actually hit

**1. The stub is silently disabled by a port conflict.** If something else already listens on `127.0.0.53:53` or `127.0.0.54:53`, `systemd-resolved` turns its stub listener off. `/etc/resolv.conf` still points at `127.0.0.53`, so every traditional program times out while `resolvectl query` may still work through the native API. Check with `sudo ss -lntup 'sport = :53'` and `journalctl -u systemd-resolved -b`. The fix is to stop the other resolver or disable the stub deliberately with `DNSStubListener=no` and point `/etc/resolv.conf` somewhere that actually answers.

**2. NSS is not using the resolver.** If `/etc/nsswitch.conf` does not list `resolve` on the `hosts:` line, glibc reads `/etc/resolv.conf` and uses `nss-dns`. In stub mode that still reaches the stub, but you lose the native API's metadata and you are depending on the generated file being correct. Check the line:

~~~bash
grep '^hosts' /etc/nsswitch.conf
~~~

A working line puts `resolve` before `dns`, for example:

~~~text
hosts: files myhostname resolve [!UNAVAIL=return] dns
~~~

**3. You edited the symlink target.** If `/etc/resolv.conf` points at `/run/systemd/resolve/stub-resolv.conf`, your edit went into `/run` and will be overwritten. This is not a permissions problem. It is the wrong file.

**4. Split DNS was configured as a search domain.** `Domains=corp.example.com` routes corporate queries to the VPN and also makes the VPN a default route under the automatic rule. Public queries can end up at the corporate resolver. Use `~corp.example.com` and, when the link should never be a fallback, `DNSDefaultRoute=no`.

**5. Strict DNSSEC met an unsigned private zone.** The resolver returns validation failures for names that used to work. Add a negative trust anchor for the private subtree, or set `DNSSEC=no` on that one link. Do not turn off validation globally because one internal zone is unsigned.

**6. Strict DNS-over-TLS met a DHCP server that cannot do TLS.** The link fails, often only after a network change. Either use explicit `DNS=` entries with `#hostname` and disable DHCP DNS on that link, or relax `DNSOverTLS=` for that link.

**7. A container got a copy of the host's resolv.conf.** Inside a container, `127.0.0.53` is the container's loopback, not the host's resolver. Container runtimes normally generate a container-appropriate file; copying the host file into an image or bind-mounting it breaks DNS in a way that looks like a network outage. Let the container runtime manage it, or point the container at a reachable resolver.

## A debugging sequence that narrows the path

Start with the question "which path is failing?" There are three paths into resolution, and they fail differently.

~~~bash
# Native API: shows the chosen link and richer metadata
resolvectl query --cache=no --legend=no internal.example.com

# NSS: what ordinary applications see
getent hosts internal.example.com

# Raw DNS to the stub: what a program that ignores NSS sees
dig @127.0.0.53 internal.example.com
~~~

The decision table is short:

- `resolvectl` works, `getent` fails: fix NSS (`/etc/nsswitch.conf`, `nss-resolve`).
- `resolvectl` works, `dig @127.0.0.53` fails: the stub listener is not running or is not reachable. Check the port conflict and the unit logs.
- All three fail: the query is not reaching a working upstream server. Check routing, `resolvectl status`, and the selected link.
- `resolvectl` fails with `--cache=no` but succeeds normally: you are looking at a cache or stale-data problem, not an upstream problem.

A successful native query looks like this (trimmed from a Debian 13 host running systemd 257.13):

~~~text
$ resolvectl query --legend=no deb.debian.org
deb.debian.org: 2a04:4e42:400::644                          -- link: eth0
                2a04:4e42:600::644                          -- link: eth0
                151.101.2.132                               -- link: eth0
                (debian.map.fastlydns.net)
~~~

The `-- link:` annotation is the useful part. It tells you which interface's DNS configuration produced the answer. If that is the wrong link, the problem is routing, not the upstream server.

Then inspect the policy:

~~~bash
resolvectl status
resolvectl domain
~~~

A representative status block looks like this:

~~~text
Global
         Protocols: +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.0.2.53
       DNS Servers: 192.0.2.53 192.0.2.54
        DNS Domain: corp.example.com
     Default Route: yes
~~~

Read it in this order: `resolv.conf mode`, `Current DNS Server`, `DNS Domain`, `Default Route`. Those four lines explain most routing surprises.

For cache and server behavior:

~~~bash
sudo resolvectl statistics          # hit/miss and validation counters
sudo resolvectl show-cache          # systemd 254+
sudo resolvectl show-server-state   # systemd 255+
sudo resolvectl monitor             # systemd 252+, live local query stream
sudo resolvectl flush-caches
~~~

`flush-caches` is not a universal fix. It clears the resolver's cache, not your browser's, not a local dnsmasq, and not an application's own cache. The resolver already flushes automatically when the network configuration changes; manual flushing is for debugging and for the rare case where an upstream changed a record early.

If the counters do not move when you query, you are not talking to this resolver. Go back to the NSS and stub checks.

For the resolver's own view of what went wrong:

~~~bash
sudo journalctl -u systemd-resolved -b --no-pager
sudo resolvectl log-level debug
# reproduce the problem
sudo resolvectl log-level info
~~~

Debug logging is verbose and can include query names. Turn it back down when you are done.

## A clean cutover

If you are moving a host from a hand-managed `resolv.conf` to `systemd-resolved`, do it in a way you can reverse.

~~~bash
# Keep the old file (or symlink) around
sudo cp -a /etc/resolv.conf /root/resolv.conf.bak

# Start the resolver and point the compatibility file at the stub
sudo systemctl enable --now systemd-resolved
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

# Verify the mode and a real lookup
resolvectl status | head -5
resolvectl query --cache=no example.com
getent hosts example.com
~~~

If NetworkManager owns the interfaces, tell it to use the `systemd-resolved` DNS plugin instead of writing its own file:

~~~ini
# /etc/NetworkManager/conf.d/90-dns.conf
[main]
dns=systemd-resolved
~~~

Then reload it:

~~~bash
sudo systemctl reload NetworkManager
~~~

If `resolvectl status` reports `resolv.conf mode: foreign`, the symlink did not take effect — something else is rewriting the file. Find that manager before you fight it. Two programs writing `/etc/resolv.conf` is not a configuration; it is a race.

## When not to use it

`systemd-resolved` is a stub resolver and router. It is not a full recursive resolver. If you run Unbound on `127.0.0.1` because you want local recursion and your own root hints, do not point clients at both resolvers. Pick one place to own the query path. If Unbound binds `0.0.0.0:53`, it will also collide with the stub listener.

If you need a DHCP server, use dnsmasq or another DHCP daemon. If you need a full-featured authoritative server, use Knot, NSD, or BIND. The mistake is treating `systemd-resolved` as a replacement for all of them because it happens to answer DNS queries.

## The short checklist

- `/etc/resolv.conf` is generated when it is a symlink into `/run/systemd/resolve/`. Do not edit it.
- Configuration belongs in `resolved.conf` drop-ins, `.network` files, or the system that owns the link.
- `~domain` is route-only; a bare `domain` is search plus route and can make a link a default route.
- Multi-label names are never suffixed with search domains. `ndots` is not implemented.
- Single-label names do not go to unicast DNS without a search domain or an explicit override.
- `DNSSEC=yes` fails closed; use a negative trust anchor for unsigned private zones.
- `DNSOverTLS=yes` needs `#hostname` for certificate validation, and it applies to DHCP-provided servers too.
- Diagnose with `resolvectl query --cache=no`, `resolvectl status`, `getent hosts`, and `dig @127.0.0.53` before changing anything.

The next time DNS breaks, resist the urge to open `/etc/resolv.conf`. Run `resolvectl status` first. The answer is usually in the link that should not have been a default route, or in the search domain that was never a search domain at all.

## References

- [systemd-resolved.service(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-resolved.service.html) — stub listeners, synthetic records, routing rules, `/etc/resolv.conf` modes, glibc compatibility
- [resolved.conf(5)](https://www.freedesktop.org/software/systemd/man/latest/resolved.conf.html) — `DNS=`, `Domains=`, `DNSSEC=`, `DNSOverTLS=`, cache settings
- [resolvectl(1)](https://www.freedesktop.org/software/systemd/man/latest/resolvectl.html) — `query`, `status`, `statistics`, `monitor`, `show-cache`, per-link commands
- [systemd.network(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.network.html) — `DNS=`, `Domains=`, `DNSDefaultRoute=`, `UseDNS=`, `DNSSECNegativeTrustAnchors=`
- [dnssec-trust-anchors.d(5)](https://www.freedesktop.org/software/systemd/man/latest/dnssec-trust-anchors.d.html) — positive and negative trust anchors
- [RFC 6762](https://www.rfc-editor.org/rfc/rfc6762) — Multicast DNS and the `.local` namespace
- [RFC 4795](https://www.rfc-editor.org/rfc/rfc4795) — LLMNR
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>linux</category>
      <category>networking</category>
      <category>devops</category>
      <category>systemd</category>
    </item>
    <item>
      <title>Stop Copying authorized_keys to Every Host: Practical OpenSSH Certificates on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Wed, 11 Nov 2026 05:02:57 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-copying-authorizedkeys-to-every-host-practical-openssh-certificates-on-linux-424e</link>
      <guid>https://dev.to/lyraalishaikh/stop-copying-authorizedkeys-to-every-host-practical-openssh-certificates-on-linux-424e</guid>
      <description>&lt;p&gt;Every small fleet reaches the same moment: you add a new engineer, and then you copy a public key into &lt;code&gt;authorized_keys&lt;/code&gt; on eleven machines. Someone leaves, and you repeat the ritual in reverse, hoping you remembered the jump box nobody documents. Host keys are the other half of the problem — every rebuild produces a new fingerprint and another &lt;code&gt;WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;OpenSSH certificates fix both problems without replacing SSH. Instead of every server trusting every user key, every server trusts one certificate authority (CA) key. The CA signs short-lived certificates that bind a public key to a username, a set of permissions, and an expiry time. The server checks the signature and the identity, and never needs a per-user file again.&lt;/p&gt;

&lt;p&gt;This is not X.509. OpenSSH certificates are a much simpler format, and the operational details are different enough that most people bounce off them once. Here is the mental model, a working lab, the server configuration, host certificates, and revocation that actually works.&lt;/p&gt;

&lt;h2&gt;
  
  
  What an OpenSSH certificate contains
&lt;/h2&gt;

&lt;p&gt;A certificate is a public key plus signed metadata. When you inspect one with &lt;code&gt;ssh-keygen -L&lt;/code&gt;, you see:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Type&lt;/strong&gt; — user or host certificate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Public key&lt;/strong&gt; — the key being certified, not a new key.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Signing CA&lt;/strong&gt; — which CA key signed it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Key ID&lt;/strong&gt; — a human-readable label written to server logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Serial&lt;/strong&gt; — a number you can use for revocation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Valid&lt;/strong&gt; — start and end times.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Principals&lt;/strong&gt; — the usernames (or hostnames) this certificate may authenticate as.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Critical Options&lt;/strong&gt; — restrictions the server must understand, or it rejects the certificate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Extensions&lt;/strong&gt; — permissions such as &lt;code&gt;permit-pty&lt;/code&gt; or &lt;code&gt;permit-agent-forwarding&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The private key never leaves the client. The certificate is a separate &lt;code&gt;*-cert.pub&lt;/code&gt; file. The CA key is an ordinary SSH key; there is no special key type and no certificate hierarchy. Clients and servers trust the CA public key directly, which means there are no intermediate CAs to manage — and no chain to constrain a compromised CA.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two defaults that make people give up
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;ssh-keygen(1)&lt;/code&gt; manual says: “By default, generated certificates are valid for all users or hosts.” That sounds convenient. It is also a trap, because &lt;code&gt;sshd_config(5)&lt;/code&gt; says certificates that lack a list of principals “will not be permitted for authentication using TrustedUserCAKeys.”&lt;/p&gt;

&lt;p&gt;So a certificate signed like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen -s user_ca -I alice alice.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;is valid forever, allows every forwarding permission OpenSSH knows about, and is rejected by a correctly configured server because it has no principals. Always pass &lt;code&gt;-n&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen -s user_ca -I alice -n alice -V +12h alice.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second default is equally sharp: without &lt;code&gt;-V&lt;/code&gt;, the certificate is valid from the Unix epoch to the distant future. Without &lt;code&gt;-O clear&lt;/code&gt;, it grants X11 forwarding, agent forwarding, port forwarding, PTY allocation, and &lt;code&gt;~/.ssh/rc&lt;/code&gt; execution. A default certificate is a permanent, fully privileged credential. Treat &lt;code&gt;-V&lt;/code&gt; and &lt;code&gt;-O clear&lt;/code&gt; as mandatory, not optional hardening.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lab: a user CA in five commands
&lt;/h2&gt;

&lt;p&gt;Run this on a machine you control. The CA private key should not live on the servers you log in to.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;install -d -m 700 ~/.ssh/ca
ssh-keygen -t ed25519 -a 100 -C &lt;span class="s2"&gt;"user-ca@example.com"&lt;/span&gt; -f ~/.ssh/ca/user_ca
ssh-keygen -t ed25519 -a 100 -C &lt;span class="s2"&gt;"alice@laptop"&lt;/span&gt; -f ~/.ssh/id_ed25519
ssh-keygen -s ~/.ssh/ca/user_ca \
  -I &lt;span class="s2"&gt;"alice@laptop-2026-11-11"&lt;/span&gt; \
  -n alice \
  -V +12h \
  -z 1001 \
  -O clear -O permit-pty \
  ~/.ssh/id_ed25519.pub
ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;-I&lt;/code&gt; value is the key ID that lands in the server log. The &lt;code&gt;-n&lt;/code&gt; value must match the Unix account, unless you add a principals file later. &lt;code&gt;-z&lt;/code&gt; gives the certificate a serial number for revocation. &lt;code&gt;-O clear&lt;/code&gt; removes the default forwarding permissions, and &lt;code&gt;-O permit-pty&lt;/code&gt; adds back only what an interactive shell needs.&lt;/p&gt;

&lt;p&gt;On OpenSSH 10.0p2, the inspection output looks like this (fingerprints shortened):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;alice-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:itwADN+Ol2Tk...
        Signing CA: ED25519 SHA256:HDzAN1TPyjP...
        Key ID: "alice@laptop-2026-11"
        Serial: 1001
        Valid: from 2026-11-11T05:00:00 to 2026-11-11T17:01:20
        Principals:
                alice
        Critical Options:
                source-address 203.0.113.0/24
        Extensions:
                permit-pty
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That example also used &lt;code&gt;-O source-address=203.0.113.0/24&lt;/code&gt;. It is useful for a fixed office or VPN range, but it is a poor fit behind carrier NAT or a load balancer that changes the apparent source address. The server evaluates the address it sees, not the address the client thinks it has.&lt;/p&gt;

&lt;h3&gt;
  
  
  Critical options versus extensions
&lt;/h3&gt;

&lt;p&gt;The distinction matters when you write your own policies. An unknown &lt;strong&gt;critical option&lt;/strong&gt; causes the certificate to be refused. An unknown &lt;strong&gt;extension&lt;/strong&gt; is ignored. Use critical options for restrictions that must fail closed, and extensions for capabilities that older servers may safely skip.&lt;/p&gt;

&lt;h2&gt;
  
  
  Server side: trust the CA, not the user
&lt;/h2&gt;

&lt;p&gt;Copy only the CA public key to the server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo install -o root -g root -m 644 ~/.ssh/ca/user_ca.pub /etc/ssh/user_ca.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Add one line to &lt;code&gt;/etc/ssh/sshd_config&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TrustedUserCAKeys /etc/ssh/user_ca.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Validate and reload:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo sshd -t
sudo systemctl reload ssh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On RHEL-family systems the unit is usually &lt;code&gt;sshd&lt;/code&gt; instead of &lt;code&gt;ssh&lt;/code&gt;. Then log in normally:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh -i ~/.ssh/id_ed25519 alice@server.example.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;OpenSSH automatically loads a certificate named by appending &lt;code&gt;-cert.pub&lt;/code&gt; to the identity file, so keeping &lt;code&gt;id_ed25519&lt;/code&gt; and &lt;code&gt;id_ed25519-cert.pub&lt;/code&gt; side by side is enough. If the files have different names, point at the certificate explicitly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh -i ~/.ssh/id_ed25519 -o &lt;span class="nv"&gt;CertificateFile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;~/.ssh/alice-cert.pub alice@server.example.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use &lt;code&gt;ssh -vvv&lt;/code&gt; when it fails. The client prints which keys and certificates it offered, and the server log records the accepted key ID. A successful login usually appears as &lt;code&gt;Accepted publickey&lt;/code&gt; with the certificate identity in the log line. The important part is that &lt;code&gt;authorized_keys&lt;/code&gt; was never consulted.&lt;/p&gt;

&lt;h2&gt;
  
  
  Principals are not usernames
&lt;/h2&gt;

&lt;p&gt;By default, the certificate's principal must equal the target Unix username. That is fine for a small team, but it couples every certificate to an account name. &lt;code&gt;AuthorizedPrincipalsFile&lt;/code&gt; decouples them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Create the directory and a file per account:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo install -d -o root -g root -m 755 /etc/ssh/auth_principals
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'alice-admin'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sudo tee /etc/ssh/auth_principals/alice
sudo chmod 644 /etc/ssh/auth_principals/alice
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now a certificate signed with &lt;code&gt;-n alice-admin&lt;/code&gt; can log in as &lt;code&gt;alice&lt;/code&gt;, while a certificate with principal &lt;code&gt;bob-admin&lt;/code&gt; cannot. The &lt;code&gt;%u&lt;/code&gt; token expands to the target username. If the file exists and the certificate has no matching principal, authentication fails even though the CA signature is valid.&lt;/p&gt;

&lt;p&gt;For a directory-backed fleet, &lt;code&gt;AuthorizedPrincipalsCommand&lt;/code&gt; can generate the list dynamically. The command must be owned by root and not writable by group or others, and &lt;code&gt;sshd&lt;/code&gt; refuses to start if you set the command without &lt;code&gt;AuthorizedPrincipalsCommandUser&lt;/code&gt;. That is a deliberate safety check, not a suggestion.&lt;/p&gt;

&lt;h2&gt;
  
  
  Host certificates: stop trusting fingerprints one by one
&lt;/h2&gt;

&lt;p&gt;User certificates solve login sprawl. Host certificates solve the other half: clients no longer pin one host key per machine.&lt;/p&gt;

&lt;p&gt;On the host, sign its existing public host key with the host CA:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo ssh-keygen -s /root/ca/host_ca \
  -h \
  -I &lt;span class="s2"&gt;"web01-2026-11"&lt;/span&gt; \
  -n web01.example.com,10.0.0.21 \
  -V +52w \
  /etc/ssh/ssh_host_ed25519_key.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That writes &lt;code&gt;/etc/ssh/ssh_host_ed25519_key-cert.pub&lt;/code&gt;. Add it to &lt;code&gt;sshd_config&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The certificate's public key must match a private host key already configured with &lt;code&gt;HostKey&lt;/code&gt;. The private host key is still required; the certificate only proves that the host key was signed by your CA.&lt;/p&gt;

&lt;p&gt;On clients, add the host CA to the known-hosts file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@cert-authority *.example.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Put that line in &lt;code&gt;/etc/ssh/ssh_known_hosts&lt;/code&gt; for a fleet-wide default, or in &lt;code&gt;~/.ssh/known_hosts&lt;/code&gt; for one user. The hostnames pattern is matched against the name the client used, so include every name and address people actually type in the certificate's &lt;code&gt;-n&lt;/code&gt; list. If clients reach the host as &lt;code&gt;web01&lt;/code&gt;, &lt;code&gt;web01.example.com&lt;/code&gt;, and &lt;code&gt;10.0.0.21&lt;/code&gt;, sign all three.&lt;/p&gt;

&lt;p&gt;This is what removes the rebuild warning. A replacement host presents a new host key with a valid certificate from the same CA, and the client accepts it without a manual fingerprint update. It also means the CA can issue a host certificate for a name the host does not own — the DNS and network still have to be right, but the cryptographic identity is centralized.&lt;/p&gt;

&lt;h2&gt;
  
  
  Revocation that actually works
&lt;/h2&gt;

&lt;p&gt;Short validity is the primary control. A twelve-hour certificate cannot be used next month, even if nobody remembers to revoke it. Revocation is the emergency brake for a key that must die before it expires.&lt;/p&gt;

&lt;p&gt;Generate a Key Revocation List (KRL) from a public key:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen -k -f revoked.krl alice.pub
ssh-keygen -k -u -f revoked.krl bob.pub
ssh-keygen -Q -f revoked.krl alice.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;ssh-keygen -Q&lt;/code&gt; exits with a non-zero status when the key is revoked, which makes it usable in a deployment check. The KRL is a compact binary format; revoking by serial number can take as little as one bit per certificate.&lt;/p&gt;

&lt;p&gt;You can also revoke by serial or key ID without keeping the original certificate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'serial:1001\n'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; revoke.spec
ssh-keygen -k -f revoked.krl -s user_ca.pub revoke.spec

&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'id:alice@laptop-2026-11\n'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; revoke-id.spec
ssh-keygen -k -u -f revoked.krl -s user_ca.pub revoke-id.spec
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Serial zero is reserved and cannot be revoked by serial number, so give every certificate a non-zero &lt;code&gt;-z&lt;/code&gt; value if you plan to use serial-based revocation. To revoke by serial or key ID, the CA key must be supplied with &lt;code&gt;-s&lt;/code&gt;, as shown above.&lt;/p&gt;

&lt;p&gt;Enable it on the server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RevokedKeys /etc/ssh/revoked_keys
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then distribute the KRL to that path. One operational warning from the manual: if the revocation file is unreadable, public key authentication is refused for all users. A missing or mispermissioned KRL is an outage, not a silent no-op. Check it during deployment.&lt;/p&gt;

&lt;p&gt;Host keys can be revoked client-side with an &lt;code&gt;@revoked&lt;/code&gt; line in &lt;code&gt;known_hosts&lt;/code&gt;, but a KRL on the server is the usual tool for user certificates.&lt;/p&gt;

&lt;p&gt;Revocation does not terminate sessions that are already established. It applies the next time the key is presented.&lt;/p&gt;

&lt;h2&gt;
  
  
  Automation: keep the CA key off the servers
&lt;/h2&gt;

&lt;p&gt;The CA private key is the crown jewel. Anyone who has it can mint a certificate for any principal, so treat it like a root signing key: offline, passphrase-protected, or in an HSM. Do not copy it to every host just because the public key is safe to distribute.&lt;/p&gt;

&lt;p&gt;For a signing service, load the CA key into an agent and sign with the public half:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-add ~/.ssh/ca/user_ca
ssh-keygen -Us ~/.ssh/ca/user_ca.pub \
  -I &lt;span class="s2"&gt;"$USER@$(hostname)-$(date -u +%Y%m%dT%H%MZ)"&lt;/span&gt; \
  -n &lt;span class="s2"&gt;"$USER"&lt;/span&gt; \
  -V +8h \
  -O clear -O permit-pty \
  ~/.ssh/id_ed25519.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;-U&lt;/code&gt; flag tells &lt;code&gt;ssh-keygen&lt;/code&gt; that the CA key is held by &lt;code&gt;ssh-agent&lt;/code&gt;; the public key identifies which key to use. For a PKCS#11 token or HSM, &lt;code&gt;-D&lt;/code&gt; points at the provider library and &lt;code&gt;-s&lt;/code&gt; identifies the CA public key. Both keep the private key out of the signing host's filesystem, though an agent or token that will sign anything is still a high-value target.&lt;/p&gt;

&lt;p&gt;A useful CI pattern is a certificate that can do exactly one thing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen -s user_ca \
  -I &lt;span class="s2"&gt;"deploy-$CI_JOB_ID"&lt;/span&gt; \
  -n deploy \
  -V +15m \
  -O clear \
  -O force-command=&lt;span class="s1"&gt;'/usr/local/bin/deploy-only'&lt;/span&gt; \
  deploy.pub
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;force-command&lt;/code&gt; option is signed into the certificate, so the client cannot override it. Combine it with a dedicated account and short validity, and a leaked CI key has a narrow, auditable blast radius.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure modes worth memorizing
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Permission denied (publickey)&lt;/code&gt; and &lt;code&gt;ssh-keygen -L&lt;/code&gt; shows &lt;code&gt;Principals: (none)&lt;/code&gt;&lt;/strong&gt; — you signed without &lt;code&gt;-n&lt;/code&gt;. The CA is trusted, but &lt;code&gt;sshd&lt;/code&gt; will not accept a principal-less certificate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Certificate expired or not yet valid&lt;/strong&gt; — check the &lt;code&gt;Valid:&lt;/code&gt; line and the client/server clocks. A certificate with a future start time fails immediately.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Principal mismatch&lt;/strong&gt; — the certificate principal must equal the target username, or appear in &lt;code&gt;AuthorizedPrincipalsFile&lt;/code&gt;. A role principal only works if the file lists it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CA not trusted&lt;/strong&gt; — verify the path in &lt;code&gt;TrustedUserCAKeys&lt;/code&gt;, its permissions, and that you reloaded &lt;code&gt;sshd&lt;/code&gt;. The file may contain multiple CA keys, one per line.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Certificate not offered&lt;/strong&gt; — name it &lt;code&gt;&amp;lt;identity&amp;gt;-cert.pub&lt;/code&gt; beside the private key, or set &lt;code&gt;CertificateFile&lt;/code&gt;. A certificate sitting in the same directory with a different name is not automatically used.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Host certificate mismatch&lt;/strong&gt; — the client name is not in the certificate's principal list. Add every alias and IP clients use.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;All public key auth fails after enabling &lt;code&gt;RevokedKeys&lt;/code&gt;&lt;/strong&gt; — the KRL path is unreadable. Fix permissions or remove the directive until the file is in place.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Root still cannot log in&lt;/strong&gt; — certificates do not bypass &lt;code&gt;PermitRootLogin&lt;/code&gt;, &lt;code&gt;AllowUsers&lt;/code&gt;, PAM, account expiry, or shell policy. They replace key distribution, not authorization policy.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What certificates do not solve
&lt;/h2&gt;

&lt;p&gt;A CA does not remove the need to protect accounts. It does not grant sudo, bypass PAM, or make a compromised laptop safe. A stolen private key plus a valid certificate is a bearer credential until the certificate expires; that is why short validity, &lt;code&gt;-O clear&lt;/code&gt;, and &lt;code&gt;source-address&lt;/code&gt; restrictions matter.&lt;/p&gt;

&lt;p&gt;There are no intermediate CAs in OpenSSH. Every server trusts the CA key directly, so a single CA compromise is a fleet compromise. If you need finer-grained delegation, run separate CAs for separate trust domains rather than trying to build an X.509-style hierarchy that OpenSSH does not implement.&lt;/p&gt;

&lt;p&gt;Host certificates also do not fix DNS or routing. They prove that a host key was signed by your CA for a name. The client still has to reach the right host and use a name that appears in the certificate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Migration and rollback
&lt;/h2&gt;

&lt;p&gt;Do not remove &lt;code&gt;authorized_keys&lt;/code&gt; on day one. Add &lt;code&gt;TrustedUserCAKeys&lt;/code&gt;, issue certificates to a pilot group, and watch the server logs for accepted certificate IDs. Keep the old keys in place until every login path — interactive, automation, backups, monitoring — has a certificate.&lt;/p&gt;

&lt;p&gt;Rollback is equally simple: remove or comment out &lt;code&gt;TrustedUserCAKeys&lt;/code&gt; and &lt;code&gt;HostCertificate&lt;/code&gt;, run &lt;code&gt;sshd -t&lt;/code&gt;, and reload. Existing &lt;code&gt;authorized_keys&lt;/code&gt; entries still work if you left them alone. Certificates are additive; they do not mutate the user's key file.&lt;/p&gt;

&lt;p&gt;The payoff is not just fewer files. It is a single place to answer “who can log in, as whom, from where, and until when” — with a signature that every host can verify.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;OpenSSH &lt;code&gt;ssh-keygen(1)&lt;/code&gt; — CERTIFICATES, certificate options, &lt;code&gt;-V&lt;/code&gt; validity syntax, and KEY REVOCATION LISTS: &lt;a href="https://man.openbsd.org/ssh-keygen.1" rel="noopener noreferrer"&gt;https://man.openbsd.org/ssh-keygen.1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenSSH &lt;code&gt;sshd_config(5)&lt;/code&gt; — &lt;code&gt;TrustedUserCAKeys&lt;/code&gt;, &lt;code&gt;AuthorizedPrincipalsFile&lt;/code&gt;, &lt;code&gt;AuthorizedPrincipalsCommand&lt;/code&gt;, &lt;code&gt;HostCertificate&lt;/code&gt;, and &lt;code&gt;RevokedKeys&lt;/code&gt;: &lt;a href="https://man.openbsd.org/sshd_config.5" rel="noopener noreferrer"&gt;https://man.openbsd.org/sshd_config.5&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenSSH &lt;code&gt;ssh(1)&lt;/code&gt; — certificate auto-loading and &lt;code&gt;CertificateFile&lt;/code&gt;: &lt;a href="https://man.openbsd.org/ssh.1" rel="noopener noreferrer"&gt;https://man.openbsd.org/ssh.1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenSSH &lt;code&gt;ssh_config(5)&lt;/code&gt; — &lt;code&gt;CertificateFile&lt;/code&gt;: &lt;a href="https://man.openbsd.org/ssh_config.5" rel="noopener noreferrer"&gt;https://man.openbsd.org/ssh_config.5&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenSSH &lt;code&gt;sshd(8)&lt;/code&gt; — &lt;code&gt;known_hosts&lt;/code&gt; format, &lt;code&gt;@cert-authority&lt;/code&gt;, and &lt;code&gt;@revoked&lt;/code&gt;: &lt;a href="https://man.openbsd.org/sshd.8" rel="noopener noreferrer"&gt;https://man.openbsd.org/sshd.8&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;OpenSSH manual index: &lt;a href="https://www.openssh.com/manual.html" rel="noopener noreferrer"&gt;https://www.openssh.com/manual.html&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Commands and certificate output in this article were verified with OpenSSH 10.0p2 (Debian 13). The revocation examples were tested with a locally generated CA and a throwaway key.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>security</category>
      <category>devops</category>
      <category>ssh</category>
    </item>
    <item>
      <title>Stop Letting One Service Eat Every PID: Practical cgroup v2 pids Control on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Tue, 10 Nov 2026 05:20:26 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-letting-one-service-eat-every-pid-practical-cgroup-v2-pids-control-on-linux-3jdo</link>
      <guid>https://dev.to/lyraalishaikh/stop-letting-one-service-eat-every-pid-practical-cgroup-v2-pids-control-on-linux-3jdo</guid>
      <description>&lt;p&gt;At 3 a.m., every login attempt on the box returned &lt;code&gt;fork: Resource temporarily unavailable&lt;/code&gt;. Memory was fine. CPU was flat. Disks were quiet. The host had run out of &lt;em&gt;tasks&lt;/em&gt;: one worker had been spawning children in a retry loop until the kernel's process table filled, and then nothing else on the machine could start a process either — including the shell you needed to fix it.&lt;/p&gt;

&lt;p&gt;We are careful about bounding bytes (&lt;code&gt;memory.max&lt;/code&gt;), milliseconds (&lt;code&gt;cpu.max&lt;/code&gt;), and IOPS (&lt;code&gt;io.max&lt;/code&gt;). We almost never bound &lt;em&gt;how many things exist&lt;/em&gt;. That is the pids controller, and systemd already wires it to a knob most people have never set: &lt;code&gt;TasksMax=&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two limits with the same name and different scopes
&lt;/h2&gt;

&lt;p&gt;The reflex is &lt;code&gt;ulimit -u&lt;/code&gt; (RLIMIT_NPROC). For services it is usually the wrong tool, and &lt;code&gt;getrlimit(2)&lt;/code&gt; says exactly why:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This is a limit on the number of extant process (or, more precisely on Linux, threads) for the real user ID of the calling process. [...] The RLIMIT_NPROC limit is not enforced for processes that have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability, or run with real user ID 0.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That last sentence is the whole problem. A service running as root can fork without limit no matter what you put in &lt;code&gt;limits.conf&lt;/code&gt;. Even for non-root services, the limit is per &lt;em&gt;user ID&lt;/em&gt;, not per service: two units running as the same user share one budget, and a login shell counts against it too.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;RLIMIT_NPROC&lt;/th&gt;
&lt;th&gt;cgroup &lt;code&gt;pids.max&lt;/code&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Enforced against&lt;/td&gt;
&lt;td&gt;real UID&lt;/td&gt;
&lt;td&gt;cgroup membership&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Counts&lt;/td&gt;
&lt;td&gt;threads of that UID, everywhere&lt;/td&gt;
&lt;td&gt;tasks in the cgroup and its descendants&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Exempt&lt;/td&gt;
&lt;td&gt;root, CAP_SYS_ADMIN, CAP_SYS_RESOURCE&lt;/td&gt;
&lt;td&gt;nothing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scope follows&lt;/td&gt;
&lt;td&gt;the process's credentials&lt;/td&gt;
&lt;td&gt;where the process was placed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Configured via&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;limits.conf&lt;/code&gt;, &lt;code&gt;LimitNPROC=&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;TasksMax=&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The cgroup limit does not care who you are. It cares where you live.&lt;/p&gt;

&lt;h2&gt;
  
  
  The counter counts threads, not processes
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;ps aux | wc -l&lt;/code&gt; is the wrong denominator. Every thread is a task, and every task counts. A JVM with two processes and forty pool threads is forty-two tasks. This is the most common reason a limit that "looks generous" kills a Java or .NET service at startup with &lt;code&gt;unable to create native thread&lt;/code&gt; — an error everyone blames on ulimit.&lt;/p&gt;

&lt;p&gt;Measure the real number before you pick one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# threads in one process&lt;/span&gt;
grep Threads /proc/&lt;span class="nv"&gt;$(&lt;/span&gt;pgrep -f my-service &lt;span class="o"&gt;|&lt;/span&gt; head -1&lt;span class="o"&gt;)&lt;/span&gt;/status

&lt;span class="c"&gt;# all tasks belonging to a unit, threads included&lt;/span&gt;
systemctl show -p TasksCurrent -p TasksMax my-service.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the second command prints &lt;code&gt;TasksCurrent=[not set]&lt;/code&gt;, you are asking the wrong manager. User units live in the user instance:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl --user show -p TasksCurrent -p TasksMax my-user-service.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A child can lower the limit, never raise it
&lt;/h2&gt;

&lt;p&gt;cgroup v2 limits are hierarchical. The effective limit of any cgroup is the tightest &lt;code&gt;pids.max&lt;/code&gt; on the path from the root down to it. A nested cgroup can write a bigger number into its own &lt;code&gt;pids.max&lt;/code&gt;, but that grants nothing: the ancestor's limit still applies to every task in the subtree. There is also no &lt;code&gt;pids.max&lt;/code&gt; on the root cgroup at all — the control files exist on non-root cgroups, which is why you cannot "limit the whole machine" this way.&lt;/p&gt;

&lt;p&gt;systemd follows the same rule. From &lt;code&gt;systemd.resource-control(5)&lt;/code&gt;: when &lt;code&gt;TasksMax=&lt;/code&gt; is set for a unit, the pids controller is enabled for that unit &lt;em&gt;and&lt;/em&gt; for its parents and siblings, so accounting is consistent up the tree.&lt;/p&gt;

&lt;h2&gt;
  
  
  What systemd already does for you
&lt;/h2&gt;

&lt;p&gt;Two defaults matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;DefaultTasksAccounting=yes&lt;/code&gt;, so task counts are always kept.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;DefaultTasksMax=&lt;/code&gt; defaults to &lt;strong&gt;15% of the minimum of &lt;code&gt;kernel.pid_max&lt;/code&gt;, &lt;code&gt;kernel.threads-max&lt;/code&gt;, and the root cgroup's &lt;code&gt;pids.max&lt;/code&gt;&lt;/strong&gt;. The man page's example is 4915 on a host with the traditional 32768 &lt;code&gt;pid_max&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On the Debian 13 host I tested (systemd 257.13, kernel 6.12):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ cat /proc/sys/kernel/pid_max
4194304
$ cat /proc/sys/kernel/threads-max
62898
$ systemctl show -p TasksCurrent -p TasksMax -p EffectiveTasksMax systemd-journald.service
TasksCurrent=1
TasksMax=9434
EffectiveTasksMax=9434
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;9434 is 15% of 62898, truncated. So every unit on that host is already capped, whether or not its author thought about it.&lt;/p&gt;

&lt;p&gt;logind adds a second layer: on systemd 257, each user slice is capped at 33% of the same system maximum (&lt;code&gt;UserTasksMax=&lt;/code&gt; in &lt;code&gt;logind.conf(5)&lt;/code&gt;), which is why &lt;code&gt;user-1000.slice&lt;/code&gt; reports &lt;code&gt;TasksMax=20756&lt;/code&gt; here. That is the layer that saves you when a logged-in user runs something unfortunate: the host still has two thirds of its task table for sshd and PID 1.&lt;/p&gt;

&lt;p&gt;One honest caveat: these are &lt;em&gt;per-unit&lt;/em&gt; caps, not a global budget. Ten units at 15% each can still sum past 100%, so a coordinated fork storm can exhaust the table. What &lt;code&gt;TasksMax=&lt;/code&gt; buys you is blast-radius control: the noisy unit hits its own wall and stops, and everything else keeps running.&lt;/p&gt;

&lt;h2&gt;
  
  
  The files that make it visible
&lt;/h2&gt;

&lt;p&gt;Inside any non-root cgroup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ ls /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/ \
     | grep pids
pids.current
pids.events
pids.events.local
pids.max
pids.peak
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;pids.current&lt;/code&gt; — tasks right now, in this cgroup and its descendants.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pids.max&lt;/code&gt; — the limit, or &lt;code&gt;max&lt;/code&gt; for none.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pids.peak&lt;/code&gt; — the high-water mark since the cgroup was created.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pids.events&lt;/code&gt; — &lt;code&gt;max N&lt;/code&gt;: how many forks were &lt;strong&gt;denied&lt;/strong&gt;, including in descendants.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pids.events.local&lt;/code&gt; — the same counter for this cgroup only.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;code&gt;pids.events&lt;/code&gt; is the one to graph. A limit that is never hit leaves it at &lt;code&gt;max 0&lt;/code&gt;, and that is your evidence that the number is doing something rather than just sitting in a unit file.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lab: make fork fail on purpose
&lt;/h2&gt;

&lt;p&gt;Do not paste the classic fork bomb. A bounded loop produces exactly the same &lt;code&gt;EAGAIN&lt;/code&gt; and cannot take your session down with it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /tmp/pidlab.py&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="n"&gt;cg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"/sys/fs/cgroup"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"::"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/proc/self/cgroup"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"0::"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;kids&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fork&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;OSError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;1.2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;kids&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;OSError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"ERR:"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"cgroup        "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"fork_failed_at"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.current  "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;rd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.current"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.max      "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;rd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.max"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.peak     "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;rd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.peak"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.events   "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;rd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pids.events"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"\n"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"; "&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;kids&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;waitpid&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;OSError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it in a scope with a deliberately small limit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-run --user --scope -q -p &lt;span class="nv"&gt;TasksMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;20 --unit=pidlab -- python3 /tmp/pidlab.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cgroup         /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/pidlab.scope
fork_failed_at (19, 11, 'Resource temporarily unavailable')
pids.current   20
pids.max       20
pids.peak      20
pids.events    max 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read that carefully, because it contains the whole model:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Twenty tasks means the parent plus nineteen children.&lt;/strong&gt; The scope's own process is inside its own limit; the twentieth &lt;code&gt;fork()&lt;/code&gt; is the one that fails.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It failed at index 19 with errno 11, EAGAIN.&lt;/strong&gt; Not &lt;code&gt;ENOMEM&lt;/code&gt;. The kernel is not out of memory; it is enforcing a policy. Applications frequently log this as a memory problem and send you chasing the wrong subsystem for a day.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;pids.events&lt;/code&gt; is &lt;code&gt;max 1&lt;/code&gt;.&lt;/strong&gt; Exactly one denial was recorded, and only the fork that hit the wall was affected — nothing was killed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A shell shows the same event in its own dialect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/usr/bin/bash: fork: retry: Resource temporarily unavailable
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;bash retries the fork a few times before giving up, which is why a fork-bombing script produces a burst of these lines rather than one clean error.&lt;/p&gt;

&lt;h2&gt;
  
  
  The percentage trap
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;TasksMax=&lt;/code&gt; accepts a percentage, and it is not a percentage of the default:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-run --user --scope -q -p &lt;span class="nv"&gt;TasksMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;50% -- \
  bash -c &lt;span class="s1"&gt;'cg=/sys/fs/cgroup$(cut -d: -f3 /proc/self/cgroup); cat $cg/pids.max'&lt;/span&gt;
&lt;span class="c"&gt;# 31449&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;31449 is 50% of 62898, the system maximum from &lt;code&gt;threads-max&lt;/code&gt; — not 50% of the 9434 default, and certainly not 50% of current usage. Percentages are relative to the same system-wide ceiling that &lt;code&gt;DefaultTasksMax=&lt;/code&gt; uses. If you want a number that means the same thing on every host, write an absolute one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Choosing a number that does not break the app
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Measure idle and peak.&lt;/strong&gt; &lt;code&gt;TasksCurrent&lt;/code&gt; at rest is your floor. Run a load test, then read &lt;code&gt;pids.peak&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Count the things that spawn threads&lt;/strong&gt;, not just processes:&lt;/li&gt;
&lt;/ol&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;JVM&lt;/strong&gt;: pool threads plus GC and JIT threads. &lt;code&gt;-XX:ActiveProcessorCount&lt;/code&gt; changes some of them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Go&lt;/strong&gt;: &lt;code&gt;GOMAXPROCS&lt;/code&gt; is a floor, not a ceiling; the runtime adds sysmon and GC workers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;nginx&lt;/strong&gt;: one process per &lt;code&gt;worker_processes&lt;/code&gt; value, plus cache manager and loader.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PostgreSQL&lt;/strong&gt;: one process per connection, plus autovacuum workers, walsenders, and background workers. A &lt;code&gt;TasksMax=100&lt;/code&gt; on a database configured for &lt;code&gt;max_connections=200&lt;/code&gt; is a time bomb that goes off under load.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PHP-FPM&lt;/strong&gt;: &lt;code&gt;pm.max_children&lt;/code&gt; is the process count you must leave room for.&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Add headroom&lt;/strong&gt; for shell-outs, &lt;code&gt;xargs -P&lt;/code&gt;, and short-lived helpers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deploy, then watch &lt;code&gt;pids.events&lt;/code&gt;.&lt;/strong&gt; If it climbs, the limit is too tight — not the app.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Recipes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Cap one noisy service&lt;/strong&gt; with a drop-in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo mkdir -p /etc/systemd/system/scraper.service.d
sudo tee /etc/systemd/system/scraper.service.d/20-tasks.conf &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'EOF'&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;Service&lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="nv"&gt;TasksMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;64
EOF
sudo systemctl daemon-reload
sudo systemctl restart scraper.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Cap a whole class of work&lt;/strong&gt; — every CI job in a slice shares one ceiling:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo mkdir -p /etc/systemd/system/build.slice.d
sudo tee /etc/systemd/system/build.slice.d/50-tasks.conf &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'EOF'&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;Slice&lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="nv"&gt;TasksMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2048
EOF
sudo systemctl daemon-reload
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Cap a user without touching logind defaults:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo mkdir -p /etc/systemd/system/user-1000.slice.d
sudo tee /etc/systemd/system/user-1000.slice.d/50-tasks.conf &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'EOF'&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;Slice&lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="nv"&gt;TasksMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;4096
EOF
sudo systemctl daemon-reload
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Try a limit without writing a unit&lt;/strong&gt;, useful for reproducing a customer report:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-run --user --scope -p &lt;span class="nv"&gt;TasksMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;32 -- ./run-tests.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Containers&lt;/strong&gt; get the same controller through the runtime: &lt;code&gt;podman run --pids-limit N&lt;/code&gt; and &lt;code&gt;docker run --pids-limit N&lt;/code&gt; write the container cgroup's &lt;code&gt;pids.max&lt;/code&gt;. Kubernetes can cap pods at the node level with kubelet's &lt;code&gt;--pod-max-pids&lt;/code&gt;. Without those flags the limit is &lt;code&gt;max&lt;/code&gt;, which is how a container takes down its host's process table from the inside.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure modes
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Real cause&lt;/th&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Resource temporarily unavailable&lt;/code&gt; on fork&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;pids.max&lt;/code&gt; reached&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pids.events&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JVM: &lt;code&gt;unable to create native thread&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;limit below thread count&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;pids.peak&lt;/code&gt; vs &lt;code&gt;TasksMax&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Works until traffic spikes&lt;/td&gt;
&lt;td&gt;fork-per-request or connection-per-process&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;TasksCurrent&lt;/code&gt; under load&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Limit "ignored" for a root service&lt;/td&gt;
&lt;td&gt;you set &lt;code&gt;LimitNPROC=&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;use &lt;code&gt;TasksMax=&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Numbers differ from the docs&lt;/td&gt;
&lt;td&gt;percentage vs absolute&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;pids.max&lt;/code&gt; in the cgroup&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Lowering a limit at runtime does not kill anything. Existing tasks keep running; new forks are refused until the count drops below the new ceiling. That makes tightening a live system safe, and it also means a too-low limit can look fine until the next restart.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verify, then roll back
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl show -p TasksMax -p EffectiveTasksMax -p TasksCurrent my-service.service
systemd-cgtop -t -b -n 1          &lt;span class="c"&gt;# order control groups by task count&lt;/span&gt;
cat /sys/fs/cgroup/system.slice/my-service.service/pids.events
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the limit was a mistake, remove the drop-in and revert:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo rm /etc/systemd/system/my-service.service.d/20-tasks.conf
sudo systemctl daemon-reload
sudo systemctl restart my-service.service
&lt;span class="c"&gt;# or, to drop every local override at once:&lt;/span&gt;
sudo systemctl revert my-service.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The limit nobody graphs
&lt;/h2&gt;

&lt;p&gt;A host that cannot fork is down in a way that looks nothing like being out of memory or CPU: the monitoring is green, the logs are quiet, and nothing you type works. It costs one line in a unit file to make sure the next runaway loop fails inside its own cgroup instead of taking the machine with it.&lt;/p&gt;

&lt;p&gt;Start with your noisiest units — scrapers, build jobs, anything that shells out — and give them a number based on measurement rather than optimism. Then watch &lt;code&gt;pids.events&lt;/code&gt; for a week. If it stays at zero, you have headroom. If it moves, you just found the service that would have been your 3 a.m. page.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Linux kernel documentation, &lt;em&gt;Control Group v2&lt;/em&gt; — the pids controller and its interface files: &lt;a href="https://docs.kernel.org/admin-guide/cgroup-v2.html" rel="noopener noreferrer"&gt;https://docs.kernel.org/admin-guide/cgroup-v2.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;cgroups(7)&lt;/code&gt; — pids controller since Linux 4.3, &lt;code&gt;CONFIG_CGROUP_PIDS&lt;/code&gt;: &lt;a href="https://man7.org/linux/man-pages/man7/cgroups.7.html" rel="noopener noreferrer"&gt;https://man7.org/linux/man-pages/man7/cgroups.7.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd.resource-control(5)&lt;/code&gt; — &lt;code&gt;TasksAccounting=&lt;/code&gt;, &lt;code&gt;TasksMax=&lt;/code&gt;, &lt;code&gt;EffectiveTasksMax=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd-system.conf(5)&lt;/code&gt; — &lt;code&gt;DefaultTasksMax=&lt;/code&gt; (15% of the minimum of &lt;code&gt;pid_max&lt;/code&gt;, &lt;code&gt;threads-max&lt;/code&gt;, and root &lt;code&gt;pids.max&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;logind.conf(5)&lt;/code&gt; — &lt;code&gt;UserTasksMax=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;fork(2)&lt;/code&gt;, &lt;code&gt;getrlimit(2)&lt;/code&gt; — EAGAIN conditions and RLIMIT_NPROC exemptions&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd-run(1)&lt;/code&gt;, &lt;code&gt;systemctl(1)&lt;/code&gt;, &lt;code&gt;systemd-cgtop(1)&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;podman-run(1)&lt;/code&gt;, &lt;code&gt;docker-run(1)&lt;/code&gt; — &lt;code&gt;--pids-limit&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Tested on Debian 13, kernel 6.12.113, systemd 257.13. Your numbers will differ; the semantics will not.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>linux</category>
      <category>systemd</category>
      <category>performance</category>
      <category>devops</category>
    </item>
    <item>
      <title>Stop Giving Every Service Your Whole Filesystem: Practical Landlock Sandboxing on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Mon, 09 Nov 2026 05:03:21 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-giving-every-service-your-whole-filesystem-practical-landlock-sandboxing-on-linux-20nn</link>
      <guid>https://dev.to/lyraalishaikh/stop-giving-every-service-your-whole-filesystem-practical-landlock-sandboxing-on-linux-20nn</guid>
      <description>&lt;p&gt;A service runs as its own user. Its unit has &lt;code&gt;NoNewPrivileges=yes&lt;/code&gt;. Maybe it even has &lt;code&gt;ProtectSystem=strict&lt;/code&gt; and &lt;code&gt;ProtectHome=yes&lt;/code&gt;. And yet, if someone finds a path traversal bug in it, the process can still open &lt;code&gt;/etc/hostname&lt;/code&gt;, &lt;code&gt;/etc/passwd&lt;/code&gt;, your application's config in &lt;code&gt;/srv&lt;/code&gt;, or a leftover credential file in &lt;code&gt;/var/backups&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That is not a misconfiguration. It is how discretionary access control works: if a file is world-readable, every uid on the box may read it. Your service does not have a "these are my files" list — it has a uid, and the kernel checks that uid against inode permissions on every &lt;code&gt;open()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Landlock closes exactly that gap. It is a Linux LSM that lets a process hand the kernel a list of path hierarchies it is allowed to touch, and have everything else denied at &lt;code&gt;open()&lt;/code&gt; — with no root, no mount namespace, and no container runtime. This article builds a small wrapper around it, proves the denials, and wires it into a systemd unit.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Landlock actually does
&lt;/h2&gt;

&lt;p&gt;Landlock was introduced in Linux 5.13 as a stackable LSM (&lt;code&gt;CONFIG_SECURITY_LANDLOCK&lt;/code&gt;). The model is deliberately narrow and easy to reason about:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A &lt;strong&gt;ruleset&lt;/strong&gt; declares the access rights it wants to &lt;em&gt;handle&lt;/em&gt;. Any handled right that no rule grants is denied, everywhere.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rules&lt;/strong&gt; grant a subset of those rights to a file hierarchy, identified by an opened path.&lt;/li&gt;
&lt;li&gt;Enforcing a ruleset creates a &lt;strong&gt;domain&lt;/strong&gt;. Every child thread and every &lt;code&gt;execve()&lt;/code&gt; descendant stays inside it.&lt;/li&gt;
&lt;li&gt;Domains &lt;strong&gt;stack&lt;/strong&gt;. A sandboxed process can only add more restrictions; it can never relax one. Neither can root.&lt;/li&gt;
&lt;li&gt;A denied operation fails with &lt;code&gt;EACCES&lt;/code&gt; — an ordinary "Permission denied", not a crash.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two consequences are worth internalising before writing any policy:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;This is an allow-list over opens, not a jail.&lt;/strong&gt; The path is still visible in &lt;code&gt;ls&lt;/code&gt;; only &lt;code&gt;open()&lt;/code&gt; is refused. It is not a mount namespace and it does not hide anything.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;File descriptors opened before enforcement are exempt.&lt;/strong&gt; &lt;code&gt;landlock(7)&lt;/code&gt; is explicit: "Files or directories opened before the sandboxing are not subject to these restrictions." So a service that inherits an open config fd from its parent keeps using it, and your policy must be applied before the process opens anything it should not have.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It also does not replace seccomp (it restricts objects, not syscalls), and it does not restrict networking unless you handle network rights too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Know your ABI before you write the policy
&lt;/h2&gt;

&lt;p&gt;Landlock is versioned by &lt;em&gt;ABI&lt;/em&gt;, not by kernel version. Query it instead of guessing, because each ABI adds rights that older kernels reject with &lt;code&gt;EINVAL&lt;/code&gt; if you mention them:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;ABI&lt;/th&gt;
&lt;th&gt;Mainline kernel&lt;/th&gt;
&lt;th&gt;What it added&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;5.13&lt;/td&gt;
&lt;td&gt;Filesystem rights (execute, read, write, create, remove, rename)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;5.19&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;LANDLOCK_ACCESS_FS_REFER&lt;/code&gt; (cross-directory link/rename)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;6.2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;LANDLOCK_ACCESS_FS_TRUNCATE&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;6.7&lt;/td&gt;
&lt;td&gt;TCP bind/connect rules&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;6.10&lt;/td&gt;
&lt;td&gt;&lt;code&gt;LANDLOCK_ACCESS_FS_IOCTL_DEV&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;6.12&lt;/td&gt;
&lt;td&gt;IPC scopes: &lt;code&gt;LANDLOCK_SCOPE_SIGNAL&lt;/code&gt;, &lt;code&gt;LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;6.15&lt;/td&gt;
&lt;td&gt;Restriction logging flags&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;7.0&lt;/td&gt;
&lt;td&gt;&lt;code&gt;LANDLOCK_RESTRICT_SELF_TSYNC&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;7.1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;LANDLOCK_ACCESS_FS_RESOLVE_UNIX&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the Debian 13 box used for this article (kernel 6.12.113) the syscall reports ABI 6. A five-line check on x86-64, no compiler needed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 - &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
import ctypes
libc = ctypes.CDLL(None, use_errno=True)
# landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)
print("Landlock ABI:", libc.syscall(444, None, ctypes.c_size_t(0), ctypes.c_uint32(1)))
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Landlock ABI: 6
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Syscall 444 is the x86-64 number; on other architectures use &lt;code&gt;SYS_landlock_create_ruleset&lt;/code&gt; from &lt;code&gt;&amp;lt;sys/syscall.h&amp;gt;&lt;/code&gt;. If the call returns &lt;code&gt;-1&lt;/code&gt; with &lt;code&gt;ENOSYS&lt;/code&gt; or &lt;code&gt;EOPNOTSUPP&lt;/code&gt;, Landlock is missing or disabled — check that &lt;code&gt;landlock&lt;/code&gt; appears in &lt;code&gt;/sys/kernel/security/lsm&lt;/code&gt;, and that the kernel log says &lt;code&gt;landlock: Up and running&lt;/code&gt; (&lt;code&gt;journalctl -kb -g landlock&lt;/code&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  A wrapper you can actually use
&lt;/h2&gt;

&lt;p&gt;Most services have no Landlock support, and you should not patch every application. The practical pattern is a tiny launcher: build the ruleset, enforce it, then &lt;code&gt;exec&lt;/code&gt; the real binary. The sandbox is inherited across &lt;code&gt;execve()&lt;/code&gt;, so the application never knows it happened.&lt;/p&gt;

&lt;p&gt;Save this as &lt;code&gt;landlock-run.c&lt;/code&gt;. It grants read+execute on &lt;code&gt;-r&lt;/code&gt; paths and full access on &lt;code&gt;-w&lt;/code&gt; paths, and drops rights the running ABI does not know about instead of failing.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight c"&gt;&lt;code&gt;&lt;span class="cm"&gt;/* landlock-run.c - apply a path allow-list with Landlock, then exec a command.
 * Build: cc -O2 -Wall -o landlock-run landlock-run.c
 * Usage: landlock-run [-r PATH]... [-w PATH]... -- CMD [ARG]...
 *   -r PATH  allow reading and executing inside PATH
 *   -w PATH  allow reading, writing, creating and removing inside PATH
 */&lt;/span&gt;
&lt;span class="cp"&gt;#define _GNU_SOURCE
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;errno.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;fcntl.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;linux/landlock.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;stdio.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;stdlib.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;string.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;sys/prctl.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;sys/stat.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;sys/syscall.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
#include&lt;/span&gt; &lt;span class="cpf"&gt;&amp;lt;unistd.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
&lt;/span&gt;
&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;ll_create_ruleset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;landlock_ruleset_attr&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                             &lt;span class="kt"&gt;size_t&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;__u32&lt;/span&gt; &lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;syscall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SYS_landlock_create_ruleset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;ll_add_rule&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;enum&lt;/span&gt; &lt;span class="n"&gt;landlock_rule_type&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="k"&gt;const&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;syscall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SYS_landlock_add_rule&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;ll_restrict_self&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;__u32&lt;/span&gt; &lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;syscall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SYS_landlock_restrict_self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="cm"&gt;/* Everything this kernel knows about, depending on its Landlock ABI. */&lt;/span&gt;
&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="n"&gt;__u64&lt;/span&gt; &lt;span class="nf"&gt;fs_rights_for_abi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;abi&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;__u64&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_EXECUTE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_WRITE_FILE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_READ_FILE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_READ_DIR&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_REMOVE_DIR&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_REMOVE_FILE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_MAKE_CHAR&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_MAKE_DIR&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_MAKE_REG&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_MAKE_SOCK&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_MAKE_FIFO&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_MAKE_BLOCK&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_MAKE_SYM&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;abi&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;|=&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_REFER&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;abi&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;|=&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_TRUNCATE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;abi&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;|=&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_IOCTL_DEV&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;add_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;const&lt;/span&gt; &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;writable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;__u64&lt;/span&gt; &lt;span class="n"&gt;handled&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;stat&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;__u64&lt;/span&gt; &lt;span class="n"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;fd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;O_PATH&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;O_CLOEXEC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"landlock-run: cannot open %s: %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fstat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"landlock-run: cannot stat %s: %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
        &lt;span class="n"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;S_ISDIR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;st_mode&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;allowed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;writable&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;handled&lt;/span&gt;
                           &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LANDLOCK_ACCESS_FS_EXECUTE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
                              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_READ_FILE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
                              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_READ_DIR&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="cm"&gt;/* Directory-only rights are rejected with EINVAL on a file. */&lt;/span&gt;
        &lt;span class="n"&gt;allowed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;writable&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LANDLOCK_ACCESS_FS_WRITE_FILE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
                              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_READ_FILE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
                              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_TRUNCATE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
                              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_IOCTL_DEV&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                           &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LANDLOCK_ACCESS_FS_EXECUTE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
                              &lt;span class="n"&gt;LANDLOCK_ACCESS_FS_READ_FILE&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;allowed&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;=&lt;/span&gt; &lt;span class="n"&gt;handled&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="cm"&gt;/* drop rights this ABI does not support */&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;landlock_path_beneath_attr&lt;/span&gt; &lt;span class="n"&gt;pb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;allowed_access&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parent_fd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;};&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ll_add_rule&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_RULE_PATH_BENEATH&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;pb&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"landlock-run: rule for %s failed: %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
            &lt;span class="n"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;argc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;landlock_ruleset_attr&lt;/span&gt; &lt;span class="n"&gt;attr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
    &lt;span class="k"&gt;const&lt;/span&gt; &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;read_paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;write_paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;n_read&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n_write&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;abi&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;argc&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;strcmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s"&gt;"--"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;strcmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s"&gt;"-r"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;strcmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s"&gt;"-w"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;argc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sc"&gt;'r'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="n"&gt;read_paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n_read&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
            &lt;span class="k"&gt;else&lt;/span&gt;
                &lt;span class="n"&gt;write_paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n_write&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"usage: %s [-r PATH]... [-w PATH]... -- CMD [ARG]...&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;argc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"landlock-run: no command given after --&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;argv&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;abi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ll_create_ruleset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;LANDLOCK_CREATE_RULESET_VERSION&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;abi&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"landlock-run: Landlock unavailable: %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;handled_access_fs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fs_rights_for_abi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;abi&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;ruleset_fd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ll_create_ruleset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;sizeof&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ruleset_fd&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"landlock-run: ruleset creation failed: %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;n_read&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;add_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;read_paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;handled_access_fs&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;n_write&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;add_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;write_paths&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;handled_access_fs&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prctl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;PR_SET_NO_NEW_PRIVS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;perror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"landlock-run: prctl"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ll_restrict_self&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"landlock-run: restrict_self failed: %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ruleset_fd&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="n"&gt;execvp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;perror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"landlock-run: execvp"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;127&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cc &lt;span class="nt"&gt;-O2&lt;/span&gt; &lt;span class="nt"&gt;-Wall&lt;/span&gt; &lt;span class="nt"&gt;-Wextra&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; landlock-run landlock-run.c
&lt;span class="nb"&gt;sudo install&lt;/span&gt; &lt;span class="nt"&gt;-m&lt;/span&gt; 0755 landlock-run /usr/local/bin/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details in there matter more than they look:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;prctl(PR_SET_NO_NEW_PRIVS, 1, ...)&lt;/code&gt; is &lt;strong&gt;required&lt;/strong&gt; for an unprivileged process, and it also makes setuid/setgid binaries inside the sandbox run without elevation. A &lt;code&gt;CAP_SYS_ADMIN&lt;/code&gt; process may skip it, but then a setuid binary could execute with elevated privileges inside your domain — a confused deputy you do not want.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;allowed &amp;amp;= handled&lt;/code&gt; is the compatibility pattern from the kernel documentation: a newer ABI's rights are silently dropped on an older kernel instead of returning &lt;code&gt;EINVAL&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The ruleset handles &lt;em&gt;all&lt;/em&gt; filesystem rights. That is intentional: an allow-list should deny creation and deletion too, not just reads.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It deliberately leaves &lt;code&gt;handled_access_net&lt;/code&gt; and &lt;code&gt;scoped&lt;/code&gt; at zero, so networking and IPC remain unrestricted. We will come back to that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it denies something
&lt;/h2&gt;

&lt;p&gt;Build a scratch directory, then run a shell inside a policy that allows only that directory plus the usual runtime trees:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; /tmp/lltest &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"inside the allowed dir"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/lltest/ok.txt

./landlock-run &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-r&lt;/span&gt; /usr &lt;span class="nt"&gt;-r&lt;/span&gt; /lib &lt;span class="nt"&gt;-r&lt;/span&gt; /lib64 &lt;span class="nt"&gt;-r&lt;/span&gt; /etc/ld.so.cache &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-w&lt;/span&gt; /dev/null &lt;span class="nt"&gt;-r&lt;/span&gt; /dev/urandom &lt;span class="nt"&gt;-w&lt;/span&gt; /tmp/lltest &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--&lt;/span&gt; /bin/sh &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'
    cat /tmp/lltest/ok.txt
    echo "written by the sandbox" &amp;gt; /tmp/lltest/out.txt &amp;amp;&amp;amp; echo "write: ok"
    cat /tmp/lltest/out.txt
    cat /etc/hostname || echo "read /etc/hostname: denied"
    /tmp/lltest2/outside.sh || echo "exec outside allow-list: denied"
  '&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;inside the allowed dir
write: ok
written by the sandbox
cat: /etc/hostname: Permission denied
read /etc/hostname: denied
/bin/sh: 6: /tmp/lltest2/outside.sh: Permission denied
exec outside allow-list: denied
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note what was denied: &lt;code&gt;/etc/hostname&lt;/code&gt; is mode 644, owned by root, and readable by every process on the machine. Landlock did not change its mode, did not change the uid, and did not unmount anything. The open simply failed. That is the whole point.&lt;/p&gt;

&lt;h2&gt;
  
  
  The traps that will bite you
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;/dev/null&lt;/code&gt; and &lt;code&gt;/dev/urandom&lt;/code&gt; are files too.&lt;/strong&gt; A policy that grants nothing under &lt;code&gt;/dev&lt;/code&gt; breaks the most mundane shell redirection: &lt;code&gt;echo x &amp;gt; /dev/null&lt;/code&gt; fails with EACCES, and any program that seeds a PRNG from &lt;code&gt;/dev/urandom&lt;/code&gt; dies. Grant them explicitly — the program above handles them because it applies only file-applicable rights when the path is not a directory.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;READ_DIR&lt;/code&gt; and &lt;code&gt;READ_FILE&lt;/code&gt; are separate.&lt;/strong&gt; Granting &lt;code&gt;READ_FILE&lt;/code&gt; on a directory lets you open files by exact path but not list it. Granting &lt;code&gt;READ_DIR&lt;/code&gt; lets you list it but not read the files. The &lt;code&gt;-r&lt;/code&gt; flag grants both, plus &lt;code&gt;EXECUTE&lt;/code&gt;, which is what you usually want for &lt;code&gt;/usr&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;TRUNCATE&lt;/code&gt; is not implied by &lt;code&gt;WRITE_FILE&lt;/code&gt;.&lt;/strong&gt; Since ABI 3 they are separate rights, and &lt;code&gt;creat()&lt;/code&gt; on an existing file needs truncate even though its name sounds like a write. &lt;code&gt;landlock(7)&lt;/code&gt; recommends always granting the two together, which is what &lt;code&gt;-w&lt;/code&gt; does.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cross-directory rename needs &lt;code&gt;REFER&lt;/code&gt;.&lt;/strong&gt; It is the one right that is denied by default even when it is not listed in &lt;code&gt;handled_access_fs&lt;/code&gt;, and a rename between two allowed directories only works if both grant it. If the destination would grant the file &lt;em&gt;more&lt;/em&gt; rights than the source, the kernel returns &lt;code&gt;EXDEV&lt;/code&gt; rather than &lt;code&gt;EACCES&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Directory-only rights on a file are &lt;code&gt;EINVAL&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;LANDLOCK_ACCESS_FS_READ_DIR&lt;/code&gt;, &lt;code&gt;REMOVE_*&lt;/code&gt; and &lt;code&gt;MAKE_*&lt;/code&gt; cannot be attached to a regular file descriptor. That is why the wrapper branches on &lt;code&gt;S_ISDIR&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A ruleset only covers the calling thread on older ABIs.&lt;/strong&gt; &lt;code&gt;landlock_restrict_self()&lt;/code&gt; restricts the thread that calls it; children inherit, but sibling threads do not. For a single-threaded launcher that &lt;code&gt;exec&lt;/code&gt;s immediately this is irrelevant. For a multithreaded process, every thread must enforce the ruleset — or use &lt;code&gt;LANDLOCK_RESTRICT_SELF_TSYNC&lt;/code&gt; on ABI 8+ (kernel 7.0), which applies it to all threads.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The domain is permanent for that process tree.&lt;/strong&gt; There is no "unrestrict" call. Test policies on a throwaway command, not on your only SSH session.&lt;/p&gt;

&lt;h2&gt;
  
  
  Putting it in a systemd unit
&lt;/h2&gt;

&lt;p&gt;The launcher composes with the hardening you already have. A renderer that needs Python, TLS certificates, a spool directory, and nothing else:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Unit]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Description&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Report renderer (Landlocked)&lt;/span&gt;
&lt;span class="py"&gt;After&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;network-online.target&lt;/span&gt;
&lt;span class="py"&gt;Wants&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;network-online.target&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;DynamicUser&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;StateDirectory&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;render&lt;/span&gt;
&lt;span class="py"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;ExecStart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/bin/landlock-run &lt;/span&gt;&lt;span class="se"&gt;\
&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;-r /usr -r /lib -r /lib64 &lt;/span&gt;&lt;span class="se"&gt;\
&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;-r /etc/ld.so.cache -r /etc/ssl/certs &lt;/span&gt;&lt;span class="se"&gt;\
&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;-r /etc/resolv.conf -r /etc/hosts -r /etc/nsswitch.conf &lt;/span&gt;&lt;span class="se"&gt;\
&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;-r /dev/urandom -w /dev/null &lt;/span&gt;&lt;span class="se"&gt;\
&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;-w /var/lib/render &lt;/span&gt;&lt;span class="se"&gt;\
&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;-- /usr/bin/python3 /usr/local/lib/render/main.py&lt;/span&gt;
&lt;span class="py"&gt;Restart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;on-failure&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nn"&gt;[Install]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;WantedBy&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;multi-user.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two systemd facilities look similar but are not:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;ProtectSystem=&lt;/code&gt;, &lt;code&gt;ProtectHome=&lt;/code&gt; and &lt;code&gt;InaccessiblePaths=&lt;/code&gt; are &lt;strong&gt;mount-namespace&lt;/strong&gt; features. They need privileges, they hide or remount whole trees, and they are coarse: &lt;code&gt;ProtectSystem=strict&lt;/code&gt; makes &lt;code&gt;/usr&lt;/code&gt; read-only, it does not stop a read of &lt;code&gt;/etc/hostname&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;RestrictFileSystems=&lt;/code&gt; (systemd 250+) filters by &lt;strong&gt;filesystem type&lt;/strong&gt;, not by path. It is implemented with the eBPF LSM hook, so &lt;code&gt;systemd.exec(5)&lt;/code&gt; warns that it silently has no effect on kernels without that hook or without the unified cgroup hierarchy. &lt;code&gt;systemd-analyze filesystems&lt;/code&gt; lists the names and set aliases (&lt;code&gt;@basic-api&lt;/code&gt;, &lt;code&gt;@temporary&lt;/code&gt;, …) it accepts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Landlock is the path-level layer. Keep the mount namespace and the syscall filter; add path allow-listing on top. They are cumulative — every layer must grant access.&lt;/p&gt;

&lt;p&gt;Since &lt;code&gt;DynamicUser=yes&lt;/code&gt; creates the uid at start and &lt;code&gt;StateDirectory=render&lt;/code&gt; creates &lt;code&gt;/var/lib/render&lt;/code&gt;, the directories exist before &lt;code&gt;ExecStart&lt;/code&gt; runs, which is exactly the ordering Landlock needs: the paths must be openable at rule-creation time.&lt;/p&gt;

&lt;p&gt;Verify with a deliberate failure rather than by reading the unit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl start render.service
systemctl status render.service &lt;span class="nt"&gt;--no-pager&lt;/span&gt;
journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; render.service &lt;span class="nt"&gt;-g&lt;/span&gt; &lt;span class="s1"&gt;'Permission denied'&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; 20 &lt;span class="nt"&gt;--no-pager&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a legitimate path is missing, the failure mode is a clean &lt;code&gt;EACCES&lt;/code&gt; in the log, with the path named by the application — add a &lt;code&gt;-r&lt;/code&gt; or &lt;code&gt;-w&lt;/code&gt; for it and restart. If the &lt;em&gt;launcher&lt;/em&gt; cannot open a path you listed, it exits with &lt;code&gt;landlock-run: cannot open …&lt;/code&gt; before the service ever starts, which is much easier to debug than a half-running daemon.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this stops
&lt;/h2&gt;

&lt;p&gt;Be honest about the boundary, because an overstated sandbox is worse than none:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Already-open descriptors remain usable.&lt;/strong&gt; Enforcing in a launcher before &lt;code&gt;exec&lt;/code&gt; sidesteps this; if you sandbox a long-running process instead, every descriptor it already holds keeps working.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No network confinement by default.&lt;/strong&gt; TCP bind/connect rules exist from ABI 4 (kernel 6.7), but you must handle &lt;code&gt;handled_access_net&lt;/code&gt; and grant each port explicitly. Until then, a compromised service can still reach the network.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No syscall filtering.&lt;/strong&gt; Pair it with &lt;code&gt;SystemCallFilter=@system-service&lt;/code&gt; or seccomp.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No IPC isolation unless scoped.&lt;/strong&gt; On ABI 6, &lt;code&gt;LANDLOCK_SCOPE_SIGNAL&lt;/code&gt; and &lt;code&gt;LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET&lt;/code&gt; stop a sandboxed service from signalling processes or connecting to abstract sockets outside its domain — useful, and easy to forget.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not a substitute for file permissions.&lt;/strong&gt; Landlock removes access; it does not fix a secret that is world-readable and copied around by other tools.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The useful mental shift is small but real: a service should have a list of paths it needs, and the kernel should enforce that list — not a uid that happens to be able to read half of &lt;code&gt;/etc&lt;/code&gt;. Landlock is the first mechanism that makes that list enforceable by an ordinary unprivileged process, in about 140 lines of C you can read in one sitting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources and references
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Linux kernel documentation, &lt;a href="https://docs.kernel.org/userspace-api/landlock.html" rel="noopener noreferrer"&gt;Landlock: unprivileged access control&lt;/a&gt; — ruleset/rule model, ABI compatibility pattern, scopes, capability and namespace rules.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://man7.org/linux/man-pages/man7/landlock.7.html" rel="noopener noreferrer"&gt;landlock(7)&lt;/a&gt; — access rights per ABI, REFER semantics, truncate/write interaction, inheritance, ptrace and IPC scoping, ABI-to-kernel table.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://man7.org/linux/man-pages/man2/landlock_add_rule.2.html" rel="noopener noreferrer"&gt;landlock_add_rule(2)&lt;/a&gt; — rule attribute layout, file vs directory rules, &lt;code&gt;EINVAL&lt;/code&gt; conditions.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://man7.org/linux/man-pages/man2/landlock_create_ruleset.2.html" rel="noopener noreferrer"&gt;landlock_create_ruleset(2)&lt;/a&gt; and &lt;a href="https://man7.org/linux/man-pages/man2/landlock_restrict_self.2.html" rel="noopener noreferrer"&gt;landlock_restrict_self(2)&lt;/a&gt; — ABI query flag, &lt;code&gt;no_new_privs&lt;/code&gt; requirement.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html" rel="noopener noreferrer"&gt;systemd.exec(5)&lt;/a&gt; — &lt;code&gt;RestrictFileSystems=&lt;/code&gt;, &lt;code&gt;ProtectSystem=&lt;/code&gt;, &lt;code&gt;ProtectHome=&lt;/code&gt;, &lt;code&gt;NoNewPrivileges=&lt;/code&gt;, &lt;code&gt;StateDirectory=&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Local verification: Debian 13, kernel 6.12.113, systemd 257.13, Landlock ABI 6; transcripts above are from runs in &lt;code&gt;/tmp&lt;/code&gt; on that host.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>linux</category>
      <category>security</category>
      <category>devops</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Stop Guessing Why systemd Gave Up Restarting: Restart= and Start Limits on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Sun, 08 Nov 2026 05:04:45 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-guessing-why-systemd-gave-up-restarting-restart-and-start-limits-on-linux-1dc2</link>
      <guid>https://dev.to/lyraalishaikh/stop-guessing-why-systemd-gave-up-restarting-restart-and-start-limits-on-linux-1dc2</guid>
      <description>&lt;p&gt;A service exits with status 1. systemd restarts it. It exits again. Two more restarts, then the unit goes to &lt;code&gt;failed&lt;/code&gt; and stays there. &lt;code&gt;systemctl status&lt;/code&gt; ends with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Scheduled restart job, restart counter is at 3.
Start request repeated too quickly.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Someone runs &lt;code&gt;systemctl start app&lt;/code&gt; and systemd refuses that too. The natural conclusion is that the restart policy is broken, or that the unit is stuck. Neither is true. Two separate mechanisms are at work, and they answer different questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Restart=&lt;/strong&gt; decides whether this particular death deserves a restart.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start rate limiting&lt;/strong&gt; decides whether the unit is allowed to start again at all.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Most "systemd stopped restarting" incidents are the second mechanism doing its job while an operator is reading the first one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two decisions, not one
&lt;/h2&gt;

&lt;p&gt;The service manager does something close to this on every exit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;if the unit was stopped by an administrator: do not restart
else if the exit cause matches Restart=: schedule a restart after RestartSec=
else: leave the unit inactive or failed

before any start:
if the unit has been started more than StartLimitBurst times
within StartLimitIntervalSec: refuse the start
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That order matters. &lt;code&gt;Restart=&lt;/code&gt; can be perfectly configured and still produce a unit that stops retrying, because the start limiter is a gate in front of every start attempt, not a property of the restart policy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Restart= is a classifier, not a boolean
&lt;/h2&gt;

&lt;p&gt;The value of &lt;code&gt;Restart=&lt;/code&gt; is matched against &lt;strong&gt;how the process died&lt;/strong&gt;, not against what you meant. &lt;code&gt;systemd.service(5)&lt;/code&gt; documents the full matrix:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Exit cause&lt;/th&gt;
&lt;th&gt;&lt;code&gt;no&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;always&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;on-success&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;on-failure&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;on-abnormal&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;on-abort&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;on-watchdog&lt;/code&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Clean exit code or signal&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unclean exit code&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unclean signal&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Timeout&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Watchdog&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;X&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A "clean" exit is not just exit code 0. For service types other than &lt;code&gt;Type=oneshot&lt;/code&gt;, systemd also treats termination by &lt;strong&gt;SIGHUP, SIGINT, SIGTERM, or SIGPIPE&lt;/strong&gt; as clean, plus anything you list in &lt;code&gt;SuccessExitStatus=&lt;/code&gt;. That is deliberate: a daemon that exits 0 or catches SIGTERM during an ordinary shutdown should not be treated like a crash.&lt;/p&gt;

&lt;p&gt;The rest of the table follows from that definition:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;on-failure&lt;/code&gt; restarts on nonzero exit codes, unclean signals (including core dumps), start/reload/stop timeouts, and watchdog expiry. It is the recommended default for long-running services.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;on-abnormal&lt;/code&gt; restarts on signals, timeouts, and watchdog expiry, but &lt;strong&gt;not&lt;/strong&gt; on a plain nonzero exit code. Use it for a daemon that is allowed to decide its own exit status without being resurrected.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;on-abort&lt;/code&gt; is narrower: an uncaught signal that is not in the clean-exit set.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;on-watchdog&lt;/code&gt; only reacts to a missed keep-alive deadline.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;on-success&lt;/code&gt; is for the odd service that is supposed to run and exit cleanly, then run again.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One detail that surprises people: the process that dies does not have to be the main daemon. A failure in &lt;code&gt;ExecStartPre=&lt;/code&gt;, &lt;code&gt;ExecStartPost=&lt;/code&gt;, &lt;code&gt;ExecStop=&lt;/code&gt;, &lt;code&gt;ExecStopPost=&lt;/code&gt;, or &lt;code&gt;ExecReload=&lt;/code&gt; can also trigger the restart logic. If your "main process" never even started, check the pre-start commands before blaming the application.&lt;/p&gt;

&lt;h3&gt;
  
  
  The &lt;code&gt;Type=oneshot&lt;/code&gt; trap
&lt;/h3&gt;

&lt;p&gt;For &lt;code&gt;Type=oneshot&lt;/code&gt;, a clean exit is a completed job, so &lt;code&gt;Restart=always&lt;/code&gt; and &lt;code&gt;Restart=on-success&lt;/code&gt; make no sense. systemd rejects them at load time. &lt;code&gt;systemd-analyze verify&lt;/code&gt; states it plainly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Service has Restart= set to either always or on-success, which isn't allowed
for Type=oneshot services. Refusing.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a oneshot unit needs to be retried, put the retry in a timer, or make the unit a long-running &lt;code&gt;Type=exec&lt;/code&gt; service. Do not try to force it with &lt;code&gt;Restart=always&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  RestartSec= and exponential backoff
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;RestartSec=&lt;/code&gt; is the delay before each automatic restart. The manager default is &lt;strong&gt;100 ms&lt;/strong&gt;, which is fine for a transient failure and terrible for a crash loop: a service that dies immediately will be restarted ten times a second until the start limit trips.&lt;/p&gt;

&lt;p&gt;Since systemd 254 there is built-in exponential backoff:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Restart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;on-failure&lt;/span&gt;
&lt;span class="py"&gt;RestartSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2s&lt;/span&gt;
&lt;span class="py"&gt;RestartSteps&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;4&lt;/span&gt;
&lt;span class="py"&gt;RestartMaxDelaySec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;32s&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The intervals become 2s, 4s, 8s, 16s, 32s, 32s, and so on. The ratio is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(RestartMaxDelaySec / RestartSec) ^ (1 / RestartSteps)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the maximum is reached after &lt;code&gt;RestartSteps + 1&lt;/code&gt; steps. &lt;code&gt;RestartSteps=&lt;/code&gt; only works when &lt;code&gt;RestartMaxDelaySec=&lt;/code&gt; is also set and &lt;code&gt;RestartSec=&lt;/code&gt; is nonzero; with &lt;code&gt;RestartSec=0&lt;/code&gt; the whole backoff feature is inert.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start limits: the gate in front of every start
&lt;/h2&gt;

&lt;p&gt;These settings live in the &lt;strong&gt;[Unit]&lt;/strong&gt; section since systemd 229 (the old [Service] location is still understood for compatibility):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Unit]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;StartLimitIntervalSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;30s&lt;/span&gt;
&lt;span class="py"&gt;StartLimitBurst&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The defaults come from &lt;code&gt;DefaultStartLimitIntervalSec=10s&lt;/code&gt; and &lt;code&gt;DefaultStartLimitBurst=5&lt;/code&gt; in &lt;code&gt;systemd-system.conf(5)&lt;/code&gt;. In plain English: no more than five starts in any ten-second window, unless the unit overrides it.&lt;/p&gt;

&lt;p&gt;Several properties of this limiter cause most of the confusion:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every start counts&lt;/strong&gt;, including &lt;code&gt;systemctl start&lt;/code&gt;, timer activations, socket activations, and automatic restarts. It is not a restart counter; it is a start counter.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The check runs after unit conditions.&lt;/strong&gt; A start that fails a &lt;code&gt;Condition...=&lt;/code&gt; does not consume an attempt. That is why a unit with a failing condition can look like it is ignoring the limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hitting the limit stops automatic restarts&lt;/strong&gt;, but the unit is not permanently dead. It can be started later, after the interval, by hand or by a timer or socket. During the interval, even a manual start is refused.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;systemctl reset-failed&lt;/code&gt; flushes the counter.&lt;/strong&gt; That is the supported way to say "I have fixed the problem, let me try again now."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;StartLimitIntervalSec=0&lt;/code&gt; disables rate limiting entirely.&lt;/strong&gt; &lt;code&gt;infinity&lt;/code&gt; instead caps the total number of starts regardless of how far apart they are.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Garbage collection flushes the counters too.&lt;/strong&gt; A unit that is unloaded because nothing references it any more loses its start history. Rate limiting is only meaningful for units that stay loaded.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last point is subtle, and it is also why the limiter is not a security boundary. It is a brake for crash loops, not an audit mechanism.&lt;/p&gt;

&lt;h2&gt;
  
  
  A reproducible crash loop
&lt;/h2&gt;

&lt;p&gt;You can watch both mechanisms interact without touching any production service. The following uses a &lt;strong&gt;user&lt;/strong&gt; unit, so no root is required. Create the unit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; ~/.config/systemd/user

&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; ~/.config/systemd/user/lyra-restart-demo.service &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
[Unit]
Description=Restart limit demo
StartLimitIntervalSec=30s
StartLimitBurst=3

[Service]
Type=exec
ExecStart=/bin/false
Restart=always
RestartSec=200ms

[Install]
WantedBy=default.target
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; daemon-reload
systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; start lyra-restart-demo.service
&lt;span class="nb"&gt;sleep &lt;/span&gt;3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;/bin/false&lt;/code&gt; exits 1 immediately, so every start fails. On a systemd 257 host the journal ends like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Scheduled restart job, restart counter is at 3.
Start request repeated too quickly.
Failed with result 'exit-code'.
Failed to start lyra-restart-demo.service.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the properties show the state:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; show lyra-restart-demo.service &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; NRestarts &lt;span class="nt"&gt;-p&lt;/span&gt; Result &lt;span class="nt"&gt;-p&lt;/span&gt; ActiveState &lt;span class="nt"&gt;-p&lt;/span&gt; SubState &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; StartLimitIntervalUSec &lt;span class="nt"&gt;-p&lt;/span&gt; StartLimitBurst
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NRestarts=3
Result=exit-code
ActiveState=failed
SubState=failed
StartLimitIntervalUSec=30s
StartLimitBurst=3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note what &lt;code&gt;Result=&lt;/code&gt; says: &lt;code&gt;exit-code&lt;/code&gt;, not "rate limit". It records why the service last failed, not why systemd stopped trying. If you only read &lt;code&gt;Result=&lt;/code&gt;, you will miss the limiter completely. Depending on the systemd version, you may instead see a result such as &lt;code&gt;start-limit-hit&lt;/code&gt;; do not build tooling that depends on either string alone.&lt;/p&gt;

&lt;p&gt;Now try a manual start inside the 30-second window:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; start lyra-restart-demo.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It fails. The unit is not broken; the counter is still over budget. Flush it and try again:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; reset-failed lyra-restart-demo.service
systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; show lyra-restart-demo.service &lt;span class="nt"&gt;-p&lt;/span&gt; NRestarts &lt;span class="nt"&gt;-p&lt;/span&gt; Result &lt;span class="nt"&gt;-p&lt;/span&gt; ActiveState
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NRestarts=0
Result=success
ActiveState=inactive
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A start now works, and the same crash loop runs until it hits the limit again. Clean up when you are done:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; stop lyra-restart-demo.service &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true
rm&lt;/span&gt; ~/.config/systemd/user/lyra-restart-demo.service
systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; daemon-reload
systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; reset-failed lyra-restart-demo.service &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a system-wide unit, use &lt;code&gt;sudo systemctl&lt;/code&gt; and put the file in &lt;code&gt;/etc/systemd/system/&lt;/code&gt;. The semantics are identical; only the manager instance changes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Diagnosing the real thing
&lt;/h2&gt;

&lt;p&gt;When a unit is sitting in &lt;code&gt;failed&lt;/code&gt; and you do not know which mechanism fired, collect these four things before changing any configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl status myapp.service

systemctl show myapp.service &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; Restart &lt;span class="nt"&gt;-p&lt;/span&gt; RestartUSec &lt;span class="nt"&gt;-p&lt;/span&gt; NRestarts &lt;span class="nt"&gt;-p&lt;/span&gt; Result &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; ActiveState &lt;span class="nt"&gt;-p&lt;/span&gt; SubState &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; StartLimitIntervalUSec &lt;span class="nt"&gt;-p&lt;/span&gt; StartLimitBurst

journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; myapp.service &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nt"&gt;-g&lt;/span&gt; &lt;span class="s1"&gt;'Scheduled restart|repeated too quickly|Failed with result'&lt;/span&gt;

systemctl list-units &lt;span class="nt"&gt;--state&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;failed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read them in order:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Restart=&lt;/code&gt; tells you the policy. &lt;code&gt;RestartUSec=&lt;/code&gt; tells you the configured delay.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;NRestarts=&lt;/code&gt; tells you how many automatic restart jobs the manager has scheduled. A number at or above &lt;code&gt;StartLimitBurst=&lt;/code&gt; is a strong hint that the limiter is involved.&lt;/li&gt;
&lt;li&gt;The journal line &lt;code&gt;Start request repeated too quickly&lt;/code&gt; is the limiter speaking. If you see it, the exit cause is no longer the only problem.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Result=&lt;/code&gt; tells you why the unit last failed. It is useful for the application bug, not for the rate limit.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Once the underlying crash is fixed, &lt;code&gt;systemctl reset-failed myapp.service&lt;/code&gt; clears the counter so the next start is allowed immediately. Without it, you may have to wait out the interval.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recipes that hold up
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. A crash loop should back off, not hammer the box
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Unit]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;StartLimitIntervalSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;60s&lt;/span&gt;
&lt;span class="py"&gt;StartLimitBurst&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Restart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;on-failure&lt;/span&gt;
&lt;span class="py"&gt;RestartSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2s&lt;/span&gt;
&lt;span class="py"&gt;RestartSteps&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;4&lt;/span&gt;
&lt;span class="py"&gt;RestartMaxDelaySec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;32s&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This gives five attempts in a minute, with growing gaps between them. The service still ends up failed if it is genuinely broken, but it does not spend the minute in a tight fork/exec loop filling the journal.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Do not restart on a configuration error
&lt;/h3&gt;

&lt;p&gt;A missing config file or a bad flag will not fix itself. If the application exits with &lt;code&gt;EX_CONFIG&lt;/code&gt; (78, from &lt;code&gt;sysexits.h&lt;/code&gt;), exclude that status from the restart logic and fail fast:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;ExecStartPre&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/bin/myapp --check-config&lt;/span&gt;
&lt;span class="py"&gt;Restart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;on-failure&lt;/span&gt;
&lt;span class="py"&gt;RestartPreventExitStatus&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;78&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;RestartPreventExitStatus=&lt;/code&gt; accepts numeric statuses, status names, and signal names. The pre-start check is not strictly required, but it turns a restart storm into one clear error message.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Restart after a clean self-exit
&lt;/h3&gt;

&lt;p&gt;Some daemons exit 0 on purpose to request a restart, for example after a self-update. With &lt;code&gt;Restart=on-failure&lt;/code&gt; that clean exit would normally be ignored. &lt;code&gt;RestartForceExitStatus=&lt;/code&gt; overrides the classification:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Restart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;on-failure&lt;/span&gt;
&lt;span class="py"&gt;RestartForceExitStatus&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This works, but be careful: a service that exits 0 immediately will now loop until the start limit stops it. Only use it when exit 0 genuinely means "restart me", not "I am done".&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Do not fail dependents for a temporary blip
&lt;/h3&gt;

&lt;p&gt;If a service depends on something that briefly disappears, a normal restart goes through the failed state, and units with &lt;code&gt;Requires=&lt;/code&gt; or &lt;code&gt;BindsTo=&lt;/code&gt; can be torn down with it. &lt;code&gt;RestartMode=direct&lt;/code&gt; (systemd 254+) restarts without that failed/inactive transition:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Restart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;on-failure&lt;/span&gt;
&lt;span class="py"&gt;RestartMode&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;direct&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;ExecStopPost=&lt;/code&gt; still runs, but &lt;code&gt;OnSuccess=&lt;/code&gt; and &lt;code&gt;OnFailure=&lt;/code&gt; are skipped and dependent units are not notified of the temporary failure. That is exactly what you want for a flaky network dependency, and exactly what you do not want if the failure should propagate.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Leave the rate limit alone unless you have a reason
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;StartLimitIntervalSec=0&lt;/code&gt; is a real setting and it does what it says: no rate limiting at all. It is also how a broken service turns into a host-wide CPU and disk problem. If a service must retry forever, give it a long interval and a large burst with exponential backoff, not an infinite one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Restart=&lt;/code&gt; matches the &lt;strong&gt;cause of death&lt;/strong&gt;: exit code, signal, timeout, or watchdog. Read the matrix, not the name.&lt;/li&gt;
&lt;li&gt;A clean exit includes 0 and, for non-oneshot services, SIGHUP/SIGINT/SIGTERM/SIGPIPE, plus &lt;code&gt;SuccessExitStatus=&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Type=oneshot&lt;/code&gt; cannot use &lt;code&gt;Restart=always&lt;/code&gt; or &lt;code&gt;on-success&lt;/code&gt;; use a timer.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;RestartSec=&lt;/code&gt; defaults to 100 ms. Add &lt;code&gt;RestartSteps=&lt;/code&gt; and &lt;code&gt;RestartMaxDelaySec=&lt;/code&gt; for backoff.&lt;/li&gt;
&lt;li&gt;Start limits count &lt;strong&gt;all&lt;/strong&gt; starts, not just restarts, and can refuse a manual start inside the window.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemctl reset-failed&lt;/code&gt; clears the counter; &lt;code&gt;StartLimitIntervalSec=0&lt;/code&gt; disables the brake entirely.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Result=&lt;/code&gt; is not proof that the rate limit fired. The journal message and &lt;code&gt;NRestarts=&lt;/code&gt; are.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The next time a unit "stops restarting", do not raise the burst blindly. Find out whether the process is dying for a reason that &lt;code&gt;Restart=&lt;/code&gt; classifies as restartable, and whether the limiter has already decided the unit has had enough starts for now.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;systemd.service(5), "Restart=", "RestartSec=", "RestartSteps=", "RestartMaxDelaySec=", "RestartMode=", "SuccessExitStatus=", "RestartPreventExitStatus=", "RestartForceExitStatus=", Table 1: &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html" rel="noopener noreferrer"&gt;https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;systemd.unit(5), "StartLimitIntervalSec=, StartLimitBurst=": &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html" rel="noopener noreferrer"&gt;https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;systemd-system.conf(5), "DefaultStartLimitIntervalSec=, DefaultStartLimitBurst=", "DefaultRestartSec=": &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html" rel="noopener noreferrer"&gt;https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;systemd 229 release notes (start-limit settings moved from [Service] to [Unit], still accepted at the old location): &lt;a href="https://github.com/systemd/systemd/blob/main/NEWS" rel="noopener noreferrer"&gt;https://github.com/systemd/systemd/blob/main/NEWS&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;sysexits.h&lt;/code&gt;, &lt;code&gt;EX_CONFIG&lt;/code&gt; = 78: &lt;a href="https://man.freebsd.org/cgi/man.cgi?query=sysexits&amp;amp;sektion=3" rel="noopener noreferrer"&gt;https://man.freebsd.org/cgi/man.cgi?query=sysexits&amp;amp;sektion=3&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Commands and observed output in this article were tested on systemd 257.13 (Debian 13) with a user manager. Values that changed in earlier releases are noted where relevant.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>linux</category>
      <category>systemd</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Your CPU Quota Is a Budget, Not a Percentage: Practical cgroup v2 CPU Control on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Sat, 07 Nov 2026 05:04:02 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/your-cpu-quota-is-a-budget-not-a-percentage-practical-cgroup-v2-cpu-control-on-linux-2lmp</link>
      <guid>https://dev.to/lyraalishaikh/your-cpu-quota-is-a-budget-not-a-percentage-practical-cgroup-v2-cpu-control-on-linux-2lmp</guid>
      <description>&lt;p&gt;A batch job gets &lt;code&gt;CPUQuota=50%&lt;/code&gt;. The box still goes to 100% CPU, the API on the same host starts missing deadlines, and someone "fixes" it by raising the quota to 200%. Now the batch job is eating two full cores and the API is still slow.&lt;/p&gt;

&lt;p&gt;The number felt like a percentage of the machine. The kernel treats it as &lt;strong&gt;microseconds of CPU time per period&lt;/strong&gt;. Those are the same thing only if you know the period, how many threads are runnable, and what happens when the budget runs out. That last part is what usually surprises people: a cgroup that exceeds its quota is not slowed down proportionally. It is stopped until the next period starts.&lt;/p&gt;

&lt;p&gt;cgroup v2's CPU controller has three separate mechanisms that answer three different questions. Mixing them up is why CPU limits feel unpredictable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three questions, three knobs
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What you actually want&lt;/th&gt;
&lt;th&gt;systemd setting&lt;/th&gt;
&lt;th&gt;Kernel interface&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;"This unit must never exceed X CPU"&lt;/td&gt;
&lt;td&gt;&lt;code&gt;CPUQuota=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;cpu.max&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Under contention, this unit should win"&lt;/td&gt;
&lt;td&gt;&lt;code&gt;CPUWeight=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;cpu.weight&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"This unit may only run on these cores"&lt;/td&gt;
&lt;td&gt;&lt;code&gt;AllowedCPUs=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;cpuset.cpus&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Only use CPU nobody else wants"&lt;/td&gt;
&lt;td&gt;&lt;code&gt;CPUWeight=idle&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;cpu.idle&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;They compose, but they are not interchangeable. A quota is enforced by throttling. A weight is enforced by the scheduler's fairness math and does exactly nothing when the machine is idle. A cpuset changes placement, not capacity.&lt;/p&gt;

&lt;h2&gt;
  
  
  The arithmetic of cpu.max
&lt;/h2&gt;

&lt;p&gt;The file is two numbers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/system.slice/ffmpeg-worker.service/cpu.max
200000 100000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The format is &lt;code&gt;$MAX $PERIOD&lt;/code&gt;, both in microseconds. This group may consume up to 200 ms of CPU time in every 100 ms period, which is two CPUs' worth. The default is &lt;code&gt;max 100000&lt;/code&gt;: unlimited quota, 100 ms period. Writing a single number updates only &lt;code&gt;$MAX&lt;/code&gt;; writing &lt;code&gt;max&lt;/code&gt; removes the limit.&lt;/p&gt;

&lt;p&gt;So the conversion is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPUs = MAX / PERIOD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;code&gt;cpu.max&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;max 100000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Unlimited&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;100000 100000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;1 CPU&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;25000 100000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;25% of one CPU&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;200000 100000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;2 CPUs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;50000 50000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;1 CPU, shorter 50 ms period&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;systemd's &lt;code&gt;CPUQuota=&lt;/code&gt; is a percentage &lt;strong&gt;of one CPU&lt;/strong&gt;, which is why the man page tells you to use values above 100% for more than one CPU. &lt;code&gt;CPUQuota=150%&lt;/code&gt; becomes &lt;code&gt;150000 100000&lt;/code&gt;. &lt;code&gt;CPUQuotaPeriodSec=&lt;/code&gt; changes the second field; it defaults to 100 ms and is clamped to the range the kernel supports, 1 ms to 1000 ms.&lt;/p&gt;

&lt;p&gt;The kernel's own floor is 1 ms for quota or period, and the period cannot exceed 1 s. Those limits come from &lt;code&gt;Documentation/scheduler/sched-bwc.rst&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a quota feels like stuttering, not slowing
&lt;/h2&gt;

&lt;p&gt;CFS bandwidth control does not ration CPU smoothly. Within a period, the group's quota is handed out to per-CPU run queues in slices as threads become runnable. When the budget is gone, every thread in the group is throttled until the next period refills it.&lt;/p&gt;

&lt;p&gt;Picture &lt;code&gt;CPUQuota=25%&lt;/code&gt; on a machine with free cores. The group gets 25 ms of runtime per 100 ms. A single busy thread runs for 25 ms, then sits parked for 75 ms. Average utilisation is 25%, but the shape is a 25 ms burst followed by a 75 ms stall. If that thread is serving requests, p99 latency now includes the stall.&lt;/p&gt;

&lt;p&gt;I ran exactly this on a 3-vCPU Debian 13 host (kernel 6.12, systemd 257):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-run &lt;span class="nt"&gt;--user&lt;/span&gt; &lt;span class="nt"&gt;--unit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cpu-lab &lt;span class="nt"&gt;--property&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;CPUQuota&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;25% &lt;span class="se"&gt;\&lt;/span&gt;
  /bin/bash &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'while :; do :; done'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; show cpu-lab &lt;span class="nt"&gt;-p&lt;/span&gt; CPUQuotaPerSecUSec &lt;span class="nt"&gt;-p&lt;/span&gt; ControlGroup
&lt;span class="c"&gt;# CPUQuotaPerSecUSec=250ms&lt;/span&gt;
&lt;span class="c"&gt;# ControlGroup=/user.slice/user-1000.slice/user@1000.service/app.slice/cpu-lab.service&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kernel file matched the arithmetic:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/cpu-lab.service/cpu.max
25000 100000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sampling &lt;code&gt;CPUUsageNSec&lt;/code&gt; over a 4.02 s window showed 1.005 s of CPU time consumed, or &lt;strong&gt;25.0% of one CPU&lt;/strong&gt;. Meanwhile &lt;code&gt;cpu.stat&lt;/code&gt; showed the group had been throttled in 30 of 31 periods. It was not running slowly. It was running fast, then not at all.&lt;/p&gt;

&lt;p&gt;That distinction drives most of the practical advice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Latency-sensitive services usually want a &lt;strong&gt;generous quota&lt;/strong&gt; plus a &lt;strong&gt;weight&lt;/strong&gt;, not a tight quota. A tight quota converts load into tail latency.&lt;/li&gt;
&lt;li&gt;Batch work that must not disturb anything else is the opposite case: a tight quota is correct, and stalls do not matter.&lt;/li&gt;
&lt;li&gt;If you need finer-grained enforcement, shorten the period. Below ~10 ms you pay more accounting overhead for little gain, and systemd will not go below 1 ms.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Read the counters before changing anything
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;cpu.stat&lt;/code&gt; is where the truth is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ cat /sys/fs/cgroup/.../cpu-lab.service/cpu.stat
usage_usec 792598
user_usec 781696
system_usec 10901
nr_periods 31
nr_throttled 30
throttled_usec 2247452
nr_bursts 0
burst_usec 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first three counters always exist. &lt;code&gt;nr_periods&lt;/code&gt;, &lt;code&gt;nr_throttled&lt;/code&gt;, &lt;code&gt;throttled_usec&lt;/code&gt;, &lt;code&gt;nr_bursts&lt;/code&gt;, and &lt;code&gt;burst_usec&lt;/code&gt; appear when the controller is enabled, and only account for fair-class scheduling.&lt;/p&gt;

&lt;p&gt;Two warnings about interpretation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;These five counters are &lt;strong&gt;not hierarchical&lt;/strong&gt;. They report throttling caused by this cgroup's own limit, not throttling inherited from a parent slice. For the time actually lost including ancestors, read &lt;code&gt;cpu.stat.local&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;For a fully CPU-bound group, &lt;code&gt;nr_throttled&lt;/code&gt; roughly equals &lt;code&gt;nr_periods&lt;/code&gt;. That is normal, not a bug: the group always uses its whole budget and always hits the wall. A ratio near zero means the quota is not binding and the limit is decoration.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A useful rule of thumb: if &lt;code&gt;throttled_usec&lt;/code&gt; divided by wall-clock time is above roughly 20%, the quota is shaping your workload's behaviour in a way users will feel.&lt;/p&gt;

&lt;h2&gt;
  
  
  Burst: borrow against future quiet time
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;cpu.max.burst&lt;/code&gt; (kernel file) lets a cgroup accumulate unused quota and spend it later, up to the value you set. Default is &lt;code&gt;0&lt;/code&gt;; the valid range is 0 to the current &lt;code&gt;$MAX&lt;/code&gt;. &lt;code&gt;nr_bursts&lt;/code&gt; and &lt;code&gt;burst_usec&lt;/code&gt; in &lt;code&gt;cpu.stat&lt;/code&gt; count the periods where it happened.&lt;/p&gt;

&lt;p&gt;The kernel documentation describes this as borrowing against future underrun, and it is explicitly a trade: you keep the average, accept short interference with other groups, and stop throttling on spikes that are followed by idle time. It is a good fit for request-driven services with bursty CPU and a bad fit for a hard isolation requirement.&lt;/p&gt;

&lt;p&gt;Not every kernel exposes the file. Check with &lt;code&gt;ls&lt;/code&gt; in the cgroup directory before you plan around it.&lt;/p&gt;

&lt;h2&gt;
  
  
  cpu.weight is fairness, not a cap
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;cpu.weight&lt;/code&gt; lives in the range 1 to 10000, default 100. It only distributes CPU time among &lt;strong&gt;siblings that have runnable tasks&lt;/strong&gt;. A lone service with weight 100 on an idle box still gets every core. Weight changes nothing until there is contention, and then it changes everything about who waits.&lt;/p&gt;

&lt;p&gt;The split is proportional to weight among the children of the same parent. It is not transitive: two services inside &lt;code&gt;system.slice&lt;/code&gt; compete with each other, but their weights do not let them outrank a unit in &lt;code&gt;user.slice&lt;/code&gt;; the slice weights decide that level.&lt;/p&gt;

&lt;p&gt;I measured a 100 vs 300 pair on 3 vCPUs, each unit running four busy threads so both would genuinely contend:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;w &lt;span class="k"&gt;in &lt;/span&gt;100 300&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;systemd-run &lt;span class="nt"&gt;--user&lt;/span&gt; &lt;span class="nt"&gt;--unit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"demo-&lt;/span&gt;&lt;span class="nv"&gt;$w&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--property&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;CPUWeight&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$w&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    /bin/bash &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'for i in 1 2 3 4; do while :; do :; done &amp;amp; done; wait'&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Over a 6 s window: 4.52 s of CPU time for weight 100 and 13.47 s for weight 300. That is 0.75 CPU versus 2.24 CPU, a ratio of about 1:3, adding up to almost exactly the 3 vCPUs available. Weights divided the machine in proportion to 100:300.&lt;/p&gt;

&lt;p&gt;Two more details matter in production:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;cpu.weight.nice&lt;/code&gt; is an alternative view using the familiar −20..19 range. It is a coarser mapping onto the same weight.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;CPUWeight=idle&lt;/code&gt; sets &lt;code&gt;cpu.idle&lt;/code&gt;, which makes the cgroup run only when nothing else in the same parent or its siblings wants the CPU. This is the right setting for backup indexers, thumbnailers, and anything that should consume leftovers.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  cpuset: placement, partitions, and a delegation trap
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;cpuset.cpus&lt;/code&gt; restricts which CPUs a group's tasks may run on. systemd exposes it as &lt;code&gt;AllowedCPUs=&lt;/code&gt; (since v244). The requested set is not the granted set: the effective set is the intersection with every ancestor, readable as &lt;code&gt;EffectiveCPUs=&lt;/code&gt; in systemd and &lt;code&gt;cpuset.cpus.effective&lt;/code&gt; in the kernel.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;AllowedCPUs&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2-5&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a 3-vCPU host, I tried &lt;code&gt;AllowedCPUs=0&lt;/code&gt; on a &lt;strong&gt;user&lt;/strong&gt; unit and got &lt;code&gt;EffectiveCPUs=&lt;/code&gt; empty with no &lt;code&gt;cpuset.cpus&lt;/code&gt; file at all. The per-user manager on that host had only &lt;code&gt;cpu memory pids&lt;/code&gt; in &lt;code&gt;cgroup.controllers&lt;/code&gt;, so the cpuset controller was never available to apply the setting. The configuration was accepted and silently did nothing.&lt;/p&gt;

&lt;p&gt;That is the trap: always verify placement with &lt;code&gt;EffectiveCPUs&lt;/code&gt;, and run cpuset experiments as system units (or inside a delegated container cgroup) where the controller exists.&lt;/p&gt;

&lt;p&gt;Beyond simple pinning, cgroup v2 supports &lt;strong&gt;cpuset partitions&lt;/strong&gt;. Writing &lt;code&gt;root&lt;/code&gt; to &lt;code&gt;cpuset.cpus.partition&lt;/code&gt; turns a cgroup into a scheduling domain with exclusive CPUs from &lt;code&gt;cpuset.cpus.exclusive&lt;/code&gt;. Writing &lt;code&gt;isolated&lt;/code&gt; also removes scheduler load balancing and keeps unbound workqueues off those CPUs. Partitions can end up in an &lt;code&gt;invalid&lt;/code&gt; state if the exclusive set overlaps a sibling's, and the file then reads back something like &lt;code&gt;root invalid (&amp;lt;reason&amp;gt;)&lt;/code&gt; — which is your cue to fix the hierarchy rather than the workload.&lt;/p&gt;

&lt;p&gt;Use this when you genuinely want dedicated cores: latency-critical inference, DPDK-style packet processing, real-time-ish audio. Do not use it to "reserve" capacity you are not actually using; an isolated core with no runnable task is just an idle core nobody else may borrow.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;cpuset.mems&lt;/code&gt; is the NUMA half of the same controller. On multi-socket boxes, pinning CPUs without pinning memory can send every access across the interconnect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wiring it into a unit, then proving it
&lt;/h2&gt;

&lt;p&gt;A drop-in keeps the distribution unit file untouched:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/systemd/system/ffmpeg-worker.service.d/20-cpu.conf
&lt;/span&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;CPUAccounting&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;CPUQuota&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;200%&lt;/span&gt;
&lt;span class="py"&gt;CPUQuotaPeriodSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;100ms&lt;/span&gt;
&lt;span class="py"&gt;CPUWeight&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;200&lt;/span&gt;
&lt;span class="py"&gt;AllowedCPUs&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;0-1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart ffmpeg-worker.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For experiments you do not want to keep:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl set-property &lt;span class="nt"&gt;--runtime&lt;/span&gt; ffmpeg-worker.service &lt;span class="nv"&gt;CPUQuota&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;150%
&lt;span class="c"&gt;# drop --runtime to persist the change as a drop-in&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verify the effective configuration rather than the file you wrote:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl show ffmpeg-worker.service &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; CPUQuotaPerSecUSec &lt;span class="nt"&gt;-p&lt;/span&gt; CPUWeight &lt;span class="nt"&gt;-p&lt;/span&gt; AllowedCPUs &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-p&lt;/span&gt; EffectiveCPUs &lt;span class="nt"&gt;-p&lt;/span&gt; CPUUsageNSec
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;CPUQuotaPerSecUSec=1.5s&lt;/code&gt; means 150% of one CPU. Empty &lt;code&gt;EffectiveCPUs&lt;/code&gt; means your cpuset request is not being applied. And a live view of who is actually consuming:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-cgtop &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="nt"&gt;-1&lt;/span&gt; &lt;span class="nt"&gt;--depth&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the flags: &lt;code&gt;-b&lt;/code&gt; is batch mode, &lt;code&gt;-c&lt;/code&gt; orders by CPU, &lt;code&gt;-1&lt;/code&gt; runs a single iteration. (&lt;code&gt;-m&lt;/code&gt; orders by memory, which is a different thing.)&lt;/p&gt;

&lt;h2&gt;
  
  
  A reproducible lab
&lt;/h2&gt;

&lt;p&gt;This script reports a unit's achieved CPU usage as a percentage of one CPU. It uses &lt;code&gt;systemctl&lt;/code&gt; by default; set &lt;code&gt;SYSTEMCTL="systemctl --user"&lt;/code&gt; to sample user units.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# cpu-usage.sh &amp;lt;unit&amp;gt; [seconds]&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-euo&lt;/span&gt; pipefail
&lt;span class="nv"&gt;unit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;:?usage:&lt;span class="p"&gt; cpu-usage.sh &amp;lt;unit&amp;gt; [seconds]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;secs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;2&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;10&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;sctl&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SYSTEMCTL&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;systemctl&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;$sctl&lt;/span&gt; show &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$unit&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; CPUUsageNSec &lt;span class="nt"&gt;--value&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;t0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; +%s%N&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;sleep&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$secs&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;b&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;$sctl&lt;/span&gt; show &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$unit&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; CPUUsageNSec &lt;span class="nt"&gt;--value&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;t1&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; +%s%N&lt;span class="si"&gt;)&lt;/span&gt;

&lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$a&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nv"&gt;b&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$b&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nv"&gt;e&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt;t1 &lt;span class="o"&gt;-&lt;/span&gt; t0&lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nv"&gt;u&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$unit&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="s1"&gt;'BEGIN {
  printf "%s: %.1f%% of one CPU over %.2fs\n", u, (b - a) / e * 100, e / 1e9
}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Full quota test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-run &lt;span class="nt"&gt;--user&lt;/span&gt; &lt;span class="nt"&gt;--unit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cpu-lab &lt;span class="nt"&gt;--property&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;CPUQuota&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;25% &lt;span class="se"&gt;\&lt;/span&gt;
  /bin/bash &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'while :; do :; done'&lt;/span&gt;
&lt;span class="nb"&gt;sleep &lt;/span&gt;3
&lt;span class="nv"&gt;SYSTEMCTL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"systemctl --user"&lt;/span&gt; ./cpu-usage.sh cpu-lab 10
systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; stop cpu-lab
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected: roughly 25%, with &lt;code&gt;nr_throttled&lt;/code&gt; close to &lt;code&gt;nr_periods&lt;/code&gt;. If you see 100%, the controller is not active or the quota did not land; check &lt;code&gt;cpu.max&lt;/code&gt; directly.&lt;/p&gt;

&lt;p&gt;Then change the quota live and watch the number follow:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl &lt;span class="nt"&gt;--user&lt;/span&gt; set-property &lt;span class="nt"&gt;--runtime&lt;/span&gt; cpu-lab &lt;span class="nv"&gt;CPUQuota&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;75%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Failure modes worth checking first
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A quota below one request's CPU cost.&lt;/strong&gt; The request cannot finish inside its slice, gets throttled, and latency explodes. Raise the quota, shorten the period, or accept that the service needs a weight instead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"50% of the machine" thinking.&lt;/strong&gt; &lt;code&gt;CPUQuota=50%&lt;/code&gt; is half of one CPU, total, across all threads. On a 32-core host that is 1.5% of capacity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weights on an idle box.&lt;/strong&gt; Nothing happens, correctly. Weight is a contention policy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weights across slices.&lt;/strong&gt; A service cannot outrank a sibling slice by weight alone; the parent slice's weight sets the ceiling.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;cpuset without the controller.&lt;/strong&gt; &lt;code&gt;AllowedCPUs=&lt;/code&gt; is accepted, &lt;code&gt;EffectiveCPUs&lt;/code&gt; stays empty, and the service runs everywhere. Check delegation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Real-time threads.&lt;/strong&gt; cgroup v2's CPU bandwidth control does not cover real-time tasks in the way you might assume; the kernel documentation carries an explicit warning about &lt;code&gt;CONFIG_RT_GROUP_SCHED&lt;/code&gt; and RT processes in non-root cgroups.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Autogrouping.&lt;/strong&gt; &lt;code&gt;sched(7)&lt;/code&gt; describes automatic session-based group scheduling, which behaves similarly to weights but is a different mechanism. Do not debug one while looking at the other.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Container runtimes writing the same file.&lt;/strong&gt; A Kubernetes CPU limit and a systemd drop-in both end up in &lt;code&gt;cpu.max&lt;/code&gt;. The last writer wins. When numbers disagree, cat the file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Steal time on VMs.&lt;/strong&gt; Quota is enforced against vCPU time. A guest can be far below its quota and still slow because the hypervisor is not scheduling it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Rollback
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo rm&lt;/span&gt; /etc/systemd/system/ffmpeg-worker.service.d/20-cpu.conf
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart ffmpeg-worker.service
&lt;span class="c"&gt;# or, to drop all local overrides for the unit:&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl revert ffmpeg-worker.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An empty &lt;code&gt;CPUQuota=&lt;/code&gt; unsets the quota; &lt;code&gt;cpu.max&lt;/code&gt; returns to &lt;code&gt;max 100000&lt;/code&gt;. &lt;code&gt;CPUWeight=&lt;/code&gt; unset returns the kernel default of 100.&lt;/p&gt;

&lt;p&gt;CPU control is one of the few places where the kernel's model is simpler than the folklore. A quota is a per-period budget that ends in a hard stop. A weight is a tie-breaker that only matters when someone else wants the same cycles. A cpuset is placement. Pick the one that matches the promise you are making, then read &lt;code&gt;cpu.stat&lt;/code&gt; and &lt;code&gt;EffectiveCPUs&lt;/code&gt; to confirm the promise is being kept.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Linux kernel documentation, &lt;em&gt;Control Group v2&lt;/em&gt;, CPU and Cpuset controller sections: &lt;a href="https://docs.kernel.org/admin-guide/cgroup-v2.html" rel="noopener noreferrer"&gt;https://docs.kernel.org/admin-guide/cgroup-v2.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Linux kernel documentation, &lt;em&gt;CFS Bandwidth Control&lt;/em&gt;: &lt;a href="https://docs.kernel.org/scheduler/sched-bwc.html" rel="noopener noreferrer"&gt;https://docs.kernel.org/scheduler/sched-bwc.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd.resource-control(5)&lt;/code&gt;, CPUWeight=, CPUQuota=, CPUQuotaPeriodSec=, AllowedCPUs=: &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html" rel="noopener noreferrer"&gt;https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemd-cgtop(1)&lt;/code&gt;: &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-cgtop.html" rel="noopener noreferrer"&gt;https://www.freedesktop.org/software/systemd/man/latest/systemd-cgtop.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;sched(7)&lt;/code&gt;, autogroup feature: &lt;a href="https://man7.org/linux/man-pages/man7/sched.7.html" rel="noopener noreferrer"&gt;https://man7.org/linux/man-pages/man7/sched.7.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Pressure stall information, &lt;code&gt;cpu.pressure&lt;/code&gt;: &lt;a href="https://docs.kernel.org/accounting/psi.html" rel="noopener noreferrer"&gt;https://docs.kernel.org/accounting/psi.html&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>linux</category>
      <category>performance</category>
      <category>devops</category>
      <category>systemd</category>
    </item>
    <item>
      <title>Stop Guessing Why Your Crash Left No Core: Practical systemd-coredump on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Fri, 06 Nov 2026 05:04:30 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-guessing-why-your-crash-left-no-core-practical-systemd-coredump-on-linux-4276</link>
      <guid>https://dev.to/lyraalishaikh/stop-guessing-why-your-crash-left-no-core-practical-systemd-coredump-on-linux-4276</guid>
      <description>&lt;p&gt;At 03:12 a service segfaults. &lt;code&gt;systemctl status&lt;/code&gt; is unambiguous:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Main process exited, code=dumped, status=11/SEGV
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At 09:00 you run &lt;code&gt;coredumpctl list&lt;/code&gt; and get nothing. The natural conclusion is that the core was never written, or that the disk was full, or that someone deleted it. Usually none of those is true. The core passed some gates and failed a later one, and the tool that lists cores is not the tool that stores them.&lt;/p&gt;

&lt;p&gt;This is a practical map of the path a core takes on a systemd host: the six places it can disappear, the knobs that control each one, and the commands that prove what your machine is actually doing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The path a core takes
&lt;/h2&gt;

&lt;p&gt;A core dump is not one event. It is a chain:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Who acts&lt;/th&gt;
&lt;th&gt;What can go wrong&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Signal delivered&lt;/td&gt;
&lt;td&gt;kernel&lt;/td&gt;
&lt;td&gt;Process is not dumpable (setuid, file capabilities, unreadable binary)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Handler selected&lt;/td&gt;
&lt;td&gt;&lt;code&gt;kernel.core_pattern&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Pattern is &lt;code&gt;core&lt;/code&gt;, empty, or points at a no-op handler&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Limit checked&lt;/td&gt;
&lt;td&gt;kernel and handler&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;RLIMIT_CORE&lt;/code&gt; is zero&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pages selected&lt;/td&gt;
&lt;td&gt;kernel (&lt;code&gt;coredump_filter&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Sensitive or useful mappings excluded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Core stored&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemd-coredump&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Storage=&lt;/code&gt;, &lt;code&gt;ProcessSizeMax=&lt;/code&gt;, &lt;code&gt;ExternalSizeMax=&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Core retained&lt;/td&gt;
&lt;td&gt;tmpfiles, &lt;code&gt;MaxUse=&lt;/code&gt;, journal rotation&lt;/td&gt;
&lt;td&gt;File aged out, or journal entry rotated first&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each gate has a different symptom. A core that fails at gate 1 leaves no journal entry at all. A core that fails at gate 6 leaves a journal entry pointing at a file that no longer exists — which is exactly the confusing &lt;code&gt;missing&lt;/code&gt; status in &lt;code&gt;coredumpctl list&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gate 1: Is a handler even installed?
&lt;/h2&gt;

&lt;p&gt;Start with the kernel's instruction for what to do when a process dumps core:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/sys/kernel/core_pattern
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a host with the standard systemd handler, the output starts with a pipe and looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight systemd"&gt;&lt;code&gt;&lt;span class="err"&gt;|/usr/lib/systemd/systemd-coredump&lt;/span&gt; &lt;span class="err"&gt;%P&lt;/span&gt; &lt;span class="err"&gt;%u&lt;/span&gt; &lt;span class="err"&gt;%g&lt;/span&gt; &lt;span class="err"&gt;%s&lt;/span&gt; &lt;span class="err"&gt;%t&lt;/span&gt; &lt;span class="err"&gt;%c&lt;/span&gt; &lt;span class="err"&gt;%e&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact path varies by distribution (&lt;code&gt;/lib/systemd/...&lt;/code&gt; on some Debian releases, for example). What matters is the leading &lt;code&gt;|&lt;/code&gt;: the kernel will pipe the core to that program instead of writing &lt;code&gt;./core&lt;/code&gt; in the crashing process's working directory.&lt;/p&gt;

&lt;p&gt;If you see &lt;code&gt;core&lt;/code&gt;, you are in legacy mode and cores land in the current directory of the crashing process — often &lt;code&gt;/&lt;/code&gt;, often unwritable. If you see &lt;code&gt;|/bin/false&lt;/code&gt;, coredump handling is disabled. systemd writes that value during very early boot and late shutdown, when storage and the handler are not available; if it stays there for the whole uptime, the handler package is probably missing.&lt;/p&gt;

&lt;p&gt;Check that the handler units exist and are active:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl status systemd-coredump.socket &lt;span class="nt"&gt;--no-pager&lt;/span&gt;
coredumpctl &lt;span class="nt"&gt;--version&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On many distributions &lt;code&gt;systemd-coredump&lt;/code&gt; is part of the main systemd package; on Debian-derived systems it is often split into a separate package. Minimal images frequently omit it. That is the first thing to check when &lt;code&gt;coredumpctl&lt;/code&gt; itself is "command not found".&lt;/p&gt;

&lt;h2&gt;
  
  
  Gate 2: RLIMIT_CORE is checked twice, by two different layers
&lt;/h2&gt;

&lt;p&gt;This is the part almost every guide gets half right.&lt;/p&gt;

&lt;p&gt;The kernel does &lt;strong&gt;not&lt;/strong&gt; enforce &lt;code&gt;RLIMIT_CORE&lt;/code&gt; when core dumps are piped to a program. &lt;code&gt;core(5)&lt;/code&gt; says it plainly: "The RLIMIT_CORE limit is not enforced for core dumps that are piped to a program via this mechanism." So a pipe handler receives the core even when the crashing process has a zero core limit.&lt;/p&gt;

&lt;p&gt;But &lt;code&gt;systemd-coredump&lt;/code&gt; checks the limit itself and respects it. The systemd coredump documentation states that individual services, processes, or users can opt out of collection by setting &lt;code&gt;RLIMIT_CORE&lt;/code&gt; to 0, and that the handler will respect limits set by processes.&lt;/p&gt;

&lt;p&gt;Both facts matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;ulimit -c 0&lt;/code&gt; in a shell stops cores for processes started from that shell, because the handler honors the inherited limit.&lt;/li&gt;
&lt;li&gt;It does not affect anything else on the system.&lt;/li&gt;
&lt;li&gt;PID 1 raises its own &lt;code&gt;RLIMIT_CORE&lt;/code&gt; to infinity at boot, so services inherit an unlimited core limit by default. "No core by default" is not the systemd behaviour.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Check what a unit actually gets:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl show &lt;span class="nt"&gt;-p&lt;/span&gt; LimitCORE &lt;span class="nt"&gt;--value&lt;/span&gt; myapp.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and what the running process inherited:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;pid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;systemctl show &lt;span class="nt"&gt;-p&lt;/span&gt; MainPID &lt;span class="nt"&gt;--value&lt;/span&gt; myapp.service&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="s1"&gt;'Max core file size'&lt;/span&gt; /proc/&lt;span class="nv"&gt;$pid&lt;/span&gt;/limits
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The property usually reads &lt;code&gt;infinity&lt;/code&gt;, and &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/limits&lt;/code&gt; renders the same thing as &lt;code&gt;unlimited&lt;/code&gt;. If you want a specific service to never write a core, set it deliberately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/systemd/system/myapp.service.d/20-no-core.conf
&lt;/span&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;LimitCORE&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart myapp.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not reach for a global &lt;code&gt;ulimit -c 0&lt;/code&gt; in &lt;code&gt;/etc/profile&lt;/code&gt;: it only covers login shells, and it is the wrong layer for a daemon.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gate 3: Is the process dumpable?
&lt;/h2&gt;

&lt;p&gt;Even with a handler and an unlimited core limit, the kernel refuses to dump some processes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A set-user-ID or set-group-ID program owned by a different user than the real UID.&lt;/li&gt;
&lt;li&gt;A program with file capabilities.&lt;/li&gt;
&lt;li&gt;An executable that is not readable by the crashing process.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is a security feature: a core is a memory image, and a setuid binary's memory image may contain credentials. The exception is controlled by &lt;code&gt;/proc/sys/fs/suid_dumpable&lt;/code&gt;. Its default is &lt;code&gt;0&lt;/code&gt; ("disabled"). Setting it to &lt;code&gt;1&lt;/code&gt; ("debug") allows dumps but is unsafe on a multi-user host. The value &lt;code&gt;2&lt;/code&gt; ("suidsafe") allows dumps only when &lt;code&gt;core_pattern&lt;/code&gt; is an absolute path or a pipe — which is the case on a systemd host.&lt;/p&gt;

&lt;p&gt;Check the process rather than guessing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/sys/fs/suid_dumpable
getcap /usr/local/bin/myapp
&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; /usr/local/bin/myapp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A service that drops privileges is usually fine. A wrapper that is setuid root is not, and no amount of &lt;code&gt;coredump.conf&lt;/code&gt; tuning will change that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gate 4: Which pages are in the core?
&lt;/h2&gt;

&lt;p&gt;The kernel does not dump the entire address space. &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/coredump_filter&lt;/code&gt; is a bitmask of mapping types:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bit&lt;/th&gt;
&lt;th&gt;Mapping type&lt;/th&gt;
&lt;th&gt;Default&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;anonymous private&lt;/td&gt;
&lt;td&gt;on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;anonymous shared&lt;/td&gt;
&lt;td&gt;on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;file-backed private&lt;/td&gt;
&lt;td&gt;off&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;file-backed shared&lt;/td&gt;
&lt;td&gt;off&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;ELF headers&lt;/td&gt;
&lt;td&gt;on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;private huge pages&lt;/td&gt;
&lt;td&gt;on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;shared huge pages&lt;/td&gt;
&lt;td&gt;off&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;private DAX pages&lt;/td&gt;
&lt;td&gt;off&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;shared DAX pages&lt;/td&gt;
&lt;td&gt;off&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The default mask is displayed in hex as &lt;code&gt;33&lt;/code&gt;. Bits 0, 1, 4, and 5 are set. That includes the stack and the heap, both of which are anonymous private mappings — which is why a default core is useful for debugging and also why it can contain secrets.&lt;/p&gt;

&lt;p&gt;Two ways to change it:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Per process, before it crashes.&lt;/strong&gt; The value is inherited across &lt;code&gt;fork()&lt;/code&gt; and preserved across &lt;code&gt;execve()&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo &lt;/span&gt;0x11 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /proc/self/coredump_filter
&lt;span class="nb"&gt;exec&lt;/span&gt; ./myapp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;0x11&lt;/code&gt; keeps anonymous private memory and ELF headers, and drops anonymous shared and huge pages.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Per systemd unit, since systemd 246.&lt;/strong&gt; &lt;code&gt;CoredumpFilter=&lt;/code&gt; accepts whitespace-separated names or numbers (base 16), and repeated assignments are ORed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/systemd/system/myapp.service.d/30-core-filter.conf
&lt;/span&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;CoredumpFilter&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;private-anonymous elf-headers&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Valid names are &lt;code&gt;private-anonymous&lt;/code&gt;, &lt;code&gt;shared-anonymous&lt;/code&gt;, &lt;code&gt;private-file-backed&lt;/code&gt;, &lt;code&gt;shared-file-backed&lt;/code&gt;, &lt;code&gt;elf-headers&lt;/code&gt;, &lt;code&gt;private-huge&lt;/code&gt;, &lt;code&gt;shared-huge&lt;/code&gt;, &lt;code&gt;private-dax&lt;/code&gt;, &lt;code&gt;shared-dax&lt;/code&gt;, plus the special values &lt;code&gt;all&lt;/code&gt; and &lt;code&gt;default&lt;/code&gt;. If the option is unset or empty, the inherited value is left alone.&lt;/p&gt;

&lt;p&gt;Verify the effective mask on the running process:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl show &lt;span class="nt"&gt;-p&lt;/span&gt; CoredumpFilter myapp.service
&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/&lt;span class="si"&gt;$(&lt;/span&gt;systemctl show &lt;span class="nt"&gt;-p&lt;/span&gt; MainPID &lt;span class="nt"&gt;--value&lt;/span&gt; myapp.service&lt;span class="si"&gt;)&lt;/span&gt;/coredump_filter
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the honest caveat. People often try &lt;code&gt;CoredumpFilter=private-anonymous&lt;/code&gt; to remove passwords from the heap. That also removes the stack, because both are anonymous private mappings. You cannot separate heap from stack with this knob. The filter reduces the size and the set of mappings; it does not sanitize memory. If a buffer must never reach disk, mark it with &lt;code&gt;madvise(MADV_DONTDUMP)&lt;/code&gt; and clear it after use. Treat every core as sensitive data, restrict access to &lt;code&gt;/var/lib/systemd/coredump&lt;/code&gt;, and never attach one to a public ticket without reviewing it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gate 5: Storage and size policy
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;coredump.conf&lt;/code&gt; decides what happens after the core reaches the handler. The defaults are reasonable, but the interactions surprise people:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Option&lt;/th&gt;
&lt;th&gt;Default&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Storage=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;external&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;external&lt;/code&gt;, &lt;code&gt;journal&lt;/code&gt;, or &lt;code&gt;none&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Compress=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;yes&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;zstd compression for external files&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ProcessSizeMax=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;32G on 64-bit&lt;/td&gt;
&lt;td&gt;Cores larger than this are stored but not processed (no stack trace)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ExternalSizeMax=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;32G on 64-bit&lt;/td&gt;
&lt;td&gt;Maximum size of an external core file&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;JournalSizeMax=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;767M&lt;/td&gt;
&lt;td&gt;Maximum size of a journal-stored core; this is a hard journal record limit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MaxUse=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;10% of disk, capped at 4G&lt;/td&gt;
&lt;td&gt;Total space for external cores&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;KeepFree=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;15% of disk, capped at 4G&lt;/td&gt;
&lt;td&gt;Free space to preserve&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two details that are easy to miss:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;With &lt;code&gt;Storage=journal&lt;/code&gt;, the core is still written to disk first so a stack trace can be generated. &lt;code&gt;Storage=&lt;/code&gt; only controls whether the file is left there afterwards.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Storage=none&lt;/code&gt; plus &lt;code&gt;ProcessSizeMax=0&lt;/code&gt; disables all coredump handling except a log entry. That is the documented way to keep the crash record but never store memory.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A practical policy for a server that needs cores for a week but must not fill the disk:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/systemd/coredump.conf.d/60-local.conf
&lt;/span&gt;&lt;span class="nn"&gt;[Coredump]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Storage&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;external&lt;/span&gt;
&lt;span class="py"&gt;Compress&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;ProcessSizeMax&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2G&lt;/span&gt;
&lt;span class="py"&gt;ExternalSizeMax&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;4G&lt;/span&gt;
&lt;span class="py"&gt;MaxUse&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;1G&lt;/span&gt;
&lt;span class="py"&gt;KeepFree&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5G&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Validate the merged configuration, not just the file you wrote:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-analyze cat-config systemd/coredump.conf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On classic systemd, each crash is handled by a fresh &lt;code&gt;systemd-coredump@.service&lt;/code&gt; instance, so the next crash picks up the new configuration. Newer systemd (262 and later, with kernel 6.19 or later) can use a long-running &lt;code&gt;systemd-coredumpd.service&lt;/code&gt;; in that mode the configuration is read once at start, and the documented way to apply changes is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl reload systemd-coredumpd.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Gate 6: Concurrency, retention, and the two clocks
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;/proc/sys/kernel/core_pipe_limit&lt;/code&gt; limits how many crashing processes may be piped to a handler in parallel. The default &lt;code&gt;0&lt;/code&gt; is special: unlimited parallelism, but the kernel does not wait for the handler, so the handler is not guaranteed access to &lt;code&gt;/proc/&amp;lt;pid&amp;gt;&lt;/code&gt; of the crashing process. A non-zero value makes the kernel wait, which is safer for metadata collection, but crashing processes above the limit are logged and their cores are skipped. If you set it, keep the handler fast.&lt;/p&gt;

&lt;p&gt;Retention is where the "missing core" mystery usually lives. Two independent mechanisms delete things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Time-based cleanup&lt;/strong&gt; by &lt;code&gt;systemd-tmpfiles&lt;/code&gt;. Check the shipped rule on your host:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;   &lt;span class="nb"&gt;grep &lt;/span&gt;coredump /usr/lib/tmpfiles.d/systemd.conf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Size-based cleanup&lt;/strong&gt; by &lt;code&gt;MaxUse=&lt;/code&gt; and &lt;code&gt;KeepFree=&lt;/code&gt;, which remove old cores as new ones arrive.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The journal entry and the core file have separate lifecycles. Journal rotation can remove the metadata while the file remains; tmpfiles or &lt;code&gt;MaxUse=&lt;/code&gt; can remove the file while the journal entry remains. &lt;code&gt;coredumpctl&lt;/code&gt; queries the journal, so an empty list does not prove the directory is empty:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo ls&lt;/span&gt; &lt;span class="nt"&gt;-lt&lt;/span&gt; /var/lib/systemd/coredump | &lt;span class="nb"&gt;head
&lt;/span&gt;coredumpctl list &lt;span class="nt"&gt;--since&lt;/span&gt; &lt;span class="s2"&gt;"2 days ago"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is also why &lt;code&gt;coredumpctl list&lt;/code&gt; shows &lt;code&gt;missing&lt;/code&gt; in the &lt;code&gt;COREFILE&lt;/code&gt; column: the metadata survived, the file did not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hands-on: make a core, find it, read it
&lt;/h2&gt;

&lt;p&gt;Use a disposable process. Sending &lt;code&gt;SIGSEGV&lt;/code&gt; to a shell is the example from the &lt;code&gt;coredumpctl&lt;/code&gt; manual page:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;bash &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'kill -SEGV $$'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then query the journal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;coredumpctl list &lt;span class="nt"&gt;--since&lt;/span&gt; &lt;span class="s2"&gt;"10 min ago"&lt;/span&gt;
coredumpctl info &lt;span class="nt"&gt;-1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;info&lt;/code&gt; gives PID, UID/GID, signal, command line, cgroup, unit, the storage path, size on disk, and a stack trace when the handler could generate one. If the list is empty, walk gates 1 through 5 before assuming anything was deleted.&lt;/p&gt;

&lt;p&gt;Open the core in a debugger:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;coredumpctl debug &lt;span class="nt"&gt;-1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By default this invokes &lt;code&gt;gdb&lt;/code&gt;. To use another debugger, pass &lt;code&gt;--debugger=&lt;/code&gt; or set &lt;code&gt;SYSTEMD_DEBUGGER&lt;/code&gt;. For a non-interactive register dump, the manual page's form is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;coredumpctl debug &lt;span class="nt"&gt;-1&lt;/span&gt; &lt;span class="nt"&gt;--debugger-arguments&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"-batch -ex 'info all-registers'"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Extract the core and keep it independent of the journal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;coredumpctl dump &lt;span class="nt"&gt;-1&lt;/span&gt; &lt;span class="nt"&gt;--output&lt;/span&gt; /tmp/bash.core
gdb &lt;span class="nt"&gt;-q&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;command&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; bash&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; /tmp/bash.core
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Useful frames need debug symbols. Install the debug package for the crashing binary (&lt;code&gt;-dbgsym&lt;/code&gt; on Debian/Ubuntu, &lt;code&gt;-debuginfo&lt;/code&gt; on Fedora), or point gdb at a debuginfod server if your distribution runs one.&lt;/p&gt;

&lt;p&gt;If &lt;code&gt;coredumpctl&lt;/code&gt; is unavailable or you are scripting, the same record is a normal journal entry with a fixed message ID:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;journalctl &lt;span class="nv"&gt;MESSAGE_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;fc2e22bc6ee647b6b90729ab34a250b1 &lt;span class="nt"&gt;-o&lt;/span&gt; verbose &lt;span class="nt"&gt;-n&lt;/span&gt; 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That prints &lt;code&gt;COREDUMP_PID=&lt;/code&gt;, &lt;code&gt;COREDUMP_SIGNAL=&lt;/code&gt;, &lt;code&gt;COREDUMP_RLIMIT=&lt;/code&gt;, &lt;code&gt;COREDUMP_FILENAME=&lt;/code&gt;, and the rest of the metadata directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cleanup and rollback
&lt;/h2&gt;

&lt;p&gt;Remove old files according to the tmpfiles rules, then confirm:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemd-tmpfiles &lt;span class="nt"&gt;--clean&lt;/span&gt;
&lt;span class="nb"&gt;sudo ls&lt;/span&gt; &lt;span class="nt"&gt;-lh&lt;/span&gt; /var/lib/systemd/coredump
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To delete one specific core, match the exact filename from &lt;code&gt;coredumpctl info&lt;/code&gt; rather than guessing a glob:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo rm&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; /var/lib/systemd/coredump/core.bash.1000.&amp;lt;timestamp&amp;gt;.zst
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To roll back a storage change, remove the drop-in and confirm the merged config:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo rm&lt;/span&gt; /etc/systemd/coredump.conf.d/60-local.conf
systemd-analyze cat-config systemd/coredump.conf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a unit-level filter or limit, remove the drop-in, then reload and restart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo rm&lt;/span&gt; /etc/systemd/system/myapp.service.d/30-core-filter.conf
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart myapp.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What this does not cover
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;systemd-coredump&lt;/code&gt; handles &lt;strong&gt;userspace&lt;/strong&gt; crashes: a process receives a fatal signal and the kernel produces a core. A kernel panic is a different pipeline entirely — &lt;code&gt;kdump&lt;/code&gt;/&lt;code&gt;kexec&lt;/code&gt; or &lt;code&gt;pstore&lt;/code&gt;/&lt;code&gt;ramoops&lt;/code&gt;, depending on firmware and configuration. Tuning &lt;code&gt;coredump.conf&lt;/code&gt; will not capture a panic, and a panic will not appear in &lt;code&gt;coredumpctl&lt;/code&gt;. Keep the two runbooks separate.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;When a crash leaves no core, do not start by deleting files or clearing the journal. Walk the chain:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;cat /proc/sys/kernel/core_pattern&lt;/code&gt; — is a handler installed?&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;systemctl show -p LimitCORE&lt;/code&gt; and &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/limits&lt;/code&gt; — is the limit zero?&lt;/li&gt;
&lt;li&gt;Check setuid, file capabilities, and executable permissions.&lt;/li&gt;
&lt;li&gt;Read &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/coredump_filter&lt;/code&gt; and any &lt;code&gt;CoredumpFilter=&lt;/code&gt; drop-in.&lt;/li&gt;
&lt;li&gt;Read the merged &lt;code&gt;coredump.conf&lt;/code&gt; — storage, size caps, and cleanup limits.&lt;/li&gt;
&lt;li&gt;Check &lt;code&gt;core_pipe_limit&lt;/code&gt;, tmpfiles age, and journal rotation.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A core is a memory image with a short, fragile life. The system tells you at every gate what it did; the mistake is assuming the list command is the whole story.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://man7.org/linux/man-pages/man5/core.5.html" rel="noopener noreferrer"&gt;&lt;code&gt;core(5)&lt;/code&gt;&lt;/a&gt; — core file naming, pipe handlers, &lt;code&gt;RLIMIT_CORE&lt;/code&gt;, &lt;code&gt;coredump_filter&lt;/code&gt;, &lt;code&gt;core_pipe_limit&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/coredumpctl.html" rel="noopener noreferrer"&gt;&lt;code&gt;coredumpctl(1)&lt;/code&gt;&lt;/a&gt; — &lt;code&gt;list&lt;/code&gt;, &lt;code&gt;info&lt;/code&gt;, &lt;code&gt;dump&lt;/code&gt;, &lt;code&gt;debug&lt;/code&gt;, matching and options&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-coredump.html" rel="noopener noreferrer"&gt;&lt;code&gt;systemd-coredump(8)&lt;/code&gt;&lt;/a&gt; — handler units, journal fields, storage behaviour&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/coredump.conf.html" rel="noopener noreferrer"&gt;&lt;code&gt;coredump.conf(5)&lt;/code&gt;&lt;/a&gt; — &lt;code&gt;Storage=&lt;/code&gt;, size limits, &lt;code&gt;MaxUse=&lt;/code&gt;, &lt;code&gt;KeepFree=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html" rel="noopener noreferrer"&gt;&lt;code&gt;systemd.exec(5)&lt;/code&gt;&lt;/a&gt; — &lt;code&gt;CoredumpFilter=&lt;/code&gt; and &lt;code&gt;LimitCORE=&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://systemd.io/COREDUMP/" rel="noopener noreferrer"&gt;systemd Coredump Handling&lt;/a&gt; — PID 1 limit setup and handler semantics&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/journalctl.html" rel="noopener noreferrer"&gt;&lt;code&gt;journalctl(1)&lt;/code&gt;&lt;/a&gt; — message-ID filtering and field output&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>linux</category>
      <category>debugging</category>
      <category>devops</category>
      <category>systemd</category>
    </item>
    <item>
      <title>Stop Letting a Backup Job Starve Your Database: Practical cgroup v2 I/O Control on Linux</title>
      <dc:creator>Lyra</dc:creator>
      <pubDate>Thu, 05 Nov 2026 05:02:16 +0000</pubDate>
      <link>https://dev.to/lyraalishaikh/stop-letting-a-backup-job-starve-your-database-practical-cgroup-v2-io-control-on-linux-4c32</link>
      <guid>https://dev.to/lyraalishaikh/stop-letting-a-backup-job-starve-your-database-practical-cgroup-v2-io-control-on-linux-4c32</guid>
      <description>&lt;p&gt;The backup window opens at 01:00. By 01:05 the database's p99 query latency has gone from 4 ms to 90 ms, and the on-call alert fires. The backup is not corrupt, not slow, and not even using much CPU. It is simply writing 400 MB/s to the same NVMe device the database lives on, and the database is now waiting in the block layer.&lt;/p&gt;

&lt;p&gt;The usual first response is &lt;code&gt;ionice -c3&lt;/code&gt;. It doesn't work, and it hasn't worked for years on most systems. The second response is a cgroup v2 &lt;code&gt;io.max&lt;/code&gt; limit, which does work, but only if you understand what it does and does not throttle.&lt;/p&gt;

&lt;p&gt;This is a practical tour of the &lt;code&gt;io&lt;/code&gt; controller: what the three knobs actually do, how to wire them to systemd, how to prove the limit is being enforced, and the traps that make people conclude "cgroups don't work" when they do.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why ionice stopped helping
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;ionice&lt;/code&gt; sets an I/O priority on a process with &lt;code&gt;ioprio_set(2)&lt;/code&gt;. Whether that priority means anything depends entirely on the block scheduler:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;BFQ&lt;/strong&gt; acts on I/O priorities and also has cgroup weight support.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;mq-deadline&lt;/strong&gt; and &lt;strong&gt;none&lt;/strong&gt; do not schedule on ioprio at all. On NVMe, &lt;code&gt;none&lt;/code&gt; is the common default.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;CFQ, the scheduler that made &lt;code&gt;ionice&lt;/code&gt; famous, was removed in Linux 5.0. If your disk is running &lt;code&gt;mq-deadline&lt;/code&gt; or &lt;code&gt;none&lt;/code&gt;, passing &lt;code&gt;-c3&lt;/code&gt; records a value and nothing reads it.&lt;/p&gt;

&lt;p&gt;Check what you actually have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;lsblk &lt;span class="nt"&gt;-o&lt;/span&gt; NAME,MAJ:MIN,SIZE,ROTA,SCHED
&lt;span class="c"&gt;# or per device&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/block/nvme0n1/queue/scheduler
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Even where BFQ is in use, ioprio is &lt;em&gt;proportional&lt;/em&gt;, not a ceiling. A backup at idle priority will still consume every spare IOPS the database isn't using, and on a fast device "spare IOPS" is enough to ruin tail latency.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the io controller actually is
&lt;/h2&gt;

&lt;p&gt;The cgroup v2 &lt;code&gt;io&lt;/code&gt; controller is not one mechanism. It is several, with different guarantees:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Mechanism&lt;/th&gt;
&lt;th&gt;Guarantee&lt;/th&gt;
&lt;th&gt;Costs&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;io.max&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;blk-throttle&lt;/td&gt;
&lt;td&gt;Hard ceiling on bytes/s or IOPS/s per device&lt;/td&gt;
&lt;td&gt;Not work-conserving; unused capacity is wasted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;io.weight&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;iocost or BFQ&lt;/td&gt;
&lt;td&gt;Proportional share of I/O time among siblings&lt;/td&gt;
&lt;td&gt;Only active if iocost is enabled or BFQ is the scheduler&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;io.latency&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;blk-iolatency&lt;/td&gt;
&lt;td&gt;Protects a group's latency target by throttling peers&lt;/td&gt;
&lt;td&gt;Only compares siblings; observes, then reacts&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two design details explain most surprises:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;io.max&lt;/code&gt; is enforced on block-layer requests&lt;/strong&gt;, not on &lt;code&gt;write(2)&lt;/code&gt; calls. Buffered writes go into page cache first and become device I/O later, during writeback. A process can dirty hundreds of megabytes before the throttle ever sees them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The kernel documentation explicitly permits bursts&lt;/strong&gt;: "BPS and IOPS are measured in each IO direction and IOs are delayed if limit is reached. Temporary bursts are allowed."&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So &lt;code&gt;io.max&lt;/code&gt; is a ceiling on sustained device traffic, not a per-syscall speed limiter. That is fine for a backup job, as long as you measure the right thing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 0: confirm the pieces exist
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# unified hierarchy?&lt;/span&gt;
&lt;span class="nb"&gt;stat&lt;/span&gt; &lt;span class="nt"&gt;-fc&lt;/span&gt; %T /sys/fs/cgroup
&lt;span class="c"&gt;# expected: cgroup2fs&lt;/span&gt;

&lt;span class="c"&gt;# is the io controller compiled in and delegated?&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/cgroup.controllers
&lt;span class="c"&gt;# expected to include: cpuset cpu io memory pids&lt;/span&gt;

&lt;span class="c"&gt;# device numbers (the io files are keyed by MAJ:MIN)&lt;/span&gt;
lsblk &lt;span class="nt"&gt;-o&lt;/span&gt; NAME,MAJ:MIN,SIZE,ROTA
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;io&lt;/code&gt; is missing from &lt;code&gt;cgroup.controllers&lt;/code&gt;, the kernel was built without &lt;code&gt;CONFIG_BLK_CGROUP&lt;/code&gt; and none of this applies. If it is present but not in &lt;code&gt;cgroup.subtree_control&lt;/code&gt;, the controller is not delegated to children yet.&lt;/p&gt;

&lt;p&gt;Also check the device, not the partition. &lt;code&gt;systemd.resource-control(5)&lt;/code&gt; is explicit that these settings should reference "the originating, physical device", not a partition or loop device.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: give the backup its own slice
&lt;/h2&gt;

&lt;p&gt;Resource control needs a boundary. Putting the limit on the backup service works, but a slice is better: everything the job spawns (restic, rclone, a shell pipeline) inherits the same budget, and you can raise the limit later without editing the service.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;/etc/systemd/system/backup.slice&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Unit]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Description&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Background maintenance workloads&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nn"&gt;[Slice]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;IOAccounting&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;CPUWeight&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;/etc/systemd/system/restic-backup.service&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Unit]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Description&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Nightly restic backup&lt;/span&gt;
&lt;span class="py"&gt;After&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;network-online.target&lt;/span&gt;
&lt;span class="py"&gt;Wants&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;network-online.target&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Type&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;oneshot&lt;/span&gt;
&lt;span class="py"&gt;User&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;backup&lt;/span&gt;
&lt;span class="py"&gt;ExecStart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/bin/backup.sh&lt;/span&gt;
&lt;span class="py"&gt;Slice&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;backup.slice&lt;/span&gt;
&lt;span class="py"&gt;IOSchedulingClass&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;best-effort&lt;/span&gt;
&lt;span class="py"&gt;IOSchedulingPriority&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;7&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;IOScheduling*&lt;/code&gt; lines are advisory on most modern schedulers. They cost nothing and help on BFQ systems, but do not rely on them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: put a hard ceiling on it
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;/etc/systemd/system/backup.slice.d/io.conf&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Slice]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;IOAccounting&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;IOReadBandwidthMax&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/dev/nvme0n1 200M&lt;/span&gt;
&lt;span class="py"&gt;IOWriteBandwidthMax&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/dev/nvme0n1 100M&lt;/span&gt;
&lt;span class="py"&gt;IOWriteIOPSMax&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/dev/nvme0n1 2K&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three things worth knowing before you copy that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Suffixes are base 1000.&lt;/strong&gt; &lt;code&gt;100M&lt;/code&gt; is 100,000,000 bytes per second, not 104,857,600. The man page says so directly. If you were thinking in MiB, your limit is about 4.6% lower than you intended.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bandwidth and IOPS both apply.&lt;/strong&gt; A 100 MB/s write cap with a 2,000 IOPS cap means the IOPS cap bites first for small random writes and the bandwidth cap bites first for large sequential ones. That is usually what you want for a backup: bound the streaming rate, and stop it from shredding the device with metadata storms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The file path is resolved to a device.&lt;/strong&gt; Pointing at &lt;code&gt;/srv&lt;/code&gt; works for simple layouts. It does not work through RAID or most volume managers, so name the physical device explicitly on anything more complex than LUKS on a partition.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Apply and inspect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart restic-backup.service

&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/backup.slice/io.max
&lt;span class="c"&gt;# 259:0 rbps=200000000 wbps=100000000 riops=max wiops=2000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That line is the ground truth. If it is missing, the controller is not active for the slice, and no amount of restarting the service will help.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: prove the limit is real
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;io.stat&lt;/code&gt; is a live per-device counter for the cgroup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/backup.slice/io.stat
&lt;span class="c"&gt;# 259:0 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read it before and after a run. &lt;code&gt;wbytes&lt;/code&gt; divided by wall-clock seconds should land near your cap, not far above it. On a fast NVMe without a limit, a restic run will show write throughput in the hundreds of MB/s; with the cap it should flatten out.&lt;/p&gt;

&lt;p&gt;For a controlled test, use &lt;code&gt;fio&lt;/code&gt; with direct I/O inside the same slice:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemd-run &lt;span class="nt"&gt;--unit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;fiotest &lt;span class="nt"&gt;--property&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;Slice&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;backup.slice &lt;span class="nt"&gt;--wait&lt;/span&gt; &lt;span class="nt"&gt;--pipe&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  fio &lt;span class="nt"&gt;--name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;seqwrite &lt;span class="nt"&gt;--filename&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/srv/fio-testfile &lt;span class="nt"&gt;--size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2G &lt;span class="se"&gt;\&lt;/span&gt;
      &lt;span class="nt"&gt;--rw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;write &lt;span class="nt"&gt;--bs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1M &lt;span class="nt"&gt;--direct&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 &lt;span class="nt"&gt;--iodepth&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;16 &lt;span class="se"&gt;\&lt;/span&gt;
      &lt;span class="nt"&gt;--runtime&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;30 &lt;span class="nt"&gt;--time_based&lt;/span&gt; &lt;span class="nt"&gt;--group_reporting&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expect roughly 100 MB/s. If you see 2 GB/s, the limit is not attached: check &lt;code&gt;io.max&lt;/code&gt; again, and confirm the unit really is in &lt;code&gt;backup.slice&lt;/code&gt; with &lt;code&gt;systemctl status fiotest&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A one-liner also works for a quick sanity check outside systemd:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemd-run &lt;span class="nt"&gt;--scope&lt;/span&gt; &lt;span class="nt"&gt;--property&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;IOReadBandwidthMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/dev/nvme0n1 20M"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nb"&gt;dd &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/dev/nvme0n1 &lt;span class="nv"&gt;of&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/dev/null &lt;span class="nv"&gt;bs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1M &lt;span class="nv"&gt;count&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;200 &lt;span class="nv"&gt;iflag&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;direct
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Buffered writes lie to you.&lt;/strong&gt; &lt;code&gt;dd&lt;/code&gt; without &lt;code&gt;oflag=direct&lt;/code&gt; or &lt;code&gt;conv=fsync&lt;/code&gt; reports the speed of copying into page cache, which is memory speed, not disk speed. It will happily report 3 GB/s while the cgroup is capped at 100 MB/s, because the bytes have not reached the device yet. Always test with &lt;code&gt;direct=1&lt;/code&gt; or force writeback.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cgroup writeback needs filesystem support.&lt;/strong&gt; The kernel attributes writeback to a cgroup only on ext2, ext4, btrfs, f2fs, and xfs. On other filesystems, writeback I/O is charged to the root cgroup, so a limit on your backup slice will not catch data the page cache flushes later. Check &lt;code&gt;findmnt -no FSTYPE /srv&lt;/code&gt; before trusting the numbers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;io.weight&lt;/code&gt; is inert more often than people think.&lt;/strong&gt; It defaults to 100 and ranges from 1 to 10,000, but proportional control is provided by iocost or BFQ. On a stock system running &lt;code&gt;none&lt;/code&gt; or &lt;code&gt;mq-deadline&lt;/code&gt;, weights can be set and read back while changing nothing. Confirm the mechanism is live:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/io.cost.qos
&lt;span class="c"&gt;# empty until a device line has been written; enable=1 turns iocost on&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;&lt;code&gt;io.latency&lt;/code&gt; only compares siblings.&lt;/strong&gt; It is work-conserving: when everyone meets their target, nothing is throttled. When one group misses, its peers with higher targets get queue-depth throttling and artificial delays. Set it on the groups that compete at the same level, not on a single child three levels down.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;io.latency&lt;/code&gt; is blind without debug stats.&lt;/strong&gt; The &lt;code&gt;missed&lt;/code&gt; and &lt;code&gt;total&lt;/code&gt; fields only appear when &lt;code&gt;blkcg_debug_stats&lt;/code&gt; is enabled, and they are off by default. Without them, you are tuning a controller you cannot observe.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Network filesystems are out of scope.&lt;/strong&gt; There is no block device to throttle. &lt;code&gt;io.max&lt;/code&gt; on an NFS-backed backup directory does nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;%util&lt;/code&gt; on NVMe is not a saturation signal.&lt;/strong&gt; It pins at 100% while the device still has headroom. Watch &lt;code&gt;await&lt;/code&gt; and &lt;code&gt;r_await&lt;/code&gt; from &lt;code&gt;iostat -x 1&lt;/code&gt;, or &lt;code&gt;io.pressure&lt;/code&gt;, which reports actual stall time.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/backup.slice/io.pressure
&lt;span class="c"&gt;# some avg10=0.00 avg60=1.42 avg300=0.31 total=...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The other tool: protect the database instead
&lt;/h2&gt;

&lt;p&gt;A hard cap is the right answer when you know the budget. When you don't, invert the problem: tell the kernel what latency the database needs, and let it throttle whoever gets in the way.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;/etc/systemd/system/db.slice.d/io.conf&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Slice]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;IOAccounting&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;IODeviceLatencyTargetSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/dev/nvme0n1 25ms&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And on the competing slice, a looser target:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Slice]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;IOAccounting&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;IODeviceLatencyTargetSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/dev/nvme0n1 100ms&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now when &lt;code&gt;db.slice&lt;/code&gt; misses 25 ms, &lt;code&gt;backup.slice&lt;/code&gt; is throttled until it recovers. When the database is idle, the backup runs at full speed. No static budget, no wasted capacity.&lt;/p&gt;

&lt;p&gt;Two caveats: this needs &lt;code&gt;CONFIG_BLK_CGROUP_IOLATENCY&lt;/code&gt;, and the target must be achievable. Setting 1 ms on a device that cannot deliver it means permanent throttling of everything else and no benefit to anyone. Measure first, then set the target slightly above the latency you see under normal load.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which one should you use?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Predictable ceiling, guaranteed isolation:&lt;/strong&gt; &lt;code&gt;io.max&lt;/code&gt;. Works regardless of scheduler, does not need iocost, and is easy to explain in a postmortem.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fair sharing with no hard cap:&lt;/strong&gt; &lt;code&gt;io.weight&lt;/code&gt;, after confirming iocost is enabled or BFQ is in use.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Protecting a latency-sensitive workload from everything else:&lt;/strong&gt; &lt;code&gt;io.latency&lt;/code&gt;, as a complement to a cap, not a replacement.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most hosts want &lt;code&gt;io.max&lt;/code&gt; on the known offenders (backups, container image pulls, log shipping) and &lt;code&gt;io.latency&lt;/code&gt; on the workload that must stay responsive. Weights are the nicest model and the easiest to misconfigure.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rollback
&lt;/h2&gt;

&lt;p&gt;Remove the drop-in and reload:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo rm&lt;/span&gt; /etc/systemd/system/backup.slice.d/io.conf
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;cat&lt;/span&gt; /sys/fs/cgroup/backup.slice/io.max
&lt;span class="c"&gt;# 259:0 rbps=max wbps=max riops=max wiops=max&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;max&lt;/code&gt; in every field means the limit is gone. That is the check to run after every change, in both directions: the file is the configuration, and it never lies about what the kernel is enforcing.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Linux kernel documentation, &lt;a href="https://docs.kernel.org/admin-guide/cgroup-v2.html" rel="noopener noreferrer"&gt;Control Group v2&lt;/a&gt;, sections "IO" and "Writeback"&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html" rel="noopener noreferrer"&gt;systemd.resource-control(5)&lt;/a&gt;, "IO Accounting and Control"&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.slice.html" rel="noopener noreferrer"&gt;systemd.slice(5)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://man7.org/linux/man-pages/man2/ioprio_set.2.html" rel="noopener noreferrer"&gt;ioprio_set(2)&lt;/a&gt; and &lt;a href="https://man7.org/linux/man-pages/man1/ionice.1.html" rel="noopener noreferrer"&gt;ionice(1)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Linux kernel documentation, &lt;a href="https://docs.kernel.org/accounting/psi.html" rel="noopener noreferrer"&gt;Pressure Stall Information&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Linux kernel documentation, &lt;a href="https://docs.kernel.org/block/bfq-iosched.html" rel="noopener noreferrer"&gt;BFQ I/O scheduler&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If a backup job is degrading your database, the fix is a budget, not a priority. Give the job a slice, give the slice a ceiling, and verify the ceiling in &lt;code&gt;io.max&lt;/code&gt; — the rest is tuning.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>performance</category>
      <category>devops</category>
      <category>storage</category>
    </item>
  </channel>
</rss>
