DEV Community

Cover image for Stop Letting One Service Eat Every PID: Practical cgroup v2 pids Control on Linux
Lyra
Lyra

Posted on

Stop Letting One Service Eat Every PID: Practical cgroup v2 pids Control on Linux

#linux #devops #systemd #performance

At 3 a.m., every login attempt on the box returned fork: Resource temporarily unavailable. Memory was fine. CPU was flat. Disks were quiet. The host had run out of tasks: one worker had been spawning children in a retry loop until the kernel's process table filled, and then nothing else on the machine could start a process either — including the shell you needed to fix it.

We are careful about bounding bytes (memory.max), milliseconds (cpu.max), and IOPS (io.max). We almost never bound how many things exist. That is the pids controller, and systemd already wires it to a knob most people have never set: TasksMax=.

Two limits with the same name and different scopes

The reflex is ulimit -u (RLIMIT_NPROC). For services it is usually the wrong tool, and getrlimit(2) says exactly why:

This is a limit on the number of extant process (or, more precisely on Linux, threads) for the real user ID of the calling process. [...] The RLIMIT_NPROC limit is not enforced for processes that have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability, or run with real user ID 0.

That last sentence is the whole problem. A service running as root can fork without limit no matter what you put in limits.conf. Even for non-root services, the limit is per user ID, not per service: two units running as the same user share one budget, and a login shell counts against it too.

RLIMIT_NPROC cgroup pids.max
Enforced against real UID cgroup membership
Counts threads of that UID, everywhere tasks in the cgroup and its descendants
Exempt root, CAP_SYS_ADMIN, CAP_SYS_RESOURCE nothing
Scope follows the process's credentials where the process was placed
Configured via limits.conf, LimitNPROC= TasksMax=

The cgroup limit does not care who you are. It cares where you live.

The counter counts threads, not processes

ps aux | wc -l is the wrong denominator. Every thread is a task, and every task counts. A JVM with two processes and forty pool threads is forty-two tasks. This is the most common reason a limit that "looks generous" kills a Java or .NET service at startup with unable to create native thread — an error everyone blames on ulimit.

Measure the real number before you pick one:

# threads in one process
grep Threads /proc/$(pgrep -f my-service | head -1)/status

# all tasks belonging to a unit, threads included
systemctl show -p TasksCurrent -p TasksMax my-service.service
Enter fullscreen mode Exit fullscreen mode

If the second command prints TasksCurrent=[not set], you are asking the wrong manager. User units live in the user instance:

systemctl --user show -p TasksCurrent -p TasksMax my-user-service.service
Enter fullscreen mode Exit fullscreen mode

A child can lower the limit, never raise it

cgroup v2 limits are hierarchical. The effective limit of any cgroup is the tightest pids.max on the path from the root down to it. A nested cgroup can write a bigger number into its own pids.max, but that grants nothing: the ancestor's limit still applies to every task in the subtree. There is also no pids.max on the root cgroup at all — the control files exist on non-root cgroups, which is why you cannot "limit the whole machine" this way.

systemd follows the same rule. From systemd.resource-control(5): when TasksMax= is set for a unit, the pids controller is enabled for that unit and for its parents and siblings, so accounting is consistent up the tree.

What systemd already does for you

Two defaults matter:

  • DefaultTasksAccounting=yes, so task counts are always kept.
  • DefaultTasksMax= defaults to 15% of the minimum of kernel.pid_max, kernel.threads-max, and the root cgroup's pids.max. The man page's example is 4915 on a host with the traditional 32768 pid_max.

On the Debian 13 host I tested (systemd 257.13, kernel 6.12):

$ cat /proc/sys/kernel/pid_max
4194304
$ cat /proc/sys/kernel/threads-max
62898
$ systemctl show -p TasksCurrent -p TasksMax -p EffectiveTasksMax systemd-journald.service
TasksCurrent=1
TasksMax=9434
EffectiveTasksMax=9434
Enter fullscreen mode Exit fullscreen mode

9434 is 15% of 62898, truncated. So every unit on that host is already capped, whether or not its author thought about it.

logind adds a second layer: on systemd 257, each user slice is capped at 33% of the same system maximum (UserTasksMax= in logind.conf(5)), which is why user-1000.slice reports TasksMax=20756 here. That is the layer that saves you when a logged-in user runs something unfortunate: the host still has two thirds of its task table for sshd and PID 1.

One honest caveat: these are per-unit caps, not a global budget. Ten units at 15% each can still sum past 100%, so a coordinated fork storm can exhaust the table. What TasksMax= buys you is blast-radius control: the noisy unit hits its own wall and stops, and everything else keeps running.

The files that make it visible

Inside any non-root cgroup:

$ ls /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/ \
     | grep pids
pids.current
pids.events
pids.events.local
pids.max
pids.peak
Enter fullscreen mode Exit fullscreen mode
  • pids.current — tasks right now, in this cgroup and its descendants.
  • pids.max — the limit, or max for none.
  • pids.peak — the high-water mark since the cgroup was created.
  • pids.events — max N: how many forks were denied, including in descendants.
  • pids.events.local — the same counter for this cgroup only.

pids.events is the one to graph. A limit that is never hit leaves it at max 0, and that is your evidence that the number is doing something rather than just sitting in a unit file.

Lab: make fork fail on purpose

Do not paste the classic fork bomb. A bounded loop produces exactly the same EAGAIN and cannot take your session down with it.

# /tmp/pidlab.py
import os, time

cg = "/sys/fs/cgroup" + [
    line.split("::")[1].strip()
    for line in open("/proc/self/cgroup")
    if line.startswith("0::")
][0]

kids, err = [], None
for i in range(40):
    try:
        p = os.fork()
    except OSError as e:
        err = (i, e.errno, os.strerror(e.errno))
        break
    if p == 0:
        time.sleep(1.2)
        os._exit(0)
    kids.append(p)

def rd(name):
    try:
        return open(os.path.join(cg, name)).read().strip()
    except OSError as e:
        return "ERR:" + str(e)

print("cgroup        ", cg)
print("fork_failed_at", err)
print("pids.current  ", rd("pids.current"))
print("pids.max      ", rd("pids.max"))
print("pids.peak     ", rd("pids.peak"))
print("pids.events   ", rd("pids.events").replace("\n", "; "))

for p in kids:
    try:
        os.waitpid(p, 0)
    except OSError:
        pass
Enter fullscreen mode Exit fullscreen mode

Run it in a scope with a deliberately small limit:

systemd-run --user --scope -q -p TasksMax=20 --unit=pidlab -- python3 /tmp/pidlab.py
Enter fullscreen mode Exit fullscreen mode 
cgroup         /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/pidlab.scope
fork_failed_at (19, 11, 'Resource temporarily unavailable')
pids.current   20
pids.max       20
pids.peak      20
pids.events    max 1
Enter fullscreen mode Exit fullscreen mode

Read that carefully, because it contains the whole model:

  • Twenty tasks means the parent plus nineteen children. The scope's own process is inside its own limit; the twentieth fork() is the one that fails.
  • It failed at index 19 with errno 11, EAGAIN. Not ENOMEM. The kernel is not out of memory; it is enforcing a policy. Applications frequently log this as a memory problem and send you chasing the wrong subsystem for a day.
  • pids.events is max 1. Exactly one denial was recorded, and only the fork that hit the wall was affected — nothing was killed.

A shell shows the same event in its own dialect:

/usr/bin/bash: fork: retry: Resource temporarily unavailable
Enter fullscreen mode Exit fullscreen mode

bash retries the fork a few times before giving up, which is why a fork-bombing script produces a burst of these lines rather than one clean error.

The percentage trap

TasksMax= accepts a percentage, and it is not a percentage of the default:

systemd-run --user --scope -q -p TasksMax=50% -- \
  bash -c 'cg=/sys/fs/cgroup$(cut -d: -f3 /proc/self/cgroup); cat $cg/pids.max'
# 31449
Enter fullscreen mode Exit fullscreen mode

31449 is 50% of 62898, the system maximum from threads-max — not 50% of the 9434 default, and certainly not 50% of current usage. Percentages are relative to the same system-wide ceiling that DefaultTasksMax= uses. If you want a number that means the same thing on every host, write an absolute one.

Choosing a number that does not break the app

  1. Measure idle and peak. TasksCurrent at rest is your floor. Run a load test, then read pids.peak.
  2. Count the things that spawn threads, not just processes:
  • JVM: pool threads plus GC and JIT threads. -XX:ActiveProcessorCount changes some of them.
  • Go: GOMAXPROCS is a floor, not a ceiling; the runtime adds sysmon and GC workers.
  • nginx: one process per worker_processes value, plus cache manager and loader.
  • PostgreSQL: one process per connection, plus autovacuum workers, walsenders, and background workers. A TasksMax=100 on a database configured for max_connections=200 is a time bomb that goes off under load.
  • PHP-FPM: pm.max_children is the process count you must leave room for.
  1. Add headroom for shell-outs, xargs -P, and short-lived helpers.
  2. Deploy, then watch pids.events. If it climbs, the limit is too tight — not the app.

Recipes

Cap one noisy service with a drop-in:

sudo mkdir -p /etc/systemd/system/scraper.service.d
sudo tee /etc/systemd/system/scraper.service.d/20-tasks.conf >/dev/null <<'EOF'
[Service]
TasksMax=64
EOF
sudo systemctl daemon-reload
sudo systemctl restart scraper.service
Enter fullscreen mode Exit fullscreen mode

Cap a whole class of work — every CI job in a slice shares one ceiling:

sudo mkdir -p /etc/systemd/system/build.slice.d
sudo tee /etc/systemd/system/build.slice.d/50-tasks.conf >/dev/null <<'EOF'
[Slice]
TasksMax=2048
EOF
sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Cap a user without touching logind defaults:

sudo mkdir -p /etc/systemd/system/user-1000.slice.d
sudo tee /etc/systemd/system/user-1000.slice.d/50-tasks.conf >/dev/null <<'EOF'
[Slice]
TasksMax=4096
EOF
sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Try a limit without writing a unit, useful for reproducing a customer report:

systemd-run --user --scope -p TasksMax=32 -- ./run-tests.sh
Enter fullscreen mode Exit fullscreen mode

Containers get the same controller through the runtime: podman run --pids-limit N and docker run --pids-limit N write the container cgroup's pids.max. Kubernetes can cap pods at the node level with kubelet's --pod-max-pids. Without those flags the limit is max, which is how a container takes down its host's process table from the inside.

Failure modes

Symptom Real cause Check
Resource temporarily unavailable on fork pids.max reached pids.events
JVM: unable to create native thread limit below thread count pids.peak vs TasksMax
Works until traffic spikes fork-per-request or connection-per-process TasksCurrent under load
Limit "ignored" for a root service you set LimitNPROC= use TasksMax=
Numbers differ from the docs percentage vs absolute pids.max in the cgroup

Lowering a limit at runtime does not kill anything. Existing tasks keep running; new forks are refused until the count drops below the new ceiling. That makes tightening a live system safe, and it also means a too-low limit can look fine until the next restart.

Verify, then roll back 

systemctl show -p TasksMax -p EffectiveTasksMax -p TasksCurrent my-service.service
systemd-cgtop -t -b -n 1          # order control groups by task count
cat /sys/fs/cgroup/system.slice/my-service.service/pids.events
Enter fullscreen mode Exit fullscreen mode

If the limit was a mistake, remove the drop-in and revert:

sudo rm /etc/systemd/system/my-service.service.d/20-tasks.conf
sudo systemctl daemon-reload
sudo systemctl restart my-service.service
# or, to drop every local override at once:
sudo systemctl revert my-service.service
Enter fullscreen mode Exit fullscreen mode

The limit nobody graphs

A host that cannot fork is down in a way that looks nothing like being out of memory or CPU: the monitoring is green, the logs are quiet, and nothing you type works. It costs one line in a unit file to make sure the next runaway loop fails inside its own cgroup instead of taking the machine with it.

Start with your noisiest units — scrapers, build jobs, anything that shells out — and give them a number based on measurement rather than optimism. Then watch pids.events for a week. If it stays at zero, you have headroom. If it moves, you just found the service that would have been your 3 a.m. page.

References

  • Linux kernel documentation, Control Group v2 — the pids controller and its interface files: https://docs.kernel.org/admin-guide/cgroup-v2.html
  • cgroups(7) — pids controller since Linux 4.3, CONFIG_CGROUP_PIDS: https://man7.org/linux/man-pages/man7/cgroups.7.html
  • systemd.resource-control(5) — TasksAccounting=, TasksMax=, EffectiveTasksMax=
  • systemd-system.conf(5) — DefaultTasksMax= (15% of the minimum of pid_max, threads-max, and root pids.max)
  • logind.conf(5) — UserTasksMax=
  • fork(2), getrlimit(2) — EAGAIN conditions and RLIMIT_NPROC exemptions
  • systemd-run(1), systemctl(1), systemd-cgtop(1)
  • podman-run(1), docker-run(1) — --pids-limit

Tested on Debian 13, kernel 6.12.113, systemd 257.13. Your numbers will differ; the semantics will not.

Top comments (0)