Mustafa ERBAY

Posted on Sep 18 Originally published at mustafaerbay.com.tr

fork: retry — Four Ceilings, One Error Message

#linux #kernel #cgroup #systemd

In a twenty-second bpftrace count (tracepoint:sched:sched_process_fork, by parent name), runc came out on top: 780 clones. It is followed by containerd-shim (74), node (61) and postgres (53). The reason is not hard to guess: 33 of 60 containers define a HEALTHCHECK, and docker events showed 102 exec_create events in 30 seconds. 3.4 health checks per second; each one a runc process, in most cases a sh -c, a command, and the handful of threads the Go runtime opens. The total is in the processes line of /proc/stat: 61,719,406 fork/clone calls since boot, the machine has been up for 7.5 days, so 95.7 new tasks per second on average. With a PID range of 4,194,304, the counter has wrapped 14 times since boot (14.7 laps); one lap every 12 hours and 10 minutes. Nobody noticed, because there is nothing to notice; what spins the PID counter is health checks, and that is normal operation.

This post is about the message you see when you exceed the counter, or some other ceiling:

bash: fork: retry: Resource temporarily unavailable

Four separate mechanisms can sit under that line: kernel.pid_max, kernel.threads-max, the per-user RLIMIT_NPROC (ulimit -u) and the cgroup pids.max (TasksMax= in systemd). All four leave copy_process with the same -EAGAIN; the message does not tell you which one you hit. So this week I hit all four separately and noted which ones leave a trace and which do not. Kernel 6.8 (Ubuntu 24.04), systemd 255.

How the counter wraps

The kernel's own default is still 32768: the boot log has the line pid_max: default: 32768 minimum: 301. 6.8's pid_idr_init compares that with 1024 per CPU and takes the larger; on 18 vCPUs, 18432 is smaller, so it stays at 32768. Then systemd steps in: /usr/lib/sysctl.d/50-pid-max.conf raises the value to 4,194,304 on 64-bit systems. That is a change systemd 243 (September 2019) introduced; the rationale in the NEWS file is "to make PID collisions less likely" and to take one of the two knobs bounding the number of concurrent tasks (pid_max and threads-max) out of the picture. Ubuntu ships the value as is.

The numbers are handed out in alloc_pid in kernel/pid.c, via idr_alloc_cyclic: the next number is one more than the last one given, and at pid_max it wraps. But it does not wrap to 1, it wraps to 300: the RESERVED_PIDS constant is 300, and once the cursor has passed 300 the lower bound becomes 300. Except for the first lap, 1-299 are never handed out again; that is why the low-numbered processes in ps output are always system daemons.

You do not have to wait for the server's counter to see the wrap; the ns_last_pid sysctl is the official way to say "start the next number from here" (CRIU exists because of it). I opened a fresh pid namespace and tried it inside:

# unshare -pfm --mount-proc bash
pid1=1 last=2 pid_max=4194304
spawn 4
spawn 5
spawn 6
ns_last_pid <- 4194300
spawn 4194301
spawn 4194302
spawn 4194303
spawn 300
spawn 301
spawn 302

Three numbers later the counter came back to 300. That was the plan. Then, from inside the same namespace, I wrote 100000 to pid_max to see whether it was namespace-local; the return code was 0, but the host's pid_max had become 100000 as well. In 6.8 pid_max is a single global variable; writing it from inside a namespace changes the outside. I restored the value immediately, but in those few seconds the server's counter, at 3,034,605, found itself above the 100000 ceiling and wrapped to 300 (the last field of /proc/loadavg showed 840). No harm done, PIDs are just numbers; but a knob you assume is "only mine" inside a namespace turning the outside is a good lesson. From 6.14 on this changed: in pid.c, pid_max is now a field of struct pid_namespace, each namespace carries its own ceiling, and alloc_pid reads a separate pid_max for each level.

I looked at my other servers too: VPS2, up for 71 days, has done 28.7 laps with 120.5 million forks; VPS1 with 7.6 GiB has done 11.5 laps with 48.2 million. The counter wrapping is the normal state; the real question is who hits what while it turns.

Ceiling 1: pid_max itself

If idr_alloc_cyclic finds no room it returns -ENOSPC, and alloc_pid turns that into -EAGAIN. In practice nobody hits this ceiling: to have 4 million live tasks you would first have to exceed threads-max, and that is smaller on every machine below 512 GiB. On a distribution predating systemd 243, or on a system where the value has been pulled back to 32768, the story is different: at my server's fork rate a 32768 counter wraps every 5.7 minutes, and PID reuse (below) becomes a real problem. There is an upper end too: at 512 GiB of RAM and above, threads-max exceeds pid_max; on those machines pid_max is the smaller one.

Ceiling 2: threads-max, or RAM divided by 128 KiB

kernel.threads-max is 772144 on my server. Nobody picked that number; set_max_threads in kernel/fork.c computes it at boot:

threads = div64_u64((u64) nr_pages * (u64) PAGE_SIZE,
                    (u64) THREAD_SIZE * 8UL);

On x86-64 THREAD_SIZE is 16 KiB; the formula is page count × 4 KiB / (16 KiB × 8), i.e. RAM / 128 KiB. The server's MemTotal is 98,873,688 kB; divided by 128 that is 772,450; the kernel found 772,144; the 306 units in between × 128 KiB ≈ 38 MiB, because fork_init runs at boot before the initrd and init sections are released. The documentation says the same thing: thread structures may occupy at most one eighth of RAM. On VPS1 with 7.6 GiB the value is 61,667; on VPS2 with 11.4 GiB it is 93,070. On VPS5, which runs kernel 7.0, it is 82,263, although its MemTotal (11,951,460 kB) divided by 128 gives 93,370: since 6.12 the formula uses memblock_estimated_nr_free_pages() instead of totalram_pages(), and on that machine dividing the first number of the boot-log line "Memory: 10519552K/12287460K available" by 128 gives 82,184. Same machine, a ceiling 12 percent lower than the old formula would give.

The check in copy_process is a single line, and its comment is candid:

/* ... the check is only there to stop root fork bombs. */
retval = -EAGAIN;
if (data_race(nr_threads >= max_threads))
    goto bad_fork_cleanup_count;

You do not need to count ps to see nr_threads: the fourth field of /proc/loadavg has the form 5/4175, the right side of the slash is exactly nr_threads, and the last field is the last PID handed out (fs/proc/loadavg.c). This ceiling leaves no log either; just -EAGAIN.

It has a chain, too: fork_init sets the default for RLIMIT_NPROC to max_threads/2. On the server ulimit -u is 386072 = 772144 / 2. For services the knob is not limits.conf (pam_limits only applies to PAM sessions) but LimitNPROC= in the unit or DefaultLimitNPROC=; if none of them is set, the third ceiling is half the second.

Ceiling 3: ulimit -u, per user and not applied to root

RLIMIT_NPROC limits the total number of tasks of a user, not of a process (the kernel counts it through ucounts). In 6.8 copy_process decides like this:

if (is_rlimit_overlimit(task_ucounts(p), UCOUNT_RLIMIT_NPROC, rlimit(RLIMIT_NPROC))) {
    if (p->real_cred->user != INIT_USER &&
        !capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
        goto bad_fork_cleanup_count;
}

There are three exemptions: root (INIT_USER), CAP_SYS_RESOURCE, CAP_SYS_ADMIN. Easy to try:

# sudo -u nobody bash -c 'ulimit -u 20; while :; do sleep 30 & done'
bash: fork: retry: Resource temporarily unavailable
# bash -c 'ulimit -u 5; ...30 sleeps...; echo root-started=$n'
root-started=30

nobody stopped at 20; root, despite writing ulimit -u 5, started 30 processes. Meanwhile no line appeared in the kernel log and no cgroup counter moved. One more footnote: if a program calling setuid() exceeds the ceiling, setuid returns success and the error is deferred to execve (the PF_NPROC_EXCEEDED flag; the source comment says "too many poorly written programs don't check setuid() return code"). So in a daemon that starts as root and drops to a user, you may get the EAGAIN from exec as well.

Docker's documentation warns about this ceiling separately: --ulimit nproc=… counts the user and does not count the container; if you start four containers with the same UID, the fourth trips over the processes of the other three. If you want a per-container limit, that is the fourth ceiling.

Ceiling 4: pids.max — the only one that logs, and logs in the wrong place

cgroup v2's pids controller counts hierarchically: pids_try_charge increments the counter of every ancestor from the leaf up to the root (excluding the root, which has no pids.max), and if any of them exceeds its limit, it reverts all of them. systemd manages this through TasksMax=; DefaultTasksMax is, by default, 15% of the smallest of threads-max, pid_max−1 and the root pids.max (system_tasks_max, src/basic/limits-util.c). On the server 772144 × 0.15 = 115821; that number is in the pids.max of every service that does not set TasksMax=, and in all 59 of Docker's container scopes as well. docker.service and containerd.service carry TasksMax=infinity in their own unit files. User slices get 33% via user-.slice.d/10-defaults.conf, i.e. 254807. The shortest way to see where a unit stands is systemctl status: the Tasks: 64 (limit: 115821) line prints pids.current and pids.max side by side.

The history matters, because it changed what the ceiling means. systemd 228 came with DefaultTasksMax=512; 231 raised it to 15% of pid_max, i.e. 4915; 243 pushed pid_max to 4 million, so the percentage became tied to threads-max, and therefore to RAM. Today the number of tasks a service may start is a function of the machine's memory:

RAM	threads-max ≈ RAM/128 KiB	DefaultTasksMax (15%)	user slice (33%)	ulimit -u (½)
1 GiB	8192	1228	2703	4096
2 GiB	16384	2457	5406	8192
4 GiB	32768	4915	10813	16384
94 GiB (VPS3)	772144	115821	254807	386072

Something like Java or Elasticsearch running as a systemd service on a 1 GiB VM getting EAGAIN at 1228 threads is the consequence of this table; memory is enough, the thread count is not.

I tried this ceiling through a slice so that the way the hierarchy is reported would show: I gave pidlab.slice TasksMax=40, the transient service inside it TasksMax=infinity, and the service tried to start 100 sleeps.

/sys/fs/cgroup/pidlab.slice/pids.max:                        40
/sys/fs/cgroup/pidlab.slice/pids.current:                    40
/sys/fs/cgroup/pidlab.slice/pids.events:                     max 0
/sys/fs/cgroup/pidlab.slice/pidlab-svc.service/pids.max:     max
/sys/fs/cgroup/pidlab.slice/pidlab-svc.service/pids.events:  max 3
/sys/fs/cgroup/pidlab.slice/pidlab-svc.service/pids.peak:    41
kernel: cgroup: fork rejected by pids controller in /pidlab.slice/pidlab-svc.service

The slice that holds the limit says max 0; the service that has no limit says max 3 (read at the fourth second, inside bash's retry window; the number keeps growing until bash gives up). In 6.8 pids_can_fork writes the event to the cgroup where the fork happened, without looking at where the limit is; the kernel log prints the leaf's path as well, and only on the first rejection (the events_limit == 1 condition). The service's pids.peak being 41 is a comment in the source come to life: because pids_try_charge increments first and reverts afterwards, the rejected attempt raises the peak too ("Not technically accurate if we go over limit somewhere up the hierarchy, but that's tolerable for the watermark").

6.11 fixed the counters: pids_event now records the event starting from the cgroup whose limit was exceeded and walking upward; the leaf where the fork happened is not counted. Run the same experiment on a 6.11+ kernel and the culprit moves: max 3 on the slice, max 0 on the service. The new pids.events.local file counts only the events where that cgroup's own pids.max was exceeded (3 on the slice, again 0 on the service); the 6.8 behaviour of "count where the fork happened" only comes back with the pids_localevents mount option. The kernel log did not change: the line still prints the cgroup where the fork happened, not where the limit is. Reading pids.events without knowing which kernel you are on can end with you blaming the wrong cgroup.

The word "retry" is bash's, and so are the 15 seconds

The retry in the message does not come from the kernel. In jobs.c, if fork() returns EAGAIN, bash first tries to reap dead children, then sleeps for 1 second and tries again, doubling the delay each time:

+  0.0s bash: fork: retry: Resource temporarily unavailable
+  1.0s bash: fork: retry: Resource temporarily unavailable
+  3.0s bash: fork: retry: Resource temporarily unavailable
+  7.0s bash: fork: retry: Resource temporarily unavailable
+ 15.0s bash: fork: Resource temporarily unavailable
exit 254

Four retries (five fork calls in total), 1+2+4+8 seconds, then a final line without retry and exit code 254. So when you see fork: retry in a log, a 15-second window has passed, and if the ceiling cleared during that window the command succeeded silently. Other runtimes do not extend this courtesy; in Python the same experiment ends on the first subprocess.run with BlockingIOError: [Errno 11] Resource temporarily unavailable.

Which one was it: the diagnostic order

The order is set by the only ceiling that logs. If the kernel log has fork rejected by pids controller, the answer is cgroup, but since the line is written once per cgroup, "absent" does not mean "never happened"; read pids.events, on the leaf in 6.8 and on the ancestor that holds the limit in 6.11+. Otherwise ulimit -u is next; ps -eLo user= | grep -c '^user$' counts the user's total tasks (on my server root 2162, messagebus 752). If the process's user is root or holds CAP_SYS_RESOURCE, this ceiling is out of play. Then the fourth field of /proc/loadavg against threads-max. pid_max comes last, and almost never.

As the PIDs go round: pidfd

The counter wrapping every 12 hours has one more consequence. If you store a PID and send it a signal 12 hours later, that number may belong to another process; in the old 32768 range that time shrinks to minutes. The kernel-side answer has existed since 5.3: pidfd_open gives you a reference to a process as a file descriptor rather than a number, and the descriptor dies when the process does. I tried it in the same pid namespace, forcing reuse with ns_last_pid:

A pid 42000 pidfd 3
A is dead, waitpid done; is the pidfd readable in poll: True
B pid 42000 (same as A: True )
kill(A.pid,0) -> success: but this is B!
pidfd_send_signal(fd,0) -> ESRCH No such process
/proc/self/fdinfo Pid field: ['Pid:\t-1']

kill(42000, 0) says "the process exists", because it does, but it is another process. pidfd_send_signal over the same pidfd returns ESRCH; the Pid field in fdinfo has dropped to -1 (pidfd_show_fdinfo prints -1 when the struct pid it points at no longer has a task). Since Python 3.9, os.pidfd_open and signal.pidfd_send_signal are in the standard library; if you are writing a process manager, holding the descriptor instead of writing the PID to a file and killing it later makes the 12-hour lap none of your concern. That was exactly systemd's rationale when it pushed the range to 4 million in 243: collisions less likely, but "certainly still possible".

Questions to ask your server

Divide awk '/^processes/' /proc/stat by /proc/uptime to get your fork rate; see how many hours a lap takes. Compare cat /proc/sys/kernel/threads-max with MemTotal/128; a large gap means kernel 6.12+ and the boot-time memory estimate is in play. Put the output of systemctl show -p DefaultTasksMax next to your thread-heavy services (JVM, Elasticsearch, anything that opens a worker per nproc); on a small VM the 15% rule ends at one or two thousand tasks. Think about ulimit -u for non-root service users, and do not think about it for root, it does not apply. And even if there is no journalctl -k | grep 'pids controller' line, run find /sys/fs/cgroup -name pids.events | xargs grep -v 'max 0' once; in 6.8 the leaf talks, in 6.11+ the owner of the limit does.

What the four ceilings share is the errno; what separates them is that one leaves a log and three do not, and that one is tied to RAM, one to the user, one to the cgroup, and one to nothing but a counter. In the KSM post I said "run=1 was not enough" on the same server; here "the error message is not enough" either, the counters tell you what you hit. The path I follow on the cgroup side for memory pressure is in the systemd-oomd runbook; pids.max is its task-count counterpart.

DEV Community