Mustafa ERBAY

Posted on Oct 19 Originally published at mustafaerbay.com.tr

I Put Five Processes Under One Limit, the Kernel Killed One

#linux #kernel #cgroups #docker

When you put a memory limit on a container you are writing exactly one sentence: "this workload may not use more than this many bytes". What the sentence leaves unsaid is who dies when the limit is hit. Just one process, or all of them? Does the supervisor survive and bury its children, or does the whole container shut down and come back?

That answer is yours to choose, and the place you write the choice is not --memory: it is a cgroup v2 file called memory.oom.group with a single character inside it. It has been there since Linux 4.19. Docker leaves it at zero, Kubernetes sets it to one. So when you run the same image with the same limit in two places, two different things happen once memory runs out.

So I sat down and ran the measurement. Five processes in a single cgroup, a 64 MiB limit, and one of them sent off to eat memory. With the flag at zero the kernel killed only the glutton; the other four carried on as if nothing had happened. With the flag at one, all five went together. Along the way the memory.events counter turned out to report one more death than actually occurred — finding out why in the kernel source was the most enjoyable part of the run.

What follows is all of it: the two exemptions, the zombie Docker leaves behind, the three routes I tried toward a persistent setting (two of them did nothing), and the state of the 65 containers on my own server.

The setup

The measurements ran inside the linuxkit virtual machine of Docker Desktop on my own Mac: kernel 6.10.14-linuxkit, aarch64, cgroup v2 (cgroup2fs), Docker 27.4.0. The container image is alpine:3.20. I created the cgroups by hand under /sys/fs/cgroup from inside a privileged container and moved the processes in by hand, so that what Docker does and what the kernel does stay visible separately.

The version confession comes up front: 6.10 is not today's kernel. On the day I wrote this, kernel.org's stable and mainline release were both 7.3, with long-term branches at 6.18.57 and 6.12.113. My VM lags behind. So I separated two kinds of statement: everything measured here was measured on 6.10.14; every sentence about how the mechanism works was read out of the kernel source at the relevant version tag, and I say where I read it. The server that hosts this blog is a second data point: it runs 6.8.0-147-generic, also cgroup v2.

The memory glutton is tail /dev/zero — BusyBox's tail waits for a line, /dev/zero never delivers a newline, so its buffer grows and grows. The supervisor and workers are plain sleep processes; their only job is to die or not to die.

Let me also record that my first attempt was wrong from top to bottom, because a rig error is part of the measurement. To push processes into the cgroup I reached for the ( echo $$ > cgroup.procs; exec sleep 300 ) & pattern. In a POSIX shell, $$ gives the parent shell's PID even inside a subshell, so without noticing it I had moved my own shell into the limited cgroup. In the second scenario the group kill took the script itself first, the container exited with 137, and half the output was never written. The fix is simple: write 0 to cgroup.procs and the kernel reads it as "the writing process". Every later run used echo 0 > cgroup.procs, and I checked membership one process at a time through /proc/<pid>/cgroup.

Who dies when the limit fills up

The rig is the same in every scenario: a fresh cgroup, memory.max at 64 MiB, memory.swap.max at zero, one supervisor, three workers and a glutton started last. The only thing that changes is memory.oom.group.

Scenario	`memory.oom.group`	Died	Survived
A	0	the glutton only	supervisor + 3 workers
B	1	all five	—
C	1 (+ one worker at `oom_score_adj=-1000`)	glutton + supervisor + 2 workers	the protected worker

The counters move with the scenario:

Scenario	`oom`	`oom_kill`	`oom_group_kill`	Actual deaths
A	1	1	0	1
B	1	6	1	5
C	1	5	1	4

Scenario A is where people who trust memory limits get it wrong most often. The limit filled, the kernel picked a victim, killed it, and considered the matter closed. The remaining four processes do not even know the limit was hit. memory.current dropped from a 64 MiB peak to 0.38 MiB, so the pressure genuinely went away. The only thing saying something happened is two lines of counter that nobody reads.

In scenario B the kernel narrates clearly. In dmesg, right after the victim's line, comes this:

Tasks in /docker/<id>/lab_g1 are going to be killed due to memory.oom.group set

After the announcement come five more "Killed process" lines: the supervisor, the three workers, and the victim printed a second time. Supervisor included, nobody is exempt — why that second line is there is the next section.

Why the counter reports one more than the dead

Five processes died in B but oom_kill says 6. Four died in C and it says 5. If you cannot explain a measurement you have noise rather than a finding, so I went to the source. oom_kill_process() in mm/oom_kill.c works like this (line for line identical in 6.10 and in today's mainline):

oom_group = mem_cgroup_get_oom_group(victim, oc->memcg);

__oom_kill_process(victim, message);

if (oom_group) {
    memcg_memory_event(oom_group, MEMCG_OOM_GROUP_KILL);
    mem_cgroup_print_oom_group(oom_group);
    mem_cgroup_scan_tasks(oom_group, oom_kill_memcg_member, (void *)message);
    mem_cgroup_put(oom_group);
}

The chosen victim is killed on its own first. If the group flag is set, every task of the cgroup is then scanned and put through the same function — and the victim is still one of that cgroup's tasks. Inside __oom_kill_process() the counter bump comes before the signal:

count_vm_event(OOM_KILL);
memcg_memory_event_mm(mm, MEMCG_OOM_KILL);

So the victim is written to the counter twice: once as the selected victim, once as a member of the group. dmesg does not even hide it; in scenario B's records the glutton's "Killed process 9981 (tail)" line is printed twice, with four sleep lines in between.

The practical consequence: you cannot read oom_kill in memory.events as "how many of my processes died". In events involving a group kill it can read one too high. The right order of questions is this: did oom_group_kill increase — if so the kill was collective and oom_kill is inflated by one; if not, it happened one at a time and the number is accurate. The gap came out at exactly +1 in all three scenarios, and in the hierarchy runs too (four dead, five counted). Still, let me not write it down as a law: __oom_kill_process() starts with find_lock_task_mm() and returns without touching the counter if the victim has already dropped its memory by then. So +1 is an upper bound; I saw it in every run of mine, with no guarantee you will see it in every run of yours.

The two processes spared by a group kill

A group kill does not mean "everyone dies". The function that scans the members looks for two conditions at once:

static int oom_kill_memcg_member(struct task_struct *task, void *message)
{
    if (task->signal->oom_score_adj != OOM_SCORE_ADJ_MIN &&
        !is_global_init(task)) {
        get_task_struct(task);
        __oom_kill_process(task, message);
    }
    return 0;
}

The first is the documented, familiar exemption: a process whose oom_score_adj is -1000 is never killed. That is what scenario C measures; one worker got -1000 and it was the only process standing after the group kill. The kernel documentation says the same thing: tasks with OOM protection are treated as an exception and are never killed.

The second is sneakier, because its name misleads. is_global_init() sounds like "the init process" and looks as though it covers PID 1 inside a container as well. It does not:

static inline int is_global_init(struct task_struct *tsk)
{
    return task_tgid_nr(tsk) == 1;
}

task_tgid_nr() gives the process id in the initial PID namespace. A container's PID 1 is 1 in its own namespace and some other number on the host side. The exemption therefore belongs only to the machine's real init. That is measured too: the PID 1 of a container that saw memory.oom.group at one died, the container moved to exited, the exit code was 137, and docker inspect reported OOMKilled=true. A group kill takes your container down; it does not take the system's init down.

Docker's default is zero, and a zombie stays behind

Next question: who sets this flag on my behalf? Docker's answer is "nobody". The measurement is one line:

docker run --rm --memory 64m alpine:3.20 \
  sh -c 'cat /sys/fs/cgroup/memory.oom.group'
# 0

The same command showed something else: with only --memory 64m, memory.swap.max also becomes 64 MiB, so your workload can reach twice the limit in pages. To keep the comparisons clean I passed --memory-swap 64m everywhere, which brings swap.max down to zero.

Here is the real-world shape of the default behaviour. The container holds a supervisor and a glutton; the glutton fills the limit and is killed:

Measurement	Value
`docker inspect` status	`running`
Exit code	0
`OOMKilled`	true
`memory.events` → `oom_kill`	1
Restart	none — the container never exits, so no policy fires
The glutton in `ps` output	still listed, state `Z`

The container is running, nothing restarts, the load balancer counts it healthy — and the process doing the actual work is dead. The OOMKilled flag is correct; it is just that nobody looks at it. It is not a docker ps column, it sits deep in the docker inspect output.

The zombie part is funny in its own right. The dead glutton keeps showing up in ps because its parent (sleep) never reaps it. "Then let us add --init, tini reaps zombies," I thought, and added it: the zombie stayed. tini only reaps processes that have been orphaned, and this glutton's parent is alive and negligent. If your container holds a process tree, ps can show you the name of a dead process as if it were alive.

Three routes to making it stick: two did nothing

Docker's command line has no option that turns this flag on. The official docker run reference carries two OOM-related flags: --oom-kill-disable and --oom-score-adj. Neither is about group kills, and the first one is stillborn on cgroup v2: when a container started with --memory 64m --oom-kill-disable, Docker printed "Your kernel does not support OomKillDisable. OomKillDisable discarded." and threw the flag away. That leaves three routes, and all three got a run.

Route one — flip the file by hand. With the container up, I wrote 1 into memory.oom.group from the host side. It worked: when the glutton filled the limit the whole container went, the exit code was 137, and this time the restart policy did fire (RestartCount 1). But a container running with --restart=on-failure:2 lost the setting when it came back. Docker builds the cgroup from scratch for the new life, and memory.oom.group is zero again. So it dies collectively on the first crash and reverts to the old, quiet behaviour in its second life. Flipping by hand is a measurement tool, not a production setting.

Route two — put the flag on the parent cgroup. "Let me write the flag into a durable parent and have Docker create the container underneath it" sounds entirely reasonable. I created a cgroup at /sys/fs/cgroup/oomtest, set memory.oom.group to 1, and started the container with --cgroup-parent=/oomtest --memory 64m. The result: nothing happened. The glutton died, the container kept running, and there is not a single group line in dmesg.

The reason is in the first lines of the function above. The kernel walks from the victim's cgroup up to the cgroup whose limit was hit and stops there:

    for (; memcg; memcg = parent_mem_cgroup(memcg)) {
        if (READ_ONCE(memcg->oom_group))
            oom_group = memcg;

        if (memcg == oom_domain)
            break;
    }

Because --memory puts the limit on the container's own cgroup, the walk ends on its first step and the parent's flag is never read. The kernel documentation already says this in one sentence: the OOM killer is not going to kill tasks outside of this cgroup, regardless of the memory.oom.group values of ancestor cgroups. I apparently preferred to try it and be refuted rather than read the sentence; the measurement confirmed the document.

Route three — put the limit on the parent as well. This one works. With both memory.max and memory.oom.group=1 on the parent cgroup, and the container created underneath it without a limit of its own, the limit fills at the parent, the OOM scope becomes the parent, the flag is read and the kill happens collectively. The container exited with 137 on both runs; since the parent cgroup is not something Docker recreates, the setting also stayed in place across a restart.

There is a price, and it is not small: every container under that parent now shares the same limit and the same fate. That part got its own hierarchy run.

One more file belongs next to these three routes: cgroup.kill. Per its documentation, writing 1 to it causes the cgroup and all descendant cgroups to be killed with SIGKILL. The difference is intent: memory.oom.group is a policy applied to the kernel's own OOM decision, cgroup.kill is a trigger you pull yourself. It got no run here, but if "kill the group together" is what you are after, that may be the first place to look.

The kill scope follows where the flag stands

The next rig has two layers: the limit on the parent (64 MiB), with two sibling cgroups beneath it, a and b. The glutton always went into a.

Arrangement	Worker inside `a`	Two workers inside `b`	`oom_group_kill`
Flag=1 on parent, 0 on children	died	both died	1 on parent
0 on parent, flag=1 on `a`	died	both survived	1 on `a`

The first row is route three's invoice. Put the flag on the parent and processes in a neighbouring cgroup that consumed no memory at all die too — because the kill scope is the highest cgroup carrying the flag, not the child the glutton happens to sit in. The second row is the good news: with the flag on the right layer the scope really does narrow.

The rule out of this fits in one sentence: the scope of the kill is the highest cgroup carrying the flag on the path from the victim up to the cgroup whose limit filled. A flag outside that path — that is, above the cgroup whose limit filled — is never read. Route two failed not because the flag and the limit sat apart, but because the flag sat on a layer the walk never visits.

Two runtimes, two opposite defaults

The tools sitting on top of the same mechanism made different choices. Kubernetes' kubelet configuration has a field called singleProcessOOMKill, and its documentation says this: if true, it prevents the memory.oom.group flag from being set for container cgroups in cgroups v2, causing processes in the container to be OOM killed individually instead of as a group, which aligns the behaviour with cgroups v1. On cgroup v2 its default value is false. Read plainly: Kubernetes sets this flag to 1 for you unless you ask otherwise. The project's own cgroup v2 post says the same thing and draws a boundary around it: the kubelet sets the flag for each container cgroup, and the behaviour is scoped to a container, not to the whole Pod. Docker leaves it at 0 and offers no way to turn it on.

systemd is more outspoken. OOMPolicy= takes three values; with kill, in the man page's words, the kernel is instructed to kill all remaining processes of the unit too, and this is done by setting the memory.oom.group attribute to 1. With stop, the service manager terminates the processes cleanly itself. The detail the man page adds matters most in production: with both stop and kill the unit ends up in the oom-kill failed state, after which Restart= may apply. The default comes from DefaultOOMPolicy=, and in systemd's source the manager's default is set to OOM_STOP. So a plain service gives you neither "one process dies and the rest carry on" nor a group kill, but "systemd stops the unit" — different outcomes from both.

Those are the two runtimes measured here. containerd did not get a direct run and Podman got none at all, so I am saying nothing about either.

One logging detail as well: the oom_group_kill field in memory.events has existed since 5.17 (it is absent from 5.16's source), while the memory.oom.group file itself arrived in 4.19 (absent in 4.18). On an older long-term kernel the flag works, but you have no separate counter to tally group kills.

The state of my own server

I took the same question to the server that hosts this blog and my other work. The table that came back: 65 container cgroups, 31 of them carrying a memory.max value, the rest unlimited. The number of containers with memory.oom.group at 1 is zero. All 138 units under system.slice are at zero too. Not one container carries a nonzero oom_kill counter. Reading that as "it never happened" would be wrong: as route one showed, a container's cgroup is built from scratch on every restart, and the counters reset with it. What I hold is the ledger of the current lifetimes. In the older post about the host side of the OOM killer on the same machine the story ended with the victim being chosen; the question here is one step later.

One line on the unit side misled me: systemctl show docker.service reports OOMPolicy=continue, while nginx keeps the default stop. My first note read "so that is Docker's choice". Then I opened the unit file: docker.service has no OOMPolicy line, it has Delegate=yes. The value is not Docker's preference but systemd's rule — on units with Delegate= turned on the default falls back to continue. Either way that policy concerns the daemon itself, not the insides of the containers.

My own takeaway: in single-process containers (most web services) this setting means nothing, because the dying process is PID 1 and the container goes down anyway. It matters in containers that host a process tree: the supervisor-plus-worker pattern, cron-like runners, a few services under supervisord, compilers forked under npm start. There the default behaviour produces a half-dead container that looks healthy.

Checklist

If your container holds processes other than PID 1, measure once who the memory limit will kill: a 64 MiB limit and a glutton are enough.
Put the oom_kill counter from memory.events under monitoring. It is a counter that can rise while the container is "running"; a health check can read it, because a container can read its own memory.events. If you read it from the host, remember the file is hierarchical: reading the parent also sums the events of child cgroups, and memory.events.local is the one holding that cgroup's own events.
Do not mistake oom_kill for the number of dead processes; in a group kill it reads one too high. Look at oom_group_kill first.
If you genuinely want group kills, the only durable route in Docker is putting both the limit and the flag on a shared parent cgroup. The invoice: every container under that parent dies together.
With a systemd unit the job is easy: OOMPolicy=kill sets the flag to 1 for you. Account for the default being stop, and for it falling back to continue on units with Delegate= turned on.
If there is a single process you want spared (a log shipper, a drain job), give it oom_score_adj=-1000; that is the only thing a group kill exempts. The container's PID 1 is not exempt.
--init does not solve the zombie problem; it only reaps orphans. If your process tree has a negligent intermediate parent, a dead process keeps standing in ps.

Closing

Setting a memory limit is a statement about quantity; it says nothing about which unit is the unit of failure. cgroup v1 had no such switch — in the kubelet documentation's terms, the behaviour there is killing processes individually; cgroup v2 reduced the question to a one-character file and left the decision to us. That the two big runtimes went opposite ways on that decision shows it really is a preference: Kubernetes says "a container is an indivisible unit of work, if a part dies let it all die and be reborn", Docker says "you presumably know what you are doing".

The real lesson here, to my mind, is not the flag itself. The layer that holds the limit also determines the scope of the kill — a flag written on the parent does nothing as long as the limit sits on the child. So when designing resource limits, the question "how much" needs the question "in what scope" beside it. The run where a container got 64 MiB and took 400 MiB of hugepages turned up something similar: the limit is a number, the accounting is a separate story.

What I will do in my own infrastructure is modest: not turning group kills on everywhere, but adding the oom_kill counter to the health check. Because the real problem is not that the container fails to die collectively; it is that it stays half dead without telling anyone.

DEV Community