DEV Community

Cover image for Your OOM Killer Is Guessing: Practical cgroup v2 Memory Control on Linux
Lyra
Lyra

Posted on

Your OOM Killer Is Guessing: Practical cgroup v2 Memory Control on Linux

At 03:12 the kernel killed your SSH session instead of the service that was eating 14 GB. Nothing was broken. Nothing was limited either. The whole machine was one big cgroup, and the OOM killer picked a victim by badness score, not by blame.

That is the default on most Linux systems: every process competes for the same pool of memory. When the pool runs dry, the kernel chooses something to kill. It might choose the leak. It might choose your database. It might choose sshd.

cgroup v2 gives you a better answer than "hope the score is right". You can put a ceiling on a service, make it slow down before it gets there, and read exactly what happened afterwards. The knobs are small and the semantics are not obvious. This article is about the semantics.

The lab below was run on Debian 13 (kernel 6.12, systemd 257). The cgroup files are the same on any system using the unified hierarchy.

First, check what you are working with

stat -fc %T /sys/fs/cgroup
# cgroup2fs
Enter fullscreen mode Exit fullscreen mode

If that prints cgroup2fs, you are on cgroup v2 and everything below applies. If it prints tmpfs, you are on the old hierarchy and the file names are different (memory.limit_in_bytes, memory.memsw.limit_in_bytes). Check before copying any of this.

Also check that the memory controller is actually available:

cat /sys/fs/cgroup/cgroup.controllers
# cpuset cpu io memory hugetlb pids rdma misc
Enter fullscreen mode Exit fullscreen mode

If memory is missing, the controller is disabled in the kernel command line or not compiled in, and none of the unit settings will do anything.

What the kernel is measuring

Every cgroup has a memory.current file. It is the total memory currently charged to that cgroup and its descendants, in bytes.

That number is not RSS. It includes:

  • anonymous memory (heap, stacks, mmap(MAP_ANONYMOUS))
  • page cache charged to the cgroup
  • tmpfs and shared memory
  • kernel memory charged to the cgroup: page tables, kernel stacks, slab, socket buffers

memory.stat breaks it down. The two lines you will use most are anon and file:

cat /sys/fs/cgroup/system.slice/myapp.service/memory.stat | grep -E '^(anon|file|kernel|slab|sock|shmem) '
Enter fullscreen mode Exit fullscreen mode

This distinction matters because file is mostly reclaimable. A service that reads 2 GB of logs can show 2 GB in memory.current while being under no real pressure: the kernel can throw those pages away and read them again later. A service whose anon grows by 200 MB per hour has a leak, and no amount of cache dropping will fix it.

The kernel documentation is explicit that file includes tmpfs and shared memory. That is worth remembering the next time a service with a small RSS hits its limit because something wrote 500 MB to /dev/shm.

The knobs that matter

There are five memory files you will actually use, plus three for observation.

File What it does
memory.min Hard protection. Memory inside this boundary is not reclaimed under any condition.
memory.low Best-effort protection. Reclaimed only if nothing unprotected is available.
memory.high Throttle. Going over puts the cgroup under heavy reclaim pressure and slows the allocating tasks. Does not invoke the OOM killer.
memory.max Hard limit. If usage cannot be reduced below it, the OOM killer runs inside the cgroup.
memory.swap.max Ceiling on swap used by the cgroup. Default is max.
memory.events Counters: low, high, max, oom, oom_kill, oom_group_kill.
memory.peak High-water mark since cgroup creation (Linux 5.19+).
memory.reclaim Write-only trigger to ask the kernel to reclaim a specified amount (Linux 5.19+).

The important pair is memory.high and memory.max, because people treat them as interchangeable and they are not.

memory.high is a speed limit. When usage goes over it, the kernel does not kill anything. It applies heavy reclaim pressure and throttles the tasks that are trying to charge more memory. The kernel documentation is blunt about the guarantee: going over the high limit never invokes the OOM killer, and under extreme conditions the limit may be breached. memory.events counts these episodes in the high field.

memory.max is a wall. When usage reaches it and reclaim cannot bring it down, the OOM killer is invoked in that cgroup. The doc again: "If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup." It also warns that usage may go over the limit temporarily, because accounting and allocation are not atomic.

That is the whole design: high buys you time and backpressure; max is the last line of defense. systemd's own documentation recommends exactly that split, using MemoryHigh= as the main control and MemoryMax= as the fallback.

Limits compose up the tree

cgroup v2 has a no-internal-process rule: a non-root cgroup can contain processes or child cgroups, not both. systemd builds slices (which contain units) and services (which contain processes) on top of that rule.

The practical consequence is that a limit is never just yours. If myapp.service sits under myapp.slice, and the slice is capped at 2 GB, then a 4 GB MemoryMax= on the service is not reachable. systemctl show exposes the result:

systemctl show myapp.service -p MemoryMax -p EffectiveMemoryMax
Enter fullscreen mode Exit fullscreen mode

EffectiveMemoryMax is the most stringent limit of the unit and its parent slices, capped by physical memory. When a limit "does not work", this is the first field to read.

A live lab: 200 MB under two different limits

Here is a real experiment. It uses a transient user scope so nothing permanent is created:

systemd-run --user --unit=memlab --scope \
  -p MemoryAccounting=yes \
  -p MemoryHigh=256M \
  -p MemoryMax=384M \
  -p MemorySwapMax=0 \
  /bin/sleep 300 &
Enter fullscreen mode Exit fullscreen mode

Inspect what systemd actually applied:

systemctl --user show memlab.scope \
  -p ControlGroup -p MemoryHigh -p MemoryMax \
  -p EffectiveMemoryMax -p MemoryCurrent
Enter fullscreen mode Exit fullscreen mode

Output from this machine:

ControlGroup=/user.slice/user-1000.slice/user@1000.service/app.slice/memlab.scope
EffectiveMemoryMax=402653184
MemoryCurrent=188416
MemoryHigh=268435456
MemoryMax=402653184
Enter fullscreen mode Exit fullscreen mode

The numbers are bytes and the suffixes are base 1024: 384M is 402653184. Now read the cgroup files directly, using the path systemd reported:

CG=$(systemctl --user show memlab.scope -p ControlGroup --value)
for f in memory.max memory.high memory.swap.max memory.current memory.peak memory.oom.group; do
  printf '%s=' "$f"
  cat "/sys/fs/cgroup$CG/$f"
done
Enter fullscreen mode Exit fullscreen mode
memory.max=402653184
memory.high=268435456
memory.swap.max=0
memory.current=188416
memory.peak=339968
memory.oom.group=0
Enter fullscreen mode Exit fullscreen mode

Nothing controversial so far: a sleeping process, three limits, and an empty event counter.

Now the interesting part. Allocate 200 MB of anonymous memory inside a scope limited to 256 MB, with swap disabled:

systemd-run --user --unit=memtime1 \
  -p MemoryMax=256M -p MemorySwapMax=0 --wait \
  /usr/bin/python3 -c 'import time; t=time.time(); b=bytearray(200*1024*1024); print(round(time.time()-t,3))'
Enter fullscreen mode Exit fullscreen mode
Service runtime: 84ms
CPU time consumed: 68ms
Memory peak: 203.2M (swap: 0B)
Enter fullscreen mode Exit fullscreen mode

84 milliseconds. The allocation fits, the limit is never approached, and nothing is throttled.

Repeat it with MemoryHigh=128M added, while keeping MemoryMax=256M:

systemd-run --user --unit=memtime2 \
  -p MemoryHigh=128M -p MemoryMax=256M -p MemorySwapMax=0 --wait \
  /usr/bin/python3 -c 'import time; t=time.time(); b=bytearray(200*1024*1024); print(round(time.time()-t,3))'
Enter fullscreen mode Exit fullscreen mode

This is where the intuition breaks. The process did not get killed. It also did not finish. After 58 seconds it was still allocating, sitting in uninterruptible sleep (D state) inside the kernel's reclaim path. The cgroup counters told the story:

memory.current=195170304
low 0
high 494
max 0
oom 0
oom_kill 0
oom_group_kill 0
Enter fullscreen mode Exit fullscreen mode

494 throttle events, zero OOM kills. The process had charged about 186 MB, well above the 128 MB high limit, and the kernel kept throttling it instead of killing it. I stopped the unit at 58 seconds; it had not printed its completion line.

Two lessons, and they are the reason this section exists:

  1. memory.high does not cap usage. It applies backpressure. Anonymous memory cannot be reclaimed when there is no swap, so the kernel has nothing to take and the limit gets breached while the task is throttled.
  2. Backpressure can be so heavy that the workload makes almost no forward progress. A 200 MB allocation that took 84 ms with only a max limit was still not done after 58 seconds with an aggressive high limit.

If your goal is "never exceed X", use MemoryMax. If your goal is "slow down before you hit X", use MemoryHigh — but set it above the real anonymous working set, not at a number you wish the service used.

Putting limits on a real service

For a unit managed by systemd, do not edit files under /usr/lib/systemd/system. Use a drop-in:

sudo systemctl edit myapp.service
Enter fullscreen mode Exit fullscreen mode

In the editor:

[Service]
MemoryAccounting=yes
MemoryHigh=768M
MemoryMax=1G
MemorySwapMax=0
Enter fullscreen mode Exit fullscreen mode

Then:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service
systemctl show myapp.service \
  -p MemoryHigh -p MemoryMax -p MemorySwapMax \
  -p EffectiveMemoryMax -p MemoryCurrent
Enter fullscreen mode Exit fullscreen mode

A few details that are easy to get wrong:

  • Sizes accept K, M, G, T with base 1024, or a percentage of installed physical memory. MemoryMax=80% is 80% of RAM, not 80% of the slice.
  • infinity removes the limit. It is the default for both MemoryHigh and MemoryMax.
  • MemorySwapMax=0 does not disable swap for the system; it stops this cgroup's anonymous pages from being swapped out. That makes the memory limit predictable, at the cost of pushing the service into reclaim/OOM instead of letting it spill to disk.
  • MemoryMax applies to every process in the unit's cgroup. A service with 8 workers gets 1 GB total, not 1 GB per worker.
  • After editing a drop-in, daemon-reload plus a restart applies it cleanly. systemctl set-property can change limits on a running unit; without --runtime it records the change persistently too. Pick one method and stay consistent.

Choosing numbers that do not lie

The most common failure is setting MemoryMax to the RSS you see in ps on a quiet afternoon. That number excludes page cache, tmpfs, socket buffers, and the startup peak.

A better procedure:

  1. Run the service through a full workload cycle — including the nightly job, the import, the backup — with accounting on and no limit.
  2. Read memory.peak and the anon line from memory.stat at the end.
  3. Set MemoryMax above the peak with 20–30% headroom for allocator behavior and traffic growth.
  4. Set MemoryHigh at roughly 80–90% of MemoryMax, and never below the steady-state anon value.
  5. If the service uses /dev/shm or tmpfs, add that to the budget. It is charged as file, but it still counts against memory.max.
  6. If the unit runs under a slice with its own cap, size the slice first. EffectiveMemoryMax will silently ignore the rest.

Example: a Python API with a 600 MB steady-state heap, a 900 MB startup spike, and 200 MB of page cache under load. MemoryMax=1.2G and MemoryHigh=1G gives the startup room, keeps the cache, and leaves the high limit above the anonymous working set. Setting MemoryHigh=700M because "700 MB sounded right" would reproduce the stall from the lab.

Reading the evidence after an incident

When something gets OOM-killed, these are the commands that answer "what happened":

# Kernel OOM messages
journalctl -k --since "2 hours ago" | grep -i -E 'out of memory|oom-kill|killed process'

# Counters for one unit
CG=$(systemctl show myapp.service -p ControlGroup --value)
cat "/sys/fs/cgroup$CG/memory.events"
cat "/sys/fs/cgroup$CG/memory.events.local"

# Anonymous vs file memory
grep -E '^(anon|file|kernel|slab|sock|shmem) ' "/sys/fs/cgroup$CG/memory.stat"

# Live view, sorted by memory
systemd-cgtop -m --depth=1
Enter fullscreen mode Exit fullscreen mode

memory.reclaim is the manual version of the reclaim the kernel performs under pressure. On Linux 5.19+:

# Ask for 512 MiB back from the unit's cgroup
echo 512M | sudo tee "/sys/fs/cgroup$CG/memory.reclaim"
Enter fullscreen mode Exit fullscreen mode

The kernel may reclaim more or less than requested; if it reclaims less, the write fails with EAGAIN. It is a nudge, not a guarantee, and the documentation is careful to say that proactive reclaim is not itself a memory-pressure signal.

memory.events is hierarchical; memory.events.local counts only events at that cgroup level. If max is climbing but oom_kill is zero, the cgroup is touching the wall and recovering. If oom_kill is climbing, something is dying repeatedly, and a supervisor with Restart=always will turn that into a loop.

The kernel log line looks like this (values omitted because they depend on the process):

Out of memory: Killed process 4242 (python3) total-vm:... anon-rss:... file-rss:... shmem-rss:...
Enter fullscreen mode Exit fullscreen mode

The anon-rss and file-rss split tells you whether the victim was heap-heavy or cache-heavy. The PID tells you whether the killer picked the process you expected.

By default memory.oom.group is 0, so the OOM killer selects one task inside the cgroup, not the whole unit. That is why a multi-process service can lose one worker, restart it, and appear healthy while the leak continues. If you need all-or-nothing behavior, memory.oom.group=1 makes the cgroup an indivisible workload; tasks with oom_score_adj of -1000 are still exempt. Treat that as an advanced option, not a default.

systemd-oomd is a separate, optional service that watches PSI and can kill a whole cgroup before the kernel OOM path runs. It acts on units marked ManagedOOMMemoryPressure=kill and is configured through oomd.conf. It is a useful backstop on desktops and laptops; it is not a substitute for per-unit limits, and it is not installed everywhere.

Things that surprise people

  • memory.max can be exceeded briefly. The documentation says so. Accounting is per-page and allocation is not atomic; a burst can push a cgroup over the line before reclaim catches up.
  • Page cache counts. A service that reads a lot can look huge in memory.current while being perfectly healthy. Look at anon before filing a leak.
  • Kernel memory counts. Page tables, slab, and socket buffers are charged to the cgroup. A network proxy with a modest RSS can hit its limit because of socket buffers.
  • tmpfs counts. /dev/shm and tmpfs writes are charged as file memory to the cgroup that touched them. A 1 GB /dev/shm allocation is a 1 GB charge even though no process owns it as RSS.
  • The root cgroup has no limit. Per-unit limits only protect you if the units that can grow are actually limited. Ten unlimited services can still exhaust the machine.
  • Containers nest. A container runtime puts the container in its own cgroup, and limits compose. A container with a 2 GB limit inside a 1 GB slice gets 1 GB, and the container's own memory.max is not the effective one.

The short version

MemoryHigh is a throttle, not a cap. MemoryMax is the cap, and it kills inside the cgroup instead of across the machine. Set MemoryMax from measured peaks with headroom, set MemoryHigh above the anonymous working set, and read memory.events when you want to know which one fired.

The OOM killer is not malicious. It is just the only thing left when nobody set a limit.

Sources

Top comments (0)