At 03:12 the kernel killed your SSH session instead of the service that was eating 14 GB. Nothing was broken. Nothing was limited either. The whole machine was one big cgroup, and the OOM killer picked a victim by badness score, not by blame.
That is the default on most Linux systems: every process competes for the same pool of memory. When the pool runs dry, the kernel chooses something to kill. It might choose the leak. It might choose your database. It might choose sshd.
cgroup v2 gives you a better answer than "hope the score is right". You can put a ceiling on a service, make it slow down before it gets there, and read exactly what happened afterwards. The knobs are small and the semantics are not obvious. This article is about the semantics.
The lab below was run on Debian 13 (kernel 6.12, systemd 257). The cgroup files are the same on any system using the unified hierarchy.
First, check what you are working with
stat -fc %T /sys/fs/cgroup
# cgroup2fs
If that prints cgroup2fs, you are on cgroup v2 and everything below applies. If it prints tmpfs, you are on the old hierarchy and the file names are different (memory.limit_in_bytes, memory.memsw.limit_in_bytes). Check before copying any of this.
Also check that the memory controller is actually available:
cat /sys/fs/cgroup/cgroup.controllers
# cpuset cpu io memory hugetlb pids rdma misc
If memory is missing, the controller is disabled in the kernel command line or not compiled in, and none of the unit settings will do anything.
What the kernel is measuring
Every cgroup has a memory.current file. It is the total memory currently charged to that cgroup and its descendants, in bytes.
That number is not RSS. It includes:
- anonymous memory (heap, stacks,
mmap(MAP_ANONYMOUS)) - page cache charged to the cgroup
- tmpfs and shared memory
- kernel memory charged to the cgroup: page tables, kernel stacks, slab, socket buffers
memory.stat breaks it down. The two lines you will use most are anon and file:
cat /sys/fs/cgroup/system.slice/myapp.service/memory.stat | grep -E '^(anon|file|kernel|slab|sock|shmem) '
This distinction matters because file is mostly reclaimable. A service that reads 2 GB of logs can show 2 GB in memory.current while being under no real pressure: the kernel can throw those pages away and read them again later. A service whose anon grows by 200 MB per hour has a leak, and no amount of cache dropping will fix it.
The kernel documentation is explicit that file includes tmpfs and shared memory. That is worth remembering the next time a service with a small RSS hits its limit because something wrote 500 MB to /dev/shm.
The knobs that matter
There are five memory files you will actually use, plus three for observation.
| File | What it does |
|---|---|
memory.min |
Hard protection. Memory inside this boundary is not reclaimed under any condition. |
memory.low |
Best-effort protection. Reclaimed only if nothing unprotected is available. |
memory.high |
Throttle. Going over puts the cgroup under heavy reclaim pressure and slows the allocating tasks. Does not invoke the OOM killer. |
memory.max |
Hard limit. If usage cannot be reduced below it, the OOM killer runs inside the cgroup. |
memory.swap.max |
Ceiling on swap used by the cgroup. Default is max. |
memory.events |
Counters: low, high, max, oom, oom_kill, oom_group_kill. |
memory.peak |
High-water mark since cgroup creation (Linux 5.19+). |
memory.reclaim |
Write-only trigger to ask the kernel to reclaim a specified amount (Linux 5.19+). |
The important pair is memory.high and memory.max, because people treat them as interchangeable and they are not.
memory.high is a speed limit. When usage goes over it, the kernel does not kill anything. It applies heavy reclaim pressure and throttles the tasks that are trying to charge more memory. The kernel documentation is blunt about the guarantee: going over the high limit never invokes the OOM killer, and under extreme conditions the limit may be breached. memory.events counts these episodes in the high field.
memory.max is a wall. When usage reaches it and reclaim cannot bring it down, the OOM killer is invoked in that cgroup. The doc again: "If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup." It also warns that usage may go over the limit temporarily, because accounting and allocation are not atomic.
That is the whole design: high buys you time and backpressure; max is the last line of defense. systemd's own documentation recommends exactly that split, using MemoryHigh= as the main control and MemoryMax= as the fallback.
Limits compose up the tree
cgroup v2 has a no-internal-process rule: a non-root cgroup can contain processes or child cgroups, not both. systemd builds slices (which contain units) and services (which contain processes) on top of that rule.
The practical consequence is that a limit is never just yours. If myapp.service sits under myapp.slice, and the slice is capped at 2 GB, then a 4 GB MemoryMax= on the service is not reachable. systemctl show exposes the result:
systemctl show myapp.service -p MemoryMax -p EffectiveMemoryMax
EffectiveMemoryMax is the most stringent limit of the unit and its parent slices, capped by physical memory. When a limit "does not work", this is the first field to read.
A live lab: 200 MB under two different limits
Here is a real experiment. It uses a transient user scope so nothing permanent is created:
systemd-run --user --unit=memlab --scope \
-p MemoryAccounting=yes \
-p MemoryHigh=256M \
-p MemoryMax=384M \
-p MemorySwapMax=0 \
/bin/sleep 300 &
Inspect what systemd actually applied:
systemctl --user show memlab.scope \
-p ControlGroup -p MemoryHigh -p MemoryMax \
-p EffectiveMemoryMax -p MemoryCurrent
Output from this machine:
ControlGroup=/user.slice/user-1000.slice/user@1000.service/app.slice/memlab.scope
EffectiveMemoryMax=402653184
MemoryCurrent=188416
MemoryHigh=268435456
MemoryMax=402653184
The numbers are bytes and the suffixes are base 1024: 384M is 402653184. Now read the cgroup files directly, using the path systemd reported:
CG=$(systemctl --user show memlab.scope -p ControlGroup --value)
for f in memory.max memory.high memory.swap.max memory.current memory.peak memory.oom.group; do
printf '%s=' "$f"
cat "/sys/fs/cgroup$CG/$f"
done
memory.max=402653184
memory.high=268435456
memory.swap.max=0
memory.current=188416
memory.peak=339968
memory.oom.group=0
Nothing controversial so far: a sleeping process, three limits, and an empty event counter.
Now the interesting part. Allocate 200 MB of anonymous memory inside a scope limited to 256 MB, with swap disabled:
systemd-run --user --unit=memtime1 \
-p MemoryMax=256M -p MemorySwapMax=0 --wait \
/usr/bin/python3 -c 'import time; t=time.time(); b=bytearray(200*1024*1024); print(round(time.time()-t,3))'
Service runtime: 84ms
CPU time consumed: 68ms
Memory peak: 203.2M (swap: 0B)
84 milliseconds. The allocation fits, the limit is never approached, and nothing is throttled.
Repeat it with MemoryHigh=128M added, while keeping MemoryMax=256M:
systemd-run --user --unit=memtime2 \
-p MemoryHigh=128M -p MemoryMax=256M -p MemorySwapMax=0 --wait \
/usr/bin/python3 -c 'import time; t=time.time(); b=bytearray(200*1024*1024); print(round(time.time()-t,3))'
This is where the intuition breaks. The process did not get killed. It also did not finish. After 58 seconds it was still allocating, sitting in uninterruptible sleep (D state) inside the kernel's reclaim path. The cgroup counters told the story:
memory.current=195170304
low 0
high 494
max 0
oom 0
oom_kill 0
oom_group_kill 0
494 throttle events, zero OOM kills. The process had charged about 186 MB, well above the 128 MB high limit, and the kernel kept throttling it instead of killing it. I stopped the unit at 58 seconds; it had not printed its completion line.
Two lessons, and they are the reason this section exists:
-
memory.highdoes not cap usage. It applies backpressure. Anonymous memory cannot be reclaimed when there is no swap, so the kernel has nothing to take and the limit gets breached while the task is throttled. - Backpressure can be so heavy that the workload makes almost no forward progress. A 200 MB allocation that took 84 ms with only a max limit was still not done after 58 seconds with an aggressive high limit.
If your goal is "never exceed X", use MemoryMax. If your goal is "slow down before you hit X", use MemoryHigh — but set it above the real anonymous working set, not at a number you wish the service used.
Putting limits on a real service
For a unit managed by systemd, do not edit files under /usr/lib/systemd/system. Use a drop-in:
sudo systemctl edit myapp.service
In the editor:
[Service]
MemoryAccounting=yes
MemoryHigh=768M
MemoryMax=1G
MemorySwapMax=0
Then:
sudo systemctl daemon-reload
sudo systemctl restart myapp.service
systemctl show myapp.service \
-p MemoryHigh -p MemoryMax -p MemorySwapMax \
-p EffectiveMemoryMax -p MemoryCurrent
A few details that are easy to get wrong:
- Sizes accept
K,M,G,Twith base 1024, or a percentage of installed physical memory.MemoryMax=80%is 80% of RAM, not 80% of the slice. -
infinityremoves the limit. It is the default for bothMemoryHighandMemoryMax. -
MemorySwapMax=0does not disable swap for the system; it stops this cgroup's anonymous pages from being swapped out. That makes the memory limit predictable, at the cost of pushing the service into reclaim/OOM instead of letting it spill to disk. -
MemoryMaxapplies to every process in the unit's cgroup. A service with 8 workers gets 1 GB total, not 1 GB per worker. - After editing a drop-in,
daemon-reloadplus a restart applies it cleanly.systemctl set-propertycan change limits on a running unit; without--runtimeit records the change persistently too. Pick one method and stay consistent.
Choosing numbers that do not lie
The most common failure is setting MemoryMax to the RSS you see in ps on a quiet afternoon. That number excludes page cache, tmpfs, socket buffers, and the startup peak.
A better procedure:
- Run the service through a full workload cycle — including the nightly job, the import, the backup — with accounting on and no limit.
- Read
memory.peakand theanonline frommemory.statat the end. - Set
MemoryMaxabove the peak with 20–30% headroom for allocator behavior and traffic growth. - Set
MemoryHighat roughly 80–90% ofMemoryMax, and never below the steady-stateanonvalue. - If the service uses
/dev/shmor tmpfs, add that to the budget. It is charged asfile, but it still counts againstmemory.max. - If the unit runs under a slice with its own cap, size the slice first.
EffectiveMemoryMaxwill silently ignore the rest.
Example: a Python API with a 600 MB steady-state heap, a 900 MB startup spike, and 200 MB of page cache under load. MemoryMax=1.2G and MemoryHigh=1G gives the startup room, keeps the cache, and leaves the high limit above the anonymous working set. Setting MemoryHigh=700M because "700 MB sounded right" would reproduce the stall from the lab.
Reading the evidence after an incident
When something gets OOM-killed, these are the commands that answer "what happened":
# Kernel OOM messages
journalctl -k --since "2 hours ago" | grep -i -E 'out of memory|oom-kill|killed process'
# Counters for one unit
CG=$(systemctl show myapp.service -p ControlGroup --value)
cat "/sys/fs/cgroup$CG/memory.events"
cat "/sys/fs/cgroup$CG/memory.events.local"
# Anonymous vs file memory
grep -E '^(anon|file|kernel|slab|sock|shmem) ' "/sys/fs/cgroup$CG/memory.stat"
# Live view, sorted by memory
systemd-cgtop -m --depth=1
memory.reclaim is the manual version of the reclaim the kernel performs under pressure. On Linux 5.19+:
# Ask for 512 MiB back from the unit's cgroup
echo 512M | sudo tee "/sys/fs/cgroup$CG/memory.reclaim"
The kernel may reclaim more or less than requested; if it reclaims less, the write fails with EAGAIN. It is a nudge, not a guarantee, and the documentation is careful to say that proactive reclaim is not itself a memory-pressure signal.
memory.events is hierarchical; memory.events.local counts only events at that cgroup level. If max is climbing but oom_kill is zero, the cgroup is touching the wall and recovering. If oom_kill is climbing, something is dying repeatedly, and a supervisor with Restart=always will turn that into a loop.
The kernel log line looks like this (values omitted because they depend on the process):
Out of memory: Killed process 4242 (python3) total-vm:... anon-rss:... file-rss:... shmem-rss:...
The anon-rss and file-rss split tells you whether the victim was heap-heavy or cache-heavy. The PID tells you whether the killer picked the process you expected.
By default memory.oom.group is 0, so the OOM killer selects one task inside the cgroup, not the whole unit. That is why a multi-process service can lose one worker, restart it, and appear healthy while the leak continues. If you need all-or-nothing behavior, memory.oom.group=1 makes the cgroup an indivisible workload; tasks with oom_score_adj of -1000 are still exempt. Treat that as an advanced option, not a default.
systemd-oomd is a separate, optional service that watches PSI and can kill a whole cgroup before the kernel OOM path runs. It acts on units marked ManagedOOMMemoryPressure=kill and is configured through oomd.conf. It is a useful backstop on desktops and laptops; it is not a substitute for per-unit limits, and it is not installed everywhere.
Things that surprise people
-
memory.maxcan be exceeded briefly. The documentation says so. Accounting is per-page and allocation is not atomic; a burst can push a cgroup over the line before reclaim catches up. -
Page cache counts. A service that reads a lot can look huge in
memory.currentwhile being perfectly healthy. Look atanonbefore filing a leak. - Kernel memory counts. Page tables, slab, and socket buffers are charged to the cgroup. A network proxy with a modest RSS can hit its limit because of socket buffers.
-
tmpfs counts.
/dev/shmand tmpfs writes are charged as file memory to the cgroup that touched them. A 1 GB/dev/shmallocation is a 1 GB charge even though no process owns it as RSS. - The root cgroup has no limit. Per-unit limits only protect you if the units that can grow are actually limited. Ten unlimited services can still exhaust the machine.
-
Containers nest. A container runtime puts the container in its own cgroup, and limits compose. A container with a 2 GB limit inside a 1 GB slice gets 1 GB, and the container's own
memory.maxis not the effective one.
The short version
MemoryHigh is a throttle, not a cap. MemoryMax is the cap, and it kills inside the cgroup instead of across the machine. Set MemoryMax from measured peaks with headroom, set MemoryHigh above the anonymous working set, and read memory.events when you want to know which one fired.
The OOM killer is not malicious. It is just the only thing left when nobody set a limit.
Sources
- Linux kernel documentation, Control Group v2 — memory interface files,
memory.high,memory.max,memory.events,memory.reclaim,memory.peak: https://docs.kernel.org/admin-guide/cgroup-v2.html - Kernel source,
Documentation/admin-guide/cgroup-v2.rst(the exact text used above): https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/cgroup-v2.rst - systemd,
systemd.resource-control(5)—MemoryAccounting=,MemoryHigh=,MemoryMax=,MemorySwapMax=,EffectiveMemoryMax=: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html - systemd,
systemd-oomd.service(8): https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html - systemd,
systemd-cgtop(1): https://www.freedesktop.org/software/systemd/man/latest/systemd-cgtop.html - Linux kernel documentation, Pressure Stall Information: https://docs.kernel.org/accounting/psi.html
Top comments (0)