Mustafa ERBAY

Posted on Sep 22 Originally published at mustafaerbay.com.tr

The 66 MiB Nobody Wrote Down: Who Sets min_free_kbytes?

#linux #kernel #sysctl #memory

While digging through /etc/sysctl.d/ for vm.max_map_count yesterday I picked up a habit: find the owner of every number. Today it was vm.min_free_kbytes's turn, and the first query surprised me. VPS3 has 96 GB of memory; sysctl vm.min_free_kbytes says 67584. VPS5 has 12 GB; the same command says 67584 again. On neither machine does the name appear in /etc/sysctl.d, /usr/lib/sysctl.d or /etc/sysctl.conf. The kernel's own source says something else entirely: the comment table in mm/page_alloc.c ends with "16384MB: 16384k", 16 MB of reserve for a 16 GB machine. Scale that to 96 GB and you get about 39 MiB; to 12 GB, 13 MiB. Neither is 66 MiB. A number nobody wrote coming out identical on two different machines meant there was code writing it, so I went looking for that code.

The answer has two parts. First: if transparent hugepages are enabled on your server, neither you nor your distribution sets min_free_kbytes — khugepaged does, and the number it picks ignores how much memory the machine has. Second, and this is what I actually care about: the number is not "memory kept free", it is the lowest of three separate lines. A different knob has decided when kswapd wakes up since 2016, people still turn the wrong one, and in one lab experiment a ClickHouse container paid the bill for that confusion.

Three lines, three different questions

The kernel keeps three watermarks per memory zone: min, low, high. /proc/zoneinfo prints them in pages. VPS3's Normal zone shows 16,374 / 40,320 / 64,266; the zone manages 23,948,668 pages, so the lines sit at 64 MB, 157 MB and 251 MB. In a 91 GB zone.

Who looks at which line? In mm/page_alloc.c the fast path of a page request checks against ALLOC_WMARK_LOW: if the zone has more free pages than low, the page is handed out and nobody is woken. Below low the request is still served, but it drops to the slow path; the slow path first wakes kswapd, then retries with ALLOC_WMARK_MIN. Below min an ordinary request gets no page; the requesting process enters reclaim itself and the allocstall_* counters in /proc/vmstat tick up by one. kswapd, once awake, works until high: pgdat_balanced in mm/vmscan.c scans the node's zones from the bottom up and declares the node balanced as soon as one eligible zone has cleared its high line, at which point kswapd sleeps.

The region below min is not one piece either. __zone_watermark_ok builds a nested ladder (the flags are defined in v6.8 mm/internal.h, VPS3's kernel). For a request carrying __GFP_HIGH the threshold halves, to min/2. If that request also cannot block — classic GFP_ATOMIC, the receive buffers a NIC allocates in interrupt context — a further quarter of what remains is struck off, leaving a threshold of 3/8 × min; such requests can reach 62.5% of the reserve. The discounts are nested: being unable to block without __GFP_HIGH (GFP_NOWAIT) earns no privilege at all. A process marked as an OOM victim halves what is left once more. Kernel paths running with the PF_MEMALLOC flag (kswapd itself, for one) check no line at all. What we call the emergency reserve is exactly this: the band below min, set aside for requests that cannot sleep. That is why the source comment explains the square-root growth with "network bandwidth does not increase linearly with machine size"; the reserve was for network buffers.

The distance between the lines is a separate story. Until 2016 it was low = min + min/4, high = min + min/2; the only way to wake kswapd earlier was to enlarge min_free_kbytes, the emergency reserve. Johannes Weiner's patch 795ae7a (March 2016, v4.6) separated the two: "In order to get kswapd to maintain a 250M buffer of free memory, the emergency reserves need to be set to 1G. That is a lot of memory wasted for no good reason." The new knob is vm.watermark_scale_factor: its unit is one ten-thousandth, its default 10, so the gap between lines is one thousandth of the zone. The code keeps the old formula too: gap = max(min/4, managed × factor / 10000). On a small machine the old formula wins, on a big one the new. On VPS3's Normal zone min/4 is 4,093 pages and one thousandth is 23,948; the measured gap is 23,946, so the new formula (the two-page difference is the same managed drift). On VPS5 min/4 is 3,164 and one thousandth is 2,230; the gap is 3,164, the old formula. Same default, two different rules on two machines.

The arithmetic behind 67584

calculate_min_free_kbytes really does take a square root: int_sqrt(lowmem_kbytes * 16), clamped between 128 and 262,144. But the last word belongs to mm/khugepaged.c: when the THP subsystem initialises, start_stop_khugepaged is called, and it calls set_recommended_min_free_kbytes; on memory hot-add or removal, khugepaged_min_free_kbytes_update redoes the same sum. The function runs its own arithmetic on every machine where THP (always or madvise) is enabled:

recommended_min  = pageblock_nr_pages × nr_zones × 2
recommended_min += pageblock_nr_pages × nr_zones × MIGRATE_PCPTYPES × MIGRATE_PCPTYPES
recommended_min  = min(recommended_min, lowmem / 20)      # 5% cap
recommended_min <<= (PAGE_SHIFT - 10)                     # pages → KB
if recommended_min > min_free_kbytes: min_free_kbytes = recommended_min

The comment in the code spells out the reasoning: keep two pageblocks free per zone so fragmentation can be avoided, and on top of that, for each of the three migrate types (unmovable, movable, reclaimable), enough blocks to fall back into the other types, three each, nine in total. On x86-64 a pageblock is 2 MiB, 512 pages; both VPS3 and VPS5 have three populated zones (DMA, DMA32, Normal). 512 × 3 × 2 + 512 × 3 × 9 = 16,896 pages, times 4 KB equals 67,584. The machine's memory is not in the formula — only the 5% cap is, and that only bites below 1.3 GB, while the square-root formula would need 285 GB to exceed 66 MiB.

What the formula does contain, instead of memory, is the number of zones, and that means the number is not 67584 everywhere. for_each_populated_zone walks every node: on a two-socket server the second node's Normal zone counts too, making four zones and 90,112. On an x86-64 machine with less than 4 GB the Normal zone is never populated, leaving two zones and 45,056. Docker Desktop's virtual machine on the Mac (linuxkit 6.10, arm64) showed exactly that: its zones are DMA and Normal, and min_free_kbytes is 45056, which is 512 × 2 × 11 × 4 KB. So it is 67584 on every single-socket, three-zone server between a few gigabytes and a few hundred; change the node or zone count and the number changes with it. Ubuntu 24.04 ships THP as madvise and both of my machines are single-node, which is why they report the same number.

The most interesting thing about the function is its silence. When it raises the value it prints "raising min_free_kbytes from ... to help transparent hugepage allocations" with pr_info, but only if user_min_free_kbytes >= 0; that variable is -1 until you write something via sysctl. At boot nobody has, so the line is never printed. dmesg | grep min_free is empty on both servers. On VPS5 I produced the evidence by hand: writing echo never > /sys/kernel/mm/transparent_hugepage/enabled dropped min_free_kbytes from 67584 to 13769 (the square-root formula, for 12 GB); restoring madvise put it back to 67584, again silently. A sysadmin who disables THP is cutting the emergency reserve to a fifth on VPS5 (to about 60% of it on the 96 GB VPS3) without knowing it. And that switch is no longer the only one: the kernel now also looks at whether any mTHP size or the shmem setting is enabled, and if one of them is, the arithmetic stays khugepaged's.

One more trap: once you have written a value through sysctl, user_min_free_kbytes is your number and the kernel's arithmetic defers to it. When I disabled THP in the lab, dmesg got this: "min_free_kbytes is not updated to 11283 because user defined value 45056 is preferred". I had written 45056 "to go back to the old value"; the kernel recorded it as "the user wants this". There is no way to reset the variable short of a reboot. One caveat: that deference applies only to the square-root path. khugepaged's arithmetic does not consult the user value; if its own recommendation is larger, it overwrites. So if you have set something below 67584 via sysctl, toggling THP off and on, or hot-adding memory, will raise your value right back.

How the number is spread across zones

__setup_per_zone_wmarks splits the total across zones in proportion to managed pages. In the lab I wrote 200 MiB (51,200 pages): DMA got 14,420, Normal 36,779, total 51,199; DMA's share is 565,054 / 2,006,203 × 51,200 = 14,420.7. Exact — as it should be, since I wrote the value there and then, and the lines were computed at that moment. On VPS3 the 16 MB DMA zone got 2 pages, DMA32 518, Normal 16,374; redoing the same proportion with today's managed values gives 2 / 523 / 16,370. The five-page gap is not a mystery but a reminder: the lines were computed once at boot, and the zones' managed page counts have shifted by a few thousand since. Movable and highmem zones follow a separate rule: managed / 1024, clamped between 32 and 128; that is why VPS3's empty Movable zone reads min 32.

The protection line in zoneinfo is part of the same arithmetic and is usually misread. On VPS3's DMA zone it says (0, 2991, 96541, ...): when a request that could also be served from Normal wants a page from DMA, the threshold becomes min + 96,541 pages, like asking 377 MiB of a 16 MB zone, so DMA is effectively closed to ordinary requests. The number is the sum of the upper zones' managed pages divided by vm.lowmem_reserve_ratio (256 256 32 0 0): for DMA, (765,914 + 23,948,668) / 256 = 96,541, and one line down, for DMA32, 23,948,668 / 256 = 93,549. Both exactly as zoneinfo prints them.

One last reading: the "free" column of free -m does not know about the lines; the "available" column does. si_mem_available in mm/show_mem.c subtracts totalreserve_pages from free pages, and that is every zone's high line plus its lowmem_reserve share. In experiment five, when I set min_free_kbytes to 2 GiB, MemFree showed 2.5 GB while MemAvailable dropped to 833 MB. What makes up the difference is not min but the topmost line: the sum of the high watermarks alone had reached 3 GiB, and available was no longer offering that memory to anyone.

Lab: watching kswapd across four runs

I cannot do this on VPS3; every experiment sweeps the page cache. I used a privileged python:3.13-slim container in Docker Desktop's 7.8 GB virtual machine. The VM had been up for three and a half days and /proc/vmstat was telling its own story: pgscan_kswapd 17 million, allocstall_normal 686, kswapd_low_wmark_hit_quickly 684. On VPS3 the same counters have been zero for 11 days and 16 hours; the kswapd0 process (PID 154) has used a total of 0 seconds of CPU in that time. On a machine with 73 GB free, kswapd has never once woken up.

Before each run I read a 2.5 GB file to fill the cache. One script samples zoneinfo and vmstat four times a second — so the "lowest free" figures below are the lowest sampled, and the real trough may be deeper. Another grabs memory with mmap in steps, touching every page, and once it reaches 3.5 GB holds for two seconds and releases. Variables: watermark_scale_factor (10 or 500) and allocation rate (64 MiB per 50 ms, or 128 MiB with no pause).

run  scale  step            kswapd scan  direct  allocstall  lowQ  highQ  swap-out  lowest Normal free (low)
A    10     64 MiB/50 ms    434,342      0       0           11    5      1,778     9,981   (10,113)
B    500    64 MiB/50 ms    589,730      0       0           0     0      5,658     79,120  (80,148)
C    10     128 MiB/0 ms    533,604      8,832   2           23    0      18,480    9,339   (10,113; min 8,091)
D    500    128 MiB/0 ms    612,480      0       0           0     7      54,579    80,611  (80,148)

In run A kswapd appears to have woken in the sample where the Normal zone dipped below low (10,113), at 9,981, scanned 434 thousand pages of cache, and the process never entered direct reclaim. But kswapd_low_wmark_hit_quickly rose by 11: kswapd had reached high and settled into its 100 ms nap, only to be woken again, eleven times. That is the "premature sleep" the documentation describes. In run C, with the rate doubled, the line was breached: two allocstalls, 8,832 pages of direct scanning, Normal free down to 9,339 (1,248 pages above min at 8,091), and the DMA zone down to 24 pages above its min.

Setting the scale factor to 500 (5%) pushed low to 80,148 pages (313 MB) and high to 152,205; kswapd now woke with half a gigabyte still free, and in runs B and D direct reclaim was zero, lowQ zero. The price is in the table: swap-out was 18 thousand pages in C and 54 thousand in D. With a bigger target kswapd reclaims more, and once the cache is gone it writes anonymous pages to swap; in run D, 213 MB of anonymous pages went to swap. Weiner's 0.1% default expresses that balance; 500 is a measurement value, not a production recommendation.

Lab, run five: what too much does

The documentation waves this possibility away in a single sentence: "Setting this too high will OOM your machine instantly." I wanted to try it; on the 7.8 GB machine I set min_free_kbytes to 2 GiB. The Normal zone's lines became 376,620 / 470,775 / 564,930 pages; kswapd immediately swept the cache, bringing the VM to 2.5 GB free. Then I started the load script with oom_score_adj=1000, so it would be the victim.

Normal free:1506472kB boost:0kB min:1506480kB low:1883100kB high:2259720kB ... managed:5764596kB
Out of memory: Killed process 1467398 (python3) total-vm:3814056kB, ... shmem-rss:3763072kB, oom_score_adj:1000

Free 1,506,472 KB, line 1,506,480 KB; eight kilobytes under. OOM with 1.5 GB free. The documentation was right, but what followed was something I had not expected: 91 milliseconds later, a second OOM report. This time the trigger was a nats-server running in the same VM, and the victim another project's clickhouse-server container (775 MB anonymous RSS, oom_score_adj 0). The container came back on its own thanks to its restart policy (it had been up 44 seconds when I looked); it was a test environment for other work, no data lost. Still, I killed someone else's container for a blog post.

Why a second victim? Because when the OOM killer marks a process, the memory does not come back at that instant. The kernel thread that forcibly tears down a victim's memory (oom_reaper) is queued behind a two-second timer — OOM_REAPER_DELAY (2*HZ). Nothing could have been reaped within 91 milliseconds; the victim had not finished its own exit_mmap, its pages were still there, and min was still breached. nats-server asked for a page inside exactly that window and the OOM killer picked one more victim. A second detail stretches that delay: the load script's memory came from mmap(-1, ...), which Python issues as MAP_SHARED | MAP_ANONYMOUS, and the kernel accounts as shmem (the shmem-rss:3763072kB in the report). Even when the reaper's timer fired, it would have skipped those pages: if (vma_is_anonymous(vma) || !(vma->vm_flags & VM_SHARED)). The lesson is not just "don't inflate min_free_kbytes"; the few hundred milliseconds after an OOM are still an OOM, and if the line is high, whoever asks next in that window is the one who goes.

The second report's timestamp is 91 milliseconds after the first (322601.437 and 322601.528); both sit side by side in dmesg. When the experiments ended I wrote 45056 back in the VM and madvise back on VPS5; VPS3 was never touched.

What too little does

At the other end the warning is more detailed: "if you set this to lower than 1024KB, your system will become subtly broken, and prone to deadlock under high loads." In the lab I wrote 128; the Normal zone's min fell to 22 pages, DMA's to 9. For GFP_ATOMIC the threshold is 3/8 × min, nine pages, leaving a request in interrupt context a cushion of thirteen pages. The moment a NIC requests receive buffers back to back in interrupt context, those five pages run out and "page allocation failure" lands in dmesg; a packet is dropped, TCP retransmits, you see latency. I stopped the experiment there; there is no way to deliberately produce an indeterminate deadlock, and on VPS3 the count of that message over 11 days is zero. A zero counter means the reserve is enough, and whether that line is zero on your machine takes a minute to learn: journalctl -k | grep -c "page allocation failure".

Which knob for which problem

There is a third knob, aimed at a different problem: vm.watermark_boost_factor (Mel Gorman, 1c30844d, v5.0). When a pageblock gets mixed by different migrate types, this factor adds a temporary allowance on top of the high line — the default of 15000 means up to 150% of high in extra reclaim — and kswapd works until that allowance is burned down, so fragmentation eases. The boost line in zoneinfo shows the allowance; it was zero on all three of my machines, and in the Docker Desktop VM the factor itself had been set to 0 (versus 15000 on the VPSes). vm.defrag_mode, added in v6.15, is a more permanent answer to the same problem; that is another post.

Ask your machine three questions. Who owns the reserve: if sysctl vm.min_free_kbytes says 67584 it is THP's, and grep -rs min_free /etc/sysctl.d /usr/lib/sysctl.d will come back empty; disable THP and the number drops. Is the reserve enough: journalctl -k | grep -c "page allocation failure"; if zero, there is no reason to touch min_free_kbytes. Is kswapd keeping up: sample grep -E "allocstall|kswapd_low_wmark_hit_quickly" /proc/vmstat once a minute; if both are growing, processes are breaching the line and kswapd is waking late.

If the third answer is yes, the knob to turn is watermark_scale_factor, not min_free_kbytes. Weiner's 2016 reasoning still holds: enlarging the emergency reserve for a latency problem means enlarging memory that will never be given to anyone. But turning the scale is not free either, and the cost can be computed in advance: on VPS3, writing 50 instead of 10 would push the gap between two lines in the Normal zone from 23,948 to 119,743 pages, that is 468 MiB, with nearly 1 GiB between min and high. Go in steps — 10 to 20, then 50 if needed — and at each step read swap-out alongside allocstall and kswapd_low_wmark_hit_quickly; the swap increase I saw at 500 in the lab is a real cost. A counter that is non-zero but flat is not worth acting on; what matters is a counter that grows under load.

For persistence use /etc/sysctl.d/99-<yourname>.conf, and if you put vm.min_free_kbytes there, know what it means: the moment sysctl is applied at boot, user_min_free_kbytes is locked and the kernel's square-root arithmetic never runs again. Touch min_free_kbytes only if you see allocation failure anyway; khugepaged's 5% cap is not a kernel rule for you either, just the limit that function puts on its own recommendation — though in practice a sensible ceiling. On a NUMA machine all of this is per node: each node has its own kswapdN, its own set of zones and its own balance, so read zoneinfo by its Node lines. In an unprivileged container you can do none of it — these are the host's settings — while a privileged container (as in my lab) overwrites the host's sysctl directly, so be careful. What the container side does have is not watermarks but cgroups: memory.min, memory.low and memory.pressure let you write down the share your workload keeps, and the global lines know nothing about it.

Conclusion

The table in the source is still there, ending in "16384MB: 16384k", and since 2011 (v2.6.38, the release that merged THP) nobody on most x86-64 servers has seen it. khugepaged sets the number: silently, at boot, without looking at the machine's size. The number itself was not what I thought either: not "memory kept free", but a band reserved for requests that cannot sleep, and above that band a range where kswapd works, whose width a different knob decides. A process killed with 1.5 GB free in the 2 GiB experiment, and an innocent ClickHouse gone 91 milliseconds later, drilled the cost of that distinction into me. Before looking at the free column and saying "there's memory", look at available; the difference is the part the kernel will not give you.

Versions: VPS3 Ubuntu 24.04, kernel 6.8.0-139, 96 GB, THP madvise, up since 10 September. VPS5 Ubuntu 26.04, kernel 7.0.0-31, 12 GB, THP madvise. Lab: Docker Desktop, linuxkit 6.10.14 arm64, 7.8 GB, python:3.13-slim; min_free_kbytes restored to 45056 and watermark_scale_factor to 10; THP restored to madvise on VPS5; no setting was changed on VPS3. Kernel source readings from the torvalds/linux main branch (mm/page_alloc.c, mm/khugepaged.c, mm/vmscan.c, mm/show_mem.c, mm/oom_kill.c) and the v6.8 tag; watermark_scale_factor verified against v4.6, watermark_boost_factor against v5.0, defrag_mode against v6.15 documentation. On both VPS3 and VPS5 the per-zone shares drift by a few pages against today's managed values; since the lines are computed at boot this is expected, and the only place the proportion matched exactly is the lab machine where I wrote the value myself.

DEV Community