I Wrote 512 Bytes and Paid 1 GiB: THP, madvise and the 64 KiB Middle Road

Reading VPS3's /proc/vmstat this week, two lines sat next to each other: thp_deferred_split_page 3455 and thp_split_pmd 3455. Same number, two counters. Right below, thp_fault_alloc 11655: in eight days the kernel had handed out eleven thousand-odd 2 MiB huge pages. Then I looked at /proc/meminfo: AnonHugePages: 4096 kB. Eleven thousand handed out, two left. And those two belonged to a single process, a Node container running vinext.

Something is being allocated, something is being split, and almost nothing remains. This post chases those three numbers: which questions the single word in Transparent Huge Pages' (THP) enabled file does not actually answer, how a huge-page request on a system that says always silently drops to small pages, and why the multi-size THP that arrived in 6.8 changes the picture. Two lab machines: the usual VPS3 (Ubuntu 24.04, 6.8.0-139-generic, 96 GB RAM) and the linuxkit kernel inside Docker Desktop on my Mac (6.10.14, arm64, 7.8 GB). I did not touch the global setting on VPS3; the toggle experiments ran in the VM, and every one of them was reverted to its starting value.

THP in thirty seconds

On x86-64 a page is 4 KiB. A process's 512 MiB array is 131,072 pages: 131,072 page-table entries, 131,072 page faults on first touch, 131,072 separate translations in the TLB. Instead, the kernel can write a single 2 MiB entry one level up in the page table (the PMD); that is a "huge page". hugetlbfs hands these out from an explicitly reserved pool, in fixed numbers; PostgreSQL's huge_pages = try asks for that pool, and its documentation says plainly that "this setting only affects the main shared memory area", not THP. THP, by contrast, hands huge pages to anonymous memory without a reserved pool and without the process noticing: at page-fault time (the fault path) or later, when the khugepaged thread gathers small pages and merges them (the collapse path).

The gain comes from two places: fewer page faults and fewer TLB misses. So does the cost: touching a 2 MiB page once allocates all of it, and finding 2 MiB of physically contiguous memory on a fragmented system requires compaction.

One word, three settings

The file everyone looks at under /sys/kernel/mm/transparent_hugepage/ is enabled. On VPS3:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
$ cat /sys/kernel/mm/transparent_hugepage/defrag
always defer defer+madvise [madvise] never
$ grep TRANSPARENT_HUGEPAGE /boot/config-6.8.0-139-generic
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y

enabled answers one question: who may ask? Under always every anonymous mapping is a candidate; under madvise only ranges marked with madvise(MADV_HUGEPAGE); under never nobody. Ubuntu's kernel config picks madvise, while Docker Desktop's linuxkit kernel ships with always (I read both with my own eyes; I am not generalising about distributions). In the Red Hat world this value is often written by a tuned profile, which is why MongoDB warns RHEL users to "create a custom tuned profile". Out of curiosity I looked at the current throughput-performance profile: its only transparent_hugepages line is for Marvell ThunderX, and it says never. "tuned turns THP on" was true once; today you cannot say it without reading the profile file.

defrag answers the second question: when there is no contiguous 2 MiB, who waits? With always, direct reclaim and compaction happen at fault time, meaning the touching thread stalls. With madvise (the default) that stall is paid only for MADV_HUGEPAGE ranges; everything else drops to small pages without trying. defer wakes kswapd/kcompactd and moves on, leaving khugepaged to install the huge page later. The kernel documentation walks through all five; my summary is that enabled is "permission" and defrag is "who pays".

The third setting is not a file but a directory: khugepaged/. The values on VPS3:

scan_sleep_millisecs=10000   pages_to_scan=4096
max_ptes_none=511            max_ptes_swap=64   max_ptes_shared=256
full_scans=1142              pages_collapsed=2

Read max_ptes_none=511 out loud: even if 511 of the 512 pages in a 2 MiB range have never been mapped, khugepaged may collapse that range into a huge page. The 6.8 documentation puts it gently: "a higher value leads to use additional memory for programs"; the current document adds that for mTHP collapse only the values 0 and 511 are accepted. pages_collapsed=2 in eight days tells you that in madvise mode khugepaged only walks hg-flagged ranges, and on VPS3 there are almost none.

Who asked: the hg flag

Finding the owner of VPS3's two huge pages took three commands. The only process with AnonHugePages above zero in /proc/*/smaps_rollup was node (pid 9217); in smaps, exactly one of its 490 mappings had THPeligible: 1:

23b07800000-23b47800000 rw-p 00000000 00:00 0    [anon:mimalloc]
Size:            1048576 kB
Rss:                4096 kB
AnonHugePages:      4096 kB
THPeligible:           1
VmFlags: rd wr mr mw me nr sd hg

The trailing hg says that VMA was marked with MADV_HUGEPAGE. Node did not mark it; the process had loaded the @rolldown/binding-linux-x64-gnu native module, and rolldown, written in Rust, uses mimalloc as its allocator. mimalloc's Unix primitives mark every large region with madvise(MADV_HUGEPAGE) when the allow_thp option is on (default 1, only Android defaults to 0). It reserved 1 GiB, used 4 MiB, and those 4 MiB became two huge pages. The server's entire THP story came down to an allocator option nobody had consciously enabled.

The JVM walks through the same door: -XX:+UseTransparentHugePages is off by default and its documentation describes it as an option "made available for experimentation"; turn it on and the heap is marked with madvise, so it gets huge pages even on a server in madvise mode. You do not need to walk smaps to see a process's overall state: the THP_enabled: line in /proc/PID/status tells you whether it was switched off via prctl; for node on VPS3 it reads 1.

The opposite exists too. The disable-thp yes line in Redis's redis.conf (since 6.2) becomes a prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0) call in server.c; even if the global setting is always, the Redis process opts itself out, citing fork and copy-on-write latency. On VPS3's two Redis containers, config get disable-thp → yes. MongoDB's THP page, meanwhile, asks for the exact opposite from 8.0 onwards, always + defer+madvise + max_ptes_none=0, and on the same page states that "you cannot enable or disable THP per-process level". Redis's prctl does precisely that; documentation does not always read other documentation.

Lab 1: four attempts in madvise mode

I wrote a small C program: it mmaps a 2 MiB-aligned region, applies madvise or prctl depending on the mode, writes one byte per 4 KiB, then prints the minor-fault count from getrusage and AnonHugePages from /proc/self/smaps_rollup. VPS3, 512 MiB:

mode=plain     touch: 671.9 ms  minor fault=131075  AnonHugePages=0 kB
mode=madv      touch: 171.9 ms  minor fault=259     AnonHugePages=524288 kB
mode=nohuge    touch: 657.1 ms  minor fault=131075  AnonHugePages=0 kB
mode=nothp     touch: 586.2 ms  minor fault=131075  AnonHugePages=0 kB

plain is a bare mapping: since it is not a candidate in madvise mode, 131,075 faults. madv is the same region with MADV_HUGEPAGE: 259 faults (256 huge pages plus program overhead), a four-times faster touch, and thp_fault_alloc up by exactly 256. nohuge is MADV_NOHUGEPAGE; nothp is PR_SET_THP_DISABLE followed by MADV_HUGEPAGE: prctl wins, exactly as the documentation says, it "will disable THPs completely for the process, irrespective of global THP controls".

A fifth attempt has existed since 6.1: MADV_COLLAPSE. I touched a bare mapping (131,075 faults) and then called madvise(p, len, MADV_COLLAPSE); it returned OK, AnonHugePages went from 0 to 524,288 kB, thp_collapse_alloc from 2 to 258. madvise(2) describes it as a "best-effort synchronous collapse" independent of sysfs settings; even enabled=never does not stop it, only PR_SET_THP_DISABLE and MADV_NOHUGEPAGE do.

I wrote 512 bytes and paid 1 GiB

The same program's "sparse" mode writes just one byte per 2 MiB across a 1 GiB region; 512 bytes in total.

mode=plain size=1024 MiB (sparse)  minor fault=515  AnonHugePages=0 kB        Rss=3784 kB
mode=madv  size=1024 MiB (sparse)  minor fault=515  AnonHugePages=1048576 kB  Rss=1050316 kB

Same fault count, different bill: the bare mapping costs 3.7 MB of RSS, the MADV_HUGEPAGE mapping 1 GiB. A huge page's unit is 2 MiB; every sparsely touched structure (large but lightly used hash tables, lazily filled buffers, allocator arenas) pays this price. In always mode it applies to every process, no madvise required. 6.12's shrink_underused support can be the remedy for this wound, but not on its own: the documentation counts a THP as "underused" only if the number of zero-filled pages is above max_ptes_none; with the default of 511 even my 512-byte scenario would not be split, you have to lower the threshold. VPS3's 6.8 does not have the file at all.

Counter at zero, memory still there: deferred split

Now back to the 3455 from the opening. The madv-split mode fills 512 MiB with huge pages and then gives back 4 KiB from the middle of every 2 MiB with MADV_DONTNEED:

[after touch]                AnonHugePages=524288 kB  Rss=526028 kB  deferred_split=3455  split_pmd=3455
[4 KiB DONTNEED per THP]     AnonHugePages=0 kB       Rss=525004 kB  deferred_split=3711  split_pmd=3711
[2 s later]                  AnonHugePages=0 kB       Rss=525132 kB  deferred_split=3711  split_pmd=3711

AnonHugePages fell from 512 MiB to zero; RSS fell by just 1 MiB. What happened: when part of a huge page is released, the kernel splits the PMD entry into 512 PTEs (thp_split_pmd) but does not break up the physical 2 MiB page right away; it puts it on the "deferred split" queue (thp_deferred_split_page) and leaves the actual split to a shrinker that runs under memory pressure. The documentation's sentence: a partially unmapped huge page is queued, and split if splitting would free memory. thp_split_page on VPS3 is still 0; no pressure, the queue waits.

That is why the two counters were equal: in eight days on VPS3, 3455 huge pages were partially released, opened into PTEs and queued; none was split under pressure. The counter is a cumulative event count, not the queue's current length: when the owning process exits, the page leaves the queue and is freed, and thp_split_page still does not move. 6.8 does not show how many are still waiting; that information arrived with 6.12's nr_anon_partially_mapped counter. While they wait, AnonHugePages does not count them; free shows them as "used". When you see "AnonHugePages is zero but RSS won't come down" in a process that uses huge pages, this is the counter to check.

always's silent fallback

To test always mode I entered Docker Desktop's VM with a --privileged container; sysfs is writable there and the kernel already ships with always. The VM has 7.8 GB, 4.5 GB of it page cache; in the /proc/buddyinfo snapshot taken before the run, the order-9 and order-10 columns of the Normal zone were zero and only the DMA zone had eight free 2 MiB blocks (the fourteen huge pages that arrived came from there). Same program, plain mode, 512 MiB:

enabled=[always]  defrag=[madvise]
  vm: thp_fault_alloc 122334  thp_fault_fallback 513786
mode=plain  touch: 360.7 ms  minor fault=123920  AnonHugePages=28672 kB
  vm: thp_fault_alloc 122348  thp_fault_fallback 514028

It says always; instead of 256 huge pages, 14 arrived and 242 requests fell back to small pages. Over the VM's lifetime the counters are even clearer: 122,334 successful huge pages, 513,786 fallbacks; four out of every five requests got small pages. Nobody saw an error, dmesg has no line, enabled still reads always. The reason is defrag=madvise: since this mapping carries no hg flag, the kernel gives up on compaction when it cannot find a contiguous block and drops to 4 KiB.

After a manual echo 1 > /proc/sys/vm/compact_memory, 19 huge pages; after defrag=always, 73 (146 MiB) and minor fault 93,771. Direct compaction helped, but on this fragmented machine it could not cover even half of 512 MiB. Timing in the VM is noisy (the same plain run gave 245 and 360 ms on two occasions; the few dozen extra faults between runs are program overhead and small faults taken during compaction), so I trust only the huge-page counters. Two more counters matter: thp_fault_fallback_charge counts requests that found a contiguous block but were refused by the cgroup memory limit; in a container close to memory.max, THP will not arrive even if the file says always. And since sysfs is read-only in an ordinary container, I could only run this experiment with --privileged; from inside a container your only levers are madvise and prctl. The lesson: enabled=always is a permission, not a promise. What you actually get is told by the ratio of thp_fault_alloc to thp_fault_fallback.

The 64 KiB middle road: mTHP

6.8 added eight subdirectories to the same directory: hugepages-16kB … hugepages-2048kB. The kernel 6.8 release notes announced it as "multi-size THP for anonymous memory: runtime opt-in feature to transparent hugepages which improves performance by allocating larger chunks of memory during anonymous page faults". Each size has its own enabled file with four values: always, madvise, never and inherit. On VPS3:

hugepages-2048kB: always [inherit] madvise never
hugepages-64kB:   always inherit madvise [never]
hugepages-16kB … hugepages-1024kB: [never]

The default is backwards-compatible by design: only the PMD size inherits the top-level setting, every other size is off. The directories exist; none of them is on. In the Docker VM I set the top level to madvise and wrote always only into hugepages-64kB/enabled; same fragmented machine, same plain 512 MiB:

mode=plain  touch: 14.0 ms  minor fault=9379  AnonHugePages=0 kB  Rss=525620 kB
  64kB stats/anon_fault_alloc +8113   anon_fault_fallback +79

On the machine where asking for 2 MiB failed 242 times, asking for 64 KiB succeeded 8113 times and fell back 79 times. Page faults went from 131,072 to 9379, the touch took 14 ms in this run. Finding a contiguous 64 KiB is far easier than finding a contiguous 2 MiB, and bloat comes in 64 KiB units rather than 2 MiB. This is an arm64 machine; the original patch series targeted exactly this, arm64's "contiguous bit" letting 64 KiB of contiguous pages collapse into a single TLB entry. The contpte support that opens that folding to user mappings landed one release later, in 6.9; on 6.8 you can enable mTHP but get no arm64 TLB benefit. x86 has no such hardware folding at all; the gain there, as the series description lists it, is fewer page faults, batched PTE/rmap manipulation and a shorter LRU list.

Two footnotes. AnonHugePages does not count those 8113 pages; the documentation admits it "only applies to traditional PMD-sized THP for historical reasons and should have been called AnonHugePmdMapped". Per-size counters (hugepages-64kB/stats/anon_fault_alloc) arrived in 6.10, and that directory does not exist on VPS3's 6.8; 6.12 added the thp_anon= boot parameter for setting these at boot. So on 6.8 you can enable mTHP but you cannot see how much of it you get; do not enable in production a setting you cannot measure.

Questions to ask your own server

• Read enabled, defrag and khugepaged/max_ptes_none together; one word is not a decision.
• Look at thp_fault_alloc / (thp_fault_alloc + thp_fault_fallback). If it says always and you get 20 percent, fragmentation is talking, not the setting.
• If thp_deferred_split_page grows while thp_split_page stays at zero, "AnonHugePages dropped, RSS didn't" is normal; the memory is waiting for the shrinker.
• Find the process using huge pages with grep -c hg /proc/PID/smaps; on VPS3 the requester was the allocator inside a Rust module.
• Protecting sparsely touched large mappings with MADV_NOHUGEPAGE is a cheaper and more local decision than switching always off.
• On a 6.8+ kernel look at hugepages-64kB/enabled; if everything is never, mTHP "exists" but is not "on". Before 6.10 you cannot measure it even if you enable it.
• Before trusting an application's documentation, look at what the process does: Redis opts out with prctl, mimalloc opts in, MongoDB 8 wants the system on always.

Closing

The three numbers from the opening now read: 11,655 huge pages were handed out (the ones left today belong to an allocator's madvise call; for the whole eight days that is an inference, and a thousand-odd of them were added by my own lab), 3455 of them were partially released and opened into PTEs, and 4 MiB remain. The global setting is madvise; nobody on the server turned THP on, and THP was there anyway. Thinking of THP as "on/off" hides the fact that an allocator, a database and a kernel thread manage the same memory with three different ideas. The settings file holds a single word; the decision is made by whoever reads that word, whoever ignores it, and whether any contiguous 2 MiB is left. The MGLRU post, where page reclaim changed in the same quiet style, and the KSM post, where I explained why KSM has to split huge pages, are this post's two neighbours.

Official Sources

• Linux kernel — Transparent Hugepage Support (admin-guide/mm/transhuge)
• git.kernel.org — 3485b88390b0 "mm: thp: introduce multi-size THP sysfs interface" (6.8)
• git.kernel.org — ec33687c6749 "mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters" (6.10)
• git.kernel.org — dd4d30d1cdbe "mm: override mTHP 'enabled' defaults at kernel cmdline" (6.12, thp_anon=)
• redis/redis — redis.conf (disable-thp) and src/server.c (PR_SET_THP_DISABLE)
• microsoft/mimalloc — src/prim/unix/prim.c (allow_thp → MADV_HUGEPAGE)
• PostgreSQL — Resource Consumption: huge_pages (hugetlb, not THP)
• redhat-performance/tuned — throughput-performance profile (transparent_hugepages only never, for ThunderX)