Mustafa ERBAY

Posted on Oct 16 Originally published at mustafaerbay.com.tr

I Called MADV_FREE on 512 MiB and No Counter Came Down

#linux #kernel #bellekyonetimi #madvise

There are two ways to hand memory back, and they look like the same job. madvise(MADV_DONTNEED) says "stop looking at these pages"; madvise(MADV_FREE) says "I no longer need these pages". The manual descriptions are nearly siblings. Run them, though, and one returns 512 MiB instantly, while with the other not a single counter the operating system shows you comes down.

I got curious because of a familiar scene. A process frees memory, you wait for RSS to fall on the dashboard, and it does not. The first thought is "we have a leak", the second is "the counter is lying". There is a third possibility, and this piece is about it: the memory really is free and the counter really is correct, both at the same time.

Two things along the way surprised me. I could read back every byte I thought I had released. And a single 4 KiB write dropped 2 MiB off the lazy list.

The rig

The measurement runs inside the linuxkit virtual machine behind Docker Desktop on my own Mac: kernel 6.10.14-linuxkit, aarch64, cgroup v2, 4.8 GiB allocated to the VM. The container runs with --memory 1g --memory-swap 1g, so memory.max is 1,073,741,824 bytes and there is no swap. Turning swap off mattered: with swap present, "reclaim" and "pushed to disk" blur together, and I wanted to tell them apart. Base image alpine:3.20, compiler gcc -O1. The VM's THP setting is [always] madvise never.

The main test program maps a 512 MiB private anonymous region, writes 0xA5 into every byte, then in order: calls madvise(), writes 0x5A into a single page, applies memory pressure with a second region, and finally scans the region end to end counting the first byte of every page. Two smaller programs sit beside it: one compares a single page byte by byte, the other only reads the LazyFree counter repeatedly. I will say below which number comes from which.

There is a measurement trap here, and I fell into it at first: reading a reclaimed page corrupts the measurement. The read raises a page fault, the kernel installs a page, and you have polluted the question "how much survived?" with your own read. So I counted resident pages with mincore(2) first — mincore reads the map without touching the pages — and kept the data scan for the very end.

One more thing I did not assume but tested: does /proc/<pid>/smaps_rollup actually report the LazyFree field? A small probe that maps 64 MiB and calls MADV_FREE made smaps_rollup say exactly LazyFree: 65536 kB. Had my probe been blind, half of this article would be fiction.

First measurement: no counter came down

First run, transparent huge pages off (MADV_NOHUGEPAGE), region 512 MiB:

Step	Region Rss	LazyFree	VmRSS	cgroup `memory.current`
Touched (`0xA5`)	524,288 kB	0 kB	524,744 kB	684,368 kB
After `MADV_FREE`	524,288 kB	524,272 kB	524,872 kB	684,476 kB

Not one kilobyte came down. mincore agrees: 131,072 of 131,072 pages resident before the call, and 131,072 of 131,072 after. RSS is not lying — the pages really are in physical memory. The cgroup counter does not move either; as far as the container's 1 GiB quota is concerned, those 512 MiB are still spent.

The only line that changed is LazyFree: from 0 to 524,272 kB. The kernel's own documentation describes this field in one sentence — the amount of memory marked by madvise(MADV_FREE) — and adds an honest warning: the printed value may be lower than the real value because of optimisations in the current implementation.

Taking that warning seriously, I stopped reading the counter once. A separate program took 11 consecutive readings from the same VMA: the value shows 524,272 kB on the first readings and settles at 524,288 kB, with a median of 524,288. So the missing 16 kB is not a permanent deviation but a lag in the first readings right after the call. This matters because the difference in the next section is on the order of 4 kB — smaller than the noise of a single reading. Had I based that number on one reading, I would have been measuring noise.

Another detail: this field does not exist in /proc/meminfo. There is no single system-wide number telling you how much lazily freed memory exists; you have to read smaps or smaps_rollup per process. Knowing this while building monitoring saves you from hunting for a field that is not there.

Second measurement: the data I released was still there

Right after the MADV_FREE call I read the first page of the region — with the small program that compares all 4096 bytes one by one. The page was 0xA5 from start to finish. The 512 MiB I thought I had released was sitting there, byte for byte.

This is not a fault, it is the design. The manual says it plainly: the kernel may free these pages, but freeing can be delayed until memory pressure occurs; if no write follows, the kernel can free the pages at any time. "At any time" means both "right now" and "never". In the uncertainty window between those, the old data stays readable.

The practical consequence is a security one, and it is often missed: MADV_FREE is not an erase. When you free() a buffer holding a password, a session key or a private key, and the allocator underneath uses MADV_FREE, those bytes remain readable within the same process — and how long they remain depends on memory pressure, that is, on what other processes do. You have to zero a secret-bearing buffer yourself before releasing it; explicit_bzero(3) exists precisely for this (C23's memset_explicit does the same job). "I freed it anyway" is not a guarantee.

When pressure arrives: who left, who stayed

The real question is what happens when pressure arrives. In the same container I opened a second region, touched 700 MiB and let it go. The quota is 1 GiB and 512 + 700 is being asked for; the kernel has to find room somewhere.

Step	Region Rss	LazyFree	`memory.current`
Before pressure	524,288 kB	524,284 kB	684,320 kB
After pressure	181,648 kB	181,644 kB	331,732 kB

mincore: 45,412 of 131,072 pages stayed resident, 85,660 pages were reclaimed — 334 MiB. All three runs are close: 334, 334, 335 MiB.

Then I ran the full scan. In the first run the table reads: 45,411 pages still 0xA5, 85,660 pages zero-filled, and exactly 1 page 0x5A. That one page is the page I wrote to before the pressure. The manual's sentence "the free operation will be cancelled if the caller writes into the page" showed itself as a single survivor in a set of 131,072.

I went looking for where that decision is made in the kernel, because my first guess was wrong. mm/madvise.c only does the marking: inside madvise_free_pte_range, a dirty or swap-cached folio goes through folio_clear_dirty(folio), and then all of them are tagged with folio_mark_lazyfree(folio). The cancellation happens on the reclaim path, in mm/rmap.c:

if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) {
    /*
     * redirtied either using the page table or a previously
     * obtained GUP reference.
     */
    folio_set_swapbacked(folio);
    return false;
}

On false the caller restores the PTEs and the page stays mapped. So "a write cancels the free" does not happen at write time; it happens when the kernel comes to take the page and finds it dirty, and puts it back.

One more result I had not expected. After the full scan — that is, after reading all 85,660 zeroed pages — the region's Rss stayed at 181,648 kB. Not a byte more. When you touch a reclaimed private anonymous page for reading, the kernel does not allocate a new page, it maps the shared zero page, and that is not counted in RSS. "I read my data back" is not the same as "I took my memory back"; writing would have been.

I wrote 4 KiB and 2 MiB left the lazy list

I also ran it with transparent huge pages on, and saw something odd: writing to a single 4 KiB page dropped LazyFree by roughly 2 MiB rather than 4 kB. Rather than guess, I set up a control experiment. Same program, the only difference being MADV_NOHUGEPAGE applied to the region; both values below are medians of the 11-reading series described above, so they come from the same reading chain:

THP	`LazyFree` (before the write)	`LazyFree` (after a 4 KiB write)	Difference
off	524,288 kB	524,284 kB	4 kB
on	524,288 kB	522,240 kB	2,048 kB

All three runs produced an identical difference: 4 kB with THP off, 2,048 kB with it on. The post-write series had zero spread, so these numbers are not lost inside the noise.

The cause is transparent huge pages. The kernel holds this region in 2 MiB folios and tracks dirtiness per folio. Touch four kilobytes and the whole two megabytes is dirtied again and leaves the lazy list. Note that the "512 times" ratio is not a property of the mechanism: it depends entirely on the size of the write, and writing one byte would have made it millions. The accurate statement is this — however small the write, the entire folio drops off the lazy list.

I have written about the allocation-side cost of huge pages on this blog before, in the measurement where I wrote 512 bytes and paid 1 GiB. There the memory allocated on first touch ballooned; here the reclaimability of already-released memory disappears. The accounting unit writing the bill is the folio in both.

A scope note: in the runs I call "THP on", not all of the region was huge pages. AnonHugePages read 296,960 / 313,344 / 317,440 kB across the three runs — only 57–60% of the region; the rest stayed as 4 KiB pages.

What got reclaimed was exactly the part that was not a huge page

That scope note turned out to be the result itself. Putting the post-pressure Rss next to the starting AnonHugePages for the three THP-on runs:

Run	Starting `AnonHugePages`	Rss after pressure	Difference
1	296,960 kB	296,960 kB	0
2	313,344 kB	317,384 kB	4,040 kB
3	317,440 kB	317,440 kB	0

In two of the three runs the remaining memory equals the starting huge-page amount to the kilobyte. The kernel reclaimed the small-page portion entirely and left the huge-page portion untouched. The reclaim figures are the arithmetic of that: 524,288 − 296,960 = 227,328 kB (222 MiB) and 524,288 − 317,440 = 206,848 kB (202 MiB) — matching the 222 / 202 / 202 MiB measured. For comparison, with THP off the same pressure reclaimed 334 / 334 / 335 MiB.

I owe an honest note: in the second run the difference is not zero but 4,040 kB. And after pressure the LazyFree and AnonHugePages counters behaved inconsistently from run to run — in one, LazyFree went to zero while AnonHugePages stayed put; in another, both went to zero. This may have to do with deferred splitting, but I did not measure it. I am leaving the observation, not an explanation.

Same pressure, different ending

Up to here the difference may look like bookkeeping. When I raised the pressure from 700 MiB to 800 MiB, it stopped being bookkeeping:

THP	Result (3 runs)	Reclaimed
on	OOM-kill (exit 137) — 3/3	—
off	survived — 3/3	440 / 439 / 440 MiB

Three out of three. Same program, same quota, same pressure; the only variable is whether MADV_NOHUGEPAGE was applied to the region. The arithmetic of the previous section explains the outcome exactly: because the huge-page portion cannot be reclaimed, roughly 310 MiB stays locked, 800 MiB is asked for on top, and the total exceeds the 1 GiB quota.

I am not offering this as a general law — one kernel version, one architecture, one swapless quota. But I watched the combination be real across three runs: a hard memory quota, no swap, THP on, and an allocator using MADV_FREE. If those four meet in the same container, "the memory is already free" will not save you.

Three paths side by side

I ran the same program with three different behaviours; 700 MiB of pressure, THP off. The counter values come from the same step in all three — after the madvise call and the single-page write, before the pressure:

Behaviour	Region Rss	`memory.current`	Under pressure	Data
no call at all	524,288 kB	684,500 kB	OOM-kill (137)	—
`MADV_FREE`	524,288 kB	684,320 kB	survived	45,411 pages readable
`MADV_DONTNEED`	4 kB	160,192 kB	survived	131,071 pages zeroed

The 4 kB on the MADV_DONTNEED row is that one page I wrote; the other 512 MiB went the moment the call returned, and the cgroup counter fell from 684 MiB to 160 MiB. MADV_FREE moves no counter at all, the data stays, and the bill comes due under pressure. Not calling anything — at least under a swapless quota — kills the process.

Why isn't MADV_DONTNEED automatically the better option? Because you genuinely gave the pages back, so next time you pay for the page fault and the zeroing again. That is the entire reason MADV_FREE exists: if pressure never comes, you never pay that cost. It is a bet; when it pays off it is free, and when it does not you get the table above.

Which line do you put on the dashboard?

memory.stat has no field saying "this much lazily freed memory exists right now". It does have two event counters, and they do the job. Measured: after the MADV_FREE run pglazyfree reads exactly 131,072 — the number of pages I marked — and pglazyfreed reads 86,975, the ones actually reclaimed under pressure. Running the same workload without calling madvise left both counters unchanged, so they are specific to this mechanism.

The practical reading: if pglazyfree climbs while pglazyfreed stays flat, you have a growing pool of the "third state" — memory visible in RSS, genuinely free, not yet reclaimed. If both climb together, the system is already under pressure and the mechanism is working.

There is also a way to drain that pool without waiting for the OOM killer. In cgroup v2 a single line written to the target group triggers proactive reclaim:

echo "300M" > /sys/fs/cgroup/<group>/memory.reclaim

The documentation attaches two warnings: the kernel may reclaim more or less than asked, and if it reclaims less it returns -EAGAIN. That line is the cheapest antidote to the OOM table above.

As a scope limit I should add: my rig only had memory.max. Most real containers also have memory.high, where per the documentation going over "never invokes the OOM killer" and the process is instead throttled under heavy reclaim pressure. In an environment using memory.high, the OOM table above does not transfer as is.

So who actually uses this?

A mechanism being real does not mean it is running on your system. I looked at the blog's own server: Ubuntu 24.04.4, kernel 6.8.0-147-generic, 751 processes alive at that moment. Number of processes with a LazyFree value above zero in the smaps_rollup scan: zero. I had already proved with the deliberate positive above that my probe was not blind, so this is a real negative.

The reason lies in ecosystem choices. The Go team changed the default in 1.16: on Linux the runtime now returns memory promptly with MADV_DONTNEED rather than lazily under memory pressure, and the release notes state the rationale openly — so that process-level statistics like RSS reflect the physical memory actually in use more accurately. Anyone who wants the old behaviour can take it with GODEBUG=madvdontneed=0, and the runtime documentation says in the same breath that the option is more efficient but makes RSS drop only when the OS is under memory pressure.

On the jemalloc side, the "lazy purge" capability hangs off the JEMALLOC_PURGE_MADVISE_FREE macro; without it, PAGES_CAN_PURGE_LAZY is not defined. jemalloc calls pages released this way "muzzy", and per its documentation muzzy decay is disabled by default — so in a default configuration those pages are not held for long. You can tune the behaviour through MALLOC_CONF with dirty_decay_ms and muzzy_decay_ms.

A final version note: this whole setup assumes a swapless system. The manual says that before Linux 4.12, on a swapless system, MADV_FREE pages were freed instantly regardless of memory pressure. On a fleet running older kernels the behaviour is the opposite of this article's.

What I did not measure

I did not examine which pages the kernel picks under pressure. I counted how many left; I did not work out the LRU ordering.
I did not measure on a system with swap. The tables here belong to a quota where memory-swap = memory.
I did not measure with memory.high; the scope note above rests on the documentation, not on my own measurement.
One kernel (6.10.14-linuxkit) and one architecture (aarch64). The current stable release is 7.2.11 and mainline is 7.3-rc7; I did not test the differences.
I did not find the cause of the counter inconsistency in the THP-on runs, nor of the 4,040 kB deviation in the second run.
I make no speed claims. I did not time anything; every number here is memory accounting.

Questions for your own system

If a process looks like it is "leaking", run grep LazyFree /proc/<pid>/smaps_rollup before you accuse it. A large number means you are seeing deferred return.
Do not look for this field in /proc/meminfo; it is not there. On the container side, watch the pglazyfree / pglazyfreed pair in memory.stat.
If you sized a container memory quota from observed RSS, you may have set the ceiling too high for an allocator that uses MADV_FREE. Measure the value under pressure before you decide.
If you have the trio of a hard quota, no swap and THP always on, the OOM table above is not theoretical. The cheapest check is to run your workload once with MADV_NOHUGEPAGE and compare.
In an emergency, writing to memory.reclaim drains the pool without waiting for the OOM killer.
Zero secret-bearing buffers with explicit_bzero before releasing them.
Instead of assuming what your allocator does, run the smaps_rollup scan across your own processes; it takes a minute.

Closing

The real lesson I took from this measurement is not about MADV_FREE. The modern kernel no longer gives a binary answer to "is this memory in use or not"; there is a third state in between — "the owner has given up on it but it has not been reclaimed yet" — and that state appears directly in none of the common counters. RSS counts it as in use, the cgroup counter charges it against the quota, and the application has already written it off in its own ledger. Each is correct in its own way and none is complete.

Most of our dashboards do not know about this third state, and we set quotas, raise alerts and declare leaks by looking at those dashboards. What a counter counts — and what it does not — deserves to be measured once before we make it the basis of a decision. I did it here for 512 MiB; you can hold your own counter to the same question.

DEV Community