run=1 Wasn't Enough: KSM Needs Two Separate Yeses

Process number 136 on my server has been running for four days and thirteen hours and has used a grand total of zero seconds of CPU in that time. Its name is ksmd, the background thread of Kernel Samepage Merging; its job is to find memory pages with identical contents and collapse them into one copy. Nearly every "save RAM" guide carries the one-line recipe: echo 1 > /sys/kernel/mm/ksm/run. I wrote that line this morning, and for two minutes pages_scanned stayed at zero. The daemon was running. It had nothing to scan.

This post is about why that zero happened and about the three experiments I ran afterwards. KSM has not one switch but two, and nothing happens until both are flipped. What happens once they are is a long way from the "free memory" line in the guides: the same mechanism that took a gigabyte down below ten megabytes ate half a CPU core under a different setting. Every number here comes from this box: Ubuntu 24.04, kernel 6.8.0-139, systemd 255, 18 vCPUs, 94 GiB of RAM, a KVM guest.

Starting point: run=0 and nobody raising a hand

First I recorded the state as it was:

$ grep -H . /sys/kernel/mm/ksm/* | sed 's|/sys/kernel/mm/ksm/||'
run:0
pages_to_scan:100
sleep_millisecs:20
smart_scan:1
advisor_mode:[none] scan-time
use_zero_pages:0
max_page_sharing:256
pages_scanned:0
pages_shared:0
pages_sharing:0
general_profit:0
$ ps -o pid,etimes,time,comm -p $(pgrep ksmd)
    PID ELAPSED     TIME COMMAND
    136  393244 00:00:00 ksmd

run:0 was what I expected; it is the kernel's default, and Ubuntu leaves it alone. The real question was who would get scanned once I turned it on. KSM only looks at address ranges that have volunteered. When a range volunteers, the VMA gets the VM_MERGEABLE flag, which shows up as mg on the VmFlags line of /proc/<pid>/smaps. I swept every process:

$ grep -l "VmFlags:.* mg" /proc/[0-9]*/smaps 2>/dev/null | wc -l
0

Zero. This machine has dozens of processes holding close to 10 GiB of anonymous memory (AnonPages: 10664376 kB): Java, ClickHouse, MySQL, a few Node instances, Go services. Not one of them had said "you may merge my pages". I will admit that my first attempt was grep -c mg, which matched three processes and made me briefly happy; what it had matched was the string img in the path of a mapped file. Pinning the pattern to the VmFlags: line brought it down to zero.

Then I turned the daemon on and waited:

$ echo 1 | sudo tee /sys/kernel/mm/ksm/run
$ sleep 120; cat /sys/kernel/mm/ksm/pages_scanned
0

This is the part the guides leave out. run=1 wakes ksmd, but if ksmd's list is empty there is no difference between awake and asleep. The kernel documentation says it plainly: KSM "only operates on those areas of address space which an application has advised to be likely candidates for merging". The first yes belongs to the administrator, the second to the process; without the second, the first is nothing more than a wake-up alarm.

Three ways to say the second yes

On the process side there are three routes, and all three light the same flag.

The classic route is madvise(addr, len, MADV_MERGEABLE): the application calls it in its own code, for a range it chose. QEMU does this for guest memory (mem-merge=on is the default; it calls qemu_madvise(..., QEMU_MADV_MERGEABLE)); KSM was written for KVM in the first place, and in the documentation's words it was known as "Kernel Shared Memory" back then.

The second route arrived with kernel 6.4: prctl(PR_SET_MEMORY_MERGE, 1). One call makes every compatible VMA of the process a candidate, and mappings created later are added automatically. The branch in kernel/sys.c has no privilege check at all: it takes mmap_write_lock, calls ksm_enable_merge_any(), which writes the MMF_VM_MERGE_ANY flag into the mm_struct and walks the existing mappings with ksm_add_vmas(). I tried it as an ordinary user and got 0. Because the flag is part of MMF_INIT_MASK, it survives fork() into the child and, through mm_init(), into the fresh address space after execve(); a shell that turns it on once makes every program it launches a candidate. One footnote: the execve side was half-finished in vanilla 6.8.0; the flag carried over and mg showed up, but because no mm_slot was created for ksmd to walk, the process was never scanned. The fix, 3a9e567ca45f "mm/ksm: fix ksm exec support for prctl", landed in 6.10 and was backported to 6.8.12; Ubuntu's 6.8.0-139 includes it, and the fact that merging happened after exec in the Node experiment below is the proof.

The third route is built on that inheritance: since systemd 254 you write MemoryKSM=yes in the unit file, exec-invoke.c calls prctl_safe(PR_SET_MEMORY_MERGE, ...) in the child, and the service is born with the flag after exec. For a binary whose source you cannot change it is the only practical way; I will start four Node processes like that shortly.

The one thing you cannot do is tell a running process from the outside that it is now a candidate. For remote processes process_madvise(2) accepts only MADV_COLD, MADV_PAGEOUT, MADV_WILLNEED and MADV_COLLAPSE; the process_madvise_behavior_valid() list in mm/madvise.c ends there. If I want the memory-hungry Java on this box inside KSM, it has to be restarted.

Experiment 1: one gigabyte, four patterns

To see the mechanism in its cleanest form I wrote a small Python program: it allocates 1 GiB of anonymous memory, fills every 4 KiB page with one of four patterns (the bytes A, B, C, D), calls prctl(PR_SET_MEMORY_MERGE, 1) and waits. On SIGUSR1 it writes one byte into every page. When the process started, 57 of its 61 VMAs carried the mg flag, including the file-backed ones, because pages written in a private file mapping become anonymous and anonymous pages are the only thing KSM cares about.

I wrote run=1 at 10:48:37 and read the counters every ten seconds:

t=10s  scanned=44800   shared=0    sharing=0       volatile=44900   full=0 profit=-2873600
t=50s  scanned=224800  shared=0    sharing=0       volatile=225100  full=0 profit=-14406400
t=60s  scanned=270981  shared=29   sharing=7056    volatile=255481  full=1 profit=13263808
t=100s scanned=446481  shared=717  sharing=181968  volatile=79981   full=1 profit=728490944

Nothing merged in the first pass and general_profit went negative: 14 MB of loss. This is the design: when cmp_and_merge_page() sees a page for the first time it only computes and stores its checksum; if oldchecksum != checksum it returns without putting the page into any tree. A page only becomes a candidate when it is caught with the same checksum on the second pass. pages_volatile is the counter for that in-between state; it holds nothing of its own, it is derived as rmap_items - shared - sharing - unshared. The loss item is the 64-byte rmap_item allocated for every page scanned: 225 thousand pages times 64 bytes is exactly that 14 MB.

The default speed can be read off the same numbers: pages_to_scan=100, sleep_millisecs=20, so five thousand pages a second, 19.5 MiB. With 263 thousand candidates one pass took a little under a minute. Had all 10 GiB of anonymous memory on this machine been candidates, a single pass would have taken nine minutes; advisor_mode=scan-time, new in 6.8, adjusts that number against a target scan time by itself. I left the default alone.

When the second pass finished the picture was this:

pages_shared:1025   pages_sharing:261137   pages_unshared:1119
stable_node_chains:4   stable_node_dups:1024   general_profit:1052767168
$ cat /proc/4146443/ksm_stat
ksm_rmap_items 263281
ksm_merging_pages 262162
ksm_process_profit 1056965568
$ grep -E "^(Rss|Pss|Shared_Dirty|Private_Dirty):" /proc/4146443/smaps_rollup
Rss:             1060020 kB
Pss:                9837 kB
Shared_Dirty:    1048648 kB
Private_Dirty:      4476 kB

Four patterns give stable_node_chains:4, yet pages_shared:1025. The reason is max_page_sharing=256: a single KSM page may carry at most 256 mappings, and 262144 pages divided by 256 is 1024 copies, plus a handful of pages the Python interpreter merged within itself. The profit is 1,052,767,168 bytes, 1004 MiB. And Rss did not move. A KSM page is still an anonymous page mapped into the process; you cannot tell whether KSM did anything by watching RSS in top. The right column is Pss: the 1 GiB process dropped to 9837 kB, under ten megabytes. Since 6.6 smaps also has a KSM: line that counts KSM pages directly.

Then I sent kill -USR1. Three seconds later:

pages_shared:11   pages_sharing:2057   pages_volatile:260096   general_profit:-8424512
Private_Dirty:   1053052 kB
MemFree: 75033272 kB -> 73878492 kB

Writing a single byte into each page undid all of the merging; the kernel produced a fresh page on every write through copy-on-write, MemFree fell by 1.1 GiB in three seconds and the profit went negative again. Because my program pulled the same byte of every page to the same value, the pages became identical again and pages_sharing had climbed back to 4997 sixty seconds later; in a real application, instead of waiting for that second chance, the question to ask is whether the pages being merged are really not changing. If they are, KSM does not save you memory; it runs a slow-motion memcpy scheme.

Experiment 2: the same program four times, two percent

The guides' second promise is "if you run many copies of the same application, the gain is large". I tested that with a real runtime. heap.js builds a million small objects ({id, name, tags, score}), loads three modules and waits while keeping the objects alive. I launched four copies with MemoryKSM=yes:

$ for i in 1 2 3 4; do sudo systemd-run --unit ksmlab-node-$i -p MemoryKSM=yes \
    /usr/bin/node /tmp/ksm-lab/heap.js; done
node-1 pid=8797 rss=222592 kB   mg_vmas=697
node-2 pid=8817 rss=231100 kB   mg_vmas=718

In my first attempt I had not referenced the array inside setInterval, V8 garbage-collected it and RSS fell to 57 MB; I wanted that correction to stay in the post, because when you are "measuring memory" the first job is to make sure the thing you are measuring is still there. On the second attempt each of the four processes held around 38 thousand candidate pages, and after two full passes the result was:

pages_sharing:3135   pages_unshared:149134
node-1  rss=198512 kB  pss=153377 kB  ksm_rmap_items 38317  ksm_merging_pages 805
node-2  rss=200704 kB  pss=155528 kB  ksm_rmap_items 38862  ksm_merging_pages 813
node-3  rss=197428 kB  pss=152227 kB  ksm_rmap_items 38041  ksm_merging_pages 819
node-4  rss=194556 kB  pss=149335 kB  ksm_rmap_items 37323  ksm_merging_pages 824

3.3 MB per process. Same binary, same script, the same million objects; two percent of the pages merged and 149 thousand were stamped unique. The reason is obvious once you see it: V8's heap is full of pointers and the addresses differ in every process. An object that is "the same" as content is four different pages as bytes.

Suspecting ASLR, I started the four copies with setarch x86_64 -R, kernel randomization off. The [heap] range landed at the same address in all four (06b5e000), but merging stayed between 1008 and 1071 pages. V8's large regions were still in different places: 3e18d25c0000 in one, 3458756c0000 in another. V8 generates its own random hint through OS::GetRandomMmapAddr(); turning off the kernel's ASLR is none of its concern. To me this is the real lesson of the experiment: the distance between "the same program" and "the same page" is decided by the runtime, not by you. This is why virtual machines, KSM's classic home, are special: guests booted from the same image repeat the same binaries, libraries and page cache, and to the host all of it is anonymous memory.

Experiment 3: zeros, two separate bills

The third scenario is genuinely common: memory that has been touched but is empty. A JVM heap pre-allocated with -Xms, AlwaysPreTouch, large zeroed buffers. I allocated 512 MiB and wrote a zero into every page; the pages are now real, VmRSS 535720 kB. KSM has a dedicated knob for this: use_zero_pages. Off, zero pages merge with each other; on, they are mapped to the kernel's single zero page. I measured both.

use_zero_pages=1:

ksm_zero_pages 131088   ksm_merging_pages 0   ksm_process_profit 536864832
Rss:   11364 kB   Pss:  5719 kB   KSM: 0 kB
ksmd ticks/30s=1438 -> cpu%=47   (1438/3000, close to 48 percent)

This time RSS really dropped, from 535720 kB to 11364 kB; the kernel zero page is not counted in RSS. Profit equivalent to 512 MiB. But over a 30-second window ksmd consumed close to 48 percent of a core. Once the test process exited, the same measurement gave zero percent, so the load was its doing.

use_zero_pages=0:

pages_shared:513   pages_sharing:130575   stable_node_dups:513
VmRSS:  535720 kB   Pss:  7788 kB   KSM: 524352 kB
ksmd cpu%=1

Same memory, a profit figure close to the same, ksmd at one percent. RSS stays put this time, because the 131072 pages collapsed into 513 real KSM pages in chains of 256, and those are still anonymous pages.

The gap between 48 and 1 percent is in the source. scan_get_next_rmap_item() walks a VMA's pages one by one with follow_page(); if a page is not PageAnon it jumps to the next_page label and carries on without deducting from the quota. The kernel zero page is not anonymous. So the 131 thousand mappings pointing at the zero page are walked on every pass, but none of them is deducted from the pages_to_scan quota, and Python's own 1119 pages are skipped on most passes because of smart_scan. ksm_do_scan() returns early when it reaches the end of a pass and the thread sleeps for 20 ms, which means every wake-up walks a full pass of 132 thousand follow_page calls; that is why full_scans hit 1350 in 30 seconds, and that is the 48 percent. In the master branch this loop has been rewritten around walk_page_range_vma(), but the counting logic is the same: it only returns when an anonymous page is found. If you are going to turn on use_zero_pages=1 for a workload rich in zero pages, do not skip looking at ksmd's CPU; that is what I measured on 6.8. The JVM example carries a hidden bill as well: KSM only works with 4 KiB pages, and when try_to_merge_one_page() meets a transparent huge page it calls split_huge_page(). Handing a pre-touched large heap to KSM means stripping that heap of its huge pages.

The price tag and the decision

Adding up the bill from the three experiments:

• ksmd's fixed cost at default settings is 5000 pages per second. Over the nineteen-minute experiment it consumed 2 minutes 32 seconds of CPU in total, most of it in the zero-page rounds. The bigger the candidate memory, the longer a pass and the later the profit; advisor_mode=scan-time exists in 6.8 for that, with advisor_max_cpu defaulting to 70 percent.
• The memory cost is a 64-byte rmap_item per page. The documentation's threshold is explicit: if ksm_rmap_items / ksm_merging_pages exceeds 64, the profit is zero or negative. In the Node experiment the ratio was 38317 / 805 ≈ 48; under the threshold, but the gain was 3 MB per process. Gaining something and gaining something worth having are not the same.
• The write cost is copy-on-write. If a merged page changes, a copy appears; writing one byte into 262 thousand pages took 1.1 GiB out of MemFree. On a system under memory pressure that is an allocation wave arriving at the worst possible moment.
• The security cost I did not measure; I leave the claim to the sources. systemd's MemoryKSM= documentation says it "should only be enabled for jobs that share the same security domain". Proxmox ships KSM enabled by default, but its wiki page states, without citing a paper, that research has shown "it is possible to infer information about a running VM via a second VM on the same host" and offers qm set <vmid> --allow-ksm 0. Page merging is a side channel that reads which pages are shared from write latency; Flip Feng Shui combined it with Rowhammer to get the victim's page merged with the attacker's and then flip bits in it.

For my own server the decision is easy: KSM stays off. There are no candidate processes, the only application running in multiple copies is Node and the experiment said two percent, and by MemAvailable 85 percent of the RAM is free. This machine is a KVM guest; there is no practical way to see from inside whether the host is merging my pages with other customers'; the known way of doing that is precisely the side channel above. I am at peace with a ksmd at zero CPU for four days.

When you ask the same question of your own setup, I would take it in this order:

1. If grep -l "VmFlags:.* mg" /proc/[0-9]*/smaps returns zero, writing run=1 will have no effect; decide first who is going to volunteer.
2. Restart the volunteering service with MemoryKSM=yes; the flag cannot be attached to a running process from the outside.
3. Read success from Pss, the KSM: line in smaps and /proc/<pid>/ksm_stat, not from RSS in top; ksm_process_profit is the gain, ksm_rmap_items/ksm_merging_pages the efficiency.
4. Expect the first pass to come out negative; do not draw conclusions before you have seen two full passes (full_scans).
5. Measure ksmd's CPU through /proc/<pid>/stat over a 30-second window, especially if you turned on use_zero_pages=1.
6. If different trust domains share the host, the answer is independent of any measurement: off.

Closing

In the kernel documentation's own words, KSM "can be useful to any application which generates many instances of the same data". Until this morning I was underrating the condition in that sentence. The same program does not generate the same data; V8 laid a million identical objects out in four different address arrangements, and KSM only looks at bytes. A gigabyte of pattern came down to ten megabytes; four Node processes gave three megabytes each. The distance between those two results measures how identical your data really is, not KSM.

To finish the experiment I wrote run=2, which separated every page, then run=0; all the counters returned to zero and use_zero_pages is off again. ksmd is back asleep. One line from the source stuck with me: on a kernel built without sysfs, ksm_run = KSM_RUN_MERGE; /* no way for user to start it */. There the first key is permanently turned; the second is still in the process's hands, even there.

For the kernel source lines I used the v6.8 tag: kernel/sys.c (PR_SET_MEMORY_MERGE), mm/ksm.c (ksm_enable_merge_any, scan_get_next_rmap_item, cmp_and_merge_page), kernel/fork.c (mm_init/mmf_init_flags). For V8's own address randomization, platform-posix.cc. For Ubuntu's ksmtuned package, packages.ubuntu.com. For the other face of memory reclaim, the MGLRU post; for overcommit at the virtualization layer, the hardware overcommit post.

Official Sources

• Kernel Samepage Merging — kernel admin guide
• /proc documentation: the smaps KSM line and VmFlags
• kernel/sys.c — PR_SET_MEMORY_MERGE
• mm/ksm.c — the ksmd scan loop and profit counters
• mm/madvise.c — process_madvise permitted behaviours
• systemd.exec — MemoryKSM=
• systemd exec-invoke.c — the prctl(PR_SET_MEMORY_MERGE) call
• Proxmox VE wiki — Kernel Samepage Merging