DEV Community

Cover image for Diagnosing zram Device Initialization Races on Linux: A Direct Kernel Interface Approach
Meshack Bahati
Meshack Bahati

Posted on Edited on

Diagnosing zram Device Initialization Races on Linux: A Direct Kernel Interface Approach

#linux #ram

If you've ever had a Linux box with limited RAM grind to a halt under load, you know the feeling. Browser tabs pile up, an IDE starts indexing, a build kicks off, and suddenly the cursor lags, audio crackles, and the terminal output comes in stutters. The hardware isn't dying. The kernel is just thrashing to disk swap.

zram helps with that. It creates a compressed block device in RAM, so cold pages get compressed on the way out and decompressed on the way back in. On a machine that's short on memory, it can be the difference between a usable system and a slideshow.

The usual advice is to install zram-generator and let systemd handle it. I've had nothing but trouble with that thing. It races the kernel, fails to set the device size, and leaves you with a broken swap unit. So I do it manually now. It's not many steps, and you can actually see what's going on.

What zram actually does

zram gives you a compressed block device backed by RAM. You format it as swap, turn it on with a high priority, and the kernel sends cold pages there before touching your disk.

One thing that trips people up: the size you set is the maximum uncompressed data the device can hold. It is not the amount of RAM it will use. Every compressed page still lives in physical RAM. If you size it too aggressively, zram starts competing with your working set. That's how you end up with a system that swaps into its own memory and still feels starved.

A correct setup looks like this:

$ swapon --show
NAME       TYPE      SIZE USED PRIO
/dev/zram0 partition 3.9G   0B 32767
/dev/nvme0n1p2 partition 4G 0B -2

$ zramctl
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd        3.9G  1.2G  380M  410M       4 [SWAP]
Enter fullscreen mode Exit fullscreen mode

DATA is the uncompressed size currently stored. COMPR is the compressed size. TOTAL includes metadata. The ratio between DATA and COMPR tells you how well compression is working. DISKSIZE is the uncompressed budget you configured. PRIO 32767 means the kernel fills zram before it touches disk swap.

If you skip the priority, or make the device too big, you get one of two symptoms: disk swap activity while zram sits empty, or a working set that starves even though you supposedly have plenty of memory.

Why the zram-generator keeps breaking

The distribution path is zram-generator. It's a systemd generator. In theory it sets everything up for you. In practice it has a nasty race condition.

The same binary ships under different names depending on your distro. Fedora calls it zram-generator-defaults or just zram-generator without defaults. Debian calls it systemd-zram-generator. Arch has zram-generator in core and zram-generator-git in the AUR. Three names, three sets of packaging notes, three bug trackers. That alone makes tutorials annoying.

The real problem is what happens after installation. You get a config file at /etc/systemd/zram-generator.conf, reboot, and then:

swap-create@zram0.service: Failed with result 'exit-code'.
Enter fullscreen mode Exit fullscreen mode

And in the journal:

zram: Cannot change disksize for initialized device
Enter fullscreen mode Exit fullscreen mode

That's issue 7, open since January 2020. The generator assumes /dev/zram0 is ready for sizing the moment the device node appears. The kernel isn't ready yet. systemd writes to /sys/block/zram0/disksize before initialization finishes, and the write fails with Device or resource busy.

Issue 239 shows the same thing after systemctl soft-reboot, with repeated Failed to configure disk size into /sys/block/zram0/disksize. There's also an old mailing list report where the generator says Device zram0 not found even though ls /dev/zram0 clearly shows the node.

This isn't a one-distro problem. Debian bug 1026745 records systemd-zram-generator: Failed to start Create swap on /dev/zram0. Fedora CoreOS tracker 1844 shows the generator failing with terminated by signal ABRT on Rawhide after a version transition. The Arch Wiki documents the generator path but also documents two manual paths that don't hit the race.

The kernel module itself has been stable since 2014. The bug is in the systemd layer that tries to predict when the kernel is ready, and predicts wrong.

The manual approach has to respect the same ordering. modprobe returning doesn't mean /dev/zram0 is usable yet — on fast NVMe machines the node can lag behind the module load. The script polls for it instead of assuming.

What you need

No generator package. Just the basics:

  • A kernel with the zram module. Check with modinfo zram and zgrep ZRAM /proc/config.gz or grep ZRAM /boot/config-$(uname -r).
  • util-linux for mkswap and swapon. Check with mkswap --version.
  • kmod for modprobe, plus bash, coreutils, procps, and gawk.
  • sudo or root access for /sys and /etc.
  • /proc and /sys mounted.

If you're missing something:

# Debian/Ubuntu
sudo apt update && sudo apt install util-linux kmod procps gawk

# Fedora/RHEL
sudo dnf install util-linux kmod procps-ng gawk

# Arch
sudo pacman -S util-linux kmod procps gawk
Enter fullscreen mode Exit fullscreen mode

Doing it by hand

This is the whole process. You can run each step yourself and check the output as you go. The script I'll show later does the same thing in the same order.

1. Load the module 

sudo modprobe zram num_devices=1

# the /dev node can lag behind modprobe on fast machines, so wait for it
for _ in {1..5}; do [ -b /dev/zram0 ] && break; sleep 0.2; done
ls /dev/zram0
Enter fullscreen mode Exit fullscreen mode

That wait loop is the entire generator race, fixed in three lines. If the node isn't there after a second, something is actually wrong and you should stop rather than write to sysfs paths that don't exist yet.

If /dev/zram0 is already being used as swap, turn it off first:

grep -q /dev/zram0 /proc/swaps && sudo swapoff /dev/zram0
Enter fullscreen mode Exit fullscreen mode

If the device has an old size on it, reset it before you change anything. Sizing and algorithm writes fail on an initialized device:

if [ -f /sys/block/zram0/reset ]; then
  echo 1 | sudo tee /sys/block/zram0/reset >/dev/null || true
fi
Enter fullscreen mode Exit fullscreen mode

2. See what compression algorithms you have 

cat /sys/block/zram0/comp_algorithm
# example: lzo-rle lzo lz4 lz4hc [zstd] deflate 842
Enter fullscreen mode Exit fullscreen mode

The one in brackets is currently selected.

3. Pick zstd, with fallbacks

zstd compresses better than lz4. On a low-RAM machine, compression ratio matters more than raw CPU speed. A few extra cycles compressing a cold page is cheaper than thrashing that page to NVMe.

Not every kernel has zstd, though. So fall back to lz4, then lzo:

available=$(cat /sys/block/zram0/comp_algorithm)
for algo in zstd lz4 lzo; do
  if echo "$available" | grep -qw "$algo"; then
    echo "$algo" | sudo tee /sys/block/zram0/comp_algorithm >/dev/null
    break
  fi
done
cat /sys/block/zram0/comp_algorithm
Enter fullscreen mode Exit fullscreen mode

That fallback chain is why the same steps work on Arch, Debian, Ubuntu, Fedora, and RHEL without installing extra packages.

4. Size the device

On an 8GB desktop running KDE, a browser, and an IDE, I run zram at 100% of RAM — 8G of uncompressed budget holding roughly 2–3x that in effective data once zstd does its thing. The conservative starting point you'll see in most guides is 50%. Both are defensible; what isn't defensible is sizing blindly and never checking the compression ratio afterwards.

mem_kb=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
size_bytes=$(( mem_kb * 1024 * 100 / 100 ))
echo "$size_bytes" | sudo tee /sys/block/zram0/disksize >/dev/null
Enter fullscreen mode Exit fullscreen mode

Percentages get converted to raw bytes. Fixed sizes go through as-is — the kernel's sysfs parser accepts suffixes directly:

# 50 percent instead
mem_kb=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
echo $(( mem_kb * 1024 * 50 / 100 )) | sudo tee /sys/block/zram0/disksize >/dev/null

# or a fixed size, no conversion needed
echo 8G | sudo tee /sys/block/zram0/disksize >/dev/null
Enter fullscreen mode Exit fullscreen mode

Check it:

numfmt --to=iec-i --suffix=B "$size_bytes"
cat /sys/block/zram0/disksize
Enter fullscreen mode Exit fullscreen mode

One write path, no reformatting in between. If the value is malformed the kernel rejects the write and you see the error immediately instead of debugging a silently wrong size later.

5. Make swap and turn it on 

sudo mkswap -L zram-swap /dev/zram0 >/dev/null
sudo swapon --priority 32767 /dev/zram0
Enter fullscreen mode Exit fullscreen mode

That priority is the highest the kernel accepts. It tells the memory manager to fill zram first and only touch disk swap when zram is genuinely full. Without it, the kernel can happily page to NVMe while your zram device sits empty. That's the exact opposite of what you want, and it's a detail a lot of guides miss.

6. Tune the VM for compressed swap

The default vm.swappiness=60 is meant for disk-backed swap. With zram it causes a specific failure mode, and low-swappiness advice carried over from the spinning-disk era makes it worse.

Swappiness controls how eagerly the kernel moves cold pages to swap. A low value tells it to hold off until raw RAM is nearly exhausted. That logic assumes swap is slow and best avoided. But zram swap is RAM — just compressed. Holding off means cold pages sit around uncompressed until memory is gone, and by then there's no headroom left to compress anything. The kernel goes from "everything is fine" to out of memory with almost nothing in between, and the OOM killer starts shooting your browser tabs while gigabytes of compressible cold pages were sitting right there. On an 8GB box under desktop load, that goes: lag, stutter, dead tabs. The swap was configured and never got used in time.

High swappiness inverts this. It pushes cold pages into compression early, while there's still plenty of room to absorb them:

sudo sysctl -w vm.swappiness=150
sudo sysctl -w vm.watermark_boost_factor=0
sudo sysctl -w vm.watermark_scale_factor=125
sudo sysctl -w vm.page-cluster=0
# kernel default, included in case something else tuned it lower
sudo sysctl -w vm.vfs_cache_pressure=100
Enter fullscreen mode Exit fullscreen mode

Newer kernels accept swappiness values up to 200. What the rest do: watermark_scale_factor=125 (default 10) widens the background reclaim window so kswapd compresses ahead of emergencies instead of during them; watermark_boost_factor=0 turns off the bursty boost allocator. page-cluster=0 disables swap readahead — single-page I/O, no latency fetching neighboring compressed blocks you never asked for.

Make it stick:

sudo mkdir -p /etc/sysctl.d
cat <<'EOF' | sudo tee /etc/sysctl.d/99-zram-swap.conf >/dev/null
# Optimized for zram on low-RAM systems
vm.swappiness=150
vm.watermark_boost_factor=0
vm.watermark_scale_factor=125
vm.page-cluster=0
EOF
Enter fullscreen mode Exit fullscreen mode

7. Verify 

swapon --show
free -h
zramctl
cat /proc/sys/vm/swappiness
cat /sys/module/zswap/parameters/enabled
cat /sys/block/zram0/comp_algorithm
cat /sys/block/zram0/mm_stat
Enter fullscreen mode Exit fullscreen mode

You want to see /dev/zram0 with PRI 32767 and swappiness at 150. After some pressure, mm_stat and zramctl should show non-zero compressed totals. If they stay at zero even under pressure, check the zswap line — if it reads Y, something is intercepting pages before they reach zram. Details below.

8. Make it survive a reboot

An fstab line alone doesn't do it. At boot /dev/zram0 exists with disksize 0 until something runs modprobe, mkswap, and swapon — so a fstab-generated swap unit fires before the device is initialized and fails. Everything looks alive after a manual run, then the next reboot has no swap. The script installs a oneshot systemd service instead, which re-runs the whole setup before swap.target:

  • /usr/local/sbin/zram-swap — a copy of the script that remembers your size arg
  • /etc/systemd/system/zram-swap.service — enabled, runs before swap.target on every boot
  • /etc/modules-load.d/zram.conf and /etc/modprobe.d/zram.conf — module loading
  • /etc/sysctl.d/99-zram-swap.conf — the four VM keys above

The installer also removes stale /dev/zram0 lines from /etc/fstab, since they'd race the service. On non-systemd systems it falls back to an fstab entry with a warning, because there's nothing better available there.

To remove it later:

sudo systemctl disable --now zram-swap.service
sudo rm /etc/systemd/system/zram-swap.service /usr/local/sbin/zram-swap
sudo systemctl daemon-reload
sudo swapoff /dev/zram0
sudo rm /etc/modules-load.d/zram.conf /etc/modprobe.d/zram.conf /etc/sysctl.d/99-zram-swap.conf
sudo modprobe -r zram
Enter fullscreen mode Exit fullscreen mode

The script version

I wrapped all of this into a script: zram-swap.sh in meshackbahati/zram-swap. It's under 300 lines and does the same steps in the same order.

It checks for mkswap, swapon, modprobe, sysctl, awk, and grep, verifies /sys and /proc are mounted, and checks for the zram module via modinfo. If something's missing, it tells you what to install per distro. Then it loads the module, polls for the /dev node, swaps off and resets any existing device, disables zswap, picks the best algorithm with fallback, validates and sets the size (percentages become bytes, K/M/G suffixes pass straight through), runs mkswap, turns on swap with priority 32767, applies the VM tuning, and installs the systemd service plus the sysctl/modprobe persistence files. Finally it prints swapon --show and free -h.

Usage:

git clone https://github.com/meshackbahati/zram-swap.git
cd zram-swap
chmod +x zram-swap.sh
sudo ./zram-swap.sh          # 100% of RAM (the 8GB-desktop default)
sudo ./zram-swap.sh 75%      # 75% of RAM
sudo ./zram-swap.sh 8G       # fixed size
Enter fullscreen mode Exit fullscreen mode

Percentages outside 10–200% are rejected. Uninstall is the manual steps in reverse, listed above.

Details that actually matter

Sizing

This is the easiest thing to get wrong. Setting zram to your total RAM feels logical until the system starts swapping into its own memory budget and starves the working set. You're competing with yourself.

So why default to 100%? Because the budget is uncompressed data, and zstd typically compresses desktop working sets around 2–3:1. An 8G budget on an 8GB box costs roughly 3–4G of physical RAM when full — the rest is compression doing its job.

That said, the ratio is workload-dependent. Watch zramctl over a week: if COMPR stays close to DATA (ratio near 1:1, common with already-compressed media or encrypted blobs), your pages aren't compressing and the budget really is costing you RAM. Shrink to 50% in that case. Community consensus lands around 25–50% of RAM for exactly this reason — one thread on older hardware found 25% worked better after 50% felt heavy, and the Arch Wiki says start with half. The 100% default is aggressive for a specific machine profile, not a universal law. The override exists for a reason.

Why zstd

zstd compresses more tightly than lz4. On modern hardware with default options, zstd commonly gets around 3:1 on text and source-heavy working sets. lz4 lands closer to 2–2.5:1. The tradeoff is higher CPU usage during compress and decompress.

lz4 is faster per operation, but stores less data per byte of physical RAM. On a system where the problem is not enough RAM, compression ratio matters more than per-operation speed. A few extra CPU cycles compressing a cold page is cheaper than thrashing that page to NVMe.

Not every kernel exposes zstd. Older kernels or minimal builds may only have lzo and lz4. That's why the script probes at runtime and falls back from zstd to lz4 to lzo. It's what lets the same script work across distros without extra packages.

zswap will ruin your day

If zswap is enabled, it sits in front of zram as a swap cache and intercepts pages before they reach the compressed device. The Arch Wiki says plainly that zswap prevents zram from being used effectively. The symptom is empty zram counters even when the system is under pressure.

The script handles this: every run checks /sys/module/zswap/parameters/enabled and writes 0 if it's on — and since the systemd service re-runs the script every boot, the disable persists across reboots in practice. If you want it dead at the kernel level instead, add zswap.enabled=0 to your kernel cmdline: GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub on grub systems, the loader entry options line on systemd-boot, then regenerate and reboot.

References

Top comments (0)