DEV Community

Cover image for My Laptop Was Drowning in Swap. Fixing It Taught Me Three Things Linux Never Told Me.
Harmeet Singh
Harmeet Singh

Posted on AI-assisted

My Laptop Was Drowning in Swap. Fixing It Taught Me Three Things Linux Never Told Me.

7.5 GiB of RAM, three AI ingestion workers, a budget SSD, and a compressed swap cache that had been silently running the wrong algorithm for months. This is the story of a measured, reversible fix — and the two bugs hiding underneath it.

The symptom

My daily driver is an Ubuntu 26.04 laptop with an i5-8250U (8 threads), 7.5 GiB of RAM, and a no-name 256 GB SATA SSD — the kind of DRAM-less budget drive that has brutal random-4K latency. On top of it I run the worst possible workload for that hardware: a browser, VS Code, Postman, VLC, a desktop full of Electron apps, and my own project — a LangChain/FastAPI platform with Celery ingestion workers that import PyTorch, transformers, and docling.

The numbers looked dire at first glance:

MemTotal:   7880296 kB    (~7.5 GiB)
MemAvailable: 1180008 kB  (~1.15 GiB)
SwapTotal: 10485756 kB
SwapFree:   5245240 kB    (~5 GiB of swap "in use")
Enter fullscreen mode Exit fullscreen mode

Five gigabytes of swap in use on a 7.5 GiB machine. The folk wisdom says: install zram, thank me later. I very nearly did.

Instead, I spent an evening measuring before changing anything, and that decision re-shaped the entire fix. Three times over.

Lesson zero: the number you trust may be wearing a costume

The first thing to understand is what swapon --show actually reports. That "5 GiB used" does not mean 5 GiB of pages sitting on my SSD. It counts allocated swap slots.

Here's why that matters: my kernel had zswap enabled — a compressed RAM cache sitting in front of the disk swap. When the kernel evicts an anonymous page with zswap on, the page is compressed into a RAM pool, and a slot on the backing device is allocated as a placeholder. The page itself often never touches the disk at all.

So how much of my "5 GiB of disk swap" was actually on disk? The kernel's own counters gave the answer away: over 53 hours of uptime, pswpout recorded ~6.4M pages written (~25 GiB, mostly bursts), but pressure stall information (PSI) for memory was near zero at idle, and — the killer detail, which I extracted later — the zswap pool stats showed:

stored_pages      = 288076      (~1.13 GiB of pages)
pool_total_size   = 453316608   (~432 MiB compressed)
written_back_pages = 1059       (!)
Enter fullscreen mode Exit fullscreen mode

A 2.5:1 compression ratio, and barely a thousand pages ever written back to the SSD by the shrinker. The disk was almost never in the loop. What looked like "drowning in swap" was actually a working compressed-cache setup — one that deserved credit, not replacement.

This also corrected my own first diagnosis. I initially read "5 GiB swap used" as disk pressure. The counters said otherwise. Measure before you tune. The kernel leaves honest breadcrumbs — /proc/vmstat, /proc/pressure/*, and zswap's debugfs stats will out-argue any dashboard.

Lesson one: your config can be silently ignored

The second discovery came from comparing two files that should have agreed:

$ cat /proc/cmdline
... zswap.enabled=1 zswap.compressor=lz4 quiet

$ cat /sys/module/zswap/parameters/compressor
lzo
Enter fullscreen mode Exit fullscreen mode

My kernel command line demanded LZ4. The live system was running LZO — a compression algorithm roughly 30–40% slower per byte, for months, without a single loud warning.

The mechanism: CONFIG_CRYPTO_LZ4=m in Ubuntu's kernel means LZ4 support is a loadable module. zswap resolves its compressor when the first swap device registers, very early in boot — before anything has loaded crypto_lz4. The request fails, zswap falls back to its compiled-in default (LZO), and never retries the value from the cmdline. No dmesg sirens. Just a quiet downgrade.

The fix is two lines — one for now, one for every boot:

sudo modprobe lz4
echo lz4 | sudo tee /sys/module/zswap/parameters/compressor
echo "lz4" | sudo tee /etc/modules-load.d/lz4.conf   # persists across reboots
Enter fullscreen mode Exit fullscreen mode

If you've ever "enabled lz4" on a Linux box, go check right now that the live parameter matches your intention. On a shocking number of machines, they don't.

Lesson two: zram and zswap are rivals, not teammates

With the cache-audit done, the actual decision was architectural. Both zram and zswap compress evicted pages into RAM:

  • zswap is a cache in front of a backing swap device (mine: a 10 GiB swap file on the SSD).
  • zram is the swap device — a compressed RAM disk.

Since kernel 6.12's zswap shrinker, the folk wisdom "zram > zswap" is mostly obsolete: zswap-with-disk now behaves functionally like zram-with-writeback. The remaining deltas are fault latency (µs-scale decompression vs. random 4K reads on my SSD) and SSD wear — both of which matter on my hardware, so a zram trial was justified.

But here's the trap that shapes the whole migration: zswap intercepts writes to all swap devices, including zram. Enable both and every page gets compressed twice — CPU burned for nothing, accounting garbage. They are alternatives, not teammates. Adopting zram means disabling zswap, full stop.

I asked whether zswap could instead be scoped "hierarchically" — a compressed front-tier with a threshold handing off to zram. It can't; zswap has no per-device scoping. But the hierarchy I actually wanted already existed one layer down: swap priorities. Give zram priority 100 and the swap file priority −1, and the kernel fills zram first, only overflowing to the SSD past its capacity threshold. Tiering was built into the swap subsystem all along.

So the plan, in phases, each reversible:

  1. Phase 0 — make LZ4 real for the existing zswap, tune vm.swappiness=100 and vm.page-cluster=0, and cap the Celery workers' memory appetite (more on that later).
  2. Phase 1 — install systemd-zram-generator, configure 4 GiB of LZ4 zram at priority 100, disable zswap live before zram starts (ordering matters), and flip the GRUB cmdline to zswap.enabled=0.
  3. Keep /swap.img untouched as the overflow tier. No migration tricks — with 1.15 GiB free, a swapoff on the file would OOM the box; natural churn redistributes over days.

The plot thickens: two bugs hiding in "it worked"

Phase 0 + Phase 1 ran as one script. The verification output looked almost perfect:

NAME       TYPE      SIZE USED PRIO
/swap.img  file       10G 1.6G   -1
/dev/zram0 partition 3.8G   0B  100

/dev/zram0 lzo-rle       3.8G   4K   80B ...
zswap enabled : N   (want 0)
swappiness    : 100 (want 100)
page-cluster  : 0   (want 0)
Enter fullscreen mode Exit fullscreen mode

Everything correct — except lzo-rle where LZ4 was promised. The generator config was unambiguous:

[zram0]
zram-size = 4096
compression-algorithm = lz4
swap-priority = 100
Enter fullscreen mode Exit fullscreen mode

And the service had exited 0/SUCCESS. A unit that exits green while doing the wrong thing logs nothing — which is exactly why nobody notices this class of bug.

Bug the first: the hot-add ABI moved. My diagnostic script read /sys/class/zram-control to create a throwaway device — the standard incantation for a decade. On this kernel it answered:

cat: /sys/class/zram-control: Is a directory.
Enter fullscreen mode Exit fullscreen mode

Modern kernels turned the file into a directory; hot-add now lives at /sys/class/zram-control/hot_add. Because the failure mode is a filesystem error rather than a permissions error, it reads as "something's broken" rather than "the interface moved." Script defensively:

if [[ -f /sys/class/zram-control ]]; then
  ZID=$(cat /sys/class/zram-control)          # pre-6.x ABI
else
  ZID=$(cat /sys/class/zram-control/hot_add)  # 6.x+ ABI
fi
Enter fullscreen mode Exit fullscreen mode

Bug the second: the generator silently skips the algorithm write. With the hot-add fixed, I ran the kernel-acceptance probe on a throwaway device — echo lz4 > comp_algorithm, read it back: accepted. So the kernel wants LZ4, the config requests LZ4, the binary contains the config key... and the device still comes up lzo-rle. Root cause by strace — trace the generator's actual sysfs writes during setup. The syscall log shows whether it ever opens and writes comp_algorithm at all. (It doesn't need to fail loudly to be wrong; it just needs to skip.)

The recovery was to stop trusting the tool and build the setup I could verify:

sudo swapoff /dev/zram0
echo 1 | sudo tee /sys/block/zram0/reset
echo lz4 | sudo tee /sys/block/zram0/comp_algorithm   # before disksize — EBUSY after!
echo 4294967296 | sudo tee /sys/block/zram0/disksize
mkswap -q -L zram0 /dev/zram0
swapon -p 100 /dev/zram0
Enter fullscreen mode Exit fullscreen mode

Two ordering laws matter here: algorithm before disksize (writes after sizing return EBUSY, and the kernel quietly keeps its compiled-in default), and mkswap before swapon. Then, for durability, a small oneshot systemd unit (Before=swap.target, idempotent) replaces the generator's setup at boot, and the generator config is retired to .disabled. Now the boot-time behavior is deterministic and LZ4 is enforced, not requested.

The other half: why the workers ate 2.3 GiB in the first place

Fixing swap was treating the symptom. The disease: three Celery prefork processes with ~780 MB swapped out each.

This turned out to be structural, not a leak. The ingestion workers import torch 2.9.1 + transformers + docling + langextract at startup. That's a ~0.5–0.9 GiB baseline per child before any task state exists. Prefork makes this survivable — children fork from a warm master and inherit everything copy-on-write — but nothing bounded how much task state each child could accumulate before recycling. The profile already had max_tasks_per_child=100; a hundred memory-hungry tasks is a long life.

Celery has a purpose-built control for exactly this: --max-memory-per-child (KiB; the child is recycled when RSS crosses it). I wired it through the codebase the way the architecture demands — settings → profile → CLI — so the flag is profile-owned and operators can't override it per-invocation:

# settings.py
CELERY_DEFAULT_MAX_MEMORY_PER_CHILD: int = Field(default=1_000_000, ge=1)
CELERY_INGESTION_MAX_MEMORY_PER_CHILD: int = Field(default=1_000_000, ge=1)
CELERY_CRAWLER_MAX_MEMORY_PER_CHILD: int = Field(default=1_000_000, ge=1)
Enter fullscreen mode Exit fullscreen mode 
# cli.py — emitted from the profile, and added to the forbidden-override set
"--max-memory-per-child", str(profile.max_memory_per_child),
Enter fullscreen mode Exit fullscreen mode

Because children fork from the warm master, recycling is cheap — the "recycling is expensive" intuition belongs to thread/solo pools, not prefork. The contract tests were extended so the flag stays pinned to the profile forever.

The verdict so far 

$ zramctl
NAME       ALGORITHM DISKSIZE  DATA   COMPR     TOTAL
/dev/zram0 lz4          3.8G   ...   ...       ...
$ cat /sys/block/zram0/mm_stat
...   # orig_data_size / compr_data_size = the effective ratio, in a few days
Enter fullscreen mode Exit fullscreen mode

What I expect, and what I'll judge it by — not vibes:

  • App/tab switching after idle gets visibly faster. Evicted pages come back as LZ4 decompressions in RAM instead of random 4K reads from a DRAM-less SSD. That stutter was the complaint that started all of this.
  • Memory-pressure moments (agent spawns, Chrome ballooning) degrade more gracefully — eviction is cheap, so the kernel stalls less.
  • The SSD stops paying the swap tax until zram's 4 GiB fills.
  • When the active working set exceeds RAM+zram, nothing virtualizes that. This fix removes the tax of swapping; it doesn't conjure capacity. On this machine, the honest ceiling was always the three torch-importing workers.

The metrics I'm watching for a verdict: orig_data_size / compr_data_size from mm_stat (expect ~2.5:1 based on the zswap evidence), the growth rate of workingset_refault_anon in /proc/vmstat, and memory PSI during my normal day.

The rules I'd hand to a friend

  1. swapon --show "used" counts slots, not disk bytes. On any zswap machine, that number is mostly RAM wearing a disk costume. Check pswpin/pswpout and /sys/kernel/debug/zswap/* before believing it.
  2. Configs can be silently downgraded. Compare /proc/cmdline against /sys/module/zswap/parameters/*. Trust the read-back, not the write.
  3. zram and zswap are alternatives. Both on means double compression. The tiering you want is swap priority: zram high, disk low.
  4. A green exit code proves nothing. The generator exited 0 while skipping its job. When config "doesn't take", strace the sysfs writes — units that lie don't log.
  5. Kernel interfaces move under your feet. /sys/class/zram-control went from file to directory. Probe, don't assume.
  6. Ordering is law with zram: algorithm → disksize → mkswap → swapon.
  7. Fix causes too. Swap tuning made the symptoms cheaper; --max-memory-per-child made the workers stop hoarding. Do both.

All of this was done reversibly: kernel modules, sysctl files, a generator config, one GRUB word (with a backup), and a systemd unit. Every step had a rollback. That's not timidity — that's what makes the measurement trustworthy.

[Insert final mm_stat ratio and refault-counter comparison after the trial period.]

Top comments (0)