Out of inodes with 56 GB free - moving source trees into squashfs

Igor Sheko — Sat, 07 Nov 2026 00:00:00 +0000

webrtc.page is a set of MCP servers for reading real-time media codebases: libwebrtc, pion, coturn, Kamailio, Asterisk, FreeSWITCH, Janus and a few more. Each server keeps several releases of its project on disk, with a semantic index and a tree-sitter symbol index next to the source. libwebrtc is the big one: 32 Chrome milestones, the newest of them with their full DEPS checkout.

On 7 November 2026 I deployed M157 and the site went down. This post is about why, the fixes I shipped the same day, and the change that came out of it: every source tree is now a single squashfs image instead of a directory with a few hundred thousand files.

The failure

The deploy script packs changed versions into a tar archive, uploads it while the services are still running, then stops pm2, removes the old data and extracts the archive. The extraction step printed thousands of lines like this:

tar: ./versions/M157/src/third_party/blink/web_tests/fast/dom/Element/...: Cannot open: No space left on device
tar: Exiting with failure status due to previous errors

The script stopped there, with pm2 already stopped. Every endpoint returned 502 from 17:52 to 18:11 UTC, about 19 minutes, until a second deploy finished.

The disk-space check before the upload had passed. It said 59 GB would be free after extraction, and df -h agreed:

/dev/sda 199G 133G 56G 71% /mnt/volume_lon1_...

df -i did not:

/dev/sda 6553600 6553600 0 100% /mnt/volume_lon1_...

The data volume is ext4 with the default ratio of one inode per 32 KB, which gives 6.55 million inodes for 199 GB. Source code is millions of small files. The volume ran out of files long before it ran out of bytes, and my pre-flight check only counted bytes.

Where the inodes went

I counted files per directory on the server:

what	files
one libwebrtc milestone with deps	375k–396k
of which own WebRTC code	~19k
pion (210 release tags)	647k
Kamailio, FreeSWITCH, sipXcom, Asterisk (all versions)	284k

So a libwebrtc version is about 5% WebRTC and 95% third_party. That is expected once you look at DEPS: the entry src/third_party is Chromium's entire third_party repository, not a list of the libraries WebRTC links. Inside it, blink/web_tests, perf_tests and manual_tests alone were 203k files in M157. None of them were in the semantic index.

Two more things came out of the count.

Orphaned dependency trees. Locally I keep deps only for the newest versions and strip them from older ones with a compact command. On the server, 16 versions still had their third_party, not 7. compact deletes files but does not touch the version's updatedAt, and the incremental deploy only re-uploads versions whose updatedAt changed. So every compaction since the start stayed local, and the server kept nine trees of ~380k files that nobody had on disk anymore.

.git directories. Each DEPS checkout is a shallow git clone with its own .git. The deploy excluded only the top-level ./.git, so 45 nested ones per version, about 1.3 GB, went to production with every release.

Same-day fixes

To bring the site back I removed the half-extracted M157 and the orphaned third_party trees by hand, then re-ran the deploy. After that I made three changes so this would not happen again.

An inode check before anything is stopped. The deploy now counts entries in the archive and compares them to df --output=iavail on the server, keeping 200k spare. If the archive does not fit, the deploy fails while the old version is still serving.

Junk never reaches the disk. One list, PRUNE_PATHS, names the paths not worth keeping. The downloader turns it into a non-cone sparse checkout for the dependency that contains them, so they are never written:

/*
!/blink/web_tests/
!/blink/perf_tests/
!/blink/manual_tests/

The same list is used to prune versions that are already on disk, to exclude the paths from the archive, and to delete them on the server before the inode check. A version with deps went from ~396k files to ~193k.

compact follows the deploy. Compaction now bumps updatedAt, so the deploy re-uploads the version without deps and with the smaller index. The deploy also re-uploads any version that has third_party on the server but not locally, whatever its updatedAt says.

After the next deploy the volume was at 41% inodes (2.65M used) and 75 GB of disk. A good result, but the shape of the problem had not changed: one inode per source file, and every new milestone adds another ~190k.

Options

option	inodes	disk	cost
bigger volume, or `mkfs -i` with more inodes	more of them	—	money; migration of 133 GB; same growth
XFS (allocates inodes dynamically)	no fixed limit	—	migration; the files are still there
hardlink duplicates across versions	shared per file	shared	breaks with tar deploys; fragile
one squashfs image per tree	1 per version	compressed	mount management

The hardlink idea was tempting: third_party/abseil-cpp in M156 and M157 is byte-for-byte identical, all 1,655 files. But hardlinks only survive inside one tar archive, and the deploy ships one version at a time. Keeping them on the server would mean a dedup pass after each deploy.

squashfs is a read-only, compressed filesystem in a single file. The kernel mounts it with a loop device; without root there is squashfuse. My versions are already read-only in practice: a milestone is downloaded once, indexed once, and then only read. An image fits that.

Measurements

Before changing any code I packed M157 (deps, after pruning) and M150 (no deps) on my workstation: 20 cores, NVMe, mksquashfs 4.7.5, squashfuse 0.6.3. .git was excluded from the image.

tree	files	size	zstd-3	zstd-15
M157	190,823	3.3 GB	795 MB in 5.0 s	710 MB in 25.8 s
M150	18,521	276 MB	—	54 MB in 2.0 s

Reading is what the servers do all day, mostly ripgrep over the whole tree and single-file reads. Times are on a warm page cache unless marked "first run":

operation on M157	directory	squashfuse	squashfuse_ll
`rg -c RTC_GUARDED_BY` over the tree	0.17 s	17–23 s	2.1 s first run, 0.65 s after
`rg` in `pc/` only	0.01 s	0.02 s	—
`find -type f`	0.06 s	1.44 s	0.18 s

Plain squashfuse is single-threaded, and a full-tree search through it is 100 times slower than the directory. squashfuse_ll, the low-level multithreaded driver from the same project, is within 4–12x. The compression level did not change read speed (zstd-3 and zstd-15 both gave 2.1 s and 0.65–0.68 s), so I took zstd-15 for the smaller files.

On the server I do not need FUSE at all. The kernel there (Ubuntu 24.04, 6.8) has squashfs built in with CONFIG_SQUASHFS_ZSTD=y, and the services run as root, so a kernel loop mount uses the normal page cache and needs no daemon in a 4 GB box.

I repeated the test on the server with M150, the image placed on the same data volume as the directory, and the page cache dropped (echo 3 > /proc/sys/vm/drop_caches) before each cold run:

operation on M150, production	directory	kernel squashfs
`rg` over the tree, cold, run 1	7.28 s	2.21 s
`rg` over the tree, cold, run 2	4.67 s	1.95 s
`rg` over the tree, warm (3 runs)	0.12–0.14 s	0.11–0.12 s
read one file, cold	0.014 s	0.011 s

Here the image is 2.4–3.3x faster cold, and the same warm. The data volume is DigitalOcean block storage, attached over the network. Reading the tree as a directory means 16k small files scattered across it; reading the image means 54 MB of compressed data laid out together. On local NVMe the image only costs; on network storage it pays for itself.

The change

Every source tree keeps its path. What used to be a directory is now an image next to a mount point:

versions/M157/src/ directory → mount point (read-only)
versions/M157/src.sqfs → image
versions/M157/semantic.db, ast.db → unchanged

Nothing that reads source had to change: ripgrep, file reads, path sanitising with realpath and the AST indexer all see the same directory as before.

The module is small. Packing is one mksquashfs call:

export function packArgs(treeDir, out, { level = 15 } = {}) {
  return [
    treeDir, out,
    "-comp", "zstd", "-Xcompression-level", String(level),
    "-noappend", "-quiet", "-no-progress",
    "-wildcards", "-e", "... .git", // nested .git dirs stay out
  ];
}

Mounting depends on who is asking:

export function mountCommand(img, mnt, { root = false } = {}) {
  return root
    ? ["mount", ["-t", "squashfs", "-o", "loop,ro", img, mnt]]
    : ["squashfuse_ll", [img, mnt]];
}

Mounts are state outside the filesystem: a reboot or a deploy leaves empty mount points. Each MCP server therefore mounts the images of its own project when it starts. The call is idempotent, and a failed mount is logged and costs only those versions, not the server. The deploy also runs a mount-all before starting pm2, so a failure shows up in its log.

Three things broke on the way, all for the same reason: a mount point looks like a directory, so tools treat it as one.

tar walks into mounts. Packing a version whose src/ is mounted put all 190k files back into the archive. The deploy now excludes the mount point of every packed tree.
du walks into mounts too. The size estimate counted the uncompressed contents. It uses du -x now.
rm -rf cannot remove a read-only mount. Replacing a version on the server first unmounts its trees, then deletes the directory.

The migration avoided re-uploading whole versions. A version that is packed locally but is still a directory on the server ships only its src.sqfs. The server deletes the old src/ directory, the image takes its place, and semantic.db and ast.db stay where they are.

Results

On my workstation, packing all 32 libwebrtc versions took about five minutes:

	before	after
inodes used (whole disk)	8.82M	7.18M
disk used	512 GB	481 GB
libwebrtc source, 32 versions	~1.6M files	32 images, 6.16 GB

Images for versions with deps are 703–744 MB, without deps 50–60 MB.

In production the migration shipped only the 32 images, a 7.2 GB archive of 38k entries. The upload took about an hour while the services kept serving. The server then deleted the old src/ directories, extracted the images and mounted them: 32 kernel loop mounts, no failures.

production volume	before	after
inodes used	2.65M (41%)	1.02M (16%)
disk used	75 GB	44 GB
libwebrtc source	~1.6M files	32 images, 5.8 GB
downtime	—	about 4 minutes (502s from 20:55 to 20:59 UTC)

Most of those four minutes went into deleting 1.6 million files of the old trees. Future deploys of a packed version replace one file.

The other ten servers: line images

libwebrtc was the obvious first target: 32 trees that held most of the inodes. The other servers looked different. pion had 209 release tags, each with src/ and deps/, so 418 small trees of 2–3k files. The C servers had between 5 and 42 versions each. One image per tree would have meant about 570 mounts.

Every kernel squashfs mount keeps its own caches and decompressor state. From the kernel's structure that is roughly 0.7 MB per mount: a metadata cache of 8 × 8 KB, a fragment cache of 3 × 128 KB, a 128 KB data page and the zstd workspace. That is an estimate, not a measurement, but 570 of them would pin about 400 MB that cannot be reclaimed, on a box with 4 GB of RAM. Locally it would also mean 570 FUSE processes.

So for every server except libwebrtc the unit is a line, not a version:

pion/versions/.lines/v4.2.sqfs mounted at pion/versions/.lines/v4.2/
  v4.2.0/src/… v4.2.0/deps/…
  v4.2.1/src/… …
pion/versions/v4.2.1/semantic.db, ast.db (unchanged)

pion lines are minor release lines, seven in total. The other servers are small enough for one line each. The function every server uses to find a version's source root now checks two places. A loose directory with a src/ wins, which covers a version that was just downloaded and not packed yet. Otherwise the path points into the line. Packing a line rebuilds it from the current image plus any loose versions, swaps the image, and deletes the loose trees.

I expected lines to save mounts. They also saved a lot of space, because mksquashfs deduplicates identical files within an image by default, and consecutive releases of a project share most of their files:

server	versions	files	line images
pion	209	335,320	7 images, 33 MB
sipXcom	5	57,833	190 MB
Kamailio	13	90,123	24 MB
FreeSWITCH	12	72,878	52 MB
rtpengine	42	21,579	19 MB
all 10 servers	330	~635k	16 images, ~465 MB

63 pion v3.1 releases fit in 6 MB. Packing all 16 lines on my workstation took 63 seconds.

The deploy compares each line image's sha256 with the server's copy and ships only the ones that differ. The first run shipped all 16, a 1.3 GB archive (most of it the RFC index, which ships every time). Removing the 330 loose trees on the server happens after the services are back: until a loose src/ is gone the servers read it, and once it is gone they read the line. Each removal also checks first that the version is visible inside its mounted line, so a line that failed to mount leaves the old directories alone. This deploy was down for about two minutes (502s from 21:25 to 21:26 UTC).

production volume	after libwebrtc	after all servers
inodes used	1.02M (16%)	25.5k (0.4%)
disk used	44 GB	33 GB
squashfs mounts	32	48

Pros and cons

What I got:

One inode per version instead of up to ~190k. Inodes are no longer the limit; disk is, and slowly.
4–5x less disk. M157 went from 3.3 GB to 710 MB, M150 from 276 MB to 54 MB.
No .git in production , without a separate exclude rule.
A version deploys as one file. No extraction of hundreds of thousands of files while the services are stopped.
Immutable trees. A mounted image cannot be modified by a stray script, and the same file is what runs locally and in production.
Faster cold search in production. On network block storage a cold whole-tree ripgrep went from 4.7–7.3 s to 1.9–2.2 s.

What it costs:

Slower full-tree search without root. On my workstation, through squashfuse_ll, a cold whole-tree ripgrep is about 12x slower than a directory, 4x warm. Searches scoped to a subdirectory do not notice.
One FUSE process per mount locally. About 6 MB RSS idle, 36 MB after a full scan of M157. With 48 mounts that is fine; with one per tree it would not be, which is why small versions go into lines.
A new release repacks its whole line. For pion v4.2 that is 9 MB, for sipXcom 190 MB. Cheap at these sizes, and it grows with the line.
Mounts are state. A reboot, or a deploy that fails between unmount and mount, leaves empty directories. The servers remount on start, but it is one more thing that can be wrong.
Changing a version means repacking it. Adding deps to an old version or pruning it now goes extract, modify, pack. The tools do this automatically, but it is slower than deleting a directory.
No incremental upload inside a version. Any change to a tree produces a new image, uploaded whole.
Root on the server. Kernel mounts need it. My services already run as root; if yours do not, you need a mount helper or FUSE there too.

Takeaway

The data volume went from 6.55 million inodes used and a failed deploy to 25 thousand, and from 133 GB of disk to 33 GB. The source code did not change; only the way it sits on disk did.

The broader lesson is about the check that failed. Disk space is the number everyone watches, and the deploy script watched it carefully. For a workload made of small files, inodes are a separate budget, and it runs out silently.

DEV Community: Igor Sheko