- Book: System Design Pocket Guide: Fundamentals
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You start four agents on four fresh worktrees. Each one's first move is to run the tests, and each one stops on a missing module. git worktree add checks out the commit's tracked files, and node_modules is in .gitignore, so the folder every agent needs first is the one folder that never arrives.
How you fill it decides how long each agent waits before it does anything useful. On pnpm's benchmark page, a clean install of a test project that resolves to about a gigabyte takes npm 12.2.0 1m 37s. With a warm cache and a lockfile, npm needs 14.5s and pnpm 12 needs 1.93s. Those are pnpm's numbers, from pnpm's machine over an emulated 50ms, 200Mbps link, so read them as a direction. Then multiply the slow row by the number of agents you start in a day.
There are five common ways to fill the folder, and each one breaks something different. I haven't run the script below myself, so there are no numbers of mine in this post. Your lockfile, your disk and your network decide them.
Five ways to fill an empty node_modules
| Method | What happens | Extra disk | Shared with the main checkout |
|---|---|---|---|
npm ci, cold cache |
Downloads every tarball, writes every file | Full tree, plus the cache | Nothing |
npm ci --prefer-offline, warm cache |
Reads tarballs from ~/.npm, still writes every file |
Full tree | Nothing |
pnpm install, warm store |
Links or clones files from one store, symlinks the graph | Close to none, on the same disk | The store's files |
ln -s to the main checkout |
Creates one symlink | None | Everything, live |
| Copy-on-write clone | Each file shares its data blocks with the original | Close to none, until files change | Data blocks, until a write |
The npm rows are the baseline. npm ci needs a package-lock.json, removes any existing node_modules before it starts, and exits with an error instead of updating a lockfile that disagrees with package.json. The prefer-offline setting bypasses staleness checks for cached data, so a warm ~/.npm saves the network round trips. It doesn't save the writes.
pnpm keeps a content-addressable store, and its docs describe installed files as hard-linked from it, "consuming no additional disk space." Its node_modules is a tree of symlinks over a .pnpm folder (layout docs). The default packageImportMethod: auto tries the cheaper methods first and falls back to copying: hard links first on Linux, a clone first on macOS and Windows (settings).
A symlink costs nothing and shares everything. A clone sits between that and a full install. On macOS, cp -c copies files using clonefile(2) (cp man page, macOS 15.2). On Linux, GNU cp --reflink=always makes the same kind of copy (cp(1)) on file systems with reflink support, such as Btrfs (reflink docs) and XFS, where mkfs.xfs turns reflink on by default (mkfs.xfs(8)). A fresh clone pair takes up the space of one file and grows as the two copies diverge (Eclectic Light).
Five ways into the same empty folder, at very different speeds and with very different strings attached.
A script that times all five on your repo
Run it from the main checkout after a successful npm ci there. It makes one detached worktree per method next to your repo, times the fill, records the disk used, and removes everything it created. The pnpm rows run only when the repo has a pnpm-lock.yaml. For an npm repo, pnpm import generates one from package-lock.json, so try it on a throwaway branch.
#!/usr/bin/env bash
# wt-deps-bench.sh: time 5 ways to fill node_modules in
# a new git worktree. Run from the main checkout after
# `npm ci` there. Optional: CHECK="npm test" ./wt-deps-bench.sh
set -uo pipefail
ROOT=$(git rev-parse --show-toplevel)
BASE="$(dirname "$ROOT")/wt-bench-$$"
mkdir -p "$BASE"
TIMEFORMAT=' time: %R s'
if [ "$(uname)" = Darwin ]; then
CP_CLONE="cp -cR"
else
CP_CLONE="cp -R --reflink=always"
fi
free_kb() { df -Pk "$BASE" | awk 'NR==2 {print $4}'; }
run() { # run <name> <command>, inside a new worktree
local name=$1 cmd=$2 wt="$BASE/$1" before after
git -C "$ROOT" worktree add -q --detach "$wt" HEAD
echo "== $name"
before=$(free_kb)
time (cd "$wt" && eval "$cmd" >/dev/null 2>&1) \
|| echo " FAILED"
after=$(free_kb)
echo " disk: $(( (before - after) / 1024 )) MB (df)"
if [ -n "${CHECK:-}" ]; then
if (cd "$wt" && eval "$CHECK" >/dev/null 2>&1); then
echo " check: pass"
else
echo " check: FAIL"
fi
fi
}
run npm-cold 'npm ci --cache "$BASE/empty-cache"'
run npm-warm 'npm ci --prefer-offline'
if [ -f "$ROOT/pnpm-lock.yaml" ]; then
run pnpm-first 'pnpm install --frozen-lockfile'
run pnpm-warm 'pnpm install --frozen-lockfile --prefer-offline'
fi
run symlink 'ln -s "$ROOT/node_modules" node_modules'
run clone '$CP_CLONE "$ROOT/node_modules" node_modules'
# Clean up: remove the link itself first, never its target.
rm -f "$BASE/symlink/node_modules"
for wt in "$BASE"/*/; do
git -C "$ROOT" worktree remove --force "$wt" 2>/dev/null
done
git -C "$ROOT" worktree prune
rm -rf "$BASE"
Set CHECK="npm test", or a faster smoke test, to run a command in each worktree after the fill. A fill that is fast and fails the check doesn't count. The pnpm-first row warms the store, and pnpm-warm is the one a second agent would see. The cold npm row points at an empty cache inside the bench folder, so its disk number includes the cache it builds. Run the whole thing twice and compare the second run.
Two choices in the script are deliberate. The worktrees go next to the repo, not under mktemp, because /tmp can sit on a different file system, and clones and hard links don't cross file systems: clonefile and Linux's FICLONE both return EXDEV, and pnpm copies instead of linking when its store is on another drive (pnpm FAQ).
The second choice: disk comes from df, not du. du counts a file with several hard links once, but only inside the tree it's measuring (GNU du). Point it at a pnpm worktree and the store's side of each link is outside that tree, so every file counts at full size. Clones look no different: Eclectic Light notes that Finder shows each clone at "the same, full size," and that at the command line clones "appear to be completely separate files." df asks the volume what's free, so it sees what was actually allocated. It also sees every other process writing to that disk, so run it while the machine is quiet.
What each method breaks
The symlink is only correct while the branch's lockfile matches the main checkout's. The first time an agent adds a dependency on its branch, that worktree runs against the wrong tree, and any install it runs writes into the folder every other worktree shares. The npm ci docs say it removes an existing node_modules before installing, and I wouldn't find out what that does through a symlink on a repo I care about.
Node works against you here too. By default it resolves a module to its real path on disk and uses that "as both an identifier and as a root path to locate other dependency modules" (Node.js docs). Every package in the linked folder resolves as if it lives in the main checkout. Caches kept inside node_modules are shared as well: Vite stores pre-bundled dependencies in node_modules/.vite, so two dev servers in two worktrees write to one cache. Set NODE_PRESERVE_SYMLINKS=1 to change the resolution, and the same docs warn that symlinked native modules linked from more than one location can fail to load.
A symlink shares one live folder. A clone shares blocks only until something writes.
The clone is a snapshot of the main checkout's tree. It has the symlink's lockfile problem without the sharing problem: a stale clone is wrong, but running npm ci over it hurts no other worktree. The copy is byte for byte, so a file that recorded the main checkout's absolute path during install still points there. Run grep -rl for your repo's path inside the clone before you trust it. On macOS, cp -c falls back to a regular copy when the target file system can't clone, so a clone onto a non-APFS volume succeeds silently at full time and full size. On Linux, --reflink=always fails instead, which is what you want in a script.
pnpm wants its store "on the same drive and filesystem as installations, otherwise packages will be copied, not linked" (pnpm FAQ). Its symlinked layout trips tools that assume a flat node_modules, which is why the nodeLinker: hoisted setting exists. On Linux, where pnpm hard-links by default, a file in node_modules is the same file as the one in the store, so an agent that patches it edits the file every other project links to. On APFS the default is a clone, and a patch stays local.
npm ci breaks nothing. It's usually the slowest row, and the only one that never depends on another folder.
The rule that falls out of this: clone when the lockfile matches the main checkout, install when it doesn't. Run this inside the new worktree:
ROOT=~/code/app # the main checkout
if cmp -s "$ROOT/package-lock.json" package-lock.json; then
cp -cR "$ROOT/node_modules" node_modules
else
npm ci --prefer-offline
fi
On Linux, swap cp -cR for cp -R --reflink=always.
Where Hermes IDE fits
I build Hermes IDE, so weigh this section with that in mind. Its README says each task gets its own branch and worktree, and that new worktrees get their dependencies in seconds. The README doesn't say which mechanism does that, and this post doesn't either. The trade-offs above apply to whichever one you pick for your own setup.
When a repo needs more, Hermes reads a .hermes/worktree.toml recipe. Its setup commands run in order in the new worktree, and the first one that fails stops the rest. Its copy list only takes files git ignores, such as .env, and its ** wildcard doesn't descend into ignored folders like node_modules (recipe docs).
Worktrees also cost disk. In 2.3.4, the Review Desk's Worktrees tab (below) shows each worktree's branch, task, age and disk size, and the free disk space. And 2.2.9 stopped Hermes from slowing down or using up the system's file-watch limit when worktrees contain large dependency or build folders.
The Review Desk's Worktrees tab in Hermes 2.3.4: one worktree each for orders-api, web-app and orders-types under ~/code, with each worktree's hermes/ branch, task, age and disk size, and the free disk space.
Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. The bench script is bash for macOS and Linux. Hermes is not affiliated with Nous Research, Anthropic, OpenAI or any agent vendor. Download it or read the code on GitHub.
If you run the script, post your five rows and your file system in the comments. I'd like to see how far apart the clone and pnpm rows land on a real repo.
If this was useful
A cold cache, a warm one, a shared store, one live link, a private copy that shares blocks until a write: these are the same caching and shared-state trade-offs you make in any backend, played out on your own disk. The System Design Pocket Guide: Fundamentals covers caching, latency and the other core building blocks in a short read. And if you run several agents on one repo, Hermes IDE gives each task its own worktree.
Top comments (0)