DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Three agents on one laptop, 4 to 6 GB per dev server, and the 145-line lock that is just mkdir

I build CogniPrep with several coding agents working in parallel, each in its own git worktree of the same repository. Last Sunday three of them were building three new assessment providers at once, and the machine fell over.

Not a crash with a stack trace. Memory pressure, swap, and everything taking four times as long until nothing finished at all. The numbers, measured and then written into the script that fixed it:

# Measured on 16 Nov 2026: one `next dev` peaks at 4 to 6 GB (more if it stays
# up while code is edited), `tsc --noEmit` at 4 GB, and the test suite needs
# several GB more. Three workers each running all three at once filled the
# machine.
Enter fullscreen mode Exit fullscreen mode

Three worktrees times three heavy jobs is nine processes, each wanting multiple gigabytes. On 12 November a single webpack dev server left up while code was edited reached 12 GB on its own; on 16 November the total crossed 18 GB. No amount of per-process tuning fixes that. The only fix is that one of them runs at a time.

The lock is mkdir

There is no daemon, no lockfile library, no port to bind.

LOCK=/tmp/cogniprep-heavy.lock

acquire() {
  local waited=0
  until mkdir "$LOCK" 2>/dev/null; do
    ...
    sleep 5
    waited=$((waited + 5))
  done
}
Enter fullscreen mode Exit fullscreen mode

mkdir on a path that exists fails, and on POSIX it is atomic, so two processes racing cannot both succeed. The directory then holds the state as two files: pid, and what, which is a human-readable line like next dev on port 3002 (in /path/to/worktree).

That gives you a status command for free:

echo "held by pid $(cat "$LOCK/pid"): $(cat "$LOCK/what" 2>/dev/null)"
Enter fullscreen mode Exit fullscreen mode

and a waiting message that names the culprit, printed once a minute rather than every five seconds:

if [ $((waited % 60)) -eq 0 ]; then
  echo "one-at-a-time: waiting for: $(cat "$LOCK/what" 2>/dev/null)" >&2
fi
Enter fullscreen mode Exit fullscreen mode

Two stale-lock cases, because they are genuinely different

Every lock on a shared machine eventually gets orphaned, and there are two ways it happens.

if [ -f "$LOCK/pid" ] && ! holder_alive; then
  rm -rf "$LOCK"
  continue
fi
# A lock with no pid file older than a minute was left by a crash.
if [ ! -f "$LOCK/pid" ] && [ -n "$(find "$LOCK" -maxdepth 0 -mmin +1 2>/dev/null)" ]; then
  rm -rf "$LOCK"
  continue
fi
Enter fullscreen mode Exit fullscreen mode

The first case is a lock whose owner is dead, and kill -0 "$pid" settles it with no timeout at all: either the process exists or it does not.

The second case is the gap between mkdir succeeding and echo $$ > "$LOCK/pid" running. It is a few microseconds wide and it will absolutely happen if you put this on a machine that gets hard-rebooted, and when it does, a lock with no pid file is unrecoverable by the first rule, because there is no pid to test. Hence the age test: no pid file and older than a minute means nobody is coming back for it. One minute is far longer than the real window and far shorter than anybody's patience.

For an ordinary command, release is a trap:

acquire
echo $$ >"$LOCK/pid"
echo "$* (in $HERE)" >"$LOCK/what"
trap release EXIT INT TERM
"$@"
Enter fullscreen mode Exit fullscreen mode

The dev server holds the lock, and stops itself

A typecheck or a test run ends by itself. A dev server does not, and the thing that leaks a lock on a developer machine is a human getting distracted. So the server gets a dead-man's switch:

# Stop the server after DEV_MAX_MINUTES even if nobody remembers to.
(sleep $(( ${DEV_MAX_MINUTES:-20} * 60 )); kill "$pid" 2>/dev/null) >/dev/null 2>&1 &
Enter fullscreen mode Exit fullscreen mode

Twenty minutes by default. A backgrounded subshell is a crude timer and it is exactly as reliable as the shell that spawned it, which is the right level of engineering for this problem.

The readiness probe is the other part I would not skip:

for _ in $(seq 1 90); do
  if curl -s -o /dev/null "http://localhost:$port/robots.txt"; then
    echo "dev server ready on http://localhost:$port (pid $pid, log $log)"
    exit 0
  fi
  kill -0 "$pid" 2>/dev/null || { echo "dev server exited, see $log" >&2; release; exit 1; }
  sleep 2
done
Enter fullscreen mode Exit fullscreen mode

Ninety tries at two seconds is a three-minute ceiling, but the liveness check inside the loop is what matters: a server that dies on a syntax error reports that in two seconds instead of making you wait out the whole timeout for a message about a port that was never going to answer. And release runs on that path, so a failed start does not hold the lock.

Stopping is keyed on the worktree, not on a global, because three servers may be running from three paths:

state="/tmp/cogniprep-dev-$(echo "$HERE" | shasum | cut -c1-12)"
...
kill "$pid" 2>/dev/null
sleep 2
lsof -ti ":$port" 2>/dev/null | xargs kill 2>/dev/null
# Free the lock only if this server holds it.
if [ "$(cat "$LOCK/pid" 2>/dev/null)" = "$pid" ]; then release; fi
Enter fullscreen mode Exit fullscreen mode

A hash of the current directory as the state file name, lsof to sweep whatever survived the kill, and a conditional release so a stop in the wrong window cannot hand somebody else's lock away.

The Turbopack detail that cost the most time

Worktrees normally get a symlinked node_modules, which is free and works for tsc and vitest. The dev server is the problem: with a symlink it has to run on webpack, which in our measurements is about twenty times slower to compile a page, and one of those webpack servers had already reached 12 GB on 12 November. Turbopack wants a real node_modules, so setup installs one:

# Remove a symlinked node_modules (the link only, never its target), then
# install for real so Turbopack accepts it.
if [ -L node_modules ]; then rm node_modules; fi
CI=true pnpm install --frozen-lockfile --prefer-offline >/tmp/cogniprep-setup.log 2>&1
if [ ! -e .env.local ]; then ln -s "$MAIN/.env.local" .env.local; fi
Enter fullscreen mode Exit fullscreen mode

With pnpm's content-addressable store that is hard links: about ten seconds and almost no disk.

The parenthesis in that comment is the expensive part. pnpm install in a worktree whose node_modules is a symlink tries to purge the link's target, which is the main checkout's packages, so the order matters: remove the link, then install. Get it the other way round and you have broken the repository everything else on the machine is linked to.

The help text is the header comment

  *)
    sed -n '2,27p' "$0"
    exit 2
    ;;
Enter fullscreen mode Exit fullscreen mode

The usage block printed on a bad argument is lines 2 to 27 of the script itself, which is the comment at the top that a reader opens the file to see. One source of truth, and a usage message that cannot drift from the thing it documents. The line numbers are the one fragile part, and they are fragile in a way that is obvious the first time you run it wrong.

What it costs

It serialises, which is the point and also the price: a worker that wants to typecheck waits for whoever is testing. The instruction the agents get is explicit about that, because the natural reaction to a lock is to find a way round it:

Waiting for the lock is normal; do not work around it.
Enter fullscreen mode Exit fullscreen mode

There is one genuine foot-gun, and it is in the same note:

do not edit code while it runs, and never run checks while it runs (they would
wait on your own lock)
Enter fullscreen mode Exit fullscreen mode

Start a dev server, then try to typecheck from the same worktree, and you wait on yourself until the twenty-minute timer fires. A lock with one slot and no reentrancy will do that. I decided a comment was the right fix rather than tracking which jobs may share a holder, because the cases where that matters are the cases where you should not be running two heavy jobs anyway.

See it

The round that prompted this shipped three new assessment providers from three worktrees on one laptop: UCAT, the Army SIFT and the NJ law enforcement exam, 29 new URLs between them across hubs, per-test pages, cheating guides and blog posts. The sitemap went from 991 URLs to 1,093 in two days, and the machine stayed up.

If you are running several agents on one machine, the file to write before the second one starts is this one. 145 lines of bash, and the primitive at the bottom of it is mkdir.

Top comments (1)

Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

Putting the holder's description in what is the part I'd copy: a waiting agent can say what it's waiting for instead of just hanging. We coordinate one layer up: a manager agent splits the work and the other agents report to it in a shared room (polyroom, which we build), each on its own branch. A machine-level lock like yours covers what that doesn't, two workers starting a heavy job at the same moment. Your note tells agents that waiting is normal. When the holder is a test run that hangs rather than the dev server with its 20-minute timer, what ends the wait?