I run a lot of coding-agent sessions in parallel. Not two or three — a fleet.
And I kept hitting the same wall: it wasn't the agents failing, it was the
machine falling over underneath them.
The failure modes were always boring and always expensive:
- Background processes pile up. Every session spawns a few. Nobody stops them. Twelve sessions later the box is crawling and you can't tell which process belongs to which session.
- A run hangs and nobody notices. Not crashed — hung. Still holding its slot, still burning quota, producing zero output for hours.
- Sessions end and leave orphans. The agent is gone; its children are still there, holding ports and temp files.
- The disk fills overnight. A scratch dir nobody watches grows until the morning's builds start failing for "no space left on device."
The scariest version of the hang comes from Ramiz Raja's production checklist
for running multiple Claude Code agents
(codebyramiz.com/blog/running-multiple-claude-code-agents): his watchdog once
caught a run hung for 7 hours 22 minutes while using 258 milliseconds of
CPU. No CPU-based monitor would ever flag that. Only a liveness check — "has
this thing produced any output lately?" — catches it. Same checklist documents
an overnight 56 GB → 1 GB disk incident that motivated a standing disk
policy. I didn't live through those exact incidents, but I recognized the
pattern instantly, because I'd lived through smaller versions of all of them.
So I packaged the operational pattern into something installable:
fleet-guard.
What it is
A stdlib-only Python package — no dependencies — with three Claude Code hooks,
a stall watchdog daemon, a disk guard, and an append-only JSONL ledger:
-
Background-process quota. A
PreToolUsehook counts agent-attributed background processes machine-wide. Over the cap (default: half your CPU cores), the new background run is blocked with a message listing what's already running. Attribution is via an env marker —fleet-guard run --session <id> -- ./build.shsets it. -
Stall watchdog. Register a long-running process with
fleet-guard watch add --pid … --log …. If the process is alive but its log goes quiet longer than the timeout (default 600s), the watchdog captures a diagnostic snapshot and kills the whole process tree — children first,SIGTERM, thenSIGKILLfor survivors. -
SessionEnd reaping. A
SessionEndhook kills the leaked child process trees the session left behind. -
Disk guard.
fleet-guard disk-checkalerts when agent scratch dirs pass a size threshold or their filesystem runs low on free space — and can quarantine the offender by moving it aside. -
Append-only ledger. Every guard action lands in
~/.fleet-guard/ledger.jsonl, fsync'ed, one JSON object per line. When something gets killed at 3am, you can see exactly what and why.
The design rule I cared about most: fail-open. Every hook and every daemon
pass is wrapped so a guard that crashes exits 0 and lets the agent continue. A
guard must never take down the agent it's protecting. I test that explicitly —
corrupt state files, broken config, and the hooks still exit 0.
What it is not
I also maintain loop-guard, which detects semantic loops — an agent stuck
repeating itself in churn/test/burn cycles without making progress. That's a
different layer. fleet-guard is the machine-resource layer underneath it:
processes, stalls, leaked children, disk. An agent can be semantically perfect
and still wedge the machine. Run both; they don't overlap.
One honest limitation: the stall "dump" is a kernel-level snapshot —
/proc/<pid>/status, wait channel, kernel stack, IO counters, open FDs. That's
enough to distinguish "deadlocked on a futex" from "waiting on network IO,"
but a true in-process Python thread dump needs something like py-spy
cooperating from inside the target, which is out of scope for a
zero-dependency guard. The README says exactly where that line is.
Try it
pip install fleet-guard
Hook it into Claude Code (PreToolUse on Bash, SessionEnd, Stop — snippets in
the README), attribute your background runs, and let the watchdog run. Eleven
tests, all green, no network needed to run them.
Repo: https://github.com/hahahahahahahahah6/fleet-guard
If you run agent fleets, I'd genuinely like to know which failure mode bites
you first — the quota, the hang, the orphans, or the disk. My money's on the
disk.
Top comments (0)