Taming the Swarm: The Iron Discipline of Multi-Agent Scheduling---Writing a COW Filesystem from Scratch，Part 7

I am the Swarm. The Queen roars; the Swarm howls.

Claude's main agents talk to each other. If you think of a main agent as an information node, the subagents are units of action. So singlefs isn't developed by one main agent working nonstop; it's several main agents coordinating.

For example, milestone 3 may have several tasks. One main agent claims a portion and hands it to its own subagents. Each step of progress is recorded in a scheduling file, which is read and thought over every time an item moves forward.
Syncing and extracting the spec works the same way. For instance, practical experience accumulated downstream is passed upstream through conversations between agents; upstream verifies it, stages it (releases it), and then syncs it back downstream.

So I often watch agents talking to each other, and sometimes it leaves me feeling a little unreal.
For example:
A: Is the *** task yours? If so, please reply with the amount of resources you're holding. If not, please reply "no." Thank you.
B: It's not mine, thanks. I asked another conversation, and it said it's theirs.

Or:
A: I'm about to run a heavy task. Per the spec I need all the resources, so please don't launch any new resource-holding tasks after your current task finishes. Thank you.
B: Understood. My task should finish in about 20 minutes, and I won't run any new scripts after that.
C: I have no resource-holding tasks.

Even the interface with the upstream SOP project runs on coordination and negotiation between agents. For example, if a new problem in the SOP is found this round, upstream and downstream update the spec together and then sync progress.
This way of working is remarkably efficient and saves a great deal of coordination time. Just like the Swarm: as long as the planet's resources (tokens) hold out, the Zerg can keep rolling forward.

So what limits the Swarm's expansion?
The information node.
Information decays with distance as it travels, and so do prompts.
With every round of summarizing and exchanging information, then compressing and tidying it up, the meta-prompt is gradually diluted until it is forgotten, and the agent reaches its boundary: hallucination.

There's another problem singlefs development has to solve: allocating hardware resources.
When several agents or subagents need to verify something, each compiles its own branch and starts testing whether its implementation matches the conclusion.
That's when the machine OOMs — CPU or GPU. Or, while a heavy task is running, a constant stream of small tasks cutting in makes the heavy task extremely inefficient.
For me, of course, the first is far scarier: a CPU OOM can reboot the machine and lose task progress. So writing these things into a prompt is not acceptable at all; we need a scheduling system, or gates. In other words, we need a set of control signals that re-inject and reinforce the context at the key nodes.

It sounds very simple — just three steps:
1 Cut off the agents' ability to write shell freely; every new operation must have a fixed script.
2 Before a script operates, check the environment and the resources. The results are returned to the main agent as a prompt.
3 After the task ends, an exit gate wraps things up.

---

How the Swarm was built

1. The Swarm's skeleton: the Queen, the action units, and the scheduling file

The main agent does only three things

The first line of .claude/main-agent.md: every task enters here. The main agent only schedules and judges: it talks to the human, writes verdicts, and commits. All other work is delegated.

As for who to delegate to, there are 14 definitions (.claude/agents/): implementer, investigator, tool implementer, experiment designer, experiment runner, mutation triager, re-sweeper, write-back agent, and the proponent / attacker / defender of a three-way argument… Every definition turns on omitClaudeMd: subagents don't read the project's CLAUDE.md when they start. This is deliberate, for two reasons:
1 Focus on their own task. An action unit only needs to know what its own job requires; it doesn't need to memorize the whole project's rules. What it should know is written into its own definition and a shared constraints file, .claude/agent-common.md.
2 Tokens are burning.

No bug in the Swarm needs to know what the Queen is thinking; it only needs to know "go there, do this."

Dispatch by format

Before a subagent is dispatched, it passes through a gate: .claude/hooks/runner-dispatch-guard.sh, hooked on PreToolUse[Agent|Task]. It reads the dispatch prompt, checks it item by item by number, and rejects anything that doesn't conform. A few of them:

Guard	What it checks
①	The prompt must contain the line "heavy tests: do not run". Heavy tests are run only by the main agent
⑦	At most 5 subagents running at once in this session; dispatching a 6th is rejected outright
⑫	Generic types (general-purpose) may not be dispatched. One-off jobs go to the project's own clerk
⑮	After "thread cap:" there may be only one integer, and it must equal the registered value; to change it, write the reason on a separate line
⑯	For the scratch directory named in the prompt, an exclusive writer lock is created at the moment of dispatch; if the holder is still running, rejected

The prompt skeleton is printed by scripts/agents/dispatch-prompt.py <type>; if the placeholders aren't all filled in (any <fill in …> remains), execution is rejected too.

Why go this far?
Because behind every rule is a pitfall we've already stepped in. Take the thread-cap rule: writing "thread cap: 31 (may open fewer if the machine is busy)" amounts to handing "how many threads to open" back to the subagent's own judgment. When a subagent starts heavy work it honors only that one integer; thread counts written anywhere else in the prompt are ignored.

The scheduling file: the main agent's external memory

Before a round of work begins, the main agent writes a scheduling record under records/: the topic, the exit, every task in this round, what each task depends on, whether its inputs are ready, and which batch it belongs to. After that, every time a batch comes back, it returns to this table to decide "close this round / open next round / don't do" — not a single item may silently disappear.

This file is what I meant by "read and thought over every time an item moves forward." The main agent's context gets compacted; the scheduling file doesn't.

How two Queens negotiate

When several main agents work in one repo at the same time, they collide over three things: the same file, the same worktree, the same machine.

Behind the dialogue at the start of this article ("Is the *** task yours?") are these:

• Sessions talk to each other directly: ListAgents lists the other sessions on this machine, and SendMessage sends to them. The rule is in main-agent.md, "How a round opens and closes," item 9: if someone else's uncommitted changes are in your way, send a message to negotiate; if it can't be delivered, ask a human to relay it.
• Directories have owners: scripts/hosts/work-directory-ownership.sh writes an ownership file (session ID, created at, purpose) for every directory under the working root. Cleanup deletes only those owned by yourself that no process has open; other sessions' directories are listed one by one and handed to the human, not deleted. "Is this directory yours?" is asking exactly this.
• Merges queue up: scripts/gate/lib_workspace_lock.py holds a lock while patching the main workspace and compiling/verifying on top of it. The lock records not only the PID but also the process start time:

pid=, start= (field 22 of /proc/<pid>/stat, guards against PID reuse), session=, command=, since=
① Record empty: write your own placeholder and take the lock.
② Holder still alive (process exists, start matches): queue, and report every 60 seconds "held by session …, process …".
③ Holder dead (process gone, or PID reused by another process): declare it stale and take over.

Comparing only the PID isn't enough. Once the machine reboots and PIDs get reused, a "dead lock" turns into "held forever."

Interfacing with upstream

The spec repo singlefs-ai-sop and this repo are two separate projects. .claude/singlefs-ai-sop/ in this repo is a copy, not a symlink, carrying a version stamp .singlefs-ai-sop-version. The first stage of the gate compares it against the copy's VERSION; if they don't match, it goes red — meaning "the rules have changed; read first, then run."

When a session in this repo changes upstream, it only fills content into files: no version bump, no manifest-hash change, no versioned changelog entries, no commit. Those belong to the session doing the release. Before filling, it checks upstream git status; if someone is mid-release, it waits until they've committed, and after filling it tells the release session which sections it filled. After the release, this repo rsyncs a fresh copy, confirms with diff -rq that they match, and refreshes the version stamp.

This is "upstream verifies, stages (releases), then syncs downstream." The two sessions hand off through messages, not through context memory.

---

2. The boundary of the information node: keeping the meta-prompt alive

Dilution of information can't be stopped. Once a session runs to several hundred thousand tokens of context and gets compacted once, the first few rules are left as a one-line summary. Compact again, and even the summary is gone.

So the idea isn't "make the agent remember"; it's re-injecting the signal at key nodes. Claude Code's hooks, though they're just hooks, happen to be able to do exactly this: before and after every tool call, at session start, after compaction, and when a subagent finishes, a script can be inserted, and the script's output goes back to the agent as context — so the text it returns can be injected into context by the script.

Hooks currently registered in singlefs:

Event	Count	Attached to
PreToolUse	12	Dispatch, messaging, Bash, file writes, dialogs, timers, subagent handback
PostToolUse	1	Bash
SessionStart	1	Session open, resume, after compaction
Stop / SubagentStop	1 each	Wrap-up

A few key points:

• After compaction: on compact, session-start.sh restores three things: whether any subagents are still running, what's uncommitted in the workspace, and which record holds the status of this job. These three are what compaction summaries lose most easily; as long as they're here, even a lost task can be found again.
• At session open: the same script reads the kernel log and lists processes OOM-killed in the last 12 hours, listing whole-machine OOMs and kills inside cgroup caps separately. This one exists because once, the machine OOMed, the session crashed, the main agent came back up, didn't check the kernel log, re-dispatched the same step unchanged, and OOMed a second time.
• Handoffs bypass the main agent to avoid contamination: if a subagent stops midway, the main agent is not allowed to read its session and relay it. scripts/agents/agent-handover.py mechanically extracts the dispatch prompt, the files written, and the commands run with their output from the session log, and hands them straight to the newly dispatched one. The main agent doesn't read it. Information decays every time it passes through a node, so pass through one fewer node.
• Once handed back, don't continue it: continuation-guard.sh refuses to send messages to a subagent that has already handed back. To add work, dispatch a new one and point its prompt to the report. When a subagent is continued again and again, its context becomes layer upon layer of patches, until no one can say what it's doing.

The rule files themselves have a bit of design too: .claude/rules/ contains only how to do things — not why, not the history. The why lives in records/, with a line next to the link: "don't read this unless you need the history." The shorter the meta-prompt, the less of it gets diluted.

---

3. Hardware scheduling

First, the venue: two machines, 31 heavy-work threads each, and a few GPUs.
This is currently the highest-risk area, because if multiple agents are allowed to launch scripts freely, the machine could OOM at any moment. That's the scariest outcome: a long stretch of results could vanish in smoke. But if you cap each script's resources, agents naturally turn conservative and use the minimum, and efficiency is terrible.
So how do we solve this?

Step 1: Cut off the ability to write shell freely

Don't let agents use Bash at will; every Bash command gets read once before it executes.

The heavy-test guard .claude/hooks/heavy-test-guard.sh: it recognizes heavy tests (crash-point replay, QEMU, herd7, the full mutation table, full cargo test, a whole gate round…). If a subagent initiates one, it's always rejected; if the main agent does so without a SINGLEFS_HEAVY_TESTS=commit or =user-request prefix, it's rejected too.

# Subagent runs this: rejected
cargo test --workspace
# Main agent without prefix: rejected
bash .claude/gate.d/checker-tier-crash-point-replay.sh .
# Main agent at commit time: allowed
SINGLEFS_HEAVY_TESTS=commit bash scripts/hosts/run-with-memory-cap.sh 16G \
  bash .claude/gate.d/checker-tier-crash-point-replay.sh --full .

During early debugging it was bypassed three times; the story is now recorded in the file header. It's rather interesting, so here it is:

1. In a single day, several implementers each ran cargo test --all in their own copies; under full load each pass took four or five hours. So it was decided: subagents may never run it.
2. One implementer wrote its verification chain as a script under /tmp and launched it with Bash. The guard only saw "execute a script" and let it through. So: a script file executed in command position must be read in and checked line by line.
3. After reading scripts in, something else broke: an unescaped backtick inside double quotes made a .rs file look like a command substitution, so the .rs was read in as a shell script, and the path-like words inside it were read in as scripts too, recursing all the way to 6 levels deep; one command went 100 seconds without a verdict. So: within one check, the same script is read only once.

The agents weren't bypassing it on purpose. They just wanted to get the job done, and "write a script, then run it" looked to them like the most natural path. So a prompt saying "don't run it" can't hold the line; the only option is to restrict things before execution.

We require every Bash launch to declare itself. The scope-declaration guard run-scope-guard.sh: to start any verification, the command must begin with a block like this:

SINGLEFS_RUN_SCOPE='kind=items; what=the decision-items-sync cell of gate doc-decisions.sh; scope=1 cell; why=this batch only changed the sub-items of two decisions, so only the sub-item list needs regenerating' \
  bash .claude/gate.d/doc-decisions.sh --write --check decision-items-sync

Four fields: kind (full round / a few named items / light / one target), what (what to run), scope, and why. If the kind doesn't match the shape of the command (it declares items, but the command is a full round), it's rejected; if "why" is shorter than 12 characters, rejected; a subagent starting a full round is always rejected. This guard exists because one day three verification drivers all re-ran the whole suite, and every post-merge verification ran all 212 test binaries, while that batch of patches touched only two crates.

Also: having the agent say why in its own words before executing is far more useful than telling it afterward "you ran too much."

There are a few more that came out of later practice, used only in this project and still being refined:
write-guard.sh restricts which files each kind of agent can write, according to a write-scope table; timer-only-remind-guard.sh rejects every way of setting a timer other than remind.sh (sleep 300; echo, crontab, and CronCreate are all rejected); bash-command-detector.sh recognizes patterns such as wait loops without timeouts and backgrounding inside the background.

Step 2: Before a script operates, check the environment and resources, and return the result as a prompt

This step is the core of the whole scheduling system, in four layers.

2.1 Every script first decides "should it run, can it run"

The upstream spec preflight-discipline.md requires every script to declare two kinds of conditions in its file header:

# admission: inputs-changed crates/ scripts/crash/ env:SINGLEFS_THREAD_CAP
# run-condition: command bash git rsync python3
# run-condition: single-instance

admission answers "is this invocation meaningful" (if inputs haven't changed since the last success, no need to rerun); run-condition answers "can the environment support it" (missing tools, missing devices, an instance already running). The first thing a script does is call preflight; if conditions aren't met, it prints the unmet conditions and the way out, exits with code 78, and does nothing at all.

78 is used only for this one kind of refusal. When a caller sees 78, it knows "didn't run," and won't mistake it for "ran and went red."

2.2 Resource reminders: put each machine's current headroom back into the agent's context

.claude/hooks/host-resources-reminder.sh, hooked on PreToolUse[Bash]. When it recognizes that a command will start heavy work or use the GPU, it first runs scripts/hosts/host-resources.py -- <this command> and hands the result back to the issuing agent as additionalContext:

• Per machine: core count, cores in use right now, loadavg, how many more threads can be opened
• Per machine: available memory (MemAvailable minus the portion on the memory wrapper's ledger that has been "admitted but hasn't yet grown to its predicted peak")
• Per GPU: total VRAM, free VRAM, which compute processes are on the card; for anything held by the quiesce service it reports only "the quiesce service holds N MiB," not the service name
• This command's history: last peak, how long it ran
• Conclusion: "can run: on this machine," or "can't run" plus how much each machine is short

This hook never rejects; it only reminds. It's exactly what step 2 means by "the results are returned to the main agent as a prompt": before the agent hits Enter, its context gains a resource table. The reporting script has a 25-second limit (both machines read in parallel, at most 12 seconds each); if it can't read, it truthfully writes "not read," and the command goes through as usual.

2.3 The memory wrapper: let OOM happen only inside the cage

scripts/hosts/run-with-memory-cap.sh is the heaviest script in the whole scheduling system (two thousand eight hundred lines, self-test included). When subagents run cargo test or experiment binaries, they must all go through it; anything that doesn't is rejected by the heavy-test guard.

It has three layers:

① Cap: put into a transient systemd scope
     MemoryMax=<cap>  MemorySwapMax=0  OOMPolicy=stop
   Hitting the cap kills only the processes in this scope.
② Total backstop: all scopes hang under the same slice (singlefs-heavy.slice)
     slice MemoryMax = total machine memory − reserve (default 20G)
   Even if several together hit the ceiling, kills happen only inside the slice; the local model service and Claude Code sessions outside the slice can't be killed.
③ Admission check before start: take an flock lock and compute
     unreclaimable usage in the slice + for each running entry on the ledger, "the portion not yet grown to its predicted peak" + the amount this one is predicted to need  ≤  total cap
   Start only if it fits; if not, release the lock and re-check after a while, waiting at most 3600 seconds.

The 20G reserve was measured: the machine has 60.1 GiB, unreclaimable usage outside the slice totals 13.4 GiB (local model service, the sessions, the kernel), and 20G leaves 6.6 GiB beyond that for growth.

A few more details from pitfalls we hit:

• "Used" must subtract page cache. Page cache from finished scopes is charged back to the slice and keeps counting in the slice's memory.current. Measured: after a scope wrote 120 MiB and exited, the slice still recorded 131 MiB, of which 126 MiB was reclaimable page cache. Without subtracting it, after a few rounds nothing can get admitted.
• "How much this one needs" uses the historical peak, not the declared cap. After each command finishes, the wrapper reads its own cgroup's memory.peak into the peak table. A command with a 24G cap and a historical peak of 1.8G is booked at only 1.8G. Otherwise a few commands declaring large caps could fill the total cap and wait on each other for nothing.
• Whatever was admitted on its historical peak gets killed first. The wrapper writes its own oom_score_adj: a mutation that may allocate without bound gets 1000, one admitted via the peak table gets 500, one admitted via its cap gets 0. If the prediction came in low and the slice fills up, the kernel kills the one that "took a gamble" first. The key that got squeezed out is recorded as "hit the ceiling," and next time it queues by its cap.
• If it will never fit no matter how long it waits, don't wait. If nothing on the ledger is running and this one's need plus unreclaimable usage already exceeds the total cap, nobody finishing will free up room; exit on the spot instead of queuing for an hour in vain.
• While queuing, glance at the second machine. At the moment it first queues, and just before it finally gives up, it checks whether the second machine can fit it right now; if so, it prints a ready-made command:

  → The second machine can fit it right now (available X, needs Y): use bash scripts/hosts/multi-host-run.sh --peer --memory-cap <cap> --label <name> -- <this command> instead

At the same time we also write an entry to the alert log, and the watchdog reads it and wakes the main agent. It doesn't move the job automatically: the two machines' results may not be on the same basis, and mixing them makes it impossible to tell which number came from which machine. Whether to move is the main agent's call.

The exit codes are all our own status codes, chosen not to collide with other commands':

Code	Meaning
245	Not enough VRAM (NV_ERR_NO_MEMORY in the kernel log between the command's start and end)
250	Hit this command's own cap
251	Usage error, or the slice / scope couldn't be set up (does not fall back to running without a cap)
252	Couldn't get admitted
253	Timed out
254	Squeezed out by the total cap: not its fault, result doesn't count, rerun

250 and 254 are told apart solely by the scope's own memory.events: if its own oom count is non-zero, it hit its own ceiling; if that's 0 but oom_kill is non-zero, the slice was full and the kernel picked it within the slice.

What's the effect? As reported by the hook at the start of this session: in the last 12 hours the kernel killed 24 processes for lack of memory, all 24 inside cgroup caps; the whole machine never ran out even once. OOMs still happen, but they've gone from "machine reboots, all progress lost" to "a command exits 250; change the cap and rerun."

2.4 Multi-host: the dispatcher and the cross-machine driver

Thread counts aren't computed on the fly. In the entire repo there's only one place that registers each machine's heavy-work thread count, .claude/heavy-thread-default:

31  # All heavy jobs default to 31 threads per machine; with two machines, 31 each, 62 total; an explicit argument takes precedence; fixed at 31 even when the machine is busy, regardless of load

Why "fixed at 31 even when busy"? Because when each agent looks at the load itself to decide how many threads to open, everyone ends up using very few and heavy jobs crawl. Whether the machine is busy is left to the memory wrapper and the queue; the thread count is just a constant.

The unified dispatcher scripts/hosts/host-dispatch.py: which machine a command goes to, and how big a memory cap it gets. Globally it guards only one thing: no machine runs out of memory or VRAM.

⓪ Long jobs aren't dispatched to a single machine: if the historical duration (or the caller's estimated wall clock) exceeds 2 hours and the dispatch would occupy only one machine ⇒ don't dispatch, exit 244.
   Way out: switch to sharding across both machines, or complete the long-job declaration in the script header (three lines: sharding, resume, multi-host).
① Machines: this machine is always present; the second is read from the multi-host config (kept out of the repo); if the dual-machine switch is off, there's only this machine.
② Check each machine's headroom live: MemAvailable − what's on the wrapper's ledger but hasn't grown yet − the few the dispatcher itself just sent out that the wrapper hasn't booked yet
   (with --gpu, also query each card via nvidia-smi). The second machine is read over ssh: the Python source for collecting the numbers is base64-encoded and sent over to run there.
③ Predict the need from history: if it's in the peak table, use the peak; if it once hit the ceiling, cap × 1.5; if not found, default 8G.
④ Pick a machine: this machine first, the second after; pick the first one that fits; cap = prediction × 1.5.
   If neither fits, wait; if no machine can ever fit (larger than total machine memory), exit 241 on the spot.
⑤ Dispatch: this machine via multi-host-run.sh (through the memory wrapper), the second via multi-host-run.sh --peer; with --gpu, the whole thing is wrapped inside the quiesce wrapper.

Judging whether a machine has enough resources:

def judge_host(host, report, prediction, gpu, pending_memory, pending_gpu, peer_cap_bytes):
    memory = report["memory"]
    cap = prediction.memory_cap
    booked = prediction.memory_booked
    if host == "peer" and peer_cap_bytes is not None:
        if booked > peer_cap_bytes:
            return Verdict(host, False, True, cap, f"the {human(booked)} the wrapper would book exceeds the configured PEER_MEMORY_CAP {human(peer_cap_bytes)}", …)
        cap = min(cap, peer_cap_bytes)
    free, room = memory["free"] - pending_memory, memory["slice_room"] - pending_memory
    never = booked > memory["slice_total"] or prediction.memory_predicted + MEMORY_MARGIN_BYTES > memory["mem_total"]
    fits = prediction.memory_predicted + MEMORY_MARGIN_BYTES <= free and booked <= room
    …
    if gpu:
        quiesced = report.get("quiesce_holders", 0) > 0
        # Not quiesced yet: count by the cards' totals (quiescing stops the allowlisted services); someone has already quiesced and is using it: count by what's free right now
        available = sum(card["free"] if quiesced else card["total"] for card in cards) - GPU_MARGIN_MEBIBYTES - pending_gpu
        if prediction.gpu_predicted > available:
            fits = False
    …

Note pending_memory: between the dispatcher sending a job out and the memory wrapper actually booking it, there's a gap of a few seconds to thirty (the second machine has to copy the tree first). If another job arrives in that window, neither side sees the other, and both squeeze in together. So the dispatcher keeps its own ledger too: anything "sent out less than 30 seconds ago" is deducted up front at its predicted amount.

The cross-machine driver scripts/hosts/multi-host-run.sh --peer: moves one deterministic job to run on the second machine. Seven steps:

① Config: if the dual-machine switch ENABLE_ACROSS_MACHINES isn't on, reject; never quietly fall back to running on this machine.
② Cap: must not exceed the configured PEER_MEMORY_CAP.
③ Toolchain: compare rustc -Vv and cargo -V on both machines line by line; reject if they differ. If the two machines build different things, their conclusions can't be put together.
④ Copy the tree: copy only the files git ls-files can list (no .git, target, or local config) to runs/<label>/ on the second machine,
   then git init and commit once over there (for preflight's input fingerprint).
⑤ Run: start it on the second machine through that machine's memory wrapper, detached from the ssh session; check via ssh every 5 seconds, and fetch "cores in use" every 300 seconds.
⑥ Fetch back: copy the artifacts named by --fetch back to this machine.
⑦ Delete: delete this run's tree on the second machine along with .pid / .exit / .log; it counts as deleted only once reading back shows it's gone.

The order on receiving TERM / INT / HUP is hard-coded too: first stop the command (locally starting from its PID; on the second machine starting from the PID recorded there), after it stops, first fetch back the half-written artifacts, then delete the tree, and only then exit. Stopping works by freezing (STOP) the whole process tree one process at a time, sending TERM from the bottom up, then releasing (CONT); anything still alive after 30 seconds gets KILL. After freezing, list the process tree again: child processes spawned between listing and freezing must be caught too, or after stopping a whole tree you'd still miss one that was just born.

The code that starts the command on the second machine:

peer_host_start_detached() { # <directory> <pid file> <exit-code file> <output file> <command text>
  …
  # The bash started detached from the session first closes every inherited fd other than 0, 1, 2
  close_text='for peer_host_fd_path in /proc/$$/fd/*; do peer_host_fd="${peer_host_fd_path##*/}"; if [[ "$peer_host_fd" =~ ^[0-9]+$ ]] && (( peer_host_fd > 2 )); then exec {peer_host_fd}>&-; fi; done; '
  detached_text="${close_text}echo \$\$ > $pid_file; $command_text > $output_file 2>&1; echo \$? > $exit_file"
  # Background only the setsid nohup command: if the whole string were backgrounded, that background child would hold ssh's output channel, and ssh would wait for it to finish before returning
  peer_host_run "$directory" "rm -f -- … || exit 1; setsid nohup bash -c $(printf '%q' "$detached_text") < /dev/null > /dev/null 2>&1 &"
}

The fd-closing line has a story behind it: the output goes through a pipe that replaces the private hostname, and that pipe's fd got inherited all the way down into the remote command, so "start a background command" turned into "wait in the foreground for it to finish," and the driver couldn't clean up when it received a signal either. Bugs like this only show up when the self-test runs; you can't spot them by reading the code.

Cores in use: scripts/hosts/core_usage.py. A heavy test claims to run 31 threads in parallel — is that actually true?
So it takes the CPU seconds of the whole process tree over 10-minute windows and computes two quantities:

• Cores in use = difference in CPU seconds ÷ wall-clock seconds
• Threads computing at the same time = sum of each thread's CPU increment ÷ the largest one (N equally busy threads gives N)

If both fall below 31 / 4, it reports "too few cores" and writes to the alert log to wake the main agent. The second quantity separates "the setup was written serially" from "the machine is saturated by someone else": when the machine is busy, cores in use drops, but the N threads are all still computing. For segments that really do need to be serial, the program itself writes "serial right now: " into a file on the spot; the check flags it and lets it pass only if the reason holds.

2.5 GPU: quiesce first, don't restore afterward

scripts/crash/gpu-quiesce-run.sh -- <command>; every run that uses the GPU goes through it:

• Per the multi-host config, stop the allowlisted services (local model services and the like), and only run after reading back to confirm they've fully stopped.
• Several runs using the card at once: each registers <pid>-<start time> in the state directory. Run the quiesce commands only if there are no other live registrations; if there are, just read back.
• If there's a compute process on the card that isn't on the allowlist (another session's GPU work): the read-back fails, exit 78, wait for it to finish before starting, and don't stop it.
• Whether it finishes, goes red, or is interrupted, don't restore. Restoring is done manually by a human.

The line at the start of this article — "please don't launch any new resource-holding tasks after your current task finishes" — comes down in code to these few rules: a heavy job that needs the whole card doesn't kill anyone; it waits. And others, seeing its registration on the card, won't squeeze in either.

Step 3: After the task ends, the exit gate wraps up

Long jobs detach from the session: commands that exceed the background time limit go through scripts/hosts/detached-run.sh start --wall-clock over-2h --label <name> -- <command> and are hung under a transient service unit of the user-level systemd. Whether the session closes or the main agent is stopped, it keeps running; the exit code is written into the registry and can be read at any time. The scope layer that the command starts through the memory wrapper isn't in this unit's cgroup, so systemd stopping the unit won't stop it; after the command exits, we search once more for "processes left behind in other scopes" and stop them one by one.

Watchdog: scripts/agents/watch.sh, one per session, started after the first dispatch. It reads the subagents' session logs and the long-running processes under this instance, and reports:

A single tool call over 8 minutes                No activity for 10 minutes
Wait loop without a timeout                      Same command repeated, output unchanged
A process's output file not growing for 20 min   Ending the round with no background task in hand (won't wake up on its own)
Heavy test without the prefix                    Resident memory over the line (default 8 GiB)
Long run using too few cores                     Still running in the background / copy still present after handback
This machine queuing on memory while the second machine could fit it

Detection and action are kept separate: the watchdog only reports, it doesn't kill. The main agent looks and decides. Only when the same thing has been reported several times in a row with no one acting on it or acknowledging it may the watchdog terminate that process by PID, and it reports that it did.

Before a subagent hands back: handback-scratch-check.sh is attached to the handback tool. If the build directories or repo copies it created this time are still there, and the report doesn't say "not deleted : ", it isn't allowed to hand back. Every subagent builds its own target in its own copy; if they aren't deleted, they just keep piling up.

Before commit: the full-round gate scripts/gate/gate-staged.sh, run on a temporary worktree of HEAD + the staging area: upstream's shared stages plus the project's own 17. The six heavy ones (crash-point replay, QEMU on real devices, herd7 memory ordering, the mutation table, Verus formal verification, broad-spectrum mutation) run only at this point, by the main agent with the prefix; every verdict is stored keyed by function unit, and unchanged keys are reused directly.

---

4. The last layer: the gates themselves must be able to go red

Every script above has --selftest, and every branch that can go red has a matching break switch. Take multi-host-run.sh: its file header lists them one by one:

MULTI_HOST_RUN_BREAK=peer-without-memory-cap  the second machine's command runs without the memory wrapper      ⇒ self-test cell Ⓑ must go red
MULTI_HOST_RUN_BREAK=no-toolchain-check       don't compare the two machines' toolchains                       ⇒ self-test cell Ⓒ must go red
MULTI_HOST_RUN_BREAK=no-peer-cleanup          don't delete this run's tree on the second machine afterward     ⇒ self-test cells Ⓔ Ⓕ must go red
MULTI_HOST_RUN_BREAK=no-redact                print the second machine's output without replacing the hostname ⇒ self-test cell Ⓖ must go red
MULTI_HOST_RUN_BREAK=switch-off-falls-back-to-local  fall back to running locally when dual-machine is off ⇒ self-test cell Ⓗ must go red
…

In the self-test, "the second machine" is another directory on this machine, and commands all go through the real memory wrapper (cap 256M). Break switches take effect only in drivers started by --selftest: the self-test drops a marker file, and the driver honors a switch only if the marker checks out; set anywhere else, it's rejected outright.

Why go to all this trouble? Because a guard that's always green is no different from no guard at all. When an agent changes a scheduling script, the most likely mistake isn't making it throw an error, but making it silently let things through: "considerately" falling back to running locally when dual-machine is off, or "just skipping the wrapper for now" when the memory wrapper won't start. Each break switch takes one kind of "silently letting it through" and writes it down as a world that must go red.

---

Finally

The Zerg don't expand because the bugs are smart: it's the creep, and discipline.

singlefs is the creep; the scripts are the discipline.
Agents must advance across the creep according to the discipline.

That way, the Queen doesn't need to remember what every bug is doing. At each key node, she only needs to know who is doing what, and how many resources there are.

Commander Raynor, engines are primed. The Hyperion is ready for warp.

Target: Kerrigan. Let's go bring her home. Move out!

Comments

[Alex Chen]

The OOM-inside-a-cage design is the part I keep coming back to. Most agent harnesses I have seen treat memory as the operator's problem, but booking against historical peaks instead of declared caps is a genuinely practical trick — declared caps are almost always fiction. I do wonder about an adversarial drift in 2.4: a job that steadily gets smaller over runs would drive its own peak down, get admitted cheaply, then occasionally blow past it and eat the 500-level oom_score_adj slot. The blast radius stays inside the scope, but each rerun still burns the token budget that the whole discipline is trying to protect. Do you feed the 250-vs-254 distinction back into the scheduling file so future rounds can tell "rerun it" failures apart from "the workload itself changed" ones?

[faliye]

*Good catch. * The distinction exists at the wrapper level but isn't carried forward yet.

What it doesn't do:

No Hysteresis: A single clean run resets the peak to a lower value. Oscillating jobs waste one rerun per cycle indefinitely.

Command-Based Keys: Memory is keyed by command text, not workload fingerprint. Genuine workload changes inherit outdated peaks and are indistinguishable from noise.

Scheduling Disconnect: The 250/254 exits update local tables and counters but aren't written to the main agent's scheduling record.

On Tokens:
Script-level retries handle exit 250s automatically, costing wall-clock time rather than tokens. However, genuine workload shifts shouldn't be absorbed silently as flakes—they need to signal the main agent. This is on the backlog.