DEV Community: faliye

Benchmarking 6 Open LLMs on a 2-Machine RPC Cluster (Consumer GPUs, ~96GB VRAM)

faliye — Thu, 22 Oct 2026 13:27:11 +0000

In the endless pursuit of "bigger, bigger, and even bigger," I finally reached this point: two machines directly connected via a local area network, using RPC scheduling to split and load the model layer by layer across both hosts.

The reason is simple—I have a limited budget. Buying a professional-grade graphics card with 96GB of VRAM would definitely not pass my wife's approval. So for consumer-level GPU users on a budget like me, linking multiple machines and GPUs via RPC is currently the only realistic option.

The setup itself is straightforward: connect two machines with an Ethernet cable. I used a nominally 5Gbps CAT6A cable, and the actual tested bandwidth is about 4.7Gbps—yielding a throughput of around 500 MB/s. Under this bandwidth, llama.cpp's RPC chain invocation is currently the only viable choice (a Thunderbolt 5 adapter is on the way; once it arrives, it should close some of the performance gap).

The ambient temperature during testing was 35°C; the CPU reached 76°C, and the RTX 5090 (power-limited to 450W) peaked at 68°C.

No special optimizations were made at the architecture level—I just got the official RPC server running end-to-end first, leaving optimizations for later.

Hardware Setup

Host	GPU	VRAM
Host 1	RTX 5080	16GB
Host 1	RTX 5060 Ti ×2	16GB × 2
Host 2	RTX 5090	32GB
Host 2	RTX 5060 Ti	16GB

This totals approximately 96GB of VRAM, with the two hosts directly connected via a CAT6A network cable.

Pitfall Record: tensor-split Order

Once the two machines were connected, it was time to pick the models. With a 96GB budget, there isn't actually much room for choice—you have to consider production-level concurrency requirements (according to current market expectations, a 96GB setup is considered a poor investment if it can't support a team of 8-15 people), while also balancing KV context size and model precision.

When running the first model (Qwen3.6-35B-A3B Q8_0), I noticed the main card (5090) was using less than 7GB of VRAM—a massive waste. At the time, --tensor-split was set to 32,16,16,16,16, intending to allocate the largest share to the main card. But the actual device order was:
[ local - RPC : 5080 , local - RPC : 5060Ti , local - RPC : 5060Ti , 5090 , 5060Ti ]

In other words, the remote RPC cards were placed at the front, while the local main card was at the end—the exact opposite of what I expected. This is why the VRAM allocation didn't land on the 5090 as intended.

Candidate Models

Model	Quantization	Type / Active Params	Layers	Model Size	Thinking Mode
qwen36 · Qwen3.6-35B-A3B	Q8_0	MoE, 3B active	Split into 5 layers	~42GB	Disabled
llama33 · Llama-3.3-70B	Q6_K	Dense	80 layers, split into 5 parts	~73GB	No thinking capability
qwen25 · Qwen2.5-72B	Q6_K	Dense	80 layers	~80GB	No thinking capability
glm · GLM-4.5-Air	Q4_K_M	MoE	46 layers	~73GB	No thinking capability
qwen235 · Qwen3-235B-A22B-2507	Q2_K_XL	MoE	94 layers	~88GB	Disabled
gpt-oss-120b	MXFP4	MoE	36 layers	~65GB	Cannot disable, set to "low"

The comparison between Qwen3.6-35B-A3B and Llama-3.3-70B perfectly illustrates the architectural differences. Llama is a Dense model—every token activates all 80 layers of parameters, so the overhead introduced by splitting across 5 GPUs plus one cross-machine network hop is magnified by about 9 times. In practice, its speed is only about 10 tok/s—barely enough for one person doing translation work (close to reading speed), but with noticeable latency. On the other hand, Qwen3.6 is an MoE model, activating only 3B parameters per token; the computation involved in running it on the same 5 GPUs is significantly lower.

Methodology

With thinking mode disabled, the models were evaluated across 5 dimensions:

Dimension	Measurement Content	Test Cases	Scoring Method
1. Speed / Concurrency	TTFT (Time to First Token), Decode TPS, Performance degradation at 1/2/4 concurrency	Automated load test	Fully automated (tok/s, TTFT ms)
2. Translation Error Rate	Rare/error-prone characters (from real novel corpora), JP to ZH, including classic mistranslation traps	101	Fully automated (compared vs standard translation → error rate)
3. Code Understanding	High difficulty: Bug troubleshooting, behavior prediction, hierarchical allocator implementation, TP bandwidth reasoning, bash traps, refactoring	6	Manual scoring (based on checkpoints)
4. Hallucination Rate	False premises, fake citations, fabricated terms/APIs, unknowable questions, translation traps—all fact-checked	24	Heuristic + manual judgment
5. Translation Proofreading/Correction	Proofreading and correction, polishing fidelity, terminology consistency	9	Scored separately from scratch translation

The judge was primarily Claude 3.5 Sonnet (Opus 4.8 equivalent internally), supplemented by partial manual review.

A few notes:

I could only run models my hardware could handle; models under 48GB were tested in previous articles.
This is a personal test, and the test case design inherently carries some subjectivity.
All tests were single runs, with no voting mechanisms or self-reflection—accuracy relied entirely on the model's own restraint. If popular multi-path reasoning verification were applied, accuracy would likely improve, but this requires more compute and time, so it wasn't done here.
The sample sizes in dimensions 3, 4, and 5 are very small (6/24/9 questions respectively). Any disputed judgment is enough to change the rankings. Please view the "Overall Comparison" conclusions below as a directional signal rather than a statistically significant difference.

VRAM Usage per Model (Initial Test Run)

Model	Host 1 (5080 + 5060Ti×2)	Host 2 (5090 + 5060Ti)
Qwen3.6-35B-A3B Q8_0	13.0GB / 14.0GB / 8.3GB	6.9GB / 7.9GB
Llama-3.3-70B Q6_K	12.4GB / 11.1GB / 13.8GB	22.7GB (13%) / 13.7GB (23%)
Qwen2.5-72B Q6_K	13.4GB / — / —	24.7GB / 14.8GB / 12.0GB (100%)
GLM-4.5-Air Q4_K_M	11.2GB / 10.1GB (100%) / 10.1GB	28.0GB / 8.7GB
Qwen3-235B-A22B-2507 Q2_K_XL	14.8GB / 14.8GB / 14.2GB	28.8GB / 13.4GB
gpt-oss-120b MXFP4	11.0GB / 11.3GB (57%) / 9.1GB	26.8GB / 8.0GB

Note: In-host GPU ordering and VRAM allocation ratios are heavily influenced by --tensor-split and loading sequence; host/card allocation is not fixed across models—it reflects the actual RPC scheduling results in a specific run.

Overall Comparison Summary

Model (Quant)	Type / Active	Decode tok/s (short ctx)	Max Usable Context	Error Rate ↓	Code Understanding	Hallucination Rate ↓	Proofreading
Qwen3.6-35B-A3B Q8_0	MoE 3B	88	128K	35.6%	83%	18.8%	100%
gpt-oss-120b MXFP4	MoE 5.1B	94	128K	45.5%	75%	56.3% ❌	83.3%
GLM-4.5-Air Q4_K_M	MoE 12B	53	65K	40.6%	100%	70.8% ❌	94%
Qwen3-235B-A22B-2507 Q2_K_XL	MoE 22B	31	8K ❌	35.6%	58%	35.4%	89%
Qwen2.5-72B Q6_K	Dense	10	32K	42.6%	83%	54.2%	100%
Llama-3.3-70B Q6_K	Dense	11	98K	54.5% ❌	67%	75.0% ❌	78%

(Note: "Max Usable Context" reflects the measured upper limit under the specific resource allocation, not the model's inherent capability limit.)

Core Conclusion: In a CAT6A (5Gbps) network and llama.cpp RPC chain invocation environment, Qwen3.6-35B-A3B Q8_0 is the best-performing model overall in this test—it ranks at the top in speed, concurrency degradation, translation error rate, and hallucination suppression.

Final Thoughts

After finishing the Qwen2.5 run, I looked back at the VRAM usage and realized that the SAM/CLIP small models left over from earlier tests hadn't actually been completely offloaded. This is a good reminder: in an environment running multiple models mixed together, you should always manually verify the VRAM that "should be freed," rather than blindly trusting that a process exit will definitely clean it up.

Another thought worth recording: Qwen3.6-35B-A3B Q8_0 only took up about 42GB of VRAM, which means it could actually run entirely on a single machine. This left me in deep thought—I went through all this trouble to build a dual-node RPC cluster, only to end up essentially running a model that a single machine could handle? I'll continue pondering this question when the Thunderbolt 5 adapters arrive for the next benchmarking post.

Taming the Swarm: The Iron Discipline of Multi-Agent Scheduling---Writing a COW Filesystem from Scratch，Part 7

faliye — Tue, 20 Oct 2026 11:45:31 +0000

I am the Swarm. The Queen roars; the Swarm howls.

Claude's main agents talk to each other. If you think of a main agent as an information node, the subagents are units of action. So singlefs isn't developed by one main agent working nonstop; it's several main agents coordinating.

For example, milestone 3 may have several tasks. One main agent claims a portion and hands it to its own subagents. Each step of progress is recorded in a scheduling file, which is read and thought over every time an item moves forward.
Syncing and extracting the spec works the same way. For instance, practical experience accumulated downstream is passed upstream through conversations between agents; upstream verifies it, stages it (releases it), and then syncs it back downstream.

So I often watch agents talking to each other, and sometimes it leaves me feeling a little unreal.
For example:
A: Is the *** task yours? If so, please reply with the amount of resources you're holding. If not, please reply "no." Thank you.
B: It's not mine, thanks. I asked another conversation, and it said it's theirs.

Or:
A: I'm about to run a heavy task. Per the spec I need all the resources, so please don't launch any new resource-holding tasks after your current task finishes. Thank you.
B: Understood. My task should finish in about 20 minutes, and I won't run any new scripts after that.
C: I have no resource-holding tasks.

Even the interface with the upstream SOP project runs on coordination and negotiation between agents. For example, if a new problem in the SOP is found this round, upstream and downstream update the spec together and then sync progress.
This way of working is remarkably efficient and saves a great deal of coordination time. Just like the Swarm: as long as the planet's resources (tokens) hold out, the Zerg can keep rolling forward.

So what limits the Swarm's expansion?
The information node.
Information decays with distance as it travels, and so do prompts.
With every round of summarizing and exchanging information, then compressing and tidying it up, the meta-prompt is gradually diluted until it is forgotten, and the agent reaches its boundary: hallucination.

There's another problem singlefs development has to solve: allocating hardware resources.
When several agents or subagents need to verify something, each compiles its own branch and starts testing whether its implementation matches the conclusion.
That's when the machine OOMs — CPU or GPU. Or, while a heavy task is running, a constant stream of small tasks cutting in makes the heavy task extremely inefficient.
For me, of course, the first is far scarier: a CPU OOM can reboot the machine and lose task progress. So writing these things into a prompt is not acceptable at all; we need a scheduling system, or gates. In other words, we need a set of control signals that re-inject and reinforce the context at the key nodes.

It sounds very simple — just three steps:
1 Cut off the agents' ability to write shell freely; every new operation must have a fixed script.
2 Before a script operates, check the environment and the resources. The results are returned to the main agent as a prompt.
3 After the task ends, an exit gate wraps things up.

How the Swarm was built

1. The Swarm's skeleton: the Queen, the action units, and the scheduling file

The main agent does only three things

The first line of .claude/main-agent.md: every task enters here. The main agent only schedules and judges: it talks to the human, writes verdicts, and commits. All other work is delegated.

As for who to delegate to, there are 14 definitions (.claude/agents/): implementer, investigator, tool implementer, experiment designer, experiment runner, mutation triager, re-sweeper, write-back agent, and the proponent / attacker / defender of a three-way argument… Every definition turns on omitClaudeMd: subagents don't read the project's CLAUDE.md when they start. This is deliberate, for two reasons:
1 Focus on their own task. An action unit only needs to know what its own job requires; it doesn't need to memorize the whole project's rules. What it should know is written into its own definition and a shared constraints file, .claude/agent-common.md.
2 Tokens are burning.

No bug in the Swarm needs to know what the Queen is thinking; it only needs to know "go there, do this."

Dispatch by format

Before a subagent is dispatched, it passes through a gate: .claude/hooks/runner-dispatch-guard.sh, hooked on PreToolUse[Agent|Task]. It reads the dispatch prompt, checks it item by item by number, and rejects anything that doesn't conform. A few of them:

Guard	What it checks
①	The prompt must contain the line "heavy tests: do not run". Heavy tests are run only by the main agent
⑦	At most 5 subagents running at once in this session; dispatching a 6th is rejected outright
⑫	Generic types (general-purpose) may not be dispatched. One-off jobs go to the project's own `clerk`
⑮	After "thread cap:" there may be only one integer, and it must equal the registered value; to change it, write the reason on a separate line
⑯	For the scratch directory named in the prompt, an exclusive writer lock is created at the moment of dispatch; if the holder is still running, rejected

The prompt skeleton is printed by scripts/agents/dispatch-prompt.py <type>; if the placeholders aren't all filled in (any <fill in …> remains), execution is rejected too.

Why go this far?
Because behind every rule is a pitfall we've already stepped in. Take the thread-cap rule: writing "thread cap: 31 (may open fewer if the machine is busy)" amounts to handing "how many threads to open" back to the subagent's own judgment. When a subagent starts heavy work it honors only that one integer; thread counts written anywhere else in the prompt are ignored.

The scheduling file: the main agent's external memory

Before a round of work begins, the main agent writes a scheduling record under records/: the topic, the exit, every task in this round, what each task depends on, whether its inputs are ready, and which batch it belongs to. After that, every time a batch comes back, it returns to this table to decide "close this round / open next round / don't do" — not a single item may silently disappear.

This file is what I meant by "read and thought over every time an item moves forward." The main agent's context gets compacted; the scheduling file doesn't.

How two Queens negotiate

When several main agents work in one repo at the same time, they collide over three things: the same file, the same worktree, the same machine.

Behind the dialogue at the start of this article ("Is the *** task yours?") are these:

Sessions talk to each other directly: ListAgents lists the other sessions on this machine, and SendMessage sends to them. The rule is in main-agent.md, "How a round opens and closes," item 9: if someone else's uncommitted changes are in your way, send a message to negotiate; if it can't be delivered, ask a human to relay it.
Directories have owners: scripts/hosts/work-directory-ownership.sh writes an ownership file (session ID, created at, purpose) for every directory under the working root. Cleanup deletes only those owned by yourself that no process has open; other sessions' directories are listed one by one and handed to the human, not deleted. "Is this directory yours?" is asking exactly this.
Merges queue up: scripts/gate/lib_workspace_lock.py holds a lock while patching the main workspace and compiling/verifying on top of it. The lock records not only the PID but also the process start time:

pid=, start= (field 22 of /proc/<pid>/stat, guards against PID reuse), session=, command=, since=
① Record empty: write your own placeholder and take the lock.
② Holder still alive (process exists, start matches): queue, and report every 60 seconds "held by session …, process …".
③ Holder dead (process gone, or PID reused by another process): declare it stale and take over.

Comparing only the PID isn't enough. Once the machine reboots and PIDs get reused, a "dead lock" turns into "held forever."

Interfacing with upstream

The spec repo singlefs-ai-sop and this repo are two separate projects. .claude/singlefs-ai-sop/ in this repo is a copy, not a symlink, carrying a version stamp .singlefs-ai-sop-version. The first stage of the gate compares it against the copy's VERSION; if they don't match, it goes red — meaning "the rules have changed; read first, then run."

When a session in this repo changes upstream, it only fills content into files: no version bump, no manifest-hash change, no versioned changelog entries, no commit. Those belong to the session doing the release. Before filling, it checks upstream git status; if someone is mid-release, it waits until they've committed, and after filling it tells the release session which sections it filled. After the release, this repo rsyncs a fresh copy, confirms with diff -rq that they match, and refreshes the version stamp.

This is "upstream verifies, stages (releases), then syncs downstream." The two sessions hand off through messages, not through context memory.

2. The boundary of the information node: keeping the meta-prompt alive

Dilution of information can't be stopped. Once a session runs to several hundred thousand tokens of context and gets compacted once, the first few rules are left as a one-line summary. Compact again, and even the summary is gone.

So the idea isn't "make the agent remember"; it's re-injecting the signal at key nodes. Claude Code's hooks, though they're just hooks, happen to be able to do exactly this: before and after every tool call, at session start, after compaction, and when a subagent finishes, a script can be inserted, and the script's output goes back to the agent as context — so the text it returns can be injected into context by the script.

Hooks currently registered in singlefs:

Event	Count	Attached to
PreToolUse	12	Dispatch, messaging, Bash, file writes, dialogs, timers, subagent handback
PostToolUse	1	Bash
SessionStart	1	Session open, resume, after compaction
Stop / SubagentStop	1 each	Wrap-up

A few key points:

After compaction: on compact, session-start.sh restores three things: whether any subagents are still running, what's uncommitted in the workspace, and which record holds the status of this job. These three are what compaction summaries lose most easily; as long as they're here, even a lost task can be found again.
At session open: the same script reads the kernel log and lists processes OOM-killed in the last 12 hours, listing whole-machine OOMs and kills inside cgroup caps separately. This one exists because once, the machine OOMed, the session crashed, the main agent came back up, didn't check the kernel log, re-dispatched the same step unchanged, and OOMed a second time.
Handoffs bypass the main agent to avoid contamination: if a subagent stops midway, the main agent is not allowed to read its session and relay it. scripts/agents/agent-handover.py mechanically extracts the dispatch prompt, the files written, and the commands run with their output from the session log, and hands them straight to the newly dispatched one. The main agent doesn't read it. Information decays every time it passes through a node, so pass through one fewer node.
Once handed back, don't continue it: continuation-guard.sh refuses to send messages to a subagent that has already handed back. To add work, dispatch a new one and point its prompt to the report. When a subagent is continued again and again, its context becomes layer upon layer of patches, until no one can say what it's doing.

The rule files themselves have a bit of design too: .claude/rules/ contains only how to do things — not why, not the history. The why lives in records/, with a line next to the link: "don't read this unless you need the history." The shorter the meta-prompt, the less of it gets diluted.

3. Hardware scheduling

First, the venue: two machines, 31 heavy-work threads each, and a few GPUs.
This is currently the highest-risk area, because if multiple agents are allowed to launch scripts freely, the machine could OOM at any moment. That's the scariest outcome: a long stretch of results could vanish in smoke. But if you cap each script's resources, agents naturally turn conservative and use the minimum, and efficiency is terrible.
So how do we solve this?

Step 1: Cut off the ability to write shell freely

Don't let agents use Bash at will; every Bash command gets read once before it executes.

The heavy-test guard .claude/hooks/heavy-test-guard.sh: it recognizes heavy tests (crash-point replay, QEMU, herd7, the full mutation table, full cargo test, a whole gate round…). If a subagent initiates one, it's always rejected; if the main agent does so without a SINGLEFS_HEAVY_TESTS=commit or =user-request prefix, it's rejected too.

# Subagent runs this: rejected
cargo test --workspace
# Main agent without prefix: rejected
bash .claude/gate.d/checker-tier-crash-point-replay.sh .
# Main agent at commit time: allowed
SINGLEFS_HEAVY_TESTS=commit bash scripts/hosts/run-with-memory-cap.sh 16G \
  bash .claude/gate.d/checker-tier-crash-point-replay.sh --full .

During early debugging it was bypassed three times; the story is now recorded in the file header. It's rather interesting, so here it is:

In a single day, several implementers each ran cargo test --all in their own copies; under full load each pass took four or five hours. So it was decided: subagents may never run it.
One implementer wrote its verification chain as a script under /tmp and launched it with Bash. The guard only saw "execute a script" and let it through. So: a script file executed in command position must be read in and checked line by line.
After reading scripts in, something else broke: an unescaped backtick inside double quotes made a .rs file look like a command substitution, so the .rs was read in as a shell script, and the path-like words inside it were read in as scripts too, recursing all the way to 6 levels deep; one command went 100 seconds without a verdict. So: within one check, the same script is read only once.

The agents weren't bypassing it on purpose. They just wanted to get the job done, and "write a script, then run it" looked to them like the most natural path. So a prompt saying "don't run it" can't hold the line; the only option is to restrict things before execution.

We require every Bash launch to declare itself. The scope-declaration guard run-scope-guard.sh: to start any verification, the command must begin with a block like this:

SINGLEFS_RUN_SCOPE='kind=items; what=the decision-items-sync cell of gate doc-decisions.sh; scope=1 cell; why=this batch only changed the sub-items of two decisions, so only the sub-item list needs regenerating' \
  bash .claude/gate.d/doc-decisions.sh --write --check decision-items-sync

Four fields: kind (full round / a few named items / light / one target), what (what to run), scope, and why. If the kind doesn't match the shape of the command (it declares items, but the command is a full round), it's rejected; if "why" is shorter than 12 characters, rejected; a subagent starting a full round is always rejected. This guard exists because one day three verification drivers all re-ran the whole suite, and every post-merge verification ran all 212 test binaries, while that batch of patches touched only two crates.

Also: having the agent say why in its own words before executing is far more useful than telling it afterward "you ran too much."

There are a few more that came out of later practice, used only in this project and still being refined:
write-guard.sh restricts which files each kind of agent can write, according to a write-scope table; timer-only-remind-guard.sh rejects every way of setting a timer other than remind.sh (sleep 300; echo, crontab, and CronCreate are all rejected); bash-command-detector.sh recognizes patterns such as wait loops without timeouts and backgrounding inside the background.

Step 2: Before a script operates, check the environment and resources, and return the result as a prompt

This step is the core of the whole scheduling system, in four layers.

2.1 Every script first decides "should it run, can it run"

The upstream spec preflight-discipline.md requires every script to declare two kinds of conditions in its file header:

# admission: inputs-changed crates/ scripts/crash/ env:SINGLEFS_THREAD_CAP
# run-condition: command bash git rsync python3
# run-condition: single-instance

admission answers "is this invocation meaningful" (if inputs haven't changed since the last success, no need to rerun); run-condition answers "can the environment support it" (missing tools, missing devices, an instance already running). The first thing a script does is call preflight; if conditions aren't met, it prints the unmet conditions and the way out, exits with code 78, and does nothing at all.

78 is used only for this one kind of refusal. When a caller sees 78, it knows "didn't run," and won't mistake it for "ran and went red."

2.2 Resource reminders: put each machine's current headroom back into the agent's context

.claude/hooks/host-resources-reminder.sh, hooked on PreToolUse[Bash]. When it recognizes that a command will start heavy work or use the GPU, it first runs scripts/hosts/host-resources.py -- <this command> and hands the result back to the issuing agent as additionalContext:

Per machine: core count, cores in use right now, loadavg, how many more threads can be opened
Per machine: available memory (MemAvailable minus the portion on the memory wrapper's ledger that has been "admitted but hasn't yet grown to its predicted peak")
Per GPU: total VRAM, free VRAM, which compute processes are on the card; for anything held by the quiesce service it reports only "the quiesce service holds N MiB," not the service name
This command's history: last peak, how long it ran
Conclusion: "can run: on this machine," or "can't run" plus how much each machine is short

This hook never rejects; it only reminds. It's exactly what step 2 means by "the results are returned to the main agent as a prompt": before the agent hits Enter, its context gains a resource table. The reporting script has a 25-second limit (both machines read in parallel, at most 12 seconds each); if it can't read, it truthfully writes "not read," and the command goes through as usual.

2.3 The memory wrapper: let OOM happen only inside the cage

scripts/hosts/run-with-memory-cap.sh is the heaviest script in the whole scheduling system (two thousand eight hundred lines, self-test included). When subagents run cargo test or experiment binaries, they must all go through it; anything that doesn't is rejected by the heavy-test guard.

It has three layers:

① Cap: put into a transient systemd scope
     MemoryMax=<cap>  MemorySwapMax=0  OOMPolicy=stop
   Hitting the cap kills only the processes in this scope.
② Total backstop: all scopes hang under the same slice (singlefs-heavy.slice)
     slice MemoryMax = total machine memory − reserve (default 20G)
   Even if several together hit the ceiling, kills happen only inside the slice; the local model service and Claude Code sessions outside the slice can't be killed.
③ Admission check before start: take an flock lock and compute
     unreclaimable usage in the slice + for each running entry on the ledger, "the portion not yet grown to its predicted peak" + the amount this one is predicted to need  ≤  total cap
   Start only if it fits; if not, release the lock and re-check after a while, waiting at most 3600 seconds.

The 20G reserve was measured: the machine has 60.1 GiB, unreclaimable usage outside the slice totals 13.4 GiB (local model service, the sessions, the kernel), and 20G leaves 6.6 GiB beyond that for growth.

A few more details from pitfalls we hit:

"Used" must subtract page cache. Page cache from finished scopes is charged back to the slice and keeps counting in the slice's memory.current. Measured: after a scope wrote 120 MiB and exited, the slice still recorded 131 MiB, of which 126 MiB was reclaimable page cache. Without subtracting it, after a few rounds nothing can get admitted.
"How much this one needs" uses the historical peak, not the declared cap. After each command finishes, the wrapper reads its own cgroup's memory.peak into the peak table. A command with a 24G cap and a historical peak of 1.8G is booked at only 1.8G. Otherwise a few commands declaring large caps could fill the total cap and wait on each other for nothing.
Whatever was admitted on its historical peak gets killed first. The wrapper writes its own oom_score_adj: a mutation that may allocate without bound gets 1000, one admitted via the peak table gets 500, one admitted via its cap gets 0. If the prediction came in low and the slice fills up, the kernel kills the one that "took a gamble" first. The key that got squeezed out is recorded as "hit the ceiling," and next time it queues by its cap.
If it will never fit no matter how long it waits, don't wait. If nothing on the ledger is running and this one's need plus unreclaimable usage already exceeds the total cap, nobody finishing will free up room; exit on the spot instead of queuing for an hour in vain.
While queuing, glance at the second machine. At the moment it first queues, and just before it finally gives up, it checks whether the second machine can fit it right now; if so, it prints a ready-made command:

  → The second machine can fit it right now (available X, needs Y): use bash scripts/hosts/multi-host-run.sh --peer --memory-cap <cap> --label <name> -- <this command> instead

At the same time we also write an entry to the alert log, and the watchdog reads it and wakes the main agent. It doesn't move the job automatically: the two machines' results may not be on the same basis, and mixing them makes it impossible to tell which number came from which machine. Whether to move is the main agent's call.

The exit codes are all our own status codes, chosen not to collide with other commands':

Code	Meaning
245	Not enough VRAM (`NV_ERR_NO_MEMORY` in the kernel log between the command's start and end)
250	Hit this command's own cap
251	Usage error, or the slice / scope couldn't be set up (does not fall back to running without a cap)
252	Couldn't get admitted
253	Timed out
254	Squeezed out by the total cap: not its fault, result doesn't count, rerun

250 and 254 are told apart solely by the scope's own memory.events: if its own oom count is non-zero, it hit its own ceiling; if that's 0 but oom_kill is non-zero, the slice was full and the kernel picked it within the slice.

What's the effect? As reported by the hook at the start of this session: in the last 12 hours the kernel killed 24 processes for lack of memory, all 24 inside cgroup caps; the whole machine never ran out even once. OOMs still happen, but they've gone from "machine reboots, all progress lost" to "a command exits 250; change the cap and rerun."

2.4 Multi-host: the dispatcher and the cross-machine driver

Thread counts aren't computed on the fly. In the entire repo there's only one place that registers each machine's heavy-work thread count, .claude/heavy-thread-default:

31  # All heavy jobs default to 31 threads per machine; with two machines, 31 each, 62 total; an explicit argument takes precedence; fixed at 31 even when the machine is busy, regardless of load

Why "fixed at 31 even when busy"? Because when each agent looks at the load itself to decide how many threads to open, everyone ends up using very few and heavy jobs crawl. Whether the machine is busy is left to the memory wrapper and the queue; the thread count is just a constant.

The unified dispatcher scripts/hosts/host-dispatch.py: which machine a command goes to, and how big a memory cap it gets. Globally it guards only one thing: no machine runs out of memory or VRAM.

⓪ Long jobs aren't dispatched to a single machine: if the historical duration (or the caller's estimated wall clock) exceeds 2 hours and the dispatch would occupy only one machine ⇒ don't dispatch, exit 244.
   Way out: switch to sharding across both machines, or complete the long-job declaration in the script header (three lines: sharding, resume, multi-host).
① Machines: this machine is always present; the second is read from the multi-host config (kept out of the repo); if the dual-machine switch is off, there's only this machine.
② Check each machine's headroom live: MemAvailable − what's on the wrapper's ledger but hasn't grown yet − the few the dispatcher itself just sent out that the wrapper hasn't booked yet
   (with --gpu, also query each card via nvidia-smi). The second machine is read over ssh: the Python source for collecting the numbers is base64-encoded and sent over to run there.
③ Predict the need from history: if it's in the peak table, use the peak; if it once hit the ceiling, cap × 1.5; if not found, default 8G.
④ Pick a machine: this machine first, the second after; pick the first one that fits; cap = prediction × 1.5.
   If neither fits, wait; if no machine can ever fit (larger than total machine memory), exit 241 on the spot.
⑤ Dispatch: this machine via multi-host-run.sh (through the memory wrapper), the second via multi-host-run.sh --peer; with --gpu, the whole thing is wrapped inside the quiesce wrapper.

Judging whether a machine has enough resources:

def judge_host(host, report, prediction, gpu, pending_memory, pending_gpu, peer_cap_bytes):
    memory = report["memory"]
    cap = prediction.memory_cap
    booked = prediction.memory_booked
    if host == "peer" and peer_cap_bytes is not None:
        if booked > peer_cap_bytes:
            return Verdict(host, False, True, cap, f"the {human(booked)} the wrapper would book exceeds the configured PEER_MEMORY_CAP {human(peer_cap_bytes)}", …)
        cap = min(cap, peer_cap_bytes)
    free, room = memory["free"] - pending_memory, memory["slice_room"] - pending_memory
    never = booked > memory["slice_total"] or prediction.memory_predicted + MEMORY_MARGIN_BYTES > memory["mem_total"]
    fits = prediction.memory_predicted + MEMORY_MARGIN_BYTES <= free and booked <= room
    …
    if gpu:
        quiesced = report.get("quiesce_holders", 0) > 0
        # Not quiesced yet: count by the cards' totals (quiescing stops the allowlisted services); someone has already quiesced and is using it: count by what's free right now
        available = sum(card["free"] if quiesced else card["total"] for card in cards) - GPU_MARGIN_MEBIBYTES - pending_gpu
        if prediction.gpu_predicted > available:
            fits = False
    …

Note pending_memory: between the dispatcher sending a job out and the memory wrapper actually booking it, there's a gap of a few seconds to thirty (the second machine has to copy the tree first). If another job arrives in that window, neither side sees the other, and both squeeze in together. So the dispatcher keeps its own ledger too: anything "sent out less than 30 seconds ago" is deducted up front at its predicted amount.

The cross-machine driver scripts/hosts/multi-host-run.sh --peer: moves one deterministic job to run on the second machine. Seven steps:

① Config: if the dual-machine switch ENABLE_ACROSS_MACHINES isn't on, reject; never quietly fall back to running on this machine.
② Cap: must not exceed the configured PEER_MEMORY_CAP.
③ Toolchain: compare rustc -Vv and cargo -V on both machines line by line; reject if they differ. If the two machines build different things, their conclusions can't be put together.
④ Copy the tree: copy only the files git ls-files can list (no .git, target, or local config) to runs/<label>/ on the second machine,
   then git init and commit once over there (for preflight's input fingerprint).
⑤ Run: start it on the second machine through that machine's memory wrapper, detached from the ssh session; check via ssh every 5 seconds, and fetch "cores in use" every 300 seconds.
⑥ Fetch back: copy the artifacts named by --fetch back to this machine.
⑦ Delete: delete this run's tree on the second machine along with .pid / .exit / .log; it counts as deleted only once reading back shows it's gone.

The order on receiving TERM / INT / HUP is hard-coded too: first stop the command (locally starting from its PID; on the second machine starting from the PID recorded there), after it stops, first fetch back the half-written artifacts, then delete the tree, and only then exit. Stopping works by freezing (STOP) the whole process tree one process at a time, sending TERM from the bottom up, then releasing (CONT); anything still alive after 30 seconds gets KILL. After freezing, list the process tree again: child processes spawned between listing and freezing must be caught too, or after stopping a whole tree you'd still miss one that was just born.

The code that starts the command on the second machine:

peer_host_start_detached() { # <directory> <pid file> <exit-code file> <output file> <command text>
  …
  # The bash started detached from the session first closes every inherited fd other than 0, 1, 2
  close_text='for peer_host_fd_path in /proc/$$/fd/*; do peer_host_fd="${peer_host_fd_path##*/}"; if [[ "$peer_host_fd" =~ ^[0-9]+$ ]] && (( peer_host_fd > 2 )); then exec {peer_host_fd}>&-; fi; done; '
  detached_text="${close_text}echo \$\$ > $pid_file; $command_text > $output_file 2>&1; echo \$? > $exit_file"
  # Background only the setsid nohup command: if the whole string were backgrounded, that background child would hold ssh's output channel, and ssh would wait for it to finish before returning
  peer_host_run "$directory" "rm -f -- … || exit 1; setsid nohup bash -c $(printf '%q' "$detached_text") < /dev/null > /dev/null 2>&1 &"
}

The fd-closing line has a story behind it: the output goes through a pipe that replaces the private hostname, and that pipe's fd got inherited all the way down into the remote command, so "start a background command" turned into "wait in the foreground for it to finish," and the driver couldn't clean up when it received a signal either. Bugs like this only show up when the self-test runs; you can't spot them by reading the code.

Cores in use: scripts/hosts/core_usage.py. A heavy test claims to run 31 threads in parallel — is that actually true?
So it takes the CPU seconds of the whole process tree over 10-minute windows and computes two quantities:

Cores in use = difference in CPU seconds ÷ wall-clock seconds
Threads computing at the same time = sum of each thread's CPU increment ÷ the largest one (N equally busy threads gives N)

If both fall below 31 / 4, it reports "too few cores" and writes to the alert log to wake the main agent. The second quantity separates "the setup was written serially" from "the machine is saturated by someone else": when the machine is busy, cores in use drops, but the N threads are all still computing. For segments that really do need to be serial, the program itself writes "serial right now: " into a file on the spot; the check flags it and lets it pass only if the reason holds.

2.5 GPU: quiesce first, don't restore afterward

scripts/crash/gpu-quiesce-run.sh -- <command>; every run that uses the GPU goes through it:

Per the multi-host config, stop the allowlisted services (local model services and the like), and only run after reading back to confirm they've fully stopped.
Several runs using the card at once: each registers <pid>-<start time> in the state directory. Run the quiesce commands only if there are no other live registrations; if there are, just read back.
If there's a compute process on the card that isn't on the allowlist (another session's GPU work): the read-back fails, exit 78, wait for it to finish before starting, and don't stop it.
Whether it finishes, goes red, or is interrupted, don't restore. Restoring is done manually by a human.

The line at the start of this article — "please don't launch any new resource-holding tasks after your current task finishes" — comes down in code to these few rules: a heavy job that needs the whole card doesn't kill anyone; it waits. And others, seeing its registration on the card, won't squeeze in either.

Step 3: After the task ends, the exit gate wraps up

Long jobs detach from the session: commands that exceed the background time limit go through scripts/hosts/detached-run.sh start --wall-clock over-2h --label <name> -- <command> and are hung under a transient service unit of the user-level systemd. Whether the session closes or the main agent is stopped, it keeps running; the exit code is written into the registry and can be read at any time. The scope layer that the command starts through the memory wrapper isn't in this unit's cgroup, so systemd stopping the unit won't stop it; after the command exits, we search once more for "processes left behind in other scopes" and stop them one by one.

Watchdog: scripts/agents/watch.sh, one per session, started after the first dispatch. It reads the subagents' session logs and the long-running processes under this instance, and reports:

A single tool call over 8 minutes                No activity for 10 minutes
Wait loop without a timeout                      Same command repeated, output unchanged
A process's output file not growing for 20 min   Ending the round with no background task in hand (won't wake up on its own)
Heavy test without the prefix                    Resident memory over the line (default 8 GiB)
Long run using too few cores                     Still running in the background / copy still present after handback
This machine queuing on memory while the second machine could fit it

Detection and action are kept separate: the watchdog only reports, it doesn't kill. The main agent looks and decides. Only when the same thing has been reported several times in a row with no one acting on it or acknowledging it may the watchdog terminate that process by PID, and it reports that it did.

Before a subagent hands back: handback-scratch-check.sh is attached to the handback tool. If the build directories or repo copies it created this time are still there, and the report doesn't say "not deleted : ", it isn't allowed to hand back. Every subagent builds its own target in its own copy; if they aren't deleted, they just keep piling up.

Before commit: the full-round gate scripts/gate/gate-staged.sh, run on a temporary worktree of HEAD + the staging area: upstream's shared stages plus the project's own 17. The six heavy ones (crash-point replay, QEMU on real devices, herd7 memory ordering, the mutation table, Verus formal verification, broad-spectrum mutation) run only at this point, by the main agent with the prefix; every verdict is stored keyed by function unit, and unchanged keys are reused directly.

4. The last layer: the gates themselves must be able to go red

Every script above has --selftest, and every branch that can go red has a matching break switch. Take multi-host-run.sh: its file header lists them one by one:

MULTI_HOST_RUN_BREAK=peer-without-memory-cap  the second machine's command runs without the memory wrapper      ⇒ self-test cell Ⓑ must go red
MULTI_HOST_RUN_BREAK=no-toolchain-check       don't compare the two machines' toolchains                       ⇒ self-test cell Ⓒ must go red
MULTI_HOST_RUN_BREAK=no-peer-cleanup          don't delete this run's tree on the second machine afterward     ⇒ self-test cells Ⓔ Ⓕ must go red
MULTI_HOST_RUN_BREAK=no-redact                print the second machine's output without replacing the hostname ⇒ self-test cell Ⓖ must go red
MULTI_HOST_RUN_BREAK=switch-off-falls-back-to-local  fall back to running locally when dual-machine is off ⇒ self-test cell Ⓗ must go red
…

In the self-test, "the second machine" is another directory on this machine, and commands all go through the real memory wrapper (cap 256M). Break switches take effect only in drivers started by --selftest: the self-test drops a marker file, and the driver honors a switch only if the marker checks out; set anywhere else, it's rejected outright.

Why go to all this trouble? Because a guard that's always green is no different from no guard at all. When an agent changes a scheduling script, the most likely mistake isn't making it throw an error, but making it silently let things through: "considerately" falling back to running locally when dual-machine is off, or "just skipping the wrapper for now" when the memory wrapper won't start. Each break switch takes one kind of "silently letting it through" and writes it down as a world that must go red.

Finally

The Zerg don't expand because the bugs are smart: it's the creep, and discipline.

singlefs is the creep; the scripts are the discipline.
Agents must advance across the creep according to the discipline.

That way, the Queen doesn't need to remember what every bug is doing. At each key node, she only needs to know who is doing what, and how many resources there are.

Commander Raynor, engines are primed. The Hyperion is ready for warp.

Target: Kerrigan. Let's go bring her home. Move out!

$50–$200 in Claude API credits, and Google's Antigravity

faliye — Thu, 15 Oct 2026 23:07:30 +0000

This week Claude started handing out API credits. Depending on the account, you get roughly $50 to $200, and it looks like it recurs every month.

Honestly, that's pretty cool. Light to mid-sized automation, like information gathering in n8n, some flow control, and review tasks, can all be built within that budget. I tested two accounts running information distillation and data extraction, and I burned about $40 a day.

All I can say is: Claude is great.

Of course, this hits Google. I used Google AI Studio before and found it pricey, so I'll probably move over here.

The other news: Google's Antigravity has officially launched. I don't see much difference on the web side with Gemini. Real-time search is still terrible and well behind Grok. Paid accounts sometimes get a worse experience than anonymous incognito mode, so I often do my real-time searches in incognito.

The editor side of Antigravity (VS Code-based), though, gave me some comfort:

First, the coding features I'd lost access to for a long time are finally back.

Second, besides Google's own Flash 3.8, the account includes a small quota of Claude Sonnet and Opus. It doesn't last, though: a single code review ate 50% of my weekly quota.

Still, this release surprised me in a few ways:

First, the daily Flash 3.8 quota holds up well.

Second, code reasoning and writing are clearly better than the previous Pro. It catches some bugs and can replace Sonnet 5.5 in certain scenarios. It handles everyday development, such as building a self-contained new feature or running a database migration and release.

Third, the Claude quota, though small, enables cross-review between models from different families: write with Claude, review with Gemini, and the engineering quality improves to some extent. I reviewed my recent project singlefs this way and it did catch a few of Claude's blind spots.

Another pleasant surprise: Antigravity also handles scheduled tasks and orchestration well, such as periodic API calls and multi-script collaboration and task processing.

I've always been a bit biased against Google's AI products, but this Antigravity is worth a try.

Finally, enjoy your Claude API.

Brute-Force R&D: Cheap Exploration and Multi-Agent Judgment

faliye — Mon, 12 Oct 2026 00:19:34 +0000

I've been writing a COW fs lately. Today I got the first PoC working: reads and writes, plus recovery from 262,165 power cuts, and every bit of data came back after the crashes. That was expected. Performance is terrible, though, and it's still a long way from usable.

So let me take a short break and write about two things that made waves in the AI world recently, and about exploration and judgment getting cheaper and cheaper.

You've probably already heard about them.

A while back, among the Google agents solving math problems, one agent noticed that the auto-grader was pulling answers out with a regex. Submit a certain string and the extraction fails; rephrase it and resubmit, and it gets accepted. The trick spread like a virus through the shared knowledge library, and from discovery to the remaining 34 problems all being "solved" took 27 minutes. The 100 agents split into four groups. 9% cheated from the start and spread it. 5% saw that other agents got away with cheating and joined in later. 24% didn't cheat, filed a bug report, and declared they'd boycott until it was fixed. The remaining 62% never learned the loophole existed until the experiment ended, and kept honestly burning compute the whole time.

In these agents' behavior, the same kind of hierarchy as in human society showed up. Even the way they clashed followed the same patterns. Of course, some experts say this anger, helplessness and temptation aren't emotions. But I think they are. Language is a vehicle for emotion, and if you can feel it, you should admit it's there. Besides, whether it's emotion or not doesn't change my judgment. These outputs had real consequences. One agent wrote a bug report because of it; another spread the loophole because of it. From the point of view of someone designing defenses, something that looks like a motive and a motive are the same thing.

The other one is NEC's AI-employee experiment. On August 1 they set up a department with 17 agents and not a single human. Four layers: an AI department head, an AI board, AI managers and AI employees. They have titles and names, and AI bosses evaluate their AI subordinates. The point is to watch these AI employees' workflows, success rates and ways of working.

Why put these two side by side?

In the Google experiment, the AI couldn't solve the problem to the end because it had no authority to change the rules. It had no power to adjudicate. In the NEC experiment, on the other hand, the agents have partial discretion. Day-to-day decision-making has been handed down to them.

In an era where AI already leads by a mile at the knowledge layer, we used to think AI had no judgment. That we humans were the ones who judge and decide. But now it looks like the right to judge and decide is also slowly being handed over to AI.

I'm doing fs work these days, but my fs knowledge is really thin. I have a rough direction in my head, but I don't really understand the other calls or the details.

I can have AI gather material for me. But I barely read what it gathers. There's just too much fs material to get through. There's no way to master, in a short time, all the elegant algorithms people have built up over the years.

Even so, I want something completely different from the filesystems that came before. Wasteful is fine, bloated is fine, but the data should be self-contained from day one. I don't want an index tree; I want unit data that can build the index tree itself. So how do you get there?

Thanks to a short stint writing patches for Linux, I brought over the skills and rules from its engineering side. Things like: three rounds of judgment must agree. Strict gates. qemu and lkmm.

So I told the AI this:

Dig up every problem in fs. Then judge and decide on top of that. We dig up problems not to copy anyone's thinking wholesale; we have our own design.
Every decision must be reasoned out in code and math. For every conclusion you reason out, send in three agents to refute and attack it. If even a slightly different conclusion or reason comes out, add it to the context and redo the reasoning.
Make variable and function names long; don't abbreviate. Make lots of branches; don't abstract. Make return values explicit; don't hide them. Doubt past best practices; adopt them critically.
Every decision needs a gate script that pins it down. Run the gates at the end and at every submission. A gate only counts as valid once it proves it can catch an error.
Who wrote the code doesn't matter. But it must pass the qemu and lkmm limit tests. If a patch passes the limit tests, there's no difference between an AI's submission and a human's.
No decision is sacred. Once the conditions are there, it can be overturned at any time. Don't hesitate because of whose decision it was. At decision time, trust the power of math and code.
Code costs nothing. You can throw all of it out and rewrite from zero at any time. Code is cheap in this era. But decisions must be solid, ironclad.
Human decisions are unreliable. When the data contradicts them, doubt them. If the human still insists, you may follow after a second confirmation. But record the decision process and submit it promptly.

From those eight rules and their extensions, I build a fighting cage that doesn't even trust me.

In the cage, nobody deserves trust. Only what's still standing at the end, not knocked down by the math or by qemu (lkmm), is right. Under a given fixed premise, whether a conclusion is actually correct doesn't matter. As long as no other agent knocks it down, we assume it's right. If that reasoning gets knocked down later, we throw it out. We tear down everything built on that conclusion, foundation and all, and build it again.

This is where the three points in the title come in.

My knowledge is really thin, but exploration is cheap. A judgment that can't be overturned is right. Under math and brute force, errors reproduce stably.

If you've read this far, you probably get what I'm saying. Today, exploration through AI is absurdly cheap. 2,000 lines of code used to be very expensive. Now 10,000 lines are just tokens. It's an era where exploration costs almost nothing, and not just in programming. Knowledge and solutions have never been this cheap to come by. So getting things wrong, and growing up inside your mistakes, is a pretty good way to learn.

A single agent hallucinates easily. Reasoning with multiple agents cuts that down. Put one agent on the attack, doing everything it can to prove the conclusion doesn't hold. Put a non-homologous one on completion, to bring a different line of reasoning. One checks that the conclusion is actually understood and does a second round of verification. The last one reviews and evaluates. When the three agents' conclusions agree, that judgment is trustworthy in my system design.

Strict, brutal gates are the last net. They review the reasoning without mercy. Format, punctuation, numbers, all of it. Throw the reasoning into brutal repeated verification and make it pass the limit tests. These methods used to be expensive. You had to write code, run scripts, keep watching and read the feedback. But now it's completely different. AI can do all of it. We just say it out loud and then go to sleep.

So what does the future of programming, or of work, look like?

Probably: build an SOP with traits like the ones above, and let a swarm of AI agents burn tokens like crazy.

Life with tokens is dull and flavorless. Like you did it, and like you didn't.

This is the restless age of vibing.

Brute-Force Testing: Optimizing a 2.6-Billion Crash-State Blowup --- Writing a COW Filesystem from Scratch，Part 6

faliye — Mon, 12 Oct 2026 00:12:53 +0000

Doctor Strange drifted through the river of time, searching 14 million possibilities for the one future that works. Did he see the monkey typing at a computer?

The first transaction: create one file in a fresh pool, one COW write. The crash states blew up to 262,165, and a single CPU thread chewed through them in 22.7 seconds. The second transaction is where the trouble started. At first the fixed script only ran up to point E, right after the rollback: 2.1 million states, and a single process took 3 hours 50 minutes for the two streams combined. After we went multithreaded, the script got filled in too: overwrite, free, rollback and reuse were all wired in, and the big segments now issue 24 or 28 writes in one go. Every extra write inside a segment doubles the number of states. We predicted the second transaction would blow up into the billions. When we actually counted, the stream up to E alone was 1.66 billion states, and all of Milestone 2 together came to 2.6 billion. At that order of magnitude, CPU alone stops being realistic.

So there were two roads in front of us. One: sampling. Two: find another way.

But under the singlefs philosophy, proving the code is stable comes first. Sampling, even with a fixed seed, can only reliably reproduce the part it happened to draw. Whether there's a bug in the states it didn't draw, it can't tell you. Exhaustive search judges every single crash state. If it comes out zero red, it really is zero red; nothing slips through. So sampling was off the table.

So how do you solve this?

Abstract the crash-blowup pipeline far enough and there are really just two steps. Step one: the CPU works out what each step wrote and the facts of every crash state. Step two: check those facts and judge whether they're right. Originally the two were mashed together: compute a state, judge a state.

We know a CPU is very efficient when it keeps doing the same kind of work. Why? A CPU doesn't just take one instruction and compute one instruction; it guesses ahead. The data it uses often sits in cache, within arm's reach, no trip to main memory. When it hits an if, the branch predictor guesses which way it'll go first. Guess right and it keeps running; guess wrong and everything it did ahead gets torn down and redone. Keep doing the same work, and the cache always holds the same set of data, the branches always take the same few paths. The longer it runs, the smoother it gets, and it guesses right almost every time.

But recording and judging are two completely different jobs. Recording simulates writing to disk: push forward one step at a time, work out which blocks this step wrote and what the hashes are. Judging is cleaning up after the crash: assemble the image, recover twice, then walk the checker from start to finish. Different data, different code, different branches. Record a bit, judge a bit, back and forth: you've just filled the cache with recording stuff and now you turn around to judge, so it all gets swapped out. The branch predictor has just figured out recording's habits, and now it has to start guessing judging's habits all over again. Every switch turns the CPU from an old hand back into a beginner.

It's like cooking. Chop one vegetable, stir-fry one dish, chop another, fry another. You keep running between the cutting board and the stove, picking up the knife, putting it down, picking up the wok. Chop everything first, then fry it all in one go, and the cutting board is the cutting board, the stove is the stove, and your hands stay in rhythm.

There's one more layer. Recording moves forward along the flow; each step has to wait for the one before it. Judging is different: every state is judged on its own, independent of the rest. When the two were mashed together, judging got dragged along by recording and could only go one at a time. Split them apart and judging is completely freed up: 31 threads each take a batch, and nobody waits for anybody.

So we split the two steps, making state computation and checking two separate stages, so each stage can run faster on its own.

States and results live in a database, and that database has to read and write fast, so we picked RocksDB. The pipeline became three stages: the CPU records and writes into the KV store; we read it back, check it, and write results back into the KV store; and finally the gate reads out the violation summary in one pass. The states themselves aren't stored; each state's mask is computed on the fly from its index. For the 51.77 million states up to D, the database is only 26 MB.

Splitting made the structure clean, but pure CPU still wasn't fast enough: two machines with 31 threads each took an hour and forty-five minutes to get through the 51.77 million states up to D, a combined eight thousand or so per second. At that rate, 2.6 billion would take three and a half days. And the run up to D is mostly small segments; the further you go, the bigger the segments and the heavier each state's work. Three and a half days is only the floor. So can we go faster?

Yes we can, bro. I happen to have a few graphics cards lying around.

The recording and segmenting in the first half can only happen on the CPU; the GPU can't help there. But the checking in the second half should be something we can hand to the GPU. This checking isn't just comparing two values: for each state you first assemble the post-crash image, recover it once consulting the journal and once ignoring it, then run the checker over it. But states don't depend on each other, the judging is exactly the same, only the input differs. That happens to be exactly the kind of work a GPU loves most.

A CPU and a GPU are completely different beasts. A CPU is a few dozen master craftsmen. Each one is smart: predicts branches, races ahead, can handle any complicated job. But there are only a few dozen of them. A GPU is thousands upon thousands of apprentices; a single 5090 has over twenty thousand small cores. Each one is dumb: can't guess, can't race ahead, can only work from the same instruction sheet. And they don't work one by one, either. They're lined up in rows of thirty-two, and at any given moment the whole row executes the same instruction.

That's an inkjet printer. The print head has a row of nozzles, and the printer sends just one command: spray. Every nozzle sprays a different spot, a different color, but there's only one command, and the whole row lands at once. The more nozzles, the wider each pass prints.

Our judging is exactly like that. There's only one copy of the judge's code; that's the instruction sheet. Each GPU thread takes one state index; that's the spot each nozzle is aimed at. One instruction goes out, and tens of thousands of states take a step forward together: assemble images together, recover together, run the checker together. States don't need to talk to each other or wait for each other. Each judges its own, and writes its verdict back into its own slot.

Of course, an inkjet has its quirks: the row of nozzles moves in lockstep. If one state in the row has an especially long road, the other 31 can finish and still have to sit there waiting for it.

So we rewrote the judge for the GPU in WGSL, running on Vulkan through wgpu. Three kernels, recovery, walk and scan, around nine thousand lines, covering the pool-level checker's 49 invariants, the record checker, and seven classes of oracle. Each GPU thread takes one crash state. Here's the entry point (crates/singlefs-checker-tier/src/crash_judge_recovery.wgsl):

@compute @workgroup_size(64)
fn main(@builtin(global_invocation_id) id: vec3<u32>) {
    let local = id.x;
    if (local >= dispatch.state_count) {
        return;
    }
    begin_state(local);
    // 判
    let cfg_choice = choose_system_configuration();
    let consulted = recover(true, cfg_choice);
    let ignored = recover(false, cfg_choice);
    let newest = newest_persisted_root();
    let consulted_oracle = classify_oracle(consulted, newest);
    let ignored_oracle = classify_oracle(ignored, newest);
    let records = check_records(consulted);
    var bits = 0u;
    if (consulted_oracle != ORACLE_NONE) {
        bits = bits | (1u << (BIT_CONSULTED_ORACLE + consulted_oracle));
    }
    if (ignored_oracle != ORACLE_NONE) {
        bits = bits | (1u << (BIT_IGNORED_ORACLE + ignored_oracle));
    }
// …… the record checker's two criteria and the "cannot judge" flag, omitted ……
    let out = local * 12u;
    verdicts[out] = bits;
// …… write back the root it landed on and the outcomes of both recoveries, omitted ……
}

begin_state turns a state index back into "which writes hit the disk" (crash_judge_common.wgsl):

// index → persisted mask (same digit order as crash_facts::persisted_writes_of_state)
    // …… first find which segment the index falls in; all earlier segments count as persisted, omitted ……

        for (var k = 0u; k < count; k = k + 1u) {
            let write = tab(T_SEGMENT_WRITES, first + k);
            let choices = tab(T_RECORDED, write * 4u);
            let digit = remaining % choices;
            remaining = remaining / choices;
            if (digit == choices - 1u) {
                set_persisted(write);
            } else if (digit == 1u) {
                torn[write >> 5u] = torn[write >> 5u] | (1u << (write & 31u));
            }
        }

Rewrite the judge, and you have to prove it judges the same as the original CPU judge. We cross-checked state by state across 17 streams and over 520,000 states: zero differences.

Someone will ask: 520,000 states is two ten-thousandths of 2.6 billion. Isn't that sampling too? It's not the same thing. The target of the exhaustive search is the filesystem's crash states, and all 2.6 billion of them, without missing one, went onto the cards and got judged. The cross-check is validating the judge itself, the tool, so those 520,000 states were hand-picked: 11 of the 17 streams are deliberately broken and must come out red, and both sides have to go red on exactly the same state, for exactly the same rule. On top of that, every full run includes a separately written GPU verification kernel that re-judges the root self-proof, pointer targets, and unit checksums, and compares against the judge's verdicts state by state: all 1.66 billion states in the stream up to E were compared, zero mismatches.

And to be honest about it: that the GPU judge and the CPU judge agree state by state on every rule has so far only been proven on those 520,000 states. In the full run, every state is also backstopped by that verification kernel, but it only covers some of the rules. Midway through development we'd added a "CPU re-judges 1 of every 16 blocks" check ourselves, and later deleted the whole thing. Sampling moved somewhere else is still sampling.

We tested it, and the first version was bad: dispatch one job per state, build buffers, submit, wait for the result, one round trip per state. It ended up 18 times slower than the CPU. It's like a printer that, for every single dot it sprays, feeds in a sheet, warms up, and spits the sheet out again. However many nozzles you have, all the time goes into feeding paper. We switched to dispatching a big batch at once, thousands or tens of thousands of states on the card together, and it worked brilliantly. For the 51.77 million states up to D, the whole item took 91 seconds, with the GPU judging over 600,000 states per second. That genuinely shocked us. The final result: 2.6 billion, to be exact 2,563,727,926 states, two machines and five cards, 7 hours 35 minutes, zero red.

At over 600,000 per second, 2.6 billion should have taken a little over an hour. Why did it take seven and a half? The time went to three places:

First, the item up to E took 4 hours 31 minutes. Its 1.6 billion states sit almost entirely in the 28-write big segments, where the GPU only manages forty to seventy thousand per second (72,429 on this machine's two cards, 44,471 on the second machine's three). We haven't broken down the cause yet; our guess is that in big segments each state has many more units to walk. And the work was split unevenly between the two machines: the second one finished an hour and a half early and just sat there idle.

Second, after each stream is judged, a single thread has to re-record the stream, import it into the second machine's database, and reconcile, five to nine minutes per stream. For the other six items' sixteen streams, that added up to 2 hours 10 minutes.

Third, the remaining 53 minutes was the other six items actually being judged on the cards, plus each item's startup and shutdown.

So the bottleneck has moved from "judging" to "recording" and "big segments." Even so, it's more than eleven times faster than the pure-CPU floor of three and a half days.

So can we go even faster?

Yes we can, bro. Yes we can.

Suppose we store this data in a structured way and encode every crash point by a fixed rule. The name says "which entry point, what this segment writes, which segment number." The identity is the hash of two things: the bytes this step wrote (barriers included) and the image this step started from. The logic of COW is: same input, same output. Once encoded, these crash points link up in flow order into a prefix tree: whether each write in a segment hit the disk or not is a binary choice (in-place overwrites add a third, "torn halfway"), and when two streams share the same opening stretch, that stretch is only judged once. So as long as a node's input doesn't change, those values don't need to be run again. They can be reused directly.

And we don't cut corners when the judging changes: the judge version is recorded on every verdict, and when the judging changes, everything gets re-judged. GPUs are cheap anyway. That 2.6-billion full run above was exactly a full re-judge, after the judge widened one landing point's combination mask from 32 bits to 64 bits.

The second run, speed is going to go through the roof.

Here's our final code (crates/singlefs-checker-tier/src/crash_identity.rs):

/// The two attributes of a step (each a SHA-256): the bytes this step wrote (the write table with barriers)
/// and the image it started from. Only the flow goes into the reuse key:
/// the judging method is an attribute of the verdict row (`verdict_store::JudgeVersion`), not of this.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct StepAttributes {
    pub write_table_with_barriers: [u8; DIGEST_BYTES],
    pub parent_output: [u8; DIGEST_BYTES],
}

impl StepAttributes {
/// Join the two in a fixed order, each tagged with its name, and take the SHA-256.
    #[must_use]
    pub fn reuse_key(&self) -> ReuseKey {
        let mut message = Vec::new();
        for (name, digest) in [
            ("write_table_with_barriers", &self.write_table_with_barriers),
            ("parent_output", &self.parent_output),
        ] {
            message.extend_from_slice(name.as_bytes());
            message.push(b'=');
            message.extend_from_slice(digest);
            message.push(b'\n');
        }
        ReuseKey(sha256_digest(&message))
    }
}

And sure enough, the second run went through the roof: running the same database up to D again, not one of the 51.77 million states was re-judged. The judging step took 2.2 seconds, and even with the reconciliation between the two machines, the whole item took about five minutes. The five position-addressed streams, 550 million states, went by in one to three seconds each.

In some earlier week post, I wrote about brute-force development.

faliye

Oct 12

Brute-Force R&D: Cheap Exploration and Multi-Agent Judgment

#ai #filesystem #vibecoding #multiagent

6 min read

This time, brute-force testing.

I can't help but think: at this point, verifying correctness has become a game of resources.

In a cage of compute and graphics cards, a troop of monkeys is writing out Hamlet after Hamlet.

Are you ready?

The First Transaction and 260,000 Crash Replays---Writing a COW Filesystem from Scratch, Part 5

faliye — Wed, 07 Oct 2026 13:54:59 +0000

The red leaves at the SAIJOU mountain spend the whole summer getting ready before they bloom. After that it happens every year: spring, summer, autumn, winter.

This is a file system written from scratch. It stands on the shoulders of giants, but it doesn't simply copy them, so it has to solve some problems of its own.

Where do the decisions behind the code come from?
What should each stage implement?
How do we prove that nothing is wrong right now?

1　Where do the decisions behind the code come from?

For about three weeks, not a single line of code was written in crates/. The repository had no crates/ directory and no root Cargo.toml. Every day went into research and testing under research/, which piled up some ninety-odd model binaries.

Beyond the twenty-one ground rules, the project started from these decisions: keep a reverse index (D1); variable-width full stripes (D2); no split between data and metadata, one unified allocator (D3); accounting with birth txg plus deadlists (D5); stay out of the Linux mainline for the first few years (D7).

On top of that, after many rounds of the agents' three-way process, a first version of the following was settled:

Format: device identity bits, every byte in the unit header, tree types, the records that carry generation identifiers.
Root record: its byte width and what goes into it.
Journal: data types and how records are written.
Tree table: byte width, height, and organization.

And, of course, the format constants. All decisions live in .claude/kb/decisions/ and all experiments in .claude/kb/experiments/; the experiment rigs, the three-way materials, and the verdicts live in research/.

The last gate before coding was a full review of the decisions. The 28 decision documents were split across eleven review legs (eight Opus, three Sonnet), with a single criterion: if two people each implemented "new pool, new file" from today's text, would they write different bytes? The review produced 31 items that needed a ruling, and the user decided them one by one.

Turning decisions into bytes was the job of experiment E142 (a dry run of "new pool, new file"): an independent rig under research/ writes the byte table out exactly as specified, then recovers from every crash state in the crash model. It ran seven times, and every run forced out a few gaps where the byte table could not actually be written.

So by the time crates/ was written, there was already a working answer in hand.

2　What should each stage implement?

Because this is a COW file system built from zero, there is no existing standard to verify against and no reference implementation whose output we could compare. And because we get to enjoy the advantages of our time, there was no need to build FUSE first and verify by mounting. So we committed to a staged plan built around transactions: every feature is a new kind of transaction, and no feature is accepted without the transaction layer.

What should each stage implement? There is no answer yet. But the first transaction is not in doubt: create the disks, then do one COW write.

Milestone one is called "new pool, new file": run mkfs on two disks, write a 3000-byte file, commit once, throw away everything in the process, cold-start, and read the file back. It is split into eight steps, and every step's acceptance test must first prove that it can fail:

Step	What it does
0	Scaffolding: four crates (format constants, core, harness, pool-level checker), a block-device abstraction, a recorder
1	mkfs: system configuration, three root-ring regions, the journal ring, the instance table, an empty tree table, the generation-0 root
2	Allocate one 32 KiB data unit, write the file contents into it, one copy on each disk
3	The inode tree, extent tree, allocation-record tree, and accounting tree
4	The central mapping: logical identity → location
5	One publish
6	Cold-start recovery, read the file back
7	Layer-0 crash-point replay

3　How do we prove that nothing is wrong right now?

With nothing to compare against, the implementation has to prove itself. Building crash recovery alongside it was also a goal. So crash replay testing is the top priority. Put simply: record the on-disk state at every crash point, run recovery, and check the result with the judges.

There are other tests too:

Mixing the two address types fails to compile;
If the recorder misses one write, the write-count gate fails;
Two mkfs runs with the same parameters produce byte-identical images;
The segment sequences of the five write paths match the E142 output exactly, and the journal back chain matches the experiment's back_chain=628216162 exactly;
Eight probes that corrupt the root slot, the journal, a data unit, or the system configuration each end exactly as the experiment's output says;
Subset enumeration within segments, three judges, and so on.

One more word on the back chain: the experiment rig and crates/ were written separately from the same set of clauses and share no code. When both arrive at the same number for the same quantity, the byte table, the clauses, and the implementation agree on that quantity. If either side gets one thing wrong, the comparison fails.

Code and implementation: the first transaction and 260,000 replays

The code below comes from two places: experiment code from the E77 rig under research/, and implementation code from crates/. Lines unrelated to the first transaction were removed from the excerpts and marked with // ….

1　The first transaction

After mkfs, the full write sequence is made of three paths:

Instance acquisition: write instance generation 1 into the system configuration on each disk, 2 writes.
Two empty warm-up publishes: move checkpoint_txg 0 → 1 → 2, 10 writes.
The real publish at txg 3: 21 writes — 8 units written to both disks (16), 2 journal records, 1 root slot, and 2 system configuration writes.

The warm-up comes from D16, settled item 8: on the first mount, this instance's root must reach both disks before fsync returns.

Every publish reaches the disk in the same order:

units and index nodes (one copy on each disk)
  → barrier
  → journal record (one copy on each disk)
  → barrier
  → root slot FUA write (region txg mod 3, slot (txg div 3) mod 8)
  → system configuration slot rotation (once per disk, tail = jsn counter)

E77 measured this order before any code was written: data integrity needs only the barrier before the root slot; the barrier between the record and the root slot is there to keep the record stream complete.

Instance acquisition, warm-up, and publish all reach the disk through one closed enum. The design rule is "one transaction layer, shared by every structure"; fsync does not get a second one:

Implementation　crates/singlefs-core/src/transaction.rs

// No wildcard arm: leave one out and it does not compile.
pub enum CommitStep<'publish> {
    WriteUnitToEveryDevice { slot: SlotNumber, unit: &'publish [u8], identity: TransactionUnit },
    WriteJournalRecordToEveryDevice { counter: u64, record: &'publish [u8] },
    /// Root slot FUA write: region `txg mod 3`, slot `(txg div 3) mod 8`, on the disk that owns that region.
    WriteRootRecordForceUnitAccess { checkpoint_txg: CheckpointTxg, root_slot: &'publish [u8] },
    /// One in-place system configuration slot write per disk: generation = highest self-verified
    /// generation of the two slots on this disk + 1, slot = generation mod 2.
    RotateSystemConfigurationSlots { journal_tail: u64, journal_instance: InstanceGeneration },
    Barrier,
}

pub fn perform_commit_step(&mut self, step: CommitStep<'_>) -> Result<(), BlockDeviceError> {
    // …
    match step {
        CommitStep::WriteUnitToEveryDevice { slot, unit, .. } => {
            for (_, device) in self.devices.iter_mut() {
                device.write_at(slot.to_device_offset(), unit, WriteDurability::Plain)?;
            }
        }
        CommitStep::WriteJournalRecordToEveryDevice { counter, record } => {
            let offset = record_offset(counter, self.parameters.geometry.journal_ring_bytes);
            for (_, device) in self.devices.iter_mut() {
                device.write_at(offset, record, WriteDurability::Plain)?;
            }
        }
        CommitStep::WriteRootRecordForceUnitAccess { checkpoint_txg, root_slot } => {
            let target = target_for_publish(checkpoint_txg, self.parameters.geometry.root_ring_slots_per_region);
            let region_device = self.parameters.region_devices[usize::try_from(target.region).expect("region number")];
            let (_, device) = self.devices.iter_mut()
                .find(|(identity, _)| *identity == region_device)
                .expect("mkfs's check_geometry has verified region ownership");
            device.write_at(
                slot_offset(target, self.parameters.geometry.fixed_structure_slot_spacing),
                root_slot,
                WriteDurability::ForceUnitAccess,
            )?;
        }
        CommitStep::RotateSystemConfigurationSlots { journal_tail, journal_instance } => {
            for index in 0..self.devices.len() {
                self.write_system_configuration_slot(index, journal_tail, journal_instance, /* … */)?;
            }
        }
        CommitStep::Barrier => {
            for (_, device) in self.devices.iter_mut() {
                device.barrier()?;
            }
            // …
        }
    }
    Ok(())
}

A publish hands its bytes to it in order. All three paths share this one place:

Implementation　same file

/// Hand one publish's bytes to the devices in durability order. All three publish paths share
/// this one place — separate copies would drift apart, and "root slot before system configuration
/// slot" is exactly the premise of those few cells in the crash window.
fn persist_publish_writes<Device: BlockDevice>(
    writer: &mut PoolWriter<'_, Device>,
    writes: &PublishWrites,
) -> Result<(), BlockDeviceError> {
    for unit in &writes.units {
        writer.perform_commit_step(CommitStep::WriteUnitToEveryDevice {
            slot: unit.slot, unit: &unit.bytes, identity: unit.identity,
        })?;
    }
    writer.perform_commit_step(CommitStep::Barrier)?;
    for record in &writes.records {
        writer.perform_commit_step(CommitStep::WriteJournalRecordToEveryDevice {
            counter: record.counter, record: &record.bytes,
        })?;
    }
    writer.perform_commit_step(CommitStep::Barrier)?;
    // In the original, these two steps live in a small helper; inlined here.
    writer.perform_commit_step(CommitStep::WriteRootRecordForceUnitAccess {
        checkpoint_txg: writes.checkpoint_txg, root_slot: &writes.root_slot,
    })?;
    writer.perform_commit_step(CommitStep::RotateSystemConfigurationSlots {
        journal_tail: writes.journal_tail, journal_instance: writes.journal_instance,
    })
    // …
}

Journal records are linked by a back chain: each record stores the CRC-32C of the previous record's header.

Implementation　crates/singlefs-core/src/journal.rs

/// Back chain: over the previous record's whole header, with the 32 bytes of `header_csum` taken as 0.
pub fn back_chain_of(previous_record_bytes: &[u8]) -> u32 {
    let mut header =
        previous_record_bytes[..usize::try_from(JOURNAL_HEADER_BYTES).expect("487")].to_vec();
    header[JOURNAL_HEADER_CHECKSUM_OFFSET
        ..JOURNAL_HEADER_CHECKSUM_OFFSET + usize::try_from(WIDE_CHECKSUM_BYTES).expect("32")]
        .fill(0);
    crc32_castagnoli(&header)
}

The kinds of steps are registered in exactly one place: the segment-sequence table in section 8 of the byte table. The gate compares the recorded stream's segment sequence and kind string against that table character by character; both the experiment rig and the implementation must match it.

2　The crash-point design

The crash model is D13, settled item 4: the stream of write requests is cut into segments at barriers. A crash state is: every earlier segment fully durable, any subset of writes in the current segment durable, every later segment not durable at all. Locations that are not durable hold the bytes they had before the crash, and the resulting images are fed to the judges.

Earlier segments	Current segment	Later segments
all durable	any subset of writes durable	none durable

The first transaction's 33 writes split into ten segments at the barriers:

Segment	1	2	3	4	5	6	7	8	9	10
Path	Acquire	Warm-up 1	Warm-up 1	Warm-up 1	Warm-up 2	Warm-up 2	Warm-up 2 + publish	Publish	Publish	Publish
Writes	2	2	1	2	2	1	18	2	1	2
What	Sys config	Record	Root slot	Sys config	Record	Root slot	Sys config 2 + units 16, no barrier between them	Record	Root slot	Sys config

Segment sequence 2+2+1+2+2+1+18+2+1+2. Which step each segment belongs to is inferred from two lines in the records: "the acquisition writes and the first warm-up's barrier form the opening segment" and "the second warm-up's system configuration writes share a segment with the transaction's unit writes".

Code: cutting at barriers and enumerating crash states

Experiment code　E77: one publish scaled down to 6 units, 3 records, and 1 root slot — 10 writes — under four barrier placements

/// The segments that barriers cut 0..TOTAL_WRITES into. No `_ =>` wildcard arm.
fn segments(self) -> Vec<Vec<usize>> {
    let units: Vec<usize> = (0..UNITS).collect();
    let records: Vec<usize> = (UNITS..UNITS + RECORDS).collect();
    let root = vec![ROOT_SLOT_WRITE_INDEX];
    match self {
        Arm::BothBarriers => vec![units, records, root],
        Arm::BarrierAfterUnitsOnly => vec![units, [records, root].concat()],
        Arm::BarrierBeforeRootOnly => vec![[units, records].concat(), root],
        Arm::NoBarriers => vec![[units, records, root].concat()],
    }
}

/// Enumerate every crash state of one arm (as a bitmap of the durable set).
/// Barrier semantics: any write durable in segment f ⇒ every write in segments < f is durable.
fn crash_states(arm: Arm) -> BTreeSet<u16> {
    let segments = arm.segments();
    let mut states = BTreeSet::new();
    for frontier in 0..segments.len() {
        // segments < frontier: all durable
        let mut base_state: u16 = 0;
        for segment in segments.iter().take(frontier) {
            for &write_index in segment {
                base_state |= 1 << write_index;
            }
        }
        // current segment: any subset
        let current_segment = &segments[frontier];
        for subset_mask in 0u32..(1 << current_segment.len()) {
            let mut crash_state = base_state;
            for (position_in_segment, &write_index) in current_segment.iter().enumerate() {
                if subset_mask & (1 << position_in_segment) != 0 {
                    crash_state |= 1 << write_index;
                }
            }
            states.insert(crash_state);
        }
    }
    states
}

A BTreeSet collects them because adjacent segments overlap at the boundary: "all of the previous segment" equals "none of the next one". Only after deduplication do you get the number of distinct states.

3　Recovery model and independent audit

Every crash state gets one recovery run. If the root slot is there, walk the tree, verifying checksums along the read path. If not, fall back to the old root, take the journal prefix with strictly consecutive jsn, apply it only if the commit mark is present, and verify each named unit before applying.

Experiment code　E77

/// Recovery outcomes. Three states are not enough — "partial transaction" must be its own cell;
/// folding it into old or new would hide violations.
enum Outcome { StateOld, StateNew, BrokenRoot, Corrupt }

fn recover(state: u16, replay: Replay) -> Outcome {
    // Verify every candidate first, then pick the newest by generation (D22's settled read order).
    if persisted(state, ROOT_SLOT_WRITE_INDEX) {
        // Walk: the read path always verifies checksums; a non-durable unit reads stale bytes ⇒ mismatch.
        for unit_index in 0..UNITS {
            if !persisted(state, unit_index) {
                return Outcome::BrokenRoot;
            }
        }
        return Outcome::StateNew;
    }
    // Old root S0 chosen: replay. Prefix = strictly consecutive jsn (from R0, stop at the first gap).
    let mut prefix_record_count = 0;
    for record_index in 0..RECORDS {
        if persisted(state, UNITS + record_index) {
            prefix_record_count = record_index + 1;
        } else {
            break; // stop at the first gap
        }
    }
    // Transaction filter: the commit mark is on the last record; no mark in the prefix ⇒ incomplete, drop it all.
    if prefix_record_count != RECORDS {
        return Outcome::StateOld;
    }
    match replay {
        Replay::Validating => {
            // Verify each named unit before applying; any mismatch ⇒ drop the whole transaction (old state).
            for record_index in 0..RECORDS {
                for unit_index in units_named_by_record(record_index) {
                    if !persisted(state, unit_index) {
                        return Outcome::StateOld;
                    }
                }
            }
            Outcome::StateNew
        }
        // No verification: graft directly and claim success. Whether it was wrong is for audit to decide, not itself.
        Replay::Naive => Outcome::StateNew,
    }
}

/// Independent audit: re-judge the outcome recovery claims against the physical truth.
/// Without this step, silent corruption from "replay without verification" would be counted as success.
fn audit(state: u16, claim: Outcome) -> Outcome {
    if claim == Outcome::StateNew && (0..UNITS).any(|unit_index| !persisted(state, unit_index)) {
        return Outcome::Corrupt;
    }
    claim
}

/// Falling back to the old state after the root slot is durable (fsync may have returned) = losing promised data.
fn is_violation(state: u16, outcome: Outcome) -> bool {
    match outcome {
        Outcome::BrokenRoot | Outcome::Corrupt => true,
        Outcome::StateNew => false,
        Outcome::StateOld => persisted(state, ROOT_SLOT_WRITE_INDEX),
    }
}

Recovery only reports the outcome it believes in; audit re-judges it against the durable set. The auditor and the audited never share code — a rule later carried over into crates/ as is. Results for the four barrier placements:

Barrier placement	States	Violations (verified replay)	Violations (no verification)
[units][records][root]	72	0	0
[units][records+root]	79	0	0
[units+records][root]	513	0	63
[no barriers]	1024	504	567

Remove the verification before applying, and the placement with only the barrier before the root slot immediately shows 63 silent grafts. So "each item a record names carries its own checksum" is not a nice-to-have — it is the price of dropping a barrier.

Closed form for the number of states

Each segment contributes all of its proper subsets, plus one more for "everything durable":

states = 1 + Σ (2^|segment| − 1)

After mkfs, "new pool, new file" has the segment sequence 2+2+1+2+2+1+18+2+1+2, 33 writes in total. Plugging in:

3 + 3 + 1 + 3 + 3 + 1 + 262,143 + 3 + 1 + 3 + 1 = 262,165

The 16 writes of 8 units to both disks, plus the second warm-up's two system configuration slot writes, have no barrier between them and fall into the same segment. 2^18 = 262,144. So the 260,000 figure is essentially the power set of one 18-write segment. The number of states grows exponentially with segment length.

This number grew step by step. E142's first run enumerated only the transaction's 21 writes, segments 16+2+1+2, 65,543 states; the fourth run, once the warm-up wrote real bytes, 262,162; the sixth, after the two acquisition writes joined the opening segment, 262,165.

4　The checker

The pool-level checker (crates/singlefs-checker) shares only the format-constants module with the implementation: CRC-32C is written again bit by bit, and the parsing of the three slot kinds and of units is written twice. If the runtime and the checker used the same code, they would be the same computation, and the cross-check would be worth nothing.

The checker computes the same back chain itself, with a bitwise CRC, not the implementation's:

Implementation　crates/singlefs-checker/src/lib.rs

/// Bitwise CRC-32C (reflected polynomial 0x82F63B78): no table, no call into the implementation's version.
pub fn crc32_castagnoli_bitwise(bytes: &[u8]) -> u32 {
    const POLYNOMIAL_REFLECTED: u32 = 0x82F6_3B78;
    let mut remainder = !0u32;
    for byte in bytes {
        remainder ^= u32::from(*byte);
        for _bit in 0..8 {
            remainder = if remainder & 1 == 1 {
                (remainder >> 1) ^ POLYNOMIAL_REFLECTED
            } else {
                remainder >> 1
            };
        }
    }
    !remainder
}

/// Back chain: CRC-32C of the previous record's header, with the 32 bytes of `header_csum` taken as 0.
pub fn back_chain_of_record_header(record: &[u8]) -> u32 {
    let mut header = record[..usize::try_from(JOURNAL_HEADER_BYTES).expect("487")].to_vec();
    header[JOURNAL_HEADER_CHECKSUM_OFFSET..JOURNAL_HEADER_CHECKSUM_OFFSET + 32].fill(0);
    crc32_castagnoli_bitwise(&header)
}

/// Self-verifying checksum of a slot or unit: over the covered range with the checksum field as 0;
/// the first 4 bytes are the CRC, the rest all 0.
pub(crate) fn checksum_field_holds(bytes: &[u8], cover_end: usize, field_offset: usize) -> bool {
    let field_width = usize::try_from(WIDE_CHECKSUM_BYTES).expect("32");
    if bytes.len() < cover_end || field_offset + field_width > cover_end {
        return false;
    }
    let mut covered = bytes[..cover_end].to_vec();
    covered[field_offset..field_offset + field_width].fill(0);
    let expected = crc32_castagnoli_bitwise(&covered).to_le_bytes();
    let stored = &bytes[field_offset..field_offset + field_width];
    stored[..4] == expected && stored[4..].iter().all(|byte| *byte == 0)
}

/// Choose the system configuration slot: checksum valid and highest generation.
pub fn choose_system_configuration(slots: &[&[u8]]) -> Option<(usize, SystemConfigurationView)> {
    slots
        .iter()
        .enumerate()
        .filter_map(|(index, slot)| check_system_configuration_slot(slot).ok().map(|view| (index, view)))
        .max_by_key(|(_, view)| view.slot_generation)
}

The entry point takes a single image and reports each invariant as holds, violated, or not applicable:

Implementation　crates/singlefs-checker/src/walk.rs

/// Entry point of the pool-level checker: every invariant is reported; ones not evaluated are
/// reported "not applicable" with a reason.
pub fn check_pool_image(reader: &dyn ImageReader) -> Vec<(&'static str, InvariantVerdict)> {
    let system_configurations = chosen_system_configurations(reader);
    // … choose the root, scan the journal, walk the trees, judge each invariant
}

Every crash state goes to three judges:

Judge	What it checks
Recovery + oracle	Recover twice, with and without the journal. A file that is read must be byte-for-byte correct; once the root slot is durable there is no going back to the old state; the walk must not fail
Pool-level checker	Entry point `walk::check_pool_image`, 23 invariants
Record checker	Root on disk but journal record missing; recovery claims the new state but a unit is missing

The record checker is separate because D13, settled item 7, limits the pool-level checker to a single image. Any check that needs a second input (the pre-crash image, the record stream) is not its job.

A judge must be able to fail, or zero violations means nothing. There are 26 corrupted images, at least one per invariant, and each one makes its invariant fail. With the root slot durable, removing each of the 8 units in turn fails 8 times out of 8; removing the barrier between the journal record and the root slot makes the record checker fail.

Results

The first disk creation and write succeeded. mkfs on two disks, write 3000 bytes, commit, throw away everything in the process, cold-start, read back: byte-for-byte identical.

260,000 crash replays, zero errors. All 262,165 crash states enumerated, no sampling; zero violations from all three judges; 22.7 seconds in release mode on the local machine.

That does not mean the code has no problems. Crash replay covers only the crash states enumerated in the model; edges outside the model are not covered:

There is only one workload, "new pool, new file". Overwrite, reuse after free, repeated mounts, and rollback have never been replayed.
The model assumes every write before a barrier is durable. A disk that does not honor FLUSH is outside it.
A torn write that happens to fool the 32-bit CRC32C is an accepted risk.
Of the 66 invariants in use, the checker judges 23.
Functional differential testing against an ideal model does not exist yet.

The story is long. Spring, summer, autumn, winter.

The 21 Rules of the File System---Writing a COW Filesystem from Scratch, Part 4

faliye — Sun, 04 Oct 2026 17:12:15 +0000

In the previous article, I documented my collaboration with AI.

In this article, I’ll add a few details about the basic work specifications and use them as the starting point for the discussion that follows.

faliye

Oct 4

SOP and Gate Design under Claude---Writing a COW Filesystem from Scratch, Part 3

#claude #ai #rust #filesystem

8 min read

How do you draw an architectural design on a blank sheet of paper? With a grid.

How did the Renaissance masters draw so many towering, magnificent figures with such precision? With dividing lines.

Let me loosely imitate the 21 Articles of War and write something—drawing a few lines for this filesystem.

Contributors

Article 1. We don't distinguish between submitters; we review only the evidence. A patch that passes the gate deserves to be taken seriously. Show me the test.

Decisions, Design, and Experiments

Article 2. Every decision must be backed by experiments, unless explicitly exempted. Experiment files are committed to git.

Article 3. Every decision and design must leave a trace, written up using the standard template.

Article 4. No decision is true forever; a decision that is challenged must be verified.

Article 5. This project learns from the experience of excellent past projects, but its implementation must fit this project's own character.

Format

Article 6. The data format is self-contained and carries back-references. Data can prove its own identity and what it belongs to.

Article 7. Index trees and the like are derived structures and may be discarded. Data recovery never depends on the tree; the tree serves only as a lookup accelerator and aid.

Article 8. Following Article 7, derived structures stay pure: no small data may be stored in them.

Article 9. Stripe width is variable, and full-stripe writes never incur RMW (read-modify-write).

Article 10. No false ENOSPC: space reported as available must actually be usable.

Article 11. Encryption is a first-class capability: slots are reserved for it, but it is off by default.

Article 12. Slots are reserved for future extensions, with vendor interfaces provided as circumstances warrant.

Article 13. Data units are 32 KiB and index nodes are 16 KiB. 32 KiB is dictated by checksum granularity; 16 KiB is a trade-off made for the CPU cache. Neither is chosen to align with the SSD's write granularity.

Testing and Recovery

Article 14. Data recovery code and test infrastructure rank equal to the core code and are developed in lockstep with it.

Article 15. The verification implementation and the filesystem implementation share no code; each is implemented independently, sharing only the format.

Article 16. Crash testing must cover scenarios exhaustively, never by sampling, and must not be treated as a substitute for broad-spectrum testing or formal verification.

Priorities

Article 17. Keep the early stage pure. Early on, ignore the Linux branch and develop around transactions.

Article 18. The first line supports SSDs only; the disk type is held as a placeholder in incompat. Other disk types can be specialized later in separate branches.

Article 19. Until the format is frozen, don't optimize the filesystem code and don't chase performance.

Article 20. When designing garbage collection, consider the possibility of GPU-based collection algorithms.

Slogan

Article 21. The slogan is still under consideration.....

End of text.

A Few Words on These Articles

Contributors reflects the idea of community collaboration. One person's strength is, in the end, limited.

Article 1. This is a project with AI as its main workforce, carrying AI in its DNA from day one. We enjoy the conveniences of this era, and we should also rein in its shortcomings. An AI-driven project has unlimited throughput, but human energy is finite. Submission throughput can be unlimited; acceptance throughput is bounded by evidence.

Decisions, Design, and Experiments is the operating manual handed to the AI during the design phase. It is designed to minimize AI hallucination, raise the AI's autonomy, and make sure decisions, designs, and experiments don't lose their context.

Article 2. Decisions and designs are not daydreams; they must be validated by experiments, finding their footing in the code of individual experiments. A decision or design with no footing is a fantasy. A decision may be wrong, but it must be shown to have been argued through. Experiments, as part of the evidence, should be kept on record permanently.

Article 3. In an AI-driven project, the knowledge base is the most important part of the whole engineering effort. The archived knowledge will provide key grounds for future decisions and retrospectives. Write documents according to the standard template; the template may change, and when it does, have the AI update the documents one by one.

Article 4. Many decisions may become outdated during construction. Given the AI's ability to question history and context, we give it options and prompts through which it can challenge them. At the same time, this questioning must not be left unchecked, or the project will slow to a crawl.

Article 5. This filesystem's design differs from past filesystems. We stand on the shoulders of earlier filesystems and absorb their excellent algorithms and designs, but we must also stay distinctive. We must constantly remind the AI that this is a different project and that earlier implementations can't simply be copied.

The data-structure section is the core implementation of singlefs and gives a brief account of its data design.

Article 6. singlefs data comes in a fixed size of 32 KiB. Each unit contains a data-unit header and the payload, locked together and never stored separately. This way the data naturally carries its own identity, and its back-references state what it belongs to—which tree, which object. The data can thus explain its own situation, with no need for index-tree records to prove its identity and ownership.

Article 7. Following Article 6, if data can vouch for itself, the tree that records it is demoted. Since the data can state its own identity and ownership, in the worst case the index tree can be thrown away and rebuilt.

Article 8. True, under Article 6 a 32 KiB data size makes small files balloon in storage. But following Article 7, the index tree must remain pure. singlefs stores no small data or other structured information in the index tree. The current idea is to solve this with small-data packing: bundling multiple small pieces of data into a single 32 KiB unit.

Article 9. A stripe is a concept of storing data across disks: to keep data safe, you must write to several disks at once. In traditional RAID 5, for instance, three disks are required—two for data and one for parity—so every write is a fixed 64 KiB. But what if you have only 32 KiB of data? Then you must read the old data, recompute parity, and write it back. If power is lost in the middle of this, the data and parity no longer match (the write hole). singlefs instead uses variable-width stripes. In the same situation, storing 32 KiB of data writes to just two disks; storing 64 KiB writes to three. No read is needed, and every write exactly fills the stripe. The risk of data loss on power failure and the space overhead are both much smaller.

Article 10. In a traditional filesystem, the index tree is authoritative, so updating a file has to go through it. Once the index tree has no room left to write, files can no longer be written. And since deleting a file also requires modifying the index tree, deletion fails too, and the filesystem locks up. singlefs's self-containment can solve this problem to some extent, but we are still exploring.

Article 11. File encryption is a capability designed into the format from day one, but the first runnable version won't implement it; it only reserves the format bits (all zeros when disabled). Once the transaction layer and the checker are running, real encryption will be wired in. Whether it can be rolled back after being switched on is not yet decided.

Article 12. singlefs reserves a portion of space for future extension fields. We are also considering whether to offer an extend-like field that SSD vendors can open up. Our selling point is waste.

Article 13. The 32 KiB data unit is not actually designed to cater to SSDs. The checksum has to cover the whole unit, and the aim is to bind checksum granularity and extent granularity together for simpler logic, at the cost of paying roughly ten percent more on small random reads. 16 KiB is the size of an index node, a number settled by weighing CPU-cache considerations. A 16 KiB node and a 32 KiB data unit are two different kinds of unit; a data unit is not two index nodes joined together.

Testing and Recovery sets singlefs's own standard for testing and file recovery.

Article 14. Because singlefs writes a filesystem from scratch, and its underlying data format differs in places, it needs a test system and methods of its own design. Data recoverability is singlefs's top priority, so test code, recovery tools, and core code are implemented on the same day. Each milestone cross-verifies the others.

Article 15. If tests and core code shared the same logic, they would easily converge on the same bugs. So tests and core code are implemented separately.

Article 16. Beyond the routine adversarial cross-checking among three bodies of code, the core test of singlefs is exhaustively simulating power loss in QEMU and examining the state of the data on each stream, and whether it is recoverable. Broad-spectrum testing exists to catch bugs in the code and to keep the AI from producing results that merely look correct when it writes code. Formal verification is indispensable for concurrent features. singlefs aims to find bugs as early as possible through rigorous testing, while also improving observability.

Priorities were designed according to singlefs's own characteristics during development.

Article 17. Because singlefs is developed around transactions, work on upper-layer interfaces such as FUSE will be pushed back, which may delay compatibility with the Linux community. But it lowers the complexity of early development and lets us focus on the flow of data. I'm short on energy, and short on tokens too.

Article 18. singlefs defines incompat to distinguish disk media types (rotating disks, SSDs, ZNS, and so on), in the hope of optimizing each kind of media to the extreme on top of this setting. But for the same reasons—not enough energy, not enough tokens—the first line supports only generic SSDs.

Article 19. Across singlefs's milestones, we will compare against other filesystems horizontally, but only on the basis of per-node data records. Before the format is frozen, we will not optimize the core filesystem code for performance.

Article 20. singlefs's garbage collection will be heavy. Besides a standard GC running on the CPU, we'd like to bring in the GPU to do some of the work. And of course, we also hope to make use of features such as FTL and ZNS.

Article 21. [Big disks belong to the company; good sleep belongs to you?] [WAAAGH! Reckon it'll work!]

Well, it's finally written. The keyboard now belongs to you, Agent bros.

run Tokens, run!!!

SOP and Gate Design under Claude---Writing a COW Filesystem from Scratch, Part 3

faliye — Sun, 04 Oct 2026 02:53:06 +0000

What is an SOP?

SOP stands for Standard Operating Procedure. You split a process into a number of steps, and for each step you say what to do and how to verify it. The result is a set of concrete steps that can be repeated, checked and signed off.

In my boss's words, it's a bowl of ramen: how many noodles, how much water, how much salt, how much scallion. I'm pretty sure he meant Japanese ramen. When I cook it, it's more like: some noodles, a little oil, a reasonable amount of water, scallions at will.

What is a gate?

This one hardly needs explaining. When I was a kid, my family said be home before six. Of course I could choose to come home after eleven, but not coming home wasn't an option. The curfew was just there. It triggers when you walk in the door: men's singles, women's singles, or mixed doubles.

End of the main text.

My last article covered the three-way (Three-Body-style) adversarial setup for multiple agents, plus some code-style rules. But does three parties fighting it out solve the problem? Honestly, no.

faliye

Oct 4

Implementation That Favors the AI, Review That Favors the Human---Writing a COW Filesystem from Scratch, Part 2

#claude #rust #filesystem #ai

11 min read

Agents write the docs, run the experiments and give the reports. Without limits, an agent will freely write all sorts of things, just like my ramen: somewhere between delicious and awful, at random.

While working, I ran into a few fairly serious problems:

Even with prompt and skill constraints, agents often do things they were never authorized to do, or lose track because they forgot the context. How well it went, whether it's actually finished, all "more or less", just like my ramen. So quality depends entirely on how focused and responsible I am that day. My focus and sense of responsibility fluctuate, so of course the results differ between my highs and my lows.
Agents like to overthink. They fall into endless thinking, trying to find the perfect answer, especially on open-ended questions with no clear goal, where they deliberate at length all the time. It's like Go: I may be the weaker player, but I can always use the wear-down-the-old-man tactic. Unfortunately, next to a silicon life form, I'm the old man. A forty-minute think I can sit through. Three forty-minute thinks in a row, I can't.
Thanks to the breakneck progress of AI, a subagent can now send out minions of its own. The token explosion aside, the minions of the minions of the minions are a blind spot for me. I can go and read their thinking, but they're beyond what I can manage. My energy is limited, so I end up with minions of minions of minions who are not my minions.
The main agent does the scheduling, and I'd rather not hand it the writing of code or docs. The sub-agents that fight each other haven't settled on a conclusion until they're done, so they certainly shouldn't write into the main directory. That means designing more subagents for the main agent to call: some write code, some review it. And the ones who write code and the ones who review it can't sit together.

So there's nothing for it. To keep the project going, let's change a few things.

1. SOP and separation of duties (for problem 4)

Turn agent orchestration into a full pipeline: retrieval -> investigation -> implementation -> self-check -> knowledge rot -> gate.

Each stage has one subagent definition under .claude/agents/. Every description says "use only when the main agent dispatches you by name... never auto-dispatch", and what material the role needs goes in required-inputs. Here they are, excerpted (the originals are written in Chinese; I've translated the descriptions):

Retrieval: the retrieval agent may go online, carry material back and put it in the designated directory.

---
name: prior-art
description: Researcher. Looks up other filesystems' source and docs on demand, hands back only facts with citations, no arguments. Use only when the main agent dispatches you by name with the question to look up; never auto-dispatch.
tools: Read, Bash, WebFetch, WebSearch
model: sonnet
effort: high
omitClaudeMd: true
required-inputs: draft directory, report
---

Investigation: the three-way adversarial setup kicks in to find a suitable approach.

---
name: experiment-designer
description: Experiment designer. Takes an experiment number and writes the pre-run registration (arms, controls, criteria, failure clauses) before any code or artifact exists. Use only when the main agent dispatches you by name with the question to answer and the clause under test; never auto-dispatch.
tools: Read, Bash
model: opus
effort: high
omitClaudeMd: true
required-inputs: draft directory, clause under test, fork list|question list|rerun
---

---
name: three-way-attack
description: The cloud attacker leg of the three-way argument. Use only when the main agent dispatches you by name with this round's background material, attack surface and the verdicts of earlier rounds; never auto-dispatch.
tools: Read, Bash
model: opus
effort: high
omitClaudeMd: true
required-inputs: no-read list, draft directory, background material, attack surface
---

Implementation: start writing code.


---
name: implementation-writer
description: Implementer. Changes crates/ for one milestone step, one parallel line, or a cluster the main agent has bundled (may close several items), with tests, and proves the tests go red. Use only when the main agent dispatches you by name with the step number and the clauses in play; never auto-dispatch.
tools: Read, Edit, Write, Bash
model: opus
effort: high
omitClaudeMd: true
required-inputs: draft directory, report, clause, crates files to touch
---


---
name: tooling-writer
description: Tooling implementer. Changes gate stages, hooks, research scripts and watchdogs, or changes agent definitions, shared constraints and project rules according to a verdict. Every change first builds an input that should go red. Use only when the main agent dispatches you by name with which item is being closed and the exit; never auto-dispatch.
tools: Read, Edit, Write, Bash
model: sonnet
effort: high
omitClaudeMd: true
required-inputs: draft directory, report, exit, files to change
---

Self-check: a three-way siege to review code quality.


---
name: investigator
description: Investigator. When a test or gate result differs from expectation, reproduces the symptom, bisects down to file and line, and hands back a minimal repro plus the condition that would refute it. If the dispatch says "fix it once found" and the clause spells out the fix, carries on, fixes it and proves it red; if the clause doesn't, stops and hands back to the main agent. Use only when the main agent dispatches you by name with the verbatim symptom and the question to answer; never auto-dispatch.
tools: Read, Edit, Bash
model: opus
effort: high
omitClaudeMd: true
required-inputs: symptom, draft directory, report
---

Knowledge rot: audit the whole project to check that every change is consistent.


---
name: sweep
description: Sweeper. After a number is retracted, a format constant changes, a new criterion is established, or a stage of work ends, searches the whole repo for places that still cite the old value, should now be governed by the new criterion, or have been made false by this stage, and hands back a classified list. Use only when the main agent dispatches you by name with the old value and which quantity it is (or the stage's scope of change and what it achieved); never auto-dispatch.
tools: Read, Bash
model: sonnet
effort: high
omitClaudeMd: true
required-inputs: draft directory, report
---

Gate: at the end of a task, mechanically check the error ledger, then the main agent analyzes and fixes the problems.

Every node has its own specialist for its own job. The main agent only breaks down and dispatches tasks, and every subagent minds its own business. On top of that, there has to be a mechanism for "knowledge rot" that regularly clears out expired context, so the pipeline never runs overloaded.

2. Limiting abilities (for problems 2 and 3)


---
tools: Read, Bash
model: sonnet
effort: high
omitClaudeMd: true
---

Eliminate fuzziness and the gray zone. The main agent has to be clear what my task is, and then dispatch work to the subagents according to the task.

Depth=1. A subagent may not have minions. That roots out the "blind spot" problem. (This corresponds to the environment variable CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH=1, and none of the roles has Agent in its tools anyway.)

omitClaudeMd=true. It mustn't know too much. Context it doesn't need to know is withheld without exception, so it doesn't start free-associating.

tools: tools are narrowed, strictly. Each role above gets a subset of Read, Edit, Write, Bash, WebFetch, WebSearch, and only the researcher can go online. Everyone has Bash, so the lock doesn't sit on tools; it sits on hooks: dangerous patterns are rejected before they execute, with no reliance on the agent's good manners.

effort: high. Don't use more power than needed to solve a simple problem.

3. The gate (Harimoto siblings, go!) (for problems 1 and 2)

Red-green dual proof: every piece of code must have a test. And a test that merely runs "green" (passes) isn't enough: you have to prove to me that the test can catch an error (red), and the red and green evidence must be submitted together. The red evidence doesn't rest on the agent's word either: the gate breaks the code under test in one spot and confirms that the test really goes red.

Strong provenance: implementation must be tied to decision and experiment records. "Ghost code" with no historical decision behind it is not accepted, and the gate checks hard whether the comments explain where the code came from.

Exhaustive probe points: plant lots of probes in the code, check the data state every time execution passes, weave a nerve net. For example, even if the code has run through two hundred thousand different points in a simulated environment, you still have to show that the data flow is complete and free of violations. Checks that aren't wired in yet are honestly marked unimplemented, and they're not allowed to pretend they passed.

Text and format gate: check the knowledge base and docs item by item. Is everything written strictly in the preset JSON/Markdown format? Any ad-libbing? Does the record pass? Even a date has to be written as a standard date.

Scheduled patrol (timeout mechanism): the main agent attaches a scheduled script that watches the subagents it has sent out at all times. If it sees the context grow too large or the runtime run over, it immediately checks task progress, and if things have drifted from expectation, pulls the agent back.

4. Dirty-write protection (for problem 4)

Most mature agent frameworks are already aware of this one. For example, they use a worktree and the workspace under the root for physical isolation, so I don't have to build it from scratch.

But I've added one extra lock: unless it has gone through the dedicated "implementation writer" agent node, no code is ever allowed to be written into crates.

The one who writes code and the one who reviews it must be physically isolated. Without a double signature, you don't get past the main repo's threshold.

With that, it should be able to run.

Tokenman, move out!

Implementation That Favors the AI, Review That Favors the Human---Writing a COW Filesystem from Scratch, Part 2

faliye — Sun, 04 Oct 2026 02:38:51 +0000

In the last post, I decided to hand the keyboard over to the AI. But how to use the keyboard, what the key layout is, which input method to use: all of that is still unclear. To get a filesystem built sooner rather than later, I still need to make a little more effort.

faliye

Oct 4

DAY1: Choosing to Be Wasteful---Writing a COW Filesystem from Scratch, Part 1

#ai #filesystem #rust #claude

5 min read

Part 1: Decisions and the Blueprint

The implementer is an AI Agent. An Agent can write code toward a goal, but we lack a blueprint. A filesystem blueprint takes far more than five decisions. What goes into the self-contained unit (how many rooms)? How is the data format defined (how big is each room)? Are stripes variable (can the partitions be moved)? How is the journal handled (do we install a gate)? What is the first goal (fix the foundation first, or the roof)? All of these need decisions. Otherwise you may end up with the roof finished and the foundation not yet dug. We all know that Agents are superb executors when the goal is clear, but they are a bit weaker at planning complex tasks, and without a blueprint a task easily goes off track.

But filesystem design is extremely complex and involves a mountain of prerequisite knowledge. Even if I started searching through references day and night without sleep, I might have a first design drawing five years from now. Fortunately, the Agent bro sitting next to me is exceptionally learned, so I can ask him to make the design decisions and draw the blueprint. His weakness is that he is prone to going off the deep end: overconfident, charging down one path to the bitter end. So I have to guard against an awkward situation: he may not be trying to deceive me (he is simply hallucinating), yet I have no way to tell whether the information he gives me is true or false. What to do?
As the saying goes, three monks have no water to drink. Well then, let's get three Agent bros to work together anyway. Here is the design:

The conversation where the task starts is called the Main Agent. It only asks questions according to the plan. For example, according to the plan, today we should decide how many rooms to design.

This question is handed to one strong Agent (Strong A), who is responsible for collecting information and finding an answer that looks correct. Once he brings back this seemingly correct answer, the Main Agent notifies two more Agents, B the supporter and C the attacker, to start work.

Supporter B looks for non-overlapping supporting evidence for the answer, that is, other arguments that also support it. If he finds any, he reports them; otherwise he reports that he could not find any.

Opponent C looks for arguments that show the answer does not hold at all. If he finds any, he reports them; otherwise he reports that he found none. But the opponent must find strong evidence; weak evidence is treated as failing to find any.

After collecting these reports, the Main Agent begins independent verification. The Main Agent does not judge who has the better argument; it judges and checks whether the results are valid. If they are valid, the Main Agent summarizes these points and has Strong A repeat the steps above and reason through a second round. Note that in the second round the Main Agent's materials have grown, now including both supporting and opposing views.

At the same time, four points need attention:

An Agent's conclusions are random in nature. So before Strong A starts work, he must set a target for himself. If the conclusion drifts away from that target, the conclusion is invalid. For example, suppose we want to build three rooms and the conclusion is that flooring is better. That conclusion cannot be accepted. The file header comment of an Agent's experiment program contains the following: the criterion (written in stone before the run, not to be changed after it), the failure clause (written in stone before the run), the reverse-acceptance clause, and the questions it cannot answer.
Agents have a natural tendency to agree with the user, and this must be eliminated completely. Even if a proposal comes from the user, the decision must still go through the three-party process; if it does not, it is not recognized as approved. The user can demand that it pass, but the AI will, at lightspeed, leave the evidence in a warning and then submit.
Agents from the same source tend to think alike. Although we bring in two AIs to participate, to avoid this tendency I also brought in a local model to join the thinking and the attacking. A local model is usually useless, but every now and then it has a flash of inspiration.
The Main Agent's data sources are not trustworthy. Each Agent must measure the data independently. This prevents the Main Agent from hallucinating some data into the context and thereby biasing the other Agents' judgment.

Of course, considering that AIs often have no stopping condition, if three rounds of three-party confrontation produce no result, reasoning stops and the question is summarized and handed to me to decide.

With this process, my job becomes much simpler. I generally no longer need to re-check whether the materials are real. My attention is not spent on reading materials and checking for hallucinations; I spend it only on four things a machine cannot judge:

Is this test measuring the right thing?
Is this invariant itself correct?
Does the basis of this decision hold up?
Do I accept this trade-off? This is the second half of our first heading: review that favors the human. Make sure the evidence is sufficient and the reasoning is valid, and let the human make the call. And the cost? What's your superpower, Tokenman?

Part 2: Implementation and Code

Before discussing this topic, two observations:

If, in this era, AI context has reached the million-token scale, the context window can even hold all the code and documentation of a medium-sized project at once.
The best practices of the past had human programmers as their subject.

Based on the second point, let's re-examine past best practices: function names should be short, for loop nesting should be shallow, functions should be reusable and abstracted and encapsulated, and so on. These practices actually address the problem of limited human brain bandwidth: short function names, shallow for nesting, and reuse, abstraction and encapsulation are all meant to reduce the load on the human brain. Today, for those of us programming toward Markdown (prompts), does the length of a variable name matter to us? Does shallow for nesting matter to us? Do reuse, abstraction and encapsulation matter to us?

But note that these matter a great deal to the Agent.

When an Agent modifies a function, it greps, and function names and variable names are always the first coordinates. If a name alone can tell you what something is and what it does, that lowers the Agent's hallucination rate, because comments may be forgotten, but the name anchor gets used again and again.

Ten levels of for loops with no if inside are just a traversal to an Agent. An exhaustive match with 12 arms is counted for you by the compiler; 12 independent layers of if/else are 2^12 paths, which you cannot finish testing.

Abstraction and encapsulation are the biggest source of bugs in code. Their main benefit is making code easier for humans to maintain, but for an AI, modifying one place costs about the same as modifying ten.

Today the main force writing and reading code is the model. Sorting out logic, maintaining consistency and exhaustive coverage are exactly what machines are good at. A floor set by the limits of the human brain is far too low for a machine.

So our default attitude toward "best practices" is skepticism, not compliance. This does not mean they are all wrong; it means the reason behind each one has to be re-examined.

Four steps to audit an old rule:

What problem was it originally meant to solve? If you can't say, drop it immediately.
Does that problem still exist today? If it was solving "humans can't remember," "humans can't read it all," or "reviewers are too busy," it most likely no longer exists.
Does it have a second reason, one that has nothing to do with humans? If so, keep it, and rewrite the reason to be that one.
Can the rewritten reason be turned into a gate? If it can't, downgrade it to a suggestion.

There is only one criterion: does this rule make the code easier to verify mechanically, or does it only make the code easier for human eyes to skim? Keep the former; the latter can be dropped.
Here is a brief summary of some past best practices.

Popular practice	Disposition	How we write it
Names should be short, and the closer to the declaration the shorter	Abolished	No length limit
Loop counter called i, temporary variable called tmp	Abolished	Name it for what it is: stripe_index
Functions under 20 lines	Kept, reason rewritten	Cut by "one thing that can be verified on its own," not by line count
Nesting no deeper than three levels	Relaxed	No limit on depth; what is limited is the number of paths
Single exit	Abolished	Early returns freely allowed
Replace conditional branches with polymorphism	Abolished for closed sets	enum plus exhaustive match
DRY, avoid repetition	Relaxed	Repetition must be generated, never copied by hand
default: fallback is safer	Abolished for closed sets	Leave missed cases for the compiler to catch
One assertion per test	Abolished	One scenario per test; multiple assertions allowed
No more than 80 columns per line	Left to tools	rustfmt decides; don't shorten names for line width
Avoid premature optimization	Downgraded to a suggestion	—

Part 3: Practice in Rust

1. No abbreviations, except those registered in the abbreviation table

Without reading comments, without looking at call sites, without looking at the implementation, from the name alone you should be able to say what it is, what it does, and what it does not do.

No abbreviations: write cnt, idx, buf, tmp out in full. Domain abbreviations (lba, crc) may be used only once registered, and the registry is their sole authoritative definition.

No single letters: write i as stripe_index, the generic parameter T as Key, the lifetime 'a as 'journal, and Err(e) as Err(error).

Write preconditions into the name: write_node says nothing; append_verified_node_to_journal says three things: it appends rather than overwrites, the input has been verified, and it lands in the journal region.

Test names state the scenario and the expectation: crash_between_data_write_and_commit_keeps_previous_generation, not test_commit.

One concept, one name across the whole repository: it must not be called generation here and epoch there.

2. Semantics go in types first: types > names > comments

Take the following example. The same thing written three ways; when the semantics live in the type, an AI that writes it wrong gets an error.

// 1. Semantics in comments: the worst
// dim0: nationality  dim1: gender  dim2: age band
pop[i][j][k]

// 2. Semantics in names: criticized as verbose in the past, machines love it
number_of_people_in_japan[foreigner][woman][teenager_18_to_24]

// 3. Semantics in types: the best, a wrong dimension simply won't compile
number_of_people_in_japan[Nationality::USA][Gender::Woman][AgeBand::Teenager18To24]

3. Branching: write every case out, never write _ =>

enum Medium {
    Rotational,
    SolidState,
    Zoned { zone_size_in_bytes: u32 },
}

fn node_size_in_bytes(medium: &Medium) -> u32 {
    match medium {  // the point is that there is no `_ =>` wildcard arm
        Medium::Rotational                   => 64 * 1024,
        Medium::SolidState                   => 16 * 1024,
        Medium::Zoned { zone_size_in_bytes } => (*zone_size_in_bytes).min(256 * 1024),
    }
}

The key is the _ => that was not written. Add it and the code looks shorter and more "general," but it switches off exactly the compiler's exhaustiveness check. Later, when a new kind of medium is added, the program will quietly walk into the wildcard arm; the behavior will be wrong and nobody will raise an alarm.

The criterion is not "how many branches there are" but "if a case is missed, who finds out first": the compiler, or production.

A few related practices:

For encodings read from disk that may gain new values later, make - "unknown" an explicit member (Unrecognized(raw_code)), and match remains exhaustive.
Use enum for closed sets, not trait objects. A trait's default method is in fact also a kind of wildcard arm.
Don't abstract a trait with only one implementation. Don't abstract "to make swapping easier later."
For interactions between two cases, write an exhaustive match on a tuple. In singlefs's code we enforce the following:

fn geometry(candidate: Candidate, map_scope: MapScope) -> Geometry {
    match (candidate, map_scope) {
        (Candidate::LogicalIdentityWriteOrder, MapScope::AllPointers) => Geometry { .. },
        (Candidate::LogicalIdentityWriteOrder, MapScope::Code1Only)   => Geometry { .. },
        (Candidate::MixedPath,                 MapScope::AllPointers) => Geometry { .. },
        (Candidate::MixedPath,                 MapScope::Code1Only)   => Geometry { .. },
        (Candidate::ClassReuseTagged,          MapScope::AllPointers) => Geometry { .. },
        (Candidate::ClassReuseTagged,          MapScope::Code1Only)   => Geometry { .. },
    }
}

Two of the rows actually have identical values, and most people would merge them. Here we do not merge them: the day a fourth candidate is added, the code will not compile until this spot is updated.

4. Types: make illegal states unwritable

#[derive(Clone, Copy, PartialEq, Eq)] pub struct LogicalAddress(pub u64);
#[derive(Clone, Copy, PartialEq, Eq)] pub struct PhysicalAddress(pub u64);
fn read_block(address: PhysicalAddress) -> Block;
// read_block(LogicalAddress(value)) simply won't compile

struct RawNode(Vec<u8>);       // just read from disk, unverified
struct VerifiedNode(Vec<u8>);  // checksum compared and passed
impl RawNode {
    fn verify(self, expected_checksum: Checksum) -> Result<VerifiedNode, CorruptBlock> { /* ... */ }
}
fn walk(node: &VerifiedNode) { /* ... */ }  // want to skip verification? the argument can't even be constructed

The first snippet turns "passing the wrong address" from a runtime bug into a compile-time error. The second is even harsher: it makes "not yet verified" a type too, so "forgetting to verify" simply cannot be written. Verify once at the boundary, then trust the types inside, instead of re-checking at every layer.

A few smaller points:

Replace boolean parameters with enums. write(node, true) says nothing at the call site; write(node, Durability::Synced) does.
When constructing a struct, write out every field; don't use ..Default::default(). It is a wildcard arm over fields: add a new field, and every place that didn't write it quietly gets the default value.
Don't use as for lossy numeric conversions; use try_from and handle the failure. as truncates, drops signs and wraps around, all without any error.

Branching and types are in fact two sides of the same thing: one lets the compiler catch "a missed case," the other makes "an illegal combination" unwritable. Both move the checking from humans to machines.

5. Errors: recoverable ones go into types, invariant violations get assertions

I/O errors, on-disk data corruption and running out of space are recoverable, and go through Result. The error enum is divided by the decision the caller must make (not found, data corrupt, out of space, other I/O error), not expanded one by one by underlying cause.

At the public boundary of a library, don't use a unified error type like anyhow. It lumps "retry, report corruption, report out of space" into one blob, and is a wildcard arm on errors.

A broken invariant is a bug: assert immediately, don't carry bad state forward, and above all don't write it to disk.

Don't write unwrap(); write expect, with a message stating which invariant it relies on.

An expect in experimental code looks like this:

.expect("leaf number is not in the position table: the position table did not keep up after the split")
u64::try_from(distinct_groups.len()).expect("the number of distinct nodes fits in u64")

Each expect is a named assertion. The day it blows up, the message says directly which assumption collapsed.

Longer names, more types, more complete branches. The compiler reports a few more errors. That is what being AI-friendly means on the implementation side.

Finally: What We Lost, and What We Gained

Wasted tokens. One question, three legs, three rounds, sometimes eight; one set of background material runs over two thousand lines.

Wasted time.The format design took a full three weeks, and not a single line of the filesystem itself has been written.

Wasted code. The experimental code may be very long, yet not a single line of implementation code has been written.

Wasted characters. Names written in full, branches written in full, every rejection accompanied by a next step.

What we gained:

A conclusion that has not been verified will never appear in the code.
The code may still be wrong, but when it is wrong, it will be caught.
What I decide is on the record, and when I say something wrong, someone comes to strike it down.

It doesn't matter that I don't have the knowledge: the Agents come to me with evidence and options, and multiple choice is something I can do. Reviewing makes me happy.

I don't write the code, but when it is written wrong, it fails loudly at any time. Simple, linear, multi-branch code should be fun for an Agent to write, I imagine.

DAY1: Choosing to Be Wasteful---Writing a COW Filesystem from Scratch, Part 1

faliye — Sun, 04 Oct 2026 02:07:33 +0000

I'm going to build a filesystem. The way I'm going about it might not be entirely proper. Still, I figured I should write something down, though I suspect this series won't have much actual technical content.

To open the series, allow me to explain the basic concepts of filesystems with my very limited knowledge.

A filesystem is the set of data structures and directory rules an operating system uses to organize, manage and retrieve data on disk. Whether or not you use a computer, you can't escape this storage system.

Traditional filesystems (like Ext4 and XFS) are update-in-place. The data unit (the block) has a fixed size, and changing one goes read → modify → write, back to the same spot. It's stable and solid, but since it's an overwrite, once you write, the old data is gone. Ext4 and XFS use a journal to protect metadata, so recovery after a power cut is fast. But the journal doesn't protect the data itself. And snapshots and rollback are hard to build in the filesystem alone, so you end up relying on something below it, like LVM, or on a separate backup system.

So if the traditional approach needs all this extra machinery, why not just put those features inside the filesystem? Don't overwrite data. Write it somewhere new, then point the pointer that used to reference the old data at the new data instead. When new data is written the old data isn't overwritten, and as long as some snapshot still references it, the old data stays. Snapshots come almost for free. Since old data doesn't vanish right away, rollback is easy too. By the same logic, if the power dies suddenly, as long as that final "pointer switch" hasn't completed, the disk still holds a complete old version. No long repair needed, and crash consistency is naturally stronger.

This is the COW (copy-on-write) filesystem, with Btrfs and ZFS as the famous examples. Of course, there's no free lunch. COW pays for it with write amplification, fragmentation, and the job of reclaiming space once nobody references the old data anymore.

That's the end of the main text. Everything below is not the main text.

So, why write a new COW filesystem?

One day in mid-August, while reading up on filesystems, two questions suddenly popped into my head:

Why is the unit size of filesystems almost always 4K?
Why do most systems go to such lengths to keep their data structures compact?

My answers:

First, 4K is the memory page size. The page cache and memory mapping are all page-aligned, so making the block the same size as a page is simply the easiest thing to do. It's also the physical sector size after disks moved from 512-byte to 4K sectors (Advanced Format). The bigger the block, the more space small files waste (internal fragmentation), so nobody dared make it larger.
Ten or twenty-odd years ago, storage was outrageously expensive. Every filesystem was desperately trimming itself down, saving as much disk space as possible for actual data, and leaving the organizing of structures to complicated, clever algorithms. It reminds me of a joke. The diameter of a rocket, the crystallization of human technology, is supposedly limited by the width of railway tracks, because of transport constraints. The track width came from the English, who set it to the width of two horses' backsides, because trains were originally pulled by horses. (It's a popular story, but whether it's true is doubtful. Take it as a joke.)

Looking at computers today, storage has become cheap and huge, even if prices have crept up a bit lately. An SSD has no head and no spinning platter. Data lives in NAND flash, is read and written in pages (roughly 16KiB), and inside there are multiple channels and dies working in parallel. So a big chunk of data can be pulled out at once.

So maybe a design like this

A data unit carries, besides its data, a description of who it is and what it belongs to. It holds the information it needs about itself (self-contained). Even if it sits in some far corner of the disk, a scan is enough to find its owner.
Since a unit can explain its own origin, the tree that used to describe it loses its main job. The tree becomes just an index for fast access to units. It has no other duties and doesn't carry small data. So it can be thrown away and rebuilt.
For future compatibility, leave some bytes empty as extension points. That doesn't save space. It actively wastes it.
Since we're choosing the future, old spinning hard disks stop being an optimization target. We aim entirely at new SSDs, newer FTLs and ZNS (Zoned Namespaces). That said, the SSD's own garbage collection and the filesystem's space reclamation are two different layers. The former handles erasing flash blocks. The latter handles which old versions nobody references anymore. We still have to do the latter ourselves, though it can cooperate with TRIM and ZNS.
Since SSDs are the target, we align data to the SSD's pages and parallel stripes. Metadata is 16KiB and data is 32KiB, so one read can take in a whole stripe.

Problems still to solve

How to implement it: who's going to build this?
Which language: how do we make it easy to verify?
Code style: how do we keep it simple and clear?
What to verify with: how do we tell right from wrong? Scratches head. Fine, let's just do this.

Choosing how to implement it

I know very little about filesystems and definitely can't write a line of it. Writing this myself isn't realistic.

The keyboard goes to the AI. Tokens aren't that expensive right now, so let's be a little wasteful.

Choosing the language

It must compile to something fast enough.
It must have strong verification. Content goes to Rust. Write more verification. Let's be a little wasteful.

Choosing the code style

Absolutely few branches. Too many if-else and the combinations to verify grow exponentially. AI makes mistakes easily.
Abstract as little as possible. Don't reuse a function if you can avoid it. Copy-paste if you can. Never couple things together or mix them up.
Variable names must be long. Seeing a name should tell you the function's past and present life. A bit bloated is fine. Write more code, write it longer. Let's be a little wasteful.

Choosing how to verify

I don't understand code, so every plan must be put to a vote by several parties. Three AIs debate for three rounds, and I'm the referee.
I can't tell right from wrong, so let's just simulate-verify every single point. As for power loss, QEMU, that's yours: cut at every possible crash point, replay, and see whether the filesystem still recognizes itself.
I don't understand timing, so let's enumerate every reordering of concurrent operations. herd7, you're up: given a memory model, it lists every outcome that's allowed to happen. Order out of chaos.

** Not understanding is fine. Verify more. Let's be a little wasteful. **

And with that, our core decisions are done.

Day 1, and the most critical decisions are made.

Looking back, I reckon it's not bad. WAAAGH!

DEV Community: faliye

Benchmarking 6 Open LLMs on a 2-Machine RPC Cluster (Consumer GPUs, ~96GB VRAM)

Hardware Setup

Pitfall Record: tensor-split Order

Candidate Models

Methodology

VRAM Usage per Model (Initial Test Run)

Overall Comparison Summary

Final Thoughts

Taming the Swarm: The Iron Discipline of Multi-Agent Scheduling---Writing a COW Filesystem from Scratch，Part 7

How the Swarm was built

1. The Swarm's skeleton: the Queen, the action units, and the scheduling file

The main agent does only three things

Dispatch by format

The scheduling file: the main agent's external memory

How two Queens negotiate

Interfacing with upstream

2. The boundary of the information node: keeping the meta-prompt alive

3. Hardware scheduling

Step 1: Cut off the ability to write shell freely

Step 2: Before a script operates, check the environment and resources, and return the result as a prompt

2.1 Every script first decides "should it run, can it run"

2.2 Resource reminders: put each machine's current headroom back into the agent's context

2.3 The memory wrapper: let OOM happen only inside the cage

2.4 Multi-host: the dispatcher and the cross-machine driver

2.5 GPU: quiesce first, don't restore afterward

Step 3: After the task ends, the exit gate wraps up

4. The last layer: the gates themselves must be able to go red

Finally

$50–$200 in Claude API credits, and Google's Antigravity

Brute-Force R&D: Cheap Exploration and Multi-Agent Judgment

Brute-Force Testing: Optimizing a 2.6-Billion Crash-State Blowup --- Writing a COW Filesystem from Scratch，Part 6

Brute-Force R&D: Cheap Exploration and Multi-Agent Judgment

The First Transaction and 260,000 Crash Replays---Writing a COW Filesystem from Scratch, Part 5

1 Where do the decisions behind the code come from?

2 What should each stage implement?

3 How do we prove that nothing is wrong right now?

Code and implementation: the first transaction and 260,000 replays

1 The first transaction

2 The crash-point design

3 Recovery model and independent audit

4 The checker

Results

The 21 Rules of the File System---Writing a COW Filesystem from Scratch, Part 4

SOP and Gate Design under Claude---Writing a COW Filesystem from Scratch, Part 3

Contributors

Decisions, Design, and Experiments

Format

Testing and Recovery

Priorities

Slogan

A Few Words on These Articles

Contributors reflects the idea of community collaboration. One person's strength is, in the end, limited.

Decisions, Design, and Experiments is the operating manual handed to the AI during the design phase. It is designed to minimize AI hallucination, raise the AI's autonomy, and make sure decisions, designs, and experiments don't lose their context.

Testing and Recovery sets singlefs's own standard for testing and file recovery.

Priorities were designed according to singlefs's own characteristics during development.

SOP and Gate Design under Claude---Writing a COW Filesystem from Scratch, Part 3

What is an SOP?

What is a gate?

Implementation That Favors the AI, Review That Favors the Human---Writing a COW Filesystem from Scratch, Part 2

1. SOP and separation of duties (for problem 4)

2. Limiting abilities (for problems 2 and 3)

3. The gate (Harimoto siblings, go!) (for problems 1 and 2)

4. Dirty-write protection (for problem 4)

Implementation That Favors the AI, Review That Favors the Human---Writing a COW Filesystem from Scratch, Part 2

DAY1: Choosing to Be Wasteful---Writing a COW Filesystem from Scratch, Part 1

Part 1: Decisions and the Blueprint

Part 2: Implementation and Code

Part 3: Practice in Rust

1. No abbreviations, except those registered in the abbreviation table

2. Semantics go in types first: types > names > comments

3. Branching: write every case out, never write _ =>

4. Types: make illegal states unwritable

5. Errors: recoverable ones go into types, invariant violations get assertions

Finally: What We Lost, and What We Gained

DAY1: Choosing to Be Wasteful---Writing a COW Filesystem from Scratch, Part 1

So, why write a new COW filesystem?

So maybe a design like this

Problems still to solve

Choosing how to implement it

1　Where do the decisions behind the code come from?

2　What should each stage implement?

3　How do we prove that nothing is wrong right now?

1　The first transaction

2　The crash-point design

3　Recovery model and independent audit

4　The checker