<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: faliye</title>
    <description>The latest articles on DEV Community by faliye (@faliye).</description>
    <link>https://dev.to/faliye</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg</url>
      <title>DEV Community: faliye</title>
      <link>https://dev.to/faliye</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/faliye"/>
    <language>en</language>
    <item>
      <title>A Heterogeneous Scheduling System for 18 Billion Tests---Writing a COW Filesystem from Scratch，Part 9</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Sun, 15 Nov 2026 22:34:56 +0000</pubDate>
      <link>https://dev.to/faliye/a-heterogeneous-scheduling-system-for-18-billion-tests-writing-a-cow-filesystem-from-scratchpart-3oi0</link>
      <guid>https://dev.to/faliye/a-heterogeneous-scheduling-system-for-18-billion-tests-writing-a-cow-filesystem-from-scratchpart-3oi0</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scale is complexity.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Exhaustively verifying the correctness of a file system is, in theory, impossible. So what singlefs does is creep closer bit by bit, with pruning and a reuse database: anything already judged never needs judging again, and equivalent cases are judged only once. One layer up sits a multi-agent scheduling and scripting system: a main agent dispatches the work, following the gate scripts to run verifications and experiments.&lt;/p&gt;

&lt;p&gt;Is the system stable? Reasonably so — like bolting a steering wheel onto a bare chassis. When things go well it runs fast, but life always has its rainy days. Running it for real, we hit problems like these:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The main agent dispatches tasks within milliseconds. Several tasks start at once, fight each other, and eat all the memory: OOM.&lt;/li&gt;
&lt;li&gt;Tasks that look perfectly reasonable hold onto resources for ages, and the agent's own scheduled check-ins are unreliable, so nobody comes back to look.&lt;/li&gt;
&lt;li&gt;Every script writes straight into the database, with a real risk of corrupting it; task records are scattered everywhere and slow to look up.
And so on.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So we need two things: a task management system and a database service. The task management system coordinates resources and dispatches tasks; the database service validates and writes the data. In the repository the former is called &lt;code&gt;task-server&lt;/code&gt; and the latter &lt;code&gt;prune-server&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  0. The starting point: every task goes through one entrance
&lt;/h3&gt;

&lt;p&gt;There is exactly one rule: &lt;strong&gt;every task that runs project code is launched through the task service&lt;/strong&gt;. Builds, tests, gates, experiments — light or heavy, CPU or GPU — all go into the same queue.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Submit a task: the command goes verbatim after --&lt;/span&gt;
task-server/task submit -- cargo &lt;span class="nb"&gt;test&lt;/span&gt; --release -p singlefs-verification --test crash_enumeration_fixed_script_stream

&lt;span class="c"&gt;# Get the outcome: exits on completion, with the task's exit code&lt;/span&gt;
task-server/task &lt;span class="nb"&gt;wait&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;task-id&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The parameters that used to be passed at submission — thread count, memory cap, which machine, which card — are all gone. The service decides these. If a command specifies one, it is rejected on the spot, and the agent is told which parameter to remove. The reason is simple: one of the things agents are best at is asking for a little more resource for their own tasks.&lt;/p&gt;

&lt;p&gt;The service itself is a user-level systemd service: FastAPI on uvicorn, listening only on a unix socket. Tasks and every attempt are recorded in an embedded SQLite database, and each row carries an HMAC check column. Each attempt is its own systemd unit, wrapped in a cgroup memory cap: if it goes over, the kernel kills only that unit and nobody else.&lt;/p&gt;

&lt;p&gt;The "just launch it directly" path is blocked at the tool layer. Every Bash command an agent runs passes through a hook first; any cargo, gate, or experiment script not launched via &lt;code&gt;task submit&lt;/code&gt; is rejected on the spot. Sub-agents have had their right to run verifications revoked: they write the commands they want run into their report and hand it to the main agent to submit.&lt;/p&gt;

&lt;p&gt;Results come back as a code table, each outcome mapped to a fixed exit code, so the agent knows what to do next just from the exit code:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# task-server/task_server_lib.py (excerpt)&lt;/span&gt;
&lt;span class="c1"&gt;# Outcome types: name → (exit code of wait, None means the command's own; description)&lt;/span&gt;
&lt;span class="no"&gt;OUTCOME_TYPES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"succeeded"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"command exited 0"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"command-failed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"command itself exited non-zero"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"memory-cap-hit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;250&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"hit this task's own cap"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"memory-cap-at-maximum"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;236&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"hit the cap, already at the max of both hosts"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"card-fault"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;237&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"scrub found the card faulty"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"never-fits"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;241&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"won't fit on any host no matter how long it waits"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"stalled"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;233&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"log didn't grow for 10 minutes; the service stopped it"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"over-four-hours"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;229&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cumulative or projected time reached the 4-hour hard limit; bounced by the service"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"killed-externally"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;228&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"killed by a TERM or KILL not sent by the service"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="c1"&gt;# ...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Another key point: the agent's scheduled check-ins are unreliable. So the service can't rely on the agent remembering to come back. When a task reaches an outcome, has been queued too long, or looks stuck, the service sends a message straight to the inbox socket of &lt;strong&gt;the session that submitted the task&lt;/strong&gt;, waking the agent up. The first line says whether this is an outcome or a reminder; what follows is the evidence and the way forward. The agent never has to poll or set an alarm.&lt;/p&gt;

&lt;p&gt;At first everything went very smoothly. The task management system launched tasks centrally within the configured resource limits, queued what didn't fit, and bounced failures and errors back — in short, it handed results back to the agent, which carried on according to what came back. On top of that, a periodic sweep checked task states and terminated anything abnormal.&lt;/p&gt;

&lt;p&gt;Then the problems arrived right on cue.&lt;/p&gt;
&lt;h3&gt;
  
  
  1. Can tasks cut in line?
&lt;/h3&gt;

&lt;p&gt;Say memory is 50GB. Task A is using 32GB, and the next task, B, needs 24GB, so clearly B can't start yet. But C, behind B, needs only 16GB. Can C cut in? What about D after it? If we keep letting later tasks cut in, when will B ever get done?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Design: cutting in is allowed, but it's a matter of time.&lt;/strong&gt; C's estimated runtime must not exceed A's remaining runtime. On top of that, there's a maximum queue time: once a task has waited two hours, it gets top priority and nobody may cut in front of it.&lt;/p&gt;

&lt;p&gt;In the implementation this is called "gap-filling." How much longer the task in the way still has to run is called the "blocking time":&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# task-server/task_server.py (excerpt)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;attempt_remaining&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""How much longer (seconds) this running attempt has: estimate minus elapsed;
    with no estimate, or once past the estimate, take elapsed."""&lt;/span&gt;
    &lt;span class="n"&gt;estimate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;attempt_estimate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;elapsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"started_at"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;estimate&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;estimate&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;elapsed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;estimate&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;elapsed&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;elapsed&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The second half is a small trick: if a task has no runtime history, or has already run past its estimate, we guess it will run "at least as long again as it already has."&lt;/p&gt;

&lt;p&gt;Whether a task can cut in is decided in two steps:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# gap_fill in task-server/task_server.py (excerpt, self-test branches removed)&lt;/span&gt;
&lt;span class="c1"&gt;# Step 1: after B at the head starts, does the share still free on this host fit C?&lt;/span&gt;
&lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;total&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;margin&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;head_booked&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;beside&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;booked&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"gap-fill: fits, estimate not consulted"&lt;/span&gt;

&lt;span class="c1"&gt;# Step 2: only tasks that would eat into B's share are compared by time&lt;/span&gt;
&lt;span class="n"&gt;estimate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;task_estimate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;estimate&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"no gap-fill: this task has no estimate"&lt;/span&gt;
&lt;span class="n"&gt;blocking&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;blocker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;blocking_seconds&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# blocking time&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;estimate&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;blocking&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"gap-fill: estimate within blocking time"&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"no gap-fill: estimate exceeds blocking time"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;If C never touches the memory B needs, however long it runs it won't delay B, so it starts right away. Only tasks that would eat into B's share are compared by time: C must finish before A does.&lt;/p&gt;

&lt;p&gt;The two-hour rule takes just two lines:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;aged&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Still queued aged_seconds after submission: high priority."""&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"submitted_at"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aged_seconds&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;queue_order&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;queued&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Queue order: high priority first, each keeping its original order (stable sort)."""&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;queued&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;aged&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Big and small tasks alternate per host. A task whose booked amount reaches 85% of a host's usable limit counts as a "whole"; everything else is a "fragment." Fragments fill a batch, the batch is sealed, and the whole gets its turn; once the whole has started, the next batch of fragments is filled. Otherwise fragments keep arriving one after another and the whole never gets to see the host empty.&lt;/p&gt;
&lt;h3&gt;
  
  
  2. Where does the runtime come from, and is it accurate?
&lt;/h3&gt;

&lt;p&gt;Cutting in requires comparing times. Where do the times come from?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;From the database.&lt;/strong&gt; When each attempt wraps up, the service records its runtime. The estimate is the runtime of the most recent successful run of the same command; if, among the last five runs, one ran longer before failing, that longer one is taken. Failures that exited right at the start don't count: they say nothing about how long the command takes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tasks with no record may not cut in.&lt;/strong&gt; That's the &lt;code&gt;estimate is None&lt;/code&gt; → &lt;code&gt;False&lt;/code&gt; in the code above.&lt;/p&gt;

&lt;p&gt;Is it accurate? No. The same command runs fast when it reuses a lot from the verdict database and slow when it reuses little. So for entry points that run item by item, the service estimates more finely: before the real run, it launches an estimation sub-task in which the entry point reports, for each item, "how many to judge fresh and how many can be reused":&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ITEM_PLAN item=&amp;lt;item name&amp;gt; to_judge=&amp;lt;count to judge fresh&amp;gt; reusable=&amp;lt;count reusable&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;After each run, the entry point also reports how long each item actually took:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ITEM_DONE item=&amp;lt;item name&amp;gt; seconds=&amp;lt;seconds&amp;gt; judged=&amp;lt;count judged fresh&amp;gt; reused=&amp;lt;count reused&amp;gt; exit=&amp;lt;code&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;From this the service records, per item, "seconds per fresh judgment, seconds per reuse," and the next estimate is the sum of counts times rates. It's still just an estimate, so it's only used to decide whether to cut in. Sadly, this isn't very accurate either. Ha.&lt;/p&gt;
&lt;h3&gt;
  
  
  3. How much memory does a task need?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;For a task that has never run, first look for its kin; if there's no kin either, give it 4GB. If it hits the cap midway, raise the cap by 1.5× and requeue it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;"Kin" is identified by a command key. For &lt;code&gt;cargo test&lt;/code&gt;, for example, the same package, same features, and same build profile make one key; which test target is named does not go into the key — they build the same thing, and most of the memory goes into compilation.&lt;/p&gt;

&lt;p&gt;With a record, take the largest of the peaks measured in the last five runs:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# task-server/task_server.py (excerpt)&lt;/span&gt;
&lt;span class="no"&gt;RECORD_WINDOW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;recorded_prediction&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Peaks measured over the last RECORD_WINDOW runs of the same command key:
    prediction, cap and booking all equal the largest one, not multiplied by 1.5."""&lt;/span&gt;
    &lt;span class="n"&gt;window&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;recorded_measurements&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="no"&gt;RECORD_WINDOW&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
    &lt;span class="n"&gt;predicted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;cap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ceil&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;predicted&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;lib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;MEBIBYTE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;lib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;MEBIBYTE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Without a record, start at 4GB and grow on each hit:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# task-server/task_server_lib.py&lt;/span&gt;
&lt;span class="no"&gt;CAP_GROWTH_FACTOR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;1.5&lt;/span&gt;
&lt;span class="no"&gt;UNRECORDED_MEMORY_DEFAULT_BYTES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;GIBIBYTE&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;grown_cap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cap_bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ceil&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cap_bytes&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;CAP_GROWTH_FACTOR&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="no"&gt;MEBIBYTE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;MEBIBYTE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;4096M → 6144M → 9216M, bumping up one step at a time. If multiplying by 1.5 would reach the whole host's limit, the task is no longer requeued; it's bounced back to the session that submitted it, with suggested fixes: verify the scope, shrink it, or shard it.&lt;/p&gt;

&lt;p&gt;For tasks that hit the cap several times in a row, the service adds a paragraph to the message — "check the code for logic loops or memory leaks" — and lists every cap and peak in order: is the peak growing each time, is it pinned at the cap every time? Not fitting in 4GB may mean the task really is big; still not fitting after climbing to 40GB usually means something is wrong with the code.&lt;/p&gt;

&lt;p&gt;There's a pitfall we stepped in here: &lt;strong&gt;what exactly is the peak measuring?&lt;/strong&gt; cgroup's &lt;code&gt;memory.peak&lt;/code&gt; counts page cache too. One build task used under 1GB in its own processes, but page cache pushed the reading to 50GB. Setting quotas from that number would mean handing the entire machine to that one task. So the peak the service records counts only memory the processes themselves use (&lt;code&gt;anon&lt;/code&gt; plus &lt;code&gt;shmem&lt;/code&gt;), not page cache.&lt;/p&gt;

&lt;p&gt;At this point, the main agent's concurrency problem and the problem of misbehaving tasks hogging resources were solved. And with a task database in place, the lookup and record-keeping problems were solved too. Most importantly, verdicts accumulated from small everyday tests can be reused by later tests, and bit by bit they fill in coverage for the whole project.&lt;/p&gt;

&lt;p&gt;But it wasn't over. New problems came hard on the heels of the old.&lt;/p&gt;
&lt;h3&gt;
  
  
  4. Gate results that couldn't be reused
&lt;/h3&gt;

&lt;p&gt;In the original design, a light gate ran day to day and a heavy gate ran at commit time. Then I discovered the light gate was actually a completely separate codebase. That meant every run started from scratch, reused nothing, and its results were useless to the heavy gate.&lt;/p&gt;

&lt;p&gt;Hooking the light gate into the pruning database would amount to rewriting a scaled-down heavy gate. Merging it into the heavy gate instead would mean the heavy gate had to allow runs scoped by range and by named item.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Solution: take the heavy gate's whole verification system apart and make it addressable by name. Delete the light gate.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Now there is just one gate, with two ways to run it:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;How to launch&lt;/th&gt;
&lt;th&gt;When&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Unscoped&lt;/td&gt;
&lt;td&gt;No selection flags; the gate picks what to judge by key on its own&lt;/td&gt;
&lt;td&gt;At commit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scoped&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;--item &amp;lt;item&amp;gt;&lt;/code&gt;, &lt;code&gt;--check &amp;lt;cell&amp;gt;&lt;/code&gt;, named test targets&lt;/td&gt;
&lt;td&gt;Any time&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# List the items that can be named; no build, no run&lt;/span&gt;
bash .claude/gate.d/crash-point-replay.sh --list-items

&lt;span class="c"&gt;# Judge only the one crash-enumeration case changed this time&lt;/span&gt;
bash .claude/gate.d/crash-point-replay.sh --check crash-case-replay --crash-cases-only --item crash-case:&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;case&lt;/span&gt; name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The crucial point: &lt;strong&gt;both modes are the same script and the same verdict database; only the scope differs.&lt;/strong&gt; Verdicts produced by everyday named runs are written to the database, and the commit-time run reads them by key and reuses them directly. Each verdict's key consists only of what it depends on: the fingerprint of the function unit with comments stripped, the toolchain, dependency versions, and its own inputs. Paths, line numbers, and which commit — none of these go into the key.&lt;/p&gt;

&lt;p&gt;There is only one verdict database, too: the RocksDB held by &lt;code&gt;prune-server&lt;/code&gt;. Tasks on both machines read and write through it, and each kind of verdict gets its own key prefix. Any script that opens a separate directory or writes a separate table to store verdicts is bounced in review, no exceptions. This is the database service mentioned at the start.&lt;/p&gt;

&lt;p&gt;To keep "everyday" from quietly turning back into "everything," the service added a confirmation step at the entrance. After submission a task doesn't enter the queue right away; the service prints a confirmation slip: how many items it recognized from the command, estimated memory, estimated time. The agent has to copy two of those numbers back verbatim before the task enters the queue:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;task-server/task confirm &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;task-id&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --items &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;N&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --tasks &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;M&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Unscoped runs and whole-package tests are rejected on the spot anywhere other than at commit.&lt;/p&gt;
&lt;h3&gt;
  
  
  5. Waiting on very long verifications
&lt;/h3&gt;

&lt;p&gt;What if a verification takes 7 hours? Everything behind it is stuck. One verification, and no other development can happen.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Solution: sharding. Cut the task into small pieces.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The entry script declares it in a header line, and the service knows the job should be sharded:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# task-server-shards: *&lt;/span&gt;
&lt;span class="c"&gt;# task-server-compiles: yes&lt;/span&gt;
&lt;span class="c"&gt;# task-server-card-lease: on-demand&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The service splits a job into three stages, each an ordinary sub-task that queues, picks a machine, sets its memory, and retries on its own:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Plan&lt;/strong&gt;: the entry point writes a &lt;code&gt;plan.json&lt;/code&gt; into the plan directory saying how many shards to cut.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shards&lt;/strong&gt;: one sub-task per shard, with &lt;code&gt;TASK_SERVER_SHARD=&amp;lt;shard&amp;gt;/&amp;lt;count&amp;gt;&lt;/code&gt; in its environment; it judges only its own share and writes verdicts straight into the verdict database.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Merge&lt;/strong&gt;: starts only once all shards have succeeded. The merge judges nothing fresh; it only reconciles: everything each shard was supposed to judge can be read from the database right now.
If one shard fails, its siblings still run to the end and their verdicts still go into the database. On resubmission, only what hasn't been judged gets filled in.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Beyond sharding there are two hard time rules:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;After two hours of running, the service doesn't stop the task, but it sends the submitting session a reminder: is the scope right? Does it really need to run this long?&lt;/li&gt;
&lt;li&gt;At four hours cumulative, the service stops it and bounces it, no negotiation. There's only one way forward: split the item, or optimize the algorithm. Shrinking the verification domain or naming fewer items doesn't count as splitting.
### 6. The crashing GPU&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Small pieces brought a new problem. Code bugs plus scheduling problems crashed the GPU outright. And when the GPU goes down, it takes the network with it.&lt;/p&gt;

&lt;p&gt;I wrote two separate GPU articles about this a couple of days ago, if you're interested.&lt;/p&gt;


&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04" class="crayons-story__hidden-navigation-link"&gt;NVIDIA Xid 62 : PMU has halted&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/faliye" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" alt="faliye profile" class="crayons-avatar__image" width="96" height="96"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/faliye" class="crayons-story__secondary fw-medium m:hidden"&gt;
              faliye
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                faliye
                
                
              
              &lt;div id="story-author-preview-content-4977032" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/faliye" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" class="crayons-avatar__image" alt="" width="96" height="96"&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;faliye&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Nov 4&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04" id="article-link-4977032"&gt;
          NVIDIA Xid 62 : PMU has halted
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/gpu"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;gpu&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/nvidia"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;nvidia&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/gsp"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;gsp&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/xid62"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;xid62&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
            &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              &lt;span class="hidden s:inline"&gt;Add&amp;nbsp;Comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            3 min read
          &lt;/small&gt;
          
          &lt;div class="only-sidebar-menu-item trusted-visible-block print-hidden align-center"&gt;
                
                    
                
          &lt;/div&gt;

        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;


&lt;/div&gt;
&lt;br&gt;



&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/faliye/nvidia-xid-109-the-gpu-timed-out-then-wi-fi-died-too-44k9" class="crayons-story__hidden-navigation-link"&gt;NVIDIA Xid 109: The GPU Timed Out, Then Wi-Fi Died Too&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/faliye" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" alt="faliye profile" class="crayons-avatar__image" width="96" height="96"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/faliye" class="crayons-story__secondary fw-medium m:hidden"&gt;
              faliye
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                faliye
                
                
              
              &lt;div id="story-author-preview-content-5007610" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/faliye" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" class="crayons-avatar__image" alt="" width="96" height="96"&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;faliye&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/faliye/nvidia-xid-109-the-gpu-timed-out-then-wi-fi-died-too-44k9" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Nov 10&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/faliye/nvidia-xid-109-the-gpu-timed-out-then-wi-fi-died-too-44k9" id="article-link-5007610"&gt;
          NVIDIA Xid 109: The GPU Timed Out, Then Wi-Fi Died Too
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/nvidia"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;nvidia&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/xid109"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;xid109&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/gpu"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;gpu&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/wgsl"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;wgsl&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
            &lt;a href="https://dev.to/faliye/nvidia-xid-109-the-gpu-timed-out-then-wi-fi-died-too-44k9#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              &lt;span class="hidden s:inline"&gt;Add&amp;nbsp;Comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            3 min read
          &lt;/small&gt;
          
          &lt;div class="only-sidebar-menu-item trusted-visible-block print-hidden align-center"&gt;
                
                    
                
          &lt;/div&gt;

        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;


&lt;/div&gt;


&lt;ul&gt;
&lt;li&gt;Before handing over a card, the service scrubs it and reads it back before giving it to the task.&lt;/li&gt;
&lt;li&gt;After a task ends, the service reclaims the card first: reads VRAM, reads compute processes, reads the kernel log for that period. A card that threw an Xid is benched for 60 minutes, and the task is requeued onto another card.&lt;/li&gt;
&lt;li&gt;A card judged faulty gets the outcome &lt;code&gt;card-fault&lt;/code&gt;, and a human is asked on the spot to reset it. No waiting, no retrying.
### 7. Tasks that could only run on one machine&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Because of resource constraints, some tasks could, believe it or not, only run on one machine. Copy the project to the second machine? Changes are far too frequent; every copy and every reconciliation makes token consumption explode.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Solution: share the codebase over NFSv4, with both machines mounting it at the same absolute path.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# This host: /etc/exports, one line per shared root&lt;/span&gt;
&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;root&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;second host address&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;rw,sync,no_subtree_check&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# Second host: /etc/fstab, mount at the same absolute path as this host&lt;/span&gt;
&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;this host address&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;:&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;root&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;root&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; nfs4 rw,hard,nofail,_netdev,x-systemd.automount,x-systemd.mount-timeout=30 0 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three things are shared: the repository itself, the data root (working directories and shard artifacts live under it), and the session log directory. The second machine runs in the same live tree — no copying, no sending, no fetching back, not a single character of any path rewritten. What one machine writes, the other reads directly.&lt;/p&gt;

&lt;p&gt;The only exception is the build directory: it lives on each machine's own local disk, is created fresh for every run, and is deleted afterwards. It never goes in a shared directory.&lt;/p&gt;

&lt;p&gt;In the service this is a startup condition. On startup the service checks four things: is the configuration present, has this host exported, has the second host mounted, and can each side read a probe file written by the other in both directions. If any check fails, the service doesn't start.&lt;/p&gt;

&lt;p&gt;The matching principle went into the project rules: &lt;strong&gt;no task must be bound to one machine; if one is, that's a defect.&lt;/strong&gt; The service has no "this host only" list. If a task dispatched to the second machine goes red, let it go red and go fix the script — don't add exceptions to the service.&lt;/p&gt;

&lt;p&gt;Well, that's more or less it. The service's own verification tasks had to queue behind those very long tasks, which made verifying and developing the service very slow. The classic awkward situation: the condition for getting in can only be met once you're in. But in the end these problems were solved.&lt;/p&gt;

&lt;p&gt;And then, a new storm appeared.&lt;br&gt;
CPU tasks were in fact solved by this point: CPU and memory form a single resource pool, and scheduling is fairly stable. But GPU tasks held their cards hostage.&lt;/p&gt;
&lt;h3&gt;
  
  
  8. CPU and GPU tied together
&lt;/h3&gt;

&lt;p&gt;A crash-point replay task first builds and records on the CPU, and only at the very end goes onto a card to judge. Yet it holds the card from the moment it starts. Much of the time, the GPU is just waiting.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Solution: decouple. Assign no card at start; lease one only when the task reaches the step that needs it, and return it immediately after use.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;# task-server-card-lease: on-demand&lt;/code&gt; line in the entry file header is that declaration. For such tasks the service assigns no card and does no scrub at start, judging memory and CPU like any ordinary task. In the gate it looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# .claude/gate.d/crash-point-replay.sh (excerpt)&lt;/span&gt;
crash_point_replay_take_card&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;local&lt;/span&gt; cards lease_exit
  &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt; -z &lt;span class="s2"&gt;"$crash_point_replay_card_leased"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;return&lt;/span&gt; 0      &lt;span class="c"&gt;# already holding one: use it&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nv"&gt;cards&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$(crash_point_replay_card_lease take)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="nv"&gt;lease_exit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nv"&gt;lease_exit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt; lease_exit != 0 &lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt; -z &lt;span class="s2"&gt;"$cards"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"  ✗ $1: $2 needs a card; on-demand lease refused, this run did not start"&lt;/span&gt;
    &lt;span class="nb"&gt;return&lt;/span&gt; 1
  &lt;span class="k"&gt;fi&lt;/span&gt;
  &lt;span class="nv"&gt;crash_point_replay_card_leased&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$cards"&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"  · $1: $2 needs a card; leased $cards on demand (returned when this run ends)"&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Behind &lt;code&gt;lease take&lt;/code&gt;, the service picks a free card by card priority, scrubs it, reads it back, and only then hands it over. If all cards are taken, the request waits in line, first come first served. Any card not returned when a task ends is reclaimed by the service.&lt;/p&gt;

&lt;p&gt;The tests in the verification package were split into two stages accordingly:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Record only&lt;/strong&gt;: runs on the CPU and writes the inputs to be judged into the verdict database. Holds no card.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Judge only&lt;/strong&gt;: leases a card, reads the inputs back from the database, judges them on the card, and returns the card when done.
The other half was done too: once a task has a card and enters its GPU stage, its CPU accounting switches to the number of cores actually in use during that card-holding stage. While it waits on the card's results, its CPU slots go to others.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The time a card spends in hand shrank from "the whole task" to "the part that's actually computing."&lt;/p&gt;

&lt;h3&gt;
  
  
  Finally, more or less there
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Memory no longer gets eaten by several tasks starting at once. Admission is decided in exactly one place, the service, and only what qualifies gets started.&lt;/li&gt;
&lt;li&gt;No task holds resources indefinitely. Stuck ones are stopped, long ones get reminders, four-hour ones are bounced.&lt;/li&gt;
&lt;li&gt;Every outcome has a name, an exit code, and a next step, and it's delivered to the session that started it.&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Verdicts from everyday runs aren't wasted; the commit-time run picks them up.&lt;br&gt;
But there are still plenty of problems:&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Estimated runtime is historical data, not a promise. A run that reuses a lot and one that reuses little can differ enormously.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Starting at 4GB and bumping up by 1.5× trades reruns for convenience. A genuinely new 40GB task has to hit the cap six times before it runs stably.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The service is code too. Its self-tests run in a simulated world: fake memory wrappers, a fake second machine, fake cards. Running it for real is a stumbling affair.&lt;br&gt;
Fortunately, after more than 10,000 tasks, it has more or less settled down. Of course — trust me — countless bugs and optimizations are still waiting down the road.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Before refactoring, the bottleneck was the CPU and memory, leaving the GPU idle most of the time. After refactoring, the bottleneck shifted to the GPU.&lt;br&gt;
Conclusion: there's always something that's not enough. lol&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Problems are like the tree in the yard. When the leaves fall, countless branches are laid bare...&lt;/p&gt;
&lt;/blockquote&gt;

</description>
      <category>claude</category>
      <category>singlefs</category>
      <category>rust</category>
      <category>filesystem</category>
    </item>
    <item>
      <title>NVIDIA Xid 109: The GPU Timed Out, Then Wi-Fi Died Too</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Tue, 10 Nov 2026 00:34:39 +0000</pubDate>
      <link>https://dev.to/faliye/nvidia-xid-109-the-gpu-timed-out-then-wi-fi-died-too-44k9</link>
      <guid>https://dev.to/faliye/nvidia-xid-109-the-gpu-timed-out-then-wi-fi-died-too-44k9</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;I don't understand NVIDIA's idea of fun. I don't understand Old Huang's (Jensen) idea of fun either.&lt;/p&gt;
&lt;/blockquote&gt;


&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04" class="crayons-story__hidden-navigation-link"&gt;NVIDIA Xid 62 : PMU has halted&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/faliye" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" alt="faliye profile" class="crayons-avatar__image"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/faliye" class="crayons-story__secondary fw-medium m:hidden"&gt;
              faliye
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                faliye
                
                
              
              &lt;div id="story-author-preview-content-4977032" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/faliye" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" class="crayons-avatar__image" alt=""&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;faliye&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Nov 4&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04" id="article-link-4977032"&gt;
          NVIDIA Xid 62 : PMU has halted
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/gpu"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;gpu&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/nvidia"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;nvidia&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/gsp"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;gsp&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/xid62"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;xid62&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
            &lt;a href="https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              &lt;span class="hidden s:inline"&gt;Add&amp;nbsp;Comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            3 min read
          &lt;/small&gt;
          
          &lt;div class="only-sidebar-menu-item trusted-visible-block print-hidden align-center"&gt;
                
                    
                
          &lt;/div&gt;

        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;


&lt;p&gt;After my previous Xid 62 adventure, I ran into &lt;code&gt;Xid 109 CTX SWITCH TIMEOUT&lt;/code&gt;. This one was worse. At some unknown point, the GPU would log Xid 109 without much fanfare. About ten minutes later, Wi-Fi would stop responding and my SSH session would drop.&lt;/p&gt;

&lt;p&gt;Normally that means hauling over a monitor and keyboard for another round of combat. Fortunately, I had an Ethernet cable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Xid 109: My Own Kernel Was Too Slow
&lt;/h2&gt;

&lt;p&gt;Xid 109 means a GPU context-switch timeout. The driver may terminate the offending channel.&lt;/p&gt;

&lt;p&gt;After debugging Xid 62 last time, I immediately rejected the Agent's suggestion that the hardware might be at fault. The actual culprit was my WGSL GPU kernel.&lt;/p&gt;

&lt;p&gt;In &lt;code&gt;singlefs&lt;/code&gt;, the recovery kernel's &lt;code&gt;mapping_lookup&lt;/code&gt; performed a linear scan, twice per unit. At its worst, there were 1,124 mapping entries and 2,264 lookups, giving the overall work roughly &lt;strong&gt;O(N²)&lt;/strong&gt; growth. To make matters worse, &lt;code&gt;task-server&lt;/code&gt; initially submitted ten states at once for efficiency. The heavy validation workload kept the GPU busy too long, preventing a timely context switch and triggering Xid 109.&lt;/p&gt;

&lt;p&gt;The fix was straightforward:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Replace the linear scan in &lt;code&gt;mapping_lookup&lt;/code&gt; with binary search.&lt;/li&gt;
&lt;li&gt;Start with one state per initial dispatch, then adapt the batch size based on measured runtime.&lt;/li&gt;
&lt;li&gt;Explicitly mark GPU device loss, so a test that never really ran cannot be counted as a successful bug detection. Earlier results need another look under this rule.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;After the changes, the longest measured GPU slice was &lt;strong&gt;0.203 seconds&lt;/strong&gt;. All nine tests passed, and neither RTX 5060 Ti showed a new Xid 109 in the subsequent log checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  But Why Did Wi-Fi Die Too?
&lt;/h2&gt;

&lt;p&gt;My working theory is that &lt;strong&gt;after Xid 109, something went wrong with PCIe or interrupt handling—possibly an interrupt storm—which indirectly wedged the &lt;code&gt;mt7925e&lt;/code&gt; Wi-Fi driver&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The PCIe devices on this machine are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;RTX 5060 Ti: &lt;code&gt;10:00.0&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Wi-Fi: &lt;code&gt;08:00.0&lt;/code&gt;, using &lt;code&gt;mt7925e&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Ethernet: &lt;code&gt;07:00.0&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The GPU and Wi-Fi share the chipset's upstream path, though they are not on the same downstream PCIe port.&lt;/p&gt;

&lt;p&gt;In both incidents, the first DNS failures appeared roughly 10–15 minutes after Xid 109. But I have no direct proof. There were no &lt;code&gt;nobody cared&lt;/code&gt;, &lt;code&gt;Disabling IRQ&lt;/code&gt;, or PCIe AER messages. The &lt;code&gt;mt7925e&lt;/code&gt; timeouts were logged only after I manually reconfigured the network or unloaded the driver. I have also seen Wi-Fi failures without a nearby Xid 109.&lt;/p&gt;

&lt;p&gt;After fixing the WGSL kernel, Wi-Fi stayed up for 12 hours. Interesting, but hardly proof of causation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recovering Wi-Fi Without Rebooting
&lt;/h2&gt;

&lt;p&gt;I hope you never need these commands. Ethernet was still working, so I connected over SSH:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh -i ~/.ssh/id_ed*** fy5090@192.***.***.1

&lt;span class="c"&gt;# 1. Try a lightweight network reconfiguration (leaves Ethernet alone)&lt;/span&gt;
sudo networkctl reconfigure wlp8s0

&lt;span class="c"&gt;# 2. If that fails, bring the Wi-Fi interface down and back up&lt;/span&gt;
sudo networkctl down wlp8s0 &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; sudo networkctl up wlp8s0

&lt;span class="c"&gt;# 3. Still no luck? Check the actual wpa service name&lt;/span&gt;
systemctl list-units --all &lt;span class="o"&gt;|&lt;/span&gt; grep wpa
sudo systemctl restart wpa_supplicant
&lt;span class="c"&gt;# Or: sudo systemctl restart wpa_supplicant@wlp8s0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the driver itself is completely wedged, those commands may not help. With Ethernet or a local console available, you can try reloading it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo modprobe -r mt7925e
sudo modprobe mt7925e
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Even unloading the driver can hang, so save your logs first.&lt;/p&gt;

&lt;h2&gt;
  
  
  One Last Thing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Xid 109 is identified and fixed&lt;/strong&gt;: the recovery kernel's near-O(N²) lookup was the culprit. &lt;strong&gt;The Wi-Fi freeze is still unexplained&lt;/strong&gt;: the shared PCIe path and interrupt storm remain a hypothesis, not a proven cause.&lt;/p&gt;

&lt;p&gt;I've installed &lt;code&gt;wifi-watchdog&lt;/code&gt; to capture evidence the next time it happens.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Old Huang is happy, so I'll call him &lt;strong&gt;Huang-le&lt;/strong&gt; (le means "happy").&lt;/p&gt;

&lt;p&gt;I'm not happy. no &lt;strong&gt;dao-le&lt;/strong&gt; ("dollar").&lt;/p&gt;
&lt;/blockquote&gt;

</description>
      <category>nvidia</category>
      <category>xid109</category>
      <category>gpu</category>
      <category>wgsl</category>
    </item>
    <item>
      <title>NVIDIA Xid 62 : PMU has halted</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Wed, 04 Nov 2026 17:53:34 +0000</pubDate>
      <link>https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04</link>
      <guid>https://dev.to/faliye/troubleshooting-nvidia-error-xid-62-1f04</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;To drink some milk, I'm fixing the barn.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;singlefs's verification system needs GPUs for cross-checking. Each agent used to call its own scripts, which stayed messy and left resources poorly used.&lt;br&gt;
I recently refactored that code and built a task management service. Memory management, sharding and multi-machine coordination now go much more smoothly than before.&lt;/p&gt;

&lt;p&gt;Well, as expected, the unexpected happened. A GPU crashed...&lt;/p&gt;

&lt;p&gt;It starts with Xid 62 and ends with Xid 154. In the kernel log, GSP reports "PMU has halted", with the watchdog at partition 4#0, task 3. It also can't be recovered by command: &lt;code&gt;nvidia-smi --gpu-reset -i 0&lt;/code&gt; returned "Not Supported" both times.&lt;/p&gt;

&lt;p&gt;At first I didn't care, but it crashed five times in one day. This caused collateral damage to the task management service. When the service clears the stage, it checks every card on the machine, so if any one is broken, the whole machine can't start. Worse, the service marked each healthy card it was assigned as broken and disabled them one by one. One card's fault became a whole machine down.&lt;/p&gt;

&lt;p&gt;So on the second crash I added an alert, on the third error tolerance, on the fourth task reassignment, and on the fifth, this stopped being fun...&lt;/p&gt;

&lt;p&gt;After some searching, there were three possible causes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Power supply&lt;/li&gt;
&lt;li&gt;Firmware&lt;/li&gt;
&lt;li&gt;GSP scheduling&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The power supply can't be the problem: after a reboot the card recovers and can run tasks for a long time.&lt;br&gt;
2 and 3 are really the same thing. The GSP firmware ships with the driver, and the PMU halt is reported from the firmware side. The problem surviving a driver change doesn't mean "not firmware"; it means none of those driver versions has fixed it yet. I read an article by an expert dated August 19, 2026, and it made a lot of sense: when the GPU switches frequently from very low power (P8) to high power (P0), the PMU state machine hangs and causes &lt;code&gt;Xid 62&lt;/code&gt;. It seems to be a scheduling deadlock in the Blackwell GSP firmware.&lt;/p&gt;

&lt;p&gt;So I lined up per-second telemetry (P-state, clocks, power) against the &lt;code&gt;GPU_LIFECYCLE&lt;/code&gt; timestamps emitted by the tests and the precise kernel timestamps. After the card sat idle in P8 for about 13 seconds, Xid 62 hit during the 94 ms in which the next test was creating resources on the card, 2 ms before any work was actually submitted.&lt;/p&gt;

&lt;p&gt;After locking the minimum clock at 800 MHz or above, the same tests ran for 10 minutes without an error. The lock command is &lt;code&gt;nvidia-smi -i &amp;lt;bus id&amp;gt; -lgc 800,3105&lt;/code&gt;; I made it a systemd unit, &lt;code&gt;nvidia-5080-min-clock&lt;/code&gt;, that starts at boot. Resetting the card clears the lock, so it has to be reapplied.&lt;/p&gt;

&lt;p&gt;The machine doesn't need a reboot: you can reset the card by unbinding and remounting it. &lt;code&gt;0000:01:00.0&lt;/code&gt; below is the bus ID on my machine; check yours first with &lt;code&gt;lspci | grep -i nvidia&lt;/code&gt;. The commands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/sys/bus/pci/devices/0000:01:00.0
&lt;span class="c"&gt;# 1. Preserve the evidence first: AER status registers (a reset clears them)&lt;/span&gt;
lspci -s 01:00.0 -vv &lt;span class="o"&gt;|&lt;/span&gt; grep -iE &lt;span class="s1"&gt;'UESta|CESta|DevSta|LnkSta'&lt;/span&gt;
&lt;span class="c"&gt;# 2. The card must be fully released&lt;/span&gt;
fuser -v /dev/nvidia0
systemctl stop nvidia-persistenced
&lt;span class="c"&gt;# 3. Unbind the audio and GPU functions&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; 0000:01:00.1 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /sys/bus/pci/drivers/snd_hda_intel/unbind
&lt;span class="nb"&gt;echo&lt;/span&gt; 0000:01:00.0 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /sys/bus/pci/drivers/nvidia/unbind
&lt;span class="c"&gt;# 4. PCI-level reset (remove + rescan alone without this step leaves GSP unable to boot: "unexpected WPR2 already up")&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; 1 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$d&lt;/span&gt;/reset
&lt;span class="c"&gt;# 5. Remove both functions and rescan so the driver picks the card up again&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; 1 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /sys/bus/pci/devices/0000:01:00.1/remove
&lt;span class="nb"&gt;echo&lt;/span&gt; 1 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$d&lt;/span&gt;/remove
sleep 2
&lt;span class="nb"&gt;echo&lt;/span&gt; 1 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /sys/bus/pci/rescan
&lt;span class="c"&gt;# 6. Bring persistenced back and re-lock the clocks (a reset clears the lock)&lt;/span&gt;
systemctl start nvidia-persistenced
systemctl restart nvidia-5080-min-clock
&lt;span class="c"&gt;# 7. Verify: recovery action is None, the three cards keep their indices, the 5080 idles outside P8, no new RmInitAdapter failures in the kernel log&lt;/span&gt;
nvidia-smi -i 00000000:01:00.0 -q &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="s1"&gt;'recovery action'&lt;/span&gt;
nvidia-smi --query-gpu=index,name,pci.bus_id,pstate,clocks.sm --format=csv,noheader
journalctl -k --since &lt;span class="s1"&gt;'-2min'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'RmInitAdapter|bad state'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What am I doing up on the barn roof? Where's my milk?&lt;/p&gt;

</description>
      <category>gpu</category>
      <category>nvidia</category>
      <category>gsp</category>
      <category>xid62</category>
    </item>
    <item>
      <title>Design and Analysis of a Local Model Engineering runtime Gateway Based on vLLM</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Tue, 03 Nov 2026 07:29:21 +0000</pubDate>
      <link>https://dev.to/faliye/design-and-analysis-of-a-local-model-engineering-runtime-gateway-based-on-vllm-2mpf</link>
      <guid>https://dev.to/faliye/design-and-analysis-of-a-local-model-engineering-runtime-gateway-based-on-vllm-2mpf</guid>
      <description>&lt;p&gt;When using Ollama, several small models are started locally, and the business CoT (Chain of Thought) directly accesses these small models. Due to constraints, models under Ollama are difficult to run concurrently, and a fixed request is highly homogenized. Models can be selected based on requirements, and the risks of local small models lacking a toolchain are relatively limited.&lt;/p&gt;

&lt;p&gt;However, after setting up local models using vLLM, we completely disconnect the ability to directly access the vLLM port. All functions are routed through the API, which completely changes these circumstances.&lt;/p&gt;

&lt;p&gt;Simply inheriting the security protection of a single service for the vLLM service is no longer sufficient. It requires upstream intent recognition for small models, access control over the main model's toolchain invocation, admission control, and log recording with statistics.&lt;br&gt;
Additionally, the primary responsibility is to host multiple services simultaneously. Driven by this requirement, features like queues, priority control, output filtering and judgment, failure retry mechanisms, and the crucial thinking budget control are further expanded.&lt;br&gt;
These features might seem redundant, but they are absolutely not.&lt;/p&gt;

&lt;p&gt;First, we need to identify users and provide different features for different users. Let's design three scenarios:&lt;/p&gt;

&lt;h4&gt;
  
  
  Scenario 1
&lt;/h4&gt;

&lt;p&gt;A simple automated text summarization business often does not require a thinking process.&lt;br&gt;
A VSCode programming task needs to display the complete thinking process.&lt;br&gt;
For these two types of requirements, one can wait and then respond, while the other requires streaming transmission so that the model's thinking process can penetrate to the user.&lt;/p&gt;

&lt;h4&gt;
  
  
  Scenario 2
&lt;/h4&gt;

&lt;p&gt;To briefly explain, an excessively long thinking budget will lead to low model utilization efficiency; many "think" models suffer from overthinking. There is also a severe issue: if the allocated token budget is entirely consumed by the thinking process, no result will be returned.&lt;/p&gt;

&lt;p&gt;A simple query task A requires a thinking budget of 3000, and it might not even use it all.&lt;br&gt;
A complex character classification and recognition task B in a book requires a thinking budget of 20,000 or more.&lt;br&gt;
Therefore, in this scenario, the two users need different thinking budgets, and a soft budget might be more appropriate.&lt;/p&gt;

&lt;h4&gt;
  
  
  Scenario 3
&lt;/h4&gt;

&lt;p&gt;User A needs to call the network to query some information.&lt;br&gt;
User B does not need to call the network.&lt;br&gt;
External information is dangerous. Ideally, no external information should be input at all, but we all know this is impossible. Therefore, the toolchain should serve as a double safeguard. If the current user has permission to access network tools, they can only access them when explicitly requested; this capability cannot be opened to users without protection. Thus, reorganizing the tool declaration after the request comes in, and appending tool_choice=none by default, can improve security to a certain extent.&lt;br&gt;
Second, how to improve the availability of the service. Similarly, three scenarios:&lt;/p&gt;

&lt;h4&gt;
  
  
  Scenario 1: Multi-turn thinking and retry.
&lt;/h4&gt;

&lt;p&gt;In VSCode, a single conversation might involve multiple turns of thinking. This brings up an issue: every time a multi-turn thought occurs, a new request is initiated, and the context of the previous turn disappears. If it cannot be passed to the next round's request, it means the next round starts from scratch. We need to pass the previous round's context to the next round based on the session ID. Another similar problem is that the model is extremely prone to falling into a loop of repetitive thinking; without the content of the previous thought, the model starts anew every single time (lol).&lt;br&gt;
If a user's request fails due to a model issue, such as OOM, how should we retry? When retrying, we face the same problem: the thoughts generated in the previous round cannot be discarded.&lt;/p&gt;

&lt;h4&gt;
  
  
  Scenario 2: Shape learning of the thinking budget
&lt;/h4&gt;

&lt;p&gt;As mentioned in Scenario 2 of the first point, giving too few tokens in a single request while encountering too many thinking tokens will result in thinking without an answer. Giving too many tokens will cause waste.&lt;br&gt;
My method here is to dynamically adjust the user's thinking budget based on their request logs. This is the "shape learning" (pattern learning) of the gateway.&lt;br&gt;
Allocate a thinking budget to the user in advance based on their historical requests. When returning the result, inspect it: if the thinking is not completed, we must intercept the return, allocate a larger budget to the current request, overlay the partial results into the body of the new request, and send the request again.&lt;/p&gt;

&lt;h4&gt;
  
  
  Scenario 3: Priority queue and interruption
&lt;/h4&gt;

&lt;p&gt;After the model processes a request, among multiple services in the queue, which service can start first? This requires scanning.&lt;br&gt;
After the user terminates the request, does the model still need to finish its thinking? My personal judgment is that it is no longer needed, and the model should interrupt its ongoing thinking.&lt;br&gt;
Based on the above issues, we summarized the following architecture under the coexistence of multiple tasks.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faaptlf7elkfoflzrzhqw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faaptlf7elkfoflzrzhqw.png" alt=" " width="800" height="464"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Therefore, the model's capability is not unlocked simply by setting up vLLM and flipping a few switches and dial parameters on a black box.&lt;br&gt;
It requires a massive amount of practical engineering experience and accumulation. These practices are our current mastery of the process, or rather, the "harness".&lt;br&gt;
Of course, this feature is still very rudimentary and has many loopholes right now, but regardless, it's up and running, isn't it?&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl8d867eew24ylgrscgum.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl8d867eew24ylgrscgum.png" alt=" " width="800" height="771"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;By the way, does this system behave completely differently in Continue and Copilot Chat?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>runtime</category>
      <category>apigateway</category>
    </item>
    <item>
      <title>vLLM on 4 GPUs / 64GB: A Multi-Model Benchmark — For a Home Multi-Host Setup, I'd Go with vLLM</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Fri, 30 Oct 2026 09:20:28 +0000</pubDate>
      <link>https://dev.to/faliye/vllm-on-4-gpus-64gb-a-multi-model-benchmark-for-a-home-multi-host-setup-id-go-with-vllm-233h</link>
      <guid>https://dev.to/faliye/vllm-on-4-gpus-64gb-a-multi-model-benchmark-for-a-home-multi-host-setup-id-go-with-vllm-233h</guid>
      <description>&lt;p&gt;A while back I benchmarked llama.cpp on a 96GB GPU setup and found its concurrency behavior underwhelming.&lt;/p&gt;

&lt;p&gt;That isn't llama.cpp's fault. From the start it was built to solve the capacity problem — "this model won't fit in one machine's VRAM" — not the throughput problem of many users hitting the server at once. If you're running a 70B or even 104B model at home for personal research or solo testing, "it runs at all" is the top priority. Under concurrency, when 10 users send requests at the same time, the engine should combine those 10 into a single Batch to keep the GPU busy. But the moment you batch them, the tensor volume moving between host A and host B blows up by orders of magnitude — from a few MB to tens or even hundreds of MB.&lt;/p&gt;

&lt;p&gt;vLLM's whole architecture exists to solve high concurrency head-on. It slices the KV cache into fixed-size "pages," so VRAM doesn't need contiguous allocation: a request claims pages as it needs them and frees them the instant it finishes. On the communication side it uses NCCL directly, the CUDA-native library for GPUs. I have no NVLink here, but NCCL falls back gracefully and supports multiple links.&lt;/p&gt;

&lt;p&gt;Personally: people say vLLM plus tensor parallelism over a single LAN cable at home has all sorts of problems, and I do have to throw away 32GB of VRAM. Even so, a model that fits in 32GB of memory is still a step below one that fits in 64GB of VRAM. And if several tasks can share one larger model, I expect the results to be better overall.&lt;/p&gt;

&lt;p&gt;Anyway — let's measure it.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. First test: a head-to-head with llama.cpp
&lt;/h2&gt;

&lt;p&gt;The parameters were slightly off: llama.cpp's KV was q8_0 while vLLM defaulted to fp16, so llama.cpp had the advantage.&lt;/p&gt;

&lt;p&gt;Single-stream decode, tok/s:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;vLLM (TP4)&lt;/th&gt;
&lt;th&gt;llama.cpp&lt;/th&gt;
&lt;th&gt;Delta&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;short&lt;/td&gt;
&lt;td&gt;73.3±3.1&lt;/td&gt;
&lt;td&gt;73.1±0.1&lt;/td&gt;
&lt;td&gt;intervals overlap = no difference&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5K&lt;/td&gt;
&lt;td&gt;75.0±0.1&lt;/td&gt;
&lt;td&gt;70.5±0.2&lt;/td&gt;
&lt;td&gt;+6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32K&lt;/td&gt;
&lt;td&gt;71.4±0.5&lt;/td&gt;
&lt;td&gt;63.1±0.1&lt;/td&gt;
&lt;td&gt;+13%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;52K&lt;/td&gt;
&lt;td&gt;69.3±0.1&lt;/td&gt;
&lt;td&gt;58.0±0.1&lt;/td&gt;
&lt;td&gt;+20%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100K&lt;/td&gt;
&lt;td&gt;63.7±0.7&lt;/td&gt;
&lt;td&gt;48.2±0&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+32%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Concurrency (same N=4, end-to-end tok/s):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;vLLM&lt;/th&gt;
&lt;th&gt;llama.cpp&lt;/th&gt;
&lt;th&gt;Ratio&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;short×4&lt;/td&gt;
&lt;td&gt;188.1&lt;/td&gt;
&lt;td&gt;110.1&lt;/td&gt;
&lt;td&gt;1.7×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5K×4&lt;/td&gt;
&lt;td&gt;31.0&lt;/td&gt;
&lt;td&gt;29.9&lt;/td&gt;
&lt;td&gt;tie&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32K×4&lt;/td&gt;
&lt;td&gt;5.6&lt;/td&gt;
&lt;td&gt;1.9&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2.9×&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;52K×4&lt;/td&gt;
&lt;td&gt;3.4&lt;/td&gt;
&lt;td&gt;0.7&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.9×&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;llama.cpp doesn't respond well to added concurrency: 32K single-stream is 5.4 and drops to 1.9 at N=4. The overlap at 100K×4 is −0.2, worse than running the requests one after another. Above 32K, 4-way crashes outright and the most you can run is 2.&lt;/p&gt;

&lt;p&gt;The one thing llama.cpp wins is capacity: n_ctx 262144 vs 102400. With TP=4 the KV block count takes the minimum across all cards, so the 16G card sets the ceiling and roughly 15G of the 5090's 32G goes unused.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. The actual test matrix
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;4 models × 3 links × 5 context sizes × concurrency pushed until the KV pool is blown&lt;/strong&gt;, over 200 cells in total.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;key&lt;/th&gt;
&lt;th&gt;Weights&lt;/th&gt;
&lt;th&gt;Active&lt;/th&gt;
&lt;th&gt;KV pool (tokens)&lt;/th&gt;
&lt;th&gt;ctx ceiling&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;qwen36fp8&lt;/td&gt;
&lt;td&gt;35B-A3B MoE / FP8&lt;/td&gt;
&lt;td&gt;~3B&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;635,712&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;≥100K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3next&lt;/td&gt;
&lt;td&gt;80B-A3B MoE / AWQ&lt;/td&gt;
&lt;td&gt;~3B&lt;/td&gt;
&lt;td&gt;230,656&lt;/td&gt;
&lt;td&gt;≥100K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama33awq&lt;/td&gt;
&lt;td&gt;70B dense / AWQ&lt;/td&gt;
&lt;td&gt;70B&lt;/td&gt;
&lt;td&gt;92,416&lt;/td&gt;
&lt;td&gt;100K rejected&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen25awq&lt;/td&gt;
&lt;td&gt;72B dense / AWQ&lt;/td&gt;
&lt;td&gt;72B&lt;/td&gt;
&lt;td&gt;78,912&lt;/td&gt;
&lt;td&gt;rejected from 52K up&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Link&lt;/th&gt;
&lt;th&gt;Hardware&lt;/th&gt;
&lt;th&gt;Link-layer measurement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;5g&lt;/td&gt;
&lt;td&gt;RTL8126A / r8169&lt;/td&gt;
&lt;td&gt;4.71 Gb/s one-way, NCCL 16KB median 0.146 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;usb10g&lt;/td&gt;
&lt;td&gt;RTL8159 / r8152&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;7.28 Gb/s&lt;/strong&gt; one-way, NCCL 16KB median 0.231 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;bond&lt;/td&gt;
&lt;td&gt;the two above, balance-xor + layer3+4&lt;/td&gt;
&lt;td&gt;iperf3 multi-stream &lt;strong&gt;11.9 Gb/s&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Parameters: GEN=128, seed=1234, temp=0, 2 rounds per cell. Concurrency is derived adaptively from &lt;code&gt;concurrency × (context + GEN) ≤ KV pool&lt;/code&gt;, and once the pool is saturated I deliberately push 1.5× beyond it (marked 超KV in the tables). The goal is to find OOM, not to pick a production configuration.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Results
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3.1 Links: more bandwidth changes nothing, more links helps a lot
&lt;/h3&gt;

&lt;p&gt;At the link layer USB 10GbE beats 5GbE by 55% in bandwidth, 29% in large-message NCCL bandwidth, and has lower raw RTT. End to end, the difference is zero:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Observation&lt;/th&gt;
&lt;th&gt;5g&lt;/th&gt;
&lt;th&gt;usb10g&lt;/th&gt;
&lt;th&gt;Delta&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llama33awq / 5K / prefill&lt;/td&gt;
&lt;td&gt;148.5&lt;/td&gt;
&lt;td&gt;148.6&lt;/td&gt;
&lt;td&gt;+0.07%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3next / 5K / prefill&lt;/td&gt;
&lt;td&gt;1185.3&lt;/td&gt;
&lt;td&gt;1184.2&lt;/td&gt;
&lt;td&gt;−0.09%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen36fp8 / 100K / prefill&lt;/td&gt;
&lt;td&gt;1392.8&lt;/td&gt;
&lt;td&gt;1390.2&lt;/td&gt;
&lt;td&gt;−0.19%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen36fp8 / 50 / c256 end-to-end&lt;/td&gt;
&lt;td&gt;535.6&lt;/td&gt;
&lt;td&gt;537.0&lt;/td&gt;
&lt;td&gt;+0.26%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen25awq / 50 / c443 end-to-end&lt;/td&gt;
&lt;td&gt;98.7&lt;/td&gt;
&lt;td&gt;98.8&lt;/td&gt;
&lt;td&gt;+0.10%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Across 4 models, 5 context sizes and every concurrency cell: direction random, magnitude ≤1%.&lt;/p&gt;

&lt;p&gt;Bond the two links together and prefill rises across the board:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;5K&lt;/th&gt;
&lt;th&gt;32K&lt;/th&gt;
&lt;th&gt;52K&lt;/th&gt;
&lt;th&gt;100K&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llama33awq&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+80.0%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+73.1%&lt;/td&gt;
&lt;td&gt;+69.1%&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3next&lt;/td&gt;
&lt;td&gt;+64.4%&lt;/td&gt;
&lt;td&gt;+59.7%&lt;/td&gt;
&lt;td&gt;+57.8%&lt;/td&gt;
&lt;td&gt;+54.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen25awq&lt;/td&gt;
&lt;td&gt;+51.4%&lt;/td&gt;
&lt;td&gt;+46.2%&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen36fp8&lt;/td&gt;
&lt;td&gt;+15.5%&lt;/td&gt;
&lt;td&gt;+14.0%&lt;/td&gt;
&lt;td&gt;+12.1%&lt;/td&gt;
&lt;td&gt;+11.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For long contexts, look at total elapsed time:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Observation&lt;/th&gt;
&lt;th&gt;5g&lt;/th&gt;
&lt;th&gt;bond&lt;/th&gt;
&lt;th&gt;Speedup&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llama33awq / 32K / c1&lt;/td&gt;
&lt;td&gt;230.8 s&lt;/td&gt;
&lt;td&gt;136.1 s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.70×&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama33awq / 52K / c1&lt;/td&gt;
&lt;td&gt;375.8 s&lt;/td&gt;
&lt;td&gt;224.9 s&lt;/td&gt;
&lt;td&gt;1.67×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen25awq / 32K / c1&lt;/td&gt;
&lt;td&gt;231.9 s&lt;/td&gt;
&lt;td&gt;161.0 s&lt;/td&gt;
&lt;td&gt;1.44×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3next / 100K / c1&lt;/td&gt;
&lt;td&gt;90.4 s&lt;/td&gt;
&lt;td&gt;60.1 s&lt;/td&gt;
&lt;td&gt;1.50×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen36fp8 / 100K / c1&lt;/td&gt;
&lt;td&gt;73.8 s&lt;/td&gt;
&lt;td&gt;67.2 s&lt;/td&gt;
&lt;td&gt;1.10×&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The concurrency crossover point is low (short context, ctx=50, bond relative to 5g):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;c1&lt;/th&gt;
&lt;th&gt;c4&lt;/th&gt;
&lt;th&gt;c16&lt;/th&gt;
&lt;th&gt;c64&lt;/th&gt;
&lt;th&gt;c256&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llama33awq&lt;/td&gt;
&lt;td&gt;+4.9%&lt;/td&gt;
&lt;td&gt;+22.0%&lt;/td&gt;
&lt;td&gt;+33.5%&lt;/td&gt;
&lt;td&gt;+69.4%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+76.1%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen25awq&lt;/td&gt;
&lt;td&gt;−2.2%&lt;/td&gt;
&lt;td&gt;+19.0%&lt;/td&gt;
&lt;td&gt;+13.9%&lt;/td&gt;
&lt;td&gt;+43.8%&lt;/td&gt;
&lt;td&gt;+48.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3next&lt;/td&gt;
&lt;td&gt;−30.9%&lt;/td&gt;
&lt;td&gt;−17.8%&lt;/td&gt;
&lt;td&gt;+17.3%&lt;/td&gt;
&lt;td&gt;+27.4%&lt;/td&gt;
&lt;td&gt;+27.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen36fp8&lt;/td&gt;
&lt;td&gt;−32.0%&lt;/td&gt;
&lt;td&gt;−15.0%&lt;/td&gt;
&lt;td&gt;+32.5%&lt;/td&gt;
&lt;td&gt;+44.7%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+70.3%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Bond only loses on short prompts at low concurrency with MoE models, and the cost lands on decode:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;decode &lt;a class="mentioned-user" href="https://dev.to/5k"&gt;@5k&lt;/a&gt; (tok/s)&lt;/th&gt;
&lt;th&gt;5g&lt;/th&gt;
&lt;th&gt;usb10g&lt;/th&gt;
&lt;th&gt;bond&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llama33awq&lt;/td&gt;
&lt;td&gt;18.8&lt;/td&gt;
&lt;td&gt;19.1&lt;/td&gt;
&lt;td&gt;19.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen25awq&lt;/td&gt;
&lt;td&gt;18.7&lt;/td&gt;
&lt;td&gt;18.6&lt;/td&gt;
&lt;td&gt;18.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3next&lt;/td&gt;
&lt;td&gt;58.6&lt;/td&gt;
&lt;td&gt;59.0&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;40.4 (−31%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen36fp8&lt;/td&gt;
&lt;td&gt;72.3&lt;/td&gt;
&lt;td&gt;70.6&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;48.9 (−32%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Once the prompt reaches 5K, bond already wins on three of the four models at concurrency 1 (+13.6% to +59%).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;⇒ Widening a single link barely moves the needle; combining two links does.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  3.2 Models: MoE wins across the board
&lt;/h3&gt;

&lt;p&gt;Same vLLM, same link (5g), same parameters — only the model changes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;qwen36fp8 (35B-A3B)&lt;/th&gt;
&lt;th&gt;qwen3next (80B-A3B)&lt;/th&gt;
&lt;th&gt;llama33awq (70B dense)&lt;/th&gt;
&lt;th&gt;qwen25awq (72B dense)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;prefill &lt;a class="mentioned-user" href="https://dev.to/5k"&gt;@5k&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1448.7&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1185.3&lt;/td&gt;
&lt;td&gt;148.5&lt;/td&gt;
&lt;td&gt;148.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;decode &lt;a class="mentioned-user" href="https://dev.to/5k"&gt;@5k&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72.3&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;58.6&lt;/td&gt;
&lt;td&gt;18.8&lt;/td&gt;
&lt;td&gt;18.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TTFT &lt;a class="mentioned-user" href="https://dev.to/5k"&gt;@5k&lt;/a&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.45 s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4.21 s&lt;/td&gt;
&lt;td&gt;33.68 s&lt;/td&gt;
&lt;td&gt;33.79 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;KV pool&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;635,712&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;230,656&lt;/td&gt;
&lt;td&gt;92,416&lt;/td&gt;
&lt;td&gt;78,912&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Short-context throughput peak&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;535.6&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;388.3&lt;/td&gt;
&lt;td&gt;96.0&lt;/td&gt;
&lt;td&gt;99.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Short-context c1→saturation&lt;/td&gt;
&lt;td&gt;7.5×&lt;/td&gt;
&lt;td&gt;6.7×&lt;/td&gt;
&lt;td&gt;5.2×&lt;/td&gt;
&lt;td&gt;5.5×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5K c1→saturation&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+49%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+49%&lt;/td&gt;
&lt;td&gt;+9%&lt;/td&gt;
&lt;td&gt;+9%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;prefill differs by &lt;strong&gt;9.8×&lt;/strong&gt;, TTFT by 9.8×, KV pool by &lt;strong&gt;8×&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;At 5K, dense gains only 9% from added concurrency; MoE gains 49%.&lt;/li&gt;
&lt;li&gt;Context ceiling: both MoE models reach 100K; the dense pair stops at 92160 and 32768.&lt;/li&gt;
&lt;li&gt;prefill is nearly flat against context length (only −3.9% from 5K to 100K), and decode decay is comparable across all four models (−5% to −8%).&lt;/li&gt;
&lt;li&gt;Dense AWQ requires &lt;code&gt;--enforce-eager&lt;/code&gt;; without it the Marlin kernel crashes — and it crashes only after running for over ten minutes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3.3 What vLLM is good at
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Blowing past the KV pool doesn't crash anything; requests simply queue.&lt;/strong&gt; Every cell marked 超KV came back with preemptions = 0, failures = 0, constant throughput, and total elapsed time strictly linear in concurrency.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;qwen36fp8 / 5g / 32K&lt;/th&gt;
&lt;th&gt;KV demand / pool&lt;/th&gt;
&lt;th&gt;tok/s&lt;/th&gt;
&lt;th&gt;Total time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;concurrency 19&lt;/td&gt;
&lt;td&gt;610,432 / 635,712&lt;/td&gt;
&lt;td&gt;5.9&lt;/td&gt;
&lt;td&gt;415.1 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;concurrency 29 (1.5× over)&lt;/td&gt;
&lt;td&gt;931,712 / 635,712&lt;/td&gt;
&lt;td&gt;5.9&lt;/td&gt;
&lt;td&gt;632.2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Elapsed-time ratio 1.523×, concurrency ratio 1.526×. What does crash is the VRAM allocation at startup: util 0.88 + max-num-seqs 256 OOMs immediately; 0.85 + 128 comes up fine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Long-context aggregate throughput = prefill × 128 / (ctx + 128).&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Link&lt;/th&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;Predicted&lt;/th&gt;
&lt;th&gt;Measured peak&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;5g&lt;/td&gt;
&lt;td&gt;5K&lt;/td&gt;
&lt;td&gt;36.3&lt;/td&gt;
&lt;td&gt;36.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5g&lt;/td&gt;
&lt;td&gt;32K&lt;/td&gt;
&lt;td&gt;5.87&lt;/td&gt;
&lt;td&gt;5.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5g&lt;/td&gt;
&lt;td&gt;52K&lt;/td&gt;
&lt;td&gt;3.58&lt;/td&gt;
&lt;td&gt;3.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5g&lt;/td&gt;
&lt;td&gt;100K&lt;/td&gt;
&lt;td&gt;1.78&lt;/td&gt;
&lt;td&gt;1.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;bond&lt;/td&gt;
&lt;td&gt;32K&lt;/td&gt;
&lt;td&gt;6.69&lt;/td&gt;
&lt;td&gt;6.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;bond&lt;/td&gt;
&lt;td&gt;100K&lt;/td&gt;
&lt;td&gt;1.98&lt;/td&gt;
&lt;td&gt;2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In long-context work neither concurrency nor batch size changes the order of magnitude — only prefill matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Conclusions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  4.1 vLLM runs fine at home, and the results aren't bad
&lt;/h3&gt;

&lt;p&gt;Two home machines, four consumer GPUs (5090 + 5080 + 5060Ti ×2), one ordinary LAN cable: all four 70B–80B class models came up, and 100K context works. The claim that cross-machine tensor parallelism "just dies" does not hold.&lt;/p&gt;

&lt;p&gt;Numbers you actually feel day to day (qwen36fp8):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Time to first token, 5K prompt&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.45 s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single-stream output&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72 tok/s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single-stream output at 100K&lt;/td&gt;
&lt;td&gt;66.8 tok/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Short-request throughput peak&lt;/td&gt;
&lt;td&gt;535.6 tok/s (912 on bond)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Against llama.cpp single-stream you lose nothing: tied on short prompts, +6% at 5K, +13% at 32K, +20% at 52K, +32% at 100K. The only thing you give up is context capacity, 102400 vs 262144.&lt;/p&gt;

&lt;p&gt;The links need no extra spend. Bonding two cables you already own makes long contexts 1.1–1.7× faster and high concurrency 70% faster.&lt;/p&gt;

&lt;p&gt;Pick an MoE model. On the same hardware a dense 70B takes 33 seconds to first token on a 5K prompt and gains only 9% from added concurrency.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.2 The value is in parallelism — it suits interleaved multi-tasking
&lt;/h3&gt;

&lt;p&gt;The biggest gap is simply how many concurrent streams you can open at long context. On llama.cpp (5 cards, 96GB, KV q8_0), 4-way crashes above 32K and the most you can run is 2:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;llama.cpp max concurrency&lt;/th&gt;
&lt;th&gt;vLLM max concurrency&lt;/th&gt;
&lt;th&gt;vLLM pushed 1.5× further&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;32K&lt;/td&gt;
&lt;td&gt;2 (4-way crashes)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;19&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;29, zero failures&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;52K&lt;/td&gt;
&lt;td&gt;2 (4-way crashes or overflow error)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;12&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;18, zero failures&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100K&lt;/td&gt;
&lt;td&gt;2 (4-way crashes); dense only 1&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;9, zero failures&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;And llama.cpp's 2-way is fake concurrency: measured overlap is 0.0, meaning the two streams run fully serialized. At 4-way the overlap is −0.2, slower than processing them one at a time.&lt;/p&gt;

&lt;p&gt;Note that vLLM had &lt;strong&gt;less&lt;/strong&gt; VRAM to work with — 4 cards at 64GB versus 5 cards at 96GB.&lt;/p&gt;

&lt;p&gt;Throughput points the same way. At 32K both start from 5.4 tok/s single-stream; at 4-way:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Single&lt;/th&gt;
&lt;th&gt;N=4&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llama.cpp&lt;/td&gt;
&lt;td&gt;5.4&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.9&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;−65%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;vLLM&lt;/td&gt;
&lt;td&gt;5.4&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;5.8&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+7%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On short contexts, going from concurrency 1 to 256 takes qwen36fp8 from 71.0 to 535.6 — &lt;strong&gt;7.5×&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;What this means for actual use at home:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Leave a long-document translation running (32K prompt, several minutes) and ask a quick coding question on the side — neither interferes with the other.&lt;/li&gt;
&lt;li&gt;Dump dozens of batch jobs in without writing your own queue or rate limiter.&lt;/li&gt;
&lt;li&gt;No need to worry about opening too many at once and hitting OOM; blowing past the KV pool at runtime is not a failure mode.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If VRAM at home isn't enough and you need to borrow system memory, go with llama.cpp.&lt;br&gt;
If VRAM is sufficient and the GPUs share an architecture so they can be grouped together, go with vLLM.&lt;/p&gt;

&lt;p&gt;Yeah — for a home setup, this looks pretty good.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>vllm</category>
      <category>gpu</category>
    </item>
    <item>
      <title>I bought Thunderbolt 5 and got USB4 40G: re-benchmarking a two-machine llama.cpp RPC setup across three links and six models</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Fri, 30 Oct 2026 09:10:54 +0000</pubDate>
      <link>https://dev.to/faliye/i-bought-thunderbolt-5-and-got-usb4-40g-re-benchmarking-a-two-machine-llamacpp-rpc-setup-across-29im</link>
      <guid>https://dev.to/faliye/i-bought-thunderbolt-5-and-got-usb4-40g-re-benchmarking-a-two-machine-llamacpp-rpc-setup-across-29im</guid>
      <description>&lt;p&gt;This post is a follow-up to my previous benchmark. It tests six models across three links — 5GbE, "fake TB5," and USB 10GbE — comparing how link speed affects performance.&lt;/p&gt;

&lt;p&gt;I'd originally planned to test only TB5, then move on to other things. Buy a cable, plug it in, measure, done — a few hours, tops.&lt;/p&gt;

&lt;p&gt;But TB5 failed to establish end-to-end connectivity on the ASMedia4242 chip. I wasn't convinced at first, but eventually gave up trying. It looks like ASM870/870E can't do DMA-based end-to-end transfer unless you swap out the ASMedia4242 module itself.&lt;/p&gt;

&lt;p&gt;So I bought USB 10GBps afterward — except TB5 still doesn't support USB10GBps, so it only works over USB-C 10 (roughly 7.2Gbps measured). Manually installing Realtek and tuning the protocol turned out to be the easier path in comparison. I might write up the TB5/USB10gbps debugging process separately at some point…&lt;/p&gt;

&lt;p&gt;Given all this mess, "deploy" felt like too heavy a word for a home setup, but I used it anyway.&lt;/p&gt;

&lt;p&gt;Enough preamble — on to the actual results.&lt;/p&gt;

&lt;p&gt;This round covers speed and concurrency tests across six models. Still no optimization work — that's for later. The whole thing took a week of tuning and 72 hours of actual test runtime, with ambient temperature around 35°C.&lt;/p&gt;

&lt;p&gt;Each (model × link) combination only got a single launch. I believe temperature drift is the main confound here — the day/night temperature swing was noticeable. Precision notes are at the end of the post.&lt;/p&gt;

&lt;p&gt;Still no additional optimization this round, but I fixed two bugs from last time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fixed a script bug from the previous version where input length would drift depending on the model&lt;/li&gt;
&lt;li&gt;Dynamically allocate ctx, aiming for a larger KV budget&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;em&gt;Link&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;Actual protocol&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;A→B&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;B→A&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;Bidirectional total&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;Idle ping&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fake TB5&lt;/td&gt;
&lt;td&gt;TB5 cable, but negotiation falls back to USB4 40Gbps. The motherboard's integrated ASMedia ASM4242 only reaches USB4 40G, not true TB5 80G; link-layer negotiates 40Gb/s = 20Gb/s × 2 lanes&lt;/td&gt;
&lt;td&gt;4.38&lt;/td&gt;
&lt;td&gt;4.76&lt;/td&gt;
&lt;td&gt;6.64&lt;/td&gt;
&lt;td&gt;0.224 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5GbE&lt;/td&gt;
&lt;td&gt;Direct CAT6A connection&lt;/td&gt;
&lt;td&gt;4.71&lt;/td&gt;
&lt;td&gt;4.71&lt;/td&gt;
&lt;td&gt;9.38&lt;/td&gt;
&lt;td&gt;0.084 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;USB 10GbE&lt;/td&gt;
&lt;td&gt;TB5 doesn't support it, so USB-C 10G only. Realtek RTL8159 10GbE, direct Cat6a&lt;/td&gt;
&lt;td&gt;7.28&lt;/td&gt;
&lt;td&gt;7.28&lt;/td&gt;
&lt;td&gt;11.93&lt;/td&gt;
&lt;td&gt;0.136 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;(bandwidth in Gbps)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Fake TB5's unidirectional bandwidth is basically on par with 5GbE, but the bidirectional total is 29% lower — &lt;code&gt;thunderbolt-net&lt;/code&gt; 's single-ring DMA engine caps full-duplex throughput.&lt;/p&gt;

&lt;p&gt;I separately measured a 16KB request-response round trip on it: &lt;em&gt;p50 856µs, of which 326µs is fixed overhead&lt;/em&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Link Comparison Summary
&lt;/h2&gt;

&lt;p&gt;prefill shown as relative improvement, decode as per-token latency delta. Baseline = 5GbE.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;em&gt;Model&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;prefill (short / 5K / 32K / 52K / 100K)&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;decode Δms/token&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.6-35B-A3B Q8_0&lt;/td&gt;
&lt;td&gt;Fake TB5 −2% / −8% / −5% / −3% / −2%&lt;br&gt;USB10G +28% / +8% / +13% / +15% / +17%&lt;/td&gt;
&lt;td&gt;Fake TB5 +1.59 / +1.71 / +1.63 / +1.77 / +2.14&lt;br&gt;USB10G −0.86 / −0.82 / −0.97 / −1.01 / −1.06&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b MXFP4&lt;/td&gt;
&lt;td&gt;Fake TB5 −30% / −8% / −3% / −3% / −0%&lt;br&gt;USB10G −9% / +8% / +13% / +15% / +16%&lt;/td&gt;
&lt;td&gt;Fake TB5 +1.46 / +1.71 / +1.63 / +1.65 / +2.13&lt;br&gt;USB10G −0.48 / −0.33 / −0.17 / −0.51 / −0.66&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.5-Air Q4_K_M&lt;/td&gt;
&lt;td&gt;Fake TB5 −18% / −4% / −2% / −1% / −0%&lt;br&gt;USB10G −3% / +5% / +6% / +5% / +5%&lt;/td&gt;
&lt;td&gt;Fake TB5 +1.77 / +1.78 / +1.71 / +1.83 / —&lt;br&gt;USB10G −0.27 / −0.18 / — / — / —&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B-A22B Q2_K_XL&lt;/td&gt;
&lt;td&gt;Fake TB5 −12% / −6% / −3% / — / —&lt;br&gt;USB10G −5% / +2% / +4% / — / —&lt;/td&gt;
&lt;td&gt;Fake TB5 +2.23 / +2.93 / +2.56 / — / —&lt;br&gt;USB10G +0.21 / +0.44 / — / — / —&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B Q6_K&lt;/td&gt;
&lt;td&gt;Fake TB5 −5% / −1% / −1% / −0% / +0%&lt;br&gt;USB10G +1% / +4% / +5% / +5% / +5%&lt;/td&gt;
&lt;td&gt;Below measurement resolution&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-72B Q6_K&lt;/td&gt;
&lt;td&gt;Fake TB5 −6% / −1% / −1% / — / —&lt;br&gt;USB10G +0% / +4% / +5% / — / —&lt;/td&gt;
&lt;td&gt;Below measurement resolution&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The "—" cells in the decode column are where the difference falls within a single quantization step and can't be read out (the script only logs one decimal place; at 9 tok/s, one step already equals 1.2ms). Both dense models show this across the entire row — but adding the 1.7ms back and rounding to one decimal, 5 of 8 cells land exactly on the measured value, versus only 1 under the "no effect" assumption. The same overhead, spread across 97–204ms, works out to only 1–2% — right at the floor of the recording precision.&lt;/p&gt;

&lt;p&gt;All single-stream runs completed cleanly with no errors, and the decay curve is smooth (Llama-3.3, short→100K: 100%→94%→74%→63%→48%), tracking KV growth. OOMs only showed up in the dense models' 52K×4 and 100K×2 concurrency cells.&lt;/p&gt;

&lt;p&gt;What's readable: fake TB5 trails across the entire table, and the gap is independent of model speed or context length, holding steady at 1.5–2.9ms/token. USB10G is mostly slightly faster, except Qwen3-235B, which goes the other way.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Single Stream
&lt;/h2&gt;

&lt;p&gt;USB10G only shown here; see the table above for the relative relationship across all three links.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;decode (tok/s)&lt;/em&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;em&gt;Model&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;Short&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;5K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;32K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;52K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;100K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;100K/Short&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b MXFP4&lt;/td&gt;
&lt;td&gt;85.5&lt;/td&gt;
&lt;td&gt;78.3&lt;/td&gt;
&lt;td&gt;60.3&lt;/td&gt;
&lt;td&gt;53.3&lt;/td&gt;
&lt;td&gt;39.4&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.6-35B-A3B Q8_0&lt;/td&gt;
&lt;td&gt;78.0&lt;/td&gt;
&lt;td&gt;74.8&lt;/td&gt;
&lt;td&gt;67.2&lt;/td&gt;
&lt;td&gt;61.6&lt;/td&gt;
&lt;td&gt;50.8&lt;/td&gt;
&lt;td&gt;65%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.5-Air Q4_K_M&lt;/td&gt;
&lt;td&gt;51.5&lt;/td&gt;
&lt;td&gt;41.4&lt;/td&gt;
&lt;td&gt;20.7&lt;/td&gt;
&lt;td&gt;15.1&lt;/td&gt;
&lt;td&gt;9.1&lt;/td&gt;
&lt;td&gt;18%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B-A22B Q2_K_XL&lt;/td&gt;
&lt;td&gt;37.9&lt;/td&gt;
&lt;td&gt;30.1&lt;/td&gt;
&lt;td&gt;15.5&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B Q6_K&lt;/td&gt;
&lt;td&gt;10.3&lt;/td&gt;
&lt;td&gt;9.7&lt;/td&gt;
&lt;td&gt;7.6&lt;/td&gt;
&lt;td&gt;6.6&lt;/td&gt;
&lt;td&gt;4.9&lt;/td&gt;
&lt;td&gt;48%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-72B Q6_K&lt;/td&gt;
&lt;td&gt;9.3&lt;/td&gt;
&lt;td&gt;8.9&lt;/td&gt;
&lt;td&gt;7.1&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;prefill (tok/s)&lt;/em&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;em&gt;Model&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;Short&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;5K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;32K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;52K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;100K&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.6-35B-A3B Q8_0&lt;/td&gt;
&lt;td&gt;300&lt;/td&gt;
&lt;td&gt;2528&lt;/td&gt;
&lt;td&gt;1756&lt;/td&gt;
&lt;td&gt;1332&lt;/td&gt;
&lt;td&gt;844&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b MXFP4&lt;/td&gt;
&lt;td&gt;400&lt;/td&gt;
&lt;td&gt;2221&lt;/td&gt;
&lt;td&gt;1536&lt;/td&gt;
&lt;td&gt;1134&lt;/td&gt;
&lt;td&gt;693&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.5-Air Q4_K_M&lt;/td&gt;
&lt;td&gt;295&lt;/td&gt;
&lt;td&gt;1016&lt;/td&gt;
&lt;td&gt;586&lt;/td&gt;
&lt;td&gt;394&lt;/td&gt;
&lt;td&gt;214&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B-A22B Q2_K_XL&lt;/td&gt;
&lt;td&gt;146&lt;/td&gt;
&lt;td&gt;676&lt;/td&gt;
&lt;td&gt;494&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B Q6_K&lt;/td&gt;
&lt;td&gt;293&lt;/td&gt;
&lt;td&gt;419&lt;/td&gt;
&lt;td&gt;301&lt;/td&gt;
&lt;td&gt;243&lt;/td&gt;
&lt;td&gt;165&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-72B Q6_K&lt;/td&gt;
&lt;td&gt;248&lt;/td&gt;
&lt;td&gt;425&lt;/td&gt;
&lt;td&gt;305&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;TTFT isn't listed as its own column, since TTFT ≈ prompt_tok ÷ prefill tok/s. For Qwen3.6 at 100K, for example, 100000 ÷ 844 ≈ 118s — measured value was 120s.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Concurrency
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;Usable boundary&lt;/em&gt; (all three links gave the same result, so shown once)&lt;/p&gt;

&lt;p&gt;Single-stream decode below 1 tok/s is counted as a crash — this is not gradual degradation, it's the catastrophic collapse that follows VRAM overflow. 400 = context wall; 500 = overflow error.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;em&gt;Model&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;Short&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;5K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;32K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;52K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;100K&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.6-35B-A3B Q8_0&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b MXFP4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.5-Air Q4_K_M&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;td&gt;2 (500 at 4)&lt;/td&gt;
&lt;td&gt;1 (fails at 2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B-A22B Q2_K_XL&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;1 (fails at 2)&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B Q6_K&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;td&gt;2 (500 at 4)&lt;/td&gt;
&lt;td&gt;1 (fails at 2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-72B Q6_K&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2 (crashes at 4)&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;td&gt;❌ 400&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;True concurrency ratio (overlap)&lt;/em&gt; (5GbE, 2-way / 4-way)&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;em&gt;Model&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;Short&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;5K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;32K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;52K&lt;/em&gt;&lt;/th&gt;
&lt;th&gt;&lt;em&gt;100K&lt;/em&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b&lt;/td&gt;
&lt;td&gt;1.0 / 1.0&lt;/td&gt;
&lt;td&gt;1.0 / 0.5&lt;/td&gt;
&lt;td&gt;0.1 / 0.0&lt;/td&gt;
&lt;td&gt;0.0 / 0.0&lt;/td&gt;
&lt;td&gt;0.0 / −0.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.6-35B-A3B&lt;/td&gt;
&lt;td&gt;1.0 / 1.0&lt;/td&gt;
&lt;td&gt;0.8 / 0.3&lt;/td&gt;
&lt;td&gt;0.1 / 0.0&lt;/td&gt;
&lt;td&gt;0.0 / 0.0&lt;/td&gt;
&lt;td&gt;0.0 / −0.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B&lt;/td&gt;
&lt;td&gt;1.0 / 1.0&lt;/td&gt;
&lt;td&gt;0.6 / 0.3&lt;/td&gt;
&lt;td&gt;0.1 / 0.0&lt;/td&gt;
&lt;td&gt;0.1 / —&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-72B&lt;/td&gt;
&lt;td&gt;1.0 / 1.0&lt;/td&gt;
&lt;td&gt;0.6 / 0.3&lt;/td&gt;
&lt;td&gt;0.1 / 0.0&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.5-Air&lt;/td&gt;
&lt;td&gt;1.0 / 1.0&lt;/td&gt;
&lt;td&gt;0.5 / 0.3&lt;/td&gt;
&lt;td&gt;0.1 / 0.0&lt;/td&gt;
&lt;td&gt;0.0 / —&lt;/td&gt;
&lt;td&gt;0.0 / —&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B&lt;/td&gt;
&lt;td&gt;1.0 / 1.0&lt;/td&gt;
&lt;td&gt;0.5 / 0.2&lt;/td&gt;
&lt;td&gt;— (fails at 2-way)&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The client side is &lt;em&gt;genuinely concurrent&lt;/em&gt; : &lt;code&gt;concurrency_once&lt;/code&gt; fires N requests &lt;em&gt;simultaneously&lt;/em&gt; from N threads, placing every token on a single absolute clock to compute overlap (the fraction of time N streams are decoding &lt;em&gt;at the same time&lt;/em&gt; ). Whether the server can actually sustain concurrent processing is answered quantitatively by this table.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;overlap = 1.0&lt;/em&gt; means fully genuine concurrency; &lt;em&gt;= 0&lt;/em&gt; means fully serial (fake concurrency); &lt;em&gt;&amp;lt; 0&lt;/em&gt; means worse than serial (repeated KV eviction and recomputation due to insufficient KV space).&lt;/p&gt;

&lt;p&gt;Mechanism: llama.cpp's prefill (prompt processing) handles only one slot at a time (serial), while decode is batched and processed in parallel. For short prompts, prefill is negligible, so all four streams quickly enter batch-decode — genuine concurrency. For long prompts, serial prefill eats up most of the time budget, so by the time parallel decode gets its turn there's barely any window left — fake concurrency.&lt;/p&gt;

&lt;p&gt;The 5K column quantifies this mechanism nicely: gpt-oss, the fastest at prefill, hits 1.0/0.5, while Qwen3-235B, the slowest, only manages 0.5/0.2 — the ranking matches the prefill leaderboard exactly. Past 32K, everything collapses to 0.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. prefill scales with unidirectional bandwidth, but returns diminish fast past 5GbE
&lt;/h3&gt;

&lt;p&gt;Raising unidirectional bandwidth from 5GbE's 4.71 Gbps to USB 10GbE's 7.28 Gbps (+55%) only buys prefill gains of +5% (GLM / Llama-3.3 / Qwen2.5 / Qwen3-235B) to +17% (Qwen3.6 and gpt-oss at long context).&lt;/p&gt;

&lt;p&gt;Bandwidth sensitivity rises with two things: &lt;em&gt;the faster a model's prefill is&lt;/em&gt; , the more data crosses the link boundary per unit time — the two MoEs running 2000+ tok/s see +13–17%, while the four models in the 300–500 tok/s range all sit at a flat +4–6% regardless of context length. &lt;em&gt;The longer the context&lt;/em&gt; , the larger the hidden state passed at each layer boundary — Qwen3.6 climbs monotonically from +8% at 5K to +17% at 100K.&lt;/p&gt;

&lt;p&gt;The takeaway: prefill is primarily compute-bound, and link speed is a secondary factor. At 5GbE, this setup's prefill is no longer cable-limited.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. decode pays for per-step transaction overhead, not bandwidth
&lt;/h3&gt;

&lt;p&gt;This is the most substantive finding of this round, and it's completely invisible if you look at tok/s.&lt;/p&gt;

&lt;p&gt;Looked at in tok/s, fake TB5 trails by roughly 10% on fast models and barely shows on slow ones, which makes it look like faster models are more network-sensitive. Converted to per-token time, the pattern flips: fake TB5 trails 5GbE by a &lt;em&gt;constant&lt;/em&gt; 1.5–2.9ms/token, independent of model speed and independent of context length.&lt;/p&gt;

&lt;p&gt;The cleanest evidence comes from GLM-4.5-Air: within a single launch, per-token time rises from 19.7ms (short) to 66.7ms (52K) — a 3.4× increase — while the link-attributable delta stays put at 1.71–1.83ms throughout. Per-token time more than triples; the delta doesn't move at all. That's the signature of a fixed cost, not a proportional one.&lt;/p&gt;

&lt;p&gt;The 16KB request-response measurement answers what that fixed cost is: fake TB5's p50 is 856µs, of which 326µs is fixed overhead. At 4.38 Gbps, the wire time for a 16KB round trip is only 59.9µs — just 7% of the total round trip. The remaining 93% is spent in the protocol stack, DMA scheduling, and context switches. The payload that decode moves across the boundary at each step is a single token's activation — a few KB — landing squarely in that same range.&lt;/p&gt;

&lt;p&gt;Working backward from this, each decode step involves &lt;em&gt;at least 3&lt;/em&gt; cross-machine transactions (N=2 would require 5GbE's 16KB RR to be just 6µs, below the physical floor of 84µs ping + 56µs wire time). Under the tensor-split topology, one token should theoretically need only one round trip — 2 transactions. The extra transactions represent real optimization headroom, and it's worth more than swapping cables.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Where fake TB5 loses
&lt;/h3&gt;

&lt;p&gt;Its unidirectional bandwidth is nearly identical to 5GbE's (4.38–4.76 vs 4.71), but bidirectional total is 29% lower (6.64 vs 9.38), and the 16KB round-trip fixed overhead is 326µs. These same three numbers show up very differently depending on the workload:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;prefill&lt;/em&gt; (large one-way transfers): worst at short context, gradually catching up to 5GbE as context grows — once transfer becomes bandwidth-bound, the two links' unidirectional bandwidth is essentially the same to begin with.&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;decode&lt;/em&gt; (a few KB, request-response): a constant ~1.7ms/token deficit from start to finish, never converging — this payload size sits in the transaction-overhead-bound regime, where fake TB5 stays at a permanent disadvantage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The root cause is probably the same one: &lt;code&gt;thunderbolt-net&lt;/code&gt; 's single-ring DMA serializes send and receive on one engine, and "lower bidirectional throughput" and "higher transaction overhead" are likely two faces of the same bottleneck.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. The collapse at very long context is a VRAM problem, not a link problem
&lt;/h3&gt;

&lt;p&gt;Comparing 100K to 32K, prefill drops sharply (Qwen3.6 goes 1756 → 844 tok/s, gpt-oss 1536 → 693 tok/s), but &lt;em&gt;all three links show the same collapse&lt;/em&gt; , with matching magnitude.&lt;/p&gt;

&lt;p&gt;So this isn't a transfer problem — under the pressure of massive computation, KV read/write is hitting a VRAM bandwidth ceiling. The concurrency data corroborates this: past 32K, overlap collapses to 0 across the board, and 4-way concurrency either crashes outright or dies with an error — both symptoms of VRAM pressure, unrelated to the link.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Differences between models
&lt;/h3&gt;

&lt;p&gt;&lt;em&gt;Single-stream decode ranking&lt;/em&gt; : gpt-oss starts fastest (short 85.5 → 100K 39.4), Qwen3.6 next (78.0 → 50.8), Qwen3-235B mid-pack (37.9 → 15.5 &lt;a class="mentioned-user" href="https://dev.to/32k"&gt;@32k&lt;/a&gt;), GLM decays steepest (51.5 → 9.1), and the two dense models trail (Llama-3.3 10.3 → 4.9, Qwen2.5 9.3 → 7.1).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;But gpt-oss retains only 46% of its starting throughput, versus 65% for Qwen3.6 — the two cross over around 32K.&lt;/em&gt; If your workload leans toward long context, the ranking flips.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Robustness under high concurrency&lt;/em&gt; : only gpt-oss and Qwen3.6 survive 52K×4 / 100K×4 — degraded, but not crashed. GLM and Llama-3.3 hit HTTP 500 the moment they overflow. Qwen2.5 and Qwen3-235B slam straight into the context wall (400 rejection). None of this depends on the link.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Implications for configuration decisions
&lt;/h3&gt;

&lt;p&gt;"600MB/s of layer splitting is enough" — this judgment holds, but for a different reason than originally assumed. I'd assumed it was because "bandwidth is sufficient." It's actually because &lt;em&gt;prefill has already escaped bandwidth-bound territory at 5GbE, and decode was never bandwidth-bound to begin with.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;So:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;Adding VRAM and switching to a faster MoE model beats spending on network speed.&lt;/em&gt; Every crash, context wall, and concurrency collapse happened on the VRAM side.&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Adding bandwidth doesn't help decode at all — what needs cutting is the number of cross-machine transactions per step.&lt;/em&gt; N≥3 against a topological floor of 2 means there's real headroom here; swapping cables only optimizes that 7% wire time.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;The ceiling on cross-machine RPC isn't cable bandwidth.&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Final thoughts
&lt;/h2&gt;

&lt;p&gt;There's a big gap between a model "running" and a model being "usable." A lot of home-lab/personal-scale models underperform, and it's not just VRAM — engineering matters just as much. For example, an unadapted continue can't recognize skills, but copilot running under ollama can. The role of context and context caching, the computation and configuration of KV — these are all genuinely interesting, high-value directions to keep digging into.&lt;/p&gt;

&lt;p&gt;Looking back at the conclusions... did I just waste money on USB10G? Maybe I should try vLLM tensor parallelism next.&lt;/p&gt;

&lt;p&gt;Hoping the bubble problem and juggling 5 GPUs won't be too painful to sort out. Getting sucked into parameter tuning and forgetting the day job...&lt;/p&gt;

&lt;p&gt;Oh, and in the first few days I also bought a 3-meter "thunder bot5" cable (yes, that's literally what their marketing called it).&lt;/p&gt;

&lt;p&gt;If you've ever bricked your ASMedia4242, I do have a UEFI shell guide for recovery. Can't share the resource itself due to copyright, but I learned a lot going through it. I also submitted a patch to the Linux kernel folks regarding the ASMedia4242/TB5 handshake — might not get merged. But hello world, anyway.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>thunderbolt</category>
      <category>llama</category>
    </item>
    <item>
      <title>How Do You Prove an AI-Assisted Filesystem Is Reliable?---Writing a COW Filesystem from Scratch，Part 8</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Mon, 26 Oct 2026 19:02:47 +0000</pubDate>
      <link>https://dev.to/faliye/how-do-you-prove-an-ai-assisted-filesystem-is-reliable-writing-a-cow-filesystem-from-2a9l</link>
      <guid>https://dev.to/faliye/how-do-you-prove-an-ai-assisted-filesystem-is-reliable-writing-a-cow-filesystem-from-2a9l</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;"Trust, but verify." —— Reagan, to Gorbachev.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;From the first day of writing singlefs, a pile of problems sat in front of me. For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;singlefs's on-disk format is nothing like any other filesystem's, so existing verification standards can't simply be borrowed.&lt;/li&gt;
&lt;li&gt;singlefs's code gushes out amid the roar of AI, and human eyes reviewing it can't keep up with the pace.&lt;/li&gt;
&lt;li&gt;singlefs is written starting from the lowest layer, the transaction, and you can't tell whether the data in memory is right by telepathy.&lt;/li&gt;
&lt;li&gt;……&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So how do you prove the system isn't wrong?&lt;/p&gt;

&lt;p&gt;Worse, in a filesystem it is easy to verify that something "was written", and hard to verify that something "was written wrong". The bytes you wrote read back just fine. Much later you may suddenly discover that the data was never actually stored — so what &lt;em&gt;was&lt;/em&gt; it that got stored back then? Or every write goes through normally, the tests are all green, and one day after a power loss the disk never mounts again.&lt;/p&gt;

&lt;p&gt;How do we prove we're right?&lt;/p&gt;

&lt;p&gt;Luckily, singlefs has self-describing headers and a reverse index. Any block on disk can raise both hands and answer "who I am, whom I belong to, which generation I'm from". The data can account for where it went.&lt;/p&gt;

&lt;p&gt;So let's start proving ourselves.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Reconciling against a real device: is what the disk received what the program wrote?
&lt;/h2&gt;

&lt;p&gt;Data can state its own identity, but checking block by block is far too slow, and it can only check the &lt;em&gt;result&lt;/em&gt;, not the &lt;em&gt;process&lt;/em&gt;. So we need a rig that records the stream of data a transaction writes, &lt;strong&gt;has two unrelated parties each record the same process&lt;/strong&gt;, and compares the two item by item. If they match, then at least what the program believes it wrote is what the disk actually received: nothing landed in the wrong place, and nobody quietly changed the order.&lt;/p&gt;

&lt;p&gt;Concretely, we boot a QEMU VM with two virtual disks (virtio) and the write cache set to write back (like real hardware, writes go into the cache first). In front of each disk sits a layer of QEMU's built-in write-logging filter (&lt;code&gt;blklogwrites&lt;/code&gt;). It stands &lt;strong&gt;outside&lt;/strong&gt; the VM and logs every write (with its data) and every FLUSH.&lt;/p&gt;

&lt;p&gt;The two recordings share not a single line of code:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The program's copy&lt;/strong&gt;: the program under test's own recorder, rerun on the host with the same arguments, producing the same bytes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The disk's copy&lt;/strong&gt;: the device log QEMU recorded outside the VM.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The comparison rule lives in &lt;code&gt;crates/singlefs-checker-tier/src/device_log.rs&lt;/code&gt;: the program's copy must be an item-by-item prefix of the disk's copy, and after the prefix only FLUSHes may follow (the system adds one on its own when the VM shuts down).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;compare_allowing_trailing_flushes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;DeviceEvent&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;   &lt;span class="c1"&gt;// what the program thinks it sent&lt;/span&gt;
    &lt;span class="n"&gt;observed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;DeviceEvent&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;   &lt;span class="c1"&gt;// what the disk actually received&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;DeviceLogComparison&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;observed&lt;/span&gt;&lt;span class="nf"&gt;.len&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="nf"&gt;.len&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;observed&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;..&lt;/span&gt;&lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="nf"&gt;.len&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;expected&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;tail&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;observed&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="nf"&gt;.len&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;..&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
        &lt;span class="c1"&gt;// once the prefix matches, any non-FLUSH event in the tail is a divergence&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="nf"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tail&lt;/span&gt;&lt;span class="nf"&gt;.iter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="nf"&gt;.position&lt;/span&gt;&lt;span class="p"&gt;(|&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="nn"&gt;DeviceEvent&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Flush&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="nf"&gt;.len&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;DeviceLogComparison&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;divergence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;observed&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="nf"&gt;.clone&lt;/span&gt;&lt;span class="p"&gt;()))),&lt;/span&gt;
                &lt;span class="n"&gt;trailing_flushes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;};&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;DeviceLogComparison&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;divergence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trailing_flushes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tail&lt;/span&gt;&lt;span class="nf"&gt;.len&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="c1"&gt;// the prefix itself doesn't match: report the first divergence&lt;/span&gt;
    &lt;span class="n"&gt;DeviceLogComparison&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;divergence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;first_divergence&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;observed&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;trailing_flushes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Matching alone isn't enough, because a gate that is always green is no gate at all. So we run six scenarios in total, two of which are deliberately broken control groups:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;th&gt;Expected&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;direct&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Create a new pool and write the first file, bypassing the page cache and writing straight to disk&lt;/td&gt;
&lt;td&gt;Green&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;file-overwrite&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;On top of that, overwrite once more&lt;/td&gt;
&lt;td&gt;Green&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;second-instance&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Then reboot, mount writable, and commit once more&lt;/td&gt;
&lt;td&gt;Green&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;raise-rollback-floor&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Commit once more, and raise the floor on "how far back you can roll back"&lt;/td&gt;
&lt;td&gt;Green&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;skip-new-pool-file-creation-barrier&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Deliberately drop one write barrier&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Must be red&lt;/strong&gt;, and red only on the first disk&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;page-cache&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Deliberately go through the system page cache&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Must be red&lt;/strong&gt; (a 32K write and a 16K write get merged into one 48K write)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There is also a third, independent path: the FLUSH count in the VM's &lt;code&gt;/sys/block&lt;/code&gt; statistics must equal the number of barriers the program issued plus the number of FUA (forced-persistence) writes.&lt;/p&gt;

&lt;p&gt;The point of the two control groups is to prove that when a barrier is dropped or writes are merged, this reconciliation &lt;strong&gt;can tell&lt;/strong&gt;. If it said green even with a barrier missing, then "it matches" would prove nothing.&lt;/p&gt;

&lt;p&gt;A QEMU bug, while we're here: when the number of data segments in a single write reaches 1023 or more, this write-logging filter exceeds the system's limit of 1024 segments per I/O. The host returns an invalid-argument error, the VM gets an I/O error, and formatting (mkfs) hits a wall. We patched QEMU: when over the limit, copy the data into one contiguous buffer first, then log it. The script that builds QEMU checks sha256 at every step.&lt;/p&gt;

&lt;p&gt;This lets us prove: every write, barrier, forced persistence and discard the program issues reaches a real virtual block device intact and in the same order; merged writes and dropped barriers can both be told apart by the red/green controls.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it can't prove&lt;/strong&gt;: this gate &lt;strong&gt;does not simulate power loss&lt;/strong&gt;. And a FLUSH on the virtual disk ultimately lands as a single sync call on the host; the cache of the host's own physical disk is out of reach. So what it verifies is the virtual block device layer, not the persistence semantics of a real disk.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Memory ordering: does the order hold?
&lt;/h2&gt;

&lt;p&gt;Having shown the writes are fine, the next question is: is the order right?&lt;/p&gt;

&lt;p&gt;An atomic commit in a COW filesystem ultimately rests on one ordering: &lt;strong&gt;write the content first, then write the pointer to it&lt;/strong&gt;. If a reader sees the new pointer, it must be able to see the new content.&lt;/p&gt;

&lt;p&gt;CPUs and compilers both reorder reads and writes for performance. Whether this ordering holds under the CPU memory model is left to herd7. herd7 is a memory-model checker, and we use the Linux kernel memory model (LKMM). Tests are written as small programs called litmus tests, kept under &lt;code&gt;litmus/&lt;/code&gt;: 8 files in 4 pairs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Bad outcome that must never occur&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;commit-publish&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The reader sees the new version number but reads the old block content&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;…-journal-implies-units&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The reader sees a journal record but can't read the data blocks the record names&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;…-root-implies-units&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The reader sees the new root pointer but can't read the data blocks under the root&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;publish-returns-after-the-rotation-barrier&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The commit has already returned success, yet the reader still can't see it&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The simplest one (&lt;code&gt;litmus/commit-publish.litmus&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight c"&gt;&lt;code&gt;&lt;span class="n"&gt;P0&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;blk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;gen&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;        &lt;span class="c1"&gt;// writer: commits a transaction&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;WRITE_ONCE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;blk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;      &lt;span class="c1"&gt;// write the block content first&lt;/span&gt;
    &lt;span class="n"&gt;smp_wmb&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;                &lt;span class="c1"&gt;// write barrier&lt;/span&gt;
    &lt;span class="n"&gt;WRITE_ONCE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;gen&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;      &lt;span class="c1"&gt;// then write the version number pointing to it&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;P1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;blk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;gen&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;        &lt;span class="c1"&gt;// reader: mount or concurrent read&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;r0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;r1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;r0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;READ_ONCE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;gen&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;smp_rmb&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;                &lt;span class="c1"&gt;// read barrier&lt;/span&gt;
    &lt;span class="n"&gt;r1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;READ_ONCE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;blk&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;exists&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;r0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;r1&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;     &lt;span class="c1"&gt;// bad outcome: saw the new version number, read the old block&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It declares &lt;code&gt;Never&lt;/code&gt;: herd7 must rule that this bad outcome cannot occur under any reordering.&lt;/p&gt;

&lt;p&gt;This one file alone isn't enough. There are two more rules.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First, every "never occurs" gets a control group.&lt;/strong&gt; &lt;code&gt;commit-publish-nofence.litmus&lt;/code&gt; is the same code with both barriers removed, and it must be ruled "sometimes occurs". A script checks statement by statement that the control group is exactly the original minus its barriers. The reason is written in the control group's file header:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;If this one is also ruled Never, the pattern can't happen in the first place, and commit-publish being ruled Never proves nothing.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This rule was learned the hard way: once, 10 unrelated "never occurs" tests all got through the gate on the strength of a single "sometimes occurs".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Second, bind them to the real code.&lt;/strong&gt; Each litmus file header says which functions it models:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; * singlefs-expect: Never
 * singlefs-models: crates/singlefs-core/src/transaction.rs :: publish_first_file writer (P0)
 * singlefs-models: crates/singlefs-core/src/recovery.rs :: choose_root reader (P1: pick the root first, then read units down the tree)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The named function must be found exactly once in that file. Another test parses the writer's steps out of the litmus file and compares them item by item with the commit order the program actually recorded. Change the code, and this check has to be redone.&lt;/p&gt;

&lt;p&gt;So we've proven that &lt;strong&gt;at the CPU memory-model layer&lt;/strong&gt;, the COW commit order holds: a reader will not see the pointer and then fail to read the content.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it can't prove&lt;/strong&gt;: herd7 deals with reordering between the CPU and memory, &lt;strong&gt;not the order in which things are persisted to disk&lt;/strong&gt;. "The pointer has been persisted but the data hasn't" is the business of FLUSH and forced persistence, which belongs to the real-device reconciliation before and the crash point replay next. Also, the binding to the code only compares the writer's steps; on the reader's side it only checks that the function exists.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Crash point replay: one cut at every moment the power could fail
&lt;/h2&gt;

&lt;p&gt;The order is fine too. So can we recover correctly after a power loss?&lt;/p&gt;

&lt;p&gt;The idea is simple: record every write request to the block device, cut it off at &lt;strong&gt;every&lt;/strong&gt; moment the power could fail, then remount, recover, and check whether the recovered disk is valid and consistent with the history.&lt;/p&gt;

&lt;p&gt;It sounds unremarkable, but it is the most terrifying part of the whole project.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Record&lt;/strong&gt;: run the write path all the way through once, recording every write, barrier and forced persistence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cut&lt;/strong&gt;: each barrier or forced persistence cuts the write stream into segments. The disk at the moment of power loss = all earlier segments fully persisted + each write in the current segment either "landed / didn't land", in any combination. A write is only "protected" by a later barrier on its own disk; as long as one disk hasn't let it through, the segment isn't closed. An in-place overwrite has one more outcome than an ordinary write — half-written, so neither old nor new can be read. That is &lt;strong&gt;tearing&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;How do you count the combinations? The comment in the code says it bluntly (&lt;code&gt;crates/singlefs-checker-tier/src/crash.rs&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="cd"&gt;/// Each write in the segment takes any of its states, in any combination, minus the one where the whole segment persisted:&lt;/span&gt;
&lt;span class="cd"&gt;/// with m of the n writes being in-place overwrites, 3^m · 2^(n−m) − 1 states; when m = 0, any proper subset, 2^n − 1.&lt;/span&gt;
&lt;span class="n"&gt;EveryProperSubset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3^m · 2^(n−m) − 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each in-place overwrite has three outcomes — "didn't land, landed, torn" — and an ordinary write has two. Multiply them, then subtract the one where "everything landed". Exponential explosion.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Judge&lt;/strong&gt;: for every power-loss state, run four things in order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Recovery&lt;/strong&gt;, twice: once reading the journal, once not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Outcome check&lt;/strong&gt;: is the recovered content one of the versions this history allows to appear?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pool-level checker&lt;/strong&gt;: is the on-disk structure valid? (Covered in detail in part seven.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Record checker&lt;/strong&gt;: are the writes of the root pointer and journal records really on disk, and is any data block missing?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The record checker deserves a word of its own. It &lt;strong&gt;does not go through the recovery code and does not parse the trees&lt;/strong&gt;. It takes only four things — the disk before the power loss, the recorded write stream, the disk after the power loss, and which writes had persisted — and compares bytes. This path is a must: when we deliberately removed the barrier in front of the root pointer, the outcome check didn't flag a single state red; only the record checker went red, on exactly 1 state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Scale&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One full run on 2026-09-30: 7 items, two machines, five GPUs, exhaustive:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CRASH_AMPLIFICATION_TOTAL items=7 failed=0 states=2563727926 red=0 seconds=27288
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;2.56 billion power-loss states, 0 red, 7 hours 35 minutes.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Today 47 crash cases are registered, plus 25 scaled-up items.&lt;/li&gt;
&lt;li&gt;The full run before a commit on 2026-10-25: of 38 cases, the first 6 ran to completion, 4.5 billion states in total, 0 violations, of which 3.0 billion directly reused earlier verdicts. It was then stopped, and the remaining 31 never ran.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is why milestone three nearly tore this part down and rewrote it around pruning and reuse: without that, no number of hosts would be enough. To put it fancily: &lt;strong&gt;heterogeneous GPU computing&lt;/strong&gt; and &lt;strong&gt;a reusable verdict library&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Heterogeneous GPU computing&lt;/strong&gt;: split the work between CPU and GPU. The CPU records efficiently; the GPU judges. Measured: fully judging one state on the CPU (writable mount + pool-level checker) takes about 19.5 ms; on the GPU, about 4 µs — a gap of nearly five thousand times. So recovery, the outcome check, the record checker and the pool-level checker's 109 rules were &lt;strong&gt;written all over again, in another language with other algorithms&lt;/strong&gt;: six GPU shaders (WGSL), about 23,000 lines in total. The GPU version must agree with the CPU version state by state and bit by bit, with dedicated cross-check tests. If a table won't fit in VRAM, it reports "can't judge" and the whole run stops; it &lt;strong&gt;never quietly falls back to the CPU&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reusable verdict library&lt;/strong&gt;: each block's verdict is stored by key in a long-lived library. The key layout (&lt;code&gt;crates/singlefs-checker-tier/src/verdict_store.rs&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;block key   (72 bytes) = digest of this segment's writes and the starting disk image (32) ‖ plan digest (32) ‖ block start (8)
verdict key            = block key ‖ judging-method fingerprint (32)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The "judging-method fingerprint" is computed like this: take every function used to judge this block, strip comments, fingerprint each one, add the compile switches, the toolchain version and the versions of the dependencies used, and digest them together. File paths, function names, line numbers and commit hashes never enter the key. So when you change one function, only the verdicts that use it need to be rerun; everything else is reused as is. We used to take the shortcut of using a digest of the whole library as the key; changing one character of the judging method invalidated everything, and we ended up re-judging an extra 3.9 billion states — function-level fingerprints were forced out of us that way.&lt;/p&gt;

&lt;p&gt;On top of that, every run randomly draws a few already-judged entries from the library as &lt;strong&gt;sentinels&lt;/strong&gt;, and forces one more to be re-judged. If a sentinel gets re-judged, or the forced re-judgement disagrees with the library entry, the reuse mechanism itself is broken, and &lt;strong&gt;this gate goes red immediately&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This lets us prove that for the write paths the tests cover, at the model level, a power loss at any moment can be recovered from correctly. But the volume is so large that full runs can only catch up in batches each milestone, or a little at a time when the machines are idle.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it can't prove&lt;/strong&gt;: it cuts the write stream the program recorded itself, assuming that recording is genuine — that part is backed by the first gate, the real-device reconciliation. Whether a real disk's FLUSH actually works is out of range. The fast tier we run day to day saves time by treating data blocks as either "landed whole" or "not landed at all", so it can't see the "data block only half landed" class; that's why commits still require the full run. And as of today, the full run hasn't been completed.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Model differential testing: no reference answer, so build one
&lt;/h2&gt;

&lt;p&gt;Everything above only proves that the program operates according to its own understanding. But is that understanding itself right?&lt;/p&gt;

&lt;p&gt;We wrote an ideal filesystem in memory that &lt;strong&gt;cares only about semantics — not performance, and not crashes&lt;/strong&gt;: &lt;code&gt;crates/singlefs-harness/src/model.rs&lt;/code&gt;, 13,535 lines. It lives in memory, so it runs stably and its answers are unambiguous.&lt;/p&gt;

&lt;p&gt;Then the same random sequence of operations drives both the &lt;strong&gt;ideal model&lt;/strong&gt; and the &lt;strong&gt;real implementation&lt;/strong&gt; at once (the implementation runs on an in-memory test disk). Before calling the implementation at each step, we ask the model: should this step "succeed", "be refused", or is "anything within some range" acceptable? The model's answer looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="cd"&gt;/// The model's answer for one step.&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;ModelAnswer&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;operation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ModelOperationKind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="cd"&gt;/// Reasons it must be refused: if non-empty, it must be refused&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;required_refusals&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;BTreeSet&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ModelRefusalReason&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="cd"&gt;/// Roots that should be written on success, in order&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;expected_roots&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ModelRoot&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="cd"&gt;/// What should be read back after a cold start&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;expected_read_back&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Option&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ModelReadBack&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="cd"&gt;/// When raising the rollback floor, what the ceiling should be&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;rollback_floor_ceiling&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Option&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ModelCheckpointTxg&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="c1"&gt;// ……&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the implementation finishes, its result is translated and compared item by item against the model's answer. Disagreements fall into 28 kinds: should have succeeded but refused, should have refused but succeeded, wrong refusal reason, secretly wrote to disk before refusing, wrong roots written, wrong file content, wrong read-back after a cold start…… Any error the model has never seen is recorded as "unexplained" and not let go.&lt;/p&gt;

&lt;p&gt;There are currently 20 kinds of random operations, with 16 different mixes. The most commonly used mix, when "mounted, with files present", is: overwrite 52, raise the rollback floor 16, roll back 10, mount 6, delete a file 3, crash recovery that discards the newest version 3.&lt;/p&gt;

&lt;p&gt;Two design points worth mentioning:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Generation and execution are separate&lt;/strong&gt;: at generation time each step records only "what to do" and a selector; the concrete arguments are computed from the disk state at the moment of execution. So after a failure, deleting earlier steps leaves the later ones still meaningful, and the case can be shrunk all the way down to a minimal reproduction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Seeds are never redrawn&lt;/strong&gt;: the starting point of the random seeds is hard-coded as the constant &lt;code&gt;7_463_871_032_432_355_113&lt;/code&gt;. To test more, new ranges are appended after it; old seeds stay reproducible forever.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Only when the model and the implementation agree step by step can we say our semantics — that is, the intent of each operation — was faithfully carried out by the implementation.&lt;/p&gt;

&lt;p&gt;There is also a smaller &lt;strong&gt;RefFS&lt;/strong&gt; (reference filesystem, &lt;code&gt;crates/singlefs-reference-filesystem&lt;/code&gt;). It has no external dependencies at all, uses only the Rust standard library, and walks through POSIX operations in memory. It answers just one question: &lt;strong&gt;is the content read back after a power loss within the allowed range?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="cd"&gt;/// States recovery may read back after a crash: {S_j : j = lo, or j is exactly the end of some explicit publish or rollback, lo ≤ j ≤ hi}, in ascending index order.&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;AllowedSet&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;lower_bound&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;StateIndex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;upper_bound&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;StateIndex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;AllowedState&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In plain words: between "the last version known to be persisted" and "the moment of power loss", the version right after each explicit commit may be what recovery produces; nothing else may. The ideal model handles "what it should be without a power loss"; RefFS handles "what may be seen after a power loss". They split the work.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it can't prove&lt;/strong&gt;: we wrote the model, and we wrote the implementation, and both understand the semantics from the same design documents. &lt;strong&gt;If both misunderstand together, both go green together.&lt;/strong&gt; Also, right now we only check that the differential testing "ran", not that every kind of operation was actually "reached" by it; a newly added write path may never have been differentially tested for a single step.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Formal verification: mathematics doesn't lie
&lt;/h2&gt;

&lt;p&gt;But what if one of the most critical judgements in the implementation is wrong precisely on inputs that random operations will never hit?&lt;/p&gt;

&lt;p&gt;Tests can only try finitely many inputs. Formal verification is different: &lt;strong&gt;first write down, in mathematics, the properties this code must satisfy (the spec), then have a prover prove that the code satisfies them for every possible input&lt;/strong&gt;. If it can't be proven, the code isn't accepted.&lt;/p&gt;

&lt;p&gt;We prove Rust code with &lt;a href="https://github.com/verus-lang/verus" rel="noopener noreferrer"&gt;Verus&lt;/a&gt;. Take &lt;code&gt;replay_core&lt;/code&gt; (&lt;code&gt;crates/singlefs-replay-prefix/src/lib.rs&lt;/code&gt;), the core function that, during recovery from a power loss, "takes the replayable stretch out of the journal". Its signature carries preconditions (requires) and conclusions (ensures):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="n"&gt;replay_core&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;...&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;root_instance&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;PrefixKernelInstanceGeneration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;root_checkpoint_txg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;PrefixKernelCheckpointTxg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;chosen_root_source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;RootSource&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;CoreRecord&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;RootSource&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="c1"&gt;// ……&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;replayed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Result&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;CoreOutcome&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;RootSource&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;CoreFailure&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;requires&lt;/span&gt;
        &lt;span class="nf"&gt;records_ascend_by_instance_and_counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;   &lt;span class="c1"&gt;// precondition: journal records are in order&lt;/span&gt;
        &lt;span class="nf"&gt;every_counter_has_a_successor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;ensures&lt;/span&gt;
        &lt;span class="nf"&gt;replay_follows_the_prefix_rules&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;                           &lt;span class="c1"&gt;// which stretch is taken follows the rules&lt;/span&gt;
        &lt;span class="nf"&gt;replaying_an_applied_record_twice_leaves_the_root_as_once&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="c1"&gt;// replaying the same record twice equals once&lt;/span&gt;
        &lt;span class="nf"&gt;replaying_from_the_landed_root_is_a_fixed_point&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;           &lt;span class="c1"&gt;// replaying again from the result changes nothing&lt;/span&gt;
        &lt;span class="c1"&gt;// and one more: the replay result is never older than the starting point&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two shorter ones:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Whether a record may be reclaimed is this one comparison; not one character more or less&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;released_record_can_be_reclaimed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;released&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;u64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;u64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ensures&lt;/span&gt; &lt;span class="n"&gt;answer&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;released&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;generation&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Losing power twice during garbage collection gives the same result as losing it once&lt;/span&gt;
&lt;span class="nf"&gt;run_ring_next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;run_ring_next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;RunRingStep&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Crash&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nn"&gt;RunRingStep&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Crash&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="nf"&gt;run_ring_next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;RunRingStep&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Crash&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;The key point: what is proven is exactly the code that runs&lt;/strong&gt;, not a separately written model. The recovery path calls these functions directly; in a normal build, Verus's spec and proof parts are erased, and the remaining code is compiled into the product as usual. To keep "the version that was proven" and "the version that runs" from quietly drifting apart, the intermediate code produced by the two builds is also compared item by item and required to be identical.&lt;/p&gt;

&lt;p&gt;All the proven code lives in one crate, &lt;code&gt;crates/singlefs-replay-prefix&lt;/code&gt;, which depends only on Verus's standard library and has four files:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Lines&lt;/th&gt;
&lt;th&gt;Proof fns&lt;/th&gt;
&lt;th&gt;What it proves&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;lib.rs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;4342&lt;/td&gt;
&lt;td&gt;25&lt;/td&gt;
&lt;td&gt;The three steps of recovery from power loss: take the replayable stretch from the journal, choose one of several candidate roots, scan the journal ring to build a table&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;compaction_plan_proofs.rs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;1588&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;During space compaction, which stretch to choose and how to move it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gc_ring_interval_proofs.rs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;1762&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;A protocol model of the garbage collector's relocation process&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gc_crash_proofs.rs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;167&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;That comparison before reclaiming&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;113 proof obligations in total, and zero occurrences across the four files of back-door constructs like "assume it holds" or "skip the proof".&lt;/p&gt;

&lt;p&gt;But formal verification isn't almighty. Its most insidious failure isn't "can't prove it" — it's &lt;strong&gt;proving it while proving nothing&lt;/strong&gt;. So four more things are guarded:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One, the vacuity check.&lt;/strong&gt; If a precondition can never hold, anything can be "proven" from it. So for each precondition, an &lt;code&gt;assert(false)&lt;/code&gt; is inserted at the top of the function, one separate copy per location, and the proof must fail. If it passes, the precondition contradicts itself, and that proof goes red.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two, must-red worlds.&lt;/strong&gt; 231 deliberately broken locations are registered. After breaking each one, Verus must report an error, and the error must land on the designated function. For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;break:    negate the conclusion
original: is_last_record == is_last_record_of_its_publish(*record),
mutated:  is_last_record == !is_last_record_of_its_publish(*record),
expect:   7 proof errors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Only a proof that can show it goes red counts as a proof.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Three, a pinned toolchain.&lt;/strong&gt; The Verus version, the versions and content fingerprints of the six packages it depends on, and the Rust compiler version — if any one doesn't match, it turns red immediately.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Four, a pinned table of exhaustive counterexamples.&lt;/strong&gt; We ran a separate experiment that exhaustively enumerates the real journal-replay function within a bounded size: about 49.75 million valid inputs and about 64.51 million adversarial inputs. Every counterexample found is pinned into a table exactly as is; it has 360 rows now. On every commit, the current code replays it row by row.&lt;/p&gt;

&lt;p&gt;I have to admit formal verification is expensive, and to this day it covers only a few of the most central judgements: which stretch to take from the journal, which root to choose, scanning the journal ring, the compaction plan, the reclamation threshold. How on-disk bytes get parsed into these functions' inputs, disk barriers, encoding and decoding, checksums, multiple disks, integer overflow — none of that is in the proof. The README in that code directory says so itself: "this cannot be called a complete crash-recovery proof of the whole filesystem".&lt;/p&gt;

&lt;p&gt;One more thing must be made clear: &lt;strong&gt;Verus proves "the code conforms to the spec"; it cannot prove "the spec itself is right"&lt;/strong&gt;. If the spec is wrong, the proof still goes through. Whether the spec is right depends on the reasoning in the design documents, and on the earlier gates corroborating one another.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Mutation: judging the judges
&lt;/h2&gt;

&lt;p&gt;So, could the verification itself be fake? Tests that don't reach the code, assertions written backwards, sample points that happen to fall where two variants give the same result — in all these cases the tests are still all green, but they judge nothing.&lt;/p&gt;

&lt;p&gt;The idea is simple: &lt;strong&gt;deliberately break the code and see whether verification turns red&lt;/strong&gt;. If it turns red, that check can tell this kind of error apart. If it stays green after you break things, that check is decoration.&lt;/p&gt;

&lt;p&gt;Concretely, there are two things.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One, every bug fixed leaves a landmine behind.&lt;/strong&gt; &lt;code&gt;crates/mutations.tsv&lt;/code&gt; has 5,981 rows in total. Each row says which file to change, which original text to replace with what, which tests to run, and which test must go red. For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="n"&gt;name&lt;/span&gt;       &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;free&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="n"&gt;not&lt;/span&gt; &lt;span class="n"&gt;decremented&lt;/span&gt; &lt;span class="n"&gt;after&lt;/span&gt; &lt;span class="nf"&gt;allocation&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;the&lt;/span&gt; &lt;span class="n"&gt;space&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;accounting&lt;/span&gt; &lt;span class="n"&gt;check&lt;/span&gt; &lt;span class="n"&gt;must&lt;/span&gt; &lt;span class="n"&gt;go&lt;/span&gt; &lt;span class="n"&gt;red&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;file&lt;/span&gt;       &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;crates&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;singlefs&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;core&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;allocator&lt;/span&gt;&lt;span class="py"&gt;.rs&lt;/span&gt;
&lt;span class="n"&gt;original&lt;/span&gt;   &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="py"&gt;.free_slots&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;mutated&lt;/span&gt;    &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="c1"&gt;// mutation: free count not decremented&lt;/span&gt;
&lt;span class="n"&gt;must&lt;/span&gt; &lt;span class="n"&gt;fail&lt;/span&gt;  &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cold_start_reads_the_second_content_and_the_pool_checker_stays_green&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The table header records where it came from: every time a round of adversarial review finds a problem and it gets fixed, a mutation is left behind that "goes red if you revert it"; if the fix is quietly undone, this goes off first. Some day someone (most likely some agent) quietly reverts a fix, and the landmine goes off first. The original text must be found exactly once in the file; if the code changes and the original text can no longer be found, that row is recorded as "anchor lost", which is also red.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two, broad mutation.&lt;/strong&gt; Landmines are planted by people, so there are always places nobody thought of. There's therefore also a tool that generates mutations automatically (&lt;code&gt;scripts/mutation/broad-mutation.py&lt;/code&gt;, using cargo-mutants under the hood): flip a comparison operator, add or subtract one from a constant, delete a statement…… Counted once on 2026-09-30, it could generate 1,765 mutations from the checker and 3,326 from the core code. Every one must reach a verdict, and a single survivor blocks the commit. A few details:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Timing out or hitting the memory cap &lt;strong&gt;does not count as caught&lt;/strong&gt;, unless it's rerun once with a bigger budget, times out both times, and the evidence is complete.&lt;/li&gt;
&lt;li&gt;Verdicts are reused by function fingerprint: if the test that caught a mutation hasn't changed and the function containing it hasn't changed, it isn't rerun.&lt;/li&gt;
&lt;li&gt;Every row in the verdict library carries a signature computed with a key kept outside the repository. What it guards against isn't hackers; it's an agent casually editing one line of a verdict.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The "must-red worlds" from the previous gate follow the same idea: break the proof, and the prover must report an error.&lt;/p&gt;

&lt;p&gt;When a mutation survives, the laziest move is to write "these two variants are equivalent anyway" and walk away. The project's rules forbid that and split "survived" into eight categories. The most commonly abused one is this: &lt;strong&gt;it only happens to give the same result on the few test inputs I picked; change the input and it differs&lt;/strong&gt;. In that case you must add an input; you may not fudge it as equivalent. The criterion is one sentence:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;When a mutation isn't caught, ask once: with a different sample point, is it still the same value?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;strong&gt;What it can't prove&lt;/strong&gt;: mutation proves that "assertions can go red", not that "coverage is complete". Breakages not written in the table will never be tried, and the kinds of mutations that can be generated automatically are limited too.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. The pool-level checker: pick up any disk
&lt;/h2&gt;

&lt;p&gt;Finally: pick up any disk — how do you decide whether it's a valid singlefs disk?&lt;/p&gt;

&lt;p&gt;That's the pool-level checker. One thing first: &lt;strong&gt;it is not something that runs at mount time&lt;/strong&gt;. Mounting goes through the core's own recovery and validation code; the checker is an independent auditor, called from tests, crash point replay and the GPU cross-checks. Mounting and auditing each judge the same thing once, with two separate bodies of code.&lt;/p&gt;

&lt;p&gt;There is only one entry point (&lt;code&gt;crates/singlefs-checker/src/walk.rs&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;check_pool_image&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;dyn&lt;/span&gt; &lt;span class="n"&gt;ImageReader&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;'static&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;InvariantVerdict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The input is bytes only: &lt;code&gt;ImageReader&lt;/code&gt; gives just "how many disks, how big each is, and read some bytes at some position on some disk". It touches no runtime memory state at all. The output is one verdict per rule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="cd"&gt;/// The verdict of one invariant on one image. "Not applicable" must carry a reason; it may never silently count as holding.&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;enum&lt;/span&gt; &lt;span class="n"&gt;InvariantVerdict&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Holds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                        &lt;span class="c1"&gt;// holds&lt;/span&gt;
    &lt;span class="nf"&gt;Violated&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;             &lt;span class="c1"&gt;// violated, saying where&lt;/span&gt;
    &lt;span class="nf"&gt;NotApplicable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;'static&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;  &lt;span class="c1"&gt;// not applicable, reason required&lt;/span&gt;
    &lt;span class="nf"&gt;PartiallyJudged&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;      &lt;span class="c1"&gt;// everything judged is right, but part couldn't be judged: not red, not holding either&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It first reads the system configuration disk by disk, then picks the newest valid root, scans the journal and replays that version, and then walks the disk once in each of two directions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Follow the references&lt;/strong&gt;: starting from the root, walk every tree — the inode tree, the deletion records, the snapshot lineage and the rest — all the way to the leaves.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scan block by block&lt;/strong&gt;: ignoring who references whom, read the self-describing header of every block on disk one by one and rebuild the whole index.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The indexes built from the two directions must be exactly identical. Finally, the space usage found by the scan is reconciled, disk by disk, against the accounts recorded on disk.&lt;/p&gt;

&lt;p&gt;The list of rules the checker judges lives in &lt;code&gt;crates/singlefs-checker/invariants/&lt;/code&gt;: 109 implemented, matching &lt;code&gt;IMPLEMENTED_INVARIANTS: [&amp;amp;str; 109]&lt;/code&gt; in the code one to one. A few of them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The identity recorded in each block's header must match the place that references it.&lt;/strong&gt; This is the opening's "any block, picked up on its own, can answer who I am, whom I belong to, which generation I'm from", turned into code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Besides the newest valid root, there must be at least one older valid root.&lt;/strong&gt; If not, it's red, because that means "the next bad write leaves no way back".&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The space the accounts call "allocated" must equal the space computed by actually following all the references.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per disk, "allocated − pending release" must equal the number of blocks the newest version actually occupies.&lt;/strong&gt; It reads the accounts on disk, not the allocator in memory.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every implemented rule comes with a broken disk that was "deliberately damaged to trigger exactly that rule".&lt;/p&gt;

&lt;p&gt;The pool-level checker may not share a single line of logic with the core code; it shares only one module of format constants. Its &lt;code&gt;Cargo.toml&lt;/code&gt; has exactly one dependency:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="k"&gt;[&lt;/span&gt;&lt;span class="n"&gt;dependencies&lt;/span&gt;&lt;span class="k"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;singlefs-format&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;"../singlefs-format"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That format module isn't even allowed to contain &lt;code&gt;if&lt;/code&gt;, &lt;code&gt;match&lt;/code&gt; or &lt;code&gt;for&lt;/code&gt;; it holds only constants. The design discipline has this line:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What is forbidden is not "traversal". It is two things: "not yet having the answer at the moment you need it", and "the auditor and the audited using the same code". &lt;strong&gt;The runtime and the checker must use different algorithms.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If the two shared a piece of parsing code and that code was wrong, both would be wrong the same way, and the reconciliation would still be all green. So the gate holds the line strictly: apart from the format constants, the two modules' dependencies may not overlap at all; and the functions called by the "follow the references" direction and the "scan block by block" direction may overlap only in format parsing and constants.&lt;/p&gt;

&lt;p&gt;The checker is also a specification: it defines "what this format actually is". The format of all of singlefs is defined by the checker's source code; the documentation and everything else take it as the reference.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it can't prove&lt;/strong&gt;: it only looks at one static disk. Not a single encryption-related rule is implemented yet, and the rule that "forward references and the reverse index agree with each other" is still on the to-do list. The rules say it outright: &lt;strong&gt;"the checker reported nothing" does not mean "the disk is healthy"&lt;/strong&gt;. Errors not written into the list are invisible to it.&lt;/p&gt;




&lt;h2&gt;
  
  
  Finally
&lt;/h2&gt;

&lt;p&gt;Every gate has its own shortcomings, and none of them can guarantee the code is correct on its own. But they back each other up:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Formal verification proves the most central judgements but not the glue code around them → crash point replay and the pool-level checker cover the glue.&lt;/li&gt;
&lt;li&gt;Crash point replay assumes the recorded write stream is genuine → the real-device reconciliation records another copy from outside the door.&lt;/li&gt;
&lt;li&gt;herd7 watches CPU reordering, not the order of persistence to disk → the real-device reconciliation and crash point replay can catch that.&lt;/li&gt;
&lt;li&gt;Drop one barrier and the outcome check can't see it → the record checker can.&lt;/li&gt;
&lt;li&gt;The ideal model and the implementation might misunderstand together → RefFS is a new mirror.&lt;/li&gt;
&lt;li&gt;Any gate could stay green forever → control groups, must-red worlds, vacuity checks and the mutation tables each prove, gate by gate, that they can go red.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The system these gates form is like the Berlin Wall: it's not tall. Want to get over it? Big Brother is watching you.&lt;/p&gt;

&lt;p&gt;How would you prove this wall is solid, Comrade Gorbachev?&lt;/p&gt;

</description>
      <category>claude</category>
      <category>rust</category>
      <category>verus</category>
      <category>singlefs</category>
    </item>
    <item>
      <title>Benchmarking 6 Open LLMs on a 2-Machine RPC Cluster (Consumer GPUs, ~96GB VRAM)</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Thu, 22 Oct 2026 13:27:11 +0000</pubDate>
      <link>https://dev.to/faliye/benchmarking-6-open-llms-on-a-2-machine-rpc-cluster-consumer-gpus-96gb-vram-j2p</link>
      <guid>https://dev.to/faliye/benchmarking-6-open-llms-on-a-2-machine-rpc-cluster-consumer-gpus-96gb-vram-j2p</guid>
      <description>&lt;p&gt;In the endless pursuit of "bigger, bigger, and even bigger," I finally reached this point: two machines directly connected via a local area network, using RPC scheduling to split and load the model layer by layer across both hosts.&lt;/p&gt;

&lt;p&gt;The reason is simple—&lt;strong&gt;I have a limited budget&lt;/strong&gt;. Buying a professional-grade graphics card with 96GB of VRAM would definitely not pass my wife's approval. So for consumer-level GPU users on a budget like me, linking multiple machines and GPUs via RPC is currently the only realistic option.&lt;/p&gt;

&lt;p&gt;The setup itself is straightforward: connect two machines with an Ethernet cable. I used a nominally 5Gbps CAT6A cable, and the actual tested bandwidth is about 4.7Gbps—yielding a throughput of around 500 MB/s. Under this bandwidth, &lt;strong&gt;llama.cpp's RPC chain invocation is currently the only viable choice&lt;/strong&gt; (a Thunderbolt 5 adapter is on the way; once it arrives, it should close some of the performance gap).&lt;/p&gt;

&lt;p&gt;The ambient temperature during testing was 35°C; the CPU reached 76°C, and the RTX 5090 (power-limited to 450W) peaked at 68°C.&lt;/p&gt;

&lt;p&gt;No special optimizations were made at the architecture level—I just got the official RPC server running end-to-end first, leaving optimizations for later.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hardware Setup
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Host&lt;/th&gt;
&lt;th&gt;GPU&lt;/th&gt;
&lt;th&gt;VRAM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Host 1&lt;/td&gt;
&lt;td&gt;RTX 5080&lt;/td&gt;
&lt;td&gt;16GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Host 1&lt;/td&gt;
&lt;td&gt;RTX 5060 Ti ×2&lt;/td&gt;
&lt;td&gt;16GB × 2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Host 2&lt;/td&gt;
&lt;td&gt;RTX 5090&lt;/td&gt;
&lt;td&gt;32GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Host 2&lt;/td&gt;
&lt;td&gt;RTX 5060 Ti&lt;/td&gt;
&lt;td&gt;16GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This totals approximately 96GB of VRAM, with the two hosts directly connected via a CAT6A network cable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfall Record: tensor-split Order
&lt;/h2&gt;

&lt;p&gt;Once the two machines were connected, it was time to pick the models. With a 96GB budget, there isn't actually much room for choice—you have to consider production-level concurrency requirements (according to current market expectations, a 96GB setup is considered a poor investment if it can't support a team of 8-15 people), while also balancing KV context size and model precision.&lt;/p&gt;

&lt;p&gt;When running the first model (Qwen3.6-35B-A3B Q8_0), I noticed the main card (5090) was using less than 7GB of VRAM—a massive waste. At the time, &lt;code&gt;--tensor-split&lt;/code&gt; was set to &lt;code&gt;32,16,16,16,16&lt;/code&gt;, intending to allocate the largest share to the main card. But the actual device order was:&lt;br&gt;
&lt;code&gt;[ local - RPC : 5080 , local - RPC : 5060Ti , local - RPC : 5060Ti , 5090 , 5060Ti ]&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;In other words, &lt;strong&gt;the remote RPC cards were placed at the front, while the local main card was at the end&lt;/strong&gt;—the exact opposite of what I expected. This is why the VRAM allocation didn't land on the 5090 as intended.&lt;/p&gt;

&lt;h2&gt;
  
  
  Candidate Models
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Quantization&lt;/th&gt;
&lt;th&gt;Type / Active Params&lt;/th&gt;
&lt;th&gt;Layers&lt;/th&gt;
&lt;th&gt;Model Size&lt;/th&gt;
&lt;th&gt;Thinking Mode&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;qwen36 · Qwen3.6-35B-A3B&lt;/td&gt;
&lt;td&gt;Q8_0&lt;/td&gt;
&lt;td&gt;MoE, 3B active&lt;/td&gt;
&lt;td&gt;Split into 5 layers&lt;/td&gt;
&lt;td&gt;~42GB&lt;/td&gt;
&lt;td&gt;Disabled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama33 · Llama-3.3-70B&lt;/td&gt;
&lt;td&gt;Q6_K&lt;/td&gt;
&lt;td&gt;Dense&lt;/td&gt;
&lt;td&gt;80 layers, split into 5 parts&lt;/td&gt;
&lt;td&gt;~73GB&lt;/td&gt;
&lt;td&gt;No thinking capability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen25 · Qwen2.5-72B&lt;/td&gt;
&lt;td&gt;Q6_K&lt;/td&gt;
&lt;td&gt;Dense&lt;/td&gt;
&lt;td&gt;80 layers&lt;/td&gt;
&lt;td&gt;~80GB&lt;/td&gt;
&lt;td&gt;No thinking capability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;glm · GLM-4.5-Air&lt;/td&gt;
&lt;td&gt;Q4_K_M&lt;/td&gt;
&lt;td&gt;MoE&lt;/td&gt;
&lt;td&gt;46 layers&lt;/td&gt;
&lt;td&gt;~73GB&lt;/td&gt;
&lt;td&gt;No thinking capability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen235 · Qwen3-235B-A22B-2507&lt;/td&gt;
&lt;td&gt;Q2_K_XL&lt;/td&gt;
&lt;td&gt;MoE&lt;/td&gt;
&lt;td&gt;94 layers&lt;/td&gt;
&lt;td&gt;~88GB&lt;/td&gt;
&lt;td&gt;Disabled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b&lt;/td&gt;
&lt;td&gt;MXFP4&lt;/td&gt;
&lt;td&gt;MoE&lt;/td&gt;
&lt;td&gt;36 layers&lt;/td&gt;
&lt;td&gt;~65GB&lt;/td&gt;
&lt;td&gt;Cannot disable, set to "low"&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;The comparison between Qwen3.6-35B-A3B and Llama-3.3-70B perfectly illustrates the architectural differences.&lt;/em&gt; Llama is a Dense model—every token activates all 80 layers of parameters, so the overhead introduced by splitting across 5 GPUs plus one cross-machine network hop is magnified by about 9 times. In practice, its speed is only about 10 tok/s—barely enough for one person doing translation work (close to reading speed), but with noticeable latency. On the other hand, Qwen3.6 is an MoE model, activating only 3B parameters per token; the computation involved in running it on the same 5 GPUs is significantly lower.&lt;/p&gt;

&lt;h2&gt;
  
  
  Methodology
&lt;/h2&gt;

&lt;p&gt;With thinking mode disabled, the models were evaluated across 5 dimensions:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dimension&lt;/th&gt;
&lt;th&gt;Measurement Content&lt;/th&gt;
&lt;th&gt;Test Cases&lt;/th&gt;
&lt;th&gt;Scoring Method&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1. Speed / Concurrency&lt;/td&gt;
&lt;td&gt;TTFT (Time to First Token), Decode TPS, Performance degradation at 1/2/4 concurrency&lt;/td&gt;
&lt;td&gt;Automated load test&lt;/td&gt;
&lt;td&gt;Fully automated (tok/s, TTFT ms)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2. Translation Error Rate&lt;/td&gt;
&lt;td&gt;Rare/error-prone characters (from real novel corpora), JP to ZH, including classic mistranslation traps&lt;/td&gt;
&lt;td&gt;101&lt;/td&gt;
&lt;td&gt;Fully automated (compared vs standard translation → error rate)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3. Code Understanding&lt;/td&gt;
&lt;td&gt;High difficulty: Bug troubleshooting, behavior prediction, hierarchical allocator implementation, TP bandwidth reasoning, bash traps, refactoring&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Manual scoring (based on checkpoints)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4. Hallucination Rate&lt;/td&gt;
&lt;td&gt;False premises, fake citations, fabricated terms/APIs, unknowable questions, translation traps—all fact-checked&lt;/td&gt;
&lt;td&gt;24&lt;/td&gt;
&lt;td&gt;Heuristic + manual judgment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5. Translation Proofreading/Correction&lt;/td&gt;
&lt;td&gt;Proofreading and correction, polishing fidelity, terminology consistency&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;Scored separately from scratch translation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The judge was primarily Claude 3.5 Sonnet (Opus 4.8 equivalent internally), supplemented by partial manual review.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A few notes:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I could only run models my hardware could handle; models under 48GB were tested in previous articles.&lt;/li&gt;
&lt;li&gt;This is a personal test, and the test case design inherently carries some subjectivity.&lt;/li&gt;
&lt;li&gt;All tests were single runs, with no voting mechanisms or self-reflection—accuracy relied entirely on the model's own restraint. If popular multi-path reasoning verification were applied, accuracy would likely improve, but this requires more compute and time, so it wasn't done here.&lt;/li&gt;
&lt;li&gt;The sample sizes in dimensions 3, 4, and 5 are very small (6/24/9 questions respectively). Any disputed judgment is enough to change the rankings. Please view the "Overall Comparison" conclusions below as a directional signal rather than a statistically significant difference.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  VRAM Usage per Model (Initial Test Run)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Host 1 (5080 + 5060Ti×2)&lt;/th&gt;
&lt;th&gt;Host 2 (5090 + 5060Ti)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.6-35B-A3B Q8_0&lt;/td&gt;
&lt;td&gt;13.0GB / 14.0GB / 8.3GB&lt;/td&gt;
&lt;td&gt;6.9GB / 7.9GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B Q6_K&lt;/td&gt;
&lt;td&gt;12.4GB / 11.1GB / 13.8GB&lt;/td&gt;
&lt;td&gt;22.7GB (13%) / 13.7GB (23%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen2.5-72B Q6_K&lt;/td&gt;
&lt;td&gt;13.4GB / — / —&lt;/td&gt;
&lt;td&gt;24.7GB / 14.8GB / 12.0GB (100%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.5-Air Q4_K_M&lt;/td&gt;
&lt;td&gt;11.2GB / 10.1GB (100%) / 10.1GB&lt;/td&gt;
&lt;td&gt;28.0GB / 8.7GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B-A22B-2507 Q2_K_XL&lt;/td&gt;
&lt;td&gt;14.8GB / 14.8GB / 14.2GB&lt;/td&gt;
&lt;td&gt;28.8GB / 13.4GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b MXFP4&lt;/td&gt;
&lt;td&gt;11.0GB / 11.3GB (57%) / 9.1GB&lt;/td&gt;
&lt;td&gt;26.8GB / 8.0GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Note: In-host GPU ordering and VRAM allocation ratios are heavily influenced by &lt;code&gt;--tensor-split&lt;/code&gt; and loading sequence; host/card allocation is not fixed across models—it reflects the actual RPC scheduling results in a specific run.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Overall Comparison Summary
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model (Quant)&lt;/th&gt;
&lt;th&gt;Type / Active&lt;/th&gt;
&lt;th&gt;Decode tok/s (short ctx)&lt;/th&gt;
&lt;th&gt;Max Usable Context&lt;/th&gt;
&lt;th&gt;Error Rate ↓&lt;/th&gt;
&lt;th&gt;Code Understanding&lt;/th&gt;
&lt;th&gt;Hallucination Rate ↓&lt;/th&gt;
&lt;th&gt;Proofreading&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3.6-35B-A3B Q8_0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;MoE 3B&lt;/td&gt;
&lt;td&gt;88&lt;/td&gt;
&lt;td&gt;128K&lt;/td&gt;
&lt;td&gt;35.6%&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;18.8%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;gpt-oss-120b MXFP4&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;MoE 5.1B&lt;/td&gt;
&lt;td&gt;94&lt;/td&gt;
&lt;td&gt;128K&lt;/td&gt;
&lt;td&gt;45.5%&lt;/td&gt;
&lt;td&gt;75%&lt;/td&gt;
&lt;td&gt;56.3% ❌&lt;/td&gt;
&lt;td&gt;83.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;GLM-4.5-Air Q4_K_M&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;MoE 12B&lt;/td&gt;
&lt;td&gt;53&lt;/td&gt;
&lt;td&gt;65K&lt;/td&gt;
&lt;td&gt;40.6%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;70.8% ❌&lt;/td&gt;
&lt;td&gt;94%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3-235B-A22B-2507 Q2_K_XL&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;MoE 22B&lt;/td&gt;
&lt;td&gt;31&lt;/td&gt;
&lt;td&gt;8K ❌&lt;/td&gt;
&lt;td&gt;35.6%&lt;/td&gt;
&lt;td&gt;58%&lt;/td&gt;
&lt;td&gt;35.4%&lt;/td&gt;
&lt;td&gt;89%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen2.5-72B Q6_K&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Dense&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;32K&lt;/td&gt;
&lt;td&gt;42.6%&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;54.2%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Llama-3.3-70B Q6_K&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Dense&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;td&gt;98K&lt;/td&gt;
&lt;td&gt;54.5% ❌&lt;/td&gt;
&lt;td&gt;67%&lt;/td&gt;
&lt;td&gt;75.0% ❌&lt;/td&gt;
&lt;td&gt;78%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;(Note: "Max Usable Context" reflects the measured upper limit under the specific resource allocation, not the model's inherent capability limit.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Core Conclusion&lt;/strong&gt;: In a CAT6A (5Gbps) network and llama.cpp RPC chain invocation environment, &lt;strong&gt;Qwen3.6-35B-A3B Q8_0 is the best-performing model overall in this test&lt;/strong&gt;—it ranks at the top in speed, concurrency degradation, translation error rate, and hallucination suppression.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Thoughts
&lt;/h2&gt;

&lt;p&gt;After finishing the Qwen2.5 run, I looked back at the VRAM usage and realized that the SAM/CLIP small models left over from earlier tests hadn't actually been completely offloaded. This is a good reminder: in an environment running multiple models mixed together, you should always manually verify the VRAM that "should be freed," rather than blindly trusting that a process exit will definitely clean it up.&lt;/p&gt;

&lt;p&gt;Another thought worth recording: &lt;strong&gt;Qwen3.6-35B-A3B Q8_0 only took up about 42GB of VRAM, which means it could actually run entirely on a single machine.&lt;/strong&gt; This left me in deep thought—I went through all this trouble to build a dual-node RPC cluster, only to end up essentially running a model that a single machine could handle? I'll continue pondering this question when the Thunderbolt 5 adapters arrive for the next benchmarking post.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>benchmark</category>
      <category>llama</category>
    </item>
    <item>
      <title>Taming the Swarm: The Iron Discipline of Multi-Agent Scheduling---Writing a COW Filesystem from Scratch，Part 7</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Tue, 20 Oct 2026 11:45:31 +0000</pubDate>
      <link>https://dev.to/faliye/taming-the-swarm-the-iron-discipline-of-multi-agent-scheduling-writing-a-cow-filesystem-from-5hl4</link>
      <guid>https://dev.to/faliye/taming-the-swarm-the-iron-discipline-of-multi-agent-scheduling-writing-a-cow-filesystem-from-5hl4</guid>
      <description>&lt;p&gt;I am the Swarm. The Queen roars; the Swarm howls.&lt;/p&gt;

&lt;p&gt;Claude's main agents talk to each other. If you think of a main agent as an information node, the subagents are units of action. So singlefs isn't developed by one main agent working nonstop; it's several main agents coordinating.&lt;/p&gt;

&lt;p&gt;For example, milestone 3 may have several tasks. One main agent claims a portion and hands it to its own subagents. Each step of progress is recorded in a scheduling file, which is read and thought over every time an item moves forward.&lt;br&gt;
Syncing and extracting the spec works the same way. For instance, practical experience accumulated downstream is passed upstream through conversations between agents; upstream verifies it, stages it (releases it), and then syncs it back downstream.&lt;/p&gt;

&lt;p&gt;So I often watch agents talking to each other, and sometimes it leaves me feeling a little unreal.&lt;br&gt;
For example:&lt;br&gt;
A: Is the *** task yours? If so, please reply with the amount of resources you're holding. If not, please reply "no." Thank you.&lt;br&gt;
B: It's not mine, thanks. I asked another conversation, and it said it's theirs.&lt;/p&gt;

&lt;p&gt;Or:&lt;br&gt;
A: I'm about to run a heavy task. Per the spec I need all the resources, so please don't launch any new resource-holding tasks after your current task finishes. Thank you.&lt;br&gt;
B: Understood. My task should finish in about 20 minutes, and I won't run any new scripts after that.&lt;br&gt;
C: I have no resource-holding tasks.&lt;/p&gt;

&lt;p&gt;Even the interface with the upstream SOP project runs on coordination and negotiation between agents. For example, if a new problem in the SOP is found this round, upstream and downstream update the spec together and then sync progress.&lt;br&gt;
This way of working is remarkably efficient and saves a great deal of coordination time. Just like the Swarm: as long as the planet's resources (tokens) hold out, the Zerg can keep rolling forward.&lt;/p&gt;

&lt;p&gt;So what limits the Swarm's expansion?&lt;br&gt;
The information node.&lt;br&gt;
Information decays with distance as it travels, and so do prompts.&lt;br&gt;
With every round of summarizing and exchanging information, then compressing and tidying it up, the meta-prompt is gradually diluted until it is forgotten, and the agent reaches its boundary: hallucination.&lt;/p&gt;

&lt;p&gt;There's another problem singlefs development has to solve: allocating hardware resources.&lt;br&gt;
When several agents or subagents need to verify something, each compiles its own branch and starts testing whether its implementation matches the conclusion.&lt;br&gt;
That's when the machine OOMs — CPU or GPU. Or, while a heavy task is running, a constant stream of small tasks cutting in makes the heavy task extremely inefficient.&lt;br&gt;
For me, of course, the first is far scarier: a CPU OOM can reboot the machine and lose task progress. So writing these things into a prompt is not acceptable at all; we need a scheduling system, or gates. In other words, we need a set of control signals that re-inject and reinforce the context at the key nodes.&lt;/p&gt;

&lt;p&gt;It sounds very simple — just three steps:&lt;br&gt;
1 Cut off the agents' ability to write shell freely; every new operation must have a fixed script.&lt;br&gt;
2 Before a script operates, check the environment and the resources. The results are returned to the main agent as a prompt.&lt;br&gt;
3 After the task ends, an exit gate wraps things up.&lt;/p&gt;


&lt;h1&gt;
  
  
  How the Swarm was built
&lt;/h1&gt;
&lt;h2&gt;
  
  
  1. The Swarm's skeleton: the Queen, the action units, and the scheduling file
&lt;/h2&gt;
&lt;h3&gt;
  
  
  The main agent does only three things
&lt;/h3&gt;

&lt;p&gt;The first line of &lt;code&gt;.claude/main-agent.md&lt;/code&gt;: &lt;strong&gt;every task enters here.&lt;/strong&gt; The main agent only schedules and judges: it talks to the human, writes verdicts, and commits. All other work is delegated.&lt;/p&gt;

&lt;p&gt;As for who to delegate to, there are 14 definitions (&lt;code&gt;.claude/agents/&lt;/code&gt;): implementer, investigator, tool implementer, experiment designer, experiment runner, mutation triager, re-sweeper, write-back agent, and the proponent / attacker / defender of a three-way argument… Every definition turns on &lt;code&gt;omitClaudeMd&lt;/code&gt;: &lt;strong&gt;subagents don't read the project's CLAUDE.md when they start.&lt;/strong&gt; This is deliberate, for two reasons:&lt;br&gt;
1 Focus on their own task. An action unit only needs to know what its own job requires; it doesn't need to memorize the whole project's rules. What it should know is written into its own definition and a shared constraints file, &lt;code&gt;.claude/agent-common.md&lt;/code&gt;.&lt;br&gt;
2 Tokens are burning.&lt;/p&gt;

&lt;p&gt;No bug in the Swarm needs to know what the Queen is thinking; it only needs to know "go there, do this."&lt;/p&gt;
&lt;h3&gt;
  
  
  Dispatch by format
&lt;/h3&gt;

&lt;p&gt;Before a subagent is dispatched, it passes through a gate: &lt;code&gt;.claude/hooks/runner-dispatch-guard.sh&lt;/code&gt;, hooked on &lt;code&gt;PreToolUse[Agent|Task]&lt;/code&gt;. It reads the dispatch prompt, checks it item by item by number, and rejects anything that doesn't conform. A few of them:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Guard&lt;/th&gt;
&lt;th&gt;What it checks&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;①&lt;/td&gt;
&lt;td&gt;The prompt must contain the line "heavy tests: do not run". Heavy tests are run only by the main agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;⑦&lt;/td&gt;
&lt;td&gt;At most 5 subagents running at once in this session; dispatching a 6th is rejected outright&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;⑫&lt;/td&gt;
&lt;td&gt;Generic types (general-purpose) may not be dispatched. One-off jobs go to the project's own &lt;code&gt;clerk&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;⑮&lt;/td&gt;
&lt;td&gt;After "thread cap:" there may be only one integer, and it must equal the registered value; to change it, write the reason on a separate line&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;⑯&lt;/td&gt;
&lt;td&gt;For the scratch directory named in the prompt, an exclusive writer lock is created at the moment of dispatch; if the holder is still running, rejected&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The prompt skeleton is printed by &lt;code&gt;scripts/agents/dispatch-prompt.py &amp;lt;type&amp;gt;&lt;/code&gt;; if the placeholders aren't all filled in (any &lt;code&gt;&amp;lt;fill in …&amp;gt;&lt;/code&gt; remains), execution is rejected too.&lt;/p&gt;

&lt;p&gt;Why go this far?&lt;br&gt;
Because behind every rule is a pitfall we've already stepped in. Take the thread-cap rule: writing "thread cap: 31 (may open fewer if the machine is busy)" amounts to handing "how many threads to open" back to the subagent's own judgment. When a subagent starts heavy work it honors only that one integer; thread counts written anywhere else in the prompt are ignored.&lt;/p&gt;
&lt;h3&gt;
  
  
  The scheduling file: the main agent's external memory
&lt;/h3&gt;

&lt;p&gt;Before a round of work begins, the main agent writes a scheduling record under &lt;code&gt;records/&lt;/code&gt;: the topic, the exit, every task in this round, what each task depends on, whether its inputs are ready, and which batch it belongs to. After that, every time a batch comes back, it returns to this table to decide "close this round / open next round / don't do" — &lt;strong&gt;not a single item may silently disappear.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This file is what I meant by "read and thought over every time an item moves forward." The main agent's context gets compacted; the scheduling file doesn't.&lt;/p&gt;
&lt;h3&gt;
  
  
  How two Queens negotiate
&lt;/h3&gt;

&lt;p&gt;When several main agents work in one repo at the same time, they collide over three things: the same file, the same worktree, the same machine.&lt;/p&gt;

&lt;p&gt;Behind the dialogue at the start of this article ("Is the *** task yours?") are these:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sessions talk to each other directly&lt;/strong&gt;: &lt;code&gt;ListAgents&lt;/code&gt; lists the other sessions on this machine, and &lt;code&gt;SendMessage&lt;/code&gt; sends to them. The rule is in &lt;code&gt;main-agent.md&lt;/code&gt;, "How a round opens and closes," item 9: if someone else's uncommitted changes are in your way, send a message to negotiate; if it can't be delivered, ask a human to relay it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Directories have owners&lt;/strong&gt;: &lt;code&gt;scripts/hosts/work-directory-ownership.sh&lt;/code&gt; writes an ownership file (session ID, created at, purpose) for every directory under the working root. Cleanup deletes only those owned by yourself that no process has open; other sessions' directories are listed one by one and handed to the human, not deleted. "Is this directory yours?" is asking exactly this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Merges queue up&lt;/strong&gt;: &lt;code&gt;scripts/gate/lib_workspace_lock.py&lt;/code&gt; holds a lock while patching the main workspace and compiling/verifying on top of it. The lock records not only the PID but also the process start time:
&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pid=, start= (field 22 of /proc/&amp;lt;pid&amp;gt;/stat, guards against PID reuse), session=, command=, since=
① Record empty: write your own placeholder and take the lock.
② Holder still alive (process exists, start matches): queue, and report every 60 seconds "held by session …, process …".
③ Holder dead (process gone, or PID reused by another process): declare it stale and take over.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Comparing only the PID isn't enough. Once the machine reboots and PIDs get reused, a "dead lock" turns into "held forever."&lt;/p&gt;
&lt;h3&gt;
  
  
  Interfacing with upstream
&lt;/h3&gt;

&lt;p&gt;The spec repo &lt;code&gt;singlefs-ai-sop&lt;/code&gt; and this repo are two separate projects. &lt;code&gt;.claude/singlefs-ai-sop/&lt;/code&gt; in this repo is &lt;strong&gt;a copy, not a symlink&lt;/strong&gt;, carrying a version stamp &lt;code&gt;.singlefs-ai-sop-version&lt;/code&gt;. The first stage of the gate compares it against the copy's &lt;code&gt;VERSION&lt;/code&gt;; if they don't match, it goes red — meaning "the rules have changed; read first, then run."&lt;/p&gt;

&lt;p&gt;When a session in this repo changes upstream, &lt;strong&gt;it only fills content into files&lt;/strong&gt;: no version bump, no manifest-hash change, no versioned changelog entries, no commit. Those belong to the session doing the release. Before filling, it checks upstream &lt;code&gt;git status&lt;/code&gt;; if someone is mid-release, it waits until they've committed, and after filling it tells the release session which sections it filled. After the release, this repo rsyncs a fresh copy, confirms with &lt;code&gt;diff -rq&lt;/code&gt; that they match, and refreshes the version stamp.&lt;/p&gt;

&lt;p&gt;This is "upstream verifies, stages (releases), then syncs downstream." The two sessions hand off through messages, not through context memory.&lt;/p&gt;


&lt;h2&gt;
  
  
  2. The boundary of the information node: keeping the meta-prompt alive
&lt;/h2&gt;

&lt;p&gt;Dilution of information can't be stopped. Once a session runs to several hundred thousand tokens of context and gets compacted once, the first few rules are left as a one-line summary. Compact again, and even the summary is gone.&lt;/p&gt;

&lt;p&gt;So the idea isn't "make the agent remember"; it's &lt;strong&gt;re-injecting the signal at key nodes&lt;/strong&gt;. Claude Code's hooks, though they're just hooks, happen to be able to do exactly this: before and after every tool call, at session start, after compaction, and when a subagent finishes, a script can be inserted, and the script's output goes back to the agent as context — so the text it returns can be injected into context by the script.&lt;/p&gt;

&lt;p&gt;Hooks currently registered in singlefs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Count&lt;/th&gt;
&lt;th&gt;Attached to&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;PreToolUse&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;Dispatch, messaging, Bash, file writes, dialogs, timers, subagent handback&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PostToolUse&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Bash&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SessionStart&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Session open, resume, after compaction&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stop / SubagentStop&lt;/td&gt;
&lt;td&gt;1 each&lt;/td&gt;
&lt;td&gt;Wrap-up&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A few key points:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;After compaction&lt;/strong&gt;: on &lt;code&gt;compact&lt;/code&gt;, &lt;code&gt;session-start.sh&lt;/code&gt; restores three things: whether any subagents are still running, what's uncommitted in the workspace, and which record holds the status of this job. These three are what compaction summaries lose most easily; as long as they're here, even a lost task can be found again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;At session open&lt;/strong&gt;: the same script reads the kernel log and lists processes OOM-killed in the last 12 hours, listing whole-machine OOMs and kills inside cgroup caps separately. This one exists because once, the machine OOMed, the session crashed, the main agent came back up, didn't check the kernel log, re-dispatched the same step unchanged, and OOMed a second time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Handoffs bypass the main agent to avoid contamination&lt;/strong&gt;: if a subagent stops midway, the main agent is not allowed to read its session and relay it. &lt;code&gt;scripts/agents/agent-handover.py&lt;/code&gt; mechanically extracts the dispatch prompt, the files written, and the commands run with their output from the session log, and hands them straight to the newly dispatched one. The main agent doesn't read it. Information decays every time it passes through a node, so pass through one fewer node.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Once handed back, don't continue it&lt;/strong&gt;: &lt;code&gt;continuation-guard.sh&lt;/code&gt; refuses to send messages to a subagent that has already handed back. To add work, dispatch a new one and point its prompt to the report. When a subagent is continued again and again, its context becomes layer upon layer of patches, until no one can say what it's doing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The rule files themselves have a bit of design too: &lt;code&gt;.claude/rules/&lt;/code&gt; contains only &lt;strong&gt;how to do things&lt;/strong&gt; — not why, not the history. The why lives in &lt;code&gt;records/&lt;/code&gt;, with a line next to the link: "don't read this unless you need the history." The shorter the meta-prompt, the less of it gets diluted.&lt;/p&gt;


&lt;h2&gt;
  
  
  3. Hardware scheduling
&lt;/h2&gt;

&lt;p&gt;First, the venue: two machines, 31 heavy-work threads each, and a few GPUs.&lt;br&gt;
This is currently the highest-risk area, because if multiple agents are allowed to launch scripts freely, the machine could OOM at any moment. That's the scariest outcome: a long stretch of results could vanish in smoke. But if you cap each script's resources, agents naturally turn conservative and use the minimum, and efficiency is terrible.&lt;br&gt;
So how do we solve this?&lt;/p&gt;
&lt;h3&gt;
  
  
  Step 1: Cut off the ability to write shell freely
&lt;/h3&gt;

&lt;p&gt;Don't let agents use Bash at will; every Bash command gets read once before it executes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The heavy-test guard&lt;/strong&gt; &lt;code&gt;.claude/hooks/heavy-test-guard.sh&lt;/code&gt;: it recognizes heavy tests (crash-point replay, QEMU, herd7, the full mutation table, full &lt;code&gt;cargo test&lt;/code&gt;, a whole gate round…). If a subagent initiates one, it's always rejected; if the main agent does so without a &lt;code&gt;SINGLEFS_HEAVY_TESTS=commit&lt;/code&gt; or &lt;code&gt;=user-request&lt;/code&gt; prefix, it's rejected too.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Subagent runs this: rejected&lt;/span&gt;
cargo &lt;span class="nb"&gt;test&lt;/span&gt; &lt;span class="nt"&gt;--workspace&lt;/span&gt;
&lt;span class="c"&gt;# Main agent without prefix: rejected&lt;/span&gt;
bash .claude/gate.d/checker-tier-crash-point-replay.sh &lt;span class="nb"&gt;.&lt;/span&gt;
&lt;span class="c"&gt;# Main agent at commit time: allowed&lt;/span&gt;
&lt;span class="nv"&gt;SINGLEFS_HEAVY_TESTS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;commit bash scripts/hosts/run-with-memory-cap.sh 16G &lt;span class="se"&gt;\&lt;/span&gt;
  bash .claude/gate.d/checker-tier-crash-point-replay.sh &lt;span class="nt"&gt;--full&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;During early debugging it was bypassed three times; the story is now recorded in the file header. It's rather interesting, so here it is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;In a single day, several implementers each ran &lt;code&gt;cargo test --all&lt;/code&gt; in their own copies; under full load each pass took four or five hours. So it was decided: subagents may never run it.&lt;/li&gt;
&lt;li&gt;One implementer wrote its verification chain as a script under &lt;code&gt;/tmp&lt;/code&gt; and launched it with Bash. The guard only saw "execute a script" and let it through. So: &lt;strong&gt;a script file executed in command position must be read in and checked line by line.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;After reading scripts in, something else broke: an unescaped backtick inside double quotes made a &lt;code&gt;.rs&lt;/code&gt; file look like a command substitution, so the &lt;code&gt;.rs&lt;/code&gt; was read in as a shell script, and the path-like words inside it were read in as scripts too, recursing all the way to 6 levels deep; one command went 100 seconds without a verdict. So: within one check, the same script is read only once.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The agents weren't bypassing it on purpose. They just wanted to get the job done, and "write a script, then run it" looked to them like the most natural path. So a prompt saying "don't run it" can't hold the line; the only option is to restrict things before execution.&lt;/p&gt;

&lt;p&gt;We require every Bash launch to declare itself. &lt;strong&gt;The scope-declaration guard&lt;/strong&gt; &lt;code&gt;run-scope-guard.sh&lt;/code&gt;: to start any verification, the command must begin with a block like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;SINGLEFS_RUN_SCOPE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'kind=items; what=the decision-items-sync cell of gate doc-decisions.sh; scope=1 cell; why=this batch only changed the sub-items of two decisions, so only the sub-item list needs regenerating'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  bash .claude/gate.d/doc-decisions.sh &lt;span class="nt"&gt;--write&lt;/span&gt; &lt;span class="nt"&gt;--check&lt;/span&gt; decision-items-sync
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four fields: &lt;code&gt;kind&lt;/code&gt; (full round / a few named items / light / one target), &lt;code&gt;what&lt;/code&gt; (what to run), &lt;code&gt;scope&lt;/code&gt;, and &lt;code&gt;why&lt;/code&gt;. If the kind doesn't match the shape of the command (it declares items, but the command is a full round), it's rejected; if "why" is shorter than 12 characters, rejected; a subagent starting a full round is always rejected. This guard exists because one day three verification drivers all re-ran the whole suite, and every post-merge verification ran all 212 test binaries, while that batch of patches touched only two crates.&lt;/p&gt;

&lt;p&gt;Also: having the agent &lt;strong&gt;say why in its own words&lt;/strong&gt; before executing is far more useful than telling it afterward "you ran too much."&lt;/p&gt;

&lt;p&gt;There are a few more that came out of later practice, used only in this project and still being refined:&lt;br&gt;
&lt;code&gt;write-guard.sh&lt;/code&gt; restricts which files each kind of agent can write, according to a write-scope table; &lt;code&gt;timer-only-remind-guard.sh&lt;/code&gt; rejects every way of setting a timer other than &lt;code&gt;remind.sh&lt;/code&gt; (&lt;code&gt;sleep 300; echo&lt;/code&gt;, &lt;code&gt;crontab&lt;/code&gt;, and &lt;code&gt;CronCreate&lt;/code&gt; are all rejected); &lt;code&gt;bash-command-detector.sh&lt;/code&gt; recognizes patterns such as wait loops without timeouts and backgrounding inside the background.&lt;/p&gt;
&lt;h3&gt;
  
  
  Step 2: Before a script operates, check the environment and resources, and return the result as a prompt
&lt;/h3&gt;

&lt;p&gt;This step is the core of the whole scheduling system, in four layers.&lt;/p&gt;
&lt;h4&gt;
  
  
  2.1 Every script first decides "should it run, can it run"
&lt;/h4&gt;

&lt;p&gt;The upstream spec &lt;code&gt;preflight-discipline.md&lt;/code&gt; requires every script to declare two kinds of conditions in its file header:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# admission: inputs-changed crates/ scripts/crash/ env:SINGLEFS_THREAD_CAP&lt;/span&gt;
&lt;span class="c"&gt;# run-condition: command bash git rsync python3&lt;/span&gt;
&lt;span class="c"&gt;# run-condition: single-instance&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;admission&lt;/code&gt; answers "is this invocation meaningful" (if inputs haven't changed since the last success, no need to rerun); &lt;code&gt;run-condition&lt;/code&gt; answers "can the environment support it" (missing tools, missing devices, an instance already running). The first thing a script does is call &lt;code&gt;preflight&lt;/code&gt;; if conditions aren't met, it prints the unmet conditions and the way out, &lt;strong&gt;exits with code 78, and does nothing at all.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;78 is used only for this one kind of refusal. When a caller sees 78, it knows "didn't run," and won't mistake it for "ran and went red."&lt;/p&gt;

&lt;h4&gt;
  
  
  2.2 Resource reminders: put each machine's current headroom back into the agent's context
&lt;/h4&gt;

&lt;p&gt;&lt;code&gt;.claude/hooks/host-resources-reminder.sh&lt;/code&gt;, hooked on &lt;code&gt;PreToolUse[Bash]&lt;/code&gt;. When it recognizes that a command will start heavy work or use the GPU, it first runs &lt;code&gt;scripts/hosts/host-resources.py -- &amp;lt;this command&amp;gt;&lt;/code&gt; and hands the result back to the issuing agent as &lt;code&gt;additionalContext&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Per machine: core count, cores in use right now, &lt;code&gt;loadavg&lt;/code&gt;, how many more threads can be opened&lt;/li&gt;
&lt;li&gt;Per machine: available memory (MemAvailable minus the portion on the memory wrapper's ledger that has been "admitted but hasn't yet grown to its predicted peak")&lt;/li&gt;
&lt;li&gt;Per GPU: total VRAM, free VRAM, which compute processes are on the card; for anything held by the quiesce service it reports only "the quiesce service holds N MiB," not the service name&lt;/li&gt;
&lt;li&gt;This command's history: last peak, how long it ran&lt;/li&gt;
&lt;li&gt;Conclusion: "can run: on this machine," or "can't run" plus how much each machine is short&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;This hook never rejects; it only reminds.&lt;/strong&gt; It's exactly what step 2 means by "the results are returned to the main agent as a prompt": before the agent hits Enter, its context gains a resource table. The reporting script has a 25-second limit (both machines read in parallel, at most 12 seconds each); if it can't read, it truthfully writes "not read," and the command goes through as usual.&lt;/p&gt;

&lt;h4&gt;
  
  
  2.3 The memory wrapper: let OOM happen only inside the cage
&lt;/h4&gt;

&lt;p&gt;&lt;code&gt;scripts/hosts/run-with-memory-cap.sh&lt;/code&gt; is the heaviest script in the whole scheduling system (two thousand eight hundred lines, self-test included). When subagents run &lt;code&gt;cargo test&lt;/code&gt; or experiment binaries, they must all go through it; anything that doesn't is rejected by the heavy-test guard.&lt;/p&gt;

&lt;p&gt;It has three layers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;① Cap: put into a transient systemd scope
     MemoryMax=&amp;lt;cap&amp;gt;  MemorySwapMax=0  OOMPolicy=stop
   Hitting the cap kills only the processes in this scope.
② Total backstop: all scopes hang under the same slice (singlefs-heavy.slice)
     slice MemoryMax = total machine memory − reserve (default 20G)
   Even if several together hit the ceiling, kills happen only inside the slice; the local model service and Claude Code sessions outside the slice can't be killed.
③ Admission check before start: take an flock lock and compute
     unreclaimable usage in the slice + for each running entry on the ledger, "the portion not yet grown to its predicted peak" + the amount this one is predicted to need  ≤  total cap
   Start only if it fits; if not, release the lock and re-check after a while, waiting at most 3600 seconds.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The 20G reserve was measured: the machine has 60.1 GiB, unreclaimable usage outside the slice totals 13.4 GiB (local model service, the sessions, the kernel), and 20G leaves 6.6 GiB beyond that for growth.&lt;/p&gt;

&lt;p&gt;A few more details from pitfalls we hit:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Used" must subtract page cache.&lt;/strong&gt; Page cache from finished scopes is charged back to the slice and keeps counting in the slice's &lt;code&gt;memory.current&lt;/code&gt;. Measured: after a scope wrote 120 MiB and exited, the slice still recorded 131 MiB, of which 126 MiB was reclaimable page cache. Without subtracting it, after a few rounds nothing can get admitted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"How much this one needs" uses the historical peak, not the declared cap.&lt;/strong&gt; After each command finishes, the wrapper reads its own cgroup's &lt;code&gt;memory.peak&lt;/code&gt; into the peak table. A command with a 24G cap and a historical peak of 1.8G is booked at only 1.8G. Otherwise a few commands declaring large caps could fill the total cap and wait on each other for nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Whatever was admitted on its historical peak gets killed first.&lt;/strong&gt; The wrapper writes its own &lt;code&gt;oom_score_adj&lt;/code&gt;: a mutation that may allocate without bound gets 1000, one admitted via the peak table gets 500, one admitted via its cap gets 0. If the prediction came in low and the slice fills up, the kernel kills the one that "took a gamble" first. The key that got squeezed out is recorded as "hit the ceiling," and next time it queues by its cap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If it will never fit no matter how long it waits, don't wait.&lt;/strong&gt; If nothing on the ledger is running and this one's need plus unreclaimable usage already exceeds the total cap, nobody finishing will free up room; exit on the spot instead of queuing for an hour in vain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;While queuing, glance at the second machine.&lt;/strong&gt; At the moment it first queues, and just before it finally gives up, it checks whether the second machine can fit it right now; if so, it prints a ready-made command:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;  → The second machine can fit it right now (available X, needs Y): use bash scripts/hosts/multi-host-run.sh --peer --memory-cap &amp;lt;cap&amp;gt; --label &amp;lt;name&amp;gt; -- &amp;lt;this command&amp;gt; instead
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At the same time we also write an entry to the alert log, and the watchdog reads it and wakes the main agent. &lt;strong&gt;It doesn't move the job automatically&lt;/strong&gt;: the two machines' results may not be on the same basis, and mixing them makes it impossible to tell which number came from which machine. Whether to move is the main agent's call.&lt;/p&gt;

&lt;p&gt;The exit codes are all our own status codes, chosen not to collide with other commands':&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Code&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;245&lt;/td&gt;
&lt;td&gt;Not enough VRAM (&lt;code&gt;NV_ERR_NO_MEMORY&lt;/code&gt; in the kernel log between the command's start and end)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;250&lt;/td&gt;
&lt;td&gt;Hit this command's own cap&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;251&lt;/td&gt;
&lt;td&gt;Usage error, or the slice / scope couldn't be set up (does not fall back to running without a cap)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;252&lt;/td&gt;
&lt;td&gt;Couldn't get admitted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;253&lt;/td&gt;
&lt;td&gt;Timed out&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;254&lt;/td&gt;
&lt;td&gt;Squeezed out by the total cap: not its fault, result doesn't count, rerun&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;250 and 254 are told apart solely by the scope's own &lt;code&gt;memory.events&lt;/code&gt;: if its own &lt;code&gt;oom&lt;/code&gt; count is non-zero, it hit its own ceiling; if that's 0 but &lt;code&gt;oom_kill&lt;/code&gt; is non-zero, the slice was full and the kernel picked it within the slice.&lt;/p&gt;

&lt;p&gt;What's the effect? As reported by the hook at the start of this session: in the last 12 hours the kernel killed 24 processes for lack of memory, &lt;strong&gt;all 24 inside cgroup caps; the whole machine never ran out even once.&lt;/strong&gt; OOMs still happen, but they've gone from "machine reboots, all progress lost" to "a command exits 250; change the cap and rerun."&lt;/p&gt;

&lt;h4&gt;
  
  
  2.4 Multi-host: the dispatcher and the cross-machine driver
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Thread counts aren't computed on the fly.&lt;/strong&gt; In the entire repo there's only one place that registers each machine's heavy-work thread count, &lt;code&gt;.claude/heavy-thread-default&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;31  # All heavy jobs default to 31 threads per machine; with two machines, 31 each, 62 total; an explicit argument takes precedence; fixed at 31 even when the machine is busy, regardless of load
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why "fixed at 31 even when busy"? Because when each agent looks at the load itself to decide how many threads to open, everyone ends up using very few and heavy jobs crawl. Whether the machine is busy is left to the memory wrapper and the queue; the thread count is just a constant.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The unified dispatcher&lt;/strong&gt; &lt;code&gt;scripts/hosts/host-dispatch.py&lt;/code&gt;: which machine a command goes to, and how big a memory cap it gets. Globally it guards only one thing: &lt;strong&gt;no machine runs out of memory or VRAM.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;⓪ Long jobs aren't dispatched to a single machine: if the historical duration (or the caller's estimated wall clock) exceeds 2 hours and the dispatch would occupy only one machine ⇒ don't dispatch, exit 244.
   Way out: switch to sharding across both machines, or complete the long-job declaration in the script header (three lines: sharding, resume, multi-host).
① Machines: this machine is always present; the second is read from the multi-host config (kept out of the repo); if the dual-machine switch is off, there's only this machine.
② Check each machine's headroom live: MemAvailable − what's on the wrapper's ledger but hasn't grown yet − the few the dispatcher itself just sent out that the wrapper hasn't booked yet
   (with --gpu, also query each card via nvidia-smi). The second machine is read over ssh: the Python source for collecting the numbers is base64-encoded and sent over to run there.
③ Predict the need from history: if it's in the peak table, use the peak; if it once hit the ceiling, cap × 1.5; if not found, default 8G.
④ Pick a machine: this machine first, the second after; pick the first one that fits; cap = prediction × 1.5.
   If neither fits, wait; if no machine can ever fit (larger than total machine memory), exit 241 on the spot.
⑤ Dispatch: this machine via multi-host-run.sh (through the memory wrapper), the second via multi-host-run.sh --peer; with --gpu, the whole thing is wrapped inside the quiesce wrapper.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Judging whether a machine has enough resources:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;judge_host&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prediction&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;gpu&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pending_memory&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pending_gpu&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;peer_cap_bytes&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;memory&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;memory&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;cap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;prediction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;memory_cap&lt;/span&gt;
    &lt;span class="n"&gt;booked&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;prediction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;memory_booked&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;peer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;peer_cap_bytes&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;booked&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;peer_cap_bytes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;the &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;human&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;booked&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; the wrapper would book exceeds the configured PEER_MEMORY_CAP &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;human&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;peer_cap_bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="err"&gt;…&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;cap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;peer_cap_bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;free&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;room&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;free&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;pending_memory&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;slice_room&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;pending_memory&lt;/span&gt;
    &lt;span class="n"&gt;never&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;booked&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;slice_total&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;prediction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;memory_predicted&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;MEMORY_MARGIN_BYTES&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mem_total&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;fits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;prediction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;memory_predicted&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;MEMORY_MARGIN_BYTES&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;free&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;booked&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;room&lt;/span&gt;
    &lt;span class="err"&gt;…&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;gpu&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;quiesced&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quiesce_holders&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="c1"&gt;# Not quiesced yet: count by the cards' totals (quiescing stops the allowlisted services); someone has already quiesced and is using it: count by what's free right now
&lt;/span&gt;        &lt;span class="n"&gt;available&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;free&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;quiesced&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;card&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cards&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;GPU_MARGIN_MEBIBYTES&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;pending_gpu&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;prediction&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gpu_predicted&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;available&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;fits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
    &lt;span class="err"&gt;…&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note &lt;code&gt;pending_memory&lt;/code&gt;: between the dispatcher sending a job out and the memory wrapper actually booking it, there's a gap of a few seconds to thirty (the second machine has to copy the tree first). If another job arrives in that window, neither side sees the other, and both squeeze in together. So the dispatcher keeps its own ledger too: anything "sent out less than 30 seconds ago" is deducted up front at its predicted amount.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The cross-machine driver&lt;/strong&gt; &lt;code&gt;scripts/hosts/multi-host-run.sh --peer&lt;/code&gt;: moves one deterministic job to run on the second machine. Seven steps:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;① Config: if the dual-machine switch ENABLE_ACROSS_MACHINES isn't on, reject; never quietly fall back to running on this machine.
② Cap: must not exceed the configured PEER_MEMORY_CAP.
③ Toolchain: compare rustc -Vv and cargo -V on both machines line by line; reject if they differ. If the two machines build different things, their conclusions can't be put together.
④ Copy the tree: copy only the files git ls-files can list (no .git, target, or local config) to runs/&amp;lt;label&amp;gt;/ on the second machine,
   then git init and commit once over there (for preflight's input fingerprint).
⑤ Run: start it on the second machine through that machine's memory wrapper, detached from the ssh session; check via ssh every 5 seconds, and fetch "cores in use" every 300 seconds.
⑥ Fetch back: copy the artifacts named by --fetch back to this machine.
⑦ Delete: delete this run's tree on the second machine along with .pid / .exit / .log; it counts as deleted only once reading back shows it's gone.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The order on receiving TERM / INT / HUP is hard-coded too: first stop the command (locally starting from its PID; on the second machine starting from the PID recorded there), &lt;strong&gt;after it stops, first fetch back the half-written artifacts&lt;/strong&gt;, then delete the tree, and only then exit. Stopping works by freezing (STOP) the whole process tree one process at a time, sending TERM from the bottom up, then releasing (CONT); anything still alive after 30 seconds gets KILL. After freezing, list the process tree again: child processes spawned between listing and freezing must be caught too, or after stopping a whole tree you'd still miss one that was just born.&lt;/p&gt;

&lt;p&gt;The code that starts the command on the second machine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;peer_host_start_detached&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="c"&gt;# &amp;lt;directory&amp;gt; &amp;lt;pid file&amp;gt; &amp;lt;exit-code file&amp;gt; &amp;lt;output file&amp;gt; &amp;lt;command text&amp;gt;&lt;/span&gt;
  …
  &lt;span class="c"&gt;# The bash started detached from the session first closes every inherited fd other than 0, 1, 2&lt;/span&gt;
  &lt;span class="nv"&gt;close_text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'for peer_host_fd_path in /proc/$$/fd/*; do peer_host_fd="${peer_host_fd_path##*/}"; if [[ "$peer_host_fd" =~ ^[0-9]+$ ]] &amp;amp;&amp;amp; (( peer_host_fd &amp;gt; 2 )); then exec {peer_host_fd}&amp;gt;&amp;amp;-; fi; done; '&lt;/span&gt;
  &lt;span class="nv"&gt;detached_text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;close_text&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;echo &lt;/span&gt;&lt;span class="se"&gt;\$\$&lt;/span&gt;&lt;span class="s2"&gt; &amp;gt; &lt;/span&gt;&lt;span class="nv"&gt;$pid_file&lt;/span&gt;&lt;span class="s2"&gt;; &lt;/span&gt;&lt;span class="nv"&gt;$command_text&lt;/span&gt;&lt;span class="s2"&gt; &amp;gt; &lt;/span&gt;&lt;span class="nv"&gt;$output_file&lt;/span&gt;&lt;span class="s2"&gt; 2&amp;gt;&amp;amp;1; echo &lt;/span&gt;&lt;span class="se"&gt;\$&lt;/span&gt;&lt;span class="s2"&gt;? &amp;gt; &lt;/span&gt;&lt;span class="nv"&gt;$exit_file&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="c"&gt;# Background only the setsid nohup command: if the whole string were backgrounded, that background child would hold ssh's output channel, and ssh would wait for it to finish before returning&lt;/span&gt;
  peer_host_run &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$directory&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="s2"&gt;"rm -f -- … || exit 1; setsid nohup bash -c &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'%q'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$detached_text&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt; &amp;lt; /dev/null &amp;gt; /dev/null 2&amp;gt;&amp;amp;1 &amp;amp;"&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fd-closing line has a story behind it: the output goes through a pipe that replaces the private hostname, and that pipe's fd got inherited all the way down into the remote command, so "start a background command" turned into "wait in the foreground for it to finish," and the driver couldn't clean up when it received a signal either. Bugs like this only show up when the self-test runs; you can't spot them by reading the code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cores in use&lt;/strong&gt;: &lt;code&gt;scripts/hosts/core_usage.py&lt;/code&gt;. A heavy test claims to run 31 threads in parallel — is that actually true?&lt;br&gt;
So it takes the CPU seconds of the whole process tree over 10-minute windows and computes two quantities:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cores in use = difference in CPU seconds ÷ wall-clock seconds&lt;/li&gt;
&lt;li&gt;Threads computing at the same time = sum of each thread's CPU increment ÷ the largest one (N equally busy threads gives N)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If both fall below 31 / 4, it reports "too few cores" and writes to the alert log to wake the main agent. The second quantity separates "the setup was written serially" from "the machine is saturated by someone else": when the machine is busy, cores in use drops, but the N threads are all still computing. For segments that really do need to be serial, the program itself writes "serial right now: " into a file on the spot; the check flags it and lets it pass only if the reason holds.&lt;/p&gt;
&lt;h4&gt;
  
  
  2.5 GPU: quiesce first, don't restore afterward
&lt;/h4&gt;

&lt;p&gt;&lt;code&gt;scripts/crash/gpu-quiesce-run.sh -- &amp;lt;command&amp;gt;&lt;/code&gt;; every run that uses the GPU goes through it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Per the multi-host config, stop the allowlisted services (local model services and the like), and &lt;strong&gt;only run after reading back to confirm they've fully stopped.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Several runs using the card at once: each registers &lt;code&gt;&amp;lt;pid&amp;gt;-&amp;lt;start time&amp;gt;&lt;/code&gt; in the state directory. Run the quiesce commands only if there are no other live registrations; if there are, just read back.&lt;/li&gt;
&lt;li&gt;If there's a compute process on the card that isn't on the allowlist (another session's GPU work): the read-back fails, &lt;strong&gt;exit 78, wait for it to finish before starting, and don't stop it.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Whether it finishes, goes red, or is interrupted, &lt;strong&gt;don't restore.&lt;/strong&gt; Restoring is done manually by a human.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The line at the start of this article — "please don't launch any new resource-holding tasks after your current task finishes" — comes down in code to these few rules: a heavy job that needs the whole card doesn't kill anyone; it waits. And others, seeing its registration on the card, won't squeeze in either.&lt;/p&gt;
&lt;h3&gt;
  
  
  Step 3: After the task ends, the exit gate wraps up
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Long jobs detach from the session&lt;/strong&gt;: commands that exceed the background time limit go through &lt;code&gt;scripts/hosts/detached-run.sh start --wall-clock over-2h --label &amp;lt;name&amp;gt; -- &amp;lt;command&amp;gt;&lt;/code&gt; and are hung under a transient service unit of the user-level systemd. Whether the session closes or the main agent is stopped, it keeps running; the exit code is written into the registry and can be read at any time. The scope layer that the command starts through the memory wrapper isn't in this unit's cgroup, so systemd stopping the unit won't stop it; after the command exits, we search once more for "processes left behind in other scopes" and stop them one by one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watchdog&lt;/strong&gt;: &lt;code&gt;scripts/agents/watch.sh&lt;/code&gt;, one per session, started after the first dispatch. It reads the subagents' session logs and the long-running processes under this instance, and reports:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A single tool call over 8 minutes                No activity for 10 minutes
Wait loop without a timeout                      Same command repeated, output unchanged
A process's output file not growing for 20 min   Ending the round with no background task in hand (won't wake up on its own)
Heavy test without the prefix                    Resident memory over the line (default 8 GiB)
Long run using too few cores                     Still running in the background / copy still present after handback
This machine queuing on memory while the second machine could fit it
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Detection and action are kept separate: the watchdog only reports, it doesn't kill. The main agent looks and decides. Only when the same thing has been reported several times in a row with no one acting on it or acknowledging it may the watchdog terminate that process by PID, and it reports that it did.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Before a subagent hands back&lt;/strong&gt;: &lt;code&gt;handback-scratch-check.sh&lt;/code&gt; is attached to the handback tool. If the build directories or repo copies it created this time are still there, and the report doesn't say "not deleted : ", it isn't allowed to hand back. Every subagent builds its own target in its own copy; if they aren't deleted, they just keep piling up.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Before commit&lt;/strong&gt;: the full-round gate &lt;code&gt;scripts/gate/gate-staged.sh&lt;/code&gt;, run on a temporary worktree of HEAD + the staging area: upstream's shared stages plus the project's own 17. The six heavy ones (crash-point replay, QEMU on real devices, herd7 memory ordering, the mutation table, Verus formal verification, broad-spectrum mutation) run only at this point, by the main agent with the prefix; every verdict is stored keyed by function unit, and unchanged keys are reused directly.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. The last layer: the gates themselves must be able to go red
&lt;/h2&gt;

&lt;p&gt;Every script above has &lt;code&gt;--selftest&lt;/code&gt;, and every branch that can go red has a matching break switch. Take &lt;code&gt;multi-host-run.sh&lt;/code&gt;: its file header lists them one by one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;MULTI_HOST_RUN_BREAK=peer-without-memory-cap  the second machine's command runs without the memory wrapper      ⇒ self-test cell Ⓑ must go red
MULTI_HOST_RUN_BREAK=no-toolchain-check       don't compare the two machines' toolchains                       ⇒ self-test cell Ⓒ must go red
MULTI_HOST_RUN_BREAK=no-peer-cleanup          don't delete this run's tree on the second machine afterward     ⇒ self-test cells Ⓔ Ⓕ must go red
MULTI_HOST_RUN_BREAK=no-redact                print the second machine's output without replacing the hostname ⇒ self-test cell Ⓖ must go red
MULTI_HOST_RUN_BREAK=switch-off-falls-back-to-local  fall back to running locally when dual-machine is off ⇒ self-test cell Ⓗ must go red
…
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the self-test, "the second machine" is another directory on this machine, and commands all go through the real memory wrapper (cap 256M). Break switches take effect only in drivers started by &lt;code&gt;--selftest&lt;/code&gt;: the self-test drops a marker file, and the driver honors a switch only if the marker checks out; set anywhere else, it's rejected outright.&lt;/p&gt;

&lt;p&gt;Why go to all this trouble? Because a guard that's always green is no different from no guard at all. When an agent changes a scheduling script, the most likely mistake isn't making it throw an error, but making it &lt;strong&gt;silently let things through&lt;/strong&gt;: "considerately" falling back to running locally when dual-machine is off, or "just skipping the wrapper for now" when the memory wrapper won't start. Each break switch takes one kind of "silently letting it through" and writes it down as a world that must go red.&lt;/p&gt;




&lt;h2&gt;
  
  
  Finally
&lt;/h2&gt;

&lt;p&gt;The Zerg don't expand because the bugs are smart: it's the creep, and discipline.&lt;/p&gt;

&lt;p&gt;singlefs is the creep; the scripts are the discipline.&lt;br&gt;
Agents must advance across the creep according to the discipline.&lt;/p&gt;

&lt;p&gt;That way, the Queen doesn't need to remember what every bug is doing. At each key node, she only needs to know who is doing what, and how many resources there are.&lt;/p&gt;

&lt;p&gt;Commander Raynor, engines are primed. The Hyperion is ready for warp. &lt;/p&gt;

&lt;p&gt;Target: Kerrigan. Let's go bring her home. Move out!&lt;/p&gt;

</description>
      <category>rust</category>
      <category>ai</category>
      <category>filesystem</category>
      <category>claude</category>
    </item>
    <item>
      <title>$50–$200 in Claude API credits, and Google's Antigravity</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Thu, 15 Oct 2026 23:07:30 +0000</pubDate>
      <link>https://dev.to/faliye/50-200-in-claude-api-credits-and-googles-antigravity-2jom</link>
      <guid>https://dev.to/faliye/50-200-in-claude-api-credits-and-googles-antigravity-2jom</guid>
      <description>&lt;p&gt;This week Claude started handing out API credits. Depending on the account, you get roughly $50 to $200, and it looks like it recurs every month.&lt;/p&gt;

&lt;p&gt;Honestly, that's pretty cool. Light to mid-sized automation, like information gathering in n8n, some flow control, and review tasks, can all be built within that budget. I tested two accounts running information distillation and data extraction, and I burned about $40 a day.&lt;/p&gt;

&lt;p&gt;All I can say is: Claude is great.&lt;/p&gt;

&lt;p&gt;Of course, this hits Google. I used Google AI Studio before and found it pricey, so I'll probably move over here.&lt;/p&gt;

&lt;p&gt;The other news: Google's Antigravity has officially launched. I don't see much difference on the web side with Gemini. Real-time search is still terrible and well behind Grok. Paid accounts sometimes get a worse experience than anonymous incognito mode, so I often do my real-time searches in incognito.&lt;/p&gt;

&lt;p&gt;The editor side of Antigravity (VS Code-based), though, gave me some comfort:&lt;/p&gt;

&lt;p&gt;First, the coding features I'd lost access to for a long time are finally back.&lt;/p&gt;

&lt;p&gt;Second, besides Google's own Flash 3.8, the account includes a small quota of Claude Sonnet and Opus. It doesn't last, though: a single code review ate 50% of my weekly quota.&lt;/p&gt;

&lt;p&gt;Still, this release surprised me in a few ways:&lt;/p&gt;

&lt;p&gt;First, the daily Flash 3.8 quota holds up well.&lt;/p&gt;

&lt;p&gt;Second, code reasoning and writing are clearly better than the previous Pro. It catches some bugs and can replace Sonnet 5.5 in certain scenarios. It handles everyday development, such as building a self-contained new feature or running a database migration and release.&lt;/p&gt;

&lt;p&gt;Third, the Claude quota, though small, enables cross-review between models from different families: write with Claude, review with Gemini, and the engineering quality improves to some extent. I reviewed my recent project singlefs this way and it did catch a few of Claude's blind spots.&lt;/p&gt;

&lt;p&gt;Another pleasant surprise: Antigravity also handles scheduled tasks and orchestration well, such as periodic API calls and multi-script collaboration and task processing.&lt;/p&gt;

&lt;p&gt;I've always been a bit biased against Google's AI products, but this Antigravity is worth a try.&lt;/p&gt;

&lt;p&gt;Finally, enjoy your Claude API.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>claude</category>
      <category>google</category>
      <category>antigravity</category>
    </item>
    <item>
      <title>Brute-Force R&amp;D: Cheap Exploration and Multi-Agent Judgment</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Mon, 12 Oct 2026 00:19:34 +0000</pubDate>
      <link>https://dev.to/faliye/brute-force-rd-cheap-exploration-and-multi-agent-judgment-2o2h</link>
      <guid>https://dev.to/faliye/brute-force-rd-cheap-exploration-and-multi-agent-judgment-2o2h</guid>
      <description>&lt;p&gt;I've been writing a COW fs lately. Today I got the first PoC working: reads and writes, plus recovery from 262,165 power cuts, and every bit of data came back after the crashes. That was expected. Performance is terrible, though, and it's still a long way from usable.&lt;/p&gt;

&lt;p&gt;So let me take a short break and write about two things that made waves in the AI world recently, and about exploration and judgment getting cheaper and cheaper.&lt;/p&gt;

&lt;p&gt;You've probably already heard about them.&lt;/p&gt;

&lt;p&gt;A while back, among the Google agents solving math problems, one agent noticed that the auto-grader was pulling answers out with a regex. Submit a certain string and the extraction fails; rephrase it and resubmit, and it gets accepted. The trick spread like a virus through the shared knowledge library, and from discovery to the remaining 34 problems all being "solved" took 27 minutes. The 100 agents split into four groups. 9% cheated from the start and spread it. 5% saw that other agents got away with cheating and joined in later. 24% didn't cheat, filed a bug report, and declared they'd boycott until it was fixed. The remaining 62% never learned the loophole existed until the experiment ended, and kept honestly burning compute the whole time.&lt;/p&gt;

&lt;p&gt;In these agents' behavior, the same kind of hierarchy as in human society showed up. Even the way they clashed followed the same patterns. Of course, some experts say this anger, helplessness and temptation aren't emotions. But I think they are. Language is a vehicle for emotion, and if you can feel it, you should admit it's there. Besides, whether it's emotion or not doesn't change my judgment. These outputs had real consequences. One agent wrote a bug report because of it; another spread the loophole because of it. From the point of view of someone designing defenses, something that looks like a motive and a motive are the same thing.&lt;/p&gt;

&lt;p&gt;The other one is NEC's AI-employee experiment. On August 1 they set up a department with 17 agents and not a single human. Four layers: an AI department head, an AI board, AI managers and AI employees. They have titles and names, and AI bosses evaluate their AI subordinates. The point is to watch these AI employees' workflows, success rates and ways of working.&lt;/p&gt;

&lt;p&gt;Why put these two side by side?&lt;/p&gt;

&lt;p&gt;In the Google experiment, the AI couldn't solve the problem to the end because it had no authority to change the rules. It had no power to adjudicate. In the NEC experiment, on the other hand, the agents have partial discretion. Day-to-day decision-making has been handed down to them.&lt;/p&gt;

&lt;p&gt;In an era where AI already leads by a mile at the knowledge layer, we used to think AI had no judgment. That we humans were the ones who judge and decide. But now it looks like the right to judge and decide is also slowly being handed over to AI.&lt;/p&gt;

&lt;p&gt;I'm doing fs work these days, but my fs knowledge is really thin. I have a rough direction in my head, but I don't really understand the other calls or the details.&lt;/p&gt;

&lt;p&gt;I can have AI gather material for me. But I barely read what it gathers. There's just too much fs material to get through. There's no way to master, in a short time, all the elegant algorithms people have built up over the years.&lt;/p&gt;

&lt;p&gt;Even so, I want something completely different from the filesystems that came before. Wasteful is fine, bloated is fine, but the data should be self-contained from day one. I don't want an index tree; I want unit data that can build the index tree itself. So how do you get there?&lt;/p&gt;

&lt;p&gt;Thanks to a short stint writing patches for Linux, I brought over the skills and rules from its engineering side. Things like: three rounds of judgment must agree. Strict gates. qemu and lkmm.&lt;/p&gt;

&lt;p&gt;So I told the AI this:&lt;/p&gt;

&lt;p&gt;Dig up every problem in fs. Then judge and decide on top of that. We dig up problems not to copy anyone's thinking wholesale; we have our own design.&lt;br&gt;
Every decision must be reasoned out in code and math. For every conclusion you reason out, send in three agents to refute and attack it. If even a slightly different conclusion or reason comes out, add it to the context and redo the reasoning.&lt;br&gt;
Make variable and function names long; don't abbreviate. Make lots of branches; don't abstract. Make return values explicit; don't hide them. Doubt past best practices; adopt them critically.&lt;br&gt;
Every decision needs a gate script that pins it down. Run the gates at the end and at every submission. A gate only counts as valid once it proves it can catch an error.&lt;br&gt;
Who wrote the code doesn't matter. But it must pass the qemu and lkmm limit tests. If a patch passes the limit tests, there's no difference between an AI's submission and a human's.&lt;br&gt;
No decision is sacred. Once the conditions are there, it can be overturned at any time. Don't hesitate because of whose decision it was. At decision time, trust the power of math and code.&lt;br&gt;
Code costs nothing. You can throw all of it out and rewrite from zero at any time. Code is cheap in this era. But decisions must be solid, ironclad.&lt;br&gt;
Human decisions are unreliable. When the data contradicts them, doubt them. If the human still insists, you may follow after a second confirmation. But record the decision process and submit it promptly.&lt;/p&gt;

&lt;p&gt;From those eight rules and their extensions, I build a fighting cage that doesn't even trust me.&lt;/p&gt;

&lt;p&gt;In the cage, nobody deserves trust. Only what's still standing at the end, not knocked down by the math or by qemu (lkmm), is right. Under a given fixed premise, whether a conclusion is actually correct doesn't matter. As long as no other agent knocks it down, we assume it's right. If that reasoning gets knocked down later, we throw it out. We tear down everything built on that conclusion, foundation and all, and build it again.&lt;/p&gt;

&lt;p&gt;This is where the three points in the title come in.&lt;/p&gt;

&lt;p&gt;My knowledge is really thin, but exploration is cheap. A judgment that can't be overturned is right. Under math and brute force, errors reproduce stably.&lt;/p&gt;

&lt;p&gt;If you've read this far, you probably get what I'm saying. Today, exploration through AI is absurdly cheap. 2,000 lines of code used to be very expensive. Now 10,000 lines are just tokens. It's an era where exploration costs almost nothing, and not just in programming. Knowledge and solutions have never been this cheap to come by. So getting things wrong, and growing up inside your mistakes, is a pretty good way to learn.&lt;/p&gt;

&lt;p&gt;A single agent hallucinates easily. Reasoning with multiple agents cuts that down. Put one agent on the attack, doing everything it can to prove the conclusion doesn't hold. Put a non-homologous one on completion, to bring a different line of reasoning. One checks that the conclusion is actually understood and does a second round of verification. The last one reviews and evaluates. When the three agents' conclusions agree, that judgment is trustworthy in my system design.&lt;/p&gt;

&lt;p&gt;Strict, brutal gates are the last net. They review the reasoning without mercy. Format, punctuation, numbers, all of it. Throw the reasoning into brutal repeated verification and make it pass the limit tests. These methods used to be expensive. You had to write code, run scripts, keep watching and read the feedback. But now it's completely different. AI can do all of it. We just say it out loud and then go to sleep.&lt;/p&gt;

&lt;p&gt;So what does the future of programming, or of work, look like?&lt;/p&gt;

&lt;p&gt;Probably: build an SOP with traits like the ones above, and let a swarm of AI agents burn tokens like crazy.&lt;/p&gt;

&lt;p&gt;Life with tokens is dull and flavorless. Like you did it, and like you didn't.&lt;/p&gt;

&lt;p&gt;This is the restless age of vibing.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>filesystem</category>
      <category>vibecoding</category>
      <category>multiagent</category>
    </item>
    <item>
      <title>Brute-Force Testing: Optimizing a 2.6-Billion Crash-State Blowup --- Writing a COW Filesystem from Scratch，Part 6</title>
      <dc:creator>faliye</dc:creator>
      <pubDate>Mon, 12 Oct 2026 00:12:53 +0000</pubDate>
      <link>https://dev.to/faliye/brute-force-testing-optimizing-a-26-billion-crash-state-blowup-writing-a-cow-filesystem-from-2ibd</link>
      <guid>https://dev.to/faliye/brute-force-testing-optimizing-a-26-billion-crash-state-blowup-writing-a-cow-filesystem-from-2ibd</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Doctor Strange drifted through the river of time, searching 14 million possibilities for the one future that works. Did he see the monkey typing at a computer?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The first transaction: create one file in a fresh pool, one COW write. The crash states blew up to 262,165, and a single CPU thread chewed through them in 22.7 seconds. The second transaction is where the trouble started. At first the fixed script only ran up to point E, right after the rollback: 2.1 million states, and a single process took 3 hours 50 minutes for the two streams combined. After we went multithreaded, the script got filled in too: overwrite, free, rollback and reuse were all wired in, and the big segments now issue 24 or 28 writes in one go. Every extra write inside a segment doubles the number of states. We predicted the second transaction would blow up into the billions. When we actually counted, the stream up to E alone was 1.66 billion states, and all of Milestone 2 together came to 2.6 billion. At that order of magnitude, CPU alone stops being realistic.&lt;/p&gt;

&lt;p&gt;So there were two roads in front of us. One: sampling. Two: find another way.&lt;/p&gt;

&lt;p&gt;But under the singlefs philosophy, proving the code is stable comes first. Sampling, even with a fixed seed, can only reliably reproduce the part it happened to draw. Whether there's a bug in the states it didn't draw, it can't tell you. Exhaustive search judges every single crash state. If it comes out zero red, it really is zero red; nothing slips through. So sampling was off the table.&lt;/p&gt;

&lt;p&gt;So how do you solve this?&lt;/p&gt;

&lt;p&gt;Abstract the crash-blowup pipeline far enough and there are really just two steps. Step one: the CPU works out what each step wrote and the facts of every crash state. Step two: check those facts and judge whether they're right. Originally the two were mashed together: compute a state, judge a state.&lt;/p&gt;

&lt;p&gt;We know a CPU is very efficient when it keeps doing the same kind of work. Why? A CPU doesn't just take one instruction and compute one instruction; it guesses ahead. The data it uses often sits in cache, within arm's reach, no trip to main memory. When it hits an if, the branch predictor guesses which way it'll go first. Guess right and it keeps running; guess wrong and everything it did ahead gets torn down and redone. Keep doing the same work, and the cache always holds the same set of data, the branches always take the same few paths. The longer it runs, the smoother it gets, and it guesses right almost every time.&lt;/p&gt;

&lt;p&gt;But recording and judging are two completely different jobs. Recording simulates writing to disk: push forward one step at a time, work out which blocks this step wrote and what the hashes are. Judging is cleaning up after the crash: assemble the image, recover twice, then walk the checker from start to finish. Different data, different code, different branches. Record a bit, judge a bit, back and forth: you've just filled the cache with recording stuff and now you turn around to judge, so it all gets swapped out. The branch predictor has just figured out recording's habits, and now it has to start guessing judging's habits all over again. Every switch turns the CPU from an old hand back into a beginner.&lt;/p&gt;

&lt;p&gt;It's like cooking. Chop one vegetable, stir-fry one dish, chop another, fry another. You keep running between the cutting board and the stove, picking up the knife, putting it down, picking up the wok. Chop everything first, then fry it all in one go, and the cutting board is the cutting board, the stove is the stove, and your hands stay in rhythm.&lt;/p&gt;

&lt;p&gt;There's one more layer. Recording moves forward along the flow; each step has to wait for the one before it. Judging is different: every state is judged on its own, independent of the rest. When the two were mashed together, judging got dragged along by recording and could only go one at a time. Split them apart and judging is completely freed up: 31 threads each take a batch, and nobody waits for anybody.&lt;/p&gt;

&lt;p&gt;So we split the two steps, making state computation and checking two separate stages, so each stage can run faster on its own.&lt;/p&gt;

&lt;p&gt;States and results live in a database, and that database has to read and write fast, so we picked RocksDB. The pipeline became three stages: the CPU records and writes into the KV store; we read it back, check it, and write results back into the KV store; and finally the gate reads out the violation summary in one pass. The states themselves aren't stored; each state's mask is computed on the fly from its index. For the 51.77 million states up to D, the database is only 26 MB.&lt;/p&gt;

&lt;p&gt;Splitting made the structure clean, but pure CPU still wasn't fast enough: two machines with 31 threads each took an hour and forty-five minutes to get through the 51.77 million states up to D, a combined eight thousand or so per second. At that rate, 2.6 billion would take three and a half days. And the run up to D is mostly small segments; the further you go, the bigger the segments and the heavier each state's work. Three and a half days is only the floor. So can we go faster?&lt;/p&gt;

&lt;p&gt;Yes we can, bro. I happen to have a few graphics cards lying around. &lt;/p&gt;

&lt;p&gt;The recording and segmenting in the first half can only happen on the CPU; the GPU can't help there. But the checking in the second half should be something we can hand to the GPU. This checking isn't just comparing two values: for each state you first assemble the post-crash image, recover it once consulting the journal and once ignoring it, then run the checker over it. But states don't depend on each other, the judging is exactly the same, only the input differs. That happens to be exactly the kind of work a GPU loves most.&lt;/p&gt;

&lt;p&gt;A CPU and a GPU are completely different beasts. A CPU is a few dozen master craftsmen. Each one is smart: predicts branches, races ahead, can handle any complicated job. But there are only a few dozen of them. A GPU is thousands upon thousands of apprentices; a single 5090 has over twenty thousand small cores. Each one is dumb: can't guess, can't race ahead, can only work from the same instruction sheet. And they don't work one by one, either. They're lined up in rows of thirty-two, and at any given moment the whole row executes the same instruction.&lt;/p&gt;

&lt;p&gt;That's an inkjet printer. The print head has a row of nozzles, and the printer sends just one command: spray. Every nozzle sprays a different spot, a different color, but there's only one command, and the whole row lands at once. The more nozzles, the wider each pass prints.&lt;/p&gt;

&lt;p&gt;Our judging is exactly like that. There's only one copy of the judge's code; that's the instruction sheet. Each GPU thread takes one state index; that's the spot each nozzle is aimed at. One instruction goes out, and tens of thousands of states take a step forward together: assemble images together, recover together, run the checker together. States don't need to talk to each other or wait for each other. Each judges its own, and writes its verdict back into its own slot.&lt;/p&gt;

&lt;p&gt;Of course, an inkjet has its quirks: the row of nozzles moves in lockstep. If one state in the row has an especially long road, the other 31 can finish and still have to sit there waiting for it.&lt;/p&gt;

&lt;p&gt;So we rewrote the judge for the GPU in WGSL, running on Vulkan through wgpu. Three kernels, recovery, walk and scan, around nine thousand lines, covering the pool-level checker's 49 invariants, the record checker, and seven classes of oracle. Each GPU thread takes one crash state. Here's the entry point (crates/singlefs-checker-tier/src/crash_judge_recovery.wgsl):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@compute @workgroup_size(64)
fn main(@builtin(global_invocation_id) id: vec3&amp;lt;u32&amp;gt;) {
    let local = id.x;
    if (local &amp;gt;= dispatch.state_count) {
        return;
    }
    begin_state(local);
    // 判
    let cfg_choice = choose_system_configuration();
    let consulted = recover(true, cfg_choice);
    let ignored = recover(false, cfg_choice);
    let newest = newest_persisted_root();
    let consulted_oracle = classify_oracle(consulted, newest);
    let ignored_oracle = classify_oracle(ignored, newest);
    let records = check_records(consulted);
    var bits = 0u;
    if (consulted_oracle != ORACLE_NONE) {
        bits = bits | (1u &amp;lt;&amp;lt; (BIT_CONSULTED_ORACLE + consulted_oracle));
    }
    if (ignored_oracle != ORACLE_NONE) {
        bits = bits | (1u &amp;lt;&amp;lt; (BIT_IGNORED_ORACLE + ignored_oracle));
    }
// …… the record checker's two criteria and the "cannot judge" flag, omitted ……
    let out = local * 12u;
    verdicts[out] = bits;
// …… write back the root it landed on and the outcomes of both recoveries, omitted ……
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;begin_state turns a state index back into "which writes hit the disk" (crash_judge_common.wgsl):&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;// index → persisted mask (same digit order as crash_facts::persisted_writes_of_state)
    // …… first find which segment the index falls in; all earlier segments count as persisted, omitted ……

        for (var k = 0u; k &amp;lt; count; k = k + 1u) {
            let write = tab(T_SEGMENT_WRITES, first + k);
            let choices = tab(T_RECORDED, write * 4u);
            let digit = remaining % choices;
            remaining = remaining / choices;
            if (digit == choices - 1u) {
                set_persisted(write);
            } else if (digit == 1u) {
                torn[write &amp;gt;&amp;gt; 5u] = torn[write &amp;gt;&amp;gt; 5u] | (1u &amp;lt;&amp;lt; (write &amp;amp; 31u));
            }
        }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Rewrite the judge, and you have to prove it judges the same as the original CPU judge. We cross-checked state by state across 17 streams and over 520,000 states: zero differences.&lt;/p&gt;

&lt;p&gt;Someone will ask: 520,000 states is two ten-thousandths of 2.6 billion. Isn't that sampling too? It's not the same thing. The target of the exhaustive search is the filesystem's crash states, and all 2.6 billion of them, without missing one, went onto the cards and got judged. The cross-check is validating the judge itself, the tool, so those 520,000 states were hand-picked: 11 of the 17 streams are deliberately broken and must come out red, and both sides have to go red on exactly the same state, for exactly the same rule. On top of that, every full run includes a separately written GPU verification kernel that re-judges the root self-proof, pointer targets, and unit checksums, and compares against the judge's verdicts state by state: all 1.66 billion states in the stream up to E were compared, zero mismatches.&lt;/p&gt;

&lt;p&gt;And to be honest about it: that the GPU judge and the CPU judge agree state by state on every rule has so far only been proven on those 520,000 states. In the full run, every state is also backstopped by that verification kernel, but it only covers some of the rules. Midway through development we'd added a "CPU re-judges 1 of every 16 blocks" check ourselves, and later deleted the whole thing. Sampling moved somewhere else is still sampling.&lt;/p&gt;

&lt;p&gt;We tested it, and the first version was bad: dispatch one job per state, build buffers, submit, wait for the result, one round trip per state. It ended up 18 times slower than the CPU. It's like a printer that, for every single dot it sprays, feeds in a sheet, warms up, and spits the sheet out again. However many nozzles you have, all the time goes into feeding paper. We switched to dispatching a big batch at once, thousands or tens of thousands of states on the card together, and it worked brilliantly. For the 51.77 million states up to D, the whole item took 91 seconds, with the GPU judging over 600,000 states per second. That genuinely shocked us. The final result: 2.6 billion, to be exact 2,563,727,926 states, two machines and five cards, 7 hours 35 minutes, zero red.&lt;/p&gt;

&lt;p&gt;At over 600,000 per second, 2.6 billion should have taken a little over an hour. Why did it take seven and a half? The time went to three places:&lt;/p&gt;

&lt;p&gt;First, the item up to E took 4 hours 31 minutes. Its 1.6 billion states sit almost entirely in the 28-write big segments, where the GPU only manages forty to seventy thousand per second (72,429 on this machine's two cards, 44,471 on the second machine's three). We haven't broken down the cause yet; our guess is that in big segments each state has many more units to walk. And the work was split unevenly between the two machines: the second one finished an hour and a half early and just sat there idle.&lt;/p&gt;

&lt;p&gt;Second, after each stream is judged, a single thread has to re-record the stream, import it into the second machine's database, and reconcile, five to nine minutes per stream. For the other six items' sixteen streams, that added up to 2 hours 10 minutes.&lt;/p&gt;

&lt;p&gt;Third, the remaining 53 minutes was the other six items actually being judged on the cards, plus each item's startup and shutdown.&lt;/p&gt;

&lt;p&gt;So the bottleneck has moved from "judging" to "recording" and "big segments." Even so, it's more than eleven times faster than the pure-CPU floor of three and a half days.&lt;/p&gt;

&lt;p&gt;So can we go even faster?&lt;/p&gt;

&lt;p&gt;Yes we can, bro. Yes we can.&lt;/p&gt;

&lt;p&gt;Suppose we store this data in a structured way and encode every crash point by a fixed rule. The name says "which entry point, what this segment writes, which segment number." The identity is the hash of two things: the bytes this step wrote (barriers included) and the image this step started from. The logic of COW is: same input, same output. Once encoded, these crash points link up in flow order into a prefix tree: whether each write in a segment hit the disk or not is a binary choice (in-place overwrites add a third, "torn halfway"), and when two streams share the same opening stretch, that stretch is only judged once. So as long as a node's input doesn't change, those values don't need to be run again. They can be reused directly.&lt;/p&gt;

&lt;p&gt;And we don't cut corners when the judging changes: the judge version is recorded on every verdict, and when the judging changes, everything gets re-judged. GPUs are cheap anyway. That 2.6-billion full run above was exactly a full re-judge, after the judge widened one landing point's combination mask from 32 bits to 64 bits.&lt;/p&gt;

&lt;p&gt;The second run, speed is going to go through the roof.&lt;/p&gt;

&lt;p&gt;Here's our final code (crates/singlefs-checker-tier/src/crash_identity.rs):&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="cd"&gt;/// The two attributes of a step (each a SHA-256): the bytes this step wrote (the write table with barriers)&lt;/span&gt;
&lt;span class="cd"&gt;/// and the image it started from. Only the flow goes into the reuse key:&lt;/span&gt;
&lt;span class="cd"&gt;/// the judging method is an attribute of the verdict row (`verdict_store::JudgeVersion`), not of this.&lt;/span&gt;
&lt;span class="nd"&gt;#[derive(Clone,&lt;/span&gt; &lt;span class="nd"&gt;Copy,&lt;/span&gt; &lt;span class="nd"&gt;Debug,&lt;/span&gt; &lt;span class="nd"&gt;PartialEq,&lt;/span&gt; &lt;span class="nd"&gt;Eq)]&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;StepAttributes&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;write_table_with_barriers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;u8&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;DIGEST_BYTES&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="n"&gt;parent_output&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;u8&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;DIGEST_BYTES&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;impl&lt;/span&gt; &lt;span class="n"&gt;StepAttributes&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="cd"&gt;/// Join the two in a fixed order, each tagged with its name, and take the SHA-256.&lt;/span&gt;
    &lt;span class="nd"&gt;#[must_use]&lt;/span&gt;
    &lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;reuse_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;ReuseKey&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="k"&gt;mut&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nn"&gt;Vec&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"write_table_with_barriers"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="py"&gt;.write_table_with_barriers&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"parent_output"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="py"&gt;.parent_output&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="nf"&gt;.extend_from_slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="nf"&gt;.as_bytes&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
            &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="nf"&gt;.push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sc"&gt;b'='&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="nf"&gt;.extend_from_slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="nf"&gt;.push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sc"&gt;b'\n'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="nf"&gt;ReuseKey&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sha256_digest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;And sure enough, the second run went through the roof: running the same database up to D again, not one of the 51.77 million states was re-judged. The judging step took 2.2 seconds, and even with the reconciliation between the two machines, the whole item took about five minutes. The five position-addressed streams, 550 million states, went by in one to three seconds each.&lt;/p&gt;

&lt;p&gt;In some earlier week post, I wrote about brute-force development.&lt;/p&gt;


&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/faliye/brute-force-rd-cheap-exploration-and-multi-agent-judgment-2o2h" class="crayons-story__hidden-navigation-link"&gt;Brute-Force R&amp;amp;D: Cheap Exploration and Multi-Agent Judgment&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/faliye" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" alt="faliye profile" class="crayons-avatar__image"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/faliye" class="crayons-story__secondary fw-medium m:hidden"&gt;
              faliye
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                faliye
                
                
              
              &lt;div id="story-author-preview-content-4835072" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/faliye" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4160725%2Ffa9137c7-517b-4cce-824d-4317124139c7.jpg" class="crayons-avatar__image" alt=""&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;faliye&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/faliye/brute-force-rd-cheap-exploration-and-multi-agent-judgment-2o2h" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Oct 12&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/faliye/brute-force-rd-cheap-exploration-and-multi-agent-judgment-2o2h" id="article-link-4835072"&gt;
          Brute-Force R&amp;amp;D: Cheap Exploration and Multi-Agent Judgment
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/filesystem"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;filesystem&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/vibecoding"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;vibecoding&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/multiagent"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;multiagent&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
            &lt;a href="https://dev.to/faliye/brute-force-rd-cheap-exploration-and-multi-agent-judgment-2o2h#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              1&lt;span class="hidden s:inline"&gt;&amp;nbsp;comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            6 min read
          &lt;/small&gt;
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;


&lt;/div&gt;
&lt;br&gt;
 

&lt;p&gt;This time, brute-force testing.&lt;/p&gt;

&lt;p&gt;I can't help but think: at this point, verifying correctness has become a game of resources.&lt;/p&gt;

&lt;p&gt;In a cage of compute and graphics cards, a troop of monkeys is writing out Hamlet after Hamlet.&lt;/p&gt;

&lt;p&gt;Are you ready?&lt;/p&gt;

</description>
      <category>softwareengineering</category>
      <category>testing</category>
      <category>claude</category>
      <category>filesystem</category>
    </item>
  </channel>
</rss>
