From late June into July 2026, I rebuilt the execution backbone of my development onto my own local large language models (LLMs). The trigger was cost.
The more you hand development over to AI, the more every task-routing decision and every actual code generation gets sent to a cloud AI, each and every time. Run it 24 hours a day without stopping, and that usage-based billing piles up every month, exactly in proportion to how much you used. And this is something you keep paying, month after month, forever.
But, I thought. If I buy the hardware once and shift the execution beyond that point onto my own local LLMs, couldn't I erase most of this ongoing cost? Couldn't I replace usage-based billing — which grows with every bit you use — with a one-time hardware cost?
Here is what I did. I moved the task-routing decisions (which I'll call the "orchestrator" in this article) and much of the hands-on work from the cloud AI to my own local LLMs. To do that, I bought one NVIDIA DGX Spark and combined it with the four Macs I already had to build an execution backbone that development tasks flow through. Leave the hands-on work to the AI, and keep only the decisions for the human. Push that division far enough and you arrive at Human-Out-Of-The-Loop (HOOTL), where the human steps outside the loop.
Let me say this up front. This is not a bragging-rights story about "I cut ¥X." What I want to convey here is not a proof of some dollar amount, but the way you wire things so you erase as much of the recurring cost — the usage-based billing — as the design lets you erase.
Let me put the conclusion first (reading the body takes about 8 minutes).
- Erased as much of the recurring billing as the design allowed — took both the decisions and the hands-on work locally first, and sent to the cloud only what genuinely needed it
- Locked the buy/no-buy criteria with an ADR (a record of design decisions) before purchasing — ruled out "buy it because I want it," and decided the purchase on measured speed alone
- Decided which model goes where, per role, from measured speed and cost — routing decisions on 14B, hands-on code generation on the zero-billing 72B lane
- Ran one model per host — avoided swap costs and made the split between local and cloud visible in numbers
When you actually do this replacement, the things to decide narrow down to three. ① Whether to buy your own hardware, and if so, which. ② Which model to put on which job. ③ How to line up multiple machines. I'll write them in order.
The period in question breaks down like this. The purchase decision itself was an immediate call from the benchmark results alone (late June 2026). From there the DGX Spark arrived (July 10), and it took about two weeks to get the basic form working. After that, in late July, an incident occurs. From the purchase decision to here is about one month of records.
Before buying: to erase the recurring billing, I first decided "whether to buy my own hardware" by measurement
Why did I want to shift the decisions onto my own local LLMs in the first place? There are two reasons. One is the recurring cost. The other is that I don't want the decision-making itself to depend on external billing.
The routing decision is the central process that runs 24 hours a day. Leave this to the cloud, and the cost of the central process piles up every month in proportion to use, and if it stops for reasons on the external service's side, all of development stops with it. So I put no cloud LLM on the orchestrator. This is a design principle of local-commander (that's the name of this development orchestrator).
Cloud costs stop development in two forms. On a free tier you hit the usage limit and stop; on usage-based billing it piles up without limit, in proportion to use. Neither is a fantasy for us. Our organization's GitHub Actions stopped from April 2026 on suspicion of hitting the free-tier ceiling.
I ran the numbers, too. Running CI on GitHub Actions' standard runners (Linux, 2 cores, $0.008/min) for two machines' worth, even estimating utilization at half (12 hours a day), comes to about ¥55,000/month. Against that, a CI server is ¥130,000 each, ¥260,000 for two, one time. It pays for itself in about five months, and beyond that there is no recurring billing.
That said, shifting the decisions local requires speed. Running a 31B-class model on the MacBook Pro (M3) at hand gives an effective 5 tok/s, 60–200 seconds per decision. That doesn't reach the speed needed for 24-hour autonomous operation.
Here, so that "I want a DGX Spark" as a want-it impulse wouldn't come first, I bound myself before buying with an ADR (Architecture Decision Record — a document that records design decisions).
ADR-0003: Gate the DGX purchase decision on PoC (proof-of-concept) benchmark results only
On top of that, I ran a 16-case classification benchmark. The results are these.
- The system's final classification matched 16 of 16 (100%), with 0 dangerous-side misclassifications (qwen3.6-35b)
- But this 100% does not mean "the LLM alone is perfect." The raw LLM's JSON-formatting success was 69% (31% fallback), and broken output was absorbed by deterministic safety overrides on the code side, lifting the final decision to 100%. The 100% is not the model's cleverness but the result of a mechanism that fails toward the safe side even when it breaks
- The only bottleneck is speed (60–200 seconds per decision on M3)
The correctness and safety of the final decision can be guaranteed by safety rules on the code side even when the LLM breaks. What's missing is only speed, and speed can be solved with hardware. The day after this measurement lined up, I decided to buy.
This connects to the recurring-cost story. Hardware is done once you buy it. The execution beyond that carries no per-token billing. Replace the monthly usage-based billing with a one-time expense. I positioned it as a purchase for exactly that. The PoC report and ADR I used for the decision are all kept in the repository.
The day it arrived: the first thing I did after ssh
2026-07-10, the DGX Spark went live. This is the first session where I ssh dgx'd in.
aarch64 on GNU/Linux 6.17, 3.67TB of storage. From the day it arrived, I'd already started running model-selection benchmarks
The terminal and dashboard images in this article have only the text — product names, hostnames, IPs, and so on — replaced with anonymous dummies from the actual screens (layout and numbers are real data; only verified numbers are shown). Some screens are anonymized diagrams with the layout reconstructed.
I bought this machine intending to place it at the center of the execution backbone that replaces the monthly usage-based billing. So from day one, I began setup on the premise of running local-commander (the first thing I placed was a marker called .lc-bootstrap-allow that permits its initial startup).
Selection: "bigger models are smarter" did not hold for classification
The DGX has 121GiB of memory. It loads a 72B class with room to spare. Naturally the urge comes up: "I want the biggest one as the orchestrator." Here too I chose by measurement. Here is the result of comparing 14B / 32B / 72B on the same cases for the orchestrator's job (task classification).
| Model | Classification accuracy | Time per decision |
|---|---|---|
| qwen2.5-coder:14b | 100% | 4.4 s |
| qwen2.5-coder:32b | 88% | (intermediate) |
| qwen2.5:72b | 100% | 22.8 s |
14B and 72B tie on accuracy, and 72B is 5× slower. And only the intermediate 32B drops to 88% from broken JSON formatting — a counterintuitive result. For a task like classification, "bigger is smarter" does not hold. So I decided on 14B for the orchestrator.
So is 72B wasted? No. I turned 72B into the "zero-billing work lane" that runs the hands-on code generation. This is where the recurring-cost story bites. Throwing the hands-on work at the cloud incurs usage-based billing, but running it on the 72B already loaded on the DGX incurs no per-token billing.
Just to be safe, though — I put 72B on the hands-on work not because "72B is smarter at code generation." In the generation benchmark, 14B and 72B tie at a 57% success rate, and 72B is about 6× slower, and even the primary source concludes "promotion to 72B is shelved." I still wired up the 72B lane because I can reuse an asset already loaded on the hardware at zero per-token billing. The reason I put 72B on the work even though 14B is faster is that the four 14B Macs are always full with classification and have no spare, while the DGX has 72B loaded from the start (how these hosts are split is the next section). I limit its use to low-risk tasks classified as local-ok (judged able to complete safely on local), and if it fails, escalate to the cloud (Claude). On that premise, 72B's slowness is acceptable, and I gain one more lane that carries no billing.
Configuration: the DGX is a 72B-only machine, the Macs are 14B classifiers
The final configuration is one model, one host.
-
DGX Spark: a qwen2.5:72b (111.4GB) dedicated machine. As a "work lane" that carries no per-token billing, it processes low-risk (
local-ok) code-implementation tasks 6 in parallel, and escalates to the cloud on failure - Four Macs (M4 / M3 and others): a pool of qwen2.5-coder:14b classifiers, bundled with a weighted load balancer (a mechanism that distributes load by weight). Apple Silicon, with its GPU and unified memory, is inherently suited to LLM inference. On top of that they double as CI runners, saving the pricey cloud-build (macOS runner) billing for iPhone apps. They pull many roles, so if I was going to buy, they were the optimal choice
- Cloud (Claude / Codex): only tasks that don't complete locally flow through, passing a human approval gate
- CI server: separately from the above, I've also introduced a CI server to keep the orchestrator (local-commander) from stopping (the cost estimate is above). Availability (HA) and configuration details I'll write in a separate article
The aim of this split is to flow to the cloud, which carries recurring billing, only the tasks that truly need the cloud. Both routing decisions and hands-on work are taken locally first. Only the part that doesn't complete locally goes out to the cloud.
Seen on the admin screen, it looks like this.
Execution lanes: the cloud (Claude) is idle, classification 14B has all four Macs busy, and the work 72B (DGX) is also idle. Only the routing decisions run without pause
The reason for not loading multiple models on one machine is simple: swapping models (load/unload) is the most expensive part. Measuring tok/s continuously with Grafana (a tool that visualizes measured results) gives DGX 24.27 tok/s, M4 16.09 tok/s. Watching these numbers, I'm growing an allocation table that decides "which task flows to which lane" by win rate and cost.
Operation: 24 hours seen on the admin screen
Once the configuration takes shape, development tasks flow into the inbox (INBOX), get routed, are executed on each lane, become PRs (pull requests), pass multiple review gates, and wait for human approval. This flow starts running 24 hours a day.
One day's execution state: decisions-pending pile up, and queues for several products line up side by side. In the last 24 hours: 2.37 million tokens, 102 jobs, about $15.76 by reference conversion
In the last 24 hours, 2.37 million tokens, 102 jobs. The $15.76 shown on the admin screen is a reference-conversion amount for 24 hours computed from a registered rate table (token-rates.json), not the cloud's actual invoiced amount itself (when models with unknown rates are mixed in, it becomes a lower bound from the known portion only).
What matters is that this $15.76 carries none of what ran on the 72B lane (DGX). Recurring billing applies only to tasks sent to the cloud. The portion taken locally carries no per-token billing.
Of course, the DGX's hardware cost and electricity cost apply separately. It isn't running for zero yen. Still, I was able to replace the usage-based billing that grows every month in proportion to use with a one-time hardware cost and a fixed electricity cost. That the split became visible in numbers is the biggest gain from building this backbone.
It wasn't all smooth sailing
It's not only a clean story. In late July, the classification cluster stopped in a chain. Retries concentrated on a bloated classification prompt (4,251 tokens), and one node's four slots all fell into a 37-second serial queue and went silent. Before reaching the true cause, I misdiagnosed for 3 days from misreading the dashboard. The fixes were co-primary'ing (distributing traffic 50/50 across two nodes) and bounding the prompt (capping its size).
This was not an incident where the cloud's usage-based billing ran wild. It's an incident on hardware I bought once — one I caused myself and can fix myself. Being able to settle in and chase the true cause without worrying about billing is also a byproduct of shifting execution onto my own gear. I'll write the incident's details in a separate article.
Summary
- Erase as much of the recurring billing as the design allows — take both the decisions and the hands-on work locally first, and send to the cloud only what's genuinely needed. The idea of replacing monthly usage-based billing with a one-time hardware cost
- Lock the buy/no-buy criteria with an ADR before buying — rule out "buy it because I want it," and decide the purchase on measured speed alone
- Choose models per role by measurement — 14B is enough for routing decisions (tied on accuracy with 72B, and 5× faster). There's also the trap where only 32B degrades. Run hands-on code generation limited to the zero-billing 72B lane
- Run one model per host — avoid swap costs, and decide allocation by measuring tok/s continuously
- Keep records — because the ADRs, benchmark reports, and postmortems are all in the repository, this article can be written
At the root of this replacement is a way of working where the hands-on work is turned over to AI and the human concentrates on the final approval = the decision. Right now it's Human-in-the-loop (HITL), with a person remaining at the approval gate; shift even that approval onto the mechanism, and it approaches HOOTL, where the human steps outside the loop. Placing the center of decisions (the orchestrator) on my own gear and cutting it off from external usage-based billing was one move toward running that foundation cheaply, for the long haul.
Top comments (0)