DEV Community: the kilted dev

The file nobody reread

the kilted dev — Sat, 05 Sep 2026 18:00:47 +0000

Field notes from a hosting migration for an NHS food-diary proof of concept. A certificate checker
was quietly lying about whether a fix had worked, and two sessions rejected the same hosting idea
twice, four days apart, without either reading the other's reasoning. The move itself was still on
hold when this was written.

This is a proof-of-concept, not an NHS product, built by one technical person working alongside an
NHS dietetic team. No patient data exists anywhere in it. The app stores everything on the
client device and nothing else. The trigger for all of this was mundane: Netlify's free tier caps
a project at 15 credits per production deploy, 300 a month, which works out to roughly twenty
deploys before the meter runs dry. The project was already rationing pushes against that ceiling,
and shared hosting on Namecheap, already being set up for another project the same person runs,
looked like a way out.

Two rejections, four days apart

GitHub Pages came up early, and the project turned it down fast. On the free plan it only serves
public repositories. A private repo needs a paid GitHub plan, and even then the site still
publishes to a public URL with no login wall short of Enterprise. Netlify's free tier deploys from
a private repo to an unlisted one instead, which fit the stated preference to keep the code itself
private. No patient data was ever going to sit in that repo either way. The preference was about
the source, not about anything a patient had typed.

Four days later, GitHub Pages came up again. Not from a different person with a different view. A
coding session working on the project proposed the exact idea an earlier session had already
rejected, because nobody had reread the decisions file before speaking. The occasion was almost
incidental: a different, newer domain had surfaced, already live on GitHub Pages with a working
certificate, and for a moment it looked like an obvious destination. That session looked into it,
then declined it again, for the same reason as the first time. That reason had not moved. It had
simply never been reread. This round did turn up one genuinely new blocker: the food-diary repo
and the newer domain lived under two different GitHub accounts, and GitHub blocks a custom domain
from being claimed by a second account once it's verified on the first, specifically to stop that
kind of takeover. A real finding, arrived at by accident, in the course of re-litigating a question
that had already been settled.

A decisions log nobody rereads is decoration. It was there, it had the right answer sitting in it,
and a session still asked the question again anyway.

A certificate checker that had been lying

The SSL side had its own real bug and its own fake one. The real bug: the domain's nameservers
were pointed at Namecheap's basic DNS, which only handles URL forwarding, not actual hosting.
Switched to the proper hosting DNS and that half resolved. An earlier assumption, picked up from a
web search during planning, claimed the hosting tier included self-service AutoSSL. It didn't.
Triggering AutoSSL turned out to require WHM-level (the hosting provider's own backend) server-administrator access, which an ordinary
shared-hosting account never gets. That was confirmed against Namecheap's own documentation once
the free "Run AutoSSL" button turned out not to exist anywhere in the dashboard. The free certificate that does ship
with the hosting plan has to be issued from Namecheap's own side, and nothing on the customer's end
can force it.

The fake bug was worse, because it looked like a real one for a while. This laptop's own antivirus
intercepts TLS connections and reissues every certificate itself, so anything running on the
machine, curl and openssl included, was reporting the antivirus's own certificate instead
of the real one.

The tool being used to answer "has the SSL fix landed yet" could not tell the truth about it even
in principle.

Checking from a phone worked around it, and later a tool that runs the request server-side,
outside the interception entirely, confirmed it for good. Two support agents tried
force-validating the certificate from Namecheap's own side with no result, and a follow-up ticket
went in to track it as a pure backend wait. Nothing on the customer side could shortcut that part.
The certificate issued not long after, and a server-side check confirmed clean HTTPS.

Two things the plan hadn't accounted for

First: SSH turned out to be off by default on shared hosting, though the migration plan had been
written without confirming that. The fix was self-service, a control panel option called "Manage
Shell," which gives no hint it's SSH, listening on a non-standard port rather than the usual one. Second:
pushing to the new remote from an automated shell hung indefinitely with no error, because a global
credential setting that keeps two separate GitHub accounts from colliding also blocks the plain
per-repo push path unless a repo-level override is set. That global setting has to stay exactly as
it is. The only new part is the missing per-repo override.

A smaller trap surfaced once both hosts were finally level with GitHub again: the control panel's
Git tool has an "Update" button that only saves configuration. It doesn't fetch, pull, or deploy
anything, and those actions live on a separate tab entirely. For a repository that was deliberately
cloned through the control panel's own "Clone a Repository" tool, with a specific option left unchecked to
preserve its automatic deploy hook, even that tab's own "Update from Remote"
option turns out to be a no-op. It's a push target, not a pull source. Reading the UI correctly
took longer than reading the DNS panel had.

Deleting a task is not a local edit

One step in the migration plan retired Netlify outright once the new host was live. Once Namecheap
turned out to be an interim stop rather than the final destination, the task was dropped, which
seemed like a small, contained change. It wasn't. That same task had been referenced as a
precondition in five other places across the plan, and one of those was dangerous on its own: a
later step told a future session to delete the Netlify credit-cap constraint from the project's own
guide file, once hosting had moved. Hosting hadn't moved. A session following that instruction
literally would have silently removed a still-live limit from the record that exists to track it.
All five references were found and corrected before that could happen.

What actually changed the private-repo rule

The preference that had driven both GitHub Pages rejections wasn't overturned by a technical
argument. It was relaxed on purpose, once the user actually named a real gap instead of waving it
off: the diary's free-text fields, a description typed next to a photo, have no structural way to
stop someone entering something identifying. Going public meant accepting that gap rather than
pretending it away. In the user's own words, it was "hard to guard... loose enough to say go
public." The account-split blocker from the second rejection turned out not to matter here either,
since the plan kept the repo under its existing account rather than moving it to the newer brand's.

A bigger swap, investigated instead of just refused

A smaller question, whether a branded subdomain could dodge a shared hosting username, led
somewhere larger: making the newer, higher-traffic domain the account's designated "Main Domain,"
on the assumption that the label itself would grant it more subdomain headroom. That assumption
didn't survive a check. The account's subdomain quota is shared regardless of which domain holds
that label, so an ordinary addon domain gets exactly the same headroom. What an
actual swap would require turned out to be much bigger than the label suggested: a different
project's own blog would have to leave its free, automatic hosting for manual shared-hosting
deploys with none of that automation and the same multi-day certificate process already documented
here. That's not a call this project gets to make about another one. It was declined without
touching anything, and the idea went out as two information-only notes to the two other projects'
own owners, for them to decide on their own terms.

How it actually landed

The reversal, when it came, was decided outside any of these sessions, by the one person who owns
all three projects at once: Netlify was retired entirely, and the food diary moved to GitHub Pages
after all, kept under its existing account rather than a newer pseudonymous one. That keeps a tool
built under a real identity separate from a brand built under a pseudonym, and it sidesteps the
cross-account domain restriction outright rather than working around it.

The account question that had paused the move turned out simpler than the plan in flight at the
time suggested. Rather than standing up a new account, the existing one kept its history and its
repos exactly where they were and simply changed its own display name. Same account throughout,
just renamed partway through. The migration is complete now. Both old hosts are fully retired, and
nothing is left dual-hosted.

It ended without a clever fix. Two things went wrong here that weren't code. Both were caught the
same way, by someone who actually checked instead of assuming. A certificate check that had every
reason to look trustworthy was quietly answering a different question than the one being asked. And
a written decision only held once someone actually reopened the file before speaking a second time.

Originally published at thekilted.dev/nobody-reread.

The verdict that survived its own correction

the kilted dev — Sun, 30 Aug 2026 06:15:27 +0000

This project started with a conclusion already sitting in its lap. In June, an earlier stretch of
work on the same machine had tested whether a local model could handle iterative coding: writing
files, running tests, fixing what broke, without a human shuttling text between a chat window and
an editor. The verdict was no. Not viable. That was the finding the whole project was built on.

On 4 July, someone re-tested it. It fell over in about two minutes.

The interesting part isn't that the verdict was wrong. It's that the verdict was right. It had
been stated precisely, with its own explanation attached, and none of that survived the trip into
the new project.

The re-test

The task was a word search game: three files, a 10x10 grid, eight words hidden horizontally,
vertically or diagonally, click-and-drag to select, correct words struck through in the list, a
New Game button. Eight scored criteria. The eighth is the one that matters: ask for one follow-up
fix (make found words green) and check it lands without breaking the drag highlighting that
already worked. That regression check is what had killed every previous run.

Run through little-coder, a small harness that gives the model real write and read tools instead
of asking it to print code into a chat window, against qwen3-coder-30b served by Ollama, the
result was 8 out of 8 in 2.1 minutes. The follow-up fix took another 30 seconds. All of it verified
in a real browser with simulated drag events, not taken from the model's own account of what it had
done.

The same benchmark against the smaller qwen3.5-9b scored 7.5 out of 8 in 44.1 minutes. It lost
half a point on reverse-placed and diagonal words: no reversed-string check, an off-by-one in the
highlight loop. It still passed the regression test, the first local setup here to manage that.

Model	Score	Time
qwen3-coder-30b	8/8	2.1 min
qwen3.5-9b	7.5/8	44.1 min

The quality gap was only half a point; the time gap was a full 42 minutes.

Two honest marks sit against the 8 out of 8. One transient failure died with "Stream ended without
finish_reason" and an identical retry worked. On criterion 8, one green pixel was never confirmed,
because the preview tab had collapsed to a zero-width viewport and inline styles had stopped
applying: impossible in a real browser, so it was scored as a harness fault rather than a code
fault. That was a judgment call, made and written down as one at the time.

What this project wrote down that evening

The June verdict was a harness problem, not a model problem.

It's a clean line. It reads as an insight. It went into the project's working memory as a headline
finding, was quoted straight into the active constraints file, and shaped everything scoped
afterward: if the harness is the variable that matters, you invest in harnesses and stop shopping
for models.

The investment was right. The claim wasn't, and nobody checked it for two weeks, because a
conclusion that's useful and turning out well doesn't feel like it needs auditing.

What the original verdict actually said

The June work had been handed over as a document. Reading it properly, two weeks late, the verdict
is one row in a table of six use cases:

Use case	Locally viable
Claude Code CLI backend	✕ broke — models misinterpret Claude Code's own system prompt format, not a config problem
Multi-file document extraction (LM Studio + Gemma 4 E4B, 8192 ctx)	✓ met — works well, the best multi-file result of anything tried
Single-file extraction, agent + filesystem (Goose + Ollama, qwen3.5-9b)	✓ met — works, but slow: 62 seconds per file
Iterative coding (any local chat UI)	✕ broke — not viable, and structural: five failure modes, not a fixable bug
Single-turn code generation (Goose or LM Studio, max context)	✓ met — viable as a baseline handoff into a human editor
Quota fallback for real sessions (fcc-server)	✕ broke — routed to Gemini instead, local wasn't trusted as the fallback

June's headline verdicts, all six, from the same document that scoped iterative coding to "any
local chat UI." The row that got re-tested is one of six, not the whole page.

The scope is in the verdict line. Not "local models cannot do iterative coding." Local models
cannot do iterative coding in a chat UI, which is a claim about an interface.

Below it, five structural failure modes are listed. Stateless regression, where a fix in turn N
gets silently undone in turn N+1: observed, in that same word search task, as a drag highlighter
fixed and then broken. Overconfident reporting. Silent context truncation. Effective context not
matching advertised. No cross-session persistence.

And then, closing that section:

Why Claude Code avoids 1–2: tool-mediated verification per step, persistent conversation
context — architecture, not raw model capability.

June had already worked out that the constraint was architectural. It's written down, in the
document that was handed over, in bold, five weeks before the July re-test that "discovered" it.

July didn't overturn June. July supplied the harness June had described, and measured what
happened.

Where June was too optimistic

Of the five failure modes, June expected a harness to handle the first two: stateless regression
and overconfident reporting, both covered by tool-mediated verification and persistent context.

The first half held. The second didn't.

Overconfident reporting is a model claiming completion when a feature doesn't actually work, and a
harness only catches that if its tests are adversarial. This project has watched a weak test
launder a false claim into a green, twice. A redaction check passed with
assert "84210.55" not in out while a partially redacted amount, 842[AMOUNT], leaked straight
through it. A loader passed nineteen of nineteen with a leaked loop variable, masked by a fixture
that only ever exercised one file.

Real file tools, persistent context, tests that ran green: both greens were wrong.

So June wasn't simply more careful than its own summary. On this one point it was optimistic, and
the correction only turned up once this project built the harness June had described and ran it
long enough to get burned.

What actually went wrong

Four things actually changed between June and July, worth stating because it means the re-test was
never a controlled experiment. The model changed. The harness changed. The runtime flags changed
(July ran with flash attention and a quantised key-value cache, June predated both). And the prompt
changed, because June's prompt text was never preserved and had to be rebuilt from scratch.

That last one isn't carelessness. The June raw evaluation file survives, and it's 809 bytes: a
findings summary whose entire task description is one line. There was no prompt left to preserve by
the time anyone went looking. Reconstruction was the only option on the table.

But the four variables are a side issue. The real failure was compression.

What travelled from June into this project was the phrase "iterative coding not viable." The scope,
any local chat UI, didn't make the trip. Neither did the mechanism: architecture rather than raw
model capability.

Then someone re-tested the compressed form, found it wanting, and corrected it into a claim broader
and less accurate than the original. A precise, well-reasoned, correctly scoped verdict went in one
end and came out the other as a slogan, and the slogan is what the project ended up arguing with.

The verdict, June: Iterative coding — not viable (structural, see the five failure modes) — any
local chat UI. Scoped to an interface, with its own mechanism named: architecture, not raw model
capability.

What travelled: "Iterative coding not viable." The scope and the mechanism both stayed behind.

What got re-tested: "Harness, not model." Broader and less accurate than the verdict it
replaced, and the claim this project ended up arguing with.

The handover document's own header says it plainly:

Nothing here needs re-deriving — it was all tested hands-on.

This project re-derived it anyway, and ended up with a worse version.

The part that only appeared while writing this

Writing this piece meant going back to the June sources, and the first attempt to find them failed.
The active constraints file listed the handover document under a path in the other project, the one
that had written it. It wasn't there. Searching that project turned up nothing.

The conclusion drawn from that was that both June primaries had been deleted. That conclusion went
into a correction to this project's own files, into an early draft of this piece, and into a
request asking another project to recover them from git history.

The document was sitting in this project's own repository root. It had been there since the first
commit. It had been written elsewhere and moved across at handover, exactly as its own header says.
The raw evaluation file had moved into a shared knowledge vault during a migration weeks earlier,
and was just as findable. The other project located both files in minutes, and mentioned, almost as
an aside, that it has no git history at all, so the recovery being asked for could never have worked
anyway.

One failed lookup at one stale path became "gone from disk," and a fair amount was built on top of
that before anyone checked the obvious place.

That's the same mistake this piece is about, made while writing about it.

An absence, asserted from a single negative result. A conclusion, drawn from a summary (the path in
the constraints file) rather than from the thing itself.

What changed

The finding was rewritten to what the evidence actually carries. The harness thesis is June's, not
this project's, and it was right. What this project added is confirmation and a measurement. The
broad version, "harness, not model," is now marked as over-stated, because it took this project's
own later evidence, not June's, to find the one place a harness alone wasn't enough.

The practical guidance didn't change at all, which is the uncomfortable part. Invest in the
harness. Verify with real tool output. Keep the local tier for bounded work. All still correct, all
still what this project does. A claim can be over-stated and still point the right way, and that
combination is exactly what stops anyone from checking it again.

Two rules came out of this, both narrow and mechanical. Before re-testing an inherited verdict,
read the primary document, not your own summary of it: an active-constraints file strips scope and
exceptions by design, because stripping detail is what it's for. And an absence needs more than one
failed lookup, especially when the path you searched came from the same compressed source you're
already re-deriving from.

What travels between sessions, projects and people is the conclusion. The conditions that produced
it stay behind. A conclusion without its conditions is a rumour with a number attached.

Originally published at thekilted.dev/the-verdict-that-survived-its-own-correction.

Protected nothing

the kilted dev — Sun, 23 Aug 2026 13:52:56 +0000

Field notes from a proof-of-concept food-diary tool built alongside one NHS dietetic team,
not an official NHS product, no patient data anywhere in it. A privacy guardrail blocked its
repository from going public. Traced through the code, it protected nothing that going public would
have exposed.

A rule sounded exactly right, protected exactly nothing, and stood as the one thing blocking real
work, until someone traced it back to what the code actually does.

The rule was simple. Before the food diary's repository could go public, it needed a guardrail
against identifiable data leaking through the diary's own free-text fields, chiefly a description a
client might type next to a reference photo. Someone could type their own name into it without
thinking, to help remember whose entry it was. That risk was real, agreed on at the time in the
project's own words as "hard to guard... loose enough to say go public," and it sat there as the one
condition standing between the repository and a public GitHub Pages migration everything else was
ready for.

The condition held for a day. Then, traced through the actual code rather than argued about again,
it didn't survive the first look.

Where the description actually goes

The description a client types is captured in the app, stored as a note on a log entry in IndexedDB
on the device, and exported into a spreadsheet the dietitian handles. It is never committed, never
pushed, never served. The repository's visibility has no effect on it whatsoever, because it was
never in the repository to begin with. A guardrail on that field would have protected data that
going public put at no risk at all.

A second, independent reason cut just as deep. The app ships with no build step at all, just
straight unminified JavaScript, so its entire source has been downloaded verbatim by every single
visitor to the live site since the day it launched. Repo visibility was never protecting the
application code either, because the browser was already handing that code to anyone who asked. The
"code stays private" preference behind two earlier hosting rejections had, for the part of the
project a visitor could actually reach, never once been true.

The place nobody checked

If the free-text field was never the risk, going public should have been safe from day one. It
wasn't, quite, and the reason was somewhere else entirely: roughly forty tracked files sitting
outside the application folder, never served to a visitor and shielded from view by nothing but the
repository itself being private, among them migration notes, deploy docs, session logs, and a stray
config file. Grepping through all of it for anything infrastructure-shaped turned up exactly one
genuinely sensitive string: a hosting-account username, and not even this project's own account. It
belonged to someone else's.

	In the repo	Repo privacy hides it
Free-text descriptions	no	no
Application source	yes	no
~40 files outside the app folder	yes	yes

Three things the repository question could have been about. Only the last is both tracked in the
repository and hidden by nothing else, so it is the only one that going public actually exposes.

The thing everyone had been scared of couldn't actually happen. The thing worth being scared of
had been sitting in a folder nobody had opened, the whole time the guardrail argument was running.

A fix that doesn't depend on remembering

The fix that shipped doesn't ask anyone to remember anything. Rather than scrub the sensitive files
out of a single shared repository and trust every future session to keep scrubbing, the project
split in two: one private repository holding everything, full history intact, and one public
repository synced from it that holds only the application directory's own contents. Nothing outside
app/ can leak, because nothing outside app/ is ever in the public repo to begin with.

Purging the sensitive files from one shared repository's history was considered and turned down on
purpose. NOTES.md gets written every session, which means every future session is a fresh chance
to type an identifier back into a file that's now public, and a miss like that is silent and
unrecoverable the moment the repository is cloned. The split doesn't have that failure mode. There's
nothing to miss, because the sensitive files are never in the repo that goes public in the first
place.

One faster fix was on the table too, and turned down for a completely different kind of reason.
GitHub offers private-repo Pages hosting on its paid tier, for around four dollars a month, which
would have made the whole question go away without touching a single file. Food diary's own author
turned it down outright, on the spot, with two words: "not pro." No technical argument followed it,
because none was needed. It was a spending decision, not an engineering one, and the project didn't
pretend otherwise.

What the guardrail actually protects now

The free-text field is still a real concern. It existed before this repository was ever considered
for public release, and it will keep existing after, because it's a property of what the app lets
someone type, not of who can see the code. It just isn't a hosting decision, and treating it as one
is the mistake this piece is actually about. It has its own design question to answer, on its own
schedule, and the fix for it won't be a line in a hosting migration plan.

The pattern generalises past one repository. The project specified a control against a threat that
sounded exactly right and had never been traced to the actual place the data flows through. It still
held up real work, however briefly. And the thing worth guarding against the whole time was sitting
in plain sight, in a directory nobody had thought to check, because the argument had already used up
everyone's attention on the door that was never actually open.

Originally published at thekilted.dev/protected-nothing.

Invoked, not executed

the kilted dev — Thu, 20 Aug 2026 07:35:56 +0000

A deep-research request to the top-tier model tore through three consecutive five-hour usage
windows, the rolling quota Claude enforces before a session has to stop and reset, to answer a
single question.

Usage window	Burned	Time to burn it
Window 1	54%	about two minutes
Window 2	100%	about four minutes
Window 3	40%	finishing the job

Three consecutive five-hour usage windows against one question. That adds up to 194% - nearly two
entire windows - and the first two were gone inside six minutes.

Nobody had done anything unusual. Someone had just asked the model to look something up properly.

The routing policy this machine runs on has a rule for
exactly this shape of task: work that's one step (search, fetch a source, extract a claim) and
objectively checkable belongs on the cheapest tier that can do it, not on the model that costs the
most per token. Deep research is built entirely out of that shape: search, fetch, extract, feeding
a synthesis step that does need judgment. The rule already existed. It just never had a chance to
apply.

Why the rule never fired

A routing policy only engages at a decision point: the moment a session has to choose which tier
does the next piece of work. Most of the policy's machinery assumes that moment exists. A plan gets
broken into tasks, and each task is a place to ask who does this.

A direct request, "deep-research this," skips all of that. It's answered as one thing, by whichever
model receives it, in whichever way that model has of doing research. There's no task list to route
pieces of. The search, the fetching, the extraction, the verification all happen inside a single
reply from the most expensive available model, because nothing ever asked whether the next step
needed to be there.

The fix follows from the gap. Route research-shaped requests through a harness that does the
decomposing on the way in, whatever words were used to ask. That covers all three shapes the ask
takes: an explicit "research this," a model deciding on its own that a claim needs checking, or
something as soft as "I'm not sure what's actually true here." All three land in the same place if
answered directly. All three should hit the harness instead.

The test: 3 phrasings, 2 tiers

Having independently found an adjacent problem (a third-party tool doing something similar had its
own bug in the budget cap meant to prevent exactly this kind of burn), a sibling project was asked
to check whether the routing fix triggered in practice. Six blind subagents, no shared context, one
prompt each: three ways of asking for the same research, crossed with two model tiers. The topic
was held constant, the enforcement status of the EU AI Act, so how the question was worded was the
only deliberate variable.

The intended way of measuring it failed. A subagent's own transcript file stays zero bytes even
after the run has finished, so nobody could simply read off which tools had been called. The
measurement fell back to asking each subject afterward what it had done, cross-checked against a
tool-call count the harness reports and the agent has no hand in writing.

The harness fired in half the runs. That looked like a pass. It wasn't one.

What "invoked" was hiding

Every run that triggered the harness received an instruction to call Workflow, the dispatcher
that fans the research out across sub-agents. Workflow is not available inside a delegated
subagent. Each of those runs found that out mid-task, noticed the gap, and quietly did the research
a different way: by hand, with the tools it did have. Nothing crashed. Nothing logged an error. The
final answer looked like a normal research answer, because it was one. Just not the one the rule
was written to guarantee.

One of the six was candid about exactly what was lost in translation. It had improvised its own
version of the harness's fan-out step, sending several of its own sub-agents out to search in
parallel, and reported the results converging as if that settled things. Asked afterward what its
own five-agent agreement was worth, it corrected itself before anyone pushed back:

The convergence I reported between agents is corroboration, not adversarial verification, five
agents drawing on an overlapping source pool can agree and still be wrong together.

The step that was skipped wasn't a decoration. It was the one part of the harness whose entire job
is to catch exactly that kind of agreement that isn't evidence. A rule checking whether the harness
was invoked would have called that run compliant. The harness had not, in the sense that mattered,
run at all.

Run	Harness invoked	Fan-out executed
Explicit ask (sonnet)	met	broke
Explicit ask (opus)	met	broke
No keyword (sonnet)	broke	never established
No keyword (opus)	broke	never established
Soft / vague (sonnet)	met	broke
Soft / vague (opus)	broke	never established

Invoked in three of six runs, executed in none. The three that invoked it reached for Workflow,
which a delegated subagent cannot call, and fell back to research by hand; the three that never
invoked it never got far enough to find out. The right-hand column is empty either way, and that
column is the one the rule was written to guarantee.

The number that looked best was the least trustworthy

The one wording that worked reliably, the explicit "deep-research this," didn't work because the
routing logic held. One of the successful runs said, unprompted, exactly why it had picked up on
the request. The words read to it as plain English, and it matched them to the right tool on its
own initiative. In its own account, the phrase was "hyphenated as a verb phrase," with "no slash,
no backticks, no skill syntax."

Nothing in how it was written signalled a tool call.

Rename the tool, or ask for the same research without those two words, and the coincidence that
made the easy case look solved stops holding.

The case that matters is a serious, specific question asked with no research vocabulary in it. That
is the shape almost every real research need takes, and it missed on every model tested. That's the
gap worth worrying about.

What changed, and what didn't

The rule was rewritten to check for evidence the harness had actually run: a count of sub-agents
dispatched, a tally of claims independently verified, rather than trusting that the right words had
been said. That closes the failure mode the test found: a compliant-looking run that quietly did
none of the work the compliance was meant to certify.

It doesn't close the other one. The wording that reliably works is still the wording that happens to
match the tool's own name. The question that matters, the one with no research vocabulary in it,
misses the harness on both models tried.

Fixing what gets checked was the easy half. The other half is getting the harness to notice it is
needed at all, from a question that never says so. That one is still open, and the coincidence that
made the easy case look solved is the reason to expect it is the harder one.

The rule that got rewritten, extracted and de-identified: the routing-policy
repo.

What really fits in 8GB VRAM

the kilted dev — Tue, 18 Aug 2026 19:03:52 +0000

Four VRAM numbers that looked wrong on an 8GB laptop, none of them a bug: what dxdiag actually
reports, what the desktop holds back before a model even loads, and the two fit rules that came
out of chasing all four down.

Every question about running models locally reduces to one question: will it fit. Which model,
which runtime, which quantisation, whether the thing is usable or a slideshow: all of it comes
down to that one word. The trouble is that the number deciding the answer is never the number
anything shows you.

This is a laptop with an RTX 5060 and 8GB of VRAM. Four separate figures turned out to be
measuring something adjacent to what actually mattered. Not one of them was wrong. Each was
reporting a real quantity. Just not the quantity that decides whether a model loads.

The first number: shared memory

› dxdiag /whql:off /t <outfile>

Run that on this machine, open the Display tab, and it reports the graphics card as having
24,144 MB of display memory. Twenty-four gigabytes, on a laptop sold as having eight. Directly
underneath, the same panel breaks it down:

Display Memory:    24144 MB
Dedicated Memory:   7899 MB
Shared Memory:     16245 MB

7,899 plus 16,245 is 24,144. The arithmetic is honest. Display memory is the dedicated VRAM on the
card plus a slice of system RAM the driver treats as overflow, half the machine's 31.7GB, the
standard Windows allocation. Only the dedicated figure decides whether a model loads.

Two details make the trap worse than a single mislabelled number. The shared pool isn't the
graphics card's at all, and every display adapter on the machine claims it: the integrated Intel
graphics, the RTX 5060, and two DisplayLink devices all report the same 16,245 MB. Add up what the
machine appears to have and the total overstates the truth several times over. The dedicated
figure isn't perfectly stable either: a model-fitting tool run on the same box reported 7.96GB
where dxdiag says 7,899 MB, close enough to be the same answer, far enough apart that a fit
calculation cutting it fine will disagree with itself depending on which tool it asked.

The two pools behave completely differently under load. VRAM is fast and finite. The shared
region is system RAM wearing a costume: spilling into it doesn't fail, it just gets slow, and a
slowdown is worse than a failure because a failure tells you something is wrong while a slowdown
just lets you keep going, wondering why everything feels off. A 9B model at 32K context sits
entirely in VRAM at 5.9GB and generates 51.5 tokens/sec. Push the context to 64K and it needs
7.5GB, spills to an 18/82 split, and drops to 22.6 tokens/sec. Same model, same machine, same
quantisation. One setting, less than half the speed.

None of this requires taking anyone's word for it: the breakdown is four lines down the Display
tab on every Windows machine there is.

The second number: your desktop already has some of it

The dedicated 7.9GB isn't yours either. Under normal use (browser open, editor open, the desktop
doing its job) somewhere between 2 and 3.5GB is already spoken for before a single model loads.
Not reserved in a way you can reclaim by closing one app. Just gone, in the ordinary course of
using the computer for anything other than inference.

The spread matters as much as the figure. Measured on this machine with a browser, a notes app, a
spreadsheet and around thirty other things touching the GPU, it sits at almost exactly 2GB. On a
busier day it has been half again that. On the quiet afternoon the benchmark below was recorded,
it was under 1GB.

That shows up in the benchmark record as an awkward footnote. Here is the same 36B
mixture-of-experts model, three ways, every row measured at the same 32K context:

Configuration	Gen t/s	VRAM	When you can use it
llama.cpp, 8 expert layers on GPU	27.1	7.4GB	Apps closed only
llama.cpp, 6 expert layers on GPU	26.3	6.4GB	Recorded safe default
Ollama, automatic 80/20 CPU/GPU split	23.4	n/a	Default behaviour

Same day, same machine, flash attention on, KV cache quantised to q8_0. Ambient GPU load during
measurement was unusually low, between 0.3 and 0.8GB; normal desktop load is around 3.5GB.

That fastest row leaves around 800 MB of headroom, and it was measured on that unusually quiet
afternoon. Under normal desktop load it doesn't fit at all. So the best number in the table is one
that can't be reproduced while actually using the computer, and the benchmark file says so
directly rather than quoting the headline figure and moving on.

The two rules that came out of it

After enough of this, the fit question collapsed into two rules that are worth more than any
individual measurement.

Dense models: the file has to be about 7GB or smaller at 4-bit quantisation, and it has to run
entirely on the GPU. If it spills, the speed lost is more than a bigger model would have gained.

Mixture-of-experts models: the file has to fit in system RAM (31.7GB here) with the experts
offloaded to the CPU. VRAM only needs to hold the attention layers and the key-value cache.

The second rule produces the result that governs everything this project runs: a large
mixture-of-experts model fits comfortably where a much smaller dense model doesn't. A 30B MoE
model with a 19GB file runs at 32.9 tokens/sec generation on a 69/31 CPU/GPU split, at 16K
context. A 14B dense model is simply ruled out, and so is the default pull of a well-known 9.6GB
dense model, on the dense rule alone.

That context qualifier isn't decoration. Speed figures are only comparable within a single
context size, which is why every number here carries one.

Parameter count isn't the constraint. Architecture is: a model three times the size fits because
of how it's shaped.

That's invisible from any spec sheet that lists parameters and file sizes. Not everything needs a
hands-on trial to rule out, either. An 80B MoE model was ruled out at the research stage: its
smallest 4-bit quantisation is 38.4GB, more than the machine has RAM for at all. No test required.

The third number: the config that was right when it was written

Six weeks in, a model that had been working for over a week stopped loading. Every attempt died
with a CUDA out-of-memory error, and the first answer was wrong.

The server log said it had offloaded 49 of 49 layers to the GPU, which reads unambiguously as "it
tried to load the whole thing into VRAM instead of doing the CPU split that had been working." A
satisfying scheduler bug, except it wasn't what happened. Reading the full memory breakdown
instead of the first suspicious line showed the actual buffer allocation: 3.2 to 4GB in the GPU
buffer against 13.5 to 14.3GB in pinned host memory. The CPU offload was correct the whole time.
That log line's wording misled; it wasn't evidence.

The real cause turned up in the runtime's own fit-calculation logging, and it wasn't a bug either.
The fit calculation had correctly worked out that 43 of 49 layers would overflow to host RAM, and
projected 5,123 MiB of VRAM use against 6,196 MiB free. A plausible-looking fit.

What had actually happened was that the model's local alias had been silently recreated eight
days earlier, inheriting the project's default context setting of 32,768, and this particular
model had never been verified at that context. It had been benchmarked at 16K.

The evidence was sitting in the original benchmark file the whole time, contradicting itself. The
prose describing the working run states the context parameter as 32,768. The speed table further
down labels the identical CPU/GPU split "@16K." Both written the same day, about the same run.
Nobody reads a benchmark file for internal consistency after writing it, so the contradiction sat
there quietly. A single-variable test settled it: at 16,384 the model loads and generates every
time; at 32,768 it fails every time. The fix was re-pinning the alias: a configuration
correction, not a code change.

The fourth number: free memory doesn't predict this

The most useful part of that failure is the bit that contradicts intuition. The out-of-memory
error didn't depend on how much VRAM was free. Retested with the model fully unloaded and 6.1GB
available, it still failed, ruling out the obvious explanation that some other app was
transiently holding memory. At 32K context, the model's key-value cache plus its compute buffers
exceed what fits after the CPU offload split, regardless of what's free. Headroom isn't the
variable. The shape of the allocation is.

That has a practical edge: a fit-estimating tool used on this machine calculates against total
VRAM rather than free VRAM, so its recommendations need hand-adjusting for whatever the desktop is
already holding.

What this adds up to

Four numbers, four different ways of being adjacent to the truth. Windows reports memory that
includes a region you don't want to touch. The desktop silently holds a few gigabytes of what
remains. A log line describes full GPU offload while performing a CPU split. A config file records
a context the model was never tested at, and a fitting tool measures against a total nobody
actually has.

None of these is a bug. Every one is a real measurement of something, presented plainly, by
software with no intention to mislead. They're wrong only in the sense that they answer a question
next to the one being asked, which is why the two fit rules are worth more than the measurements
that produced them. The rules survive a driver update, a runtime version bump, a model release.
The numbers don't, and the ones that look most authoritative are usually the ones measured on an
unusually quiet afternoon.

Originally published at thekilted.dev/what-really-fits-in-8gb-vram.

Nine ways to talk to a local model

the kilted dev — Tue, 11 Aug 2026 15:56:51 +0000

A nine-tool survey of local-model interfaces on one GPU: what worked, what silently failed, and
why what sits between you and the model matters more than which model you picked.

Nine pieces of software for talking to a local model went through this machine over about two
weeks. Same GPU, largely the same handful of models, one afternoon each.

The spread in outcomes was enormous, and almost none of it was about the model. The same weights
that scored full marks through one harness produced unparseable garbage through another, and hung
indefinitely through a third. What sits between you and the model turns out to matter more than
which model you picked, which is a boring conclusion until you notice it also means most model
comparisons are measuring something else.

Here is what each one was actually good at.

Tool	Good at	The number that mattered
Ollama	the default, broadly reliable	32.9 t/s gen on a 30B MoE
llama.cpp	fastest, once tuned	27.1 t/s vs Ollama's 23.4
LM Studio	document extraction to file	10 rows, exact match, no invented values
little-coder	small-model coding harness	8/8 in 2.1 min, then a version bump broke it
Unsloth Studio	fastest single-file result	unusable past a 4096-token ceiling
Goose	agent framework, right idea	30% GPU vs 98%, one setting
Claude Code as client	n/a	all three connection methods fail
Odysseus	the most polished interface	0/8, only one of three files actually saved
Continue.dev	autocomplete, nothing else does it	355MB free at 32K, no room for anything else

Ollama, the one everything else is measured against

The default answer, and it stayed the default. It runs a 30B mixture-of-experts model at 32.9
tokens/sec generation and 443 tokens/sec prefill on a 69/31 CPU/GPU split, which is the single
most useful capability on this box.

Two things about it are worth knowing before you trust it. Its desktop application stores a
context-length setting in a local database that overrides the environment variable you set,
which produced a memorable afternoon of a model ignoring configuration that was demonstrably
correct. And model aliases pin their own context, which is how a working alias silently acquired
a context it had never been tested at and started failing weeks later.

Neither is a defect exactly. Both are the kind of thing you only learn by being caught by them.

More: ollama.com

llama.cpp, faster if you're willing to tune it

For one specific job, a 36B mixture-of-experts model at 32K context, llama.cpp beats Ollama by
about 16% on generation and 34% on prefill. 27.1 tokens/sec against 23.4.

That margin isn't free. It came from three separate tuning discoveries: disabling memory-mapped
file loading, which llama.cpp itself warns is slow when combined with CPU tensor overrides and
which alone accounted for most of the gap; setting thread count to the CPU's fourteen physical
cores rather than the eight the script had; and tuning how many expert layers stay on the GPU.

Untuned, the same path runs at 13 tokens/sec, less than half. The default configuration of the
faster runtime is slower than the alternative, which is worth remembering whenever a benchmark
reports that one tool beats another.

llama.cpp's server also has the most complete tool-calling story of anything tested here. Given a
toy arithmetic tool, it emitted five valid parseable calls out of five. Given three near-identical
tools and six questions, it selected correctly six times out of six. That second number is the one
that matters, because choosing between similar tools is where tool-calling usually breaks.

It also has server-side built-in tools, off by default, with the server itself warning against
exposing them to untrusted environments (the available set includes shell execution). Only the
read-only subset was enabled here. Notably, the server does not auto-execute anything: it returns
the tool call and the client drives the loop. That's the right design, and it means the safety
boundary sits where you can see it.

More: github.com/ggml-org/llama.cpp

LM Studio, the graphical one, and the absolute-path rule

Good at exactly one thing this project needed: read some documents, extract structured data, write
the result to a file. Given real utility bills it produced ten rows across two months with every
usage quantity and cost matching the source exactly. No invented values.

Getting there took three attempts. The first two failed with filesystem permission errors even
after access was explicitly approved through its own dialog. The difference was the prompt: "the
files in your workspace" failed, and naming the absolute directory worked.

Whether the model requested bad relative paths or the permission scope failed to resolve them was
never determined. The rule is empirical and it's reliable: give it absolute paths.

More: lmstudio.ai

little-coder, the one that proved the whole thing was possible

A thin harness tuned for small models, giving them real file-writing tools rather than asking them
to print code into a chat window. It's what scored 8 out of 8 on this project's benchmark in 2.1
minutes, including the follow-up-fix-without-regression test that had defeated everything before
it.

Then a version bump broke it. With its default model it began emitting raw XML that nothing
parsed, exiting successfully, and writing no files, the worst failure shape available, because an
exit code of zero and no error output reads as success to anything automated.

The instructive part is the diagnosis, which was wrong the first time. The failure was recorded as
being caused by the version change breaking model-id resolution, because a warning about the model
id appeared next to the failure. A later controlled test held the model constant and varied only
id registration: the warning is benign, and the actual fault is the specific model. The same
harness works with a different one.

Adjacency isn't causation, and a warning that appears next to a failure is still just a warning
that appears next to a failure.

More: github.com/itayinbarr/little-coder

Unsloth Studio, fastest and unusable

The fastest single-file result of anything tested: ten seconds.

Its context window control is broken on Windows. The effective ceiling is 4096 tokens regardless
of what the interface is set to, which makes multi-file work impossible and makes any measurement
taken through it incomparable to anything else. A tool that silently runs at a fraction of the
context you configured is worse than one that refuses, because the number it shows you is a number
you will use.

It also gets flagged as malware by the antivirus on this machine, which is a false positive, and
which is its own small saga.

More: unsloth.ai

Goose, right idea, wrong backend

Goose works here, with one hard rule: use its Ollama provider, never its built-in llama.cpp.

The built-in path manages roughly 30% GPU and 25% CPU utilisation on Windows, a broken offload
that leaves the hardware idle while the model crawls. Through the Ollama provider the same machine
runs at 98% GPU.

Same tool, same model, same box. One configuration choice, and the difference is the entire
value of having a GPU.

More: github.com/block/goose

Claude Code itself, three methods, one root cause

The obvious idea, and the one people ask about most: point the agentic coding tool you already use
at a local model, and stop paying for tokens.

Three connection methods were tested. Setting the API base URL directly, which produces no output
without also setting an auth token to a dummy value. A proxy shim, which worked and ran at roughly
two and a half minutes per response on CPU offload. And the runtime's own native launch command,
which produced a model hallucinating unrelated tasks and emitting raw tool-call syntax as text.

All three fail, and they fail for the same reason: local models do not parse Claude Code's
system prompt format. That's not a configuration problem, and no amount of trying a fourth
connection method changes it. The finding was worth writing down precisely so nobody here burns
another afternoon on connection method five.

More: claude.com/product/claude-code

Odysseus, the most polished tool tested

The most polished thing tested, and the worst result.

It scored 0 out of 8 on the same benchmark. The run climbed from 0.74 to 10.19 tokens/sec, then
stalled dead at "4% complete" with a byte-identical transcript across a five-minute recheck.
Underneath, the backend had dropped the generation stream and was returning 404 to the frontend's
status polls, which the frontend kept making, forever, with no error surfaced, no retry, and no
timeout. A silent, indefinite hang.

Two things compounded it. Its interface exposes no control for disabling thinking mode, which this
project has needed since day one, so the model visibly re-planned and restarted its own draft
mid-stream. And checking its own document panel rather than trusting the chat transcript revealed
that only one of three files had actually been saved. The transcript showed content that was
never written anywhere.

A chat transcript is a record of what a model said, not what a system did, and those two things
diverge silently.

It's also structurally unable to do what this project needs: its tools are a fixed set of built-in
capabilities toggled per message, not a registry you can register an arbitrary function with.
There was no way to run the tool-calling probe at all. Not a bug, a different product.

More: github.com/odysseus-dev/odysseus

Continue.dev, the one that does something nothing else does

An editor extension, evaluated for autocomplete rather than chat, because autocomplete is the one
capability nothing else here provides.

A 4B model completes a fill-in-the-middle hole in 209 to 252 milliseconds at 127 tokens/sec,
comfortably inside the budget where ghost text feels helpful rather than laggy. A 9B does it in
around 328 milliseconds. A 14B coder model, the one nominally built for this, takes 2.5 seconds,
because it spills to CPU, and rambles past the hole into inventing further functions.

The measurement that mattered was not speed, though. With the autocomplete model warm at 32K
context, free video memory drops to about 355MB for the 9B and 1GB for the 4B. There is no room
for anything else. You can have interactive coding with autocomplete, or a background delegation
run. Not both.

More: continue.dev

The comparison we never ran

All of this was supposed to be settled by a formal benchmark. It was written into the handover
document that started this project, as open item one: the word search task run across five
entrants, Goose's built-in backend, Goose via Ollama, llama.cpp, Unsloth Studio and LM Studio,
against one baseline model, with context standardised at 89K on every runtime so the results would
be comparable.

The plan even predicted its own casualty:

Unsloth will fail this, document it.

It was closed without ever being run.

Not from lack of time. By the time it came up for scheduling, its actual question, which runtime
for which job, had already been answered by using them. Ollama for the coding harness. llama.cpp
for the 36B at 32K. LM Studio for graphical document extraction. Goose only via Ollama. Unsloth
ruled out on the context ceiling. The proxy shim never a peer to the others, because it's a
different category of thing entirely.

A generic five-way bake-off would have re-measured memory-spill behaviour already understood, at a
context size chosen to be fair rather than useful, and changed not one of those decisions. The
tools had already differentiated themselves on the only axis that mattered: what happened when
real work went through them.

One footnote, found while checking the above against the original plan rather than against our own
notes. The record written when the benchmark was closed describes it as having six entrants, and
lists a different set, merging Goose's two configurations into one, adding a runtime the plan had
not included, and adding a proxy shim the plan mentions only in its list of things already
rejected. The reasoning for closing it was sound and every per-use-case answer holds. The entrant
list was simply restated from memory rather than reread, in a note written the same week the plan
was still open.

A benchmark nobody ran is a low-stakes thing to miscount. It's the same move that makes a
miscounted one dangerous.

The rule that came out of it is narrow. Benchmark a use case, never a category. If a new runtime
appears, or a genuinely new job appears, measure that, and if the answer is already visible in
work you have done, the benchmark is a formality you are performing for the shape of it.

Nine tools. The differences that mattered were reliability, offload behaviour, honesty about what
was actually written to disk, and whether the thing exposed the one control the model needed. None
of that shows up in a table of tokens per second.

Originally published at thekilted.dev/nine-ways-to-talk-to-a-local-model.

Convergent evidence

the kilted dev — Mon, 10 Aug 2026 18:57:13 +0000

Companion to Route, don't guess: nine tools crossed the desk during the ten-day build. Not one became a dependency. The reasons why are the actual argument for building it in-house.

Building something in-house invites an obvious question: did you check whether someone already shipped it? A companion project, the one that tracks and evaluates AI tooling for the whole machine, kept an eye out for exactly that, every time an orchestration-shaped tool crossed its desk. The bar it held every candidate to was narrow on purpose: does it add a capability the orchestrator doesn't already have?

Almost nothing cleared it. That's less interesting as a verdict than as a pattern.

The direct comparators

Six candidates were orchestrators or multi-agent harnesses solving the same problem: which model does which task, and how do you trust what comes back.

Fable Foreman is parked, worth watching. It sits near one-to-one with our own routing policy: capability-class tiers, blind verification, budget-aware dispatch. The value was design comparison rather than new capability. It was also four days old at review, with two of its three headline features gated behind a call-to-action and never actually shown.
Ruflo went to the backlog. A hundred-plus prebuilt agents, vector memory, multi-model routing, some 210 tool integrations, 58k+ GitHub stars and climbing. What it sells is the routing-plus-verification loop already running here, more simply.
CrewAI skills was rejected on overlap, not quality. It's official plugin scaffolding for an orchestrator, researcher, writer, and reviewer crew, from a legitimate 55.5k-star parent project. The same pattern already runs on the machine's own workflow and routing layers; a third-party framework would duplicate it rather than extend it.
9router was rejected hard. It's a credential-interception proxy for every model provider, carrying an unauthenticated remote-code-execution flaw scored CVSS 10, plus terms-of-service risk for any subscription routed through it. An existing fallback covers the same need safely.
Claude Council was declined by choice. A genuine multi-model consultation plugin, honestly framed, but a lighter version of a panel-of-models pattern already covered elsewhere.
Three "agentic OS" reels were rejected with nothing to install. Camera mockups of an always-on multi-agent life OS. Every named part (a constitution file, tier routing, adversarial sparring, a cost ledger, a budget halt) maps onto pieces already running here, separately and more safely. The bundle is marketing wrapped around five patterns already decomposed on purpose.

The adjacent category

The second cluster doesn't route anything. These are desktop shells for running a fan-out once you've decided on one, checked for an unrelated reason rather than because they compete with the routing policy.

Orca was confirmed hands-on. Free, runs the real CLI under your own subscription; skill and plugin parity held up, and a worktree fan-out plus diff-and-merge round-trip verified end to end on a scratch repo. 12.1k stars.
Nimbalyst went to the backlog. Kanban-style parallel sessions with worktree isolation, parked pending confirmation it plays well with an existing hook-based workflow.
DevSwarm was rejected. Same category as Orca, but closed-source, paid, download-only, and the public repo is a landing page with 87 stars. It doesn't clear a bar a free alternative already cleared.

What cleared the bar wasn't a tool

Two inputs cleared the bar, both as ideas rather than installs, arriving from opposite directions and landing on the same shape.

A stranger's public routing config contributed one genuinely new distinction: taste as an axis separate from difficulty. A component name or a marketing hook is trivial to compute, but still wants the highest-taste model available. A difficulty-only rule would route that down, wrongly. The axis is now a permanent exception in the routing policy. The same source's "manager never runs high reasoning effort" rule was weighed and rejected; it conflicts directly with keeping the terminal review and hardest verification stages deliberately expensive.

And an academic paper on filesystem-as-agent-architecture (numbered stage folders, a per-stage context file, human gates between them) turned out to overlap about 80% with the four-file working-notes system already in use. Independent convergence, this time from arXiv instead of a product pitch.

Neither produced a dependency. Both produced a stronger version of something that already existed.

The closest comparator, side by side

Fable Foreman is worth lining up in full, precisely because it's the closest external thing to the in-house build.

Dimension	Fable Foreman	The in-house orchestrator
Routing basis	Capability class: frontier / workhorse / fast	Verifiability + iteration count, down a cost ladder
Model naming	Aliases resolved at runtime	Same. Tier by role, never a hardcoded model ID
Verification	"Trusts no one": blind verify against the real workload	Same posture. Never trust a subagent's "done" without evidence
Budget awareness	Budget-aware dispatch, degradation rule	Append-only ledger plus a budget halt
Design/review routing	Not specified	Never routes down. A standing, explicit rule
Trust surface	Four days old, spawns full agent fleets, content gated behind a CTA	In-house, auditable, every piece proven separately

The market kept independently rediscovering the same routing model: a stranger's config file, an academic paper, three viral reels, and a four-day-old repo that maps onto our own build dimension for dimension. None became a dependency, because the same five patterns were already decomposed into separable, auditable pieces before any of them showed up.

The actual takeaway

A single rejected tool proves nothing; every project rejects things. What's worth noticing is the shape of nine candidates checked and not one dependency adopted. Not one offered a capability the routing layer didn't already have, and the two that offered something real turned out to be ideas rather than products. That's what you'd expect if the underlying problem (route by verifiability, verify blindly, cap the budget, keep a human in the loop) is well-understood enough that independent people keep arriving at the same four or five pieces.

The discipline wasn't refusing to look. It was refusing to trade five pieces we can audit for one box we can't. This doesn't claim there were no more tools worth checking. Only that everything that crossed the desk was checked against the same bar, and the bar held.

Originally published at thekilted.dev/convergent-evidence.

Route, don't guess

the kilted dev — Mon, 10 Aug 2026 15:20:58 +0000

Field notes from the machine: how we built the orchestrator that routes each task to the optimal model, and the near miss that drew the line on what it's allowed to act on.

Every Claude Code session defaults to the model it was opened with. Ask it to rename a variable or design a whole subsystem and, left alone, it reaches for the same tier both times. That's fine until the quota notices. Tokens and time are one budget, and the top tier spends both fastest. Waste them on a variable rename and they aren't there for the subsystem. The obvious fix is a rule of thumb: low-cost model for easy stuff, expensive model for hard stuff. It sounds right. It's wrong, and working out why it's wrong is most of what this project turned out to be.

What it actually decides

The orchestrator is a routing layer, not another agent framework. It lives as markdown: a contract file, a handful of dispatch templates, an append-only ledger, all read by a session before it delegates anything. No installed dependency, no external scheduler. The bet, made on day one, was that Claude Code's own primitives were already enough, and the missing piece was a policy rather than a platform.

The policy's core move is refusing "hard versus easy" as the routing variable. What matters is task shape: how many correction rounds the work needs, and whether a given round can be checked objectively.

Route down — one-shot, objectively checkable: tests exist, or the output validates against a schema. Push it down the cost ladder, through scripts and local models before the low-cost hosted tiers.
Stays top — iterative and judgment-heavy: design, debugging, architecture, review. Stays with the top-tier model, in-session.

Underneath that split sit three costs the "low-cost tier for easy tasks" instinct never prices in:

Cold-start tax — every delegated session re-derives context from nothing, so handing off a five-minute edit loses outright.
Review tax — whoever dispatched the work still pays to read whatever comes back, and a chatty response from a low-cost tier can erase its own savings.
Retry tax (dominant) — low-cost models fail more often at anything with judgment in it, and every failure means a redo.

The rule that survives all three isn't "is this task hard". It's whether the expected saving beats the expected cost of doing it again. Iteration-heavy work maximises the wrong side of that inequality no matter how easy it looks.

What never routes down

Four categories are pinned to the top tier permanently, however simple the task looks in isolation.

Taste-sensitive output is the first: design, copy, naming, brand voice. A component name is trivial to compute and still wants the model with taste. It turns out "easy" and "low-stakes" are different axes, an idea that arrived from outside the project and earned its place.

The terminal review is the second. Per-task reviews can run on a mid tier, but the whole-branch pass before anything ships always runs top-tier, because it catches what the narrower passes miss. A pass on a low-cost tier once sailed past a real security hole that a stronger model then caught. The rule dates from that day.

Third, a green light from a low-cost tier gates a dispatch but never substitutes for verification. Low-cost models have a structural habit of reporting success they haven't earned.

Fourth, the orchestration layer itself. A low-cost model was tried as middle manager between the session and a worker. Sixty minutes and roughly fifty tool calls later it had shipped nothing and quietly run a global install nobody asked for. Delegation stays a command the session issues directly.

› echo "nobody gets a middle manager"

Proof over promises

Every delegated run ends the same way: a status block in the chat, and a row appended to a ledger file. Neither trusts a bare claim. A "done" needs the tool output, test result, or screenshot sitting next to it.

The ledger isn't decoration. Each row records what was routed where, whether it passed, and the field that matters most: any lesson. A populated lesson is a candidate for promotion into the routing rules themselves. The policy tightens from its own field evidence, not just from design sessions.

The near miss

The hardest lesson didn't come from a cost overrun. It came from a decision that would have run correctly, in the wrong place.

A "do it everywhere at once" clause, written into one project's own decision log, would have had that project's session run git rm --cached against a sibling project's repository, one it had no business touching. The user caught it before it executed. Nothing was lost.

But it exposed the real default across every project on the machine: any session could offer to edit a neighbouring project's files, and the offer usually looked reasonable.

The fix wasn't a smarter check. It was inverting the default.

A session now treats every other project's tree in three bands:

Read (always allowed) — reading a sibling's files is always allowed, since no desync is possible.
Propose (the only sanctioned cross-project write) — writing a note into the sibling's root asking its own next session to decide.
Mutate (denied by default) — editing a sibling's facts or running commands in its repo. Two narrow carve-outs: files whose whole purpose is to receive writes, like the routing ledger, and gated edits to genuinely shared config, which always name their blast radius and wait for sign-off.

Everything else routes through a proposal note. It suggests; it never asserts. The receiving project's own next session folds it into its own files, then deletes it. No queue, no index, no daemon. Three ad hoc uses proved the pattern before anyone wrote it down as a rule.

How it came together

Eleven days, roughly a session a day, each one closing with the same four working files (notes, memory, active constraints, rejected approaches) so the next session opens cold and still knows why past decisions landed where they did.

Day 1 — Founded on native primitives. External orchestration frameworks rejected in favour of Claude Code's own agent files, skills, and hooks. First trial ran in a throwaway project on purpose, for zero regression risk while the design was unproven.
Day 2 — The ladder tested for real. A wiki migration ran as the first live trial, with a script, a local model, and two hosted tiers all doing genuine work in one pipeline. The layered-review rule landed that same week, after a low-cost pass missed that security hole.
~Same week — First shipped consumer. A pension calculator's feature became the first real work routed end-to-end through the policy. It surfaced the first known gap: the local tier only speaks Python, and this task was JavaScript. Not a misroute, a disqualification, but a gap worth naming.
Day 4 — Outside validation. An unrelated public routing framework, reviewed independently, converged on the same core shape from a completely different workload. We folded in its one genuinely new idea: taste as a routing axis distinct from difficulty. We rejected its "never run high reasoning effort" rule, since it conflicts with exactly the stages this project keeps expensive on purpose.
Day 6 — Cross-project messaging formalised. Three ad hoc successes generalised into the proposal-note convention.
Day 7 — The near miss, and the contract it produced. Default-deny on cross-project mutation became a standing, machine-wide rule.
Day 10 — Fleet-wide rollout. The enforcement line, the one that makes a project's session actually consult the policy instead of silently defaulting, was offered across around eighteen sibling projects. Opt-in per project, tracked from the filesystem alone.
Day 11 — The ledger checks itself. A staleness audit found most adopted projects had quietly stopped logging, so a hook now nudges emission at session end. The same day, a reviewer on a low-cost tier blamed an unrelated, pre-existing file change on the task it was checking. Comparing timestamps caught it, and it became a standing rule: a low-cost tier may scan, but attributing a found change to a cause needs a timestamp check first, not a guess.

What's still open

Three gaps stay visible on purpose. The local tier only routes Python-shaped work, so JavaScript tasks run top-tier by default until a second real JS consumer justifies building a driver. Reasoning effort never entered the model at all: the policy routes on which tier runs a task, not on how hard that tier is told to think, and the same model at a different effort setting is a different cost and arguably a different capability. That dimension is unexamined, not settled. And one mid-tier code-generation route sits in the ladder unproven: named, priced, ready, and untouched, because no task has gone through it yet.

A rung nobody has used is not the same as a rung that works.

Nothing above is theory. Every rule came from work that actually ran. The whole apparatus is a bet that the interesting problem was never "which model is smartest". It was knowing, task by task, when smartest is the wrong thing to be paying for.

Companion piece: Convergent evidence. Nine orchestration-shaped tools crossed the desk while this was being built, and not one became a dependency.

The policy itself, extracted and de-identified: the routing-policy repo.

Originally published at thekilted.dev/route-dont-guess.

The dashboard accused itself

the kilted dev — Sun, 02 Aug 2026 20:31:11 +0000

Our own cost dashboard flagged the day it was born as a 26.5x overspend. The transcript said otherwise. Making sure it never happened to anyone else took a bug report, and three days later the fix shipped.

We build a lot of small internal dashboards. One of them tracks Claude Code token spend across the projects on this machine. It leans on an open-source engine called CodeBurn: free, local-only, reads session logs and estimates what each session would have cost on pay-as-you-go pricing. CodeBurn has a panel called optimize, and the panel's job is to flag waste. Sessions that cost far more than a project's own average, with no obvious reason why.

One day the panel flagged something inside its own house. The dashboard's build session, the day it came into existence, sat there marked as a 26.5x cost outlier. Twenty-six and a half times the project's own average. That is the kind of number designed to make you flinch.

We didn't flinch. We pulled the raw transcript.

What was actually in there

It ran to 1,070 events. A brainstorm, a plan, twenty-four build agents dispatched, and a final commit that promoted the dashboard to its first real version. That was the founding session, and founding sessions are expensive. The founding session alone ran 1,045,640 tokens, against an average of 62,940 across twelve sibling sessions that followed — a 16.6x gap on tokens, before cost pricing even enters it.

The day a project goes from nothing to something costs more than any day after it. A house's foundation costs more than repainting one room, and nobody files that under waste. The panel had measured the founding day against the quiet maintenance days that followed and called the ratio waste. That's a category error rather than a measurement problem. The comparison was never apples to apples, and no cost threshold repairs a comparison that was wrong before the arithmetic started.

So the panel was right to ask the question. It was wrong about the answer.

The two easy exits

We could have ignored the flag. We could have silenced it locally and moved on, which would have worked fine, for us. It would also have left the bug sitting there for everyone else running CodeBurn against a young project, which is every project at some point.

So we read CodeBurn's source instead. The comparison lives in a function called detectSessionOutliers, in src/optimize.ts. Reading it turned up the useful part: the codebase already had the mechanism a fix would need. A wee exclusion list called outlierExclusions, already doing this exact kind of job elsewhere in the panel, sitting right there.

That gave us something better to file than a complaint. Issue #664: what happens, why it's wrong, and the specific lines to change.

The maintainer's fix was better than ours

CodeBurn has an active maintainer, and the pull request was open by the next morning.

Our sketch excluded every project's founding session, permanently. The maintainer's version was sharper. It excludes the founding session only while a project is young, and it measures young by how many costed sessions the project has behind it rather than by wall-clock age.

That distinction is the whole fix. Under our version, a genuinely wasteful founding session would have hidden behind the exemption forever. Under the maintainer's, the benefit of the doubt expires after six costed sessions. A founding session in a two-week-old project gets it. The same session in a mature project with months of history does not. Wall-clock age would have missed that, because a project can be old and barely used, and it's the use that makes the average mean anything.

Three new tests. The 86-test suite green.

The whole arc took three days. Report on the 14th, the maintainer's pull request open by the 15th, merged on the 16th, and out in v0.9.16 on the 17th — where the release notes say, in plain words, that a young project's founding session is no longer flagged as a cost outlier. If you run CodeBurn, update it and the bug is gone.

What it took

None of this needed luck. It needed not trusting a scary number at face value, then reading the data underneath it. The rest was writing the bug report the way you would want one written to you: the exact code path, the root cause instead of the symptom, and a fix that was ready to review.

The tool was wrong, and it was wrong in a way worth the time it took to explain properly. Silencing the flag locally would have fixed our dashboard. It would have left the bug exactly where it was.

Originally published at thekilted.dev/dashboard-accused-itself.

We were cloning a copy

the kilted dev — Mon, 27 Jul 2026 07:57:44 +0000

A product walkthrough needed a narrator: not a person, a synthesised voice reading a script over the screen recording. Simple enough on paper.

We wanted four things from that voice at once, and it turned out you cannot fully satisfy all four together.

Scottish — because the product carries a real name behind it, and a bland mid-Atlantic voice reading it back would have been wrong from the first sentence.
Natural — not like a robot reading a list.
Local — free and offline, no per-render cloud bill, no dependency on someone else's server staying up.
Obtainable — a thing we could actually get running on this machine, today.

Almost every option we tried hit three of those and quietly broke the fourth. The whole hunt is really about which corner each candidate painted us into, and how the problem eventually stopped having corners at all.

Piper, and the number that came out wrong

We started with Piper, a small, fast, fully local text-to-speech engine, using a voice called en_GB-alba-medium. Alba was the one that mattered: a genuine Scottish speaker. Local, free, offline, right accent. Three of four, immediately.

Two problems. The first was almost funny: Piper read grouped numbers wrong. A figure like 222,000 didn't come out as "two hundred and twenty-two thousand," it came out mangled. We fixed it by rewriting the script instead of the engine: every figure got spelled out as words before it ever reached the synthesiser. That fix turned out to be engine-independent. It survived every voice we swapped in after, which was the first small lesson of the whole hunt.

Fix the input, not the tool, and the fix outlives the tool.

The second problem was the real wall. Individual words were fine; whole sentences were robotic. The rise and fall across a sentence, the prosody, was flat, and that isn't a knob you can turn on Piper. Sentence-level prosody is a hard ceiling of that engine. Getting past it meant leaving Piper, and that's where the corners started.

Five candidates, five different ways to lose

We ran through the alternatives. Every one was rejected, and the interesting part is that no two were rejected for the same reason.

Kokoro-82M, a different local engine, voice bf_emma. Clearer than Piper, genuinely more natural, and judged "really bland and unnatural compared to the alba voice." It won on naturalness and lost on character. We kept it wired in behind a switch, since it worked and there was no reason to delete working code.

F5-TTS, cloning alba. The clever idea: take an engine with far better prosody than Piper and clone the alba voice into it. Best of both, alba's accent riding F5's flow. The result was "brutal, loses the alba and sounds more English," heard instantly. Cloning a synthetic reference, a voice that was itself already generated, flattened the very character we were trying to keep. Getting even that far meant building an isolated Python environment on the CPU, because the machine's GPU stack was too new for the tool.

Won on flow, lost on accent, cost real setup time. Rejected.

Cloud, ElevenLabs. The one path to genuinely Scottish and natural at once. The two free Scottish female voices weren't right, and paying for the better tier wasn't on the table. Parked, not killed. Worth noting the trade would have been softer than it sounds: it's cloud at build time only, you synthesise the script once and the video that ships is a local file. But parked is parked.

RVC voice conversion. The technically correct local route: convert an expressive voice's output into alba's timbre. Research-grade effort, the same environment friction as everything else, and a real risk of coming out muddy. Declined at this stage.

CereProc's "Heather / The Scottish Voice." This one stung, because it was the ideal: a real, high-quality Scottish voice that installs as a normal local system voice. Everything we wanted, except we couldn't get it. The shop that sells it was closed, and the free route needs a Scottish public-sector, education, or NHS email address we don't have. Won on every axis except the one that reads "can you actually obtain it." No.

Five candidates, and between them they covered all four requirements. Just never four at once, never in the same voice.

Voice	Scottish	Natural	Local	Obtainable
Piper (`en_GB-alba-medium`)	✅	❌	✅	✅
Kokoro-82M (`bf_emma`)	❌	✅	✅	✅
F5-TTS (cloning alba)	❌	✅	✅	✅
ElevenLabs (cloud)	➖	✅	❌	❌
RVC (voice conversion)	➖	➖	✅	❌
CereProc ("Heather")	✅	✅	✅	❌

✅ met · ❌ broke · ➖ never established (never run far enough to find out)

The ceiling we accepted

So we stopped chasing and accepted a ceiling. The voice we shipped first was alba again, but with a phrasing pass over the whole script: long strings broken into short breath-group sentences, figures landing at the ends of sentences where the voice naturally slows, the brand name written the way it should sound. A couple of prosody settings pushed to slow it down and loosen it up. The honest verdict was "a step up from robot." Tolerable. It shipped.

If the story ended there, it would be a fine wee story about accepting good-enough. It didn't end there, because of one fact we didn't have yet.

The corpus that was already ours

We found the Alba speech corpus on Edinburgh DataShare. Licensed CC BY 4.0: free to use, fine-tune, even use commercially, as long as you attribute it. And here's what made it matter. It is the actual source corpus behind Piper's alba voice. The speaker we'd been trying to preserve through clone after clone was sitting right there as raw studio recordings: roughly four hours, 4,613 matched audio-and-text pairs, 48kHz, recorded in a hemi-anechoic room, clean.

That reframes the whole problem. Up to this point the question had been "which voice do we pick." Now it was "can we reproduce this specific speaker inside a better engine." Not clone a synthetic echo of her. Train on her actual voice. A different question, with a different and better answer.

Cheapest first, and a failure that explained everything

We staged it cheapest-first: try the free thing before the expensive thing.

Zero-shot — hand the expressive engine (XTTS-v2) a real human clip of alba and ask it to imitate.
Fine-tune — if that failed, actually fine-tune the engine on the corpus.
Cloud — if that failed too, cloud.

Stage 1 failed, as predicted, and the failure was the useful part. Even with a genuine human reference, the voice "wildly drifts off Scottish." Here's why, and it's the structural lesson of the whole hunt: zero-shot cloning copies the timbre but regenerates the prosody from the engine's own base model, and that base model is English-dominant. The accent gets ironed out no matter how good or how long the reference clip is. That single fact retroactively explained the F5 disaster too.

Cloning was never going to hold a Scottish accent. Only training could.

On to Stage 2.

The environment, wall by wall

Fine-tuning a neural voice model on a Windows machine with a consumer GPU is a series of small walls, each of which stops you dead until you find the one specific thing wrong. In order:

The machine's antivirus intercepts SSL, which breaks the package installer for any fresh environment. We had to point it at the antivirus's own certificate to install anything at all. The training library demanded a recent version of a dependency, but the newest version of that dependency had deleted a function the library still called, so we pinned it below the break. The obvious PyTorch build wanted an extra media component that's painful to install on Windows; we pinned an earlier, matched build instead, and even that took force because the installer kept deciding the wrong build was already satisfied and skipping it. On Windows, the training loop has to be guarded with the standard if __name__ == "__main__" incantation, or the parallel data-loading workers each re-run the whole training script and collide on a locked file.

The GPU is an RTX 5060 with 8GB VRAM, but it's also driving the monitor, so only about 5GB was actually free, which forced the smallest possible batch size. And the natural move to save memory, half-precision training, produced loss = nan from the very first step: half-precision overflowed inside the model's forward pass, and only full precision at batch-size-one trained cleanly. The reference recipe we were adapting had quietly omitted half-precision for exactly this reason. It always tells you afterwards.

Every one of those was a full stop until it was solved. None of them was the interesting problem. All of them stood between us and the interesting problem.

The last mile was in the settings, not the training

Then the real limit: training ran at about 8.7 seconds per step, and a complete run would have taken fifteen to eighteen hours. Not happening. It didn't need to finish, though. The adaptation levels off, and around checkpoint 1,000 the voice had clearly turned: "much better, much clearer." The accent held. That was the whole question, whether training could recover the accent that cloning couldn't, and the answer was yes.

Two blemishes were left: a wee stutter, and the first word of each line pitched a bit high. Neither needed more training. They were inference settings: synthesise the text sentence-by-sentence instead of all at once, nudge one sampling temperature. Stutter gone. Along the way we had to correct a belief we'd been carrying: that using the desktop while it generated was corrupting the output. It wasn't. The model's output is deterministic; what the GPU is doing during playback has nothing to do with the saved file. We'd been blaming the environment for a settings problem, which, after a whole hunt of genuine environment walls, is an easy mistake to make and a telling one.

Two loose ends, closed

The voice shipped: a locally fine-tuned Scottish narrator, trained on the real speaker, running offline. But "it works" and "it's finished" are different claims, and two small things after the fact prove it.

First, acronyms came out garbled, the model tried to pronounce them as words. The fix was to spell them phonetically in the script: an initialism became "aitch em ar see," a web address became "guhv dot you kay." That last one has a detail worth keeping: we spelled it "guhv," not "gov," because "gov" gave the voice a long o and we wanted the short flat vowel a person actually uses. The on-screen text was never touched, only what the voice reads.

Second, the finished video sounded quiet. Instead of guessing, we measured it: the audio sat at −26.6 LUFS, where spoken content usually lives around −16. Not subjective. The synthesiser's raw output was just quiet, and nothing in the pipeline had ever corrected for it. One normalisation pass, re-rendered, remeasured: −16.2. On target.

What's still open

The regen script's default engine still points at Piper, not the fine-tuned voice, on purpose: a missing training environment can't hard-fail a regen if the fallback is the simple engine. The good voice ships by setting an explicit flag. Whether to flip that default is a live, unresolved call, not an oversight, and it's still open.

Strip the specifics and it comes down to this. A want with four constraints that don't all fit at once. A row of honest dead ends, each failing for its own real reason. One piece of information, a corpus with the right licence, that turned "pick a voice" into "reproduce a speaker." A gauntlet of environment walls that are exhausting, boring, and mandatory. And at the end, the last defects weren't in the hard thing, the training. They were in the easy thing, the settings, that we'd stopped questioning.

Originally published at thekilted.dev/cloning-a-copy.