DEV Community: the kilted dev

Nine ways to talk to a local model

the kilted dev — Tue, 11 Aug 2026 15:56:51 +0000

A nine-tool survey of local-model interfaces on one GPU: what worked, what silently failed, and
why what sits between you and the model matters more than which model you picked.

Nine pieces of software for talking to a local model went through this machine over about two
weeks. Same GPU, largely the same handful of models, one afternoon each.

The spread in outcomes was enormous, and almost none of it was about the model. The same weights
that scored full marks through one harness produced unparseable garbage through another, and hung
indefinitely through a third. What sits between you and the model turns out to matter more than
which model you picked, which is a boring conclusion until you notice it also means most model
comparisons are measuring something else.

Here is what each one was actually good at.

Tool	Good at	The number that mattered
Ollama	the default, broadly reliable	32.9 t/s gen on a 30B MoE
llama.cpp	fastest, once tuned	27.1 t/s vs Ollama's 23.4
LM Studio	document extraction to file	10 rows, exact match, no invented values
little-coder	small-model coding harness	8/8 in 2.1 min, then a version bump broke it
Unsloth Studio	fastest single-file result	unusable past a 4096-token ceiling
Goose	agent framework, right idea	30% GPU vs 98%, one setting
Claude Code as client	n/a	all three connection methods fail
Odysseus	the most polished interface	0/8, only one of three files actually saved
Continue.dev	autocomplete, nothing else does it	355MB free at 32K, no room for anything else

Ollama, the one everything else is measured against

The default answer, and it stayed the default. It runs a 30B mixture-of-experts model at 32.9
tokens/sec generation and 443 tokens/sec prefill on a 69/31 CPU/GPU split, which is the single
most useful capability on this box.

Two things about it are worth knowing before you trust it. Its desktop application stores a
context-length setting in a local database that overrides the environment variable you set,
which produced a memorable afternoon of a model ignoring configuration that was demonstrably
correct. And model aliases pin their own context, which is how a working alias silently acquired
a context it had never been tested at and started failing weeks later.

Neither is a defect exactly. Both are the kind of thing you only learn by being caught by them.

More: ollama.com

llama.cpp, faster if you're willing to tune it

For one specific job, a 36B mixture-of-experts model at 32K context, llama.cpp beats Ollama by
about 16% on generation and 34% on prefill. 27.1 tokens/sec against 23.4.

That margin isn't free. It came from three separate tuning discoveries: disabling memory-mapped
file loading, which llama.cpp itself warns is slow when combined with CPU tensor overrides and
which alone accounted for most of the gap; setting thread count to the CPU's fourteen physical
cores rather than the eight the script had; and tuning how many expert layers stay on the GPU.

Untuned, the same path runs at 13 tokens/sec, less than half. The default configuration of the
faster runtime is slower than the alternative, which is worth remembering whenever a benchmark
reports that one tool beats another.

llama.cpp's server also has the most complete tool-calling story of anything tested here. Given a
toy arithmetic tool, it emitted five valid parseable calls out of five. Given three near-identical
tools and six questions, it selected correctly six times out of six. That second number is the one
that matters, because choosing between similar tools is where tool-calling usually breaks.

It also has server-side built-in tools, off by default, with the server itself warning against
exposing them to untrusted environments (the available set includes shell execution). Only the
read-only subset was enabled here. Notably, the server does not auto-execute anything: it returns
the tool call and the client drives the loop. That's the right design, and it means the safety
boundary sits where you can see it.

More: github.com/ggml-org/llama.cpp

LM Studio, the graphical one, and the absolute-path rule

Good at exactly one thing this project needed: read some documents, extract structured data, write
the result to a file. Given real utility bills it produced ten rows across two months with every
usage quantity and cost matching the source exactly. No invented values.

Getting there took three attempts. The first two failed with filesystem permission errors even
after access was explicitly approved through its own dialog. The difference was the prompt: "the
files in your workspace" failed, and naming the absolute directory worked.

Whether the model requested bad relative paths or the permission scope failed to resolve them was
never determined. The rule is empirical and it's reliable: give it absolute paths.

More: lmstudio.ai

little-coder, the one that proved the whole thing was possible

A thin harness tuned for small models, giving them real file-writing tools rather than asking them
to print code into a chat window. It's what scored 8 out of 8 on this project's benchmark in 2.1
minutes, including the follow-up-fix-without-regression test that had defeated everything before
it.

Then a version bump broke it. With its default model it began emitting raw XML that nothing
parsed, exiting successfully, and writing no files, the worst failure shape available, because an
exit code of zero and no error output reads as success to anything automated.

The instructive part is the diagnosis, which was wrong the first time. The failure was recorded as
being caused by the version change breaking model-id resolution, because a warning about the model
id appeared next to the failure. A later controlled test held the model constant and varied only
id registration: the warning is benign, and the actual fault is the specific model. The same
harness works with a different one.

Adjacency isn't causation, and a warning that appears next to a failure is still just a warning
that appears next to a failure.

More: github.com/itayinbarr/little-coder

Unsloth Studio, fastest and unusable

The fastest single-file result of anything tested: ten seconds.

Its context window control is broken on Windows. The effective ceiling is 4096 tokens regardless
of what the interface is set to, which makes multi-file work impossible and makes any measurement
taken through it incomparable to anything else. A tool that silently runs at a fraction of the
context you configured is worse than one that refuses, because the number it shows you is a number
you will use.

It also gets flagged as malware by the antivirus on this machine, which is a false positive, and
which is its own small saga.

More: unsloth.ai

Goose, right idea, wrong backend

Goose works here, with one hard rule: use its Ollama provider, never its built-in llama.cpp.

The built-in path manages roughly 30% GPU and 25% CPU utilisation on Windows, a broken offload
that leaves the hardware idle while the model crawls. Through the Ollama provider the same machine
runs at 98% GPU.

Same tool, same model, same box. One configuration choice, and the difference is the entire
value of having a GPU.

More: github.com/block/goose

Claude Code itself, three methods, one root cause

The obvious idea, and the one people ask about most: point the agentic coding tool you already use
at a local model, and stop paying for tokens.

Three connection methods were tested. Setting the API base URL directly, which produces no output
without also setting an auth token to a dummy value. A proxy shim, which worked and ran at roughly
two and a half minutes per response on CPU offload. And the runtime's own native launch command,
which produced a model hallucinating unrelated tasks and emitting raw tool-call syntax as text.

All three fail, and they fail for the same reason: local models do not parse Claude Code's
system prompt format. That's not a configuration problem, and no amount of trying a fourth
connection method changes it. The finding was worth writing down precisely so nobody here burns
another afternoon on connection method five.

More: claude.com/product/claude-code

Odysseus, the most polished tool tested

The most polished thing tested, and the worst result.

It scored 0 out of 8 on the same benchmark. The run climbed from 0.74 to 10.19 tokens/sec, then
stalled dead at "4% complete" with a byte-identical transcript across a five-minute recheck.
Underneath, the backend had dropped the generation stream and was returning 404 to the frontend's
status polls, which the frontend kept making, forever, with no error surfaced, no retry, and no
timeout. A silent, indefinite hang.

Two things compounded it. Its interface exposes no control for disabling thinking mode, which this
project has needed since day one, so the model visibly re-planned and restarted its own draft
mid-stream. And checking its own document panel rather than trusting the chat transcript revealed
that only one of three files had actually been saved. The transcript showed content that was
never written anywhere.

A chat transcript is a record of what a model said, not what a system did, and those two things
diverge silently.

It's also structurally unable to do what this project needs: its tools are a fixed set of built-in
capabilities toggled per message, not a registry you can register an arbitrary function with.
There was no way to run the tool-calling probe at all. Not a bug, a different product.

More: github.com/odysseus-dev/odysseus

Continue.dev, the one that does something nothing else does

An editor extension, evaluated for autocomplete rather than chat, because autocomplete is the one
capability nothing else here provides.

A 4B model completes a fill-in-the-middle hole in 209 to 252 milliseconds at 127 tokens/sec,
comfortably inside the budget where ghost text feels helpful rather than laggy. A 9B does it in
around 328 milliseconds. A 14B coder model, the one nominally built for this, takes 2.5 seconds,
because it spills to CPU, and rambles past the hole into inventing further functions.

The measurement that mattered was not speed, though. With the autocomplete model warm at 32K
context, free video memory drops to about 355MB for the 9B and 1GB for the 4B. There is no room
for anything else. You can have interactive coding with autocomplete, or a background delegation
run. Not both.

More: continue.dev

The comparison we never ran

All of this was supposed to be settled by a formal benchmark. It was written into the handover
document that started this project, as open item one: the word search task run across five
entrants, Goose's built-in backend, Goose via Ollama, llama.cpp, Unsloth Studio and LM Studio,
against one baseline model, with context standardised at 89K on every runtime so the results would
be comparable.

The plan even predicted its own casualty:

Unsloth will fail this, document it.

It was closed without ever being run.

Not from lack of time. By the time it came up for scheduling, its actual question, which runtime
for which job, had already been answered by using them. Ollama for the coding harness. llama.cpp
for the 36B at 32K. LM Studio for graphical document extraction. Goose only via Ollama. Unsloth
ruled out on the context ceiling. The proxy shim never a peer to the others, because it's a
different category of thing entirely.

A generic five-way bake-off would have re-measured memory-spill behaviour already understood, at a
context size chosen to be fair rather than useful, and changed not one of those decisions. The
tools had already differentiated themselves on the only axis that mattered: what happened when
real work went through them.

One footnote, found while checking the above against the original plan rather than against our own
notes. The record written when the benchmark was closed describes it as having six entrants, and
lists a different set, merging Goose's two configurations into one, adding a runtime the plan had
not included, and adding a proxy shim the plan mentions only in its list of things already
rejected. The reasoning for closing it was sound and every per-use-case answer holds. The entrant
list was simply restated from memory rather than reread, in a note written the same week the plan
was still open.

A benchmark nobody ran is a low-stakes thing to miscount. It's the same move that makes a
miscounted one dangerous.

The rule that came out of it is narrow. Benchmark a use case, never a category. If a new runtime
appears, or a genuinely new job appears, measure that, and if the answer is already visible in
work you have done, the benchmark is a formality you are performing for the shape of it.

Nine tools. The differences that mattered were reliability, offload behaviour, honesty about what
was actually written to disk, and whether the thing exposed the one control the model needed. None
of that shows up in a table of tokens per second.

Originally published at thekilted.dev/nine-ways-to-talk-to-a-local-model.

Convergent evidence

the kilted dev — Mon, 10 Aug 2026 18:57:13 +0000

Companion to Route, don't guess: nine tools crossed the desk during the ten-day build. Not one became a dependency. The reasons why are the actual argument for building it in-house.

Building something in-house invites an obvious question: did you check whether someone already shipped it? A companion project, the one that tracks and evaluates AI tooling for the whole machine, kept an eye out for exactly that, every time an orchestration-shaped tool crossed its desk. The bar it held every candidate to was narrow on purpose: does it add a capability the orchestrator doesn't already have?

Almost nothing cleared it. That's less interesting as a verdict than as a pattern.

The direct comparators

Six candidates were orchestrators or multi-agent harnesses solving the same problem: which model does which task, and how do you trust what comes back.

Fable Foreman is parked, worth watching. It sits near one-to-one with our own routing policy: capability-class tiers, blind verification, budget-aware dispatch. The value was design comparison rather than new capability. It was also four days old at review, with two of its three headline features gated behind a call-to-action and never actually shown.
Ruflo went to the backlog. A hundred-plus prebuilt agents, vector memory, multi-model routing, some 210 tool integrations, 58k+ GitHub stars and climbing. What it sells is the routing-plus-verification loop already running here, more simply.
CrewAI skills was rejected on overlap, not quality. It's official plugin scaffolding for an orchestrator, researcher, writer, and reviewer crew, from a legitimate 55.5k-star parent project. The same pattern already runs on the machine's own workflow and routing layers; a third-party framework would duplicate it rather than extend it.
9router was rejected hard. It's a credential-interception proxy for every model provider, carrying an unauthenticated remote-code-execution flaw scored CVSS 10, plus terms-of-service risk for any subscription routed through it. An existing fallback covers the same need safely.
Claude Council was declined by choice. A genuine multi-model consultation plugin, honestly framed, but a lighter version of a panel-of-models pattern already covered elsewhere.
Three "agentic OS" reels were rejected with nothing to install. Camera mockups of an always-on multi-agent life OS. Every named part (a constitution file, tier routing, adversarial sparring, a cost ledger, a budget halt) maps onto pieces already running here, separately and more safely. The bundle is marketing wrapped around five patterns already decomposed on purpose.

The adjacent category

The second cluster doesn't route anything. These are desktop shells for running a fan-out once you've decided on one, checked for an unrelated reason rather than because they compete with the routing policy.

Orca was confirmed hands-on. Free, runs the real CLI under your own subscription; skill and plugin parity held up, and a worktree fan-out plus diff-and-merge round-trip verified end to end on a scratch repo. 12.1k stars.
Nimbalyst went to the backlog. Kanban-style parallel sessions with worktree isolation, parked pending confirmation it plays well with an existing hook-based workflow.
DevSwarm was rejected. Same category as Orca, but closed-source, paid, download-only, and the public repo is a landing page with 87 stars. It doesn't clear a bar a free alternative already cleared.

What cleared the bar wasn't a tool

Two inputs cleared the bar, both as ideas rather than installs, arriving from opposite directions and landing on the same shape.

A stranger's public routing config contributed one genuinely new distinction: taste as an axis separate from difficulty. A component name or a marketing hook is trivial to compute, but still wants the highest-taste model available. A difficulty-only rule would route that down, wrongly. The axis is now a permanent exception in the routing policy. The same source's "manager never runs high reasoning effort" rule was weighed and rejected; it conflicts directly with keeping the terminal review and hardest verification stages deliberately expensive.

And an academic paper on filesystem-as-agent-architecture (numbered stage folders, a per-stage context file, human gates between them) turned out to overlap about 80% with the four-file working-notes system already in use. Independent convergence, this time from arXiv instead of a product pitch.

Neither produced a dependency. Both produced a stronger version of something that already existed.

The closest comparator, side by side

Fable Foreman is worth lining up in full, precisely because it's the closest external thing to the in-house build.

Dimension	Fable Foreman	The in-house orchestrator
Routing basis	Capability class: frontier / workhorse / fast	Verifiability + iteration count, down a cost ladder
Model naming	Aliases resolved at runtime	Same. Tier by role, never a hardcoded model ID
Verification	"Trusts no one": blind verify against the real workload	Same posture. Never trust a subagent's "done" without evidence
Budget awareness	Budget-aware dispatch, degradation rule	Append-only ledger plus a budget halt
Design/review routing	Not specified	Never routes down. A standing, explicit rule
Trust surface	Four days old, spawns full agent fleets, content gated behind a CTA	In-house, auditable, every piece proven separately

The market kept independently rediscovering the same routing model: a stranger's config file, an academic paper, three viral reels, and a four-day-old repo that maps onto our own build dimension for dimension. None became a dependency, because the same five patterns were already decomposed into separable, auditable pieces before any of them showed up.

The actual takeaway

A single rejected tool proves nothing; every project rejects things. What's worth noticing is the shape of nine candidates checked and not one dependency adopted. Not one offered a capability the routing layer didn't already have, and the two that offered something real turned out to be ideas rather than products. That's what you'd expect if the underlying problem (route by verifiability, verify blindly, cap the budget, keep a human in the loop) is well-understood enough that independent people keep arriving at the same four or five pieces.

The discipline wasn't refusing to look. It was refusing to trade five pieces we can audit for one box we can't. This doesn't claim there were no more tools worth checking. Only that everything that crossed the desk was checked against the same bar, and the bar held.

Originally published at thekilted.dev/convergent-evidence.

Route, don't guess

the kilted dev — Mon, 10 Aug 2026 15:20:58 +0000

Field notes from the machine: how we built the orchestrator that routes each task to the optimal model, and the near miss that drew the line on what it's allowed to act on.

Every Claude Code session defaults to the model it was opened with. Ask it to rename a variable or design a whole subsystem and, left alone, it reaches for the same tier both times. That's fine until the quota notices. Tokens and time are one budget, and the top tier spends both fastest. Waste them on a variable rename and they aren't there for the subsystem. The obvious fix is a rule of thumb: low-cost model for easy stuff, expensive model for hard stuff. It sounds right. It's wrong, and working out why it's wrong is most of what this project turned out to be.

What it actually decides

The orchestrator is a routing layer, not another agent framework. It lives as markdown: a contract file, a handful of dispatch templates, an append-only ledger, all read by a session before it delegates anything. No installed dependency, no external scheduler. The bet, made on day one, was that Claude Code's own primitives were already enough, and the missing piece was a policy rather than a platform.

The policy's core move is refusing "hard versus easy" as the routing variable. What matters is task shape: how many correction rounds the work needs, and whether a given round can be checked objectively.

Route down — one-shot, objectively checkable: tests exist, or the output validates against a schema. Push it down the cost ladder, through scripts and local models before the low-cost hosted tiers.
Stays top — iterative and judgment-heavy: design, debugging, architecture, review. Stays with the top-tier model, in-session.

Underneath that split sit three costs the "low-cost tier for easy tasks" instinct never prices in:

Cold-start tax — every delegated session re-derives context from nothing, so handing off a five-minute edit loses outright.
Review tax — whoever dispatched the work still pays to read whatever comes back, and a chatty response from a low-cost tier can erase its own savings.
Retry tax (dominant) — low-cost models fail more often at anything with judgment in it, and every failure means a redo.

The rule that survives all three isn't "is this task hard". It's whether the expected saving beats the expected cost of doing it again. Iteration-heavy work maximises the wrong side of that inequality no matter how easy it looks.

What never routes down

Four categories are pinned to the top tier permanently, however simple the task looks in isolation.

Taste-sensitive output is the first: design, copy, naming, brand voice. A component name is trivial to compute and still wants the model with taste. It turns out "easy" and "low-stakes" are different axes, an idea that arrived from outside the project and earned its place.

The terminal review is the second. Per-task reviews can run on a mid tier, but the whole-branch pass before anything ships always runs top-tier, because it catches what the narrower passes miss. A pass on a low-cost tier once sailed past a real security hole that a stronger model then caught. The rule dates from that day.

Third, a green light from a low-cost tier gates a dispatch but never substitutes for verification. Low-cost models have a structural habit of reporting success they haven't earned.

Fourth, the orchestration layer itself. A low-cost model was tried as middle manager between the session and a worker. Sixty minutes and roughly fifty tool calls later it had shipped nothing and quietly run a global install nobody asked for. Delegation stays a command the session issues directly.

› echo "nobody gets a middle manager"

Proof over promises

Every delegated run ends the same way: a status block in the chat, and a row appended to a ledger file. Neither trusts a bare claim. A "done" needs the tool output, test result, or screenshot sitting next to it.

The ledger isn't decoration. Each row records what was routed where, whether it passed, and the field that matters most: any lesson. A populated lesson is a candidate for promotion into the routing rules themselves. The policy tightens from its own field evidence, not just from design sessions.

The near miss

The hardest lesson didn't come from a cost overrun. It came from a decision that would have run correctly, in the wrong place.

A "do it everywhere at once" clause, written into one project's own decision log, would have had that project's session run git rm --cached against a sibling project's repository, one it had no business touching. The user caught it before it executed. Nothing was lost.

But it exposed the real default across every project on the machine: any session could offer to edit a neighbouring project's files, and the offer usually looked reasonable.

The fix wasn't a smarter check. It was inverting the default.

A session now treats every other project's tree in three bands:

Read (always allowed) — reading a sibling's files is always allowed, since no desync is possible.
Propose (the only sanctioned cross-project write) — writing a note into the sibling's root asking its own next session to decide.
Mutate (denied by default) — editing a sibling's facts or running commands in its repo. Two narrow carve-outs: files whose whole purpose is to receive writes, like the routing ledger, and gated edits to genuinely shared config, which always name their blast radius and wait for sign-off.

Everything else routes through a proposal note. It suggests; it never asserts. The receiving project's own next session folds it into its own files, then deletes it. No queue, no index, no daemon. Three ad hoc uses proved the pattern before anyone wrote it down as a rule.

How it came together

Eleven days, roughly a session a day, each one closing with the same four working files (notes, memory, active constraints, rejected approaches) so the next session opens cold and still knows why past decisions landed where they did.

Day 1 — Founded on native primitives. External orchestration frameworks rejected in favour of Claude Code's own agent files, skills, and hooks. First trial ran in a throwaway project on purpose, for zero regression risk while the design was unproven.
Day 2 — The ladder tested for real. A wiki migration ran as the first live trial, with a script, a local model, and two hosted tiers all doing genuine work in one pipeline. The layered-review rule landed that same week, after a low-cost pass missed that security hole.
~Same week — First shipped consumer. A pension calculator's feature became the first real work routed end-to-end through the policy. It surfaced the first known gap: the local tier only speaks Python, and this task was JavaScript. Not a misroute, a disqualification, but a gap worth naming.
Day 4 — Outside validation. An unrelated public routing framework, reviewed independently, converged on the same core shape from a completely different workload. We folded in its one genuinely new idea: taste as a routing axis distinct from difficulty. We rejected its "never run high reasoning effort" rule, since it conflicts with exactly the stages this project keeps expensive on purpose.
Day 6 — Cross-project messaging formalised. Three ad hoc successes generalised into the proposal-note convention.
Day 7 — The near miss, and the contract it produced. Default-deny on cross-project mutation became a standing, machine-wide rule.
Day 10 — Fleet-wide rollout. The enforcement line, the one that makes a project's session actually consult the policy instead of silently defaulting, was offered across around eighteen sibling projects. Opt-in per project, tracked from the filesystem alone.
Day 11 — The ledger checks itself. A staleness audit found most adopted projects had quietly stopped logging, so a hook now nudges emission at session end. The same day, a reviewer on a low-cost tier blamed an unrelated, pre-existing file change on the task it was checking. Comparing timestamps caught it, and it became a standing rule: a low-cost tier may scan, but attributing a found change to a cause needs a timestamp check first, not a guess.

What's still open

Three gaps stay visible on purpose. The local tier only routes Python-shaped work, so JavaScript tasks run top-tier by default until a second real JS consumer justifies building a driver. Reasoning effort never entered the model at all: the policy routes on which tier runs a task, not on how hard that tier is told to think, and the same model at a different effort setting is a different cost and arguably a different capability. That dimension is unexamined, not settled. And one mid-tier code-generation route sits in the ladder unproven: named, priced, ready, and untouched, because no task has gone through it yet.

A rung nobody has used is not the same as a rung that works.

Nothing above is theory. Every rule came from work that actually ran. The whole apparatus is a bet that the interesting problem was never "which model is smartest". It was knowing, task by task, when smartest is the wrong thing to be paying for.

Companion piece: Convergent evidence. Nine orchestration-shaped tools crossed the desk while this was being built, and not one became a dependency.

The policy itself, extracted and de-identified: the routing-policy repo.

Originally published at thekilted.dev/route-dont-guess.

The dashboard accused itself

the kilted dev — Sun, 02 Aug 2026 20:31:11 +0000

Our own cost dashboard flagged the day it was born as a 26.5x overspend. The transcript said otherwise. Making sure it never happened to anyone else took a bug report, and three days later the fix shipped.

We build a lot of small internal dashboards. One of them tracks Claude Code token spend across the projects on this machine. It leans on an open-source engine called CodeBurn: free, local-only, reads session logs and estimates what each session would have cost on pay-as-you-go pricing. CodeBurn has a panel called optimize, and the panel's job is to flag waste. Sessions that cost far more than a project's own average, with no obvious reason why.

One day the panel flagged something inside its own house. The dashboard's build session, the day it came into existence, sat there marked as a 26.5x cost outlier. Twenty-six and a half times the project's own average. That is the kind of number designed to make you flinch.

We didn't flinch. We pulled the raw transcript.

What was actually in there

It ran to 1,070 events. A brainstorm, a plan, twenty-four build agents dispatched, and a final commit that promoted the dashboard to its first real version. That was the founding session, and founding sessions are expensive. The founding session alone ran 1,045,640 tokens, against an average of 62,940 across twelve sibling sessions that followed — a 16.6x gap on tokens, before cost pricing even enters it.

The day a project goes from nothing to something costs more than any day after it. A house's foundation costs more than repainting one room, and nobody files that under waste. The panel had measured the founding day against the quiet maintenance days that followed and called the ratio waste. That's a category error rather than a measurement problem. The comparison was never apples to apples, and no cost threshold repairs a comparison that was wrong before the arithmetic started.

So the panel was right to ask the question. It was wrong about the answer.

The two easy exits

We could have ignored the flag. We could have silenced it locally and moved on, which would have worked fine, for us. It would also have left the bug sitting there for everyone else running CodeBurn against a young project, which is every project at some point.

So we read CodeBurn's source instead. The comparison lives in a function called detectSessionOutliers, in src/optimize.ts. Reading it turned up the useful part: the codebase already had the mechanism a fix would need. A wee exclusion list called outlierExclusions, already doing this exact kind of job elsewhere in the panel, sitting right there.

That gave us something better to file than a complaint. Issue #664: what happens, why it's wrong, and the specific lines to change.

The maintainer's fix was better than ours

CodeBurn has an active maintainer, and the pull request was open by the next morning.

Our sketch excluded every project's founding session, permanently. The maintainer's version was sharper. It excludes the founding session only while a project is young, and it measures young by how many costed sessions the project has behind it rather than by wall-clock age.

That distinction is the whole fix. Under our version, a genuinely wasteful founding session would have hidden behind the exemption forever. Under the maintainer's, the benefit of the doubt expires after six costed sessions. A founding session in a two-week-old project gets it. The same session in a mature project with months of history does not. Wall-clock age would have missed that, because a project can be old and barely used, and it's the use that makes the average mean anything.

Three new tests. The 86-test suite green.

The whole arc took three days. Report on the 14th, the maintainer's pull request open by the 15th, merged on the 16th, and out in v0.9.16 on the 17th — where the release notes say, in plain words, that a young project's founding session is no longer flagged as a cost outlier. If you run CodeBurn, update it and the bug is gone.

What it took

None of this needed luck. It needed not trusting a scary number at face value, then reading the data underneath it. The rest was writing the bug report the way you would want one written to you: the exact code path, the root cause instead of the symptom, and a fix that was ready to review.

The tool was wrong, and it was wrong in a way worth the time it took to explain properly. Silencing the flag locally would have fixed our dashboard. It would have left the bug exactly where it was.

Originally published at thekilted.dev/dashboard-accused-itself.

We were cloning a copy

the kilted dev — Mon, 27 Jul 2026 07:57:44 +0000

A product walkthrough needed a narrator: not a person, a synthesised voice reading a script over the screen recording. Simple enough on paper.

We wanted four things from that voice at once, and it turned out you cannot fully satisfy all four together.

Scottish — because the product carries a real name behind it, and a bland mid-Atlantic voice reading it back would have been wrong from the first sentence.
Natural — not like a robot reading a list.
Local — free and offline, no per-render cloud bill, no dependency on someone else's server staying up.
Obtainable — a thing we could actually get running on this machine, today.

Almost every option we tried hit three of those and quietly broke the fourth. The whole hunt is really about which corner each candidate painted us into, and how the problem eventually stopped having corners at all.

Piper, and the number that came out wrong

We started with Piper, a small, fast, fully local text-to-speech engine, using a voice called en_GB-alba-medium. Alba was the one that mattered: a genuine Scottish speaker. Local, free, offline, right accent. Three of four, immediately.

Two problems. The first was almost funny: Piper read grouped numbers wrong. A figure like 222,000 didn't come out as "two hundred and twenty-two thousand," it came out mangled. We fixed it by rewriting the script instead of the engine: every figure got spelled out as words before it ever reached the synthesiser. That fix turned out to be engine-independent. It survived every voice we swapped in after, which was the first small lesson of the whole hunt.

Fix the input, not the tool, and the fix outlives the tool.

The second problem was the real wall. Individual words were fine; whole sentences were robotic. The rise and fall across a sentence, the prosody, was flat, and that isn't a knob you can turn on Piper. Sentence-level prosody is a hard ceiling of that engine. Getting past it meant leaving Piper, and that's where the corners started.

Five candidates, five different ways to lose

We ran through the alternatives. Every one was rejected, and the interesting part is that no two were rejected for the same reason.

Kokoro-82M, a different local engine, voice bf_emma. Clearer than Piper, genuinely more natural, and judged "really bland and unnatural compared to the alba voice." It won on naturalness and lost on character. We kept it wired in behind a switch, since it worked and there was no reason to delete working code.

F5-TTS, cloning alba. The clever idea: take an engine with far better prosody than Piper and clone the alba voice into it. Best of both, alba's accent riding F5's flow. The result was "brutal, loses the alba and sounds more English," heard instantly. Cloning a synthetic reference, a voice that was itself already generated, flattened the very character we were trying to keep. Getting even that far meant building an isolated Python environment on the CPU, because the machine's GPU stack was too new for the tool.

Won on flow, lost on accent, cost real setup time. Rejected.

Cloud, ElevenLabs. The one path to genuinely Scottish and natural at once. The two free Scottish female voices weren't right, and paying for the better tier wasn't on the table. Parked, not killed. Worth noting the trade would have been softer than it sounds: it's cloud at build time only, you synthesise the script once and the video that ships is a local file. But parked is parked.

RVC voice conversion. The technically correct local route: convert an expressive voice's output into alba's timbre. Research-grade effort, the same environment friction as everything else, and a real risk of coming out muddy. Declined at this stage.

CereProc's "Heather / The Scottish Voice." This one stung, because it was the ideal: a real, high-quality Scottish voice that installs as a normal local system voice. Everything we wanted, except we couldn't get it. The shop that sells it was closed, and the free route needs a Scottish public-sector, education, or NHS email address we don't have. Won on every axis except the one that reads "can you actually obtain it." No.

Five candidates, and between them they covered all four requirements. Just never four at once, never in the same voice.

Voice	Scottish	Natural	Local	Obtainable
Piper (`en_GB-alba-medium`)	✅	❌	✅	✅
Kokoro-82M (`bf_emma`)	❌	✅	✅	✅
F5-TTS (cloning alba)	❌	✅	✅	✅
ElevenLabs (cloud)	➖	✅	❌	❌
RVC (voice conversion)	➖	➖	✅	❌
CereProc ("Heather")	✅	✅	✅	❌

✅ met · ❌ broke · ➖ never established (never run far enough to find out)

The ceiling we accepted

So we stopped chasing and accepted a ceiling. The voice we shipped first was alba again, but with a phrasing pass over the whole script: long strings broken into short breath-group sentences, figures landing at the ends of sentences where the voice naturally slows, the brand name written the way it should sound. A couple of prosody settings pushed to slow it down and loosen it up. The honest verdict was "a step up from robot." Tolerable. It shipped.

If the story ended there, it would be a fine wee story about accepting good-enough. It didn't end there, because of one fact we didn't have yet.

The corpus that was already ours

We found the Alba speech corpus on Edinburgh DataShare. Licensed CC BY 4.0: free to use, fine-tune, even use commercially, as long as you attribute it. And here's what made it matter. It is the actual source corpus behind Piper's alba voice. The speaker we'd been trying to preserve through clone after clone was sitting right there as raw studio recordings: roughly four hours, 4,613 matched audio-and-text pairs, 48kHz, recorded in a hemi-anechoic room, clean.

That reframes the whole problem. Up to this point the question had been "which voice do we pick." Now it was "can we reproduce this specific speaker inside a better engine." Not clone a synthetic echo of her. Train on her actual voice. A different question, with a different and better answer.

Cheapest first, and a failure that explained everything

We staged it cheapest-first: try the free thing before the expensive thing.

Zero-shot — hand the expressive engine (XTTS-v2) a real human clip of alba and ask it to imitate.
Fine-tune — if that failed, actually fine-tune the engine on the corpus.
Cloud — if that failed too, cloud.

Stage 1 failed, as predicted, and the failure was the useful part. Even with a genuine human reference, the voice "wildly drifts off Scottish." Here's why, and it's the structural lesson of the whole hunt: zero-shot cloning copies the timbre but regenerates the prosody from the engine's own base model, and that base model is English-dominant. The accent gets ironed out no matter how good or how long the reference clip is. That single fact retroactively explained the F5 disaster too.

Cloning was never going to hold a Scottish accent. Only training could.

On to Stage 2.

The environment, wall by wall

Fine-tuning a neural voice model on a Windows machine with a consumer GPU is a series of small walls, each of which stops you dead until you find the one specific thing wrong. In order:

The machine's antivirus intercepts SSL, which breaks the package installer for any fresh environment. We had to point it at the antivirus's own certificate to install anything at all. The training library demanded a recent version of a dependency, but the newest version of that dependency had deleted a function the library still called, so we pinned it below the break. The obvious PyTorch build wanted an extra media component that's painful to install on Windows; we pinned an earlier, matched build instead, and even that took force because the installer kept deciding the wrong build was already satisfied and skipping it. On Windows, the training loop has to be guarded with the standard if __name__ == "__main__" incantation, or the parallel data-loading workers each re-run the whole training script and collide on a locked file.

The GPU is an RTX 5060 with 8GB VRAM, but it's also driving the monitor, so only about 5GB was actually free, which forced the smallest possible batch size. And the natural move to save memory, half-precision training, produced loss = nan from the very first step: half-precision overflowed inside the model's forward pass, and only full precision at batch-size-one trained cleanly. The reference recipe we were adapting had quietly omitted half-precision for exactly this reason. It always tells you afterwards.

Every one of those was a full stop until it was solved. None of them was the interesting problem. All of them stood between us and the interesting problem.

The last mile was in the settings, not the training

Then the real limit: training ran at about 8.7 seconds per step, and a complete run would have taken fifteen to eighteen hours. Not happening. It didn't need to finish, though. The adaptation levels off, and around checkpoint 1,000 the voice had clearly turned: "much better, much clearer." The accent held. That was the whole question, whether training could recover the accent that cloning couldn't, and the answer was yes.

Two blemishes were left: a wee stutter, and the first word of each line pitched a bit high. Neither needed more training. They were inference settings: synthesise the text sentence-by-sentence instead of all at once, nudge one sampling temperature. Stutter gone. Along the way we had to correct a belief we'd been carrying: that using the desktop while it generated was corrupting the output. It wasn't. The model's output is deterministic; what the GPU is doing during playback has nothing to do with the saved file. We'd been blaming the environment for a settings problem, which, after a whole hunt of genuine environment walls, is an easy mistake to make and a telling one.

Two loose ends, closed

The voice shipped: a locally fine-tuned Scottish narrator, trained on the real speaker, running offline. But "it works" and "it's finished" are different claims, and two small things after the fact prove it.

First, acronyms came out garbled, the model tried to pronounce them as words. The fix was to spell them phonetically in the script: an initialism became "aitch em ar see," a web address became "guhv dot you kay." That last one has a detail worth keeping: we spelled it "guhv," not "gov," because "gov" gave the voice a long o and we wanted the short flat vowel a person actually uses. The on-screen text was never touched, only what the voice reads.

Second, the finished video sounded quiet. Instead of guessing, we measured it: the audio sat at −26.6 LUFS, where spoken content usually lives around −16. Not subjective. The synthesiser's raw output was just quiet, and nothing in the pipeline had ever corrected for it. One normalisation pass, re-rendered, remeasured: −16.2. On target.

What's still open

The regen script's default engine still points at Piper, not the fine-tuned voice, on purpose: a missing training environment can't hard-fail a regen if the fallback is the simple engine. The good voice ships by setting an explicit flag. Whether to flip that default is a live, unresolved call, not an oversight, and it's still open.

Strip the specifics and it comes down to this. A want with four constraints that don't all fit at once. A row of honest dead ends, each failing for its own real reason. One piece of information, a corpus with the right licence, that turned "pick a voice" into "reproduce a speaker." A gauntlet of environment walls that are exhausting, boring, and mandatory. And at the end, the last defects weren't in the hard thing, the training. They were in the easy thing, the settings, that we'd stopped questioning.

Originally published at thekilted.dev/cloning-a-copy.