10 seconds per character 😬. That's what one of the big local LLMs I tried gave me, the first time I took local models seriously. Unusable, plain and simple.
I dropped the idea for a while and went back to the API. Then a video about distilling Chinese models 🤓 made me want to run the test again, this time on small models instead of big ones. The question that came out of it: can these things, a few hundred megabytes to a few gigabytes, actually replace an API call on my tasks, or does it only work on a narrow slice of what I do every day.
10 Seconds Per Character, Then 1 Video
That first attempt wasn't a fluke. I loaded a big open model on hardware that had no business running it, and watched it type a single sentence slower than I could make coffee. I closed the terminal and didn't touch local inference for months.
What brought me back wasn't a benchmark, it was a video walking through how a Chinese lab distilled a much smaller student model from a bigger teacher and kept most of the accuracy on a narrow task. That's a different game than "run a 70B on a laptop." The question stopped being "can I run a big model locally" and became "can a small model, trained on exactly what I need, replace the API call I'm making right now."
Five tasks later, I had an answer. Not the one I expected.
The Contract, Not the Model Size
A small model isn't a pocket-sized ChatGPT. It's a tool with edges, and the edges are the point.
3 conditions have to hold at the same time for a small model to be worth the setup. The input has to be bounded (a form field, an extract, a record, not an open prompt). The output has to be bounded too (JSON, a label, a URL or null, a score, something with a fixed shape). And the correctness has to be checkable objectively, not "does this sound right" but "is this SIREN number the one on the invoice, yes or no."
When all 3 hold, a model between 0.6B and 7B parameters is usually enough. When even one doesn't, no amount of prompt engineering saves the task. This isn't a tuning problem you iterate your way out of, it's a structural fit question you answer before writing a single line of training code. Getting it wrong means you'll spend weeks polishing a prompt for a job the model was never going to be able to do, which is a more expensive mistake than it sounds like from the outside.
Local also means something beyond the contract. Data that doesn't leave the machine. Cost that moves from a per-call bill to RAM and GPU time you already own. And full control over what the model is allowed to say when it doesn't know, which for API models usually means guessing and for a model you trained yourself can mean an honest null.
A small model doesn't need to be smart, it needs to be right on a narrow slice, every time.
Skip that step and the whole approach falls apart: writing the contract down before you build anything. Same discipline, different layer.
And no, the model doesn't want to take over the world. It wants to extract a company ID and go back to sleep. No Skynet moment required.
5 Tasks Where the Small Model Won
Extracting a company ID, name, and city from a raw text block. In the US that's usually an EIN buried in an invoice or a filing, in France it's a SIREN. The input stays inside a tight box and the output does too, so checking whether the answer is right is a lookup, not a judgment call. A 1.5B model fine-tuned on about 150 labeled examples went from guessing right half the time to landing north of 90%.
Picking the right LinkedIn /in/ profile out of a page of Google results, or returning nothing when none of them match. This one's sneaky because the failure mode of a big model here is confident wrong answers, and a small model trained to say "none of these" is worth more than one that always picks something. A 3B model trained on roughly 200 examples cut the wrong-pick rate by more than half, mostly by learning when to abstain instead of guessing.
Sorting mail into invoice, follow-up, spam, or other. 4 labels in a closed set, and Karen from Accounting doesn't need to touch it. A 0.6B model trained on about 120 examples landed north of 95% accuracy, which is overkill for a task this narrow but the model barely notices the extra weight.
Random aside: half these tests ran during a home renovation with a compressor going 2 rooms over. Turns out that's less distracting than a Slack notification popping on the second monitor.
Generating uncensored text on a narrow, bounded task where an API provider's content filter kept getting in the way of something entirely legitimate. Small local model, no filter, no ticket to support explaining why I need it. No fine-tuning needed here, just a 7B base model running with the guardrails off, which turned a multi-day support back-and-forth into zero.
Better grep. Semantic search over logs or a codebase, running locally, no round trip to an API for something that's really just "find me the thing that means this." Swapping an API embedding call for a local model dropped lookup time from a couple seconds to under 100 milliseconds, on a search I run dozens of times a day.
5 for 5 isn't a coincidence, it's the contract holding 5 times in a row. Which makes you wonder where it stops holding.
LoRA, Distillation, or Just a Better Prompt
Behind each win above sits a different technical decision. Sometimes a prompt alone did the job. Sometimes I had to graft an adapter onto the base model to get it to behave.
LoRA is frozen adapters layered on top of a base model, a few megabytes, loaded at inference time. Distillation is a teacher (me, a bigger LLM, or some mix of both) labeling 100 to 500 examples, and the small model learning to copy the pattern. What gets distilled here isn't intelligence, it's a decision policy: when to double-check, when to answer "name only," when to return null instead of guessing.
Distillation isn't teaching a model to think. It's teaching it when to shut up and say null.
The decision tree I ended up using: prompt alone if the task is already easy for the base model. LoRA if the prompt drifts, invented URLs, wrong homonym picked. Distillation plus LoRA if I have a teacher and enough examples to label. Big model or API for the rare cases and anything that needs open reasoning.
First LoRA run: dead on arrival. You died, no checkpoint, 3 more hours of training. Wrong loss mask, the whole run wasted on learning to repeat the prompt back to me.
On a Mac, that's mlx_lm.lora --train --mask-prompt. The --mask-prompt flag is the one that matters, it makes sure the loss only applies to the answer, not to the context you fed it. Skip that flag and the model gets very good at echoing your input and not much else.
The same idea applied to automation tooling shows up in a completely different context, but it's the same instinct: don't reach for the biggest tool when a small, well-scoped one does the job.
Where It Breaks: Long Creative French Text
A 0.6B model was never going to write a coherent short story, obviously, but I wanted to see how badly it would fail.
Badly. Loops that repeat the same paragraph structure 3 times in a row. Adverbs stacking up like the model forgot it already used "soudainement" twice on the same page. Grammar mistakes that a spellchecker catches in half a second. And, a few hundred words in, the model quietly switching to English mid-sentence, like it forgot which language it was supposed to be writing.
This isn't a settings problem. I tried different temperatures, different system prompts, different adapters. None of it fixed the structural issue: open-ended, long-form creative generation in a language other than English is exactly the kind of task that fails the contract on all 3 counts at once. Unbounded input, unbounded output, and no objective way to check if a sentence is "good" beyond reading it yourself.
What Still Needs the API
What works is easy to state: an input you can put a box around, an output with a fixed shape, and a correctness check nobody has to eyeball. On that slice, a small model beats an API call on cost, latency, and control, every time I tested it.
What's still open, I'll say plainly instead of hedging around it. Maintaining 5 or 6 different LoRA adapters over a year, I don't have a clean answer for what that costs in upkeep. Honestly not sure if it saves more than it costs in babysitting (I think it does, but ask me again in 6 months). And the quality of a distilled model depends entirely on whoever labeled the training examples, which means the risk doesn't disappear, it just moves upstream to whoever's playing teacher.
Anything that needs open reasoning, long context, or judgment calls a human would argue about, that still goes to the API. No HAL 9000 moment where the small model refuses the request, it just quietly gives you a wrong answer with the same confidence as a right one, which is worse.
5 tasks, 1 pattern held. The sixth one will probably break it, and I haven't found it yet.
Sources
- Thread on X (@theahmadosman) reporting a Reddit r/LocalLLaMA distillation result: a 0.6B model on a Text2SQL task went from 36% accuracy to 74% after distillation on roughly 100 examples
This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure.)
Top comments (0)