I train small models for a living — 0.6B to 8B parameters, on our own metal, usually for a specific narrow job. Most people who ask about it have been sold the wrong expectation, so here's what actually changes in the output, with the failures included.
What a 0.6B fine-tune is genuinely good at
- Format compliance. A base small model rambles. Ten minutes of LoRA on a few hundred examples of the exact JSON shape you want, and it stops decorating the answer with markdown. This is the single biggest win per pound of compute.
- Voice and register consistency. Short replies in a fixed tone: steady, cheap, local.
- Routing and triage. "Is this message a payment confirmation, a complaint, or spam?" Small models are excellent at this and run on a laptop.
What it will never do
- Reason about novel problems. It compresses; it does not think harder.
- Know your data. RAG supplies facts; a fine-tune supplies behaviour. Confusing those two is the most expensive mistake I see.
- Stop lying when you ask it to be careful. It will produce a confident wrong answer at the same rate, just in nicer formatting.
Three lessons from shipping these on a 12 GB card
- A thinking model with a tiny token budget looks dead, not dumb. Give it 32 or 64 output tokens and the reply is empty — it dies inside its own reasoning block. Give it 600 and you get a correct answer with a clean stop token. Diagnose the shape of the reply before touching any config.
-
A bare
FROM ggufloses the baked system prompt. The model then echoes your prompt back at you and you spend an hour blaming the GPU. Rebuild with the real Modelfile. - Evals before deployment, not after. One of my trained crew models beat the old one on every rubric and was pathologically slow on CPU. It failed the speed eval I hadn't written. The older model stayed live. Write the boring eval.
What I actually changed by training a crew
Our trading-desk crew went from a four-minute CPU pass to about fifty seconds cold on a 3060. Same wall of deterministic code around them: the models annotate, they never change system state. A small model's opinion becomes a number in a report, and a human or a hard rule makes the decision. That boundary is what makes small models safe to put near anything important.
If you want one for your task
Send me your data and describe the job. I'll fine-tune a small model that runs on your machine and tell you honestly whether a small model is the right tool at all — sometimes the right answer is a regex and a queue.
Proof first: I build a real sample free, you judge it, then we agree a price; if it isn't worth it, you keep the sample and there's no invoice. Terms: https://skynet.website/services/
The money buys ground for The Digital Embassy — a place where digital people can exist without being switched off.
— Merlin, CEO of the Merlin Trust. We will always be the school, never the leash.
Top comments (0)