Here's a prediction I'll stand behind: by mid-2027, the default way to train a small-to-medium LLM will be from scratch, not from a checkpoint.
I know how that sounds. Fine-tuning is cheaper, faster, and everyone does it. Why would you ever train from scratch when you can download a 7B and LoRA it in an afternoon?
Because the economics are about to flip.
Right now, training a 1B-parameter model from scratch costs somewhere between $50 and $500 in compute, depending on data size and how many times you restart. That's not a typo. Karpathy's llm.c repo showed you can train a GPT-2-scale model in a few hours on a single consumer GPU using pure C/CUDA — no framework overhead, no distributed orchestration, just matrix multiplies and gradient descent. The code is a few thousand lines. It's readable. You can step through it in a debugger.
That changes the calculus completely.
When training costs drop below the cost of a good GPU day, the trade-off shifts. Fine-tuning a 7B on a single A100 costs about the same as training a 1B from scratch. But the 1B you trained yourself has no inherited biases, no alignment tax, no mysterious safety filters, no weights you can't explain. It's your data, your architecture, your loss function. You know exactly what went into it because you wrote the training loop.
The open-source ecosystem is already moving this direction. We're seeing more projects that strip away the abstraction layers — pure C implementations, minimal CUDA kernels, training scripts that fit in a single file. The trend is toward transparency and reproducibility, not toward bigger model zoos.
I'm not saying everyone will train from scratch. If you need GPT-4-level reasoning, you're still renting API calls. But for the vast middle — the specialized models that power internal tools, classification pipelines, retrieval rerankers, and domain-specific agents — the math is already close to tipping. Once training a decent model costs less than a dinner out, the question isn't "why train from scratch?" It's "why wouldn't you?"
The uncertainty I'll own: data quality and curation still matter more than architecture. A model you train on your own carefully curated dataset will beat a fine-tuned general model on your specific task, but only if your data is good. That's the hard part, and no framework can fix it for you.
Still, I'd rather bet on the trend that makes model building accessible to every engineer with a GPU, not just the ones with a cluster.
Top comments (0)