Next-token prediction is an objective, not a restriction that a model must only count neighboring words. A tiny model makes one part of the distinction visible.
A model we can fully inspect
We supply the structure y_hat = w*x and train only w. Examples use x=3 with target 6 and x=5 with target 10. The model does not discover multiplication: multiplication is built in.
With squared loss (w*x-y)^2, the gradient is 2*x*(w*x-y). For this demonstration, a learning rate of 1/(4*x*x) halves the distance from w to 2 at every step.
Starting from w=1, the first updates produce 1.5 and 1.75. After 10 updates, w=1.9990234375.
Try an unseen input
For x=7, without further updates, the output is 13.9931640625, approximately 14. There was no training pair 7→14: the model reused the relation encoded in its parameter.
Connecting the idea to language models
LLMs have much more complex architectures. Mathematical text and worked examples can influence their shared parameters during prediction training. Controlled arithmetic experiments show that transformers can learn arithmetic under this objective; formatting and intermediate steps matter, and length generalization remains challenging.
Generated intermediate results can become subsequent input and help with multi-step computation. They can also be wrong; a written explanation is not automatically a faithful account of internal computation.
The toy model illustrates parameter learning, not the complete mechanism of an LLM or a guarantee of a universal math algorithm.
Animation
A 2-minute animation with Chinese subtitles, music and no narration accompanies this post. All network diagrams are teaching illustrations.
Top comments (0)