I built a GPT from scratch in NumPy only – 34 notebooks, no PyTorch

Steven Li — Thu, 12 Nov 2026 15:52:53 +0000

I wanted to understand LLMs below the framework level, so I wrote a 34-chapter tutorial that starts from a single neuron and builds up to a ~10.8M-parameter GPT — tensors, computational graphs, autograd, attention, pre-training, sampling, KV cache, SFT and DPO — using nothing but NumPy. Every chapter is a runnable Jupyter Notebook, everything trains on CPU, and the book is free in English and Chinese.

How is this different from nanoGPT / micrograd?

Those projects are excellent and this book stands on their shoulders. The difference is scope and starting point: it starts from neural-network basics, builds its own tensor / autograd framework in NumPy, and only then gets to GPT — covering pre-training, sampling, KV cache, SFT and DPO in 34 self-contained notebooks. It is a textbook, not just a training script.

What the book covers

Part 1 (16 chapters) introduces the fundamentals of neural networks: multilayer perceptrons, loss functions, gradient descent, autograd, learning rates, iterations, batch sizes, epochs, hidden layers, and activation functions — followed by discussions on encoding, embeddings, cross-entropy, recurrent neural networks (RNNs), the Adam optimizer, and learning rate scheduling. After completing this part, you will have built a small, working deep learning framework (src/core).

Part 2 (18 chapters) introduces large language models (LLMs): tokens and positional embeddings; self-attention, causal attention, weighted attention, and multi-head attention; residual connections, normalization, dropout, and model persistence; sampling strategies (temperature, top-k, top-p), stopping criteria, KV caching, and batch inference; and finally, SFT and DPO data preprocessing and training.

Each chapter is a self-contained, runnable Jupyter notebook, and most chapters end with practice exercises. No GPU required: the default model (context window 256, 6 Transformer modules, 6 attention heads, approximately 10.8 million parameters) can be trained on a laptop’s CPU. The training data uses Karpathy’s tinyshakespeare.

Who it is for

Self-learners and career changers who seeking a complete learning path rather than fragmented tutorials.
Engineers already using PyTorch who cannot explain autograd or attention mechanisms in detail. You can read relevant chapters (such as autograd and KV caching) separately to reinforce your knowledge.
Teachers looking for experimental materials. Licensed under the MIT License, each chapter is executable and includes exercises, no GPU budget required.

This book is not intended to replace PyTorch. Its purpose is to encourage you to read it before relying on any framework — so that when you actually use a framework, you’ll understand how it works under the hood. Frameworks come and go, but the underlying principles remain constant.

If you find this book helpful, please star the repository; this will help others discover it. Feel free to submit issues and pull requests, especially for chapters that are unclear — that’s how this book continues to improve.

DEV Community: Steven Li

I built a GPT from scratch in NumPy only – 34 notebooks, no PyTorch

What the book covers

Who it is for