Over the past few weeks, I started a personal project that, at first glance, seems impractical: implementing a GPT from scratch, in Java, without using any AI framework. No TensorFlow, no PyTorch, no DJL. Just linear algebra, a few hundred lines of code, and the willingness to understand each piece that makes a language model work.
The reason is straightforward: at the pace the AI ecosystem is evolving, I felt the need to deepen my understanding of the fundamentals, starting with the piece that sustains practically everything used today in applied GenAI, the Transformer. I use this layer of applications (RAG, copilot, integrations) daily, but at a certain point I realized I wanted to understand better what supports all of this underneath, not just consume it.
That frustration became MiniGPT. The idea is simple to state but nothing simple to execute, and consists of manually building each component of a Transformer Decoder (tokenization, embeddings, self-attention, backpropagation, training, text generation) to come out the other side understanding better what I use every day in applied form.
This series of articles documents that journey. It will be a series of texts, one for each phase of the roadmap, from the mathematical foundation to inference of the trained model generating text. It's not a series of ready-to-copy-and-paste tutorials; it's the record of decisions, the whys, and also the mistakes, because implementing backpropagation manually, for example, is the kind of thing that doesn't work on the first try, and I think the value is precisely in showing that.
What is, in fact, a GPT
It's worth pausing on the name before diving into code. GPT stands for Generative Pre-trained Transformer, and each of those three words carries a specific architectural decision.
The Transformer is the architecture described by Vaswani and other Google researchers in 2017 in the paper "Attention Is All You Need". The original proposal solved machine translation with an encoder (which reads the entire input sentence) and a decoder (which generates the output sentence, token by token). GPT uses only half of that architecture—the decoder half—which is why it's called "decoder-only". It makes sense because a language model doesn't need to "read" a source sentence before translating; it just needs to predict which is the most likely next token, given everything that came before.
It's this prediction mechanism (called autoregressive modeling) that sustains both training and text generation. During training, the model sees a text snippet and learns to predict the next token; during generation, it does literally the same thing, repeatedly, token by token, feeding its own output back as input for the next prediction. It's not magic, it's technology: it's the same operation, millions of times, in a loop.
The diagram below summarizes the path a sentence takes inside the model, from text input to the choice of the next token, and is also, essentially, the map of the next phases in this series.
Each rectangle in that diagram becomes, over the course of the series, its own phase with tests and implementation: Tokenizer (Phase 2), Embeddings and Positional Embeddings (Phases 3 and 4), Self-Attention and Multi-Head Attention (Phases 5 and 6), Feed Forward (Phase 7), and so on until text generation (Phase 13).
A reference I'm using as a general compass is the book "Build a Large Language Model (From Scratch)" by Sebastian Raschka. It's one of the few materials that treats building a GPT manually as its own pedagogical goal, rather than treating it as a disposable academic exercise. I'll cite it again throughout the series whenever a design decision warrants deeper theoretical context than fits in an article.
A scope decision before tokenizer
Before diving into Phase 2 (Tokenizer), which is the subject of the next article, it's worth recording a decision I made early on that will appear implicitly throughout the series: not everything needs to be reinvented.
Phase 1 of the project is building a small math library (vectors, matrices, dot product, matrix product, transposition, basic statistics). It's the foundation everything else is built on. I even considered implementing this from scratch too, but I decided to use an external lib (EJML, in this case) for that specific layer.
The criterion I used was simple: if the operation is pure math (addition, matrix product, transposition), it can come from a mature and tested library. The moment an operation carries neural network semantics (a gradient, the forward or backward of a layer) is when it needs to be mine, written and understood line by line. Reinventing matrix product teaches me nothing about Transformers; reinventing self-attention does.
From there was born the Tensor, the class that sustains the entire project. In practice, it's a thin wrapper over EJML matrices, representing one data item per element of a batch. It knows how to add, multiply by a factor, do matrix product, transpose, calculate mean and variance per row. Basically the minimum mathematical vocabulary that the next phases will consume without stopping. I built this entire foundation in TDD, cycle by cycle, and it's a detail that will reappear often: each component of MiniGPT (from Tensor to the trained model) is born from a test that fails before any implementation line exists.
What comes next
The next article is about Tokenizer. It's the first step where the project stops being generic math and starts to actually become a language model: the moment text becomes a number, and where the first vocabulary decisions start to shape everything the model will be able to (or not) represent afterward.
I'm publishing this series over the next few weeks, one article per phase.
Top comments (0)