DEV Community: Luiz Fernando Vid

Article 1 — MiniGPT in Java: Understanding a Transformer in Detail

Luiz Fernando Vid — Sun, 09 Aug 2026 22:00:00 +0000

Over the past few weeks, I started a personal project that, at first glance, seems impractical: implementing a GPT from scratch, in Java, without using any AI framework. No TensorFlow, no PyTorch, no DJL. Just linear algebra, a few hundred lines of code, and the willingness to understand each piece that makes a language model work.

The reason is straightforward: at the pace the AI ecosystem is evolving, I felt the need to deepen my understanding of the fundamentals, starting with the piece that sustains practically everything used today in applied GenAI, the Transformer. I use this layer of applications (RAG, copilot, integrations) daily, but at a certain point I realized I wanted to understand better what supports all of this underneath, not just consume it.

That frustration became MiniGPT. The idea is simple to state but nothing simple to execute, and consists of manually building each component of a Transformer Decoder (tokenization, embeddings, self-attention, backpropagation, training, text generation) to come out the other side understanding better what I use every day in applied form.

This series of articles documents that journey. It will be a series of texts, one for each phase of the roadmap, from the mathematical foundation to inference of the trained model generating text. It's not a series of ready-to-copy-and-paste tutorials; it's the record of decisions, the whys, and also the mistakes, because implementing backpropagation manually, for example, is the kind of thing that doesn't work on the first try, and I think the value is precisely in showing that.

What is, in fact, a GPT

It's worth pausing on the name before diving into code. GPT stands for Generative Pre-trained Transformer, and each of those three words carries a specific architectural decision.

The Transformer is the architecture described by Vaswani and other Google researchers in 2017 in the paper "Attention Is All You Need". The original proposal solved machine translation with an encoder (which reads the entire input sentence) and a decoder (which generates the output sentence, token by token). GPT uses only half of that architecture—the decoder half—which is why it's called "decoder-only". It makes sense because a language model doesn't need to "read" a source sentence before translating; it just needs to predict which is the most likely next token, given everything that came before.

It's this prediction mechanism (called autoregressive modeling) that sustains both training and text generation. During training, the model sees a text snippet and learns to predict the next token; during generation, it does literally the same thing, repeatedly, token by token, feeding its own output back as input for the next prediction. It's not magic, it's technology: it's the same operation, millions of times, in a loop.

The diagram below summarizes the path a sentence takes inside the model, from text input to the choice of the next token, and is also, essentially, the map of the next phases in this series.

Each rectangle in that diagram becomes, over the course of the series, its own phase with tests and implementation: Tokenizer (Phase 2), Embeddings and Positional Embeddings (Phases 3 and 4), Self-Attention and Multi-Head Attention (Phases 5 and 6), Feed Forward (Phase 7), and so on until text generation (Phase 13).

A reference I'm using as a general compass is the book "Build a Large Language Model (From Scratch)" by Sebastian Raschka. It's one of the few materials that treats building a GPT manually as its own pedagogical goal, rather than treating it as a disposable academic exercise. I'll cite it again throughout the series whenever a design decision warrants deeper theoretical context than fits in an article.

A scope decision before tokenizer

Before diving into Phase 2 (Tokenizer), which is the subject of the next article, it's worth recording a decision I made early on that will appear implicitly throughout the series: not everything needs to be reinvented.

Phase 1 of the project is building a small math library (vectors, matrices, dot product, matrix product, transposition, basic statistics). It's the foundation everything else is built on. I even considered implementing this from scratch too, but I decided to use an external lib (EJML, in this case) for that specific layer.

The criterion I used was simple: if the operation is pure math (addition, matrix product, transposition), it can come from a mature and tested library. The moment an operation carries neural network semantics (a gradient, the forward or backward of a layer) is when it needs to be mine, written and understood line by line. Reinventing matrix product teaches me nothing about Transformers; reinventing self-attention does.

From there was born the Tensor, the class that sustains the entire project. In practice, it's a thin wrapper over EJML matrices, representing one data item per element of a batch. It knows how to add, multiply by a factor, do matrix product, transpose, calculate mean and variance per row. Basically the minimum mathematical vocabulary that the next phases will consume without stopping. I built this entire foundation in TDD, cycle by cycle, and it's a detail that will reappear often: each component of MiniGPT (from Tensor to the trained model) is born from a test that fails before any implementation line exists.

What comes next

The next article is about Tokenizer. It's the first step where the project stops being generic math and starts to actually become a language model: the moment text becomes a number, and where the first vocabulary decisions start to shape everything the model will be able to (or not) represent afterward.

I'm publishing this series over the next few weeks, one article per phase.

Article 2 — MiniGPT in Java, Phase 2: Tokenizer

Luiz Fernando Vid — Sun, 09 Aug 2026 22:00:00 +0000

After closing Phase 1 with the Tensor structure that will support the rest of the project, it's time to solve a problem that looks simpler on the surface, but carries a design decision worth pausing to think about before jumping into coding: how to turn text into a number.

The Problem

A neural network doesn't process text. It processes numbers, and more specifically, it processes linear algebra operations on vectors and matrices, which means that before anything gets near an embedding layer, an attention layer, or whatever else, the text we type needs to become a sequence of integers. That's the sole responsibility of the tokenizer: to be the bridge between the language we understand and the numerical representation the model can manipulate.

Three Ways to Build That Bridge

There are essentially three levels of granularity for tokenization, and each one solves the problem differently, with clear trade-offs.

Using the phrase "the cat sleeps" as an example, you can see how each approach tackles the problem differently.

The first is char-level, where each character (including the space) becomes a token:

Text:   "the cat sleeps"
Tokens: ['t', 'h', 'e', ' ', 'c', 'a', 't', ' ', 's', 'l', 'e', 'e', 'p', 's']
IDs:    [1, 0, 5, 2, 8, 1, 0, 4, 1, 7, 6, 3, 9, 4]

The vocabulary here is all the distinct characters appearing in the corpus (letters, space, punctuation), which for English stays in the range of 60 to 100 symbols, and the implementation is straightforward, but in return the sequences get long, since each word becomes multiple tokens.

The second is word-level, where each word is a single token (typically separating punctuation as well):

Text:   "the cat sleeps"
Tokens: ['the', 'cat', 'sleeps']
IDs:    [12, 340, 891]

The vocabulary becomes all the distinct words in the corpus, which quickly turns into tens of thousands of entries, and any new word the model hasn't seen during training (a proper name, for example) simply has no ID for it—the classic OOV (out of vocabulary) problem.

The third, which real production models use (GPT-2 and up, for example), is subword via BPE (Byte Pair Encoding). The logic is to start char-level and merge the most frequent pairs of characters until forming a vocabulary of fixed size (10,000 tokens, for example), which makes common words into a single token and rare or compound words break into smaller pieces:

Text:   "the cat sleeping peacefully"
Tokens: ['the', 'Ġcat', 'Ġsleep', 'ing', 'Ġpeace', 'ful', 'ly']
IDs:    [12, 340, 55, 891, 203, 77, 88]

(the Ġ is the GPT-2 convention to mark word start with space before). Notice that "sleeping" became two tokens ("sleep" + "ing") and "peacefully" also ("peace" + "ful" + "ly"), the algorithm learned that these fragments are frequent enough to deserve their own token, without needing an entry for every complete word in the language. It's the middle ground that solves both the long sequence problem of char-level and the OOV problem of word-level, just at the cost of a much more complex implementation.

Why Char-Level, Here and Now

For this phase of the project, I opted for char-level, not because it's the "right" approach (it's not what production uses), but because the goal here is to understand the end-to-end mechanism without the complexity of the BPE merge algorithm getting in the way of learning. Char-level closes the encode-decode cycle quickly, with an implementation that fits in a few lines, and that's what matters in a phase that exists to consolidate concepts, not to compete with production tokenizers.

Separating Responsibilities

The design split into two classes: Vocabulary, which is the mapping between character and ID (and vice versa), and Tokenizer, which uses that vocabulary to convert text to IDs and back. This separation exists because it leaves the door open to swap tokenization strategies later (char-level for BPE, for example) without needing to touch whoever consumes the Tokenizer.

Vocabulary Consistency

The vocabulary cannot be rebuilt with every execution. If the Vocabulary is regenerated every time the program runs, the IDs assigned to each character might come out differently from one session to the next, and in that case, the model trained with one vocabulary becomes invalid if used with another, because each learned weight depends directly on the ID it represents.

The vocabulary is built once, during dataset preparation, and from that point on is persisted and only loaded (never rebuilt) in all subsequent training or inference. The encode and decode, those are used all the time, always against the same frozen vocabulary.

What Stays for Next Phase

With the tokenizer closed, the text sequence already becomes a sequence of integers, but those integers still don't mean anything to the model, they're just indices. That's what the next phase solves: transforming each ID into a dense vector, capable of carrying some notion of meaning in vector space. That's the gateway to embeddings.