DEV Community: Luiz Fernando Vid

Article 4 — MiniGPT in Java, Phase 4: Positional Embeddings

Luiz Fernando Vid — Thu, 10 Sep 2026 22:00:00 +0000

Self-attention treats every token in the sequence the same way: it computes attention scores based purely on content, comparing queries against keys regardless of position. Left alone, self-attention has no notion of order — "the cat sat on the mat" and "mat the on sat cat the" would look identical to it. Positional embeddings are how we teach the model where each token sits in the sequence.

Self-Attention Doesn't See Order

The attention mechanism from Phase 3 computes a weighted sum over value vectors, where the weights come from a similarity between queries and keys. Nothing in that computation depends on the index of a token — it's a set operation, not a sequence operation. If you shuffled the input tokens and shuffled the output in the same way, the result would be unchanged (this property is called permutation equivariance).

That's a problem for language, where order carries meaning. The fix is not architectural — we don't change attention itself — it's additive: we inject positional information directly into the input embeddings before they ever reach the attention layers. Each token's embedding gets a positional vector added to it, so that by the time attention runs, position is already baked into the representation it's comparing.

Two Ways to Encode Position

There are two broad families of positional encoding: fixed (sinusoidal) and learned.

The original Transformer paper used fixed sinusoidal functions — sine and cosine waves of different frequencies, one pair per pair of embedding dimensions. This has an elegant property: the encoding for any position can be expressed as a linear function of the encoding for any other position, which in theory lets the model generalize to sequence lengths it never saw during training.

Learned positional embeddings, by contrast, are just another embedding table — exactly like the token embedding table from Phase 2, but indexed by position instead of by token id. Position 0 gets a trainable vector, position 1 gets a different trainable vector, and so on, up to some maximum length. GPT-2 and MiniGPT both use this approach: simpler to implement, and in practice it performs comparably to sinusoidal encoding for the sequence lengths these models actually train on.

MiniGPT uses learned positional embeddings for that reason — it keeps the implementation symmetric with the token embedding table, and there's no need for the extrapolation properties sinusoidal encoding offers when the context window is fixed and known in advance.

contextWindow, Not maxSeqLen

One naming decision worth calling out: MiniGPT's positional embedding table is sized by a field called contextWindow, not maxSeqLen. This isn't just a style preference. "Max sequence length" suggests an incidental limit — the longest input you happen to support. "Context window" names what the number actually is: the boundary of what the model can attend to at all. Every position beyond it simply has no embedding to look up.

That distinction matters for readability. A future maintainer reading contextWindow immediately understands why the value matters (it defines the model's attention horizon), rather than treating it as an arbitrary array-bounds constant to work around.

Implementation Guided by Testing

The positional embedding table in MiniGPT is a straightforward parallel to the token embedding table: a matrix of shape [contextWindow, dModel], where row i holds the learned vector for position i.

public class PositionalEmbedding {

    private final float[][] weights; // [contextWindow][dModel]
    private final int contextWindow;
    private final int dModel;

    public PositionalEmbedding(int contextWindow, int dModel) {
        this.contextWindow = contextWindow;
        this.dModel = dModel;
        this.weights = new float[contextWindow][dModel];
        initializeWeights();
    }

    private void initializeWeights() {
        Random random = new Random();
        float bound = 1.0f / (float) Math.sqrt(dModel);
        for (int pos = 0; pos < contextWindow; pos++) {
            for (int d = 0; d < dModel; d++) {
                weights[pos][d] = (random.nextFloat() * 2 - 1) * bound;
            }
        }
    }

    public float[] embed(int position) {
        if (position >= contextWindow) {
            throw new IllegalArgumentException(
                "Position " + position + " exceeds contextWindow " + contextWindow);
        }
        return weights[position];
    }

    public float[][] embedSequence(int sequenceLength) {
        if (sequenceLength > contextWindow) {
            throw new IllegalArgumentException(
                "Sequence length " + sequenceLength + " exceeds contextWindow " + contextWindow);
        }
        float[][] result = new float[sequenceLength][dModel];
        for (int pos = 0; pos < sequenceLength; pos++) {
            result[pos] = embed(pos);
        }
        return result;
    }
}

The bounds check in embed is not defensive boilerplate — it's the direct enforcement of the contextWindow concept from the previous section. A unit test that asserts this exception is thrown for position == contextWindow is what actually pins down the invariant; without it, an off-by-one in a caller could silently read garbage or throw an unrelated array-index exception with a far less useful message.

Combining token and positional embeddings is simple element-wise addition: the final input to the transformer blocks is tokenEmbedding[i] + positionalEmbedding[i] for each position i. Both live in the same dModel-dimensional space, which is precisely why dModel had to be decided once, up front, in Phase 3 — every component downstream depends on that shared dimensionality.

What This Means at Production Scale

The contextWindow choice has real consequences at scale. MiniGPT's table is tiny by design. GPT-2 XL's context window is 1024 tokens; Llama 3.1's is 128,000; some production systems now advertise context windows in the millions. Since the positional embedding table (for the learned variant) is [contextWindow, dModel], a naive learned-embedding approach doesn't scale gracefully to those lengths — this is exactly why modern large-scale models have largely moved to relative or rotary positional encodings (RoPE), which encode position as a function of the relative distance between tokens rather than as a fixed per-position lookup table. That's a natural next question, but outside the scope of what MiniGPT needs to demonstrate the core mechanism.

For MiniGPT's purposes, the fixed-size learned table is the right tradeoff: it's the simplest correct implementation of "the model needs to know where each token is," and it makes the connection between contextWindow and the model's attention horizon completely explicit in the code.

Article 3 — MiniGPT in Java, Phase 3: Embeddings

Luiz Fernando Vid — Sat, 05 Sep 2026 22:00:00 +0000

The tokenizer (Phase 2) solves half the problem: text becomes a sequence of integer IDs. But an ID, by itself, carries no meaning. It's just an index, a table address. This phase solves the other half: transforming each ID into a dense vector, capable of carrying some notion of meaning in vector space.

What is an Embedding, in Practice

An embedding is a lookup table. A weights matrix [vocabSize x dModel], where each row represents a token from the vocabulary. "Looking up the embedding of token 15" is simply taking row 15 of that matrix. There's no complex math here, it's indexing. The real learning happens later, via backprop adjusting the values in that matrix throughout training, until tokens with similar semantic use end up occupying nearby regions in that vector space.

Choosing dModel

dModel is the dimension of the vector representing each token throughout the entire network, not just in the embeddings table: it's the size of each row of the [vocabSize x dModel] matrix here in Phase 3, but it's also the same dimension the vectors maintain as they traverse self-attention, feed-forward, and residual connections in the next phases, what the literature calls the residual stream. Each layer of the Transformer reads a vector of size dModel and returns another vector of the same size, so this constant works as the "width" of the model, while the number of layers works as the "depth".

I defined dModel = 64 for this phase. To get a sense of scale, it's worth comparing with public architectures (frontier models like Opus, GPT-5.6, or Kimi K3 don't disclose this number, so comparison is only possible with open models):

Model	dModel	Layers	Parameters
MiniGPT	64	—	~thousands
GPT-2 Small	768	12	117M
GPT-2 XL	1600	48	1.5B
Llama 3.1 405B	16,384	126	405B
DeepSeek-V3	7,168	61	671B (MoE)

dModel=64 is 12 times smaller than the smallest public GPT-2, and 256 times smaller than Llama 3.1 405B. The distance isn't conceptual, it's scale: the mechanism that 64 dimensions demonstrate is the same one running in production with thousands of dimensions, it just fits running on a notebook instead of needing a GPU cluster.

Design, Considering the Tensor from Phase 1

Tensor was defined as an array of SimpleMatrix, a 2D matrix per element of the batch. The embeddings matrix itself (the trainable weights) is not part of the batch, it's a model parameter: a single SimpleMatrix [vocabSize x dModel]. The output of the forward pass, that does become a Tensor: for each batch item, a SimpleMatrix [seqLen x dModel].

To close the cycle quickly, the implementation started single-sequence (int[] tokenIds → SimpleMatrix), with the plan to evolve to batched (int[][] → SimpleMatrix[]) after validating basic behavior.

About DDRM, FDRM, and ZDRM

It's worth recording an infrastructure decision that came up when working with SimpleMatrix.random64: EJML (the linear algebra library used in the project) has three types of dense matrices under the hood. DDRM is double, real, and is the default used by SimpleMatrix. FDRM is float, half the precision and memory, useful when performance matters more than accuracy (not the case here). ZDRM is complex number, used in specific spectral decompositions, with no application in a typical neural network, since weights, activations, and gradients in a GPT are all real numbers. DDRM remains the right choice for the entire project.

What Stays for Next Phase

With the embedding in place, each token in the sequence now has a dense vector associated with it, but that vector is the same regardless of which position the token appears in. A GPT needs to know the order of tokens, since "the cat sleeps" and "sleeps cat the" can't become the same representation. That's the problem the next phase solves, with positional embeddings.

Article 1 — MiniGPT in Java: Understanding a Transformer in Detail

Luiz Fernando Vid — Sun, 09 Aug 2026 22:00:00 +0000

Over the past few weeks, I started a personal project that, at first glance, seems impractical: implementing a GPT from scratch, in Java, without using any AI framework. No TensorFlow, no PyTorch, no DJL. Just linear algebra, a few hundred lines of code, and the willingness to understand each piece that makes a language model work.

The reason is straightforward: at the pace the AI ecosystem is evolving, I felt the need to deepen my understanding of the fundamentals, starting with the piece that sustains practically everything used today in applied GenAI, the Transformer. I use this layer of applications (RAG, copilot, integrations) daily, but at a certain point I realized I wanted to understand better what supports all of this underneath, not just consume it.

That frustration became MiniGPT. The idea is simple to state but nothing simple to execute, and consists of manually building each component of a Transformer Decoder (tokenization, embeddings, self-attention, backpropagation, training, text generation) to come out the other side understanding better what I use every day in applied form.

This series of articles documents that journey. It will be a series of texts, one for each phase of the roadmap, from the mathematical foundation to inference of the trained model generating text. It's not a series of ready-to-copy-and-paste tutorials; it's the record of decisions, the whys, and also the mistakes, because implementing backpropagation manually, for example, is the kind of thing that doesn't work on the first try, and I think the value is precisely in showing that.

What is, in fact, a GPT

It's worth pausing on the name before diving into code. GPT stands for Generative Pre-trained Transformer, and each of those three words carries a specific architectural decision.

The Transformer is the architecture described by Vaswani and other Google researchers in 2017 in the paper "Attention Is All You Need". The original proposal solved machine translation with an encoder (which reads the entire input sentence) and a decoder (which generates the output sentence, token by token). GPT uses only half of that architecture—the decoder half—which is why it's called "decoder-only". It makes sense because a language model doesn't need to "read" a source sentence before translating; it just needs to predict which is the most likely next token, given everything that came before.

It's this prediction mechanism (called autoregressive modeling) that sustains both training and text generation. During training, the model sees a text snippet and learns to predict the next token; during generation, it does literally the same thing, repeatedly, token by token, feeding its own output back as input for the next prediction. It's not magic, it's technology: it's the same operation, millions of times, in a loop.

The diagram below summarizes the path a sentence takes inside the model, from text input to the choice of the next token, and is also, essentially, the map of the next phases in this series.

Each rectangle in that diagram becomes, over the course of the series, its own phase with tests and implementation: Tokenizer (Phase 2), Embeddings and Positional Embeddings (Phases 3 and 4), Self-Attention and Multi-Head Attention (Phases 5 and 6), Feed Forward (Phase 7), and so on until text generation (Phase 13).

A reference I'm using as a general compass is the book "Build a Large Language Model (From Scratch)" by Sebastian Raschka. It's one of the few materials that treats building a GPT manually as its own pedagogical goal, rather than treating it as a disposable academic exercise. I'll cite it again throughout the series whenever a design decision warrants deeper theoretical context than fits in an article.

A scope decision before tokenizer

Before diving into Phase 2 (Tokenizer), which is the subject of the next article, it's worth recording a decision I made early on that will appear implicitly throughout the series: not everything needs to be reinvented.

Phase 1 of the project is building a small math library (vectors, matrices, dot product, matrix product, transposition, basic statistics). It's the foundation everything else is built on. I even considered implementing this from scratch too, but I decided to use an external lib (EJML, in this case) for that specific layer.

The criterion I used was simple: if the operation is pure math (addition, matrix product, transposition), it can come from a mature and tested library. The moment an operation carries neural network semantics (a gradient, the forward or backward of a layer) is when it needs to be mine, written and understood line by line. Reinventing matrix product teaches me nothing about Transformers; reinventing self-attention does.

From there was born the Tensor, the class that sustains the entire project. In practice, it's a thin wrapper over EJML matrices, representing one data item per element of a batch. It knows how to add, multiply by a factor, do matrix product, transpose, calculate mean and variance per row. Basically the minimum mathematical vocabulary that the next phases will consume without stopping. I built this entire foundation in TDD, cycle by cycle, and it's a detail that will reappear often: each component of MiniGPT (from Tensor to the trained model) is born from a test that fails before any implementation line exists.

What comes next

The next article is about Tokenizer. It's the first step where the project stops being generic math and starts to actually become a language model: the moment text becomes a number, and where the first vocabulary decisions start to shape everything the model will be able to (or not) represent afterward.

I'm publishing this series over the next few weeks, one article per phase.

Article 2 — MiniGPT in Java, Phase 2: Tokenizer

Luiz Fernando Vid — Sun, 09 Aug 2026 22:00:00 +0000

After closing Phase 1 with the Tensor structure that will support the rest of the project, it's time to solve a problem that looks simpler on the surface, but carries a design decision worth pausing to think about before jumping into coding: how to turn text into a number.

The Problem

A neural network doesn't process text. It processes numbers, and more specifically, it processes linear algebra operations on vectors and matrices, which means that before anything gets near an embedding layer, an attention layer, or whatever else, the text we type needs to become a sequence of integers. That's the sole responsibility of the tokenizer: to be the bridge between the language we understand and the numerical representation the model can manipulate.

Three Ways to Build That Bridge

There are essentially three levels of granularity for tokenization, and each one solves the problem differently, with clear trade-offs.

Using the phrase "the cat sleeps" as an example, you can see how each approach tackles the problem differently.

The first is char-level, where each character (including the space) becomes a token:

Text:   "the cat sleeps"
Tokens: ['t', 'h', 'e', ' ', 'c', 'a', 't', ' ', 's', 'l', 'e', 'e', 'p', 's']
IDs:    [1, 0, 5, 2, 8, 1, 0, 4, 1, 7, 6, 3, 9, 4]

The vocabulary here is all the distinct characters appearing in the corpus (letters, space, punctuation), which for English stays in the range of 60 to 100 symbols, and the implementation is straightforward, but in return the sequences get long, since each word becomes multiple tokens.

The second is word-level, where each word is a single token (typically separating punctuation as well):

Text:   "the cat sleeps"
Tokens: ['the', 'cat', 'sleeps']
IDs:    [12, 340, 891]

The vocabulary becomes all the distinct words in the corpus, which quickly turns into tens of thousands of entries, and any new word the model hasn't seen during training (a proper name, for example) simply has no ID for it—the classic OOV (out of vocabulary) problem.

The third, which real production models use (GPT-2 and up, for example), is subword via BPE (Byte Pair Encoding). The logic is to start char-level and merge the most frequent pairs of characters until forming a vocabulary of fixed size (10,000 tokens, for example), which makes common words into a single token and rare or compound words break into smaller pieces:

Text:   "the cat sleeping peacefully"
Tokens: ['the', 'Ġcat', 'Ġsleep', 'ing', 'Ġpeace', 'ful', 'ly']
IDs:    [12, 340, 55, 891, 203, 77, 88]

(the Ġ is the GPT-2 convention to mark word start with space before). Notice that "sleeping" became two tokens ("sleep" + "ing") and "peacefully" also ("peace" + "ful" + "ly"), the algorithm learned that these fragments are frequent enough to deserve their own token, without needing an entry for every complete word in the language. It's the middle ground that solves both the long sequence problem of char-level and the OOV problem of word-level, just at the cost of a much more complex implementation.

Why Char-Level, Here and Now

For this phase of the project, I opted for char-level, not because it's the "right" approach (it's not what production uses), but because the goal here is to understand the end-to-end mechanism without the complexity of the BPE merge algorithm getting in the way of learning. Char-level closes the encode-decode cycle quickly, with an implementation that fits in a few lines, and that's what matters in a phase that exists to consolidate concepts, not to compete with production tokenizers.

Separating Responsibilities

The design split into two classes: Vocabulary, which is the mapping between character and ID (and vice versa), and Tokenizer, which uses that vocabulary to convert text to IDs and back. This separation exists because it leaves the door open to swap tokenization strategies later (char-level for BPE, for example) without needing to touch whoever consumes the Tokenizer.

Vocabulary Consistency

The vocabulary cannot be rebuilt with every execution. If the Vocabulary is regenerated every time the program runs, the IDs assigned to each character might come out differently from one session to the next, and in that case, the model trained with one vocabulary becomes invalid if used with another, because each learned weight depends directly on the ID it represents.

The vocabulary is built once, during dataset preparation, and from that point on is persisted and only loaded (never rebuilt) in all subsequent training or inference. The encode and decode, those are used all the time, always against the same frozen vocabulary.

What Stays for Next Phase

With the tokenizer closed, the text sequence already becomes a sequence of integers, but those integers still don't mean anything to the model, they're just indices. That's what the next phase solves: transforming each ID into a dense vector, capable of carrying some notion of meaning in vector space. That's the gateway to embeddings.