The tokenizer (Phase 2) solves half the problem: text becomes a sequence of integer IDs. But an ID, by itself, carries no meaning. It's just an index, a table address. This phase solves the other half: transforming each ID into a dense vector, capable of carrying some notion of meaning in vector space.
What is an Embedding, in Practice
An embedding is a lookup table. A weights matrix [vocabSize x dModel], where each row represents a token from the vocabulary. "Looking up the embedding of token 15" is simply taking row 15 of that matrix. There's no complex math here, it's indexing. The real learning happens later, via backprop adjusting the values in that matrix throughout training, until tokens with similar semantic use end up occupying nearby regions in that vector space.
Choosing dModel
dModel is the dimension of the vector representing each token throughout the entire network, not just in the embeddings table: it's the size of each row of the [vocabSize x dModel] matrix here in Phase 3, but it's also the same dimension the vectors maintain as they traverse self-attention, feed-forward, and residual connections in the next phases, what the literature calls the residual stream. Each layer of the Transformer reads a vector of size dModel and returns another vector of the same size, so this constant works as the "width" of the model, while the number of layers works as the "depth".
I defined dModel = 64 for this phase. To get a sense of scale, it's worth comparing with public architectures (frontier models like Opus, GPT-5.6, or Kimi K3 don't disclose this number, so comparison is only possible with open models):
| Model | dModel | Layers | Parameters |
|---|---|---|---|
| MiniGPT | 64 | — | ~thousands |
| GPT-2 Small | 768 | 12 | 117M |
| GPT-2 XL | 1600 | 48 | 1.5B |
| Llama 3.1 405B | 16,384 | 126 | 405B |
| DeepSeek-V3 | 7,168 | 61 | 671B (MoE) |
dModel=64 is 12 times smaller than the smallest public GPT-2, and 256 times smaller than Llama 3.1 405B. The distance isn't conceptual, it's scale: the mechanism that 64 dimensions demonstrate is the same one running in production with thousands of dimensions, it just fits running on a notebook instead of needing a GPU cluster.
Design, Considering the Tensor from Phase 1
Tensor was defined as an array of SimpleMatrix, a 2D matrix per element of the batch. The embeddings matrix itself (the trainable weights) is not part of the batch, it's a model parameter: a single SimpleMatrix [vocabSize x dModel]. The output of the forward pass, that does become a Tensor: for each batch item, a SimpleMatrix [seqLen x dModel].
To close the cycle quickly, the implementation started single-sequence (int[] tokenIds → SimpleMatrix), with the plan to evolve to batched (int[][] → SimpleMatrix[]) after validating basic behavior.
About DDRM, FDRM, and ZDRM
It's worth recording an infrastructure decision that came up when working with SimpleMatrix.random64: EJML (the linear algebra library used in the project) has three types of dense matrices under the hood. DDRM is double, real, and is the default used by SimpleMatrix. FDRM is float, half the precision and memory, useful when performance matters more than accuracy (not the case here). ZDRM is complex number, used in specific spectral decompositions, with no application in a typical neural network, since weights, activations, and gradients in a GPT are all real numbers. DDRM remains the right choice for the entire project.
What Stays for Next Phase
With the embedding in place, each token in the sequence now has a dense vector associated with it, but that vector is the same regardless of which position the token appears in. A GPT needs to know the order of tokens, since "the cat sleeps" and "sleeps cat the" can't become the same representation. That's the problem the next phase solves, with positional embeddings.
Top comments (0)