DEV Community: Carlos Chinchilla Corbacho

From GPT-2 to Llama: four changes and the reason for each

Carlos Chinchilla Corbacho — Thu, 15 Oct 2026 13:00:00 +0000

Originally published at cchinchilla.dev. Part 7 of From code to weights, a 12-part series on ML fundamentals for engineers.

Part 6 ended on GPT-2's five decisions and the loop that serves it. Open a model from this decade and the block is still Part 5's: two stages, each reading a normalised copy of the bus and adding its result back. Four things inside it changed.

I read modeling_llama.py from transformers 5.17 next to nanoGPT's model.py, and loaded SmolLM2-135M, a model with Llama's architecture at GPT-2 small's size. Then I undid each change on its trained weights and measured what broke.

Same block, four edits

LlamaDecoderLayer.forward, modeling_llama.py:305-323, condensed:

residual = hidden_states
hidden_states = self.input_layernorm(hidden_states)            # RMSNorm, not LayerNorm
hidden_states, _ = self.self_attn(hidden_states, position_embeddings=...)
hidden_states = residual + hidden_states                       # the bus, as in GPT-2
residual = hidden_states
hidden_states = self.mlp(self.post_attention_layernorm(hidden_states))
hidden_states = residual + hidden_states

This is Part 5's four lines with more names; the plus signs are where they were, and so is pre-LN, the norm applied before each stage rather than after. What changed sits inside the calls: how positions enter attention, what the norm computes, the shape of the feed-forward, and how many keys and values attention keeps. (Llama also drops the bias vectors from its linear projections; nothing below depends on them.)

The two models make a fair pair. GPT-2 small is 124 million parameters in 12 blocks of width 768. SmolLM2-135M is 135 million in 30 blocks of width 576. None of the four changes is about size; you could make either model with or without them.

Positions become a rotation

GPT-2 adds a learned row to each token, one row per position, and the table ends at 1,024. Llama has no position table. It rotates the query and key instead, at modeling_llama.py:158-159:

q_embed = (q * cos) + (rotate_half(q) * sin)
k_embed = (k * cos) + (rotate_half(k) * sin)

This is rotary position embedding, RoPE. In Part 4, attention scores a pair of tokens by comparing a query vector with a key vector. Each query and key is split into heads of 64 numbers. Each head's vector is read as 32 pairs, each pair a point on a circle, and each point is turned like a clock hand by an angle proportional to the position. rotate_half is how the code writes that turn without a matrix. Every pair has its own speed. One config number, the rotation base, sets all of them; SmolLM2's is 100,000. In SmolLM2 the fastest pair goes round once every 6.3 positions and the slowest once every 438,000.

The rotation is not decoration. Rotate a query by position m and a key by position n, and their dot product, the score from Part 4, depends only on m − n. I took a random query and key through SmolLM2's rotation. At positions 10 and 3 the score was −13.4724; at 1,010 and 1,003 it was −13.4723. One step closer, at 10 and 4, it was −10.786.

The whole model inherits that. I shifted every position id in a 512-token text by 500, then by 5,000. SmolLM2's loss on it, its average surprise at the next token, was 3.381 unshifted and 3.381 after each shift. GPT-2 given the same shift of 500 went from 3.85 to 8.97, because every token now reads a different row of its table. Removing the rotation altogether, by setting every position to 0, took SmolLM2 from 3.12 to 6.28 on the full 1,024-token text. Using 10,000 as the rotation base, the value in the original RoPE paper and in Llama 2, instead of SmolLM2's 100,000 cost 3%.

The original post has a widget for this that doesn't survive being pasted into a feed: drag the key's position and watch the score move, then shift both and watch it stay. Pick pair 17, the slowest of the three, and nearby positions barely differ. The score here is one pair's share, between −1 and 1.

No table also means no last row. Llama 3.1 8B's config records an original length of 8,192 positions and a maximum of 131,072. It gets there by stretching the rotation. Its heads have 128 numbers, so 64 pairs: the 29 slowest are slowed by a factor of 8, the 29 fastest left alone, and the six between blended. Meta also pre-trained the family at growing lengths up to 128K, in six stages for the 405B. There is no assert to hit: past the length a model was trained at, quality is a measurement, not a guarantee.

The norm stops centring

LayerNorm subtracts each row's mean, divides by its standard deviation, then scales and shifts. LlamaRMSNorm, modeling_llama.py:62-67, skips the mean and the shift, and divides by the root mean square instead: square each number, average, take the root. If you've standardised a column in a dataframe, you've subtracted and divided; RMSNorm keeps only the division:

variance = hidden_states.pow(2).mean(-1, keepdim=True)
hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
return self.weight * hidden_states.to(input_dtype)

The bet is that the mean is small anyway, and in SmolLM2 it is. In a typical norm layer, a typical row's mean is 2% of its root mean square; in the worst layer, 4.5%. Swap full LayerNorm back in with the same learned scale and the loss moves from 3.124 to 3.175, under 2%.

Be fair about the reason given for it. The RMSNorm paper sells it as cheaper: one statistic instead of two, no shift. On my laptop's CPU, torch's F.layer_norm took 50 to 75 microseconds on a [1, 1024, 576] tensor across runs, and F.rms_norm 110 to 135. The profiler says why: on CPU, rms_norm still runs as a chain of small operations, square, average, add, root and two multiplies, while layer_norm is a single tuned kernel, one compiled routine for the whole job.

On a single thread the order flips: layer_norm about 155 microseconds, rms_norm about 130. Less arithmetic isn't less time until someone writes the kernel, and I didn't measure a GPU.

The feed-forward grows a gate

GPT-2's feed-forward is two matrices with GELU between them. Llama's is three, modeling_llama.py:175:

down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

The input goes through two projections side by side. One passes through SiLU, a smooth activation close to GELU, and then multiplies the other element by element: a gate. This is SwiGLU, from a 2020 paper that tested it and reported better results without offering a theory for why.

Three matrices cost more than two at the same width, so the width shrinks. With d the model width, a plain feed-forward of width 4d holds 8d² numbers, biases aside. Three matrices of width f hold 3·d·f, which matches at f = 8d/3. SmolLM2 uses exactly that: 1,536 is 8/3 of 576. Llama 2 7B's 11,008 is 8/3 of 4,096 rounded up to a multiple of 256.

At 8d/3 the gate changes nothing in the budget: with full attention heads, the feed-forward still holds two thirds of a block. SmolLM2's 75% comes from sharing keys and values across query heads, covered next. Llama 3.1 8B moves the split from both sides. Its feed-forward is wider, f = 3.5d, and its attention is smaller because it shares keys and values. Together they give the 81% quoted in Part 5: 176 million of each block's 218 million parameters. The wider feed-forward alone, with full attention heads, gives 72%.

The gate carries weight. Replace SiLU with GELU in SmolLM2 and the loss goes from 3.12 to 3.79. Drop the gate's multiplication entirely and it goes to 17.7, well past 10.8, the loss of a model that gives each of 49,152 tokens equal odds.

The feed-forward is also where mixture of experts plugs in: many feed-forwards per block and a router, a small layer that picks a few of them for each token. Llama 3 stayed dense; Llama 4 and other families didn't.

Keys and values get shared

GPT-2 has as many key and value heads as query heads. SmolLM2 has 9 query heads and 3 key-value heads; each key and value head serves a group of three queries. Think of one index serving three different queries: same data, different questions. In the plain PyTorch path, repeat_kv, modeling_llama.py:179-189, copies the three key and value heads out to nine before the arithmetic; torch's fused attention kernel can skip the copy when there is no padding mask. This is grouped-query attention, GQA. Llama 2 used it only in its two largest sizes, 70B and an unreleased 34B. Every Llama 3 size has it.

At inference the sharing is exact. I built a copy of SmolLM2 with nine full key and value heads, each a duplicate of its group's, and ran both. The logits matched to the last bit, and the copy had 147.8 million parameters instead of 134.5. What GQA gives up, it gives up in training, with fewer distinct keys and values to learn.

The saving is in Part 6's cache. SmolLM2 stores 11,520 numbers per token, which I read back from transformers' own cache after 256 tokens. With full heads it would store 34,560. GPT-2 small, with 12 blocks to SmolLM2's 30, stores 18,432.

At Llama 3.1 8B's size the ratio decides what fits. It has 32 query heads and 8 key-value heads, so each token costs 131 KB in bf16, the 16-bit format from Part 3. At its full 131,072 positions, one sequence's cache is 17.2 GB, next to 16.1 GB of weights. With full heads it would be 68.7 GB.

The second widget on the original post starts from GPT-2 small. Switch the three changes on one at a time; the feed-forward share barely moves. Then drag the key-value heads down to 1 and watch the cache and the share.

What the ML engineer watches

Four habits, and none of them needs a GPU.

Recompute the cache for every model. Parameter count does not predict it: the cache is 2 × blocks × key-value heads × head width numbers per token, and the key-value heads are a config field. SmolLM2 is larger than GPT-2 small and stores three eighths less per token.

Port the architecture before the weights. Every change above is part of the contract the weights were trained under. The activation, the position scheme with its rotation base and scaling, and the key-value head count are config fields; the norm type comes with the architecture name. Get the head count wrong and loading fails, because the shapes don't match. Get the count right and the grouping wrong, tiling the shared heads instead of interleaving them, and nothing raises: the loss went past 11. The same silence covers the rest. A bias-free LayerNorm instead of RMSNorm cost 1.6%, GELU instead of SiLU 21%, and the wrong rotation base 3%. Rotation pairs laid out side by side instead of in halves took the loss to 7.24. If you port one by hand, compare your logits against a reference implementation before you trust it.

Test the lengths you serve. Rotation removes the hard limit and keeps the question. If you run a model past the length it was trained at, you are the one who has to measure it there, on your own data.

Count active parameters separately. With mixture of experts, Qwen3-30B-A3B holds 30.5 billion parameters and uses 3.35 billion per token; 95% of the weights are experts. Memory follows the first number; arithmetic per token follows the second, and you pay for both.

None of this is modelling work either. A Llama block is a GPT-2 block with four parts swapped, and each swap is a line of config or code you can read, a cost you can compute, and a mismatch you can test for before a single request.

Part 7 of twelve. Llama keeps GPT-2's block and bus and changes four parts: positions become a rotation of queries and keys, the norm drops its mean, the feed-forward gains a gate, and query heads share keys and values. Get the norm, the activation or the positions wrong and nothing raises; the loss moves instead. Rotary positions drop a table, shared keys and values shrink the cache, and the gate, at 8d/3, costs nothing. The next part asks where the weights come from, and lands on the series index when it's out. If you arrived here directly, Part 1 reads the tokenizer, Part 2 the table a token id indexes, Part 3 the shapes, Part 4 attention, Part 5 the block, and Part 6 GPT-2 and its two programs.

Written with AI assistance. See AI policy.

GPT-2 is the transformer plus five decisions, and inference is a different program

Carlos Chinchilla Corbacho — Thu, 08 Oct 2026 13:00:00 +0000

Originally published at cchinchilla.dev. Part 6 of From code to weights, a 12-part series on ML fundamentals for engineers.

Part 5 ended on a block and a stack of twelve. GPT-2 is that stack plus five decisions around it. Its one checkpoint runs two different programs. Training scores a whole sequence in one call. Generation, what serving calls inference, writes one token at a time, in a loop, and most of what breaks in production breaks there.

I loaded OpenAI's GPT-2 small weights into nanoGPT's model.py again and ran both programs on the same weights. Then I added a cache to the loop, cast the weights to bf16, and compared what came out.

Five decisions around the stack

GPTConfig, model.py:108-116, fits on one screen: 1,024 positions, 12 blocks, 12 heads, width 768. The vocabulary field says 50,304, which the comment on that line calls padding to a multiple of 64, for speed. OpenAI's checkpoint has 50,257 rows, and loading it sets the config to match. Around those numbers GPT-2 made five decisions, and three of them are already in this series.

The tokenizer is byte-level BPE, Part 1: every string encodes, so there is no unknown token. The normalisation is pre-LN, Part 5. The token table is tied to the output layer at model.py:138. One matrix serves both ends: the table Part 5 counted as 31% of the model. Untied, GPT-2 small would have 163 million parameters instead of 124 million.

The fourth is positions. GPT-2 learns them: wpe is a table of shape [1024, 768], one row per position, added to the token rows before the first block. A table has a last row. Ask forward for 1,100 positions and the assert at model.py:173 stops you:

AssertionError: Cannot forward sequence of length 1100, block size is only 1024

The fifth is the initialisation. Each block makes two writes to the bus, Part 5's name for the residual stream, so a stack of L blocks adds 2L of them. In nanoGPT, every matrix, the token and position tables included, starts as a random draw with standard deviation 0.02 (OpenAI's own code used 0.01 for the position table); biases start at zero and LayerNorm at one. model.py:143-145 starts the two layers that make those writes at 0.02/√(2L) instead, so the sum doesn't grow with depth. I measured it on untrained models. With the scaled init, the stream's norm grew about 11 times through 12 blocks, and 11 times through 48. With the plain init it grew 55 times and 112 times. Training takes it from there.

Training is one call

Hand forward a sequence and its targets, the same sequence shifted by one, and it predicts every position at once. I gave it 92 positions of text. It returned logits of shape [1, 92, 50257]. Those are raw scores for every token at every position, and softmax turns them into probabilities. It also returned one loss, the cross-entropy at model.py:187: minus the log of the probability given to the right next token, averaged over all 92. Part 4's mask is what makes this honest. Row t sees only rows 0..t, so no prediction can read its own answer.

That's the program training runs at every step, for hundreds of thousands of steps. Every position is known in advance, so all of them go through the stack together, as a few large matrix multiplications. If you've written a batch job over a table, you know the shape of this program. On my laptop's CPU, one forward over 1,024 positions took 0.23 seconds.

Without targets, forward takes a shortcut at model.py:190: it computes logits for the last position only. That line is the first sign of the second program.

Generation is a loop

To write text, the model has to see each token before it can predict the next. There is nothing to run in parallel across positions. Here is nanoGPT's generate, model.py:312-328, condensed:

for _ in range(max_new_tokens):
    idx_cond = idx[:, -1024:]                 # crop to the position table
    logits, _ = self(idx_cond)                # the whole sequence, again
    logits = logits[:, -1, :] / temperature   # [B, 50257], last position only
    probs = F.softmax(logits, dim=-1)         # top_k masking skipped here
    idx_next = torch.multinomial(probs, 1)    # draw one token id
    idx = torch.cat((idx, idx_next), dim=1)   # append it and go round

Read the self(idx_cond) line again. Every step runs the full forward over everything so far and keeps one row. If you've re-read a whole file to append one line to it, you've written this loop. With an 11-token prompt, 256 new tokens push 35,456 positions through the stack for 256 useful rows.

The fix is a KV cache. Part 4 gave every position a key, which later queries match against, and a value, which they average. Under Part 4's mask, the key and value at position t depend only on positions 0..t, so once computed they never change. Keep them, one pair of tensors per block, and each step only pushes the new token through the stack and attends over the stored rows. Same 256 tokens: 266 positions. The price is memory. For GPT-2 small in fp32, that's about 74 kB per position. nanoGPT doesn't have one; I wrote one in about thirty lines on top of its modules.

The timer agrees, but not in proportion. Generating 64, 256 and 512 tokens took 2.9, 5.0 and 7.4 times longer without the cache, for 37, 133 and 261 times the positions. At batch 1, each step still reads all 124 million weights to produce one token, and the cache doesn't shrink that. Writing 1,023 tokens with the cache took 9.3 seconds, about forty times the training forward over the same length.

The cache also breaks at the edge of decision four. Past 1,024 tokens, generate crops the left end. From the second block on, every stored key and value was computed while the dropped token was still there, and with learned positions every token also moves to a new row of wpe. The whole cache is stale. nanoGPT's loop gets away with it because it recomputes everything anyway; a cached loop has to handle it.

The original post has a widget for this that doesn't survive being pasted into a feed: drag the new tokens and watch the two counts part. Then pick XL in bf16 and raise the batch until the cache bar passes the weights.

Same weights, different bugs

Two programs over the same weights should agree. Mine did, almost. Over 200 greedy steps, always taking the highest-scoring token, both loops wrote the same text. Their logits differed by up to 0.011. Almost all of that was the same shift for every token, and softmax ignores it. The gap between the two best candidates moved by at most 0.00005; the narrowest gap at any step was 0.004, so the tokens held. A longer run or a bigger model can close that margin, and nothing will raise an error when it does.

The difference is floating point. After the prompt, every matrix multiplication inside the cached loop's blocks sees one row instead of the whole sequence, so the library picks different kernels, the routines that do the arithmetic, and those add in a different order. Float addition isn't associative, so the sums differ in the last bits.

Then change something your deployment changes routinely. I cast the same weights to bf16, Part 3's 16-bit format, and ran the same greedy loop. The fp32 model continued the prompt with " were not working."; the bf16 one with " were not being run on the same system". They split at the third token and never came back together. Running the prompt eight times in one batch kept the tokens but moved the logits by up to 0.005 against the single-prompt run.

The usual fix people reach for is temperature 0. It doesn't fix this, and in nanoGPT it isn't even a value. generate divides the logits by the temperature, so 0 turns all 50,257 logits into infinities, softmax turns those into NaN, the float for an undefined result, and torch.multinomial raises. Most APIs that accept 0 treat it as a switch to argmax, picking the top score. That's greedy decoding again. It's only as deterministic as the weights, dtype, batch shape and kernels under it.

Be fair about the scale of this. Every run here was repeatable: the same seed on the same machine sampled the same 40 tokens twice. The drift appears when something underneath changes, and in a serving stack those things change without asking you.

Sampling is a policy on top

Greedy decoding is a choice, and GPT-2 small shows why people avoid it: within 200 tokens, the loop above was repeating its own sentence. Sampling draws from the distribution instead, and three knobs shape it before the draw.

Temperature divides the logits before softmax; below 1 it sharpens the distribution, above 1 it flattens it. Top-k keeps the k highest-scoring tokens and drops the rest. Top-p, also called nucleus sampling, keeps the smallest set whose probabilities add up to at least p. nanoGPT's sample.py defaults to temperature 0.8 and top-k 200; it has no top-p.

What the knobs do depends on the prompt. After "The capital of France is", GPT-2 small's top candidate is " the", at 8.5%; " Paris" is fifth, at 3.2%. At temperature 1, covering 90% of the probability takes 1,504 tokens. After "import numpy as", " np" alone has 85%, and 90% takes three. Entropy, measured in bits, puts a number on the spread: 8.7 for the capital prompt at temperature 1, as uncertain as a fair pick among about 400 tokens, and 1.5 for numpy, about three.

Top-k can't tell those apart, so any single k you pick is wrong for one of them. At nanoGPT's defaults, those 200 tokens hold 89% of the mass after the capital prompt. After the numpy prompt they keep 199 options that share under 5%. Top-p adapts: at the same temperature and p = 0.9, it keeps 237 tokens for one and a single token for the other.

The second widget on the original post starts at temperature 1 and p = 0.9, where the capital prompt keeps 1,504 tokens and numpy three. Drag the temperature to 2 and watch numpy's count grow. The top-40 and top-200 readouts show what a fixed k keeps.

What the ML engineer watches

Four habits, and the first is a test.

Test the cached loop against the full forward, on logits. Run both programs for a few hundred steps and compare the greedy tokens and, with a tolerance, the logits. Token equality alone can pass while the logits drift, as mine did.

Budget the cache like the weights. Per sequence it holds 2·L·T·d numbers for L blocks, T positions and width d: a key and a value for every block and position. GPT-2 small at 1,024 positions in fp32 is 75 MB. GPT-2 XL in bf16 is 315 MB per sequence, so about ten full-length sequences weigh as much as its 3.1 GB of weights. When you raise the batch size, you are making a memory decision, not only a throughput one.

Pin what you compare. Dtype, batch shape, library version and kernel all move logits. If your regression test diffs model outputs, fix all of them, or compare with a tolerance and log what changed.

Log the sampling settings with every output. Temperature, top-k, top-p and the seed are part of the program. Two runs that differ only in them behave differently, and an output logged without them can't be reproduced.

None of this is modelling work either. The weights are one artifact. The code around them decides whether a position is scored or written, what survives between steps, and how a distribution becomes a token, and each of those is ordinary code you can test.

Part 6 of twelve. GPT-2 is Part 5's stack plus five decisions: byte-level BPE, pre-LN, a tied token table, learned positions, and a scaled init for the writes to the bus. Training runs one parallel forward over every position; generation runs a loop that needs a cache, a sampling policy and its own tests. The next part reads what Llama changed, and lands on the series index when it's out. If you arrived here directly, Part 1 reads the tokenizer, Part 2 the table a token id indexes, Part 3 the shapes, Part 4 attention, and Part 5 the block.

Written with AI assistance. See AI policy.

A transformer block is six tensors and a bus: read it like a pipeline

Carlos Chinchilla Corbacho — Sun, 04 Oct 2026 16:00:00 +0000

Originally published at cchinchilla.dev. Part 5 of From code to weights, a 12-part series on ML fundamentals for engineers.

Part 4 ended on one stage. A decoder is a stack of blocks, twelve in GPT-2 small, and attention is one of two stages in each. This post reads a whole block the way you'd read a pipeline: what flows in, what each stage writes, and where the parameters and the cost end up.

They don't end up in the same place.

I read Block in nanoGPT's model.py with the shapes in the margin and loaded GPT-2's weights into it. Then I broke it on purpose: I took blocks out one at a time, then made each one overwrite its input instead of adding to it.

This is part of a series for engineers moving into ML engineering. Each part names one thing that does not behave the way software behaves. This one is where the cost lives: not where the code is.

The block is four lines

nanoGPT is a minimal GPT-2 in two files. Its README now points to a successor, but model.py is still the shortest honest read of a GPT-2 block. Block.forward is model.py:103-106:

def forward(self, x):                   # x: [B, T, 768], the stream
    x = x + self.attn(self.ln_1(x))     # read a normalised copy, add attention's output
    x = x + self.mlp(self.ln_2(x))      # read again, add the feed-forward's output
    return x                            # [B, T, 768]: the shape it came in with

Inside the two calls, model.py:52-92, six tensors are worth naming. The code below is condensed, without scaling, masking, or dropout. The shapes in the margin come from running a model with GPT-2 small's shape, d = 768 and 12 heads:

qkv = self.attn.c_attn(self.ln_1(x))         # [B, T, 2304]    q, k, v side by side
q, k, v = qkv.split(768, dim=2)              # three views, [B, T, 768] each
q = q.view(B, T, 12, 64).transpose(1, 2)     # [B, 12, T, 64]  12 heads of 64; k, v alike
att = (q @ k.transpose(-2, -1)).softmax(-1)  # [B, 12, T, T]   Part 4's square
y = (att @ v).transpose(1, 2).reshape(B, T, 768)
y = self.attn.c_proj(y)                      # [B, T, 768]     attention out
x = x + y                                    # [B, T, 768]     the stream, attention added
h = F.gelu(self.mlp.c_fc(self.ln_2(x)))      # [B, T, 3072]    the feed-forward's hidden layer
out = self.mlp.c_proj(h)                     # [B, T, 768]     feed-forward out

With the stream itself, that's the block: x, the projections, the scores, attention's output, the hidden layer, and the feed-forward's output. Three of them share the stream's shape. Everything else is a view, an elementwise step, or the per-head [B, 12, T, 64] layout.

One division of labour is worth reading twice. Attention is the only stage that mixes positions: row t of its output is built from rows 0..t of the stream. LayerNorm and the feed-forward work on one position at a time.

LayerNorm subtracts each 768-number row's mean and divides by its standard deviation, then applies a learned scale and shift; model.py:27 is one call to F.layer_norm. The feed-forward widens each row to 3,072, applies GELU (a smooth version of max(0, x)), and narrows it back. If you've written a map over rows, you've written the feed-forward's control flow.

The original post has a widget for this that doesn't survive being pasted into a feed: pick the feed-forward hidden layer and read its parameters, then pick the scores, which have none. Drag T and the parameters don't move while the arithmetic does. That split is the whole point of the last two sections.

The plus sign is the architecture

Read the two lines again. Neither stage replaces x. Each reads a normalised copy and adds what it computed to the original. The stream that enters a block leaves it with two edits on top, and the next block reads the result.

That stream, [B, T, 768] from the embedding to the last block, is what the literature calls the residual stream; this post calls it the bus. Every stage reads it and writes to it; no stage owns it. If you've written total += delta in a loop, you know the contract: every stage adds to the running total, and none can overwrite it; undoing an earlier write takes a later stage adding its negative.

The numbers show the contract holding. I ran 1,024 tokens of an earlier post through GPT-2 small and measured lengths, a vector's norm, as medians over positions. In blocks 1 to 10, each stage adds a vector 16% to 40% as long as the stream it reads. The stream that comes out points almost where it went in: the cosine similarity, where 1 means the same direction, is 0.93 to 0.97. The norm also grows as it goes, from 49 after block 0 to 480 at the end. Nothing on the bus is ever normalised in place. Each stage normalises its own copy, and ln_f normalises once more before the output layer turns the last row into logits, the next-token scores.

That was a decision. Call GPT-2's order pre-LN: normalise inside the branch, x + attn(ln(x)), and leave the bus alone. The original transformer, and GPT-1, were post-LN: normalise after the add, x = ln(x + attn(x)), which rescales the bus itself at every stage. GPT-2's paper records the move in one sentence, the same one that adds ln_f, and cites a precedent but gives no reason of its own.

A small test shows one. I trained two six-block models on this blog's text, one character at a time, with the same settings and no warmup (warmup: starting with a small step size and ramping it up). Loss is the model's average surprise at each next token; lower is better, and 0 would mean certain and right.

After 300 steps the pre-LN model reached about 2.4, better than the 2.56 a model gets from knowing only the previous letter. The post-LN one stalled at 3.2, roughly what letter frequencies alone give, and one seed run to 600 steps was still there. Given a 100-step warmup, it broke loose between steps 200 and 250. It's a toy, with two seeds and losses read off training batches, but the direction matches the published analysis of why post-LN needs a warmup.

Back to GPT-2 small: I deleted blocks. The intact model scores a loss of 3.59 on those 1,024 tokens. Remove any one of blocks 1 to 10, so the stream passes straight through, and the loss comes out between 3.61 and 3.89. Remove block 0 and it jumps to 9.15.

Overwriting is worse than removing. Keep a block, but let its output replace the stream instead of adding to it. For blocks 1 to 8 the loss is above 9.6. For blocks 2 to 7 it is above 10.8, the score of a uniform guess over the vocabulary. Worse than guessing.

The second widget on the original post plots that: switch it to overwriting and blocks 2 to 7 cross the dotted line that marks a uniform guess. Then pick block 0, which barely moves.

Block 0 is the exception, and the norms say why. The embedding enters with a norm of 4.6, and block 0's attention writes a vector seven times that length. After block 0 the stream is mostly block 0's output, so it behaves less like an edit than like a second embedding. Overwriting there costs nothing, 3.59, because adding already amounted to replacing.

Two thirds of the weights

You can count one block's parameters by hand, with d = 768 and the feed-forward width f = 4d. Attention's two projections are 4d² + 4d: 2,362,368 numbers. The feed-forward's two are 2·d·f + f + d, which is 8d² + 5d: 4,722,432. The two LayerNorms are 3,072. The feed-forward holds two thirds of every block in every GPT-2 size, because the ratio comes from the 4 * config.n_embd at model.py:82 and not from the width. Llama 3.1 8B changes both stages, and its feed-forward holds 81% of a block; Part 7 reads why.

Twelve blocks are 85 million parameters, and GPT-2 small has 124,439,808. The rest is almost all the token table: 38.6 million numbers, 31% of the model. It's used twice, as Part 2's lookup at the bottom and as the projection to logits at the top; model.py:138 ties them, the same array and not a copy. In GPT-2 XL the same table is 5.2% of the model. Small models spend nearly a third of their weights on the vocabulary; large ones barely notice it.

The cost is somewhere else

In software, a function's size and its running time come from the same code. Here they come apart. The parameters sit mostly in the feed-forward. The arithmetic follows T.

A forward pass costs about two operations per token for each number in a weight matrix, a multiply and an add; Part 4 counted them as FLOPs. Call the count of those numbers N, so the parameters cost 2N per token. Add the attention term, 4·L·T·d with L the number of blocks, for the scores and the weighted sum over the full square. For GPT-2 small at 1,024 tokens that's 247 million operations per token from the parameters and 38 million from attention. torch's FlopCounterMode agrees when attention is written out, Part 4's eager path: 284.8 million per token, exactly the formula's figure (285.1 if N is nanoGPT's get_num_params(), which also counts the biases and the LayerNorms).

On sdpa, the default, the same counter reported 247 million at both 256 and 1,024 tokens. In torch 2.14, the version I ran, it has no entry for the fused CPU attention kernel, the single routine sdpa calls instead of the four lines. The attention term never reaches its total, and nothing tells you so. PyTorch's main branch now registers that kernel (pull request #195801), so the next feature release should count the attention term on sdpa too.

Inside one block, the attention term is 18% of the arithmetic at 1,024 tokens, and it passes the linear layers at 4,608 tokens, six times d. The clock agrees. On my laptop's CPU, one block's attention stage, projections included, took 38% of the block's time at 256 tokens, 47% at 1,024, and 57% at 4,096, while holding a third of its parameters.

Be fair about the split. At the lengths GPT-2 was built for, the linear layers do most of the arithmetic, and the feed-forward is the larger half of that. The attention term only takes over past a length GPT-2 never accepts. Still, when you pick a model you pick two budgets. The parameters set the memory and are fixed at load; the context sets the cost and a growing share of the memory, and it arrives with each request.

What the ML engineer watches

Four habits, and the first two are arithmetic.

Count parameters by component, not in total. 4d² for attention, 2·d·f for the feed-forward, per block. Add the vocabulary times d, once, for the table. If you compare a 124M model with a 1.5B one, ask how much of each is vocabulary.

Estimate cost as 2N per token, and 6N to train. Training is a forward and a backward pass, and the backward, which computes the gradients, costs about two forwards. That's 6N per token and 6·N·D for a run of D tokens. nanoGPT's config for GPT-2, config/train_gpt2.py, runs 491,520 tokens per iteration for 600,000 iterations: 295 billion tokens. 6·N·D is 2.2 × 10²⁰ operations, 195 hours of one A100 at its bf16 peak, a day of an eight-GPU node. The file says to expect about five days and the README four, which puts the run at a fifth to a quarter of peak. nanoGPT's own estimator adds the attention term, model.py:296, and at 1,024 tokens that term is another 15%.

Depth is latency. Spend the same 85 million block parameters four ways: 6 blocks of width 1,088, 12 of 768, 24 of 544, and 48 of 384. At batch 1, one token through the stack took 3.2, 4.0, 4.7, and 6.7 ms on my CPU. Each block adds a fixed overhead, and blocks run one after another. If you serve one request at a time, depth is what you wait for, even when the arithmetic is equal.

Measure a block before you drop it. Layer pruning (shipping with some blocks removed) and early exit (stopping at a middle block when the answer looks settled) lean on the bus, and the bus is uneven. Removing a middle block of GPT-2 small raises its loss by 0.5% to 8%; removing block 0 raises it by 155%, and the last block by 41%. The numbers above took a minute on a laptop, on text from this blog. Produce them on your own traffic.

None of this is modelling work either. A block is four lines around a sum, and the sum is what lets a stack of twelve behave like one pipeline. Where the weights sit and where the time goes are two different answers, and both are arithmetic you can do before loading anything.

Part 5 of twelve. A block reads the stream through two stages, attention and a feed-forward, and adds both results back: the plus sign is why removing a middle block costs a few percent of the loss rather than the model. The feed-forward holds two thirds of each block's weights; the attention term's share of the arithmetic grows with the context. The next part reads the five decisions GPT-2 made around the stack, and lands on the series index when it's out. If you arrived here directly, Part 1 reads the tokenizer, Part 2 the table a token id indexes, Part 3 the shapes, and Part 4 the stage this block is built around.

Written with AI assistance. See AI policy.

Attention is a learned weighted average, and the cost is in the square

Carlos Chinchilla Corbacho — Fri, 25 Sep 2026 13:00:00 +0000

Originally published at cchinchilla.dev. Part 4 of From code to weights, a 12-part series on ML fundamentals for engineers.

Part 3 ended on a square. Two tensors in one decoder block carry T twice, [B, H, T, T]: a T × T square per head, H heads, B sequences. At the widget's defaults they were 89% of the block's memory. This post reads what fills them. It's a weighted average. The average is ordinary; the weights are computed from the input, and that is the new part.

I wrote attention in four lines and checked it against F.scaled_dot_product_attention, torch's own implementation. Then I read LlamaAttention.forward with the shapes in the margin, and priced it for Llama 3.1 8B at the context lengths people promise.

This is part of a series for engineers moving into ML engineering. Each part names one thing that does not behave the way software behaves. This one is cost: you cannot read it from the code.

A dictionary that matches a little

A dictionary lookup is exact. The key matches or it doesn't, and you get one value back. Attention is the same operation with the match made soft. Every key matches a little, and you get back the average of all the values, weighted by how well each key matched.

Those are attention weights, not parameters. They aren't stored anywhere; the model computes them from the input on every call. Until the cost section, "weights" means these. Three learned matrices turn each input row into a query, a key, and a value. The query is compared against every key, and the scores become weights:

q, k, v = x @ Wq, x @ Wk, x @ Wv             # [B, T, d] each
scores = q @ k.transpose(-1, -2) / sqrt(d)  # [B, T, T]
weights = scores.softmax(-1)                # every row sums to 1
out = weights @ v                           # [B, T, d]

Read the shapes. x is Part 3's [B, T, d]. The three projections keep it. scores is [B, T, T]: one number for every pair of positions, which is the square. The softmax turns each row into weights that sum to one, and the last line is the average. The four lines agree with F.scaled_dot_product_attention to 1e-5.

Two properties follow from "average". Each output coordinate sits between the smallest and largest value at that coordinate; attention never leaves that range. And if you replace a softmax row with a one-hot row, a single 1 and zeros elsewhere, the output is exactly the value at that position. The soft dictionary contains the hard one as a limit.

Why the square root

The division by sqrt(d) looks like a detail. It's the difference between a model that trains from the first step and one that starts saturated. The reason is arithmetic you can check in a shell.

A dot product of two random vectors of width d, with unit-variance entries, has variance d. I measured it: 4,104 at d = 4096. Feed scores that wide into a softmax and it saturates: one weight goes to nearly 1 and the rest to nearly 0. With 1,024 keys, the largest weight in a row averaged 0.976. The row had 0.09 bits of entropy out of a possible 10. That's a hard lookup wearing a soft one's clothes, at initialisation; trained models sharpen rows on purpose, from a start they can learn from. Divide by sqrt(d) and the variance is 1.0, the largest weight 0.017, the entropy 9.27 bits. Averaged over the scores, the gradient that reaches them is about an order of magnitude larger: 12 to 18 times across seeds, with the loss read through the average. Through a single weight the ratio swings from a few times to many orders of magnitude, depending on which rows that key happens to win. A saturated softmax has almost nothing to learn from: its gradient is under 1e-6 for 99.8% of the scores, and what's left piles onto the few near-ties.

Llama writes it once: self.scaling = self.head_dim**-0.5, at modeling_llama.py:226. The width that matters is the head's, not the model's.

Heads are a view, the mask is a triangle

Multi-head attention is the four lines above run once per head, H times, on slices of d. The slicing is free. q.view(B, T, H, d // H).transpose(1, 2) gives [B, H, T, d/H] and shares memory with q. The head axis is a stride. Scores are then [B, H, T, T], one square per head. Head 0 of the batched call equals attention run on head 0 alone. What the heads buy is H different sets of weights over the same positions. Each lives in a slice of width d/H: 128 in Llama 3.1 8B, with 32 heads.

The mask is where a decoder stops being an average over everything. To predict the token after position t, the model may read positions 0..t and nothing after, so the scores above the diagonal are set to -inf before the softmax and their weights come out as zero. I tested it the direct way: attention over eight positions, then rewrite the keys and values at positions 5 to 7. Positions 0 to 4 returned the same output to 1e-6. Without the mask, all eight changed.

The original post has a widget for this that doesn't survive being pasted into a feed: switch between the three masks and read the live-cell count change. The third chip, the window, is the last lever in the cost section.

The triangle composes with Part 3's padding mask by addition. Left-pad a sequence by two and its first two query rows are all -inf; a row of all -inf is 0/0 in the softmax, NaN. Same trap, one part later, in the four-line form. transformers fills its eager mask with the dtype's minimum instead (masking_utils.py:602-604); those rows come out uniform. Quiet, and still meaningless.

The eager path computes the exponentials and their sum in fp32 whatever the model's dtype, then casts the row back: dtype=torch.float32 at modeling_llama.py:208. A normaliser summed in bf16's eight significant bits is the wrong place to save memory.

Reading `LlamaAttention.forward`

modeling_llama.py:243-281 in transformers 5.17.0 is the four lines with the heads, the mask, and a cache written out. Condensed, with shapes for the 8B model:

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)  # [B, 32, T, 128]
key_states   = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)  # [B, 8, T, 128]
value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)  # [B, 8, T, 128]
query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
if past_key_values is not None:
    key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
attn_output, attn_weights = attention_interface(self, query_states, key_states, value_states,
                                                attention_mask, scaling=self.scaling)
attn_output = self.o_proj(attn_output.reshape(*input_shape, -1))

Three things in it aren't in the four lines. past_key_values.update is the cache. At generation time the keys and values of every earlier token are kept, so each new token computes one query row and reads T stored keys.

The key and value have 8 heads where the query has 32: k_proj maps 4,096 to 1,024. That's grouped-query attention (GQA), against one key-value head per query head (MHA). It exists for the cache. And attention_interface is a dispatch whose fallback is eager_attention_forward, modeling_llama.py:191-213: the name means the four lines run as written, plus repeat_kv, which copies the 8 key heads to 32 at runtime so the matmul lines up. Set _attn_implementation to sdpa, the scaled_dot_product_attention from the first section, or to flash_attention_2, and a different kernel (the routine that runs the arithmetic) computes the same maths. The rotary line is Part 7's.

The cost is in the data, not the code

In software you read a function's cost from its code. A loop over the input is linear; two nested loops are quadratic; you can tell before it runs. Attention is one function; its code is the four lines. There is no loop in them. The square is inside a single @, and nothing in the code's shape says quadratic. Its cost is set by T in the request: positions squared, per head, per layer.

Take Llama 3.1 8B's shape. The scores for one sequence in one layer, in bf16, are 268 MB at 2,048 tokens, 4.3 GB at 8,192, and 1.1 TB at the 131,072 the config declares; the eager path holds three such squares at its peak, two of them in fp32: about five times those bytes. The weights of the whole model are 16 GB. Nobody materialises that last number. The kernel is why.

The second widget on the original post is that multiplication, live: drag T to 16k with eager selected and one layer's scores outgrow the whole model's weights; at 131k they are about seventy times larger. Switch to sdpa and the number is struck out — never allocated.

I timed the last three lines on my laptop's CPU with 8 heads of width 64. From 512 tokens to 8,192, sixteen times the input, the eager form went from 1.9 ms to 284 ms: 150 times. A linear layer over the same input went from 0.23 ms to 2.5 ms, eleven times. Doubling T from 4,096 to 8,192 multiplied the linear layer by 1.95 and attention by 4.3: the square, within timing noise.

Three levers move the number. The first is the kernel. F.scaled_dot_product_attention with its flash or memory-efficient backend computes the same numbers in blocks and never holds the T × T matrix; its math fallback still does. At 4,096 tokens, in fp32, the profiler charged 545 MB to the eager form's matmuls, the 537 MB square plus its output, and 8.5 MB to sdpa, and it ran 2.4 times faster.

Flash attention is that idea on a GPU. The arithmetic is still quadratic. What changes is what must exist in memory at once. The line that picks it is one config field, _attn_implementation. In 5.17.0 the default at load is sdpa; eager is what you get by asking for it. With padding in the batch, sdpa still holds the mask itself, [B, 1, T, T] booleans, 17 MB at 4,096; without padding, transformers passes no mask and lets the kernel apply the triangle (is_causal=True).

The other two change what's computed. Grouped-query attention shrinks the cache. With 8 key-value heads the 8B model stores 131 kB per token across its 32 layers, 17 GB for a 128k context, already a little more than the weights, and with 32 heads it would be 524 kB per token and 69 GB, four times the weights. A sliding window caps how far back a query reads. Llama 3.1 has none; Mistral 7B v0.1 had one, and Gemma 2 and 3 interleave windowed layers with global ones. transformers writes it as kv_idx > q_idx - sliding_window, at masking_utils.py:100. At 32,768 tokens a 4,096 window leaves 126 million live cells where the causal triangle has 537 million. Below the window the cost is a triangle. Above it, a band of width W. A band grows with T, not its square.

Be fair about the square. Count operations rather than bytes, over the prompt pass, and the linear layers of the 8B model do more of them than attention until roughly 30,000 tokens, about twice that if you count only the triangle a causal kernel computes. At 8,192, attention is about a fifth of the block's floating-point operations, its FLOPs, counting the full square, and about an eighth counting the triangle.

The square is a memory problem long before it's a compute problem. The number to compute before promising a context length is the memory one.

What the ML engineer watches

Four things, and the first is a multiplication.

Price [B, H, T, T] and the cache before promising a context. Heads times T squared times two bytes, per layer. Add the cache per token times the longest sequence times the batch. Do it at the batch you'll serve. The config's max_position_embeddings is what the model accepts after its rotary scaling (131,072 here: an 8,192 pre-training window, scaled and then trained out to 128k), not what your card can hold.

Know which kernel runs. eager returns the weights and materialises the square; sdpa and flash don't. The choice is _attn_implementation, and it decides whether the 545 MB above exists. A debugging tool that wants the weights can't get them from sdpa: output_attentions=True returns an empty tuple there, and the fix, loading with attn_implementation="eager", materialises the square on every call.

Compose the masks on purpose. Causal, padding, and window are ANDed; added, in the -inf form. A query row with nothing left to read is NaN in that form and uniform in the library's, and neither is a number you want. Part 3's habit applies twice now: the last-token index comes from the mask, and the mask travels with the batch.

Measure latency at the longest prompt you'll accept. The linear layers scale with T. Attention scales with its square. The cache read at each generated token scales with everything already in it. A p99 measured at 500 tokens says nothing about 8,000. The 1.9 ms to 284 ms above is the shape of the surprise.

None of this is modelling work either. A weighted average is four lines, and the weights come from three learned matrices. The square they live in is the first cost in this series that the input sets and the code doesn't.

Part 4 of twelve. Attention is a weighted average whose weights the model computes from its own input: the average is four lines, the T × T matrix of weights is the cost, and the kernel decides whether that square is ever held in memory. Attention is one stage. The next part reads the whole block around it — six tensors, and a residual stream that every stage adds to — and lands on the series index when it's out. If you arrived here directly, Part 1 reads the tokenizer, Part 2 the table a token id indexes, and Part 3 the shape both of them hand to this one.

Written with AI assistance. See AI policy.

Shapes are the type system of ML, and nobody checks them for you

Carlos Chinchilla Corbacho — Thu, 24 Sep 2026 13:00:00 +0000

Originally published at cchinchilla.dev. Part 3 of From code to weights, a 12-part series on ML fundamentals for engineers.

Part 2 ended on a signature. A batch of sequences of embedding rows is a tensor of shape [B, T, d]. That line is the closest thing this stack has to a type. This post reads it the way you'd read a function's signature, then shows what the runtime does when the signature is wrong: usually nothing.

I read the broadcasting rule as torch writes it. Then I wrote four shape bugs on purpose and watched which ones the runtime caught. It caught one.

This is part of a series for engineers moving into ML engineering. Each part names one thing that does not behave the way software behaves. This one is the type system: there isn't one.

A shape is a signature

A tensor is an N-dimensional array plus a description of how it sits in memory. [B, T, d] says: B sequences in the batch, T positions in each, d numbers at each position. Read it the way you'd read f(batch: List[List[Vec]]), with one difference that matters later: the runtime never looks at the names, only at the sizes.

The memory description is the part software engineers already know. A tensor of shape [4, 8, 16] has strides (128, 16, 1): move one step along the first axis and you skip 128 numbers. Transpose the first two axes and the shape becomes [8, 4, 16] with strides (16, 128, 1), the same memory read in a different order; data_ptr doesn't change. That's why transpose is free and why a view that flattens the result refuses, with a message that says so, while reshape copies when it has to. A tensor is a pointer, a shape, and a stride tuple; the view operations you'll call are arithmetic on the last two.

Every operation has a shape signature of its own: what goes in, what comes out, which axis disappears. mean(dim=1) on [B, T, d] returns [B, d]; the T is gone. A linear layer maps [B, T, d] to [B, T, d_out]; only the last axis changes. Writing the signature next to the call is the discipline this post is about; nothing else will write it.

The rule is five lines

Two tensors of different shapes can still be added, and the rule that decides how is broadcasting. torch keeps a readable copy of it in Python, under torch.broadcast_shapes, which calls _broadcast_shapes in torch/_refs/__init__.py (lines 388-470 at v2.13.0). Stripped of the symbolic-shape machinery, the rule is this:

common_shape = [1] * reduce(max, (len(shape) for shape in shapes))
for arg_idx, shape in enumerate(shapes):
    for idx in range(-1, -1 - len(shape), -1):          # align from the right
        if common_shape[idx] == 1:
            common_shape[idx] = shape[idx]              # 1 stretches to anything
        if shape[idx] == 1:
            continue                                    # ... in either direction
        torch._check(common_shape[idx] == shape[idx],   # otherwise sizes must match
            lambda: f"Attempting to broadcast a dimension of length {shape[idx]} at {idx}!")
return common_shape

Align the shapes from the right. A size of 1 stretches to match the other. Equal sizes pass. Anything else raises. That's the whole rule. The eager path runs the same rule in C++ and raises with a shorter message. It exists so that you can add a bias of shape [d] to [B, T, d] without a loop or a copy.

The rule has one property that the rest of this post is about: it decides by size, not by meaning. [8, 16] + [16] works because 16 equals 16. [8, 16] + [8] raises, because 8 is not 16 and 8 is not 1. Whether the 16 on the right is a sequence length or a hidden size or a batch, the rule has no idea, and neither will you when it's wrong. When two of your axes happen to be the same size, it can't tell them apart either. The data scientist has leaned on this rule in numpy for years; the software engineer is meeting a coercion rule with no declared types. Both should read it as the second thing, not the first.

The bug that only happens at batch size sixteen

The first bug. Attention scores have shape [B, T, T], one T × T matrix per sequence. A padding mask has shape [B, T]: which positions in each sequence are real. In its additive form (0 for real, -inf for padding), adding it to the scores switches padding off. The correct form is scores + mask[:, None, :]: the None makes the mask [B, 1, T], so it stretches across the query axis and masks the keys.

Drop the None. With B = 8 and T = 16, scores + mask raises: 16 is not 8. With B = 16 and T = 16, it runs. At B = 1 it runs and is correct by accident: 1 is what the None would have inserted.

The rule aligns [16, 16] against [16, 16, 16] from the right and reads the mask as [1, B, T]. The mask meant for sequence b lands on query row b of every sequence. On random lengths I measured a third of the positions masked wrong and a fifth of the attention weight on padding. The softmax after it is finite, the loss is finite, and the run continues. No warning, because there is nothing to warn about: the rule was followed.

The original post has a widget for this that doesn't survive being pasted into a feed: move the two sliders until B and T meet and the verdict flips from an error to a silent result. Then pick [B, T, 1]: that mask runs at every size and is wrong at every size, because it switches off queries instead of keys. It's the loud version — a padded query's row is all -inf and its softmax is NaN. The rule is the same in all three cases.

The second bug is the classic one. Predictions of shape [256], targets of shape [256, 1], subtracted for a squared error. The rule stretches both to [256, 256], every prediction against every target. The mean of that is a number. It trains.

I trained a one-feature linear regression through it for two thousand steps. The loss fell and settled at 9.7. The fitted slope was 0.000, the intercept was the mean of the targets, and the R² was 0.0. The model learned to ignore its input. That's the best it can do when the loss compares it with everyone else's target. The library loss, F.mse_loss, warns about this mismatch. A hand-written ((pred - target) ** 2).mean() doesn't, and the hand-written form is the one that ends up in notebooks.

The third bug never raises where it happens. On a tensor of shape [16, 16, 64], mean(dim=0) and mean(dim=1) both return [16, 64]. Different numbers. Same shape. Nothing downstream can tell. On [3, 5, 7] the same two calls return [5, 7] and [3, 7], and the first thing that tries to combine them raises. The bug was there in both cases; only the sizes decided whether it was visible.

The fourth is the control. [3, 5, 7] plus [3, 5] fails because 7 is not 5. It was caught for the one reason the others weren't: the sizes differ.

Be fair about the rule. It's doing what it was asked to. The alternative exists: a runtime that carries axis names and refuses to add T to B. torch shipped one as a prototype, named tensors. It never left prototype, and 2.13 removed it to cut overhead and code bloat. Part of why it never caught on: every library in the stack would have had to agree on the names, and sizes are the only thing all of them share. The rule is right. It is not a type check, and nothing in the stack is.

Padding is a shape decision

Sequences don't come in rectangles. Three sequences of lengths 5, 3, and 1 become a [3, 5] tensor only by padding, and for those lengths 40% of the rectangle is padding. Every operation over T then needs to know which cells are real. That knowledge is a second tensor, the mask, and it has to travel with the first.

Which side you pad on is part of the signature. Pad on the right and the last real token sits at index length - 1: 4, 2, 0 for the three above. Pad on the left and it sits at 4 for all of them. Decoders that generate one token at a time prefer that; the position to read is always the last. Either choice works. Mixing them is the bug. Code written for right padding and fed a left-padded batch reads padding, or the wrong token, where it expects the last one.

The mask is not optional even when the model doesn't attend. Mean-pool that [3, 5] batch without it and the length-1 sequence's vector averages one real row with four of padding. On arange(1, 31).reshape(3, 5, 2) the unmasked mean of that sequence is [25, 26]; the masked mean is [21, 22], its only real row. Part 2's pooling code divides by the mask sum for this reason. The division is the signature, written out.

Memory is shape times bytes

A tensor's cost is arithmetic. Activations of shape [32, 2048, 4096] are 268 million numbers: 1.07 GB in fp32, 537 MB in bf16, 268 MB in int8. That's the bill, and you need nothing about the model to compute it: the shape and the width of one number. It decides whether a batch fits on a card.

The second widget on the original post is that multiplication, live: drag the three sizes and switch the format. At the defaults one block costs 77 GB in fp32 and the two square rows are 89% of it; pull T down to 128 and they fall to a third. Those two rows are the next part's subject.

What each format gives up is also arithmetic. fp32 keeps 23 bits of mantissa. bf16 keeps 7 and the whole fp32 exponent: it writes π as 3.140625 and rounds 1 + 0.004 to 1.0078125, but its range is fp32's, to within the last rounding step. fp16 keeps 10 bits and only five of exponent: it tops out at 65,504, and small gradients underflow to zero unless the loss is scaled first. That's why training moved to bf16 and serving reaches for int8. Each is a decision about which precision a tensor doesn't need.

The batch axis is the one you control at serving time, and it moves two numbers in opposite directions. I timed a [B, 4096] by [4096, 4096] matmul on my laptop's CPU. It costs 447 µs per row at B = 1 and 22 µs per row at B = 512: twenty times the throughput. The call itself went from 0.45 ms to 11.4 ms; each request waited twenty-five times longer, plus whatever it spent waiting for the batch to fill. Five repetitions on one laptop: the ratio is the finding, not the digits. Dynamic batching is the knob between those two numbers, and its setting is a product decision wearing a shape.

What the ML engineer watches

Four habits, and the first two are one habit.

Annotate shapes at function boundaries. jaxtyping with beartype turns the signature into a check that runs on every call:

@jaxtyped(typechecker=beartype)
def pool(x: Float[Tensor, "B T d"], mask: Bool[Tensor, "B T"]) -> Float[Tensor, "B d"]:
    m = mask[:, :, None].float()
    return (x * m).sum(1) / m.sum(1)

Reduce the wrong axis inside and it raises on the return, because [T, d] is not [B, d]. Pass the column-shaped target into the squared error and it raises before the rule can stretch anything. Two of the bugs above, caught at the boundary (the reduce only when B ≠ T), with the shape written where the next reader looks. The batch-sixteen bug, and the wrong-axis reduce when B equals T, need the next habit.

Test with sizes that can't collide. The bug at batch size sixteen is invisible to the annotation. When B and T are both 16, the names bind to the same number and the check passes; so does the wrong-axis reduce, because [T, d] and [B, d] are the same [16, 64]. The fix is in the tests, not the code: use B = 3, T = 5, d = 7. Every dropped None and every mix-up between B, T and d then produces a size that matches nothing, and the runtime raises. Size-1 axes still stretch at any size, and the two Ts of the scores are always equal; those need the annotation or a test that checks values. Round numbers in tests are how the bug at batch size sixteen survives to production. That's why the first two are one habit: the annotation is only as strict as the sizes it sees.

Mask and padding side are one decision. Record the side next to the tokenizer config. Compute the last-token index from the mask, never from T. Reject a batch whose mask disagrees with its padding. The pooling code in Part 2 already divides by the mask sum; make sure yours does.

Price a tensor before allocating it. Shape times bytes, at the batch you'll serve, in the format you'll serve it in. That multiplication has an answer before a single weight is loaded.

None of this is modelling work either. A shape is the one type this stack never checks: a device mismatch raises, a wrong axis broadcasts. The check is yours to write.

Part 3 of twelve. A shape is a signature that nobody verifies: broadcasting decides by size, so a wrong axis raises only when the sizes happen to differ, and a bug can live at one batch size alone. The attention scores in the first widget and the two amber rows in the second carry T twice, one T × T matrix per sequence (per head, in the second). The next part reads where that square comes from and what it costs. If you arrived here directly, Part 1 reads the tokenizer and Part 2 the table a token id indexes.

Written with AI assistance. See AI policy.

An embedding is a lookup table, and everything else is how you fill it

Carlos Chinchilla Corbacho — Tue, 22 Sep 2026 13:00:00 +0000

Originally published at cchinchilla.dev. Part 2 of From code to weights, a 12-part series on ML fundamentals for engineers.

Part 1 ended on an index. The tokenizer turns text into ids, an id is a row number, and the row lives in a matrix the model was born with. This post reads that matrix. It's smaller than it sounds: a lookup table, indexed by integer, with learned contents. The science is in how the contents get there. The engineering is in what you can't do with the table once they're in.

I read two files for this post: the forward of torch.nn.Embedding, which is one call, and the pooling module of sentence-transformers, which is where the other kind of embedding, the one you put in an index, gets made. Then I built a 48-line index and broke it three ways on purpose, to measure what breaks.

This is part of a series for engineers moving into ML engineering. Each part names one thing that does not behave the way software behaves. This one is the index: it has no migration.

A matrix and an index

If you've written a hash map, you've written most of an embedding layer. nn.Embedding(V, d) allocates a matrix of V rows and d columns. Looking up id i returns row i. That's the whole forward pass, and torch's own source says so in sparse.py:188-197:

def forward(self, input: Tensor) -> Tensor:
    return F.embedding(
        input,
        self.weight,
        self.padding_idx,
        self.max_norm,
        self.norm_type,
        self.scale_grad_by_freq,
        self.sparse,
    )

F.embedding is a gather. That's checkable rather than a matter of trust: for a fresh Embedding(50257, 768) and three ids, torch.equal(emb(ids), emb.weight[ids]) is True. No transformation, no hashing, no similarity. A row per id.

The sizes are where it stops looking like a hash map. GPT-2's input table, wte.weight, is 50,257 rows by 768 columns: 38.6 million numbers, 154 MB in fp32, every one of them a trained parameter. all-MiniLM-L6-v2, the sentence model the rest of this post runs on, has a 30,522 by 384 table for the same job. Row 11,241 of GPT-2's table is the vector for " token" for the reason Part 1 gave: that's where the tokenizer sends it. The contents of the row were learned. The address was fixed before training started.

For the engineer coming from software, the surprise is that the keys mean nothing. A hash map's key carries its own identity; row 11,241 means something only inside the model that trained it. For the data scientist, the surprise runs the other way. This table isn't a word2vec artifact you produce and then use. It's a parameter inside the model, updated by the same gradient step as everything above it. It never leaves.

Two ways to fill the table

A row gets its numbers in one of two ways, and the difference decides what you can do with the result.

The first is the one above. The table is a parameter of a language model. Rows start random and gradients flow into them through the layers above. After training, each row is whatever helped the model predict the next token. Nobody designs the space; it's a side effect. Part 8 reads the gradient that fills it.

The second is the one most engineers actually deploy. You take a trained model, run a piece of text through it, and take the output as the text's vector. The row isn't stored anywhere; it's computed on demand, and it depends on the whole input, not on one id.

The word bank has one row in the input table of that sentence model, row 2,924. A position table gets added to it, but the word's row is the same in "she sat on the river bank" and in "she opened a bank account". At the output, the two vectors for that same token have a cosine of 0.77 to each other. Same lookup, different result, because a position and six layers sit between the two. That's what contextual means, and it's the whole difference between a static embedding and the kind you put in a search index.

The output is a vector per token. A sentence needs one vector, so something has to collapse T vectors, one per token, into one, and that something is pooling. sentence-transformers keeps every variant in one place, and on the default padded path the branch a mean-pooled model runs is pooling.py:200-213:

elif mode in ("mean", "mean_sqrt_len_tokens"):
    if mean_sum is None:
        mask = attention_mask.unsqueeze(-1).expand_as(token_embeddings).to(token_embeddings.dtype)
        mean_sum = (token_embeddings * mask).sum(dim=1)
        # ... mean_mask is the count of real tokens (or a supplied per-token weight sum), clamped away from zero
    if mode == "mean":
        output_vectors.append(mean_sum / mean_mask)

Sum the tokens the attention mask says are real, divide by how many there were. Six modes live in that function, each a different way to collapse the same T vectors: cls takes the first, lasttoken the last, max the largest per dimension. A model is trained with one of them, and the choice is part of the artifact.

So is the step after it. The small model's own modules.json lists what runs when you call encode:

[
  {"idx": 0, "name": "0", "path": "", "type": "sentence_transformers.models.Transformer"},
  {"idx": 1, "name": "1", "path": "1_Pooling", "type": "sentence_transformers.models.Pooling"},
  {"idx": 2, "name": "2", "path": "2_Normalize", "type": "sentence_transformers.models.Normalize"}
]

Transformer, then pooling, then normalisation to unit length. Not even the family agrees on the third: paraphrase-MiniLM-L6-v2 ships without it, and its raw vectors have norms around six. Three modules, and the last two are the ones a client tends to reimplement. Reimplemented differently, they change the answer. By how much is measured two sections down.

Close is a dot product

Once every vector is unit length, cosine similarity is a dot product, and nearest-neighbour search is a matrix multiply followed by a sort. GPT-2 reads its own table the same way on the way out: the logits for the next token are a dot product against every row of wte. That's the whole retrieval side, and it's why the index bill is arithmetic: ten million documents at 384 dimensions is 15.4 GB of fp32 before any index structure. At 768 it's 30.7 GB. At 1,536, the width of a popular hosted embedding, 61.4 GB. Dimension is a cost you pay per document for as long as the index exists. Whoever trained the model decided it, not you.

The number the search returns is less useful than it looks. On my 48-line corpus, the best match for "how do I undo a bad release" has a cosine of 0.30 to the query, and it's the right line: the rollback procedure. The best match for "the migration is stuck waiting on a lock" scores 0.71. Both are correct. A cosine isn't a confidence; it's a position in a space whose scale the model chose. Compare ranks within one model, and never compare the raw number across two.

There is no schema migration

Here is the part that's different from software. When you change a database schema you write a migration, run it, and the old rows come out the other side in the new shape. An embedding space has no migration. Two models produce two spaces, and no map between them ships with either, because neither was designed; each is whatever training settled on. You can fit one after the fact, and people do. It's a learned approximation with its own error, not a migration: a linear map fitted on four thousand unrelated sentences lifts the stale index below from 60% to 70%, and stops there. Re-embed with a new model and every vector you stored is in a coordinate system nothing else uses.

I expected that to show up as garbage. It shows up as something worse.

The test: 48 lines from a runbook, eight each on deploys, auth, billing, databases, observability, and on-call. I indexed them with the L6 model and queried through all-MiniLM-L12-v2, the upgrade an engineer would reach for (same family, same 384 dimensions, so the index accepts the vectors without complaint).

Against what the upgraded index would return, the stale one keeps 60% of the top five. Random would keep 10%, and guessing five lines from the right topic would keep 35%. The old index queried by the old model keeps 77%: the mismatch costs seventeen points, and the other twenty-three are the two models disagreeing about the answer. For "user keeps getting logged out", the first hit is still the session-expiry line, and the third is the database backup retention policy. Nothing in the response says which is which.

Be fair about the number. Sixty percent isn't a scrambled index, and the reason is lineage, not English: both models were fine-tuned from the same parent checkpoint, the L6 one after dropping every second layer, so the two spaces started in the same coordinates. The same line under the two models has, on average, a cosine of 0.54 to itself, far from orthogonal. Two models that only shared training data wouldn't share coordinates, and a stale index between them should land near random. Six tight topics of eight lines each is also the friendliest corpus a stale index could hope for. I haven't measured a production index, and I'd expect it to keep less. What the number does support is the shape: partial, plausible, silent.

The pooling module does the same thing without a model change. Take the same weights and pool with cls instead of mean, at index time and at query time, and the top five keeps 50% of the mean-pooled answer. Mix them, mean in the index and cls on the query path, and it keeps 70%: more than consistent cls, because one side of the pair is still the pooling the model was trained with. A client that reimplements pooling has changed the model.

There's an interactive version of the next bit that doesn't survive being pasted into a feed, so it lives on the original post: drag the reindex forward, then flip which model encodes the queries. On the new model, recall climbs as the table migrates. On the old one it starts at 0.77, because two consistent models already disagree on a quarter of the answers, then falls through the middle, to 0.63, gets back to 0.77 only in the last few lines, and finishes below it, at 0.73: by the end, every line in the index was written by a model the query path isn't using.

That second curve is the one production runs. The index is a batch job; the query encoder is a deployed service; the two ship on different days.

What the ML engineer watches

Four habits, all cheap next to a reindex.

The embedder is part of the index. Store the model's hash, its pooling mode, its normalisation step, and its sequence cap in the index metadata. Refuse a query whose encoder doesn't match. The modules.json above is half of that contract; the pooling mode sits next to it in 1_Pooling/config.json, and the hash you add yourself. The cap matters: the L6 model reads 256 tokens and the L12 upgrade reads 128, so the swap that broke the index also halved what each vector sees, silently. Ship all of it with the index, not in a client's config.

Cut over, don't roll. Build the new index beside the old one, switch queries and index together behind one flag, and delete the old one afterwards. A rolling reindex under a live query path is the falling curve.

Measure neighbour overlap, not cosine. Keep a fixed set of queries with their expected top-k, and check recall@k against it on every model change. A cosine threshold tuned on one model means nothing on the next; the neighbour set is the only observable that survives a swap. It measures drift, not relevance: the migrated top five for the logout query still has a cold-storage line in it.

Dimension is a bill. N times d times four bytes, before the index structure. Half-precision halves it; some newer models let you keep only the first k dimensions. Both can be applied to vectors already stored, without re-embedding; truncated vectors have to be renormalised and queries cut to the same k, and both choices belong in the index metadata with the rest. Price a larger model's dimension against the corpus you'll have in two years, not the one you have now.

None of this is modelling work either. An embedding index looks like a database with a float column, right up to the day someone upgrades the model that wrote the column.

Part 2 of twelve. An embedding is a row of a matrix, filled either by the gradient or by a model's output, and the space it lives in has no migration: change the model and every stored vector goes stale, silently. A batch of sequences of those rows is a tensor with the signature [B, T, d]. The next part reads that signature the way you'd read a function's, and finds that the runtime never checks it.

Written with AI assistance. See AI policy.

A model doesn't read text: what a tokenizer decides for you

Carlos Chinchilla Corbacho — Sun, 20 Sep 2026 19:14:20 +0000

Originally published at cchinchilla.dev. Part 1 of From code to weights, a 12-part series on ML fundamentals for engineers.

A language model has never seen a letter.

It sees integers. Somewhere between your string and the first matrix multiply there's a function that turns text into a list of ids, and that function was frozen the day the model was trained. It isn't preprocessing. It's part of the artifact. Swap it and the weights are still there, but the model you had is gone.

I came to this from the compiler side, where the parser is something you own and can change on a Tuesday. Here you own it exactly once, before training, and never again.

This is the first post of a series for engineers moving into ML engineering, from software or from data science. Each part names one thing that does not behave the way software behaves. This one is the parser: it ships inside the model.

A lexer with a different goal

If you've written a compiler, you've written a tokenizer. The lexer takes characters and emits a stream of symbols: identifiers, keywords, literals. Its goal is to recognise structure so the parser can build a tree.

A model's tokenizer has the same shape and a different goal. There's no structure to recognise. The goal is to compress: turn a string into as few ids as possible from a fixed vocabulary, so the model sees fewer positions and the vocabulary stays small enough to put a row of the embedding matrix behind every id.

Both extremes fail. Characters give a tiny vocabulary and sequences that are far too long; every position costs attention, and attention costs the square of the length. Whole words give short sequences and an unbounded vocabulary, plus a hole for every word that isn't in it. The data scientist knows this hole as out-of-vocabulary; the software engineer knows it as an unhandled case.

Subwords are the compromise. Sennrich, Haddow and Birch (2016) took byte-pair encoding, a compression algorithm from 1994, and pointed it at this problem instead: learn the sub-word units from the corpus rather than declaring them up front.

BPE is one loop

I read tiktoken's _educational.py. Its encoder is a readable stand-in for the Rust core, which I checked by running both over every sample below and comparing ids. Its trainer is a stand-in for nothing: the Rust core ships no trainer, so bpe_train (_educational.py:119-185) is the only version of the loop in the package. It is one loop. The body of bpe_train:

while len(ranks) < vocab_size:
    stats = collections.Counter()
    for piece in words:
        for pair in zip(piece[:-1], piece[1:]):
            stats[pair] += 1
    most_common_pair = max(stats, key=lambda x: stats[x])
    token_bytes = most_common_pair[0] + most_common_pair[1]
    token = len(ranks)
    # Add the new token!
    ranks[token_bytes] = token
    # ... then replace that pair through the corpus, and loop

Start with 256 tokens, one per byte. Count every adjacent pair across the corpus. Take the most frequent pair, give it the next id, replace it through the corpus left to right. Repeat until the vocabulary is the size you asked for.

The vocabulary is the list of merges, in order. That order is the whole model of the language. Encoding replays it: split the input into bytes, find the adjacent pair with the lowest merge rank, merge it, repeat until nothing merges. bpe_encode (_educational.py:83-116) is thirty-four lines. It never iterates the merge list; training takes the most frequent pair and encoding takes the lowest-ranked one, which comes to the same thing.

The original has an interactive widget that replays the merge loop on a toy corpus, one merge per click. Open it on cchinchilla.dev to use it.

The widget runs that loop on a toy corpus, one merge per click. Twelve of its eighteen steps are ties, so the tie-break rule (first pair seen wins) is doing real work.

Two details matter for the engineer. First, it works on bytes, not characters, so any input encodes and there's no unknown token. A character the merges never saw costs up to one token per UTF-8 byte instead of raising. Under GPT-2, ß is one token because its two bytes merged during training, 漢 is three, one per byte, and a Gothic letter is four. Newer vocabularies bought some of that back; o200k_base has 漢 whole. The Gothic letter still costs four everywhere. Second, before BPE runs there is a regex that splits the text into chunks, and merges never cross chunk boundaries. GPT-2's pattern keeps a leading space attached to the word that follows it, so " token" and "token" are different tokens. That's why the model treats the start of a line differently from the middle of one, and why a stray space in a prompt template changes the output.

The vocabulary is the training corpus

GPT-2's vocabulary has 50,257 entries. That number isn't round because it's a sum: 256 byte tokens, 50,000 learned merges, and one special token, <|endoftext|>, id 50256. Radford et al. (2019) describe the byte-level design those numbers come out of. The two newer OpenAI encodings I checked, cl100k_base and o200k_base, hold 100,261 and 200,000 tokens spread over id ranges of 100,277 and 200,019 (tiktoken's n_vocab); the gaps are unused ids.

The merges encode the training corpus. GPT-2 was trained mostly on English web text, so English words are cheap: " tokenizer" is two tokens. The same concept in Spanish, " tokenizador", is three. " desvanecimiento" is six. A German word with an umlaut, " Wörter", is three; one of those three is the two bytes of ö welded to the r after it. The tokenizer has no notion of a character; it has bytes and frequencies.

Numbers are the case that surprises engineers. GPT-2 tokenizes 1234567 as 123, 45, 67, because that's what the merge ranks happen to produce. cl100k_base changed the regex to cap digit runs at three, which bounds the damage without fixing it: the cap groups left to right, so 1000 becomes 100 and 0. That's a tokenizer decision with a direct effect on what the model can compute, and it was made in a regex.

Be fair about why. GPT-2's merges were learned from frequency alone, and in web text 1000 is common enough that 00, then 000, then 1000 each won the pair count in turn; a merge table that respects frequency will swallow it whole. Nobody chose to break arithmetic. The cost only became visible once people started asking these models to add.

The same paragraph costs more in Spanish

I took one paragraph of 184 bytes, wrote it in English and in Spanish, and built a JSON payload of roughly the same size, 182 bytes. Then I encoded all three with the three tokenizers.

Spanish costs about 1.7× English under GPT-2 and still 1.2× under the newest encoding. JSON costs about 2.3× under all three: braces, quotes, and keys do not compress. The 26 quotes in that payload never appear as a token on their own; each one is welded to the brace, bracket, colon, comma, or space beside it, in tokens like ", ":, ",, {", "},. Twenty-six quotes, twenty-six tokens carrying one.

The original has an interactive widget where you switch the sample text and the encoding. Open it on cchinchilla.dev to use it.

Switch the sample and the encoding. Newer vocabularies mostly close the Spanish gap; the JSON one does not move.

Three things follow, and all three are billing or capacity problems, not modelling problems. A context window quoted in tokens holds less Spanish than English, and much less JSON. A per-token price is a per-language price. And a prompt template that wraps every request in structured metadata is paying the JSON rate on every call.

Nothing in the type system says so

In software, the parser is code and the model is data. You can change the parser, rebuild, and read the same data. Here the parser is coupled to the weights. The embedding matrix has one row per id, and the meaning of row 11,241 is " token" only because that's what GPT-2's merges said when the model was trained. Under cl100k_base the same row is Seg. Encode the same text with a different tokenizer and you get ids that index the same matrix and mean something else. Whether that is loud or silent is luck. The same English paragraph under o200k_base reaches id 99,665, which a 50,257-row matrix refuses. Go the other way, GPT-2 ids into a model with a larger vocabulary, and every id lands. Nothing throws. Even the refusal is softer than it looks: production models pad the embedding to a multiple of 64 or 128 for tensor-core alignment, so the matrix has more rows than the vocabulary has ids, and on a GPU an out-of-range index surfaces as an asynchronous assert at some unrelated line rather than where it happened.

This is what makes the tokenizer part of the artifact. In GPT-2 that same matrix is also the output head, so a tokenizer swap corrupts the lookup on the way in and the sampling distribution on the way out, from one array of weights. The model file and the tokenizer file are two halves of one thing, and the failure mode when they drift is silent: a full swap whose ids stay in range produces garbage that no check flags, and a partial drift (a renumbered special token, a missing BOS) produces fluent text conditioned on the wrong input.

What the ML engineer watches

Four things to check before any of this reaches production.

Tokenizer and model versioned together. Store the tokenizer's hash next to the weights and check it at load time. A mismatch that doesn't fail on load is the worst kind.

Cost per language and per domain. Measure it on your real traffic before you estimate a bill or promise a context length. The 8k window is 8k English tokens; for the JSON payload above it is closer to 3.5k of the same content.

Special tokens. tiktoken's Encoding.encode raises by default if the text contains something that looks like <|endoftext|>, precisely because a user could type it. That default is a security decision; check what your serving layer does with it, and where your chat template inserts its own.

Length in tokens, not characters. Every limit the model has is in tokens. Truncation, context budgeting, and rate limits all need the tokenizer in the request path, not a character count.

None of this is modelling work. It's the part of the model that behaves most like ordinary software, which is exactly why it ships without review: a function that turns strings into integers looks like preprocessing right up until the moment it isn't.

Part 1 of twelve. This part starts before the model exists: the tokenizer ships inside the artifact rather than in front of it, and nothing in the type system says so. A token id is an index; the next part reads what sits at that index, the embedding matrix, which is a lookup table and not much else.

Written with AI assistance. See AI policy.

DEV Community: Carlos Chinchilla Corbacho

From GPT-2 to Llama: four changes and the reason for each

Same block, four edits

Positions become a rotation

The norm stops centring

The feed-forward grows a gate

Keys and values get shared

What the ML engineer watches

GPT-2 is the transformer plus five decisions, and inference is a different program

Five decisions around the stack

Training is one call

Generation is a loop

Same weights, different bugs

Sampling is a policy on top

What the ML engineer watches

A transformer block is six tensors and a bus: read it like a pipeline

The block is four lines

The plus sign is the architecture

Two thirds of the weights

The cost is somewhere else

What the ML engineer watches

Attention is a learned weighted average, and the cost is in the square

A dictionary that matches a little

Why the square root

Heads are a view, the mask is a triangle

Reading LlamaAttention.forward

The cost is in the data, not the code

What the ML engineer watches

Shapes are the type system of ML, and nobody checks them for you

A shape is a signature

The rule is five lines

The bug that only happens at batch size sixteen

Padding is a shape decision

Memory is shape times bytes

What the ML engineer watches

An embedding is a lookup table, and everything else is how you fill it

A matrix and an index

Two ways to fill the table

Close is a dot product

There is no schema migration

What the ML engineer watches

A model doesn't read text: what a tokenizer decides for you

A lexer with a different goal

BPE is one loop

The vocabulary is the training corpus

The same paragraph costs more in Spanish

Nothing in the type system says so

What the ML engineer watches

Reading `LlamaAttention.forward`