The cross-entropy in your training log is an average over a unit the tokeniser invented:
loss = (1/T) * sum_t -log p(token_t | context)
The sum is a property of the model and the data. The 1/T is a property of the tokeniser, because T is how many pieces it cut the data into. Two tokenisers divide the same quantity by different numbers.
The data here is an order-3 Markov source, so the floor is an exact eigenvector computation — 1.4500 bits per byte — and the models are real: a neural LM with a hand-written forward and backward pass, gradients checked to 9.0e-11 against central differences on an independently written second forward pass. https://dev48.infy.uk/dl/day71-tokenisation-and-the-loss-you-report.html
Six tokenisers, one corpus, one architecture
| tokeniser | train tokens | loss/token | perplexity | bits per byte |
|---|---|---|---|---|
| bytes | 15,000 | 1.4138 | 4.11 | 2.0397 |
| BPE, 8 merges | 9,645 | 1.9297 | 6.89 | 1.7581 |
| BPE, 24 merges | 7,253 | 2.5851 | 13.26 | 1.7482 |
| BPE, 48 merges | 5,983 | 3.2011 | 24.56 | 1.8028 |
| blocks of 2 | 7,500 | 2.4901 | 12.06 | 1.7962 |
| top 4-grams | 10,608 | 1.8925 | 6.64 | 1.8948 |
The byte tokeniser wins the column everyone quotes and loses the one that means something. 24 merges post a perplexity three times worse and the best score on the page, on 51.6% fewer training tokens. 11 of 15 pairwise comparisons invert between the two readings.
Scored as a ranker over 144 trained models against the exact excess bits per byte, held-out loss per token lands at AUC 0.369 — below the 0.500 a coin gives you, so you would do better preferring the model with the higher loss. Tokens per byte, an integer needing no model, no training and no forward pass, does 0.718.
The control is what makes it a finding rather than a complaint
Hold the tokeniser fixed and vary width, context, budget and seed, and the same number ranks the same models perfectly: AUC 1.000 over 1,656 pairs, in all six tokenisers separately. The loss is fine; the comparison is not.
Both obvious objections die to controls. Capacity: widen the byte model to 5,333 parameters, 3.9× the winner's 1,373, and it gets worse, 2.0397 → 2.1171 — what a coarse tokeniser bought was bytes of context, and width does not buy those. Context: at C = 3 the byte tokeniser covers the source's order, wins outright at 1.6734, and the inversions collapse from 11 of 15 to 4.
What the measurement contradicted
Mid-build the verifier caught a real bug in my BPE: merges indexed on raw symbol values instead of vocabulary ids, because index 0 is a pad and symbol a lives at index 1. Nothing complained. The vocabulary was still valid byte strings, greedy encoding still round-tripped the corpus perfectly, an independently written encoder still agreed token for token, and the whole page rendered.
The assertion that caught it is the one that does not share the assumption: reassemble the merged sequence and compare it against the corpus it came from. 24 merges had been buying 1.500 bytes per token instead of 2.133. Every number moved when it was fixed, and the winner changed from the largest vocabulary to the middle one.
One caveat as well: enumerating every tokenisation of all 15,625 six-symbol strings (the sum comes to 1.0000000000000, which it would not if the pad were an output class) puts 48.90% of a BPE model's probability on sequences its own encoder can never emit, worth 0.7144 bits per byte — and correcting for it flips 6 of 15 pairs.
56 in-page assertions, all passing. One file, inline CSS, no external asset of any kind.
Part of a from-scratch series — one idea a day, vanilla JS, dependency-free engine: https://dev48.infy.uk/deeplearningfromzero.php
Top comments (0)