A byte-level model has no tokeniser and no vocabulary. Its inputs are the 256 possible byte values, which removes an entire class of failures — and multiplies the sequence length by roughly four, which is the reason it is not how models are built.
What disappears
The vocabulary, and its embedding table. Concretely, for a model of dimension 4,096:
128,000-token vocabulary:
embedding matrix 128,000 * 4,096 = 524,288,000 parameters
output head another 524,288,000 if untied
-> over a billion parameters, on a 7B model, spent on the vocabulary
256-byte vocabulary:
embedding matrix 256 * 4,096 = 1,048,576
output head 1,048,576
Saving: about 1.05 billion parameters freed for actual layers.
On a 7B model that is 15 per cent of the budget moved from a lookup table into computation.
The character blindness. A model that receives strawberry as two or three opaque integers has no direct access to its letters, which is the mechanical reason for the counting and spelling failures that get treated as evidence of deeper deficiency. A byte-level model sees the letters. It still has to learn to count, but the information is present rather than destroyed at the input.
The language tax. Tokenisers are fitted to a corpus, so a language under-represented in that corpus fragments into far more tokens per sentence — and since billing and context are counted in tokens, its speakers pay more for the same meaning. At the byte level the cost is proportional to the UTF-8 encoding, which is a property of the writing system rather than of whose text happened to be in the tokeniser’s training set. It does not equalise things — UTF-8 already spends three bytes per character on many scripts — but it removes the arbitrary part.
- No out-of-vocabulary anything. Emoji, rare scripts, code with unusual identifiers, mixed-language text, binary-adjacent data: all just bytes.
- No tokeniser mismatch bugs. The class of failure where your token count disagrees with the provider’s does not exist when the unit is a byte.
- Typo robustness. A misspelling changes one byte rather than re-segmenting a word into a completely different token sequence, which is why byte-level models are consistently reported as more robust to noisy text.
The multilingual arithmetic
The language-tax claim deserves numbers, because byte-level models improve the situation without equalising it, and the difference between those two statements matters.
UTF-8 bytes per character, by script:
ASCII (English, most code) 1
Latin with diacritics, Greek, Cyrillic, 2
Hebrew, Arabic
Most CJK, Devanagari, Thai, Korean 3
Emoji, some historic scripts 4
So a byte-level model still charges a Thai or Chinese sentence roughly three times what it charges the same characters in English. That cost is a property of the encoding, and it is fixed, published and predictable.
Compare it with what a tokeniser does. A BPE vocabulary fitted to a corpus that is overwhelmingly English learns long merges for English words and few or none for an under-represented script, so that script’s text fragments toward one token per character — sometimes worse, when a single character is split across several byte tokens. The resulting multiplier is not a property of the writing system. It is a property of whose text happened to be in the tokeniser’s training data.
That is the honest framing of the improvement: byte-level modelling replaces an arbitrary, corpus-dependent, provider-specific tax with a fixed, encoding-dependent one. It does not make all languages cost the same. It makes the cost explicable, identical across providers, and impossible to make worse by choosing a different tokeniser.
What it costs, priced
English text: roughly 4 bytes per BPE token on average.
A 1,000-token document is about 4,000 bytes.
Sequential decode steps: 4x more
Quadratic attention cost: 16x more
KV cache for the same text: 4x larger
At the same context window measured in units:
a 128k-token window covers about 128,000 tokens of text
a 128k-BYTE window covers about 32,000 tokens of text
The four times on sequential steps is the one that hurts most, and it does not go away with a cleverer attention mechanism. Generating a fixed amount of text takes four times as many forward passes, and sequential depth is the thing you cannot parallelise.
There is a subtler cost too. A tokeniser is free compression, fitted to the data, that hands the model word-like units at the input. Remove it and the first several layers of the network have to spend capacity reassembling bytes into something word-shaped before any of the actual work begins. You did not delete the tokenisation; you moved it inside and started paying for it in parameters and depth.
Patching: putting the compression back inside
Every serious byte-level architecture is an answer to that cost, and they all have the same shape: a small cheap model near the bytes, a large expensive model over groups of them.
| Approach | Description |
|---|---|
| ByT5 (Google, 2021) | A byte-level version of T5 with no patching, made viable by making the encoder much deeper than the decoder. Slower than the token-level equivalent and notably more robust to noise. The honest baseline for what pure byte-level costs. |
| Charformer | Learns a soft subword segmentation as part of the model, scoring candidate spans and downsampling. Tokenisation becomes differentiable rather than a preprocessing step. |
| MegaByte (Meta, 2023) | Fixed-size patches — say 8 bytes each. A large global transformer runs over patch representations, a small local one predicts the bytes inside a patch. An 8-byte patch means the expensive model sees an eighth of the positions. |
| Byte Latent Transformer (Meta, 2024) | Patches of variable size, chosen by how predictable the next byte is. The interesting one, and the subject of the next section. |
The pattern is the same one as patch embedding in a vision transformer: the expensive stack should not run at the resolution of the raw signal.
Dynamic patches, and why entropy is the right signal
Fixed patches waste compute. The eight bytes of the cat are almost entirely predictable, and the eight bytes of a chemical name are not, yet both get one patch and the same compute.
The dynamic approach runs a small byte-level model first, measures the entropy of its next-byte prediction, and starts a new patch wherever that entropy spikes:
small model gives H(next byte | context) at each position
low entropy -> the continuation is obvious -> keep extending the patch
high entropy -> a decision point -> end the patch here
Effect: compute is allocated where the text is hard, and patch
boundaries land near real linguistic boundaries without anyone
specifying what a word is.
This is a better idea than a fixed tokeniser on its own terms. A BPE vocabulary is fitted once, offline, to one corpus, and then applied identically to every input forever. Entropy-based patching adapts per input, and it adapts to the same model that will do the work.
What you still lose
- Nothing about counting is automatic. Seeing individual letters makes counting learnable, not learned. The model still has to acquire the skill.
- Two models to train and serve. Patching architectures have a local and a global component, and the boundary logic sits on the critical path of every request.
- Patch boundaries are state. With dynamic patching, where a patch begins depends on preceding bytes, which complicates caching a shared prefix and makes batching sequences with different boundary patterns awkward.
- Everything downstream assumes tokens. Context limits, pricing, truncation logic, client-side token counting and every budgeting tool are denominated in tokens. A byte-level model that bills in bytes is a different unit for every existing integration.
What is and is not the tokeniser’s fault
Tokenisation has become the standard explanation for any model behaviour that looks stupid, and it is the correct explanation for rather fewer of them than it gets credit for. Separating the two is what decides how much removing it would actually buy.
Genuinely the tokeniser
- Character-level operations. Counting letters, spelling backwards, finding the third character, deciding whether two words rhyme. The model receives integers; the letters were discarded before the first layer. This is a data-availability failure at the input, not a reasoning failure.
- Digit grouping in arithmetic. If
1234is one token and12345splits as123plus45, the model has to learn place value across groupings that change with the number. The strongest evidence this is real is that the fix was applied at the tokeniser: several modern vocabularies deliberately split numbers into single digits or fixed three-digit groups so the segmentation is regular. - The cost multiplier on under-represented scripts, for the reasons in the section above.
Blamed on the tokeniser, but not
- Factual errors. Whether a claim is true has nothing to do with the unit the text was chopped into.
- Losing track across a long document. That is context handling and attention, and the same failure appears in models with entirely different tokenisers.
- Weak performance in a low-resource language. This is the important one. The tokeniser adds a cost multiplier and a small quality penalty, and the dominant cause is how little of that language was in the training data. Removing the tokeniser removes the multiplier; it does not add data, and a byte-level model trained on the same skewed corpus will be roughly as weak in that language for the same reason.
That last point is the honest brake on the whole idea. Tokeniser-free modelling fixes a real, bounded set of problems very cleanly. It is not the reason models are bad at the things models are bad at.
Honest status
No widely deployed frontier model is byte-level as of mid-2026. Published results at moderate scale are encouraging — patch-based byte models report matching token-level baselines at comparable training compute, with better robustness on noisy and multilingual input — and the open question is whether that holds at frontier scale, where a small constant-factor disadvantage compounds over an enormous training run.
Treat “tokenisation is going away” as a hypothesis with serious work behind it rather than a settled direction. If it turns out to be right, the visible consequences for anyone calling an API are concrete: the language tax shrinks, spelling and counting behaviour improves, and the pricing unit stops being a tokeniser artefact.
Until then the tokeniser difference is a real and measurable cost difference between models: the same prompt is a different number of tokens on each, and the gap is largest in exactly the languages that are worst represented. Multigrid reports the token counts and cost each provider billed per request, which is how to see that difference on your own traffic rather than by estimating it.
Top comments (0)