BERT's tokenizer bottleneck was hiding before WordPiece

Normalization consumed 88% of a warm English encoding pass with bert-base-uncased. WordPiece, the algorithm that chooses tokens, took 8%. Simon Brandeis reports those measurements in his October 15 Hugging Face article, which follows the work from a CPU profile to a proposed implementation.

The surprising part is where the time went. A tokenizer sounds like a problem about finding token boundaries, yet this profile mostly measured the cost of preparing characters. For an embedding or RAG service, that makes tokenization worth measuring separately from the model. It doesn't establish that tokenization dominates any particular service.

The normalizer did the same expensive work for ordinary letters

BertNormalizer prepares text before WordPiece. It cleans text and inserts spaces around Chinese characters. It then decomposes Unicode characters, strips accents and lowercases the result. These steps help determine which token IDs the model receives. Changing the implementation therefore needs a check beyond throughput. The tokens must still match.

Brandeis found a costly check for nonspacing marks, which the normalizer uses when removing accents. It searched a sorted table of 1,567 characters. The pipeline did Unicode property lookups even for ASCII letters that couldn't need decomposition or accent removal. The profile exposed repeated work on the common case, rather than an expensive rare input. Consider the letter s. A general Unicode path can decode it, look up its properties and pass it through normalization. But its UTF-8 byte already tells the implementation that it's ASCII. That distinction gives the normalizer a cheaper path without changing its output.

Skip unchanged text, then cache the remaining decisions

The proposed branch uses vector instructions to inspect 16 bytes at once. In UTF-8, ASCII bytes have a clear high bit. Checking those bits lets the normalizer find a run that doesn't need Unicode normalization, instead of decoding and examining each character. BERT's other ASCII transformations still apply, including lowercasing and whitespace handling. The branch handles those with vector operations too.

That shortcut helps English and source code most in the reported corpora. Other scripts need a different shortcut. Brandeis uses the tendency of nearby characters to belong to the same script, with related characters occupying nearby code points. A page cache stores whether normalization could change a character or combine it with a neighbor. The first encounter fills answers for 256 code points; later characters from that page use a memory read instead of another binary search. Russian, Greek and Arabic gain from this cache despite containing much less ASCII. Separately, unchanged stretches avoid copying, and an unchanged input can return without an allocation.

These changes share a useful idea. The implementation first establishes that it can leave text alone, then avoids paying for a transformation that would do nothing.

Read the speedup with its baseline attached

Brandeis reports up to another 5.7 times faster BERT encoding for the proposed branch, bringing the total to about 50 times over tokenizers 0.23. Those are different comparisons. The extra gain compares the branch with main; the larger headline includes earlier improvements over the older release.

The reproduction appendix describes single-threaded runs on two AMD EPYC hosts and an Intel Xeon host. It alternates the implementations across rounds and compares output token hashes. Those hashes match across the tested corpora and implementations. Matching hashes support output consistency for those tests, while leaving other inputs to be checked.

The limits are instructive. Chinese barely benefits in the BERT benchmarks, and some tiny inputs become slower. ModernBERT's NFC path already skips ASCII on main, so its English result improves only 1.06 times. The same proposed change encounters different remaining work in each tokenizer. These are tokenizer results. They don't measure a complete RAG request, GPU inference or application latency. And Brandeis says the changes were still in a pull request when he published.

What this changes in a service investigation

An application profile needs to distinguish text preparation from model execution. Otherwise, a faster GPU can leave the original CPU bottleneck intact. The Hugging Face measurements offer a concrete reason to inspect normalization when tokenization appears expensive, particularly for BERT-style embedding pipelines.

A useful comparison would keep the tokenizer configuration and input corpus fixed, then measure both encoding time and complete request latency. It would also compare token IDs and include the service's actual languages and input lengths. That's a proposed validation plan, not an experiment reported here. The source's uneven results make those checks part of the engineering question, rather than a footnote to the headline.