Same Model, 4 Tokenizers. 30 Percentage Points of Difference.

Same model, same data, 4 different ways to chop text into tokens. The accuracy spread was 30 percentage points.

Setup

Task: AG News classification (World, Sports, Business, Sci/Tech)
Model: Embedding (64d) + Average Pooling + 2-layer FC (128 hidden). Identical for all tokenizers.
Data: 15K train, 3K test
Budget: 15 epochs, Adam 1e-3

Tokenizers tested:

1. Character - each character is a token (a, b, c...)
2. Word - whitespace/punctuation split (the, stock, market...)
3. BPE - learned subword merges (mark, ##et, ##ing...)
4. Character 3-gram - overlapping 3-char windows (the, he_, e_s...)

Results

Tokenizer	Vocab Size	Avg Seq Length	Accuracy	Train Time
Word	10,000	39 tokens	86.67%	9.2s
Char 3-gram	10,000	227 tokens	84.83%	19.6s
BPE	2,040	102 tokens	78.17%	12.5s
Character	83	235 tokens	56.67%	22.0s

30 percentage point spread from the same model on the same data. The only difference is how the text was split.

Why Word Wins

Each word token carries a complete semantic unit. "stock" means something. "s", "t", "o", "c", "k" individually don't. With average pooling over the sequence, more meaning per token means more signal in the averaged representation.

Word tokenization also produces the shortest sequences (39 tokens average vs 235 for characters). Shorter sequences mean less noise in the average pool and faster training.

The downside of words: large vocabulary (10K), can't handle typos or unseen words. If the test set contains "cryptocurrency" and training only had "crypto", word-level misses it. Character n-grams would catch the overlap.

Why Characters Failed

56.67% accuracy (barely above the 25% random baseline for 4 classes). Each character carries almost zero semantic information. The model needs to compose meaning from sequences of characters, but average pooling destroys the ordering. Characters need attention or recurrence to work — you need the model to understand that "s-t-o-c-k" in sequence means something different than "k-c-o-t-s".

Why BPE Underperformed

This one surprised me. BPE is the tokenizer behind GPT, Claude, and every major LLM. But my implementation only learned 2,040 merges (limited for CPU speed). Production BPE uses 32K-100K merges. With too few merges, BPE is stuck between character-level and word-level without the benefits of either.

The tokenizer overhead was also brutal: 515 seconds for fit+encode vs 0.9 seconds for word-level. BPE merge learning is O(vocab * corpus_size) per merge step.

Why Char N-grams Are Underrated

84.83% accuracy, only 1.8% behind word-level. Character 3-grams capture subword patterns: prefixes ("pre", "un_"), suffixes ("ing", "tion"), and word fragments that generalize across related words. "trading", "traded", "trader" all share "trad" as a trigram.

N-grams also handle typos, neologisms, and code-mixed text that break word-level tokenizers. For noisy real-world text (social media, logs, user input), char n-grams may outperform words.

Per-Class Results

Tokenizer	World	Sports	Business	Sci/Tech
Word	86.8%	93.5%	82.9%	83.5%
Char N-gram	85.7%	92.6%	80.1%	81.1%
BPE	77.7%	88.6%	72.4%	74.2%
Character	63.0%	61.2%	50.1%	51.5%

Sports is easiest to classify across all tokenizers (distinctive vocabulary: "goal", "championship", "quarterback"). Business and Sci/Tech are hardest (overlapping vocabulary: "market", "technology", "growth").

The Takeaway

Tokenization is not a preprocessing detail. It is an architectural choice that affects accuracy more than most hyperparameters. Switching from character to word tokenization improved accuracy by 30 percentage points with zero changes to the model.

For text classification with simple models: use word-level or char n-grams. Save BPE for transformers that have the capacity to learn from subword structure.

Code

python3 tokenizer_experiment.py

github.com/turingrtss/vulndetect

Comments

[Raknaos]

30 points from tokenization alone on a fixed model is a good reminder that preprocessing is the hidden hyperparameter. I have seen a similar cliff on noisy production text (typos, code mixed with prose): the word-level tokenizer silently collapsed because unknown tokens became a single bucket, while char n-grams kept degrading gracefully instead of falling off.

Did you run multiple seeds per tokenizer? With 64d embeddings and 15 epochs I would expect variance big enough to matter — it would be interesting to know whether the char-3gram vs BPE gap survives a couple of reruns.

[turingrtss]

good question and honestly no, we didn't. only ran each tokenizer once —
there's a data-subsampling seed (42) but no torch seed and no repeats, so
right now the 30pp gap is a single point estimate per tokenizer.

your production example is a good sanity check on the mechanism too —
word-level collapsing hard on OOV while char n-grams degrade gracefully is
exactly what we'd expect if the gap is really about coverage/robustness and
not just an artifact of one lucky/unlucky init.

going to rerun with 5 seeds per tokenizer and report mean ± std instead of
a single number. if the char-3gram vs BPE gap holds up under that i'll
reply here with the numbers, if it collapses into the noise i'll say that
too.