The problem with standard tokenizers

The problem with standard tokenizers
If you have ever tried running a standard NLP tokenizer over Roman Urdu text,
you know it falls apart fast. Words like "kyunnn", "kese ho", or "btao" break the
assumptions that most tokenizers are built on: consistent spelling, a stable
lexicon, clean word boundaries.
Roman Urdu is Urdu written in Latin script, and it has no spelling standard. The
same word gets spelled five different ways depending on who is typing. People
mix English and Urdu mid-sentence. They stretch letters for emphasis
("bohatttt"). They abbreviate aggressively. A BPE or WordPiece tokenizer trained
on English just shreds this input into meaningless fragments.
I ran into this while working on a compiler design and automata visualization
project. I needed clean tokens out of noisy, code-mixed Roman Urdu text, and I
wanted the process to be inspectable — not a black box. So I built the tokenizer
around the oldest tool in the formal languages toolbox: a deterministic finite
automaton.
Why a DFA
A DFA is a good fit here for two reasons. First, tokenization is fundamentally a
regular-language problem — you are scanning a character stream and
classifying substrings into token types. Regular expressions compile down to
DFAs anyway, so writing the automaton by hand just skips the middleman and
gives you full control over the transition table.
Second, a hand-written DFA is debuggable. When the tokenizer misclassifies
something, I can point at the exact state and the exact transition that went
wrong. Try doing that with a subword model.
The pipeline has three stages: normalization, DFA execution, and token
classification.
Normalization: folding the noise
Before the DFA ever sees the input, a normalization pass folds the most
common noise patterns:
•
•
•
Repeated letters get collapsed ("kyunnn" becomes "kyun"). This is the
single highest-impact rule for Roman Urdu social text.
Phonetic variants get mapped to a canonical form. The character-level
normalizer ( normalizePhoneticChar ) handles the common substitutions
people make when typing Urdu sounds in Latin script.
Short-forms ("btao", "kese") pass through to the DFA, which has accepting
states for the common abbreviated forms rather than trying to expand
them.
This stage is responsible for most of the cleanup. On my test inputs it removed
the large majority of surface noise before tokenization even started.
The DFA core
The core is a transition table: current state plus input character maps to next
state, with a dead state for anything unrecognized. Each step also advances a
virtual tape head, which feeds the visualizer. Here is the step function:
export function stepDFA(
currentState: DFAState,
char: string,
tapeHead: number
): DFATransitionResult {
const normalizedChar = normalizePhoneticChar(char);
const nextState =
TRANSITION_TABLE[currentState]?.[normalizedChar] ?? DFAState.DEAD;
return {
previousState: currentState,
currentState: nextState,
consumedChar: char,
tapePosition: tapeHead + 1,
isAccepting: ACCEPTING_STATES.has(nextState)
};
}
Nothing fancy. The interesting part is the transition table itself, which encodes
the grammar: which character sequences form valid tokens, where word
boundaries fall, and which states are accepting. Building that table by hand for
Roman Urdu's irregular spellings was the bulk of the four-week build.
The Turing tape visualizer
Because this started as an automata education project, I built a visual debugger
on top of the DFA: a Turing tape rendered on Canvas that steps through the
input character by character, showing the current state, the tape head position,
and which cells have been consumed. It renders at 60 FPS with no UI lag, which
mattered because the whole point was watching the machine think in real time.
There is also a contextual explanation layer (via the Gemini API) that takes the
produced tokens and explains their meaning in context — useful for the code-
mixed cases where a token's meaning depends on the surrounding English or
Urdu.
What I measured
On my test corpus, the pipeline hit 10K tokens per second parsing throughput,
the visualizer held 60 FPS during state transitions, grammar coverage across the
defined token set was complete, and the normalization stage reduced surface
noise by 92% measured against the raw input.
The honest caveat: these numbers are on my own test inputs, not a published
benchmark. Roman Urdu has no standard benchmark corpus, which is part of
why the problem is interesting and part of why you should be skeptical of
anyone claiming a single accuracy number for it.
What I would do differently
The hand-built transition table does not scale. Every new spelling variant means
editing the table. The next iteration would learn transition weights from a
corpus of real Roman Urdu text and keep the DFA structure only as the skeleton
— a hybrid of the formal approach and a statistical one.
But as a starting point, the DFA got me something no off-the-shelf tokenizer
could: clean, explainable tokens from genuinely messy input, with every
decision traceable to a state transition I can point at.
Full case study with code walkthrough: https://muhammadsaaddev.netlify.app/
work/automata-tokenizer

Comments

[Zhang Enquan]

Nice — "normalize before you tokenize" generalizes past Roman Urdu, and there's a number that makes the argument for you.

I measured the cost side of this: 10,000 samples from real dev.to posts through cl100k. Median 4.47 chars per token overall, but the spread by content type was the surprise: prose 4.58, code 4.16, structured 3.78.

Your stretched-letter case ("bohatttt") should land well below prose, because character repetition is exactly what BPE merges badly — more tokens per unit of meaning, so fewer chars per token. Which means the failure isn't just wrong tokens, it's a bill.

If you have a tagged sample sitting around, the number worth reporting is chars/token before vs after your normalization pass. That delta is the whole justification for stage one, and it's denominated in the thing readers actually pay for.