Why Arabizi Confuses AI Models

#ai #llm #machinelearning #programming

Arabizi is Arabic written in Latin letters, with digits standing in for the consonants Latin script has no letter for. Models handle the letters tolerably. The digits are what breaks, and they break at the tokenizer, before any model has seen the sentence.

The convention, and why it exists

Arabic has consonants with no Latin equivalent — pharyngeals and emphatics that English simply lacks. When Arabic speakers write on Latin keyboards, they substitute digits whose shape resembles the Arabic letter. The mapping is remarkably consistent across the region for the core set:

2  →  ء  (hamza, glottal stop)
3  →  ع  (ʿayn)          — the digit 3 mirrors the letter's shape
5  →  خ  (khāʾ)          — also written 7' in some regions
6  →  ط  (ṭāʾ)
7  →  ح  (ḥāʾ)
8  →  غ  (ghayn)         — also written 3' or 4 depending on region
9  →  ص  (ṣād)           — also written 9' for ض (ḍād)

"3youni"     عيوني      "my eyes"
"7abibi"     حبيبي      "my darling"
"sa7"        صح         "correct / right"
"2ana"       أنا        "I"
"mar7aba"    مرحبا      "hello"

This is not an internet novelty of a single decade. It predates smartphones, it survived the arrival of Arabic keyboards, and it remains ordinary in messaging across the Levant, Egypt and the Gulf — often mixed with English or French in the same message, which puts it in code-switching territory as well as transliteration territory.

What the tokenizer does with it

Byte-pair encoding merges frequent character sequences into single tokens. The merges it learned come from its training corpus, and that corpus has effectively no 7abibi in it. What it does contain is a great many numbers, so it has learned merges that treat digits as a separate class from letters. The result is a split at every letter-digit boundary:

"7abibi, keefak? kel shi tamem el7amdulillah"

 tokenizes roughly as:

 "7" "ab" "ibi" "," " ke" "ef" "ak" "?" " kel" " shi"
 " tam" "em" " el" "7" "am" "du" "lil" "lah"

 versus the same greeting in Arabic script:

 "حبيبي" "،" " كيفك" "؟" " كل" " شي" " تمام" " الحمد" " لله"

Two consequences follow directly. The first is cost and context: the Arabizi form takes noticeably more tokens than the Arabic-script form of the same sentence, on top of the penalty Arabic script already carries relative to English — see the token cost of Arabic. The second matters more: el7amdulillah is a single lexical item, one of the most frequent phrases in Arabic, and it has been cut into six fragments with a bare digit in the middle. The model is not reading a word it knows badly. It is reading debris.

The digit fragments do further harm because a bare 7 token carries strong numeric associations from every invoice, date and measurement in the training data. The model is being handed a numeral in the middle of a greeting and has to overcome that prior from context alone.

That numeric prior is not merely a quality problem; it is an extraction bug waiting to happen. Any pipeline that pulls numbers out of a message — order quantities, prices, dates, reference codes — will harvest the letter-digits along with the real ones. “7abibi 3ndi order raqam 4471” contains one number and three digits, and a regex for digit runs returns four candidates. A model asked for a JSON field of numbers mentioned will frequently return the 7 and the 3, because they are digits in the input and the instruction said digits.

The generation direction is worse and less discussed. A model asked to reply in Arabizi has almost no consistent training signal for the convention, so it mixes regional variants within a single reply — spelling ghayn as 8 in one word and 3’ in the next — and drifts into plain unaccented romanisation partway through, dropping the digits entirely. There is no orthographic standard for it to converge on, so there is nothing to be consistent with.

Language detection returns the wrong answer confidently

Run Arabizi through a document-level language identifier and it will not return ar. Detectors trained on web text learn Arabic from Arabic script, so the script feature dominates; with no Arabic characters present, the candidates are whatever Latin-script languages have similar letter statistics. Reports of Arabizi being labelled as Turkish, Malay, Somali, Indonesian or simply English are the norm rather than the exception, and — this is the operationally dangerous part — the confidence is not low. The detector is not uncertain; it is answering a different question than the one you asked.

The same thing happens to romanised Hindi, Greeklish and romanised Persian, for the same structural reason. It is worth treating as one class of bug rather than four, and it is why script detection and language detection have to be separate steps.

The ambiguity that has no clean fix

Suppose you decide to convert Arabizi to Arabic script before processing. The mapping is not a function, and three separate ambiguities stack:

Digits are genuinely ambiguous with digits. “3andi 3 ktob” means “I have 3 books”. The first 3 is the letter ʿayn; the second is the number three. Only context separates them, and a rule-based converter has none.
Short vowels are unwritten in Arabic and written in Arabizi. Going back to Arabic script means deleting information the writer supplied. Going the other way means inventing it. Neither direction is lossless, which is the general abjad problem covered in abjad vowel ambiguity.
Arabizi encodes dialect; Arabic script conventionally does not. keefak is Levantine, ezayak is Egyptian, and both mean “how are you”. Normalising to Modern Standard Arabic erases the dialect, which is frequently the thing a support system most needs to know.

How to handle it

Detect it explicitly rather than inferring it. The signal is distinctive and cheap: Latin letters with digits inside word boundaries, specifically from the set 2, 3, 5, 6, 7, 8, 9. A regex for a digit flanked by letters catches nearly all of it and almost never fires on English, where digits sit at word edges or stand alone.
Tag it, do not silently convert it. Store a field saying this record is Arabizi. Converting on ingest destroys the original and every ambiguity above becomes an unrecoverable error in your store.
Ask the model in the prompt. Instruction-following models handle Arabizi far better when told what it is. A system prompt that says the input may be Arabic transliterated into Latin script using digits for ʿayn, ḥāʾ and hamza gives the model the key it cannot infer from a fragmented token stream.
Keep both forms if you must convert. Index the converted Arabic for retrieval, generate replies from the original, and never let the converted form be the only copy.
Reply in the script the user used. A user writing Arabizi on a Latin keyboard may not have an Arabic keyboard available. Answering in Arabic script can be unreadable to them in practice even though they read Arabic fluently.

Digit conventions vary by region and by generation — 8 for ghayn in some places, 3’ in others; 9 for ṣād against 9’ for ḍād. Any mapping table is a regional approximation, and a table built on one country’s messaging data will mis-convert another’s.