Ask an AI chatbot a question in formal Arabic (Fusha) and it answers beautifully. Ask it the same thing in Gazan, Egyptian, or Moroccan dialect — and it stumbles, mistranslates, or misses the point entirely.
As someone building an Arabic AI newsletter, I run into this every day. Here's what's actually going on under the hood — and why it's a great window into how large language models really work.
1. Models learn from data — and the data is lopsided
An LLM learns language by reading enormous amounts of text. For English, that's a huge chunk of the internet. For formal Arabic, there's a decent amount (news, books, Wikipedia). But for dialects? Very little is written down — dialects live in speech, voice notes, and casual chats, not in the formal text models train on.
Less data for a language variety = weaker performance. This is the well-known "low-resource language" problem, and Arabic dialects are a textbook case.
2. One language, many "languages"
Arabic isn't one thing. Fusha is the shared formal standard, but everyday speech splits into dialects that can differ more than some European languages do from each other. A model trained mostly on Fusha treats a dialect almost like a foreign language it barely saw.
3. Tokenization makes it worse
LLMs don't read words — they break text into tokens (chunks). Tokenizers are optimized for the languages they saw most. Arabic script, with its rich morphology and prefixes/suffixes, often gets chopped inefficiently — and dialectal spellings (which aren't standardized) fragment even more. More tokens = more cost and more room for the model to lose meaning.
Why this matters beyond Arabic
This isn't just an Arabic problem. It's a preview of what happens to any under-represented language or community when AI is trained mostly on high-resource data: the technology works brilliantly for some and poorly for others.
The encouraging part: open models are changing this. When powerful models are released openly, any local team can fine-tune them for their own language and dialects — instead of waiting for a big lab to care. That's real digital sovereignty.
I write simple Arabic explainers about AI (tokens, embeddings, RAG, and more) over at ai.salamthabet.com — making machine learning make sense, one term at a time. If the Arabic-AI intersection interests you, come say hi 👋
Top comments (0)