The latest articles on DEV Community by salam thabit (@salamthabet). https://dev.to/salamthabet https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4046797%2Ff06dfcdd-d292-4f74-8b6d-2c64a6d385d4.jpg https://dev.to/salamthabet en salam thabit Sat, 25 Jul 2026 11:53:30 +0000 https://dev.to/salamthabet/why-ai-understands-formal-arabic-but-gets-lost-in-dialects-29cm https://dev.to/salamthabet/why-ai-understands-formal-arabic-but-gets-lost-in-dialects-29cm <p>Ask an AI chatbot a question in formal Arabic (Fusha) and it answers beautifully. Ask it the same thing in Gazan, Egyptian, or Moroccan dialect — and it stumbles, mistranslates, or misses the point entirely.</p> <p>As someone building an Arabic AI newsletter, I run into this every day. Here's what's actually going on under the hood — and why it's a great window into how large language models really work.</p> <h2> 1. Models learn from data — and the data is lopsided </h2> <p>An LLM learns language by reading enormous amounts of text. For English, that's a huge chunk of the internet. For <strong>formal Arabic</strong>, there's a decent amount (news, books, Wikipedia). But for <strong>dialects</strong>? Very little is written down — dialects live in speech, voice notes, and casual chats, not in the formal text models train on.</p> <p>Less data for a language variety = weaker performance. This is the well-known <strong>"low-resource language"</strong> problem, and Arabic dialects are a textbook case.</p> <h2> 2. One language, many "languages" </h2> <p>Arabic isn't one thing. Fusha is the shared formal standard, but everyday speech splits into dialects that can differ more than some European languages do from each other. A model trained mostly on Fusha treats a dialect almost like a foreign language it barely saw.</p> <h2> 3. Tokenization makes it worse </h2> <p>LLMs don't read words — they break text into <strong>tokens</strong> (chunks). Tokenizers are optimized for the languages they saw most. Arabic script, with its rich morphology and prefixes/suffixes, often gets chopped inefficiently — and dialectal spellings (which aren't standardized) fragment even more. More tokens = more cost and more room for the model to lose meaning.</p> <h2> Why this matters beyond Arabic </h2> <p>This isn't just an Arabic problem. It's a preview of what happens to <strong>any</strong> under-represented language or community when AI is trained mostly on high-resource data: the technology works brilliantly for some and poorly for others.</p> <p>The encouraging part: <strong>open models</strong> are changing this. When powerful models are released openly, any local team can fine-tune them for their own language and dialects — instead of waiting for a big lab to care. That's real digital sovereignty.</p> <p>I write simple Arabic explainers about AI (tokens, embeddings, RAG, and more) over at <strong><a href="https://ai.salamthabet.com/" rel="noopener noreferrer">ai.salamthabet.com</a></strong> — making machine learning make sense, one term at a time. If the Arabic-AI intersection interests you, come say hi 👋</p> ai llm nlp