<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Abdullah Almania</title>
    <description>The latest articles on DEV Community by Abdullah Almania (@abdullah_almania_be4b69d4).</description>
    <link>https://dev.to/abdullah_almania_be4b69d4</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4258815%2F5072866c-e32c-46be-ab70-eb86a1426728.png</url>
      <title>DEV Community: Abdullah Almania</title>
      <link>https://dev.to/abdullah_almania_be4b69d4</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/abdullah_almania_be4b69d4"/>
    <language>en</language>
    <item>
      <title>Arabic Dialects and LLM Accuracy: What DAR-MGSM Found</title>
      <dc:creator>Abdullah Almania</dc:creator>
      <pubDate>Sun, 22 Nov 2026 20:17:58 +0000</pubDate>
      <link>https://dev.to/abdullah_almania_be4b69d4/arabic-dialects-and-llm-accuracy-what-dar-mgsm-found-371h</link>
      <guid>https://dev.to/abdullah_almania_be4b69d4/arabic-dialects-and-llm-accuracy-what-dar-mgsm-found-371h</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://cluetrove.com/does-saudi-arabic-change-ai-math-answers-dar-mgsm-study/" rel="noopener noreferrer"&gt;ClueTrove&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If your AI product answers Arabic speakers, a good score in formal Arabic does not carry over to the way people actually write. In the DAR-MGSM study, 11 language models averaged 70.0% on grade-school math in Modern Standard Arabic and 61.9% on the same problems written in Saudi Arabic.&lt;/p&gt;

&lt;p&gt;Swapping in local names, cities and currencies costs a little more: the average across all six Arabic varieties slips from 65.5% to 63.4%. Big models barely notice. Small Arabic-focused models can lose more than 10 points.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is DAR-MGSM, and why should product teams care?
&lt;/h2&gt;

&lt;p&gt;DAR-MGSM is a math benchmark published on arXiv on 20 October 2026. Its authors describe it as the first manually curated benchmark for math reasoning in dialectal Arabic. It starts from 250 word problems that trace back to GSM8K, a well-known English set of grade-school questions, and the authors point out that the originals are US-centric.&lt;/p&gt;

&lt;p&gt;Professional translators rewrote all 250 problems in Modern Standard Arabic (MSA) and five dialects: Egyptian, Emirati, Jordanian, Moroccan and Saudi. That gives 1,500 problems in the “translated” set. A second, “localized” set then swaps four things for regional versions: names, locations, currencies and numerals. A farmer in Toulouse selling eggs for dollars becomes a farmer in Riyadh selling them for riyals, written with Arabic-Indic digits.&lt;/p&gt;

&lt;p&gt;The math stays the same in every version. So any change in the score comes from the wording and the local details. If you build a tutoring app for Gulf students, a support bot for Egyptian customers or a budgeting assistant for Moroccan users, that is exactly the gap you need to know about.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much accuracy do LLMs lose in Arabic dialects?
&lt;/h2&gt;

&lt;p&gt;Averaged over the 11 models, on the translated problems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Emirati:&lt;/strong&gt; 70.8%, the highest of the six&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MSA:&lt;/strong&gt; 70.0%&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Saudi:&lt;/strong&gt; 61.9%, which is 8.1 points below MSA&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Moroccan:&lt;/strong&gt; 55.3%, the lowest, 14.7 points below MSA&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So “Arabic support” on a model card covers a wide range. A model can look solid in formal Arabic and still miss far more answers when a user writes the way they speak at home.&lt;/p&gt;

&lt;h2&gt;
  
  
  Do local names and riyals trip up AI models?
&lt;/h2&gt;

&lt;p&gt;For large models, hardly. For small ones, yes. Here are the Saudi results for five of the 11 models, translated first and localized second:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Translated&lt;/th&gt;
&lt;th&gt;Localized&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.4&lt;/td&gt;
&lt;td&gt;85.6%&lt;/td&gt;
&lt;td&gt;86.4%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma4-26B&lt;/td&gt;
&lt;td&gt;78.4%&lt;/td&gt;
&lt;td&gt;80.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen-3.6&lt;/td&gt;
&lt;td&gt;78.0%&lt;/td&gt;
&lt;td&gt;76.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jais2-8B-Chat&lt;/td&gt;
&lt;td&gt;62.0%&lt;/td&gt;
&lt;td&gt;54.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nile-Chat-4B&lt;/td&gt;
&lt;td&gt;35.2%&lt;/td&gt;
&lt;td&gt;24.4%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The average of all 11 models on Saudi problems goes from 61.9% to 59.4%.&lt;/p&gt;

&lt;p&gt;The three large models move between 0.8 and 1.6 points, in either direction. The two smaller Arabic-focused models drop 7.2 points (Jais2-8B-Chat) and 10.8 points (Nile-Chat-4B). The authors report that for the large flagship models the gap between translated and localized is not statistically significant, while for small models such as Nile-Chat and Atlas-Chat it is. If you are weighing a compact Arabic model to save on hosting, for example one of the open models such as &lt;a href="https://cluetrove.com/jais-3-arabic-open-models-weights-benchmarks-hosting/" rel="noopener noreferrer"&gt;Jais 3&lt;/a&gt;, this is where to look first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can a stable score hide wrong answers?
&lt;/h2&gt;

&lt;p&gt;Yes, and this is the most useful finding for anyone shipping a product. The authors looked closely at the three Gemma 4 models. Across 3,582 problem pairs that could be localized, the models changed from right to wrong, or from wrong to right, on 316 of them. That is 8.8% of the pairs, while average accuracy moved by just 1.2 points.&lt;/p&gt;

&lt;p&gt;In other words, the dashboard number looks flat while the model is quietly failing different users. The smallest model, Gemma4-2B, flipped 12.9% of its answers; the largest, Gemma4-26B, flipped 5.6%. Larger models are steadier, but none were immune. The paper’s look inside the models found that the translated and localized version of one problem can send the model down different internal reasoning paths, even when the overall score barely changes.&lt;/p&gt;

&lt;h2&gt;
  
  
  How should you test an AI model for Arabic users?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Test in your users’ dialect.&lt;/strong&gt; A score in MSA tells you little about Saudi or Moroccan users. Write test prompts the way your customers write.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use local details.&lt;/strong&gt; Put in the names, cities, currency and digit style your users see every day, not the defaults from an English test set.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compare answers one by one.&lt;/strong&gt; Run the same questions in two versions and check which items changed, not only the total. Our guide to &lt;a href="https://cluetrove.com/how-to-run-a-one-afternoon-ai-model-evaluation/" rel="noopener noreferrer"&gt;a one-afternoon AI model evaluation&lt;/a&gt; shows a simple setup.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Give small models extra scrutiny.&lt;/strong&gt; The 10.8-point drop for Nile-Chat-4B is the kind of gap a quick demo will not reveal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look beyond math.&lt;/strong&gt; Dialect and script affect other tasks too, from &lt;a href="https://cluetrove.com/arabic-tokenization-jais-vs-tiktoken-small-test/" rel="noopener noreferrer"&gt;how Arabic text is tokenized&lt;/a&gt; to &lt;a href="https://cluetrove.com/arabic-ocr-study-datalab-azure-document-ai-error-rates/" rel="noopener noreferrer"&gt;how well OCR reads Arabic documents&lt;/a&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Keep the study’s scope in mind. It covers grade-school math only, six Arabic varieties and 11 models, each run once per variety and version. Reasoning (“thinking”) was switched off for GPT-5.4, Qwen-3.6 and the Gemma 4 models, so these scores are not comparable with results where it is on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Questions and answers
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Were the dialect problems machine translated?
&lt;/h3&gt;

&lt;p&gt;No. Professional translators wrote them, with a translator and a reviewer for each Arabic variety, and native speakers then reviewed the translations. Corrections ranged from 4.0% of problems for Saudi to 49.6% for MSA.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do all localized versions use Arabic-Indic digits?
&lt;/h3&gt;

&lt;p&gt;No. Every localized variety uses Arabic-Indic numerals except Moroccan, which keeps Western digits (0 to 9).&lt;/p&gt;

&lt;h3&gt;
  
  
  Which model scored best on Saudi problems?
&lt;/h3&gt;

&lt;p&gt;GPT-5.4, with 85.6% on translated and 86.4% on localized Saudi problems.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which dialect was hardest for the models?
&lt;/h3&gt;

&lt;p&gt;Moroccan, with a translated average of 55.3% across the 11 models.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://cluetrove.com/arabic-ocr-study-datalab-azure-document-ai-error-rates/" rel="noopener noreferrer"&gt;Arabic OCR Study: Datalab, Azure and Document AI Error Rates&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://cluetrove.com/three-arabic-tasks-claude-55-what-they-can-and-cannot-show/" rel="noopener noreferrer"&gt;Three Arabic tasks with Claude 5.5: what they show&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2610.28607" rel="noopener noreferrer"&gt;arXiv: Benchmarking Arabic Dialects for Math Problems (DAR-MGSM)&lt;/a&gt;, 20 October 2026&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>nlp</category>
      <category>machinelearning</category>
    </item>
  </channel>
</rss>
