Update (Oct 2): Following reviewer feedback, the 0.95 vs 0.90 gap is not statistically significant (exact McNemar p = 0.25) — the honest reading is a four-way tie at n = 60.
DarijaBench: Do AI Models Actually Understand Moroccan Darija?
As a Moroccan student, I use AI assistants every day. They are brilliant in English and French — but I kept noticing something: ask them something in Darija (Moroccan Arabic dialect, spoken by 35+ million people), and the confident answers start wobbling. So I decided to stop guessing and start measuring. I built DarijaBench, a 60-item benchmark that tests whether today's frontier models truly understand Moroccan Darija — and ran it on four of them.
🔗 Benchmark: https://www.kaggle.com/benchmarks/soufianzaari/darijabench
What DarijaBench tests
Darija is a low-resource dialect: it is barely present in training data compared to English, French, or even Modern Standard Arabic. DarijaBench probes three practical skills, 20 items each:
Darija → French translation — can the model convert everyday Moroccan sentences into French?
Sentiment analysis — can it tell whether a Darija text is positive, negative, or neutral?
Moroccan cultural knowledge — can it answer questions asked in Darija about Morocco (the capital, couscous ingredients, the 2030 World Cup hosts...)?
Each item is graded automatically with keyword checks, and the final score is the average of the three categories. The whole thing was built with the Kaggle Benchmarks SDK, entirely on the free quota.
The results
Model
DarijaBench score
Gemini 3.7 Flash
0.95
Claude Sonnet 5
0.95
Claude Opus 4.7
0.95
GPT-5.6 Luna
0.90
What I learned
The top models have mostly cracked basic Darija. A three-way tie at 0.95 was not what I expected — I assumed Darija would still be a weak spot. Frontier labs are clearly doing better on dialectal Arabic than their reputation suggests.
Sentiment is the easiest skill. Gemini scored a perfect 20/20 on sentiment. Emotional tone ("هاد المطعم رائع!" vs "الخدمة خايبة") seems to transfer across dialects and languages with little friction.
Translation is the hardest. At 85%, Darija→French was the lowest category for Gemini. Darija is full of French loanwords used differently than in French, plus idioms with no literal equivalent — exactly where word-level understanding breaks down.
Small gaps need bigger samples. GPT-5.6 Luna's 0.90 vs 0.95 is ~3 more failures on 60 items — but a paired test shows this isn't statistically significant (p = 0.25). Detecting a 5-point gap reliably needs ~200–300 items. That's the main lesson for v2.
Honest caveats. Keyword-based grading is brittle — a correct answer phrased unexpectedly can fail. Sixty items is small. And the three-way tie tells me the benchmark is currently too easy to separate the best models. The next version needs harder items: idioms, heavy code-switching (Darija/French/Spanish mixes are everywhere in the north), and regional variants. Try it yourself The benchmark is public — run more models on it, or fork the task and make it harder. If you speak a low-resource dialect, consider building your own version: the Kaggle Benchmarks SDK makes it surprisingly painless, and measuring is always better than vibing.
Built for the DEV x Kaggle Benchmarking Challenge.
Top comments (3)
On point 4: 0.95 vs 0.90 on 60 items is 57 vs 54 correct, and since all four models answered the same items the fair comparison is paired. Even in the best case for the gap, where the three extra misses are items every other model got right, an exact McNemar test on 3 discordant items gives p = 0.25. So the honest reading is a four-way tie at this size, which fits your conclusion that the set is too easy.
Two cheap things before v2. Since the per-item results are in the run, you can check the guess that the misses cluster in cultural knowledge rather than infer it. And it's worth hand-grading the translation items once: if the keyword checker gets 2 or 3 of the 20 wrong, the grader's error is the same size as the gap between models. Harder idiom and code-switching items, as you plan, are the right fix; for a 5-point gap to show up in a paired test you need roughly 200-300 items, depending on how often two models disagree.
Thanks Arhan, this is exactly the kind of scrutiny the benchmark needs. You're right on the paired comparison... I reported the raw gap without testing it, and p = 0.25 settles it: four-way tie at n = 60, that's the honest headline. On your two suggestions: I'll pull the per-item results and check the cultural-knowledge clustering against the data instead of asserting it, and I'll hand-grade all 20 translation items to measure the keyword grader's error rate before trusting any gap smaller than it. Both going into the v2 writeup. Noted on sizing too... targeting 200–300 items with harder idioms and code-switching so a 5-point gap is actually detectable. Really appreciate the McNemar framing; it'll make v2's analysis section much stronger.
Nice to see another low-resource language in this challenge. I benchmarked Kazakh, and your v2 plan (heavy code-switching) is exactly where I saw the only real gap.
My setup: 72 scam/genuine-message scenarios, each written three times, in Russian, Kazakh, and shala (the Russian-Kazakh mix most people here actually text in). 14 models. What came out:
One design idea that might help with your sample-size problem: write the same item in each variant (Darija, MSA, a Darija/French mix). Then every comparison is paired within the item, and a language gap can't be confused with "this item was just harder." It doesn't fix n = 60, but each item gives you more information.
Darija with French and Spanish mixed in sounds like a much harder test than mine. Curious whether the 0.95 tie survives it.