Here's something most AI benchmarks get wrong: they assume everyone talks like a textbook.
I don't. My friends don't. Millions of people across Pakistan and India don't.
We talk like this:
"Yaar deadline is tomorrow, kaam jaldi finish karo."
Half Urdu, half English, all heart. We call it Roman Urdu — Urdu written in English letters — and we code-switch without even thinking about it. It's how we text our moms, argue with friends, and vent at 2 AM.
So I started wondering: when I ask an AI something in the language I actually speak... does it even understand me?
I couldn't find a benchmark that answered that. So I built one.
What I Actually Built
The Roman Urdu Code-Switch Benchmark — 35 test items across four tasks, all running on Kaggle Benchmarks:
1. Comprehension (15 questions)
I give the model a Roman Urdu sentence and three English options. Simple idea: does it know what "Mehngai ne to kamar tor di hai yaar" means? (It means inflation has broken our backs — something every Pakistani feels in their bones.)
2. Sentiment (10 questions)
Can it tell the difference between "Wah kya baat hai! Maza aa gaya" (pure joy) and "Yaar dil toot gaya" (heartbreak)? This sounds easy until you realize how much sentiment lives in cultural context, not just words.
3. Code-Switch Translation (5 questions)
The real test. Sentences like "Mujhe ek meeting attend karni hai at 3pm, phir gym jaunga" — genuinely mixed language, the way real people write. Can the model produce clean English from that?
4. Cultural Context (5 questions)
What is "load shedding"? What's a "chai dhaba"? When is Eid ul Fitr? If you've lived in Pakistan, these are obvious. If you haven't... well, that's the point. Language isn't just words — it's the world behind the words.
The Models I Put to the Test
I picked a spread — the biggest closed models and strong open ones — because I wanted to know: is this a "bigger model = better" situation, or does something else matter?
| Model | Type | Score |
|---|---|---|
| Gemini 2.5 Pro | Closed (Google) | 35/35 (100%) |
| Gemma 4 31B | Open (Google) | 35/35 (100%) |
| GPT-5.5 | Closed (OpenAI) | 31/35 (88.6%) |
| Claude Sonnet 5.5 | Closed (Anthropic) | 31/35 (88.6%) |
What the Numbers Told Me
The headline result stopped me in my tracks: an open 31B-parameter model matched Google's flagship and beat both OpenAI and Anthropic's best.
Gemma 4 31B — a model you can download and run yourself — scored a perfect 35/35 on understanding Roman Urdu. Meanwhile GPT-5.5 and Claude Sonnet 5.5, the models everyone assumes are the smartest in the room, each dropped 4 questions.
That flips the usual script. On English benchmarks, the giant closed models almost always win. But Roman Urdu isn't English. It's a low-resource, informal, code-switched way of speaking that lives in WhatsApp chats, not training-data textbooks. And here, the playing field looks very different.
What I think is happening: this isn't about raw intelligence. It's about exposure. Models that saw more multilingual, informal, real-world text during training handle Roman Urdu better — regardless of their size or price tag. The gap between 100% and 88.6% isn't a reasoning gap. It's a data gap.
The Part That Surprised Me
Honestly, I went in expecting the big models to crush this. What I found was more interesting than that.
The surprise isn't just that an open model won. It's what that implies: the next frontier of AI capability might not be "smarter models" but "models that have actually heard how real people talk." There are billions of people whose daily language looks nothing like the benchmarks we use to measure progress. Roman Urdu is just one of them — think of Hinglish, Spanglish, Arabizi, and every other hybrid language born in group chats around the world.
The models that win the future might not be the biggest. They might just be the ones that listened.
What I'd Do Differently Next Time
This is 35 questions. It's a start, not a finish. If I had more time:
- 100+ items with real dialectal variation — Punjabi-influenced Roman Urdu hits different from Karachi slang
- A generation task — understanding is one thing, but can a model reply naturally in Roman Urdu? That's the real test
- Reverse code-switching — English sentences with Urdu words dropped in
- Harder distractors — some of my multiple-choice options were too easy; the next version needs trickier wrong answers
- More open vs. closed showdowns — the Gemma result deserves a deeper investigation
Why I Think This Matters
Here's what I keep coming back to: benchmarks decide what gets better. If every benchmark is in clean, formal English, then that's what models optimize for. And that's fine — until you realize the next billion AI users won't talk like that.
They'll code-switch. They'll use slang. They'll mix three languages in one sentence and expect to be understood.
A benchmark like this is tiny. 35 questions won't change the industry. But it's a reminder: the way my people talk is worth measuring too.
Top comments (0)