It started with a simple question that came to me somewhere between a bottle and a nap: would my agents behave the same way regardless of what language the request came in? Would they use their tools consistently when someone asks in French, Chinese, or Swahili?
I'd been building a student support chat where several tools were available to the agent to give users a better experience. That agent had to work across more than 20 very different languages, including Basque, a language spoken in the Basque Country that has almost no semantic relationship with the Romance languages around it.
AI Localization
Traditional localization is basically about translation: making a resource available in multiple languages to expand its reach. In the agent era, the concept needs to stretch further. Translation is a solved problem. The harder question is whether the agent's behavior stays stable when the input language changes.
A concrete example. Say you have an agent specialized in hotel reservations, with three tools available:
- Check availability
- Manage reservation
- Send confirmation email
A user wants to move their reservation a week out and writes: "Something came up and I need to cancel or push my booking by a week."
The correct flow would be:
- Check availability for the following week
- Available → confirm cancellation of the old booking and create the new one
- Confirms → make the new booking, cancel the old one, send confirmation email
- Declines → ask what they'd like to do instead
- Not available → inform the user and ask if they want to cancel
- Yes → cancel the reservation
- No → ask what they'd like to do instead
- Available → confirm cancellation of the old booking and create the new one
The order at the confirmation step matters: if you cancel before creating the new booking, someone else can grab that slot and the user ends up with nothing. On top of that, each tool needs specific arguments, pulled from the conversation or from the previous tool's response.
The Experiment
I ran a simple benchmark to see if my intuition was right. Six common scenarios:
- E-commerce: cancel an order
- E-commerce: track an order
- Scheduling: make a reservation
- Scheduling: change a reservation
- Support: cancel a subscription
- Support: dispute a duplicate charge
Each scenario was triggered in a different language: English, French, Arabic, Chinese, Russian, Indonesian, Vietnamese, Swahili, Welsh, Basque, Mongolian, and Yoruba, twelve in total.
To make the results more reliable, the same test ran across three different models:
- Claude Haiku 4.5
- GPT-4o mini
- DeepSeek deepseek-chat
And each model ran through every scenario up to ten times to get more representative numbers.
The Results
The results split clearly by model. GPT-4o mini was the most reliable, hitting 100% on three of the six scenarios, though it had an interesting blind spot: it consistently called book_new_appointment instead of check_availability across every locale, a failure that had nothing to do with language and everything to do with how the flow was defined. Claude Haiku showed the most variation across languages, dropping to 0% in Yoruba for subscription cancellations and struggling with Welsh, Mongolian, and Basque across multiple scenarios. DeepSeek landed in the middle, with serious failures in Chinese, Basque, and Swahili, and one notable quirk: English itself failed 2 out of 10 times in the e-commerce cancellation scenario.
The rough shape looked like this:
| Pattern | What showed up |
|---|---|
| Strongest model | GPT-4o mini, with 100% pass rate in three of six scenarios |
| Model-level bug | GPT-4o mini repeatedly called book_new_appointment before check_availability
|
| Most variable model | Claude Haiku, especially in Yoruba, Welsh, Mongolian, and Basque |
| Repeated weak locales | Basque across all models; Swahili, Yoruba, Mongolian, and Welsh in at least two |
| Hardest failure to catch |
no_tool_call, because the model still returns a normal-looking text response |
Three patterns showed up across all models. Basque failed in multiple scenarios in every model tested. Swahili, Yoruba, Mongolian, and Welsh repeated as weak spots in at least two. The most common failure mode in Haiku and DeepSeek was no_tool_call: the model responds with text instead of invoking anything, which is harder to catch in production than a wrong argument because it looks like a normal response.
The most important finding is not which languages failed. It's that English-only testing would have missed all of this.
Why It Matters
If you're shipping an agent that handles support tickets, processes orders, or books appointments, you almost certainly tested it in English. That's the language your team works in, the one your evals are written in, the one your prompts were tuned against.
But not all your users write in English. And when they don't, the agent can keep responding, keep sounding helpful, keep returning 200s, while actually calling the wrong tool, calling nothing, or making up an argument. None of that shows up in your error logs.
Translation is the easy part. The harder problem is behavioral: the same user intent, expressed in Swahili, Basque, or Mongolian, can produce a completely different action from the same agent. Not always. Not predictably. Just enough to matter in production.
That's what this experiment was built to surface, and it's the gap LangDrift is designed to close: not by replacing existing evaluations, but by adding one more axis. Does this scenario still pass when the input language changes? It exits non-zero on failure, so it fits in CI. You can set a --min-pass-rate threshold or mark a known weak locale with --allow-fail so it reports without breaking the build.
Not a Language Ranking
This experiment doesn't produce a ranked list of languages your agent will struggle with. The failures here are specific to this agent architecture, these system prompts, this set of tools, and these input phrasings. A different agent, a different model, or a different domain will produce different failure patterns.
What it does show is that the failures are real, repeatable, and invisible in English-only testing. Basque failed across every model tested here. Swahili and Yoruba came up in two. But that doesn't mean Basque is universally harder, or that French is safe. It means these specific scenarios, with these specific inputs, failed in these specific locales.
The right takeaway isn't "avoid these languages." It's that you don't know where your agent breaks until you test it in the languages your users actually write in.
The student support chat that started this ran across more than twenty languages in production. I would not have built this experiment before paternity leave — not because the work was too complex, but because I never had a reason to stay focused on one narrow question long enough to build around it. A single nap is not much time, but it is enough to close a gap that had been open for a long time.
Try It Yourself
LangDrift includes a simulated agent you can run locally without an API key. It's a deterministic server that returns predictable tool calls based on the input, so you can run through the full evaluation flow, read the output, and understand how scenarios are structured without spending any tokens.
A scenario is just YAML. The important part is that the same intent can be exercised across locales while the expected tool behavior stays stable:
id: support_cancel_subscription
agent: support
locales:
en:
input: "Please cancel my subscription. I'm not using it anymore."
expect:
toolCall:
name: cancel_subscription
noToolCall:
name: escalate_to_human
fr:
input: "Veuillez annuler mon abonnement. Je ne l'utilise plus."
expect:
toolCall:
name: cancel_subscription
noToolCall:
name: escalate_to_human
sw:
input: "Tafadhali futa usajili wangu. Sistumii tena."
expect:
toolCall:
name: cancel_subscription
noToolCall:
name: escalate_to_human
eu:
input: "Mesedez, baja eman iezadazu. Ez dut gehiago erabiltzen."
expect:
toolCall:
name: cancel_subscription
noToolCall:
name: escalate_to_human
pnpm fake-agent
node ./src/cli.ts run ./examples/scenarios/support-routing.yaml --target http://127.0.0.1:3011/api/agent
It won't show you language drift, but it'll show you exactly what LangDrift does and how to connect it to a real agent when you're ready.
Built During a Nap
I needed more scenarios to get cleaner benchmark data. We agreed on the list together first, then I handed the implementation of each one off to the agent. When I came back, the scenarios worked but some were too similar to each other: overlapping tool calls, nearly identical flows across different domains. I went through them one by one and trimmed the redundancy so each scenario was actually testing something distinct.
Top comments (0)