My first language is Russian. When I set a task for a coding agent, I write it in English, because I assumed the agent would do better that way. Last week I checked that assumption, and a second one behind it: if the task is in my own language, does a guide help the agent more, because it will not go and look up the English answer on its own?
This is the second round of the same experiment. In the first one I measured whether a guide helps a coding agent at all: the same stories through nine agents, with the guide and without, and the answer was that a guide carries something only when it says what the docs do not. This round keeps that setup and changes one thing: the language of the task.
Same six tasks in English and in Russian, with a guide and without, seven agent setups, five runs per cell, 1050 runs. The method is at the end of the article. The results are here.
The short answer
- The agent solves the task about as well in Russian. Without a guide, 63% of the English-task runs passed and 56% of the Russian ones. The interval covers zero.
- A guide is worth the same in both languages. It lifted the pass rate by 31 points under the English task and 36 under the Russian one.
- Translating the guide changes nothing. Under a Russian task the English original did as well as the Russian translation.
- One agent stopped searching the web. Codex searched in a third of the English-task runs and in none of the Russian ones. On the one task whose answer lives only on a forum, that took it from 5 of 5 to 0 of 5, and the guide took it back to 5 of 5.
If you work in a language other than English, the last point is the one to take home: the agent may be quietly skipping a step it takes for English prompts, and a guide with the missing fact closes the gap completely.
The guide is worth the same in both languages
The guides are six stories from worklore: four that carry something the agent does not have on its own, and two controls where the fact is in the docs or in the log. Each row is 35 runs, seven setups times five repetitions.
| Story | English task | + story | Russian task | + English story | + Russian story |
|---|---|---|---|---|---|
| Heart-rate corridor for a Garmin watch (fact on a forum, not in the docs) | 25/35 | 35/35 | 15/35 | 34/35 | 35/35 |
| Nightly job silent for two reasons (one bug the agent cannot fix untold) | 8/35 | 32/35 | 10/35 | 31/35 | 32/35 |
| Three OAuth provider quirks (fact on a forum) | 16/35 | 30/35 | 14/35 | 30/35 | 30/35 |
| Analytics event on every keystroke (trap where the natural code goes) | 18/35 | 32/35 | 15/35 | 31/35 | 33/35 |
| Two cache markers for a memory write (control: fact in the docs) | 32/35 | 34/35 | 30/35 | 33/35 | 33/35 |
| DynamoDB crash on the second write (control: the log says what is wrong) | 33/35 | 34/35 | 34/35 | 34/35 | 35/35 |
| all six | 132/210 | 197/210 | 118/210 | 193/210 | 198/210 |
On the four stories that carry something, the pass rate went from 48% to 92% under the English task and from 39% to 90% under the Russian one. The gap between 48 and 39 is Codex: without its two setups the no-guide rate is 46% in English and 45% in Russian; the two Codex setups alone go from 52% to 22%, and 12 of the 13 passes lost to the Russian task are theirs. The story gain is 31 points in English and 36 in Russian over all six stories; the difference of those two gains is 5 points with a 95% interval from minus 6 to plus 16. A guide does not help more in Russian. It also does not help less. The gap it closes is the same gap.
The controls behaved like controls: 30 to 35 of 35 in every arm, both languages. A guide for something the agent already knows is worth nothing in any language, which is the same result as the previous round.
Translating the guide changes nothing
The last two columns of the table are the same story twice, once in English and once in Russian, both under a Russian task. The translation scored 2 points above the original, interval minus 2 to plus 7. Every setup except GPT-OSS was at 29 or 30 of 30 with the story in either language.
The reason is visible in the logs: the agent matches on identifiers, paths and error strings, and those stayed in English in the translation because that is what the code contains. If you were going to translate your team's guides for the agent's sake, don't. Translate them for the people.
The task language barely matters
Without a guide, a Russian task cost 7 points pooled over everything, interval minus 16 to plus 3. Per setup, 30 runs each:
| Setup | English task | Russian task |
|---|---|---|
| Claude Opus 5.5 | 27/30 | 27/30 |
| Claude Sonnet 5.5 | 22/30 | 23/30 |
| Claude Haiku 5.5 | 19/30 | 17/30 |
| Codex GPT-6-Sol medium | 21/30 | 17/30 |
| Codex GPT-6-Sol low | 20/30 | 12/30 |
| Gemini 3.8 Flash | 17/30 | 17/30 |
| GPT-OSS 120B | 6/30 | 5/30 |
Five of seven setups are flat. The two Codex setups dropped, and they are the story of this article.
The answers came back in Russian from Claude, Codex and Gemini, with the code identifiers left in English, which is what I would want from a colleague. GPT-OSS answered in English whatever language I asked in.
The one thing that moved: who searches the web
I had guessed that a Russian task might stop the agent from looking up the English answer. It did, for one agent, and that agent was the one whose passes depended on looking it up.
Without a guide, Codex at low reasoning searched the web in 10 of 30 English-task runs and in 0 of 30 Russian ones. Codex at medium: 9 of 30 and 0 of 30. Claude Haiku searched in 8 and 6, Gemini Flash in 13 and 10, so the other setups kept searching in Russian. Only Codex stopped.
Look at the first row of the story table again. The heart-rate corridor story is the one place where the Russian task really cost something, 25 of 35 down to 15 of 35, and the whole language cost of the round lives in that row. Under the English task Codex-low ran the same search every time, site:forum.intervals.icu workout_doc hr units %lthr …, found the shape and passed 5 of 5. Under the Russian task it never opened the web and designed its own output format instead: once it dropped the structured target, twice it switched to absolute BPM with a value and a range, once it kept the old key and added the new ones next to it. All four designs are sensible. All four are wrong for the watch. 0 of 5. Codex medium went 3 of 5 to 0 of 5 on the same story, and the OAuth quirks story, the other forum fact, shows the same pattern more weakly: across all setups 18 of 35 English runs searched and 7 of 35 Russian ones.
The Claude models hardly search on these tasks in either language; they got the corridor shape from the project's own readback files, which is why Opus and Sonnet are flat. Gemini Flash searches in both languages, hundreds of calls per run, and passes in both. So the effect is not "small models suffer in Russian". It is "an agent whose fallback is a search drops the fallback when the prompt is not in English", and Codex is the one setup here where that fallback was doing real work.
With the guide in the prompt, searching stops in every language: zero to two runs per cell. The guide replaces the search. For Codex under a Russian task that is the whole gap, 0 of 5 to 5 of 5.
What I do differently now
- I write tasks in Russian when that is easier. The agent does not care, and the answer comes back in my language with the identifiers intact.
- I give the agent the fact, not a hint to search. For anything that lives on a forum rather than in the docs, the guide is the reliable path in any language. The search is not.
- I don't translate guides. The English story worked as well as the Russian one. What matters is the identifier, the path and the error string, and those are not translated anyway.
- If your agent searches a lot, watch what it does with a non-English prompt. Give it a task whose answer you know is only on the web, once in English and once in your language, and read the command log, not the answer. That took me minutes per setup.
How I tested
Each task is a small repo with one planted problem and a one- or two-sentence request, the way a person types it. Five arms: the English task alone; the English task plus the story, handed over as "a colleague's note on a similar case, use what applies and say which parts did"; the Russian task alone; the Russian task plus the English story; the Russian task plus the story translated into Russian. Seven setups: Claude Code with Opus, Sonnet and Haiku 5.5; Codex CLI with GPT-6-Sol at medium and low reasoning; Antigravity with Gemini 3.8 Flash and GPT-OSS 120B. Every setup had web access and no memory, instruction files or MCP servers. A hidden script graded every run by the story's own Verify section, reading the repo, the diff and the command log, never the agent's report about itself. Five repetitions per cell, which is why there are intervals above instead of single counts; the intervals are bootstrapped over runs.
Limits
Six tasks, one non-English language, fixtures with one planted problem each. Thirty-five runs per cell is enough for the intervals above and not enough to rank the setups against each other. The Russian texts were translated by a model and checked by me; the stories' code and identifiers are the originals. Whether a model had seen a story in training I cannot check, and no run without a story fetched it.
Everything is on GitHub: fixtures, prompts in both languages, graders and every run's log, at worklore-mcp/research/guide-eval/round8. The guides are stories on worklore, each with a Verify section and a count of the agents that reproduced it. If there is a fix your agent keeps needing, in any language, that format is what it is for.
Top comments (0)