A short request in one language, pasted material in another: Claude sometimes answers in the wrong one. What we measured, and a check that catches it.
Mostly because the pasted text outweighs the request. When someone writes one short line in Bulgarian and pastes a long English build log under it, the model reads far more English than Bulgarian, and when it sums up the paste, the summary often comes out in whatever language the paste was written in. A neighbouring language makes it worse: German next to Dutch, Portuguese next to Spanish, Russian next to Bulgarian. Across 24 such requests, Claude Sonnet answered in the wrong language three times when nobody told it whose language counts, and not once after a short rule said so.
Read the full report on AISkills402: https://aiskills402.com/blog/llm-replies-in-wrong-language
Top comments (0)