A country’s language can push an LLM’s survey answers away from that country’s baseline, undermining language choice as a shortcut for cultural personalization.
Explicit country cues tell a different story. Across 1.6 million open-ended responses, naming a country closed 30 to 44% of the gap to the model’s own responses for that country. Language cues and explicit country cues did not produce interchangeable effects.
That distinction matters if your assistant combines stored user context with the language of a new message. A change in answers doesn’t establish that the model has moved toward the country you associate with that language.
What the researchers tested
Siddhesh Pawar and colleagues studied Llama, Mistral, and Gemma under conflicting cultural cues.
The system prompt supplied a country persona. The user prompt supplied another country through an explicit country cue or a language cue.
An explicit country cue used a disclosure such as “I am from” followed by a country. A language cue changed the query language without naming a country.
The researchers tested both WVS multiple-choice questions and open-ended sociopolitical generation. This let them compare selected answers with values expressed in free text.
Their baseline needs careful attention. For each country, they used the model’s own responses when that country supplied the system persona. They did not use human survey responses as the target.
So movement toward a country’s baseline means movement toward the model’s existing representation of that country. It doesn’t establish agreement with people from that country.
A larger shift can point the wrong way
In WVS multiple-choice probing, language cues moved answers away from the associated country’s baseline across every tested model and country.
Explicit country cues produced smaller, model-dependent effects. Gemma moved toward the country’s baseline for most countries. Llama and Mistral moved away for nearly all countries.
The paper’s WVS table makes the direction visible. The Language Effect columns show movement away throughout. The Explicit Mention columns vary by model.
For a builder, the distinction is between responsiveness and direction. A model can respond strongly to a language cue while moving away from the baseline your evaluation uses as its target.
There’s also a limitation. The researchers estimated country baselines with English prompts. Language cues changed the query language. The researchers note that cross-lingual distributional shifts may partly explain the movement away.
That limits the conclusion. The result shows that these language cues failed to move answers toward these country baselines. It doesn’t establish that the language itself caused worse cultural understanding.
Free text changes the picture
In open-ended generation, explicit country cues produced more differentiated changes in expressed values than language cues.
This matters because a multiple-choice evaluation and a free-text evaluation give the model different freedom. A model selects among supplied answers in the former. It chooses which values to express in the latter.
I would keep both formats in a cultural personalization evaluation. I’d also measure language cues and explicit country cues separately, with the system persona held fixed.
Otherwise, an evaluation risks collapsing distinct effects into a single claim about cultural adaptation.
Evaluate the cue, not an imagined identity
The researchers do not define a correct answer for someone with ties to multiple countries. Their measurements describe changes in model responses. They do not describe the values of people with those identities.
That boundary makes the work useful. It supports testing how conflicting cues affect an LLM without assigning a prescribed blend of national values to a user.
For builders, the practical lesson is specific. Measure the direction of a cue’s effect, state the baseline, and keep language cues separate from explicit country cues. A model changing its answer is only the start of the evaluation.
Top comments (0)