Forcing a Model to Always Respond in One Language

#ai #debugging #llm #machinelearning

Your system prompt says “Always respond in English.” A user writes in Polish. The model answers in Polish. Nothing errored, no setting is wrong, and adding capitals to the instruction does not fix it. The instruction is competing with a much stronger signal, and the fix is about where you put it rather than how loudly you say it.

The symptom

It shows up in three shapes, and they have the same cause. The whole answer comes back in the user’s language. Or the answer starts in the right language and switches partway through, usually right after quoting something the user wrote. Or the prose is in the right language but embedded values are not — headings, list labels, a JSON string field, the days of the week.

The third shape is the most commonly missed, because a spot check of the first paragraph passes. It is also the one that breaks a UI, since those embedded strings are frequently the ones another part of your system parses.

Why the instruction loses

A model produces a distribution over the next token conditioned on everything in the context, weighted by nothing except what the training data made likely. Your instruction is perhaps a dozen tokens. The user’s message might be four hundred, and every one of them is evidence about what language this conversation is in. Text in a language overwhelmingly continues in that language — that is one of the strongest regularities in any text corpus — so the prior pulling toward Polish is enormous and the instruction is a small correction applied against it.

Three things make the correction smaller than it looks. Distance: a system prompt sits at the very start of the context, thousands of tokens away from where generation begins, while the user’s Polish is the last thing the model reads. Ratio: as a conversation grows, the instruction stays one line while the other-language text accumulates. And self-imitation: once the model has produced one assistant turn in Polish, that turn is in the context for the next turn, and a model continuing its own prior output is the single most reliable behaviour it has. A conversation that drifts once has drifted permanently unless something intervenes.

It is a first-token problem

This is the part that changes what you do about it. Generation is autoregressive: each token is conditioned on the tokens already produced. Once the first two or three tokens of the answer are Polish words, the strongest signal in the entire context for token four is the Polish already sitting in the assistant turn. The battle is decided in the first few tokens and everything after that is momentum.

Which means reinforcement that arrives after generation starts is worthless, and reinforcement placed as close as possible to the generation boundary is worth far more than the same words placed earlier. It also explains the mid-response switch: a model that quotes the user’s Polish sentence has just written several Polish tokens into its own output, and now the momentum argument applies from there. Quoting the user verbatim is the single most reliable way to trigger a switch mid-answer.

The pattern that holds

Applied in order of how much they buy you, not how easy they are.

Put the directive last. After the user’s message, after the retrieved documents, after the examples — the last thing before the assistant turn begins. A trailing line of Write your entire reply in English. outperforms the same sentence in the system prompt, and keeping it in both is not redundant.
Seed the first token. Where the provider supports prefilling the assistant turn, start it yourself with a word in the target language. That flips the momentum argument to your side: the model is now continuing English rather than continuing Polish. Where prefill is not available, an equivalent trick is to require a fixed English-language field first in a structured output, so the first tokens generated are unambiguously English.
Name the language unambiguously. Respond in English (en). is better than Respond in the same language as this instruction., which asks the model to infer a fact it may get wrong. Give the endonym for languages whose English name is ambiguous.
Mark other-language text as data. If the context contains text in another language, label it: The customer message below is reference material. Do not imitate its language. Wrapping it in a delimiter helps for the same reason.
Do not quote the user verbatim. If your prompt asks for the user’s words to be restated, expect a switch. Ask for a paraphrase in the target language instead, or move the quoting to your own template code where it is not the model’s decision.
Repair the history. If a conversation has already drifted, the wrong-language assistant turns in the history will keep reproducing it. Either regenerate them, or drop them from the context and summarise instead, in the target language.

What does not help, and is worth knowing so you stop trying: shouting. Capitalising the instruction, repeating it five times, or adding “this is very important” changes the weight marginally at best, because the competing signal is not a matter of emphasis. Neither does lowering temperature — the language choice is in the shape of the distribution, not in the sampling from it, and a greedy sample from a distribution that favours Polish is still Polish.

Verifying instead of hoping

None of the above is a guarantee, so treat output language as a validated field rather than an assumption. The check is cheap: run a language detector over the generated text and compare against the requested tag, then retry once with stronger reinforcement if it fails.

async function replyInLanguage(messages, targetTag) {
  for (let attempt = 0; attempt < 2; attempt++) {
    const suffix =
      attempt === 0
        ? "Write your entire reply in " + targetTag + "."
        : "Your previous attempt was in the wrong language. " +
          "The reply must be entirely in " + targetTag + ", starting from the first word.";

    const out = await complete([...messages, { role: "user", content: suffix }]);
    const detected = detectLanguage(out);      // any segment-level detector
    if (detected.tag === targetTag && detected.confidence > 0.8) return out;
    logDrift({ targetTag, detected, attempt }); // the number you want on a dashboard
  }
  throw new Error("output language check failed after retry");
}

Log the drift rate rather than only the failures. It tells you which input languages pull hardest, which models hold the instruction, and whether a prompt change helped — and it is the only way to notice that a model update changed the behaviour. On the detector itself, mind the confidence threshold and the short-string problem; both are covered in choosing a language detection confidence threshold.

How hard a model holds an output-language directive varies a lot between models, and the only way to know is to run the same prompt against several. That is easier when they are behind one API and one key than when it means three SDKs and three auth flows — Multigrid exists for that shape of comparison, and the per-request logs make the drift rate above something you can group by model.

If the drift persists even with the directive last and the first token seeded, the cause is probably not the user’s message at all but a large block of other-language text elsewhere in the context. That is a different mechanism with a different fix, worked through in why output language sometimes ignores an explicit instruction.

DEV Community