A user opened our dashboard in Italian and the button said "Rigenera (VAR_1)". Not a variable. The literal text VAR_1, shipped to production, in seven languages, for weeks. Nobody noticed because we don't read Italian, or Japanese, or Hindi.
I build a localization tool. This bug was in my own product. That's the embarrassing part, and also why I think the postmortem is worth sharing: the failure mode applies to anyone piping strings through an LLM.
The setup
Our translation pipeline masks interpolation variables before sending text to the model. Regenerate ({wu} credits) becomes Regenerate (VAR_1 credits), the model translates around the token, and we swap the real variable back afterwards. Standard practice. The model can't mangle {wu} into {wu_créditos} if it never sees it.
Except we had two code paths into the translator. The direct one masked variables. The queued one, added later for batch jobs, didn't. And the prompt, shared by both, told the model "copy tokens like VAR_N through unchanged".
So for queued jobs the model received unmasked text with instructions about tokens that weren't there. Most of the time it translated fine. Sometimes it obeyed the prompt too literally and wrote VAR_1 into the translation, guessing where a token might belong. Valid JSON. Correct-looking diff. Straight into production.
Why nothing caught it
Every safety net we had was looking somewhere else:
The tests covered the masking function, which worked. The bug was that one path never called it.
TypeScript was happy. A string is a string.
The UI rendered fine. VAR_1 is just text, nothing crashes.
And the humans reviewed the English source, which was correct. The broken strings only existed in languages nobody on the team reads. That's the trap with i18n bugs: your ability to notice them is inversely proportional to how many languages you support.
What actually surfaced it
I wrote a check command for our CLI, mostly for CI: it diffs every locale against the source, validates that placeholders match, flags empty values. First time I ran it against our own projects it reported 47 broken strings across the landing page and the dashboard. I assumed the checker was wrong. It wasn't.
Fixing the data took an evening. Fixing the pipeline meant masking on both paths and adding a validation step that rejects any translation whose variables don't match the source, requeuing it with an error instead of shipping it quietly.
The check now runs in CI. A leaked token or a dropped placeholder fails the build the way a type error does, which is the only place this class of bug is catchable, because no human on the team was ever going to read the Hindi strings.
The takeaways
If you're doing AI translation, in any form:
Mask variables before the model sees them, on every path. Auditing that "every path" claim is the actual work.
Never trust the model's output structurally. Validate placeholders against the source after translation, and reject instead of shipping when they differ.
Treat locale files like compiled artifacts. Nobody proofreads a build output; you put a check in CI instead.
The check that caught this is part of i1n, the tool I work on (i1n.ai, CLI is open source). But honestly, even if you never touch it, write the placeholder diff yourself. It's an afternoon of work and it will eventually catch something a human can't, because it reads every language and your team doesn't.
Has anyone else had an LLM follow instructions too well? The model doing exactly what the prompt said, on input the prompt never anticipated, feels like a whole new bug category.
Top comments (0)