My app serves the same content in Chinese and English: personalized readings for Chinese birth charts, generated by an LLM. The readings are cached; when a user flips language, regenerating from scratch costs ~3 seconds and produces a different reading — which is exactly what you don't want, because now your two languages disagree about the user's life. The fix is obvious: translate the cached reading instead of regenerating. Language switch went from 2.8s to ~60ms once cached, and both languages finally say the same thing.
But "just translate it" walked straight into the domain-terminology buzzsaw, and the eval I ran to pick a model produced my favorite result of the year: the flagship translation model, at 15× the price, was the most dangerously wrong of the lot.
The baseline eval: fluent nonsense scales with price
The domain (BaZi astrology) has a closed set of technical terms with established English renderings. I ran four sizes of a dedicated translation model family over my own corpus, no glossary, and scored the terms:
| Term | Correct EN | flash (cheap) | plus (flagship, 15×) |
|---|---|---|---|
| 七杀 | Seven Killings | Seven Kill*ers* ✗ | Seven Killers ✗ |
| 印星 | Resource star | Inheritance Star ✗ | "Hidden Stems of the Mind" ✗✗ |
| 比劫 | Companion star | Peer & Robbery Stars ✗ | "Rat–Ox combinations" ✗✗✗ |
| 食神 | Eating God | Food God ✗ | Food God ✗ |
Every model missed terms — expected; the terms are jargon. The pattern that wasn't expected: the flagship's misses weren't near-misses, they were inventions. "Hidden Stems of the Mind" is not a bad translation of 印星 (a chart-analysis category); it's a hallucinated concept delivered with total fluency. The cheap model's "Inheritance Star" is wrong but recognizably adjacent. Paying more bought more fluent wrong answers — a bigger model has more capacity to confabulate plausibly, and in translation, plausible-but-wrong is strictly worse than clunky-but-wrong because nobody catches it in review.
The glossary is the product
The provider's API takes a terms list — source term, forced target rendering — injected at translation time (the docs call it term intervention; most serious MT APIs have an equivalent). With ~15 domain terms pinned:
All four models became term-perfect. Every one. The entire quality race collapsed into prose style, latency, and price:
| Model | Terms (with glossary) | Prose | Latency | Cost per domain snippet |
|---|---|---|---|---|
| flash | perfect | most natural | ~740ms | $0.00015 |
| lite | perfect | slightly flatter | ~610ms | $0.00011 |
| plus | perfect | marginally more idiomatic | ~1100ms | $0.0022 |
So the model decision — the thing I'd expected to agonize over — was over. The cheap-fast one wins on quality-per-dollar; the flagship's "marginal polish" is not worth 15×. All the real engineering turned out to be in what goes into the glossary. Three findings, each from a production incident or a near-miss:
1. Don't glossary everything you could glossary
My first instinct: pin all the domain vocabulary, including the sexagenary "pillar" characters (庚 → Yang Metal, 午 → Horse Fire). The output was technically correct and read like a parts catalog: "Yang Metal seated atop Horse Fire." The unglossed baseline had written "Geng seated atop Wu Fire" — pinyin, the way English-language practitioners actually write. The glossary is for terms where deviation is an error (the analysis vocabulary). Terms where the natural rendering varies by register (name-like characters) read better left to the model, with the fixed mapping reserved for UI labels. Scope the forced list to what must never drift; let prose be prose.
2. A glossary is directional
Reusing the zh→en term table for the en→zh direction doesn't fail loudly — it fails by leaving English terms embedded in Chinese output. Real production sentence: 「你的Day Master是Rén」. The table has to flip per direction, so the lookup is translationTerms(targetLang), not a constant.
3. Reverse-mapping pinyin is a minefield of English collisions
Going back into Chinese, the pinyin names in English prose (Rén, Shēn) must become characters again (壬, 申). I generate that map from the engine's own data tables — but only the tone-marked forms are included. The bare romanizations collide with ordinary English: You is a branch name (酉), Yin is one (寅), Wu is two different ones (戊/午). An unfiltered map happily rewrites every English "you" into 酉. Diacritics, which I'd initially treated as typographic fussiness, turned out to be the only thing making the mapping injective.
Practical constraints worth knowing
Dedicated MT endpoints are shaped differently from chat models, and the shape matters for cost:
Single-turn, no system prompt. You send source text; the glossary rides in a config field that isn't billed as prompt tokens. A chat model doing the same job re-bills you the instruction preamble on every call.
Hard input cap (8,192 tokens here) — long documents need a chunker.
Measured production numbers: a full reading section zh→en in ~2.1s at $0.0003; short strings at $0.00008. Output tokens dominate MT cost, so the output-price column of the pricing table is the one to sort by.
The chain still ends in a fallback: flash → lite → regenerate in the target language if the MT service is down. Degraded consistency beats an error page.
Takeaways
In a domain with fixed terminology, eval the glossary mechanism, not just the models. It flattened a 15× price range into a tie.
Model size doesn't fix domain terms — it upgrades the failure mode from clunky-wrong to fluent-wrong. Fluent-wrong is worse.
Scope the glossary to must-never-drift terms; over-pinning makes prose robotic.
Glossaries and transliteration maps are directional; test the round trip, not just the forward pass.
Translating cached content instead of regenerating buys you speed and cross-language consistency — the second one is the sleeper benefit.
The bilingual app in question: auspiceoracle.com. The eval harness (corpus, runner, full transcripts) lives in the repo and reruns with one command — evals you can't rerun are anecdotes.
Top comments (0)