My app serves the same content in Chinese and English: personalized readings for Chinese birth charts, generated by an LLM. The readings are cached; when a user flips language, regenerating from scratch costs ~3 seconds and produces a different reading — which is exactly what you don't want, because now your two languages disagree about the user's life. The fix is obvious: translate the cached reading instead of regenerating. Language switch went from 2.8s to ~60ms once cached, and both languages finally say the same thing.
But "just translate it" walked straight into the domain-terminology buzzsaw, and the eval I ran to pick a model produced my favorite result of the year: the flagship translation model, at 15× the price, was the most dangerously wrong of the lot.
The baseline eval: fluent nonsense scales with price
The domain (BaZi astrology) has a closed set of technical terms with established English renderings. I ran four sizes of a dedicated translation model family over my own corpus, no glossary, and scored the terms:
| Term | Correct EN | flash (cheap) | plus (flagship, 15×) |
|---|---|---|---|
| 七杀 | Seven Killings | Seven Kill*ers* ✗ | Seven Killers ✗ |
| 印星 | Resource star | Inheritance Star ✗ | "Hidden Stems of the Mind" ✗✗ |
| 比劫 | Companion star | Peer & Robbery Stars ✗ | "Rat–Ox combinations" ✗✗✗ |
| 食神 | Eating God | Food God ✗ | Food God ✗ |
Every model missed terms — expected; the terms are jargon. The pattern that wasn't expected: the flagship's misses weren't near-misses, they were inventions. "Hidden Stems of the Mind" is not a bad translation of 印星 (a chart-analysis category); it's a hallucinated concept delivered with total fluency. The cheap model's "Inheritance Star" is wrong but recognizably adjacent. Paying more bought more fluent wrong answers — a bigger model has more capacity to confabulate plausibly, and in translation, plausible-but-wrong is strictly worse than clunky-but-wrong because nobody catches it in review.
The glossary is the product
The provider's API takes a terms list — source term, forced target rendering — injected at translation time (the docs call it term intervention; most serious MT APIs have an equivalent). With ~15 domain terms pinned:
All four models became term-perfect. Every one. The entire quality race collapsed into prose style, latency, and price:
| Model | Terms (with glossary) | Prose | Latency | Cost per domain snippet |
|---|---|---|---|---|
| flash | perfect | most natural | ~740ms | $0.00015 |
| lite | perfect | slightly flatter | ~610ms | $0.00011 |
| plus | perfect | marginally more idiomatic | ~1100ms | $0.0022 |
So the model decision — the thing I'd expected to agonize over — was over. The cheap-fast one wins on quality-per-dollar; the flagship's "marginal polish" is not worth 15×. All the real engineering turned out to be in what goes into the glossary. Three findings, each from a production incident or a near-miss:
1. Don't glossary everything you could glossary
My first instinct: pin all the domain vocabulary, including the sexagenary "pillar" characters (庚 → Yang Metal, 午 → Horse Fire). The output was technically correct and read like a parts catalog: "Yang Metal seated atop Horse Fire." The unglossed baseline had written "Geng seated atop Wu Fire" — pinyin, the way English-language practitioners actually write. The glossary is for terms where deviation is an error (the analysis vocabulary). Terms where the natural rendering varies by register (name-like characters) read better left to the model, with the fixed mapping reserved for UI labels. Scope the forced list to what must never drift; let prose be prose.
2. A glossary is directional
Reusing the zh→en term table for the en→zh direction doesn't fail loudly — it fails by leaving English terms embedded in Chinese output. Real production sentence: 「你的Day Master是Rén」. The table has to flip per direction, so the lookup is translationTerms(targetLang), not a constant.
3. Reverse-mapping pinyin is a minefield of English collisions
Going back into Chinese, the pinyin names in English prose (Rén, Shēn) must become characters again (壬, 申). I generate that map from the engine's own data tables — but only the tone-marked forms are included. The bare romanizations collide with ordinary English: You is a branch name (酉), Yin is one (寅), Wu is two different ones (戊/午). An unfiltered map happily rewrites every English "you" into 酉. Diacritics, which I'd initially treated as typographic fussiness, turned out to be the only thing making the mapping injective.
Practical constraints worth knowing
Dedicated MT endpoints are shaped differently from chat models, and the shape matters for cost:
Single-turn, no system prompt. You send source text; the glossary rides in a config field that isn't billed as prompt tokens. A chat model doing the same job re-bills you the instruction preamble on every call.
Hard input cap (8,192 tokens here) — long documents need a chunker.
Measured production numbers: a full reading section zh→en in ~2.1s at $0.0003; short strings at $0.00008. Output tokens dominate MT cost, so the output-price column of the pricing table is the one to sort by.
The chain still ends in a fallback: flash → lite → regenerate in the target language if the MT service is down. Degraded consistency beats an error page.
Takeaways
In a domain with fixed terminology, eval the glossary mechanism, not just the models. It flattened a 15× price range into a tie.
Model size doesn't fix domain terms — it upgrades the failure mode from clunky-wrong to fluent-wrong. Fluent-wrong is worse.
Scope the glossary to must-never-drift terms; over-pinning makes prose robotic.
Glossaries and transliteration maps are directional; test the round trip, not just the forward pass.
Translating cached content instead of regenerating buys you speed and cross-language consistency — the second one is the sleeper benefit.
The bilingual app in question: auspiceoracle.com. The eval harness (corpus, runner, full transcripts) lives in the repo and reruns with one command — evals you can't rerun are anecdotes.
Top comments (3)
the fluent-wrong point generalizes way past translation and i think it's the most portable thing here.
a cheap model's errors are shaped wrong, so a reviewer's eye catches them. an expensive model's errors are shaped right, so review passes them. which means review quality degrades as model quality improves. genuinely unpleasant thing to be true, and it's why "the bigger model was more dangerous" isn't a paradox.
the open question i'd want answered on your setup: does term-perfect survive inflection and embedding. 印星 alone is pinned, but 印星的 or a compound containing it as a substring may or may not hit the term intervention depending on how the provider matches. worth twenty minutes with a probe file, because that's the case where the clean form gets a glossary hit and the inflected one gets a hallucinated concept, and your current eval wouldn't separate them.
second one, the 8192 cap. a pinned term straddling a chunk boundary is either half matched or not matched, and it'd fail silently in exactly the way 「你的Day Master是Rén」 did.
the diacritics-as-injectivity finding is the sharpest thing in the post and you nearly buried it as housekeeping. You / 酉 and Wu being either 戊 or 午 is a real collision, not a typography preference.
Both of your probe cases are ones my eval genuinely can't separate today, so I'd rather measure than guess — the inflection one especially. If 印星 pins but 印星的 (or a compound containing it as a substring) slips past the glossary into a hallucinated concept, that's the worst possible shape: the clean form keeps passing review and builds false confidence in exactly the path that's broken. Same for a pinned term straddling the 8192 boundary — it would fail precisely like 「你的Day Master是Rén」 did: silently, and only on long inputs, which is where nobody's looking.
Taking the twenty minutes on a probe file for both. Will report back with numbers either way.
And fair hit on the diacritics finding — I filed it under housekeeping because it looked like typography, but you're right that it's a collision in the output alphabet: two distinct inputs mapping to one output string is an injectivity bug, not a style preference. You/酉 and Wu being either 戊 or 午 means a reader can't invert the transliteration even in principle. That probably deserves its own short post rather than a bullet.
Report back as promised, with numbers. Probe: 12 cases × each key term in three forms — bare (印星), particle-attached (印星的), embedded/adjacent (喜用神 containing 用神; 印星生比劫 chains) — against both the production primary (qwen-mt-flash) and its fallback (qwen-mt-lite).
Result: the inflection hypothesis doesn't hold for the primary. flash went 12/12 — every inflected and embedded form got the term intervention. The provider's matching evidently isn't the naive exact-match I couldn't rule out.
The interesting failure was on an axis neither of us named: the fallback model. lite produced one genuine categorical error — on the bare form, no inflection involved: 比劫弱 came back as "weak Day Master", the concept silently replaced by its neighbour (身弱). Plus three cases where the pinned British "favourable elements" got normalized to American "favorable" — concept right, pinned form not honoured. So the probe's verdict schema needed two tiers: concept-MISS vs spelling-DRIFT, and the one MISS is exactly your fluent-wrong shape — a perfectly plausible sentence a reviewer would pass — living in the cheap fallback. Which means fallback ≠ lossless degradation: a term-sensitive translation that went through the fallback path now gets flagged for review instead of trusted.
The 8192 boundary turned out to be moot in the current architecture: each section translates in its own call, far under the cap, and no chunking code exists to straddle. But your point stands as a time bomb, so there's now a loud length assert at the call site — if a section ever grows past the limit it throws with "this section needs splitting" instead of gambling on server-side truncation.