My translation QA passed 31,020 strings. One locale had quietly lost 45% of its text.

I have a content corpus that people actually read: 282 story seeds, each with ten text slots, each slot translated into nine languages. One TypeScript file holds all of it. It is 56 MB.

Key parity passed. Placeholder integrity passed. Sentence-final punctuation passed. Then I grouped the same data by the batch that produced it, and one locale fell off a cliff.

This post is the four checks in the order I wrote them, why the first three could not possibly have caught the problem, and the one change to the translation pipeline that came out of it.

The corpus

The shape matters, so here it is exactly:

• 282 seeds (each one is a fixed piece of editorial content)
• 10 text slots per seed — two families of five variants each
• 11 locale fields per slot — the English source, the Chinese source, and nine translations
• 2,820 slots, 31,020 locale fields, 25,380 of them translated

Everything lives in one object keyed by seedId/family/variant. Translations arrived over about six weeks in batches — roughly one batch per content group — produced by a model, reviewed by me only by spot-reading. That last detail is the whole story.

Check 1 — key parity (necessary, and nearly useless)

The obvious first check: does every slot have every locale?

const LOCALES = ['es', 'ru', 'pt', 'fr', 'it', 'ar', 'de', 'hi', 'id'];

for (const [key, slot] of Object.entries(corpus)) {
  for (const loc of LOCALES) {
    if (typeof slot[loc] !== 'string' || !slot[loc].trim()) {
      missing.push(`${key} → ${loc}`);
    }
  }
}

Result: 0 missing across 25,380 translated fields.

That is a genuinely good result, and it told me almost nothing. A key-parity check answers one question — "did the write step crash?" — and it will happily pass on a string that says "Coming soon" in all nine languages. Presence is not correctness. I still run it first, because a null in the middle of a render is a crash rather than a quality problem, but I stopped treating a green result as a signal.

Check 2 — length ratio, and why the baseline has to be per locale

Translation length is not random. For prose against English, it is remarkably stable per target language. Median characters-of-translation ÷ characters-of-English, across 2,820 slots:

locale	median ratio	p01	p99
ar	0.822	0.65	1.93
ru	0.959	0.74	2.21
hi	0.967	0.77	2.16
es	1.060	0.84	2.40
pt	1.079	0.86	2.47
it	1.087	0.87	2.46
id	1.088	0.49	1.67
de	1.144	0.94	2.57
fr	1.174	0.92	2.74

Arabic at 0.82 and French at 1.17 is a 43% spread between two healthy locales. Any single global threshold — "flag anything under 60% of English" — is therefore wrong in one direction or the other for most of your languages. You have to calibrate per locale, then compare each string against its own language's baseline:

const median = arr => {
  const s = [...arr].sort((a, b) => a - b);
  return s[Math.floor((s.length - 1) / 2)];
};

const ratios = slots.map(s => s[loc].length / s.en.length);
const base = median(ratios);

const suspicious = slots.filter(
  s => s[loc].length / s.en.length < base * 0.6
);

Look at the id column above: p01 is 0.49 while its median is 1.088. That is not a tail, that is a second population hiding inside the distribution. I did not notice it at the time. Remember it for later.

Check 3 — sentence-final punctuation, and the 750 false alarms

If a translation is cut off mid-generation, it usually loses its final period. So: when the English ends with terminal punctuation and the translation does not, flag it.

I wrote the character class the way you write one when you are thinking in English:

const TERM = /[.!?。！？…؟।"'”’»)\]]\s*$/;

Run it and German explodes: 750 flagged strings out of 2,820.

My first reaction was that the German batch was badly broken. It was not. Every one of those 750 looked like this:

EN tail: ...you said: "Each of you. That was the plan."
DE tail: ...Du sagtest: „Jeder von euch. Das war der Plan.“

German typographic quotes are „…“ — they open with U+201E and close with U+201C, a character that is a left double quote everywhere else. My class had ” (U+201D) and ’ (U+2019) but not “ (U+201C), because in English “ only ever appears at the start of a quotation. So every German string that ended in dialogue ended in a character my regex considered non-terminal.

Add one code point and German goes from 750 → 5:

const TERM =
  /[.!?。！？…؟।"'”’»«“‹›)\]]\s*$/;

The lesson generalizes past German: your punctuation table is locale data, not a constant. French spaces its punctuation (« guillemets », ! with a preceding thin space). Arabic uses ؟ and ،. Hindi uses ।. Japanese and Chinese borrow full-width 。 and ！. A single flat class is a bug waiting to fire the first time a locale's typography disagrees with your assumptions — and it fires loudly, in the direction of false positives, which trains you to ignore the check entirely. That is worse than not having it.

After the fix, the real truncations were clean and obvious:

locale	truncated strings (of 2,820)
ru	30
pt	23
hi	19
fr	14
it	12
es	8
ar	6
de	5
id	3
total	120

120 out of 25,380 — 0.47%. Every one I opened was cut mid-clause, and the pattern was always the same: the model hit an output limit or a timeout, the client wrote whatever it had, and nothing downstream asked "does this sentence end?"

These are cheap to fix: re-request just the failing keys. 120 strings is one small batch.

Check 4 — the batch dimension, which is where it actually broke

id had the fewest truncations of any locale. Three. Its median length ratio was 1.088 — the healthiest-looking number in the table. Both of my checks agreed Indonesian was the best-translated language I had.

Then I did something I should have done on day one: I grouped the ratio by the batch that produced it, not by locale.

// batchOf(key) reads the content-group id off the seed
const byBatch = groupBy(slots, s => batchOf(s.id));

for (const [batch, group] of byBatch) {
  const base = median(group.map(s => s.en.length));
  const ratios = group.map(s => s[loc].length / s.en.length);
  report(batch, loc, median(ratios));
}

Per-locale median ratio, by batch:

content group	en avg chars	cn	es	de	id
group 1	1,095	0.31	1.09	1.18	1.13
group 2	1,328	0.27	1.04	1.12	1.12
group 3	1,290	0.28	1.05	1.14	1.12
group 4	1,388	0.27	1.00	1.10	1.11
group 5	1,290	0.28	1.01	1.14	1.10
group 6	1,593	0.29	1.07	1.12	1.12
group 7	945	0.28	1.04	1.11	1.11
group 8	1,096	0.30	1.10	1.18	1.11
group 9	1,170	0.31	1.16	1.22	1.02
group 10	1,262	0.31	1.18	1.23	0.99
group 11	1,518	0.30	1.06	1.14	0.64
group 12	1,741	0.28	1.04	1.12	0.57
group 13	1,226	0.30	1.08	1.16	1.09

Chinese, Spanish and German are flat across all thirteen groups. Indonesian drops to 0.64 in one group and 0.57 in the next, then returns to 1.09 — as if nothing happened.

202 strings, in two groups, roughly 45% shorter than their English source. And here is why check 3 missed them: 201 of the 202 end with a proper full stop. They are not truncated. They are condensed. The model summarised instead of translating, the output was well-formed, the writer step accepted it, and every structural check went green.

The damage is not a broken word at the end of a paragraph. It is a paragraph that no longer exists, in a language spoken by a few hundred million people, sitting in a shipped build for six weeks.

What changed in the pipeline

Four things, all cheap:

1. Verify at the write boundary, per key, with three assertions. Not "did the response parse" — that was already there — but: the key set returned equals the key set sent; the last character is terminal per a locale-aware class; the paragraph count matches the source's. A single number check would not have caught the condensing, and a single punctuation check would not have caught it either, but paragraph count diverges the moment a model drops a section.

2. Retry only the failures. 120 of 25,380 is 0.47%. Re-requesting those keys as one batch costs a fraction of regenerating a locale, and it keeps the keys that were already good — including the human-reviewed ones you do not want overwritten.

3. Store the baseline as a test fixture. The per-locale, per-batch median ratio is now a committed JSON file. Any future batch that lands more than ~25% off its locale's baseline fails CI before it reaches a build. This is a twelve-line test and it would have caught the Indonesian regression on the day it landed, not six weeks later.

4. Keep the batch id on the content, forever. This was the actual bug in my QA, not in my translations. A field that says which run produced a string costs nothing and lets you ask "was this one bad batch or a systemic decay?" — which is the difference between a targeted re-run and a two-week audit.

The general shape of the mistake

The three checks I wrote first all measured the artifact. Key present, ratio plausible, ends with a period. The failure was in the process that produced the artifact, and process failures are invisible unless the process leaves a trace you grouped by.

If I had to reduce this to one sentence: aggregate per-locale metrics are an average over your batches, and an average is exactly the thing that hides one batch going wrong.

I build NameSoul, an app that gives people a Chinese name and a story behind it — the corpus above is its content, in ten languages, and this is the audit that made me trust it.

Comments

[ArgumentMoney8117]

The German punctuation example is the part that stayed with me — treating the terminator set as locale data rather than a constant is a small change with a large effect. Grouping by batch rather than by locale is the other lesson I will take away: an aggregate can hide exactly the failure it is supposed to catch.

[Melisa Wong]

The terminator set was the one thing I nearly hardcoded — it looked like syntax, not data, so it never occurred to me it belonged with the locale files. What convinced me was how loud the false positives got: 750 warnings taught me to skim, and a check that cries wolf is worse than no check at all.

The batch dimension is what I would still be blind to. The locale with the cleanest per-locale numbers was the one that had lost two paragraphs.