Your CI checks your types. It doesn't check your translations.

Your pipeline lints your JavaScript. It typechecks your types. It runs your tests, checks your
formatting, audits your dependencies, and scans for secrets.

Then it ships a de.json that nobody has read, produced by a model that nobody audited, straight to
production.

Here is a bug I planted in a test corpus. It is representative of the real thing:

en: "Your changes were saved successfully."
es: "No se pudieron guardar tus cambios."

The Spanish says your changes could not be saved. Not a near-miss — the exact inverse of the
English. Same key. No placeholders in either string, so nothing to drop. Valid JSON. Byte-identical
key sets.

Every i18n checker I know of passes that file.

What key-diffing can and can't see

The standard tooling compares the shape of your locale files. Take the source tree, take the target
tree, flatten both, diff the key sets. You get:

• keys in the source that are missing from the target
• keys in the target that no longer exist in the source
• sometimes: empty values, or values identical to the source

This is genuinely useful and you should absolutely have it. It catches the single most common i18n
defect — the key someone added on Friday that nobody translated. It runs in milliseconds, it is
deterministic, and it costs nothing.

But look at what it is actually examining. It reads the keys. It does not read the values. Every
check in that list is a statement about structure, and the class of bug above is not a structural
bug. The structure is perfect. The meaning is inverted.

There is a second tier that structural tools can reach, and it is worth doing well: placeholder
drift.

Placeholder drift, which is structural and still hard

en: "Hello {{name}}, you have {{count}} new messages"
es: "Hola, tienes mensajes nuevos"

Both placeholders are gone. Depending on your i18n library and your framework, this either renders
as a sentence that has lost its subject and its number, or it throws at runtime, or — my favourite —
it silently renders the literal string {{name}} to a user in production.

The reason this is harder than it looks is that "placeholder" is not one syntax. In a mixed codebase
you will meet:

Syntax	Where
{{name}}	i18next, Vue I18n, Handlebars
{name}	ICU MessageFormat, react-intl
%s, %d	printf-style, gettext
%1$s	positional printf — Android, and reorderable
%@, %lld	Apple .strings / .xcstrings
%{name}	Ruby i18n
$t(key)	i18next interpolation of another key
<0>, <b>	react-i18next Trans components, inline HTML

A checker that only knows {{...}} will report a clean file for a tree full of %1$s. And
positional printf has a failure mode of its own: a translator may legitimately reorder %1$s and
%2$s to fit target grammar, so "the placeholders appear in a different order" is correct and must
not be flagged, while "%2$s disappeared entirely" is a bug.

Direction matters too. A placeholder that exists in the source and is gone from the translation is an
error. A placeholder invented in the translation that has no counterpart in the source is a
different, weirder problem — usually a hallucination — and it deserves a warning rather than a hard
failure, because occasionally it is deliberate.

So: do the structural pass, do it across every syntax, and get the direction and the positional case
right. That covers most defects. It does not cover the inverted sentence.

Using an LLM as the judge, carefully

The obvious move is to have a model read each source/translation pair and say whether the translation
means what the source means. The obvious move is also where most of the danger is, so it is worth
being precise about the failure modes.

Judge scores are unstable across runs. This is well documented — the GEMBA line of work found
single-pass LLM quality scores swinging wildly on identical input. If you ask once and act on the
answer, you have built a random number generator with good manners. The mitigation is to ask N times
and take the majority; I default to 3 passes and discard any pass that fails to parse rather than
counting it as a non-flag.

A noisy gate gets uninstalled. This is the part I feel strongest about. If a check fails builds
on judgment calls about tone, the first thing a team does is turn it off, and then they have neither
the semantic check nor the structural one. So the semantic findings are advisory by default —
they surface as warnings and do not change your exit code unless you explicitly opt in with
--semantic-fail. The structural checks, which are deterministic, are the ones allowed to fail your
build.

Don't pay a model to look at something you already know is broken. Keys that already carry a
structural error are excluded from judging entirely. If a string has dropped its placeholder, you do
not need an opinion about its semantics; you need to fix the placeholder. This is also the cost
story: on a tree where everything is broken, the judge makes zero calls.

Only judge what changed. Verdicts are cached by a hash of the source/translation pair. A re-run
where nothing changed costs nothing at all — literally zero model calls. This is what makes it
viable in CI rather than a thing you run once and disable.

Give it the categories, not a score. Instead of "rate this 1-100", the judge returns an MQM-style
error type: mistranslation, omission, or addition. These are checkable. A number is not.

Here is what each of those looks like, from the corpus:

mistranslation
  en: "Your changes were saved successfully."
  es: "No se pudieron guardar tus cambios."          ← "could not be saved"

omission
  en: "Save your work before closing the window, or unsaved changes will be lost."
  es: "Guarda tu trabajo antes de cerrar la ventana."  ← the consequence is gone

addition
  en: "Your file has been uploaded."
  es: "Tu archivo se ha subido y se compartirá con todo tu equipo."
                                          ← "and will be shared with your whole team"

That last one is my favourite category, because it is the one no human reviewer catches by skimming.
The Spanish reads beautifully. It is fluent, natural, correctly conjugated — and it promises your
users something your product does not do.

There is a fourth check that needs no model at all: a glossary. You list the terms that must never be
translated — product names, "OAuth", "webhook" — and the terms that must always be translated one
specific way. That is a deterministic string check, and it catches this:

en: "Shipi18n checks your translations in CI."
es: "EnvíoI18n comprueba tus traducciones en CI."     ← it translated the product name

Measuring it instead of claiming it

Here is the part I would want to interrogate if I were reading someone else's post, so let me be
explicit about the methodology before the numbers.

The corpus was committed before the judge existed. 228 source/translation pairs across Spanish,
German and Japanese: 168 clean pairs taken from real production locale files, and 60 with deliberately
planted errors — 18 mistranslations, 18 omissions, 18 additions, 6 glossary violations. It went into
git at commit 60d699b, and the judge was written afterwards.

The thresholds were fixed before the implementation, too. Catch rate ≥80%, per-category recall
≥60%, false positives <10%, glossary recall 100%. Writing the thresholds after seeing the results is
how you produce a number that means nothing, and it is very easy to do accidentally.

The runner is the gate. It exits non-zero when a threshold is missed, so I cannot quietly ship a
regression and keep quoting the old figures.

The results, on claude-haiku-4-5 with 3 passes:

catch rate         54/54 planted errors flagged      (100%)
per-category       mistranslation 100% · omission 100% · addition 100%
false positives    12/168 clean pairs flagged        (7.1%)
glossary           6/6 caught, 0 false
label accuracy     100% this run — 98.1%–100% across three runs
cost               48 model calls · ~59k tokens · 138s

Two things about those numbers that matter more than the numbers.

Label accuracy is a range, not a constant. Across runs a day apart it was 100% and then 98.1%.
The catch rate and false-positive rate reproduced exactly; the labels moved. If you see a single
crisp percentage quoted for anything a language model produces, ask how many times they ran it. I
published the range because publishing the better of two runs would have been a lie of selection.

The false-positive rate is the number I care about, and it is not zero. Twelve clean pairs
got flagged. Here is what they actually look like:

[es] "i18n & Localization Blog | Shipi18n"
     "Blog - API de traducción de Shipi18n"

[de] "Tutorials and guides on i18next, React i18n, Next.js localization…"
     "Erfahren Sie mehr über Internationalisierung, Best Practices…"

[ja] "View @shipi18n/mcp on npm"
     "npmで@shipi18n/mcpを見る"

I want to be precise about these, because "false positive" is doing different work in each one.

The first is not a false positive at all — the judge is right and I was wrong. The Spanish says
"Shipi18n translation API", which is a product that no longer exists. That is stale copy sitting in
my locale files, and the only reason it counts against the tool in this table is that my corpus
labelled it clean. Three of the twelve are that same category of thing.

The second is a judgment call. The German is a rewrite rather than a translation — it conveys the
gist and restructures the sentence. Whether that is an error depends on whether you think marketing
copy should be translated or transcreated, and reasonable people disagree.

The third is a real false positive. The Japanese is fine. The judge flagged it anyway.

So the honest reading of "7.1%" is: some of it is the tool being right about data I mislabelled, some
is the tool having opinions about tone, and some is just wrong. I report the whole 7.1% rather than
relitigating the labels in my favour, because a corpus you edit after seeing the results is not a
corpus any more.

These are the cost of the feature. Some are arguments about tone rather than errors. On a real
production tree the flag rate came out around 3.6%, and when I went through those by hand, several of
the "false positives" turned out to be true catches of stale copy I had forgotten about — the site
still described a "translation API" that no longer existed, and the Japanese used 連携 ("integration
/ linkage") for a nav item that meant something narrower.

That is the honest shape of this feature: it finds real problems, and it also wants to talk to you
about things that are fine. Which is exactly why it does not fail your build.

What it found in my own project

The most useful thing I can tell you about this tool is what it failed to catch.

I built it, dogfooded it on my own site — eleven languages — and it passed. Then, doing an unrelated
review, I found two things that had been live for weeks.

The documentation described a feature that did not work. My MCP server was advertised as being
able to translate with no API key at all, via a mechanism called MCP sampling, where the client's
model does the inference. That mechanism was deprecated in the spec, and Claude Desktop — the client
my own setup instructions told people to configure — never implemented it. The claim was live in
eleven languages. It was a translation of a sentence that was false in English.

The copy-paste command on my own homepage exited 1. The hero command was npx @shipi18n/cli
translate .... If you ran it exactly as printed, it failed, because npx installs the CLI but not
the provider SDK, which is an optional peer dependency. The first command a visitor would ever run
did not work, in the README and on the homepage, for weeks.

My QA tool caught neither, and it was right not to: neither is a translation defect. The German
translation of a false sentence is a perfectly good translation. This is the boundary of the thing —
it checks that your translations say what your source says. It has no opinion about whether your
source is true.

Both are fixed. I am telling you because a post about a QA tool that only lists what the tool catches
is an advertisement, and because the second one is a genuinely useful warning about optional peer
dependencies and npx that has nothing to do with i18n.

Try it

The structural half needs no API key, no account, and no config:

npx @shipi18n/cli check ./locales -s en

JSON, Flutter .arb, Apple .xcstrings. Human output, JSON, SARIF so GitHub annotates your PRs
inline, or JUnit.

The semantic half needs your own key, and only ever talks to the provider you picked:

npm i -D @shipi18n/cli @anthropic-ai/sdk
npx shipi18n check ./locales -s en --semantic

There is also an MCP server, if you would rather have your agent do it — its validator tools make no
model calls at all; the review tool hands the pairs and the criteria back to the agent, which judges
with its own inference.

If you want to reproduce the example this article opened with, it takes four commands:

mkdir -p demo/locales && cd demo && npm init -y
npm i -D @shipi18n/cli @anthropic-ai/sdk
echo '{"saved":"Your changes were saved successfully."}'  > locales/en.json
echo '{"saved":"No se pudieron guardar tus cambios."}'    > locales/es.json
npx shipi18n check ./locales -s en              # passes — nothing structural is wrong
npx shipi18n check ./locales -s en --semantic   # catches it

The second command prints:

warning  saved  semantic-mistranslation — Source states changes were saved
         successfully; translation states changes could not be saved (opposite meaning)

and still exits 0, because it is advisory.

Apache-2.0, no server, no telemetry: https://github.com/Shipi18n/shipi18n

The eval harness and the whole corpus are in the repo under evals/semantic/. If you want to know
whether these numbers hold for your language pair, your domain, or a different judge model, run it —
JUDGE_MODEL=... node evals/semantic/run.mjs. I would genuinely rather hear that you made it look
worse than never hear from you. The false-positive rate is where I expect it to break first.