DEV Community: Oscar Green

Your CI checks your types. It doesn't check your translations.

Oscar Green — Sat, 22 Aug 2026 11:26:30 +0000

Your pipeline lints your JavaScript. It typechecks your types. It runs your tests, checks your
formatting, audits your dependencies, and scans for secrets.

Then it ships a de.json that nobody has read, produced by a model that nobody audited, straight to
production.

Here is a bug I planted in a test corpus. It is representative of the real thing:

en: "Your changes were saved successfully."
es: "No se pudieron guardar tus cambios."

The Spanish says your changes could not be saved. Not a near-miss — the exact inverse of the
English. Same key. No placeholders in either string, so nothing to drop. Valid JSON. Byte-identical
key sets.

Every i18n checker I know of passes that file.

What key-diffing can and can't see

The standard tooling compares the shape of your locale files. Take the source tree, take the target
tree, flatten both, diff the key sets. You get:

keys in the source that are missing from the target
keys in the target that no longer exist in the source
sometimes: empty values, or values identical to the source

This is genuinely useful and you should absolutely have it. It catches the single most common i18n
defect — the key someone added on Friday that nobody translated. It runs in milliseconds, it is
deterministic, and it costs nothing.

But look at what it is actually examining. It reads the keys. It does not read the values. Every
check in that list is a statement about structure, and the class of bug above is not a structural
bug. The structure is perfect. The meaning is inverted.

There is a second tier that structural tools can reach, and it is worth doing well: placeholder
drift.

Placeholder drift, which is structural and still hard

en: "Hello {{name}}, you have {{count}} new messages"
es: "Hola, tienes mensajes nuevos"

Both placeholders are gone. Depending on your i18n library and your framework, this either renders
as a sentence that has lost its subject and its number, or it throws at runtime, or — my favourite —
it silently renders the literal string {{name}} to a user in production.

The reason this is harder than it looks is that "placeholder" is not one syntax. In a mixed codebase
you will meet:

Syntax	Where
`{{name}}`	i18next, Vue I18n, Handlebars
`{name}`	ICU MessageFormat, react-intl
`%s`, `%d`	printf-style, gettext
`%1$s`	positional printf — Android, and reorderable
`%@`, `%lld`	Apple `.strings` / `.xcstrings`
`%{name}`	Ruby i18n
`$t(key)`	i18next interpolation of another key
`<0>`, `<b>`	react-i18next Trans components, inline HTML

A checker that only knows {{...}} will report a clean file for a tree full of %1$s. And
positional printf has a failure mode of its own: a translator may legitimately reorder %1$s and
%2$s to fit target grammar, so "the placeholders appear in a different order" is correct and must
not be flagged, while "%2$s disappeared entirely" is a bug.

Direction matters too. A placeholder that exists in the source and is gone from the translation is an
error. A placeholder invented in the translation that has no counterpart in the source is a
different, weirder problem — usually a hallucination — and it deserves a warning rather than a hard
failure, because occasionally it is deliberate.

So: do the structural pass, do it across every syntax, and get the direction and the positional case
right. That covers most defects. It does not cover the inverted sentence.

Using an LLM as the judge, carefully

The obvious move is to have a model read each source/translation pair and say whether the translation
means what the source means. The obvious move is also where most of the danger is, so it is worth
being precise about the failure modes.

Judge scores are unstable across runs. This is well documented — the GEMBA line of work found
single-pass LLM quality scores swinging wildly on identical input. If you ask once and act on the
answer, you have built a random number generator with good manners. The mitigation is to ask N times
and take the majority; I default to 3 passes and discard any pass that fails to parse rather than
counting it as a non-flag.

A noisy gate gets uninstalled. This is the part I feel strongest about. If a check fails builds
on judgment calls about tone, the first thing a team does is turn it off, and then they have neither
the semantic check nor the structural one. So the semantic findings are advisory by default —
they surface as warnings and do not change your exit code unless you explicitly opt in with
--semantic-fail. The structural checks, which are deterministic, are the ones allowed to fail your
build.

Don't pay a model to look at something you already know is broken. Keys that already carry a
structural error are excluded from judging entirely. If a string has dropped its placeholder, you do
not need an opinion about its semantics; you need to fix the placeholder. This is also the cost
story: on a tree where everything is broken, the judge makes zero calls.

Only judge what changed. Verdicts are cached by a hash of the source/translation pair. A re-run
where nothing changed costs nothing at all — literally zero model calls. This is what makes it
viable in CI rather than a thing you run once and disable.

Give it the categories, not a score. Instead of "rate this 1-100", the judge returns an MQM-style
error type: mistranslation, omission, or addition. These are checkable. A number is not.

Here is what each of those looks like, from the corpus:

mistranslation
  en: "Your changes were saved successfully."
  es: "No se pudieron guardar tus cambios."          ← "could not be saved"

omission
  en: "Save your work before closing the window, or unsaved changes will be lost."
  es: "Guarda tu trabajo antes de cerrar la ventana."  ← the consequence is gone

addition
  en: "Your file has been uploaded."
  es: "Tu archivo se ha subido y se compartirá con todo tu equipo."
                                          ← "and will be shared with your whole team"

That last one is my favourite category, because it is the one no human reviewer catches by skimming.
The Spanish reads beautifully. It is fluent, natural, correctly conjugated — and it promises your
users something your product does not do.

There is a fourth check that needs no model at all: a glossary. You list the terms that must never be
translated — product names, "OAuth", "webhook" — and the terms that must always be translated one
specific way. That is a deterministic string check, and it catches this:

en: "Shipi18n checks your translations in CI."
es: "EnvíoI18n comprueba tus traducciones en CI."     ← it translated the product name

Measuring it instead of claiming it

Here is the part I would want to interrogate if I were reading someone else's post, so let me be
explicit about the methodology before the numbers.

The corpus was committed before the judge existed. 228 source/translation pairs across Spanish,
German and Japanese: 168 clean pairs taken from real production locale files, and 60 with deliberately
planted errors — 18 mistranslations, 18 omissions, 18 additions, 6 glossary violations. It went into
git at commit 60d699b, and the judge was written afterwards.

The thresholds were fixed before the implementation, too. Catch rate ≥80%, per-category recall
≥60%, false positives <10%, glossary recall 100%. Writing the thresholds after seeing the results is
how you produce a number that means nothing, and it is very easy to do accidentally.

The runner is the gate. It exits non-zero when a threshold is missed, so I cannot quietly ship a
regression and keep quoting the old figures.

The results, on claude-haiku-4-5 with 3 passes:

catch rate         54/54 planted errors flagged      (100%)
per-category       mistranslation 100% · omission 100% · addition 100%
false positives    12/168 clean pairs flagged        (7.1%)
glossary           6/6 caught, 0 false
label accuracy     100% this run — 98.1%–100% across three runs
cost               48 model calls · ~59k tokens · 138s

Two things about those numbers that matter more than the numbers.

Label accuracy is a range, not a constant. Across runs a day apart it was 100% and then 98.1%.
The catch rate and false-positive rate reproduced exactly; the labels moved. If you see a single
crisp percentage quoted for anything a language model produces, ask how many times they ran it. I
published the range because publishing the better of two runs would have been a lie of selection.

The false-positive rate is the number I care about, and it is not zero. Twelve clean pairs
got flagged. Here is what they actually look like:

[es] "i18n & Localization Blog | Shipi18n"
     "Blog - API de traducción de Shipi18n"

[de] "Tutorials and guides on i18next, React i18n, Next.js localization…"
     "Erfahren Sie mehr über Internationalisierung, Best Practices…"

[ja] "View @shipi18n/mcp on npm"
     "npmで@shipi18n/mcpを見る"

I want to be precise about these, because "false positive" is doing different work in each one.

The first is not a false positive at all — the judge is right and I was wrong. The Spanish says
"Shipi18n translation API", which is a product that no longer exists. That is stale copy sitting in
my locale files, and the only reason it counts against the tool in this table is that my corpus
labelled it clean. Three of the twelve are that same category of thing.

The second is a judgment call. The German is a rewrite rather than a translation — it conveys the
gist and restructures the sentence. Whether that is an error depends on whether you think marketing
copy should be translated or transcreated, and reasonable people disagree.

The third is a real false positive. The Japanese is fine. The judge flagged it anyway.

So the honest reading of "7.1%" is: some of it is the tool being right about data I mislabelled, some
is the tool having opinions about tone, and some is just wrong. I report the whole 7.1% rather than
relitigating the labels in my favour, because a corpus you edit after seeing the results is not a
corpus any more.

These are the cost of the feature. Some are arguments about tone rather than errors. On a real
production tree the flag rate came out around 3.6%, and when I went through those by hand, several of
the "false positives" turned out to be true catches of stale copy I had forgotten about — the site
still described a "translation API" that no longer existed, and the Japanese used 連携 ("integration
/ linkage") for a nav item that meant something narrower.

That is the honest shape of this feature: it finds real problems, and it also wants to talk to you
about things that are fine. Which is exactly why it does not fail your build.

What it found in my own project

The most useful thing I can tell you about this tool is what it failed to catch.

I built it, dogfooded it on my own site — eleven languages — and it passed. Then, doing an unrelated
review, I found two things that had been live for weeks.

The documentation described a feature that did not work. My MCP server was advertised as being
able to translate with no API key at all, via a mechanism called MCP sampling, where the client's
model does the inference. That mechanism was deprecated in the spec, and Claude Desktop — the client
my own setup instructions told people to configure — never implemented it. The claim was live in
eleven languages. It was a translation of a sentence that was false in English.

The copy-paste command on my own homepage exited 1. The hero command was npx @shipi18n/cli translate .... If you ran it exactly as printed, it failed, because npx installs the CLI but not
the provider SDK, which is an optional peer dependency. The first command a visitor would ever run
did not work, in the README and on the homepage, for weeks.

My QA tool caught neither, and it was right not to: neither is a translation defect. The German
translation of a false sentence is a perfectly good translation. This is the boundary of the thing —
it checks that your translations say what your source says. It has no opinion about whether your
source is true.

Both are fixed. I am telling you because a post about a QA tool that only lists what the tool catches
is an advertisement, and because the second one is a genuinely useful warning about optional peer
dependencies and npx that has nothing to do with i18n.

Try it

The structural half needs no API key, no account, and no config:

npx @shipi18n/cli check ./locales -s en

JSON, Flutter .arb, Apple .xcstrings. Human output, JSON, SARIF so GitHub annotates your PRs
inline, or JUnit.

The semantic half needs your own key, and only ever talks to the provider you picked:

npm i -D @shipi18n/cli @anthropic-ai/sdk
npx shipi18n check ./locales -s en --semantic

There is also an MCP server, if you would rather have your agent do it — its validator tools make no
model calls at all; the review tool hands the pairs and the criteria back to the agent, which judges
with its own inference.

If you want to reproduce the example this article opened with, it takes four commands:

mkdir -p demo/locales && cd demo && npm init -y
npm i -D @shipi18n/cli @anthropic-ai/sdk
echo '{"saved":"Your changes were saved successfully."}'  > locales/en.json
echo '{"saved":"No se pudieron guardar tus cambios."}'    > locales/es.json
npx shipi18n check ./locales -s en              # passes — nothing structural is wrong
npx shipi18n check ./locales -s en --semantic   # catches it

The second command prints:

warning  saved  semantic-mistranslation — Source states changes were saved
         successfully; translation states changes could not be saved (opposite meaning)

and still exits 0, because it is advisory.

Apache-2.0, no server, no telemetry: https://github.com/Shipi18n/shipi18n

The eval harness and the whole corpus are in the repo under evals/semantic/. If you want to know
whether these numbers hold for your language pair, your domain, or a different judge model, run it —
JUDGE_MODEL=... node evals/semantic/run.mjs. I would genuinely rather hear that you made it look
worse than never hear from you. The false-positive rate is where I expect it to break first.

tsc-correctness != runtime-correctness

Oscar Green — Sun, 24 May 2026 12:28:12 +0000

Here is a TypeScript error you have probably seen.

src/Header.tsx:3:10 - error TS2614: Module '"./logo.svg"' has no exported
  member 'ReactComponent'. Did you mean to use 'import Logo from "./logo.svg"'
  instead?

tsc helpfully tells you the fix. Your editor offers it as a one-click Quick Fix. An LLM doing code repair will reliably take it. After the fix:

import Logo from "./logo.svg";
//                            ← tsc is now green
//                            ← the dev server now crashes

The fixed file type-checks. The app is broken. Logo is now the asset URL string, not a React component, so <Logo /> blows up with "Logo is not a function" the moment the page renders.

This is the gap LLM-driven code repair walks into every day, and the gap tsfix is built around.

The three characters

Three things shape the LLM-repair failure in this case, and they all have to be in the room to see why it happens.

The library author. vite-plugin-svgr v4 changed how you import an SVG as a React component. The default import is now the asset URL, and you have to opt in to the React-component import with a ?react query suffix:

import Logo from "./logo.svg?react";

Reasonable choice for them: it makes the plugin's default behavior match what every other Vite asset import does, and the ?react suffix is explicit. Breaking change, but small. Their migration guide says exactly this.

The TypeScript team. TS doesn't know about Vite plugins. All it sees is a *.svg ambient module declaration. If your declaration says export const ReactComponent: ... and you import { ReactComponent }, that's a named import that doesn't exist — TS2614. Its Quick Fix logic looks for plausible alternative imports from the same module, finds the default export, and suggests import Logo from "./logo.svg". From TS's seat, that's a clean suggestion. It makes the program type-check.

TS is right about types and wrong about runtime semantics. It has no way to be otherwise.

The LLM. Trained on a corpus that is, overwhelmingly, "code that compiles." Pre-v4 vite-plugin-svgr exported ReactComponent as a named member — millions of training tokens still say so. v4 docs exist but they're a thin slice of the data. When tsc says "use the default import instead," the LLM agrees. It would have suggested the same fix on its own.

The LLM is not being lazy. It is doing exactly what its training distribution and the compiler's hint tell it to do. They both happen to be wrong.

The fix is not the LLM. The fix is the prompt context.

tsfix reads your package.json on every Layer-2 invocation. If it sees vite-plugin-svgr at v4 or higher in your dependencies, it injects this into the system prompt headline, before the model sees the errored file:

### library-migrations
- vite-plugin-svgr: v4 requires the `?react` query suffix to import an SVG
  as a React component. `import Logo from "./logo.svg"` returns the asset URL.

### task
Library migration: vite-plugin-svgr

That's it. No fine-tuning, no agent loop, no retrieval pipeline. A registry lookup against package.json and four lines of prompt.

On our benchmark, the ?react-migration case goes from 0/3 to 3/3 with this change. The model already knew about ?react; it just needed permission to override tsc's hint.

The same shape works for other libraries whose major bumps generate confidently wrong tsc fixes: next@15 (params and searchParams are now Promises and must be awaited), ai@v3/v6 (generateText API rewrite), drizzle-orm (parameterized template literals, not string concat).

Why prompt headline, not "more context"

The first version of this lived in MendContext.featureSpecText — a freeform Markdown section the model would see somewhere in the middle of the prompt. It did approximately nothing. The model still followed tsc's quick-fix.

Moving the same two sentences to the headline taskDescription — the first thing after the system instructions and before the file content — flipped the result. Same content, different position, opposite outcome.

This is consistent with what we know about long-context attention falloff and how Claude in particular interprets the "task" framing: the model treats the headline as what it's actually being asked to do and weights the rest of the prompt against it. "Library migration: vite-plugin-svgr" is read as "the user knows about this migration; whatever quick-fix tsc is suggesting, the migration is the reason." That single reframing overrides the gravity well of "tsc says X."

The same gap, with worse consequences

The svgr case is the easy version of the failure mode: a crashed dev server is loud, immediate, and obviously broken. Move the same dynamic into security territory and the failure goes quiet.

Case 1 — dangerouslySetInnerHTML as a children-type escape hatch. An LLM is asked to render some user-controlled HTML in a React component. The component signature says children: string, the input is an HTML string, tsc complains. The path of least resistance — and one I've seen models take — is to switch the rendering to dangerouslySetInnerHTML={{ __html: input }}. The error vanishes. The XSS hole opens. Same three-character collision: React's type system correctly warns that an HTML string isn't a React node; tsc enforces it; the LLM picks the dodge that makes the type-checker happy. The runtime-correct fix is to render the text as JSX ({input} auto-escapes) or sanitize via DOMPurify before mounting. The type system has no way to know which of those you wanted.

Case 2 — substituting crypto.subtle.digest for a missing bcrypt import. A repo's package.json lists bcrypt; the source imports it; an LLM-generated refactor accidentally removes the import line. tsc emits Cannot find name 'bcrypt'. Did you mean 'crypto'? — and the LLM dutifully takes the suggestion, switching bcrypt.hash(password, 10) to crypto.subtle.digest("SHA-256", encoder.encode(password)). tsc is happy. The code compiles. An unsalted, un-adaptive SHA-256 of every user password is now shipping to production. Every detail of tsc's reasoning was correct — crypto is the closest in-scope identifier, and SHA-256 does return a digest — but the runtime semantics are catastrophically different from a salted, adaptive-cost password hash.

Both cases are the same shape as svgr, just with stakes that get someone fired instead of a broken page reload. tsc is a static system reasoning about types; the program is a dynamic system that has to actually work. A repair that prioritizes the first over the second is the kind of fix that makes the build green and the security report red.

tsfix added explicit prompt-level rules against both of these (plus as keyof T to silence index-signature errors, and dropping arguments to silence TS2554). They're not magic — the model can still produce a bad fix — but they shift the prior. Across the bench, the cases that exercise these patterns went from 0/3 to 3/3 functional-and-secure.

The bet I'm making

The next moat in LLM coding tools is not on the model side. The frontier-model gap has narrowed to weeks at best; everyone codes against the same three providers. The moat is on the structured-knowledge side — the layer that injects the things a model trained on five years of mixed-version code cannot reliably know: which libraries broke, in which versions, with which migration. Library-migration registries are one form. Framework-version-aware refactoring is another. Security-pattern recognition is a third.

These are unsexy databases of "this is wrong now, do that instead," extended one entry at a time by humans who hit the failure mode and submitted a fix. The first project to ship a registry serious enough to embed into Cursor / Claude Code / Continue.dev / Cline as a sub-component wins the post-generation correctness category. That's the integration that touches every one of those tools' users, every codegen pass, every day. It compounds: every new library entry makes your tool relatively more useful versus every alternative.

We've open-sourced our registry under MIT. It currently knows about vite-plugin-svgr v4, next v15, the Vercel AI SDK v3, and drizzle-orm. Four entries is a starting line, not a finish line. The interesting thing is that adding the fifth, sixth, and hundredth entries is exactly the kind of contribution this codebase is structured to receive — see src/libraryMigrations.ts, the registry-extension guide in CONTRIBUTING.md, and the pinned discussion "Which library should the migration registry cover next?"

Try it

npx @shipispec/tsfix --workspace . --llm

On the first run, Layer 0/1 clears the trivial errors deterministically (typos, missing imports — no LLM, no network, no cost). Layer 2 takes whatever's left, with library hints firing automatically when one of the four currently-registered packages is in your package.json. You'll see a per-error tally, per-iteration token / cost numbers, and either stopReason=fixed or a list of remaining errors. Layer 0/1 by itself needs no API key. Layer 2 needs ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_GENERATIVE_AI_API_KEY — your choice via --llm-provider.

If your stack hits one of the patterns above and tsfix doesn't yet know about it, the registry-suggestion issue template is the fastest path to making sure no one else in your category hits it again.

Links

npm: @shipispec/tsfix
Source: github.com/shipispec/tsfix
Library-migration registry: src/libraryMigrations.ts
"Which library next?" discussion: github.com/owgreen-dev/tsfix/discussions