If you've ever built anything with Japanese location data — a travel app, a map layer, a delivery service — there's a coin-flip chance the English names you're showing users are wrong. Not "slightly off" wrong. Different words wrong.
I know because I checked. All 53,502 records of a nationwide restaurant dataset, evaluated independently by two frontier AI models. 50.4% of the raw romanizations were errors.
Here's how I found out, what the errors look like, and why the standard tooling can't fix this.
The problem: Japanese names aren't text
Every romanization library — pykakasi, kakasi, cutlet, all of them — treats Japanese names as text to be transliterated. But Japanese place and shop names aren't text. They're cultural artifacts: classical Chinese readings, irregular place-name pronunciations, katakana loan-words that need to be translated back to English, historical terms that predate modern Japanese.
When you feed those to a dictionary-based library, you don't get subtle errors. You get completely different words:
| Input | pykakasi (industry standard) | Correct |
|---|---|---|
| 光華園 | Hikari Hanazono ("a flower garden") | Kokaen — it's a ramen shop |
| 岩洞湖レストハウス | Iwa Hora Mizumi ("rock hole lake") | Gandoko Rest House |
| 満洲里 | Manshu Sato (a surname appeared from nowhere) | Manshuri |
| 国際展示場駅 | Ariake (a different station entirely) | Kokusai-tenjijo |
None of these are edge cases I hunted for. They fell out of the first batch I looked at.
The failure modes are structural, not random:
- Misreads: 光華 has multiple valid readings; picking the right one (Koka, not Hikari-hana) requires knowing it's a shop name in a Chinese-cuisine naming tradition.
- Oversplitting: 石垣島 (Ishigaki Island) becomes "Ishi Gaki Shima" — three tokens where the name is one unit.
- Katakana round-trips: カントリークラブ is the English word "Country Club" written in katakana. Transliterating it back as "Kantori Kurabu" is technically faithful and completely useless.
- Place-name readings: Japanese place names routinely use readings that no dictionary derives from the characters.
A dictionary can't fix this, because the correct answer isn't in the characters. It's in the culture around them.
The method: two AIs, zero collaboration
Modern LLMs actually can do this — they've absorbed enough of the cultural context to read 光華園 as Kokaen. But "AI can do it" isn't a data-quality guarantee. AIs are confidently wrong all the time.
So I didn't trust one AI. I ran two.
- Model A: Claude Sonnet 5
- Model B: Claude Opus 4.8
- Same 53,502 records, same prompt, evaluated completely independently. Neither model saw the other's output.
- 51,967 records were shared between both runs (the remainder were ASCII names or override-excluded records).
Then I compared every single verdict. If two models with different capabilities independently agree that a romanization is wrong — and agree on the fix — that's a much stronger signal than either alone.
The results
On the 51,967 shared records:
| Agreement class | Share |
|---|---|
| Both correct (no error found) | 43.0% |
| Both caught the error, same fix | 26.2% |
| Opus only — Sonnet missed it | 14.5% |
| Disagreement → human review | 9.9% |
| Sonnet only (false positive) | 6.4% |
| Full agreement (verdict + fix) | 69.2% |
Two numbers in that table matter more than the rest:
Opus caught 7,548 errors that Sonnet missed. And Sonnet flagged 3,333 records that were actually fine. The gap between "good-enough AI" and the best available AI isn't academic — it's 7,548 wrong restaurant names that would have shipped in production. I used Opus as the authority and applied all 26,975 fixes.
The 26,975 errors, classified:
| Error type | Count | Example |
|---|---|---|
| misread | 9,027 | 光華 → Hikari |
| oversplit | 8,866 | 石垣島 → Ishi Gaki Shima |
| katakana | 3,964 | カントリークラブ → Kantori Kurabu |
| mixup | 2,777 | another shop's data merged in |
| other | 1,240 | — |
| placename | 781 | 岩洞湖 → Iwa Hora Mizumi |
| chinese | 282 | 満洲里 → Manshu Sato |
| okurigana | 35 | — |
Even the "official" data was wrong
The obvious objection: "sure, scraped restaurant names are messy, but official sources are clean, right?"
I audited the 8,650 train stations in my station master — built from official transit sources — with the same pipeline. 261 stations (3.0%) had romanization errors. 227 were corrected and applied. Examples: 国際展示場 was listed as "Ariake" (a different station), 後楽園 as "Kasuga" (also a different station).
No dataset escaped the audit. If the official transit data is 3% wrong, assume everything else is worse.
Takeaways if you work with Japanese data
- Don't trust library romanization for names. It's fine for running text; it fails structurally on proper nouns, which is exactly what location data is made of.
- Don't trust a single AI either. A 6.4% false-positive rate (Sonnet, in my run) means blindly applying one model's fixes introduces thousands of new errors.
- Dual independent evaluation is cheap and brutal. Two models, same prompt, compare verdicts. Agreement gives you confidence; disagreement gives you a prioritized human-review queue (9.9% in my case — very manageable).
- Audit your "clean" sources. 3% error in official transit data surprised me more than 50% in raw data.
Where this data lives now
This audit was part of building Gachi-Ramen — a ramen-shop location API for Japan (62,173 shops, all 47 prefectures, weekly liveness checks). Every one of the 26,975 fixes is live in it. The full technical story, including how we verify shops are still open, is here: the story page.
There's a free sample of 1,000 real records on the site if you want to poke at the data yourself — no signup.
Questions about the eval setup, the prompt design, or Japanese romanization pain in general — happy to answer in the comments.
Top comments (0)