Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs

#entityresolution #knowledgegraph #nlp #python

Cross-lingual entity resolution is where knowledge graphs quietly break down.

I ran into this building er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data.

The problem is deceptively simple: Samsung Electronics appears as four different strings across source documents.

Korean: 삼성전자
English: Samsung Electronics
Japanese: サムスン電子
Chinese: 三星电子

Each string hashes to a different node in the graph. A query for Samsung Electronics misses three-quarters of the data unless you happen to use the exact string form the graph was built with. One real company becomes four phantom entities.

Why String Matching Fails

The instinct is to use fuzzy string matching — Levenshtein distance, cosine similarity on character n-grams. This works for typos and abbreviations within the same script. It fails completely across scripts.

"Samsung Electronics" and "サムスン電子" share no character sequences. No string distance metric will recognize them as the same entity without preprocessing.

Step 1: Transliterate to a Canonical Form

The first step is converting all strings to a common script before comparison.

import jaconv  # Japanese: katakana/hiragana to romaji
from hanja import translate as hanja_to_hangul
from hangul_romanize import Transliterate
from hangul_romanize.rule import RevisedRomanization

transliterator = Transliterate(RevisedRomanization())

def canonical_form(text: str, source_lang: str) -> str:
    """Convert a name to a canonical Latin-ish form for comparison."""
    if source_lang == "ja":
        text = jaconv.kata2hepburn(jaconv.hira2kata(text))
    elif source_lang == "zh":
        text = hanja_to_hangul(text)
        source_lang = "ko"

    if source_lang == "ko":
        text = transliterator.translit(text)

    return text.lower().strip()

After transliteration:

삼성전자 → "samsung jeonja"
Samsung Electronics → "samsung electronics"
サムスン電子 → "samsung denshi"
三星电子 → "삼성전자" → "samsung jeonja"

Not identical, but close enough that a similarity metric can catch them. Transliteration alone collapses a large fraction of duplicates before any similarity computation runs.

Step 2: Blocking Keys

Comparing all n-squared candidate pairs across four languages at scale is not viable. For a corpus of 100k entities across four languages, that is 10 billion pairs.

Blocking reduces candidates to a manageable set by only comparing pairs that share a blocking key — a coarse-grained signature that likely-duplicate entities share.

import re

def blocking_key(name: str, lang: str) -> str:
    """
    Generate a blocking key for grouping candidate pairs.
    Strips legal suffixes, normalizes, takes first 4 chars.
    """
    canonical = canonical_form(name, lang)

    legal_suffixes = [
        r"\b(co|corp|inc|ltd|llc|plc|gmbh|ag|sa|bv)\b\.?",
        r"\b(주식회사|kabushiki kaisha|kk|株式会社|有限公司)\b",
    ]
    for pattern in legal_suffixes:
        canonical = re.sub(pattern, "", canonical, flags=re.IGNORECASE)

    canonical = re.sub(r"[^a-z0-9]", "", canonical)
    return canonical[:4]

All entities that share the same blocking key enter the same candidate pool for pairwise comparison. Entities in different blocks never get compared.

Step 3: Per-Language-Pair Similarity Thresholds

Once you have candidate pairs within a block, you need a similarity score. Splink's EM estimator works well here — it learns match probabilities from the data without requiring fully labeled examples.

But the threshold for what counts as a match is not the same across language pairs.

Korean-English pairs tolerate more abbreviation variance. "Samsung C&T" vs. "Samsung C&T Corporation" — same entity, different abbreviation patterns. A Korean-English threshold of 0.75 captures this without over-merging.

Japanese-Chinese pairs are tighter. The transliteration paths for Japanese and Chinese overlap less cleanly than Korean-Latin paths. A threshold of 0.85+ is safer here.

LANG_PAIR_THRESHOLDS = {
    ("ko", "en"): 0.75,
    ("ko", "ja"): 0.80,
    ("ko", "zh"): 0.82,
    ("en", "ja"): 0.78,
    ("en", "zh"): 0.80,
    ("ja", "zh"): 0.85,
}

def threshold_for_pair(lang_a: str, lang_b: str) -> float:
    key = tuple(sorted([lang_a, lang_b]))
    return LANG_PAIR_THRESHOLDS.get(key, 0.80)

The Hardest Part: Labeled Training Data

Splink's EM estimator can run unsupervised, but performance is better with labeled examples.

For Korean-English, there is labeled data available from Korean financial filings — companies are required to report both the Korean and English names in disclosure documents. A financial data scraper gives you positive pairs cheaply.

For Korean-Chinese and Japanese-Chinese, there is almost no open labeled data. What actually worked:

Generating negative pairs by sampling clearly distinct companies from different industry sectors — any two companies with no overlapping industry codes and no name similarity are almost certainly different entities.
Bootstrapping positive pairs from a subset of companies with known cross-lingual identifiers (Dun & Bradstreet DUNS numbers, Bloomberg tickers) where the same company appears in all four language corpora.
Manual annotation for the ambiguous middle tier — about 2,000 pairs reviewed by a bilingual Korean-Chinese speaker.

It is not elegant. But the EM estimator is sensitive enough to training data quality that this effort pays back in precision.

The Result: The Graph Shrinks

After running cross-lingual ER on the initial corpus:

Node count decreased by approximately 18%. Four phantom nodes collapsing to one real node, repeated across the corpus.
Graph RAG retrieval precision for cross-lingual queries improved significantly. Queries for a company in English now correctly retrieve documents from Korean, Japanese, and Chinese sources.
False positive rate stayed low because blocking plus per-language-pair thresholds prevented over-merging.

The graph shrinks after you apply this. Retrieval quality improves because queries no longer split across phantom duplicates.

Building er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.

Cover: Alina Grubnyak on Unsplash