Entity resolution at scale starts with a problem most introductions skip: you can't compare every pair of records. A dataset with 500,000 entities has 125 billion possible pairs. At one millisecond per comparison, exhaustive comparison takes 1,400 days.
Blocking reduces this to a tractable set. You partition records into blocks using one or more keys, and only compare records that share a key. The constraint: your blocking decision determines which true matches you'll never see. A match that falls into different blocks is a miss by design.
Building the entity resolution layer for 2asy.ai — a corporate intelligence knowledge graph for East Asian markets — blocking was where the approach broke down first, in a way that was hard to diagnose.
The Initial Approach: Name Normalization
The first blocking key was a normalized company name: strip punctuation, lowercase, remove common suffixes.
import re
COMMON_SUFFIXES = {
"co., ltd.", "co.,ltd.", "co. ltd.", "ltd.", "inc.",
"corp.", "corporation", "co.", "llc", "plc", "gmbh",
"주식회사", "(주)", "(사)", "유한회사",
"株式会社", "有限会社", "合同会社",
}
def normalize_name(name: str) -> str:
name = name.lower().strip()
name = re.sub(r'[^ws]', '', name)
name = re.sub(r's+', ' ', name)
for suffix in sorted(COMMON_SUFFIXES, key=len, reverse=True):
if name.endswith(suffix):
name = name[:-len(suffix)].strip()
return name
Two records sharing a normalized name go into the same block. Simple, fast.
Where It Failed
The failure mode was Korean and Japanese company names. A company registered as "Samsung Electronics Co., Ltd." in a Korean regulatory filing might appear as "Samsung Electronics" in a Japanese document and "삼성전자" in a Korean tax filing.
After normalization, those three forms produce different blocking keys:
normalize_name("Samsung Electronics Co., Ltd.") # → "samsung electronics"
normalize_name("Samsung Electronics") # → "samsung electronics" ← same block
normalize_name("삼성전자") # → "삼성전자" ← different block!
The Korean script name falls into a separate block. It's never compared. The correct merge never happens.
The underlying issue: a blocking key derived from a single attribute assumes that field is stable across sources. It isn't. In East Asian corporate data, the company name field is the least stable field you have. It varies by language, script, legal suffix convention, transliteration standard, and filing jurisdiction.
Composite Blocking with Multiple Key Strategies
The approach that worked was generating multiple candidate keys per record and forming a candidate pair if they share any key.
from typing import Iterator
def generate_blocking_keys(record: dict) -> Iterator[tuple[str, str]]:
"""Yields (key_type, key_value) pairs for a company record."""
name = record.get("name", "")
normalized = normalize_name(name)
if normalized:
yield ("name_norm", normalized)
entity_type = record.get("entity_type", "")
jurisdiction = record.get("jurisdiction_code", "")
if entity_type and jurisdiction:
yield ("type_jurisdiction", f"{entity_type}_{jurisdiction}")
romanized = romanize(name)
if romanized:
phonetic = soundex(romanized)
yield ("phonetic", phonetic)
reg_num = record.get("registration_number", "")
if reg_num:
yield ("reg_prefix", reg_num[:6])
def build_candidate_pairs(records: list[dict]) -> set[tuple[int, int]]:
"""Return all candidate pairs sharing at least one blocking key."""
key_to_records: dict[tuple, list[int]] = {}
for idx, record in enumerate(records):
for key_type, key_value in generate_blocking_keys(record):
bucket = (key_type, key_value)
key_to_records.setdefault(bucket, []).append(idx)
candidates: set[tuple[int, int]] = set()
for indices in key_to_records.values():
for i in range(len(indices)):
for j in range(i + 1, len(indices)):
a, b = sorted([indices[i], indices[j]])
candidates.add((a, b))
return candidates
A pair of records is a candidate if they share any key. This creates overlap — some pairs get compared more than once — but it eliminates the category of missed merges caused by name form variation.
Measuring the Impact
Adding the phonetic key increased candidate pairs by about 40% while recovering merges we'd been systematically missing.
def evaluate_blocking(records, ground_truth_pairs, blocking_fn):
candidates = blocking_fn(records)
true_matches = set(ground_truth_pairs)
found = candidates & true_matches
missed = true_matches - candidates
recall = len(found) / len(true_matches) if true_matches else 0.0
reduction_ratio = 1.0 - len(candidates) / (len(records) * (len(records) - 1) / 2)
return {
"recall": recall,
"candidate_pairs": len(candidates),
"reduction_ratio": reduction_ratio,
"missed_true_matches": len(missed),
}
# Name-only blocking: recall 0.73, candidate_pairs 42_000, missed 12_500
# Composite blocking: recall 0.94, candidate_pairs 59_000, missed 2_800
In a corporate ownership graph, the cost of a false negative is high. Two nodes for the same company means ownership relationships assigned to one aren't visible when querying through the other. The silent failure is worse than the computational overhead.
Blocking Key Design Doesn't End at Ship
Blocking key design isn't a one-time decision. As source variety increases, keys that worked for the initial source set start missing merges from new sources.
The monitoring approach: sample records that weren't merged, check manually whether they should have been, and track false negative rate over time. When it spikes, it usually means a new source has naming conventions that none of the current keys handle.
def sample_unmerged_for_review(
records: list[dict],
merged_ids: set[int],
sample_size: int = 50,
) -> list[dict]:
unmerged = [r for i, r in enumerate(records) if i not in merged_ids]
by_jurisdiction: dict[str, list[dict]] = {}
for r in unmerged:
jur = r.get("jurisdiction_code", "UNK")
by_jurisdiction.setdefault(jur, []).append(r)
flagged = []
for jur, recs in by_jurisdiction.items():
total_in_jur = sum(1 for r in records if r.get("jurisdiction_code") == jur)
merge_rate = 1.0 - len(recs) / total_in_jur
if merge_rate < 0.3 and total_in_jur > 20:
flagged.extend(recs[:10])
return flagged[:sample_size]
The goal isn't zero false negatives — it's knowing which false negative rate your blocking tradeoffs are producing, and catching when that rate degrades without explanation.
I work on entity resolution and corporate knowledge graphs at er-api.hannune.ai and 2asy.ai. If you're designing blocking keys for multilingual or multi-script data, curious what strategies you've tried.
Top comments (0)