DEV Community

Cover image for The normalization step that breaks before your ER model does
Tae Kim
Tae Kim

Posted on Originally published at hannune.ai

The normalization step that breaks before your ER model does

#entityresolution #dataengineering #nlp #python

I spent three weeks tuning near-threshold match scores before realizing the pairs I was most worried about weren't getting bad scores. They weren't generating candidate pairs at all.

The thing I thought would fix it didn't. My first guess was blocking key design. I'd read enough about the invisibility of blocking errors that I figured the issue had to be there. I checked the blocking key function, checked the bucket sizes, didn't see anything obviously wrong. The pairs I was looking for were just absent.

Then I added a simple audit: take 30 pairs that should match, print the blocking keys side by side. First run was Japanese companies from a securities exchange I'd added a few weeks before. Out of 30 pairs, 11 weren't generating matching keys.

The exchange uses full-width ASCII characters in company names. ＡＢＣ株式会社 uses full-width Latin letters (U+FF21 and so on), not regular ASCII. I hadn't known this was a thing. Ａ and A are different code points, so the normalized name hashes to a different key.

import unicodedata
name = unicodedata.normalize("NFKC", name)
Enter fullscreen mode Exit fullscreen mode

NFKC handles this. I ran the audit again, got 30 of 30. Then I ran the same audit on Korean data from a corporate registry and found a different problem.

Korean company names have the entity type in multiple forms: 주식회사 ABC, ABC(주), sometimes just ABC㈜ where ㈜ is a single precomposed Unicode character at U+3378 in the CJK Compatibility block. My stripping code handled 주식회사 and (주). It didn't handle ㈜ because that's not a sequence of three characters; it's one codepoint that looks like a parenthetical when rendered.

KOREAN_LEGAL_FORMS = [
    r"주식회사",
    r"유한회사",
    r"\(주\)",
    r"㈜",   # U+3378, precomposed, won't match the spelled-out pattern
    r"\(유\)",
    r"㈔",   # U+3354
]
Enter fullscreen mode Exit fullscreen mode

Japanese has ㈱ (U+3231) and ㈲ (U+3232) the same way. I found those in the same audit session.

The ordering thing tripped me up a bit. My thought was to run NFKC first and that would expand the precomposed symbols so the stripping patterns would match them. It doesn't work that way. NFKC doesn't decompose those particular codepoints; they pass through intact. So the precomposed forms have to be in the stripping list explicitly. Running NFKC before stripping is still right because it handles the full-width ASCII, but it doesn't help with the precomposed legal symbols.

There's still a category I don't have a clean answer for. Some Japanese company names use older traditional character forms that predate postwar simplification. 國 and 国 refer to the same character. NFKC handles some variant pairs but not all. I've been adding them manually to a lookup table when they turn up in audits. That's obviously not comprehensive. At some point I should probably run a dedicated audit against a list of the known variant pairs, but I haven't done it, partly because I'm not sure how to get that list and partly because the other normalization issues got fixed first and this one fell off the queue.

The reason these are hard to catch is that they don't show up in any metric unless you build the blocking recall audit yourself.

I build er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.

Top comments (0)