DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

NFC(NFD(c)) Fails for 1,120 of the 2,061 Characters That Have a Canonical Decomposition

Everyone knows NFC composes, NFD decomposes, and the two undo each other.

Over the 2,061 characters that actually have a canonical decomposition, NFC(NFD(c)) === c holds for 941 and fails for 1,12054.34%, more than half.

Check any character: https://dev48.infy.uk/solve/day76-unicode-normalization.html

The page implements UAX #15, not a wrapper

Recursive canonical and compatibility decomposition, the canonical ordering algorithm, the derivation of Full_Composition_Exclusion, the blocked-composition rule and the Hangul arithmetic — written from the raw UCD fields rather than calling out to a library. Otherwise it would be testing the library's opinion rather than the standard.

Two ways it fails

It declines to recompose. U+0958 DEVANAGARI LETTER QA is one assigned character. NFD turns it into two, and NFC will never put it back — it is on the composition-exclusion list, so the round trip is lossy by design.

Or it hands back a different character. For 1,035 of the failures NFC does not merely decline:

U+212B  ANGSTROM SIGN
   NFD -> U+0041 U+030A
   NFC -> U+00C5  LATIN CAPITAL LETTER A WITH RING ABOVE
Enter fullscreen mode Exit fullscreen mode

Same rendering. Different code point. Your string went in as one character and came out as another, and every equality check downstream now depends on which normalisation ran last.

What to do with that

Normalise once, at the boundary, and compare normalised forms — never assume a round trip is identity. If you are storing a key, store the normalised form; if you are comparing user input to it, normalise the input the same way. The failure mode is not corruption, it is two spellings of the same text that never compare equal.

17,581 verifier assertions, 0 failures. Vanilla JavaScript, one file, no build step.

Top comments (0)