NFC and NFKC Normalization: What Actually Differs

#ai #machinelearning #llm #programming

NFC and NFKC differ by one letter and by an enormous amount of behaviour. One of them is guaranteed to preserve the meaning of your text; the other is guaranteed not to. Picking the wrong one for storage corrupts data in ways that are impossible to reverse.

Two axes, four forms

The four normalization forms defined in Unicode Standard Annex #15 are two binary choices, not four unrelated options.

Composed or decomposed — the C/D axis. Does é end up as one code point or as e plus a combining acute?
Canonical or compatibility — the presence or absence of the K. Do you only unify sequences that are the same character, or also fold characters that are merely related?

So NFC is composed-canonical, NFD is decomposed-canonical, NFKC is composed-compatibility, NFKD is decomposed-compatibility. The C/D choice is a representation detail and is fully reversible in the sense that both forms mean the same thing. The K choice is not: it throws information away on purpose, and no normalization brings it back.

Canonical: the same character, written twice

Canonical equivalence covers sequences that Unicode considers indistinguishable in meaning. The classic case:

"café" typed on a Mac      : 63 61 66 65 CC 81      (c a f e + U+0301)
"café" typed on Windows    : 63 61 66 C3 A9         (c a f + U+00E9)

NFC of both  -> 63 61 66 C3 A9      identical
NFD of both  -> 63 61 66 65 CC 81   identical

Two byte sequences, one character, and a naive equality check says they are different strings — which they are, byte for byte. This is the single most common cause of “the record is definitely in the database but the search returns nothing”, and normalizing both sides to the same canonical form is the entire fix. The mechanism, and which platforms produce which form, is the subject of precomposed against combining characters.

Canonical normalization also fixes up the order of multiple combining marks. Marks carry a combining class, and canonical ordering sorts them into a fixed sequence, so a base with an acute and a cedilla applied in either typing order normalizes to one representation. That matters enormously for Vietnamese and for Indic scripts, where two and three marks on one base are ordinary.

One further guarantee is worth relying on: the set of canonical decompositions is frozen by Unicode’s normalization stability policy. A string normalized to NFC under one Unicode version stays normalized under every later one. Very little else in internationalisation offers that.

Compatibility: a different character that looks similar

Compatibility mappings are the ones with a <tag> in the Unicode character database, and they cover characters that exist for round-trip compatibility with older encodings or for typographic reasons. Applying them is lossy by design. Here is a labelled string run through both composed forms:

input : "ﬁle ① Ａ² 12㎏ Ⅻ non\u00A0breaking"

NFC   : "ﬁle ① Ａ² 12㎏ Ⅻ non\u00A0breaking"     unchanged
NFKC  : "file 1 A2 12kg XII non breaking"

  U+FB01 LATIN SMALL LIGATURE FI  -> "fi"    length 1 -> 2
  U+2460 CIRCLED DIGIT ONE        -> "1"
  U+FF21 FULLWIDTH LATIN A        -> "A"
  U+00B2 SUPERSCRIPT TWO          -> "2"     "A²" and "A2" now equal
  U+338F SQUARE KG                -> "kg"    length 1 -> 2
  U+216B ROMAN NUMERAL TWELVE     -> "XII"   length 1 -> 3
  U+00A0 NO-BREAK SPACE           -> U+0020  a space, not a NBSP

Read the last two lines carefully, because they are the ones that bite. NFKC turns a superscript into a digit, which means x² and x2 become the same string — fine for a search key, catastrophic for a chemistry or maths corpus. And NFKC converts a non-breaking space into an ordinary space, silently destroying a typographic decision somebody made deliberately.

What NFKC does not do is the thing people most often expect. It does not strip accents: é stays é, because an acute accent is not a compatibility artefact. It does not expand German ß to ss. It does not map the French ligature œ (U+0153) to oe, because that character has no decomposition at all — unlike ﬁ, which does. If you need those, you need explicit rules, which is exactly why diacritic-insensitive search is a separate piece of work.

Which one to store and which one to index

The rule that survives contact with production is simple and rarely stated:

Store NFC. It is canonical, so nothing is lost; it is the shortest of the four for most text; and it is the form the web platform assumes. Normalize on write, at the boundary where input enters the system, so that everything downstream can assume it.
Index NFKC (or NFKC plus case folding). A search key wants Ａ and A to match, and it does not care that a non-breaking space became a space. Compute it as a derived column or a separate analyser chain — never in place of the original.
Compare identifiers with NFKC. Usernames, file names, package names and domain-adjacent identifiers are exactly where visually confusable forms are a security problem rather than an inconvenience, and the Unicode identifier and security annexes recommend compatibility normalization for this reason.

The failure to avoid is storing NFKC as the canonical record. Someone named Ⅻ, a product code containing a fullwidth character, a document with meaningful superscripts — once the compatibility mapping has been applied to the stored value, the original is gone.

Telling which form you already have

Before changing anything, find out what is in the data, because the answer is almost never “one form”. A table populated over several years by a web form, a mobile client, a CSV import and an admin tool will contain all of them, and the mix is what makes the symptoms intermittent.

# Python: how many rows are not already NFC?
import unicodedata
def is_nfc(s): return unicodedata.is_normalized("NFC", s)

# The check that actually finds the problem: rows that are equal
# after normalization but not before. These are your duplicate keys.
seen = {}
for row in rows:
    k = unicodedata.normalize("NFC", row.name)
    seen.setdefault(k, []).append(row.name)
collisions = [v for v in seen.values() if len(set(v)) > 1]

-- PostgreSQL 13+, the same question in SQL
SELECT count(*) FILTER (WHERE NOT name IS NORMALIZED)  AS not_nfc,
       count(*)                                        AS total
FROM customers;

-- Rows that collide only after normalization:
SELECT normalize(name, NFC) AS key, count(DISTINCT name)
FROM customers GROUP BY 1 HAVING count(DISTINCT name) > 1;

The second query is the one worth running first. A count of un-normalized rows tells you the scale of the cleanup; a list of post-normalization collisions tells you which specific records your users have already been unable to find, and whether a unique constraint you thought was enforcing something has been letting duplicates through. Fixing the second set usually requires a decision per row rather than an UPDATE, so it is better to know before you start than after.

A related check for the ingest side: log the normalization form of incoming values for a week before you enforce anything. If one client is the source of every decomposed string, fixing that client is cheaper and less risky than normalizing everything forever — though you should generally do both, because the boundary normalization is what stops the problem returning.

Four traps

Normalization changes string length. Any code that stores offsets — search highlighting, annotation spans, diff positions — must normalize before computing offsets, or the offsets point at the wrong characters. A VARCHAR(20) that fits before NFKC may not fit after.
NFC is not idempotent across concatenation. Two NFC strings joined together are not necessarily NFC: if the first ends in a base character and the second begins with a combining mark, they compose across the join. Normalize after concatenating, not only before.
Databases do not normalize for you. PostgreSQL exposes a normalize() function and an IS NORMALIZED predicate from version 13, and you have to call them. Accent-insensitive collations in other engines compare code points and do not normalize, so a decomposed string still fails to match a precomposed one no matter how insensitive the collation is.
Normalization is not case folding. They are separate operations with separate specifications, and NFKC does not change case. The combined operation you usually want for a search key is NFKC then case folding, in that order.