If you have ever built search for a language with diacritics, you have probably reached for this:
const fold = s => s.normalize('NFD').replace(/\p{M}/gu, '').toLowerCase();
Decompose, strip the combining marks, lowercase. It is the standard trick and it works beautifully for French, Spanish, Portuguese.
For Icelandic it silently does half the job, and the half it misses is the half that matters.
What NFD actually does
NFD decomposes precomposed characters into a base letter plus combining marks. á becomes a + U+0301, and stripping the mark leaves a. Same for é í ó ú ý ö.
Now try it on these:
fold('þjónusta') // 'þjonusta' — þ survived
fold('góður') // 'goður' — ð survived
fold('æfing') // 'æfing' — æ survived
þ (thorn), ð (eth) and æ (ash) are not precomposed characters. They are distinct letters with no decomposition mapping. NFD leaves them exactly where they were, and your "normalized" index still contains characters that no one is going to type on a US keyboard.
Why that is a real problem and not a cosmetic one
Icelanders on foreign keyboard layouts substitute. þ becomes th. ð becomes d. æ becomes ae. This is not a rare edge case — it is what the query log looks like.
So a user searching thjonusta finds nothing, while your index sits there holding þjonusta, which is neither the original nor what anyone typed.
In a large language you get rescued by the search engine's own fuzzy matching and a mountain of query data. In a language with a few hundred thousand speakers there is no such safety net. Nobody has trained a "did you mean" model on your queries. If your matching layer does not handle it, it does not get handled.
The fix
An explicit map, applied to both sides:
const FOLD = {
'á':'a', 'é':'e', 'í':'i', 'ó':'o', 'ú':'u', 'ý':'y',
'ð':'d', 'þ':'th', 'æ':'ae', 'ö':'o'
};
const fold = s =>
s.toLowerCase().replace(/[áéíóúýðþæö]/g, c => FOLD[c]);
Lowercase first so uppercase forms are covered by the same map.
Three things worth knowing before you ship this.
þ → th changes the string length.** One character becomes two. If you are highlighting matches by index into the original string, your offsets are now wrong. Either fold a copy and keep a position map, or highlight on the unfolded text with a separate pass.
Fold the query the same way, at the same time.** The invariant is that both sides pass through identical code. The moment the index is folded at build time and the query is folded by a slightly different function at request time, you get failures that are miserable to reproduce.
Do not fold what you display. This belongs in the matching layer only. Nothing the reader sees should lose its diacritics — góður and goður are not the same word, and showing the second one looks broken to a native speaker.
Sorting is a separate problem
While you are here: Icelandic alphabetical order puts þ, æ and ö at the end, after z. A naive .sort() on code points gets this wrong, and so does folding, since folding has already thrown the information away.
list.sort((a, b) => a.localeCompare(b, 'is'));
Use the locale-aware comparison on the original strings. Fold for matching, compare for ordering, and keep the two apart.
Where this came from
I wrote this for a retrieval index on an Icelandic site I run, where the whole search layer is one plain-text file and a Cloudflare Worker — no embeddings, no vector database. The diacritic folding took more iterations than any other part of it, which is why it seemed worth writing down.
If you work in a language with letters outside the Latin-1 comfort zone, the general lesson is probably the same: check what your normalization actually does to your alphabet before trusting it. The standard recipe was written for a different set of letters.
Top comments (0)