<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Svipan</title>
    <description>The latest articles on DEV Community by Svipan (@svipan).</description>
    <link>https://dev.to/svipan</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4258582%2F41ad9b72-f47f-46ae-a0b3-5d25157ca70c.png</url>
      <title>DEV Community: Svipan</title>
      <link>https://dev.to/svipan</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/svipan"/>
    <language>en</language>
    <item>
      <title>Why NFD normalization quietly breaks Icelandic search</title>
      <dc:creator>Svipan</dc:creator>
      <pubDate>Sun, 22 Nov 2026 10:22:16 +0000</pubDate>
      <link>https://dev.to/svipan/why-nfd-normalization-quietly-breaks-icelandic-search-19ig</link>
      <guid>https://dev.to/svipan/why-nfd-normalization-quietly-breaks-icelandic-search-19ig</guid>
      <description>&lt;p&gt;If you have ever built search for a language with diacritics, you have probably reached for this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;fold&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normalize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'NFD'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;\&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="no"&gt;M&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;gu&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toLowerCase&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Decompose, strip the combining marks, lowercase. It is the standard trick and it works beautifully for French, Spanish, Portuguese.&lt;/p&gt;

&lt;p&gt;For Icelandic it silently does half the job, and the half it misses is the half that matters.&lt;/p&gt;

&lt;p&gt;What NFD actually does&lt;/p&gt;

&lt;p&gt;NFD decomposes precomposed characters into a base letter plus combining marks. &lt;code&gt;á&lt;/code&gt; becomes &lt;code&gt;a&lt;/code&gt; + U+0301, and stripping the mark leaves &lt;code&gt;a&lt;/code&gt;. Same for &lt;code&gt;é í ó ú ý ö&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Now try it on these:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;fold&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'þjónusta'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;// 'þjonusta'  — þ survived&lt;/span&gt;
&lt;span class="nf"&gt;fold&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'góður'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;     &lt;span class="c1"&gt;// 'goður'     — ð survived&lt;/span&gt;
&lt;span class="nf"&gt;fold&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'æfing'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;     &lt;span class="c1"&gt;// 'æfing'     — æ survived&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;þ&lt;/code&gt; (thorn), &lt;code&gt;ð&lt;/code&gt; (eth) and &lt;code&gt;æ&lt;/code&gt; (ash) are not precomposed characters. They are distinct letters with no decomposition mapping. NFD leaves them exactly where they were, and your "normalized" index still contains characters that no one is going to type on a US keyboard.&lt;/p&gt;

&lt;p&gt;Why that is a real problem and not a cosmetic one&lt;/p&gt;

&lt;p&gt;Icelanders on foreign keyboard layouts substitute. &lt;code&gt;þ&lt;/code&gt; becomes &lt;code&gt;th&lt;/code&gt;. &lt;code&gt;ð&lt;/code&gt; becomes &lt;code&gt;d&lt;/code&gt;. &lt;code&gt;æ&lt;/code&gt; becomes &lt;code&gt;ae&lt;/code&gt;. This is not a rare edge case — it is what the query log looks like.&lt;/p&gt;

&lt;p&gt;So a user searching &lt;code&gt;thjonusta&lt;/code&gt; finds nothing, while your index sits there holding &lt;code&gt;þjonusta&lt;/code&gt;, which is neither the original nor what anyone typed.&lt;/p&gt;

&lt;p&gt;In a large language you get rescued by the search engine's own fuzzy matching and a mountain of query data. In a language with a few hundred thousand speakers there is no such safety net. Nobody has trained a "did you mean" model on your queries. If your matching layer does not handle it, it does not get handled.&lt;/p&gt;

&lt;p&gt;The fix&lt;/p&gt;

&lt;p&gt;An explicit map, applied to both sides:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="no"&gt;FOLD&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="s1"&gt;'á'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'a'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'é'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'e'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'í'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'i'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'ó'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'o'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'ú'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'u'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'ý'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'y'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s1"&gt;'ð'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'d'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'þ'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'th'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'æ'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'ae'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'ö'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;'o'&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;fold&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toLowerCase&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;áéíóúýðþæö&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="no"&gt;FOLD&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lowercase first so uppercase forms are covered by the same map.&lt;/p&gt;

&lt;p&gt;Three things worth knowing before you ship this.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;þ&lt;/code&gt; → &lt;code&gt;th&lt;/code&gt; changes the string length.** One character becomes two. If you are highlighting matches by index into the original string, your offsets are now wrong. Either fold a copy and keep a position map, or highlight on the unfolded text with a separate pass.&lt;/p&gt;

&lt;p&gt;Fold the query the same way, at the same time.** The invariant is that both sides pass through identical code. The moment the index is folded at build time and the query is folded by a slightly different function at request time, you get failures that are miserable to reproduce.&lt;/p&gt;

&lt;p&gt;Do not fold what you display. This belongs in the matching layer only. Nothing the reader sees should lose its diacritics — &lt;code&gt;góður&lt;/code&gt; and &lt;code&gt;goður&lt;/code&gt; are not the same word, and showing the second one looks broken to a native speaker.&lt;/p&gt;

&lt;p&gt;Sorting is a separate problem&lt;/p&gt;

&lt;p&gt;While you are here: Icelandic alphabetical order puts &lt;code&gt;þ&lt;/code&gt;, &lt;code&gt;æ&lt;/code&gt; and &lt;code&gt;ö&lt;/code&gt; at the &lt;em&gt;end&lt;/em&gt;, after &lt;code&gt;z&lt;/code&gt;. A naive &lt;code&gt;.sort()&lt;/code&gt; on code points gets this wrong, and so does folding, since folding has already thrown the information away.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="n"&gt;list&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;localeCompare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'is'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use the locale-aware comparison on the original strings. Fold for matching, compare for ordering, and keep the two apart.&lt;/p&gt;

&lt;p&gt;Where this came from&lt;/p&gt;

&lt;p&gt;I wrote this for a retrieval index on &lt;a href="https://svipan.is" rel="noopener noreferrer"&gt;an Icelandic site I run&lt;/a&gt;, where the whole search layer is one plain-text file and a Cloudflare Worker — no embeddings, no vector database. The diacritic folding took more iterations than any other part of it, which is why it seemed worth writing down.&lt;/p&gt;

&lt;p&gt;If you work in a language with letters outside the Latin-1 comfort zone, the general lesson is probably the same: check what your normalization actually does to &lt;em&gt;your&lt;/em&gt; alphabet before trusting it. The standard recipe was written for a different set of letters.&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>search</category>
      <category>i18n</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
