Stripping French Accents for Search Without Losing Meaning

#ai #machinelearning #llm #tutorial

The standard advice for French search is to run every string through an accent fold so that eleve finds élève. It works, and it also quietly merges où with ou, tâche with tache, and maïs with mais — three pairs of unrelated words.

Pairs that differ only by an accent

These are not edge cases dug out of a dictionary. They are among the most frequent words in the language, and a fold makes each pair indistinguishable:

ou    / où       or            / where
a     / à        has           / to, at
mais  / maïs     but           / maize, corn
sur   / sûr      on            / sure
tache / tâche    stain         / task
des   / dès      some          / from (the moment)
jeune / jeûne    young         / a fast (abstaining)
cote  / côte     rating, quota / coast, rib
cote  / côté     rating        / side
pecheur / pêcheur / pécheur    fisherman / sinner

The last group is the one that shows the shape of the problem best. cote, côte, coté and côté are four distinct words, and a blanket fold collapses all four into one index term. A user searching for côte (coast) in a travel corpus gets documents about ratings and about sides, ranked identically, with no way to express that they meant the one with the circumflex.

Note also what the fold does not destroy: the cedilla. ç folds to c, and there is no French pair that is distinguished only by a cedilla, because the cedilla is a pronunciation rule rather than a lexical distinction. The accents differ in how much information they carry, and the aigu, grave, circonflexe and tréma all carry more than the cédille does.

Why you have to fold anyway

Given that, the tempting conclusion is not to fold. That is worse, for a reason that has nothing to do with linguistics: French users routinely type French without accents.

Keyboards. A French AZERTY keyboard has é è à ù ç as direct keys but requires dead keys for circumflex and tréma and offers no direct capital É. On a phone, on a foreign keyboard, or in a hurry, the accents are dropped.
House style on capitals. Accents on capitals are correct French and are frequently omitted in practice, so ETAT and ÉTAT both appear in real documents — meaning the corpus itself is inconsistent, not just the queries.
Imported and legacy data. Anything that has been through an ASCII-only system arrives stripped, and typically the strip was done character-by-character with no record of what was removed.

So the query side and the document side are each independently inconsistent, which means unfolded exact matching misses a large share of genuine matches. Folding is not optional. What is optional — and what most implementations get wrong — is folding destructively.

The ligatures normalization will not fix

Before building the fold, know what your normalization pass does and does not cover. Most French accented characters decompose cleanly, so NFD followed by dropping combining marks handles them. Two characters do not:

é  U+00E9   NFD -> "e" + U+0301   combining mark, strips cleanly
ç  U+00E7   NFD -> "c" + U+0327   combining mark, strips cleanly

œ  U+0153   NFD -> "œ"            NO decomposition at all
æ  U+00E6   NFD -> "æ"            NO decomposition at all

so:  "cœur" -> strip marks -> "cœur"    the fold did nothing
     "sœur" -> strip marks -> "sœur"
     but real users type    "coeur", "soeur"

œ and æ are independent letters in Unicode with no decomposition mapping, canonical or compatibility — unlike the ﬁ ligature, which does have one, as described in the NFC and NFKC page. So neither NFD-and-strip nor NFKC will turn cœur into coeur. You need an explicit two-character mapping.

Lucene’s asciifolding filter does handle these, mapping œ to oe and æ to ae from its own table rather than from Unicode decompositions, which is one good reason to use the platform filter instead of a hand-written strip loop. The Elasticsearch documentation for the asciifolding token filter also documents the option that makes the whole strategy below possible.

Fold for recall, rank on the original

The design that keeps both properties is to treat the folded form as an_additional_ way to reach a document, never as a replacement for the real one. Concretely, every token is indexed twice: once as written, once folded. A query is likewise expanded to both forms. And the scoring is arranged so that a match on the written form always outranks a match on the folded form.

Then a search for côte returns coast documents first, because they matched the accented term, and rating documents further down, because they only matched the fold. A search for cote typed without an accent returns everything, ordered by whatever other signals exist. Nothing is lost and nothing is unreachable — which is the property a destructive fold cannot offer at any threshold. The same two-field arrangement is what keeps accented names in a database both findable and correctly spelled.

In Elasticsearch this is asciifolding with preserve_original set to true, or a multi-field with two analysers and a boosted multi_match. In PostgreSQL it is a second column, or the unaccent extension used inside a ts_vector configuration while the raw column stays for exact predicates and ranking.

PostgreSQL’s unaccent() is declared STABLE rather than IMMUTABLE, because its behaviour depends on a dictionary that can be changed. That means you cannot use it directly in an expression index; the documented workaround is an IMMUTABLE wrapper function, which you are then responsible for not lying about. See the PostgreSQL unaccent documentation.

Building it

Normalize to NFC on write. Decomposed input from macOS file names or PDF extraction otherwise fails even exact matching, for reasons unrelated to accents.
Define the fold explicitly: NFD, drop characters with a non-zero combining class, recompose, then apply the ligature table (œ→oe, æ→ae, and their capitals), then case fold. Write it as one function used by both the indexer and the query parser, because two implementations will diverge.
Index both forms. Original in the primary field, folded in a secondary field or as a second token at the same position.
Expand the query to both forms and search both fields, with the original field boosted. A factor around 2–3 is a reasonable starting point; the exact value is something to tune against your own click data, not something to copy.
Test on the pairs from the first section. A search for où must not rank ou documents first, and a search for ou must still find où. Those two assertions catch almost every way this can be built wrong.
Handle apostrophes in the same pass. French elision means l’élève and l'élève differ by the apostrophe character (U+2019 against U+0027), and the token should be split so that élève is searchable on its own. That is a tokenizer setting, not a fold, and it is the second most common cause of missing French matches after accents.