DEV Community

LingvoBerry profile image LingvoBerry
LingvoBerry for LingvoBerry

Posted on

Your German word list has no ß (and your stats may be lying because of it)

#analytics #data #programming #python

Here is a one-liner that broke a report of ours:

>>> from wordfreq import tokenize
>>> tokenize("Er weiß, dass die Straße groß ist", "de")
['er', 'weiss', 'dass', 'die', 'strasse', 'gross', 'ist']
Enter fullscreen mode Exit fullscreen mode

wordfreq is a good library and this isn't a bug: it looks like ordinary case-folding, since "ß".casefold() gives "ss". It still surprised us, because the output is spelled in a way a German book never spells these words.

How far it goes

We were using version 3.1.1, and the list has no ß in it at all:

>>> from wordfreq import top_n_list, word_frequency
>>> sum("ß" in w for w in top_n_list("de", 50_000))
0
>>> word_frequency("weiß", "de") == word_frequency("weiss", "de")
True
>>> top_n_list("de", 200).index("weiss")
166
Enter fullscreen mode Exit fullscreen mode

So asking about either spelling works. The trouble starts when you take the list out of the library and compare it with text as plain strings, which is what happens the moment you publish a word list or measure coverage against a set.

The symptom

We had a job that measures how much of a book a frequency list covers, and it prints the commonest words the list does not know. For German, the top of that report was:

weiß
saß
ließ
Enter fullscreen mode Exit fullscreen mode

These are some of the most ordinary verbs in the language. If you read the report without thinking, you conclude that frequency lists miss basic German. That is a bug of ours reported as a fact about the language.

Why you can't just put the ß back

The tempting fix is to restore the letter in the list. That can't work, because the fold is lossy:

>>> tokenize("in Maßen, in Massen", "de")
['in', 'massen', 'in', 'massen']
Enter fullscreen mode Exit fullscreen mode

"In Maßen" means in moderation and "in Massen" means in huge quantities. After the fold they are one token, and nothing in the list tells you which one you are holding. The information is gone before you get the data.

So we left the list alone and fold the other side at the moment of comparison:

_MATCH_FOLD = {
    "de": lambda w: w.replace("ß", "ss"),
    "ru": lambda w: w.replace("ё", "е"),
    "uk": lambda w: w.replace("ё", "е"),
}

def fold(word, lang):
    f = _MATCH_FOLD.get(lang)
    return f(word) if f else word
Enter fullscreen mode Exit fullscreen mode

The comparison key changes and the text you show people does not.

Russian is the same bug in a softer form

Russian print usually drops the dots on ё, and the corpus has both spellings as separate entries, with very different weights:

>>> word_frequency("ещё", "ru"), word_frequency("еще", "ru")
(0.000513, 0.00195)
Enter fullscreen mode Exit fullscreen mode

Nothing is missing here, but a book that prints ё gets matched against the rarer entry, and your numbers shift by a factor of about four for one of the commonest words in the language.

Turkish breaks plain lower()

While we were in there, we checked the other side of the comparison, our own lowercasing:

>>> "KIZ".lower()
'kiz'
>>> tokenize("KIZ", "tr")
['kız']
Enter fullscreen mode Exit fullscreen mode

Python's str.lower() doesn't know the language, so the Turkish capital I becomes the wrong letter. kiz is not the word at all. The library gets it right, so a homemade tokenizer in front of it quietly disagrees with it. Our fix is two replacements before lowering:

if lang == "tr":
    text = text.replace("İ", "i").replace("I", "ı")
text = text.lower().replace("̇", "")
Enter fullscreen mode Exit fullscreen mode

What we'd do from the start

Three habits would have caught all of this earlier:

  1. Print the commonest misses, not just the coverage number. A coverage of 94% looks fine whether or not it is wrong. A list of misses starting with weiß does not.
  2. Treat normalisation as part of the data's schema. Write down what the source folds, per language, and apply the same fold to anything you compare with it.
  3. Keep two strings. A comparison key and a display form are different things, and one field can't be both.

If you want the longer version of what these lists leave out, once the spelling is sorted, we wrote it up here: what a frequency list misses.

This article was written with the help of AI. The code comes from our own pipeline, and every output above was run against wordfreq 3.1.1 before publishing.

Top comments (0)