Your Vocabulary Coverage Report Needs a Denominator

LingvoBerry — Sat, 24 Oct 2026 13:14:14 +0000

Two vocabulary reports can disagree without either one containing an arithmetic error. They may be counting different things.

Consider this deliberately small passage:

Maya walked to the station. Maya was walking and the children were reading.

Keep the reader's declared vocabulary fixed. Counting printed forms gives 38.46%. Applying a small, explicit normalization table gives 84.62%. Counting distinct normalized items instead of occurrences gives 88.89%.

Those numbers come from the same thirteen tokens. The difference is the contract around the metric.

A complete example you can run

This example uses Python's standard library. The passage and the known-word set are hand-written fixtures. The alias table is an annotation for this passage, rather than a general-purpose lemmatizer.

"""A deliberately small, original English fixture; no production data or NLP model."""
from collections import Counter
import json
import re

TEXT = 'Maya walked to the station. Maya was walking and the children were reading.'
KNOWN = {'walk', 'to', 'the', 'station', 'be', 'and', 'child', 'read'}
ALIASES = {'walked': 'walk', 'walking': 'walk', 'was': 'be', 'were': 'be',
           'children': 'child', 'reading': 'read'}

def coverage(units, known, excluded=frozenset()):
    counts = Counter(unit for unit in units if unit not in excluded)
    hits = sum(count for unit, count in counts.items() if unit in known)
    total = sum(counts.values())
    return hits, total

def percent(pair):
    hits, total = pair
    return 'n/a' if total == 0 else f'{100 * hits / total:.2f}%'

forms = re.findall(r'[a-z]+', TEXT.casefold())
lemmas = [ALIASES.get(word, word) for word in forms]
scores = {
    'form_tokens': coverage(forms, KNOWN),
    'lemma_tokens': coverage(lemmas, KNOWN),
    'lemma_types': coverage(set(lemmas), KNOWN),
    'lemma_tokens_without_declared_name': coverage(lemmas, KNOWN, {'maya'}),
}
assert len(forms) == 13
assert scores['form_tokens'] == (5, 13)
assert scores['lemma_tokens'] == (11, 13)
assert scores['lemma_types'] == (8, 9)
assert scores['lemma_tokens_without_declared_name'] == (11, 11)
assert coverage(['walk', 'walk', 'mystery'], KNOWN) == (2, 3)
assert coverage({'walk', 'mystery'}, KNOWN) == (1, 2)
assert percent(coverage(['maya'], KNOWN, {'maya'})) == 'n/a'
result = {key: {'hits': value[0], 'total': value[1], 'percentage': percent(value)}
          for key, value in scores.items()}
print(json.dumps(result, indent=2))

The output is:

{
  "form_tokens": {
    "hits": 5,
    "total": 13,
    "percentage": "38.46%"
  },
  "lemma_tokens": {
    "hits": 11,
    "total": 13,
    "percentage": "84.62%"
  },
  "lemma_types": {
    "hits": 8,
    "total": 9,
    "percentage": "88.89%"
  },
  "lemma_tokens_without_declared_name": {
    "hits": 11,
    "total": 11,
    "percentage": "100.00%"
  }
}

Python's re documentation describes the matching operation, and Counter supplies the occurrence counts. Here the tokenizer is intentionally limited to runs of ASCII letters. It is not a production tokenizer for multilingual text, contractions, or hyphenated words.

What each percentage answers

Counting contract	Known units	Total units	Result
Printed-form occurrences	5	13	38.46%
Normalized occurrences	11	13	84.62%
Distinct normalized items	8	9	88.89%

The printed-form check recognizes to, the, station, and, and the second the. It does not match walked to the known entry walk.

The alias table maps walked and walking to walk; was and were to be; children to child; and reading to read. That leaves the two occurrences of Maya unmatched: eleven hits out of thirteen occurrences.

The type-based calculation turns the normalized sequence into a set. Repeated occurrences of Maya, walk, the, and be now count once each. Eight of the nine distinct normalized items belong to the known set.

Token coverage asks how much of this passage's running text matches the declared vocabulary. Type coverage asks how much of its distinct normalized vocabulary matches. Neither is automatically a better score; they answer different questions.

A name filter changes the question again

The last row of the JSON output excludes the declared name maya from both numerator and denominator. It returns eleven out of eleven: 100.00%.

That does not establish complete comprehension. It means every remaining item matched after the name was excluded. A reader still has to identify the character, follow the events, and interpret the grammar. Recognizing a lemma also does not prove familiarity with every sense of the word.

Here the name is explicitly annotated. A production system would need a documented policy and a way to evaluate its name detector. Capitalization alone is insufficient: sentence-initial words are capitalized too, and conventions vary across languages.

The empty-input assertion is also intentional. If a filter removes everything, the score is undefined. Returning zero would imply failure; returning one hundred would imply complete coverage. This example returns n/a.

Test the denominator, not just the final percentage

The two assertions with walk, walk, and mystery keep this distinction visible. An occurrence-based report must count two hits out of three. A type-based report must count one out of two. A refactor that silently replaces the sequence with a set would change the question even if the resulting percentage still looked plausible.

For a stored report, keep the components and the policy together:

{
  "unit": "normalized_token",
  "hits": 11,
  "total": 13,
  "tokenizer": "ascii-letter-runs-fixture-v1",
  "normalization": "explicit-fixture-aliases-v1",
  "named_entity_policy": "count",
  "known_vocabulary": "hand-written-fixture-v1"
}

In a real pipeline, also record the source-text version and the vocabulary-list version. Compare percentages only after checking that these contracts match. Otherwise a higher score may reflect a normalization change, a different treatment of names, or a new denominator rather than a change in the reader's vocabulary.

Context and limits

This example was prepared for LingvoBerry's language-learning work. Its public English word list is a concrete vocabulary-list view; the fixture above deliberately uses a separate hand-written set so that the experiment can run without a network request or product data.

The alias table is intentionally narrow. It does not resolve part of speech, ambiguous senses, phrasal verbs, or named entities. The percentages are measurements of this fixture under stated rules, not a CEFR placement result or a validated comprehension threshold.

A word's frequency rank is not its difficulty: grading words for a placement test

LingvoBerry — Fri, 02 Oct 2026 02:56:27 +0000

We run Lingvoberry, a reader where you tap a word in a graded book to save it. To place a new user on a CEFR level we built a test: show a page of a real book, ask about four words on it, and climb up or down a band depending on the answers.

The first version read one or two bands too high. Fixing it taught us that a word's frequency rank is a poor stand-in for how hard the word is. Here is what we measured, on 252 production books.

The first version: grade a word by its rank

We have a frequency list for each of 14 languages and cut it into bands, from A0 up to C1 by rank. The rarer a word, the higher the band, and a page of a C1 book was asked about its rarest words.

It looked reasonable. It was wrong in three different ways.

1. The glossary put easy words on hard pages

Every chapter of our books ends with a small vocabulary box, and we took test words from it. Measured, that box put as many top-900 words on a C1 page as on an A1 one: 29% of every word asked, and 60% of runs asked at least one. So the questions did not get harder as the walk climbed.

The fix: grade each word by its own frequency band and draw the questions out of the page's own prose. A page's prose turned out to be deep enough that the glossary is no longer the admission ticket: a C1 page carries a median of 8 distinct C1-band words.

2. A third of the "hard" words were not hard

We then looked at what a C1 probe could ask. About a third of the candidates were not C1 questions at all, and they fell into a few classes.

Forms of an easier word (20%). Our lists count printed forms, so "blau" is A2 and "blauer" ended up in C1. The fix is to band a word by the best rank of anything it is a form of: the longest prefix that is itself a list word, with a language-specific limit on the tail. German, Dutch and Swedish glue words together, so a long tail there is a compound ("Küstenstadt" is not a form of "Küste"). Turkish stacks four endings, the Slavic languages three.
International words (16%). "Konstruktion" is rare in German text and obvious to anyone who reads a European language. Frequency cannot see that. We compared the fourteen lists against each other: a word whose folded stem also stands in three lists outside its own language family is not a question from B1 up. The family rule matters: without it "город" matched three Slavic lists and was thrown out of the Russian ladder.
Names and fragments. A capitalised word that stands nowhere in lowercase is often a name, and in German every noun is capitalised, so there we ask whether the word appears several times on one page. A tokeniser also leaves aren and isn behind at an apostrophe, and aren had come out of one list as a C2 word.

3. The level was the maximum, and "I know it" was free

The old level was the maximum over the pages visited, and only the first three claims of a run were ever verified. A reader could press "I know it" on everything, fail every check and still be placed at B2 in 83% of runs, and at C1 in the rest.

Now the level is the highest band that passed with no failure under it, and nothing is reported above the highest band where a claim was actually checked. On the same adversarial runs, none reach B1.

What I would do again

Measure on your own data before tuning. The 29% and the 83% were found by counting, not by looking at a few pages.
Treat a threshold as a calibration constant. We now write every run down (bands walked, words known and asked, checks passed) so the next calibration starts from our own numbers.
Keep the engine in one place. The same walk runs in the Android app, on the web and on the logged-out page, pinned by one shared test fixture.

If you grade text for learners, or anything else where "rare" and "hard" get conflated, I would like to hear what you use instead of rank.

DEV Community: LingvoBerry