GradientGym

Posted on Nov 1 Originally published at almoghagbi.com

NLP #1: What Is a Word?

#nlp #wordsandtokens

Originally published on almoghagbi.com, where the figures are interactive.

I'm taking an introductory natural language processing (NLP) course this
semester. This series is me learning it in public. My course follows
Speech and Language Processing
by Jurafsky & Martin: the 3rd edition draft (January 2026 version), which
is free online. The authors update the draft occasionally, so a newer version may
number its sections differently. Any version works if you want to follow
along.

This first post covers sections 2.1–2.2. Before a model can read text, someone has to decide what
a "word" is. That decision is the first step of every NLP system, from a
spam filter to ChatGPT.

Each post has three parts: Intuition (no math needed), The math
(definitions and key equations), and Exercises (to check yourself).

Intuition

Counting words is a choice

Take one sentence, our running example for the whole post:

The bakery's cinnamon rolls were so good, I bought more rolls for the road.

How many words? Split on spaces and you get 14. But "rolls" appears twice,
so there are only 13 different words. NLP needs both counts. Every word as it appears in the text is an instance. Every distinct word is a type. The set of types is the vocabulary.

Then the questions start. Are "The" and "the" the same type? Is the comma a
word? There's no single right answer. A search engine may ignore capitals.
A model that writes text keeps capitals and punctuation, because it
has to produce both. Try it:

▶ Interactive figure: try it on the original post.

Spoken language adds more puzzles. In "I want, uh, the cinnamon one",
is uh a word? A transcript for reading would drop it. A speech recognizer
keeps it, because a pause like that is a hint about what comes next.

New words never stop coming

On the first page of a novel, almost every word is new to your list.
By page 100 you have seen "the", "and" and "she" hundreds of times, and new
words come slowly. But they never stop: there's always another name, place,
technical term, typo or bit of slang.

So a vocabulary has no final size. The more text you collect, the bigger it
gets. For a model this is a real problem: however large its word list, it
will meet words that aren't on it.

Words have parts

"Rolls" is roll plus -s, which means "more than one". "Bakery's" is
bake, plus -ery (a place where it's done), plus 's (belongs to).
These smallest pieces that carry meaning are called morphemes. Some
don't split cleanly: "bought" means buy + past tense, but you can't point
to the letters that mean "past".

Languages differ a lot here. In Spanish, dámelo ("give it to me") is
one written word: da (give) + me (to me) + lo (it). Chinese has no
spaces between words, so finding the words is its own task.

Put together, these facts make whole words a shaky unit for a model. They're
hard to define across languages, and new ones keep appearing. That's why
modern models split text into subwords, pieces between a letter and a
word. Reusing pieces like bake, -er and -s, they can build words
they've never seen, like baker and bakes. How they choose those pieces
is the topic of the next posts.

The math

Instances and types

Definition 1 (Corpus, word instance). A corpus is a collection of
text. After deciding what counts as a word, the corpus is a sequence of
word instances $w_1, w_2, \dots, w_N$ . $N$ is the number of instances,
the length of the corpus in words.

Definition 2 (Vocabulary, word type). The vocabulary $V$ is the set
of distinct words in the corpus: $V := {w_1, \dots, w_N}$ , with repeats
removed. Its elements are types, and $|V|$ is the number of types.

Since every type appears at least once, $|V| \le N$ . In the running example,
splitting on spaces and keeping capitals, $N = 14$ and $|V| = 13$ .
Lowercasing merges "The" and "the" into one type, so $|V| = 12$ . Counting
the comma and the period as words gives $N = 16$ .

In code, with one possible definition of "word":

import re

text = "The bakery's cinnamon rolls were so good, I bought more rolls for the road."
words = re.findall(r"[\w']+", text)  # letters, digits and apostrophes
print(len(words), len(set(words)))  # N = 14, |V| = 13

Heaps' law

Key equation (Heaps' law, also called Herdan's law). For a text of $N$
words, the number of different words is about

|V| = k N^{\beta}, \qquad k > 0, \quad 0 < \beta < 1.

It has two numbers, both measured from real text:

$\beta$ (beta) is the growth rate: how fast new words keep coming. At $\beta = 1$ every word is new; at $\beta = 0$ no word is. For English $\beta$ is between 0.4 and 0.6. At the usual 0.5, 100 times more text gives only about 10 times more different words.
$k$ is the scale: it moves the whole curve up or down. A varied text (technical, full of names) has a bigger $k$ .

For example, with $k = 20$ and $\beta = 0.5$ , a text of 10,000 words has
about $20 \cdot \sqrt{10{,}000} = 2{,}000$ different words, and a text of
1,000,000 words about 20,000.

Taking logs gives $\log |V| = \log k + \beta \log N$ : on log-log axes the
law is a straight line, and $\beta$ is its slope. Here is real text: ten
classic novels read one after another. Drag the slider to see the slope at
each point:

▶ Interactive figure: try it on the original post.

The slope starts near 1 (at first almost every word is new), settles around
0.5, and rises again when a new author brings new names. So $\beta$ isn't
one fixed number: it describes a stretch of text.

Two more ways to read the law. The derivative of $|V|$ is always positive,
but it shrinks as $N$ grows:

\frac{d|V|}{dN} = k \beta N^{\beta - 1} > 0.

New words keep coming, but more slowly. Scaling shows the same thing:
multiply the text by $c$ , and the vocabulary grows only by $c^{\beta}$ ,
which is less than $c$ :

N \to cN \;\Rightarrow\; |V| \to c^{\beta}|V|.

Morphemes

Definition 3 (Morpheme, root, affix). A morpheme is the smallest
unit of a language that carries meaning. A word is a sequence of morphemes
$m_1 m_2 \dots m_k$ . The root carries the main meaning (bake in
bakery). Affixes add to it (-ery, -s).

Definition 4 (Inflectional, derivational, clitic). An inflectional
affix adds grammar and keeps the word's class: roll → rolls (plural),
bake → baked (past). A derivational affix makes a new word, often of
a different class, with a less predictable meaning: bake (verb) →
bakery (noun). A clitic acts like a separate word but can't stand
alone, so it attaches to its neighbor: the 's in bakery's, the 'll in
I'll.

Key equation (morphemes per word). If word $i$ of a text has $k_i$
morphemes, the text averages

\bar{k} = \frac{1}{N} \sum_{i=1}^{N} k_i

morphemes per word. Languages with $\bar{k}$ close to 1 (like Vietnamese)
are called isolating. Languages that pack many morphemes into one word
(like Turkish) are synthetic. A high $\bar{k}$ means many possible word
forms, so $|V|$ grows faster. That is one more reason to split text into subwords.

Exercises

Try each one before opening the answer. Questions 6–10 use the definitions and equations above.

1. (Intuition) Is um a word? Give one application that should keep it
and one that should remove it.

Answer

It depends on the task. A speech recognizer should keep it, since a filler
signals a pause or a restart and helps predict what comes next. Subtitles
or a meeting summary for reading should remove it, because it adds nothing
for the reader.

2. (Intuition) In the counter above, paste a paragraph from any news
site and switch "ignore capitals" on. Why does $|V|$ go down? Give an
example where ignoring capitals loses information.

Answer

Words at the start of a sentence ("The") merge with the same word inside a
sentence ("the"), so there are fewer types. But capitals sometimes carry
meaning. Ignoring them merges "Apple" (the company) with "apple" (the fruit),
and "US" (the country) with "us" (the pronoun).

3. (Intuition) You double the amount of text in your collection. Does
the vocabulary double too?

Answer

No. Most of the new text reuses words you've already seen, so the
vocabulary grows by much less than double. It does still grow, because the
new text brings new names, terms and typos.

4. (Intuition) Give three reasons why a model can't simply use a list
of all the words in a language.

Answer

The list never ends: there are always new names, terms and slang.
Some languages (Chinese, Japanese, Thai) don't put spaces between words, so even deciding where words are is hard.
In languages with many morphemes per word, one root has a huge number of forms, each a separate entry.

5. (Intuition) The Spanish word dámelo means "give it to
me". Split it into its parts. Then explain why it's a problem for a
model that treats every whole word as one unit.

Answer

da (give) + me (to me) + lo (it). A whole-word model sees one rare
word. It must learn that word from scratch, even if it has seen da, me and lo
in other words. A subword model can reuse those parts.

6. (Math · Definitions 1–2) For the corpus "the dog saw the other dog
near the gate", find $N$ and $|V|$ .

Answer

The instances are: the, dog, saw, the, other, dog, near, the, gate, so
$N = 9$ . The types are the, dog, saw, other, near, gate, so $∣ V ∣ = 6$ .
"the" appears 3 times and "dog" 2 times, which accounts for the difference:
$9 - 6 = (3 - 1) + (2 - 1)$ .

7. (Math · Definition 2) Show that $|V| \le N$ for any corpus. When are
they equal?

Answer

Each type in $V$ is one of the instances $w_{1}, \dots, w_{N}$ . Different types come from different instances, so $V$ can't have more elements than there
are instances: $∣ V ∣ \leq N$ . They're equal exactly when no word repeats,
meaning every instance is a different type.

8. (Math · Heaps' law) A corpus follows Heaps' law with $k = 30$ and
$\beta = 0.5$ . (a) What is $|V|$ for $N = 40{,}000$ ? (b) How many instances
do you need for a vocabulary of 12,000 types?

Answer

(a) $∣ V ∣ = 30 \cdot 40, 00 0^{0.5} = 30 \cdot 200 = 6, 000$ .

(b) Solve $12, 000 = 30 \cdot N^{0.5}$ : $N^{0.5} = 400$ , so
$N = 40 0^{2} = 160, 000$ .

Doubling the vocabulary takes four times as much text, because with
$β = 0.5$ , $∣ V ∣$ grows like $N$ .

9. (Math · Heaps' law, log form) You measure two points on a corpus:
$N = 10^4$ gives $|V| = 2{,}000$ , and $N = 10^6$ gives $|V| = 20{,}000$ .
Find $\beta$ and $k$ .

Answer

Divide the two equations: $\frac{20 , 000}{2 , 000} = \frac{k ( 1 0 ^{6} ) ^{β}}{k ( 1 0 ^{4} ) ^{β}} = (1 0^{2})^{β}$ ,
so $10 = 10 0^{β}$ and $β = \frac{l o g 10}{l o g 100} = 0.5$ .

Then $k = \frac{2 , 000}{( 1 0 ^{4} ) ^{0.5}} = \frac{2 , 000}{100} = 20$ .

10. (Math · Definitions 3–4, morphemes per word) Split each word of
"The teachers repainted the unbreakable doors" into morphemes, label every
affix as inflectional or derivational, and compute $\bar{k}$ .

Answer

Word	Morphemes	$k_{i}$
The	the	1
teachers	teach + -er (derivational) + -s (inflectional)	3
repainted	re- (derivational) + paint + -ed (inflectional)	3
the	the	1
unbreakable	un- (derivational) + break + -able (derivational)	3
doors	door + -s (inflectional)	2

$k ˉ = \frac{1 + 3 + 3 + 1 + 3 + 2}{6} = \frac{13}{6} \approx 2.17$
morphemes per word.

Next: how computers store text as bytes: Unicode, code points and
UTF-8 (section 2.3).

Co-authored with AI.

DEV Community

NLP #1: What Is a Word?

Intuition

Counting words is a choice

New words never stop coming

Words have parts

The math

Instances and types

Heaps' law

Morphemes

Exercises

Top comments (0)