DEV Community: GradientGym

NLP #2: Unicode and UTF-8

GradientGym — Sun, 08 Nov 2026 07:00:00 +0000

Originally published on almoghagbi.com, where the figures are interactive.

Before a natural language processing (NLP) model can split text into words or
tokens, the text has to exist inside the computer, where there are only
numbers. This post covers section 2.3 of
Speech and Language Processing:
how a character becomes a number (Unicode), and how that number becomes bytes
in a file (UTF-8). The next post builds on it: modern tokenizers start from
exactly these bytes.

Intuition

Every character gets a number

Our running example for this post is a short review:

Great café, only ₪5 😀

In the 1960s, computers in the US stored English text with a code called
ASCII. It gave each character a number below 128: a is 97, G is 71,
a space is 32. Each character took one
byte. A byte is 8
bits, and each bit is a 0 or a 1. That
gives 256 possible values, from 0 to 255. ASCII has no é, no ₪, no Hebrew,
no Chinese and no emoji. Three of the characters in our review don't exist in
it.

Unicode fixes that with one big list of every character in every writing
system: Latin, Hebrew, Arabic, Devanagari, Chinese and ancient scripts like
cuneiform. It also covers math symbols and emoji. Version 16.0 has more than 150,000
characters from 168 scripts. Each one gets its own number, called a code point.

Code points are written in hexadecimal,
which counts in base 16 instead of base 10. It uses the digits 0–9, then A–F
for 10 to 15. We write U+ in front: é is U+00E9, ₪ is U+20AA and 😀 is
U+1F600. The first 128 code points are exactly ASCII, so a is still 97. In
hexadecimal that's U+0061, because 61 means 6 sixteens plus 1:
$6 \cdot 16 + 1 = 97$ .

A code point names a character. It doesn't say how the character looks.
That's the font's job: a in bold, in italics or in a typewriter font is
the same code point. The drawn shape is called a glyph.

Short codes for common characters

Now we need to write those numbers into a file. The simplest way is to give
every character 4 bytes, enough for the largest code point. Our review has
21 characters, so it would take 84 bytes, most of them zeros. That wastes
space. Worse, old software reads a zero byte as "end of text".

An encoding is a rule for turning code points into bytes. UTF-8, the
encoding used by nearly all of the web, uses short codes for
common characters and longer ones for rare characters:

1 byte for ASCII: English letters, digits, punctuation.
2 bytes for accented Latin letters like é, and for Greek, Cyrillic, Hebrew and Arabic letters.
3 bytes for most Chinese, Japanese and Korean characters, and symbols like ₪.
4 bytes for emoji and rare characters.

Our review takes 27 bytes instead of 84. Since ASCII characters keep
their single byte, every old ASCII file is already valid UTF-8. Try it with
your own text:

▶ Interactive figure: try it on the original post.

How a program knows where a character ends

A file is one long row of bytes, with no gaps between characters and no
lengths stored next to them. So how does a program know that é takes two
bytes while the 5 takes one?

Every byte starts with a small label. On a character's first byte, the label
says how long the character is: "just me", "two of us", "three of us" or "four
of us". On every later byte, it says "I belong to the one before me". The
program reads a label, takes that many bytes, then does the same for the next
character.

₪5 is saved as four bytes. The first byte says "three of us", so that byte
and the next two make ₪. The fourth says "just me", so it's the 5. The math
section shows the exact bits.

Characters and bytes are different counts

So a text has two lengths: how many characters it has, and how many bytes it
takes. For English text they are equal. For Hebrew they're not: "שלום" has 4
characters and takes 8 bytes.

Every text file is stored in some encoding. Read it with the wrong one and
you get garbage: a UTF-8 é read as an older Western-European encoding
shows up as Ã©. If you've seen "cafÃ©" on a website, that's what happened.

For NLP, bytes are a handy unit. A model whose units are characters needs
room for 150,000 of them, and it still meets characters it has never seen.
A model whose units are bytes needs only 256, and every possible text is
made of them. That's why many tokenizers start from UTF-8 bytes, as the
next post shows.

The math

Code points

Definition 1 (Code point). Unicode assigns each character a unique
integer, its code point $c$ , with $0 \leq c \leq 0x10FFFF$ (0x
marks a hexadecimal number). A code point is written U+ followed by at least four hex digits. A text is
then a sequence of code points $c_{1}, c_{2}, \dots, c_{n}$ , where $n$ is its
length in characters.

There are $0x10FFFF + 1 = 1, 114, 112$ possible code points.
Since $2^{20} = 1, 048, 576$ is too few and $2^{21} = 2, 097, 152$ is
enough, a code point needs 21 bits.

Encodings

Definition 2 (Encoding). An encoding turns a sequence of code points
into a sequence of bytes (integers from 0 to 255), in a way that can be
reversed. Turning the bytes back into code points is decoding. A
fixed-length encoding uses the same number of bytes for every code
point. A variable-length encoding doesn't.

ASCII is the old one-byte code for the first 128 code points. UTF-32 is a
fixed-length encoding: 4 bytes per code point, so a text of $n$
characters takes $4 n$ bytes. UTF-8 is variable-length.

Key equation (UTF-8 length). UTF-8 stores code point $c$ in

ℓ (c) = {1 c < 0x80 2 0x80 \leq c < 0x800 3 0x800 \leq c < 0x10000 4 0x10000 \leq c \leq 0x10FFFF

bytes, so a text takes $\sum_{i = 1 n} ℓ (c_{i})$ bytes. Take the text
Great café, only ₪5 😀. It has 18 ASCII characters, plus é (2 bytes), ₪ (3) and 😀 (4):
$18 + 2 + 3 + 4 = 27$ bytes, against $4 \cdot 21 = 84$ in UTF-32.

How UTF-8 packs the bits

Key equation (UTF-8 bit layout). The code point's bits are cut into
chunks. Each byte starts with a few marker bits that say what it is,
followed by one chunk:

Bytes	Free slots (`x`)	Byte 1	Bytes 2–4
1	7	`0xxxxxxx`	none
2	5 + 6 = 11	`110xxxxx`	`10xxxxxx`
3	4 + 6 + 6 = 16	`1110xxxx`	`10xxxxxx` ×2
4	3 + 6 + 6 + 6 = 21	`11110xxx`	`10xxxxxx` ×3

Each x is an empty slot. You fill the slots with the bits of the code point,
left to right. The 0s and 1s written in the table are the marker bits. They
never change. In a character of 2 or more bytes, the number of 1s at the start
of the first byte equals the number of bytes. Every byte after the first, called
a continuation byte, starts with 10.

The slots also explain the limits in the length equation. With 7 slots, the
largest number is $2^{7} - 1 = 127$ , so 1 byte stops at 0x80 = 128. With 11 slots,
2 bytes stop at $2^{11}$ = 0x800. With 16 slots, 3 bytes stop at $2^{16}$ =
0x10000.

The simplest case is a, code point 97, which is 1100001 in binary. Those 7
bits fill the 7 slots of 0xxxxxxx, giving 01100001 = 0x61. That's one byte,
the same as in ASCII.

Now take é, code point U+00E9 = 233. It's between 0x80 and 0x7FF, so it takes
2 bytes with 11 slots. In binary, 233 is 11101001, which is only 8 bits, so we
add zeros on the left to fill the 11 slots: 00011 101001, with a space only to
show the two groups. The first 5 bits go
after 110, the last 6 after 10:

11000011 = 0xC3, 10101001 = 0xA9 .

So é is stored as C3 A9. Click é in the widget above to see the same
packing, and then try ₪ (3 bytes) and 😀 (4 bytes).

Reading a file back

A file holds only bytes, with no separators or lengths. Here is our whole
review as saved on disk, all 27 bytes in hexadecimal:

47 72 65 61 74 20 63 61 66 C3 A9 2C 20 6F 6E 6C 79 20 E2 82 AA 35 20 F0 9F 98 80

A decoder reads the marker bits of a first byte, takes that many bytes, then
repeats. Here is the ₪5 part, E2 82 AA 35, written in bits:

11100010 10000010 10101010 00110101

The first byte starts with 1110, so this character takes 3 bytes. The next two start with 10, as they should.
Remove the marker bits and join the rest: 0010 000010 101010. That's 0x20AA, the code point of ₪.
The next byte starts with 0, so it's a character by itself: 0x35, the digit 5.

Useful properties

The layout has useful properties:

ASCII-compatible: a byte below 128 is always a whole ASCII character.
No zero bytes, except for the code point U+0000 itself.
Self-synchronizing: a byte starting with 10 is never the start of a character. Jump into the middle of a file, skip at most 3 such bytes, and you're at the start of the next character.

In Python, a string is a sequence of code points, and encode gives the
bytes:

text = "Great café, only ₪5 😀"
print(len(text))                           # 21 characters (code points)
data = text.encode("utf-8")
print(len(data))                           # 27 bytes
print(hex(ord("é")), data[9:11].hex(" "))  # 0xe9 c3 a9
print(len(text.encode("utf-32-be")))       # 84 bytes

Exercises

Each question has one correct answer. Pick your answer, then read the explanation. Questions 6–10 use the definitions and equations above.

1. (Intuition) ASCII used one byte per character and only numbers below 128. Why was that enough for English, but not for Hebrew?

A. Hebrew letters need more than one byte because they're written right to left.
B. Hebrew has more letters than English.
C. ASCII could store only capital letters.
D. English fits below 128, but ASCII has no Hebrew letters.

Answer

D. Upper- and lowercase letters, digits, punctuation and a space come to about 95 printable characters, so they all fit below 128. ASCII has no Hebrew letters at all, so a Hebrew text can't be written in it.

A is wrong: writing direction has nothing to do with how many bytes a character takes.
B is wrong: Hebrew has 22 letters and English has 26. The number of letters wasn't the problem.
C is wrong: ASCII has both cases. a is 97 and A is 65.

2. (Intuition) Are a, a (bold) and a (italic) the same code point?

A. No, each style has its own code point.
B. Yes, bold and italic are different glyphs of the same code point.
C. Only a and a are the same; italic letters have their own code points.
D. It depends on the file's encoding.

Answer

B. All three are U+0061. Bold and italic are different glyphs, chosen by the font. A code point names the character, not its look.

A and C are wrong: style doesn't change the code point. a and A do differ (U+0061 and U+0041), because they're different characters.
D is wrong: the encoding decides how a code point becomes bytes. It doesn't decide which code point a character has.

3. (Intuition) A friend's name, Zoë, shows up in a spreadsheet as "ZoÃ«". What went wrong?

A. The file was damaged during the download.
B. The spreadsheet's font doesn't include ë.
C. The file is UTF-8, but the spreadsheet read it as an older encoding.
D. The file was saved in UTF-32.

Answer

C. In UTF-8, ë takes two bytes. An older encoding with one character per byte turns each of those bytes into its own character: Ã and «. Opening the file as UTF-8 fixes it.

A is wrong: the bytes are intact, which is why reading them correctly fixes the name.
B is wrong: a missing glyph shows up as an empty box or a placeholder, not as two other letters.
D is wrong: in UTF-32 every character takes 4 bytes, most of them zeros, so "Z", "o" and all the rest would look broken too.

4. (Intuition) Compare "hello" and "שלום" in UTF-8. Which statement is true?

A. "hello" has more characters, but "שלום" takes more bytes.
B. "hello" has more characters and takes more bytes.
C. "שלום" has more characters and takes more bytes.
D. They take the same number of bytes.

Answer

A. "hello" has 5 characters against 4. Each English letter takes 1 byte, so "hello" takes 5 bytes. Each Hebrew letter takes 2, so "שלום" takes 8. More characters doesn't always mean more bytes, which is why a text has two different lengths.

B and D are wrong: they ignore that Hebrew letters take 2 bytes each.
C is wrong: "שלום" has only 4 characters.

5. (Intuition, widget) In the widget, replace the text with a thumbs-up with a skin tone: 👍🏽. How many characters does it show?

A. 1
B. 2
C. 4
D. 8

Answer

B. The widget shows 👍 (U+1F44D) and a skin-tone modifier (U+1F3FD). What you see as one symbol can be several code points, so counting "characters" is a choice, like counting words in post 1.

A is what you see on screen, not the number of code points.
C is the bytes of one of the two code points.
D is the total number of bytes: 4 for each code point.

6. (Math · Definition 1) Why does a code point need 21 bits, and not 20?

A. 20 bits give too few values for all the code points.
B. The extra bit marks where the code point ends.
C. Each byte in UTF-8 reserves one bit for a marker.
D. Unicode has 21 groups of characters.

Answer

A. The code points run from 0 to 0x10FFFF, which makes $1, 114, 112$ of them. With 20 bits there are only $2^{20} = 1, 048, 576$ different values, too few. With 21 bits there are $2^{21} = 2, 097, 152$ , enough.

B and C are wrong: these 21 bits count only the code point itself. Marker bits are added by UTF-8 on top of them, and they depend on the encoding.
D is wrong: the number of bits comes from the size of the range, not from how characters are grouped.

7. (Math · UTF-8 length) How many bytes does "naïve ₪100 🎉" take in UTF-8, and in UTF-32?

A. 12 in UTF-8, 48 in UTF-32
B. 15 in UTF-8, 48 in UTF-32
C. 18 in UTF-8, 48 in UTF-32
D. 18 in UTF-8, 72 in UTF-32

Answer

C. There are 12 characters: n, a, ï, v, e, space, ₪, 1, 0, 0, space, 🎉. In UTF-8, the 9 ASCII characters take 1 byte each, ï takes 2, ₪ takes 3 and 🎉 takes 4, so $9 + 2 + 3 + 4 = 18$ bytes. In UTF-32, every character takes 4 bytes: $4 \cdot 12 = 48$ .

A is wrong: 12 is the number of characters, not bytes.
B is wrong: it counts ï and ₪ as 1 byte each, so $9 + 1 + 1 + 4 = 15$ .
D is wrong: UTF-32 multiplies the characters by 4, not the UTF-8 bytes.

8. (Math · UTF-8 bit layout) How is the Hebrew letter ש (U+05E9) encoded in UTF-8?

A. 05 E9
B. E0 97 A9
C. C5 E9
D. D7 A9

Answer

D. 0x05E9 is between 0x80 and 0x7FF, so it takes 2 bytes with 11 slots. In binary, 0x05E9 = 101 1110 1001, so the 11 bits are 10111 101001. Then:

11010111 = 0xD7, 10101001 = 0xA9 .

Check it in the widget, or with "ש".encode("utf-8").

A is wrong: those are the code point's digits with no marker bits at all.
B is wrong: it uses the 3-byte layout, but this code point fits in 2 bytes. UTF-8 always uses the shortest form.
C is wrong: E9 = 11101001 starts with 1110, so it can't be a continuation byte.

9. (Math · UTF-8 bit layout) Which character do the bytes E2 82 AC encode?

A. ₪
B. €
C. é
D. Three separate characters

Answer

B. E2 = 1110 0010 starts with 1110, so this is a 3-byte character, and its chunk is 0010. 82 = 10 000010 gives 000010, and AC = 10 101100 gives 101100. Together: 0010 000010 101100 = 0x20AC, the euro sign.

A is wrong: ₪ is U+20AA, which is E2 82 AA. It differs only in the last byte.
C is wrong: é is C3 A9, only 2 bytes.
D is wrong: 82 and AC start with 10, so they continue the character that E2 started.

10. (Math · self-synchronizing) A download breaks in the middle of our review, and the part you got starts with the bytes 82 AA 35 20 .... Where is the first whole character?

A. 82
B. AA
C. 35, the digit 5
D. 20, the space

Answer

C. 82 = 10000010 and AA = 10101010 both start with 10, so they're continuation bytes: the end of the ₪ (E2 82 AA), whose first byte was lost. Skip them. 35 starts with 0, so it's a whole 1-byte character, and decoding starts there.

A and B are wrong: a byte starting with 10 is never the start of a character.
D is wrong: 20 is a whole character too, but it's the second one. 35 comes first.

Next: byte-pair encoding (BPE), the algorithm that builds a tokenizer's
vocabulary from these bytes (section 2.4).

Co-authored with AI.

NLP #1: What Is a Word?

GradientGym — Sun, 01 Nov 2026 07:00:00 +0000

Originally published on almoghagbi.com, where the figures are interactive.

I'm taking an introductory natural language processing (NLP) course this
semester. This series is me learning it in public. My course follows
Speech and Language Processing
by Jurafsky & Martin: the 3rd edition draft (January 2026 version), which
is free online. The authors update the draft occasionally, so a newer version may
number its sections differently. Any version works if you want to follow
along.

This first post covers sections 2.1–2.2. Before a model can read text, someone has to decide what
a "word" is. That decision is the first step of every NLP system, from a
spam filter to ChatGPT.

Each post has three parts: Intuition (no math needed), The math
(definitions and key equations), and Exercises (to check yourself).

Intuition

Counting words is a choice

Take one sentence, our running example for the whole post:

The bakery's cinnamon rolls were so good, I bought more rolls for the road.

How many words? Split on spaces and you get 14. But "rolls" appears twice,
so there are only 13 different words. NLP needs both counts. Every word as it appears in the text is an instance. Every distinct word is a type. The set of types is the vocabulary.

Then the questions start. Are "The" and "the" the same type? Is the comma a
word? There's no single right answer. A search engine may ignore capitals.
A model that writes text keeps capitals and punctuation, because it
has to produce both. Try it:

▶ Interactive figure: try it on the original post.

Spoken language adds more puzzles. In "I want, uh, the cinnamon one",
is uh a word? A transcript for reading would drop it. A speech recognizer
keeps it, because a pause like that is a hint about what comes next.

New words never stop coming

On the first page of a novel, almost every word is new to your list.
By page 100 you have seen "the", "and" and "she" hundreds of times, and new
words come slowly. But they never stop: there's always another name, place,
technical term, typo or bit of slang.

So a vocabulary has no final size. The more text you collect, the bigger it
gets. For a model this is a real problem: however large its word list, it
will meet words that aren't on it.

Words have parts

"Rolls" is roll plus -s, which means "more than one". "Bakery's" is
bake, plus -ery (a place where it's done), plus 's (belongs to).
These smallest pieces that carry meaning are called morphemes. Some
don't split cleanly: "bought" means buy + past tense, but you can't point
to the letters that mean "past".

Languages differ a lot here. In Spanish, dámelo ("give it to me") is
one written word: da (give) + me (to me) + lo (it). Chinese has no
spaces between words, so finding the words is its own task.

Put together, these facts make whole words a shaky unit for a model. They're
hard to define across languages, and new ones keep appearing. That's why
modern models split text into subwords, pieces between a letter and a
word. Reusing pieces like bake, -er and -s, they can build words
they've never seen, like baker and bakes. How they choose those pieces
is the topic of the next posts.

The math

Instances and types

Definition 1 (Corpus, word instance). A corpus is a collection of
text. After deciding what counts as a word, the corpus is a sequence of
word instances $w_{1}, w_{2}, \dots, w_{N}$ . $N$ is the number of instances,
the length of the corpus in words.

Definition 2 (Vocabulary, word type). The vocabulary $V$ is the set
of distinct words in the corpus: $V := w_{1}, \dots, w_{N}$ , with repeats
removed. Its elements are types, and $∣ V ∣$ is the number of types.

Since every type appears at least once, $∣ V ∣ \leq N$ . In the running example,
splitting on spaces and keeping capitals, $N = 14$ and $∣ V ∣ = 13$ .
Lowercasing merges "The" and "the" into one type, so $∣ V ∣ = 12$ . Counting
the comma and the period as words gives $N = 16$ .

In code, with one possible definition of "word":

import re

text = "The bakery's cinnamon rolls were so good, I bought more rolls for the road."
words = re.findall(r"[\w']+", text)  # letters, digits and apostrophes
print(len(words), len(set(words)))  # N = 14, |V| = 13

Heaps' law

Key equation (Heaps' law, also called Herdan's law). For a text of $N$
words, the number of different words is about

∣ V ∣ = k N^{β}, k > 0, 0 < β < 1.

It has two numbers, both measured from real text:

$β$ (beta) is the growth rate: how fast new words keep coming. At $β = 1$ every word is new; at $β = 0$ no word is. For English $β$ is between 0.4 and 0.6. At the usual 0.5, 100 times more text gives only about 10 times more different words.
$k$ is the scale: it moves the whole curve up or down. A varied text (technical, full of names) has a bigger $k$ .

For example, with $k = 20$ and $β = 0.5$ , a text of 10,000 words has
about $20 \cdot 10, 000 = 2, 000$ different words, and a text of
1,000,000 words about 20,000.

Taking logs gives $lo g ∣ V ∣ = lo g k + β lo g N$ : on log-log axes the
law is a straight line, and $β$ is its slope. Here is real text: ten
classic novels read one after another. Drag the slider to see the slope at
each point:

▶ Interactive figure: try it on the original post.

The slope starts near 1 (at first almost every word is new), settles around
0.5, and rises again when a new author brings new names. So $β$ isn't
one fixed number: it describes a stretch of text.

Two more ways to read the law. The derivative of $∣ V ∣$ is always positive,
but it shrinks as $N$ grows:

\frac{d ∣ V ∣}{d N} = k β N^{β - 1} > 0.

New words keep coming, but more slowly. Scaling shows the same thing:
multiply the text by $c$ , and the vocabulary grows only by $c^{β}$ ,
which is less than $c$ :

N \to c N \Rightarrow ∣ V ∣ \to c^{β} ∣ V ∣.

Morphemes

Definition 3 (Morpheme, root, affix). A morpheme is the smallest
unit of a language that carries meaning. A word is a sequence of morphemes
$m_{1} m_{2} \dots m_{k}$ . The root carries the main meaning (bake in
bakery). Affixes add to it (-ery, -s).

Definition 4 (Inflectional, derivational, clitic). An inflectional
affix adds grammar and keeps the word's class: roll → rolls (plural),
bake → baked (past). A derivational affix makes a new word, often of
a different class, with a less predictable meaning: bake (verb) →
bakery (noun). A clitic acts like a separate word but can't stand
alone, so it attaches to its neighbor: the 's in bakery's, the 'll in
I'll.

Key equation (morphemes per word). If word $i$ of a text has $k_{i}$
morphemes, the text averages

k ˉ = \frac{1}{N} i = 1 \sum N k_{i}

morphemes per word. Languages with $k ˉ$ close to 1 (like Vietnamese)
are called isolating. Languages that pack many morphemes into one word
(like Turkish) are synthetic. A high $k ˉ$ means many possible word
forms, so $∣ V ∣$ grows faster. That is one more reason to split text into subwords.

Exercises

Try each one before opening the answer. Questions 6–10 use the definitions and equations above.

1. (Intuition) Is um a word? Give one application that should keep it
and one that should remove it.

Answer

It depends on the task. A speech recognizer should keep it, since a filler
signals a pause or a restart and helps predict what comes next. Subtitles
or a meeting summary for reading should remove it, because it adds nothing
for the reader.

2. (Intuition) In the counter above, paste a paragraph from any news
site and switch "ignore capitals" on. Why does $∣ V ∣$ go down? Give an
example where ignoring capitals loses information.

Answer

Words at the start of a sentence ("The") merge with the same word inside a
sentence ("the"), so there are fewer types. But capitals sometimes carry
meaning. Ignoring them merges "Apple" (the company) with "apple" (the fruit),
and "US" (the country) with "us" (the pronoun).

3. (Intuition) You double the amount of text in your collection. Does
the vocabulary double too?

Answer

No. Most of the new text reuses words you've already seen, so the
vocabulary grows by much less than double. It does still grow, because the
new text brings new names, terms and typos.

4. (Intuition) Give three reasons why a model can't simply use a list
of all the words in a language.

Answer

The list never ends: there are always new names, terms and slang.
Some languages (Chinese, Japanese, Thai) don't put spaces between words, so even deciding where words are is hard.
In languages with many morphemes per word, one root has a huge number of forms, each a separate entry.

5. (Intuition) The Spanish word dámelo means "give it to
me". Split it into its parts. Then explain why it's a problem for a
model that treats every whole word as one unit.

Answer

da (give) + me (to me) + lo (it). A whole-word model sees one rare
word. It must learn that word from scratch, even if it has seen da, me and lo
in other words. A subword model can reuse those parts.

6. (Math · Definitions 1–2) For the corpus "the dog saw the other dog
near the gate", find $N$ and $∣ V ∣$ .

Answer

The instances are: the, dog, saw, the, other, dog, near, the, gate, so
$N = 9$ . The types are the, dog, saw, other, near, gate, so $∣ V ∣ = 6$ .
"the" appears 3 times and "dog" 2 times, which accounts for the difference:
$9 - 6 = (3 - 1) + (2 - 1)$ .

7. (Math · Definition 2) Show that $∣ V ∣ \leq N$ for any corpus. When are
they equal?

Answer

Each type in $V$ is one of the instances $w_{1}, \dots, w_{N}$ . Different types come from different instances, so $V$ can't have more elements than there
are instances: $∣ V ∣ \leq N$ . They're equal exactly when no word repeats,
meaning every instance is a different type.

8. (Math · Heaps' law) A corpus follows Heaps' law with $k = 30$ and
$β = 0.5$ . (a) What is $∣ V ∣$ for $N = 40, 000$ ? (b) How many instances
do you need for a vocabulary of 12,000 types?

Answer

(a) $∣ V ∣ = 30 \cdot 40, 00 0^{0.5} = 30 \cdot 200 = 6, 000$ .

(b) Solve $12, 000 = 30 \cdot N^{0.5}$ : $N^{0.5} = 400$ , so
$N = 40 0^{2} = 160, 000$ .

Doubling the vocabulary takes four times as much text, because with
$β = 0.5$ , $∣ V ∣$ grows like $N$ .

9. (Math · Heaps' law, log form) You measure two points on a corpus:
$N = 1 0^{4}$ gives $∣ V ∣ = 2, 000$ , and $N = 1 0^{6}$ gives $∣ V ∣ = 20, 000$ .
Find $β$ and $k$ .

Answer

Divide the two equations: $\frac{20 , 000}{2 , 000} = \frac{k ( 1 0 ^{6} ) ^{β}}{k ( 1 0 ^{4} ) ^{β}} = (1 0^{2})^{β}$ ,
so $10 = 10 0^{β}$ and $β = \frac{l o g 10}{l o g 100} = 0.5$ .

Then $k = \frac{2 , 000}{( 1 0 ^{4} ) ^{0.5}} = \frac{2 , 000}{100} = 20$ .

10. (Math · Definitions 3–4, morphemes per word) Split each word of
"The teachers repainted the unbreakable doors" into morphemes, label every
affix as inflectional or derivational, and compute $k ˉ$ .

Answer

Word	Morphemes	$k_{i}$
The	the	1
teachers	teach + -er (derivational) + -s (inflectional)	3
repainted	re- (derivational) + paint + -ed (inflectional)	3
the	the	1
unbreakable	un- (derivational) + break + -able (derivational)	3
doors	door + -s (inflectional)	2

$k ˉ = \frac{1 + 3 + 3 + 1 + 3 + 2}{6} = \frac{13}{6} \approx 2.17$
morphemes per word.

Next: how computers store text as bytes: Unicode, code points and
UTF-8 (section 2.3).

Co-authored with AI.