DEV Community: Ritesh

Build a BPE tokenizer in 30 lines of Python and you will never read a prompt the same way again

Ritesh — Mon, 04 May 2026 17:57:26 +0000

Most engineers who use language models every day cannot, on a blank piece of paper, describe what their tokenizer is doing to the prompts they send. The library returns a list of integers; the integers are passed to the API call; the model produces a response; the engineer goes back to work. Most of the time, the abstraction holds, and there is no reason to look underneath it. Then one day, there is a reason. The model breaks on a name with an unusual suffix. The cost of a Korean prompt is twice that of an English one. A jailbreak works because of the way an emoji decomposes into surrogate pairs. None of these are mysterious once you have built a tokenizer; all of them are mysterious if you have not.

This piece is the version of the chapter on Byte-Pair Encoding (BPE) that fits in a blog post. The full chapter is in Volume I of the book, but the algorithm is small enough that a thirty-line implementation is enough to change what you see when you look at a prompt. Twenty minutes, a calculator-sized corpus, and a Python REPL are all you need.

The problem BPE was invented to solve

There are two obvious ways to turn text into integers, and both of them are bad. You can split on whitespace and assign each unique word an integer; the vocabulary explodes, every typo is a new token, and the model never sees unhappy and unhappily as related. Or you can split on characters; the vocabulary stays tiny, but every word becomes a long sequence of integers and the model has to learn the structure of the from scratch every time it sees the three letters in a row.

What you want is the middle: a vocabulary of frequent subword pieces, where common words come out as one token, common prefixes and suffixes come out as their own tokens, and rare or invented words decompose into a handful of pieces the model has seen before. BPE is one of the cheapest ways to build such a vocabulary, and it serves as the basis for the tokenizers in GPT-2, GPT-3, GPT-4, Llama, and most open-weight models you might fine-tune.

The algorithm in plain English

Start with each word as a list of characters, plus a special end-of-word marker to indicate where one word ends and the next begins. Count every adjacent pair of symbols across the entire corpus. Find the most frequent pair. Merge that pair, everywhere in the corpus, into a single new symbol. Repeat until the vocabulary reaches the size you want.

That is it. Four steps. The model that emerges has the property that the most common letter combinations in the training data become single tokens first, and the rare combinations are left as smaller pieces. the becomes one token because t-h-e is a frequent pair in your corpus. understandable becomes maybe under and stand and able because each of those substrings is frequent across the corpus, even if the full word is not.

Three deterministic merges, on three small words

Take a corpus of three words: low lower lowest. After the initial split into characters with an end-of-word marker (we will use _), the corpus looks like this:

l o w _
l o w e r _
l o w e s t _

Count the adjacent pairs across all three words. The pair (l, o) appears three times, and so does (o, w). After that, (w, e) appears twice; every other pair appears once. We pick (l, o) as the first merge (a tie is broken by ordering; the choice does not matter for the example).

After merge 1, the corpus is:

lo w _
lo w e r _
lo w e s t _

Now (lo, w) is the most frequent pair, with three occurrences. Merge it.

low _
low e r _
low e s t _

(low, e) is the most frequent now, with two occurrences. Merge it.

low _
lowe r _
lowe s t _

After three merges, the word low is a single token, the word lowest is the three tokens lowe s t _, and lower is the three tokens lowe r _. The shared prefix lowe is what the algorithm has discovered. The algorithm did not know in advance that lowe would be a useful token; it found it by counting.

The thirty-line implementation

The whole algorithm fits in a small Python file. The following script trains BPE on a corpus and prints each merge as it happens. Save it, run it, change the corpus, watch the merges change.

from collections import Counter

def get_pairs(word):
    """Return the list of adjacent pairs in a tokenized word."""
    return [(word[i], word[i + 1]) for i in range(len(word) - 1)]

def merge_pair(word, pair, replacement):
    """Replace every occurrence of pair in the tokenized word."""
    out = []
    i = 0
    while i < len(word):
        if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
            out.append(replacement)
            i += 2
        else:
            out.append(word[i])
            i += 1
    return tuple(out)

def train_bpe(corpus, num_merges):
    words = [tuple(w) + ("_",) for w in corpus.split()]
    merges = []
    for step in range(num_merges):
        counts = Counter()
        for word in words:
            for pair in get_pairs(word):
                counts[pair] += 1
        if not counts:
            break
        best = counts.most_common(1)[0][0]
        replacement = best[0] + best[1]
        words = [merge_pair(word, best, replacement) for word in words]
        merges.append((best, replacement))
        print(f"Merge {step + 1}: {best} -> '{replacement}'")
    return merges, words

corpus = "low lower lowest"
merges, vocab = train_bpe(corpus, num_merges=5)
print("Final tokenization:", vocab)

Run it on the corpus from the worked example and the first three lines of output will match the merges we computed by hand. Run it on a larger corpus, say a paragraph from a book, and the merges become a small autobiography of which letter combinations are common in your text.

What this changes about your prompts tomorrow

Once you have built the tokenizer, four things stop being mysterious. The first is why the cost of an English prompt and a Korean prompt of the same character count differ; English text is in the GPT tokenizer's training distribution and decomposes into one token per common word, while Korean has to fall back to longer subword splits. The second is why typos cost more tokens than the corrected version of the word; the typo is rare and decomposes into pieces. The third is why models sometimes treat compound words as if they were two unrelated concepts; the merge that would have unified them never reached the cutoff. The fourth is why putting variable content at the end of a prompt makes prefix caching cheap; the tokenizer is deterministic, and an unchanged prefix produces an unchanged sequence of tokens, which is what the cache keys on.

None of these is mysterious once the four-step algorithm has gone through your fingers on a tiny corpus. All of them are mysterious if it has not.

The book this excerpt is drawn from is two volumes long, and the BPE chapter is one of eighteen.

The companion repository is at github.com/ritesh-modi/inside-llm;

The script above is in the chapter-three folder, with a longer corpus and a few extensions (vocabulary serialization, a tokenize function for new text, and the byte-level fallback that real tokenizers use). Clone it, run it on your own paragraph, and the next prompt you write will look different.

Build a working asyncio event loop in 30 lines of plain Python

Ritesh — Mon, 04 May 2026 17:51:25 +0000

I will keep saying this until it stops being controversial: asyncio is small. The reason it feels large is that every tutorial introduces the keywords before the runtime they describe, leaving you to guess at what await is actually doing. Strip the keywords away, and the runtime fits in 30 lines of plain Python with no asyncio import.

This post walks through the toy event loop end to end. The code runs. Paste it, save, execute. By the end, you will have built a working asyncio runtime in your terminal and watched it interleave three jobs on a single thread.

The shape of the problem

A program with three jobs, each of which spends most of its time waiting. The naive version runs them one after the other and pays the sum of the waits. A program that overlaps the waits pays only the longest one.

serial:     [job A 2s] -> [job B 1s] -> [job C 3s]   total 6s
concurrent: [A 2s] [B 1s] [C 3s]   all overlapping   total 3s

Six seconds versus three. The work is identical. The only thing that changes is whether the jobs queue up behind each other or share the time.

A job is a generator that yields

A generator is a function that can pause itself and be resumed by its caller. The pause point is yield. The caller advances the generator with next(...).

def example():
    print("step 1")
    yield
    print("step 2")
    yield
    print("step 3")

>>> g = example()
>>> next(g)
step 1
>>> next(g)
step 2
>>> next(g)
step 3
Traceback (most recent call last):
  ...
StopIteration

That is the entire mechanism. A yield is a bookmark. The caller picks up a different generator, runs it for a while, and comes back to the bookmarked one when it feels like it. Hold on to that picture; it is what await will do later, in fewer letters.

A timer that needs three ticks

For the toy loop, a "wait" is a generator that yields a target wake-up time. The loop checks the time on each pass; once the wake-up time has passed, the generator is allowed to advance.

import time

def sleep(seconds):
    deadline = time.time() + seconds
    while time.time() < deadline:
        yield deadline

A job that "waits two seconds" is a generator that yields the deadline now + 2 until that deadline passes, then returns. The loop watches deadlines; jobs advance when their deadline arrives.

The loop in 30 lines

Here it is. Save it as toy_loop.py.

import time
from collections import deque

def run(jobs):
    """Run the given jobs until all of them finish.

    Each job is a generator. A job yields a deadline (a time.time()
    value) to mean "wake me up at or after this time". When a job
    returns (StopIteration), it is removed from the queue.
    """
    ready = deque((job, 0.0) for job in jobs)
    while ready:
        job, wake_at = ready.popleft()
        if time.time() < wake_at:
            ready.append((job, wake_at))
            time.sleep(0.001)   # avoid a tight CPU spin
            continue
        try:
            new_wake_at = next(job) or 0.0
            ready.append((job, new_wake_at))
        except StopIteration:
            pass

def sleep(seconds):
    deadline = time.time() + seconds
    while time.time() < deadline:
        yield deadline

Twenty-eight lines of code. No imports beyond time and deque. No asyncio. No threads. No futures. The whole runtime is a queue and a while loop.

The while loop pops the front entry. If the job's wake-up time is in the future, push it to the back, sleep one millisecond, continue. If the job is ready, advance it with next(...); the job runs until its next yield, returns the new wake-up time, and goes back on the queue. When next(job) raises StopIteration, the job is finished and does not return to the queue.

That is the whole runtime.

Run it

Add three jobs and a main block.

def fetch(name, delay):
    print(f"  {name} started, waiting {delay}s")
    yield from sleep(delay)
    print(f"  {name} done")

if __name__ == "__main__":
    start = time.time()
    run([
        fetch("A", 2.0),
        fetch("B", 1.0),
        fetch("C", 3.0),
    ])
    print(f"total: {time.time() - start:.2f}s")

yield from sleep(...) delegates to another generator. It is the same idea await will be later. Each yield from sleep flows up through fetch to the next(job) call in the loop.

$ python toy_loop.py
  A started, waiting 2.0s
  B started, waiting 1.0s
  C started, waiting 3.0s
  B done
  A done
  C done
total: 3.00s

Three seconds. Not six. Three jobs whose waits sum to six seconds finished in the time of the longest one. Single thread. CPU idle for almost all of it. That is concurrency.

This is what asyncio is

Now look at what we built and what asyncio adds on top.

asyncio replaces the time.sleep(0.001) polling with a real selector-based wait on file descriptors (epoll on Linux, kqueue on macOS, IOCP on Windows). The selector tells the OS, "wake me up when any of these sockets has data, or when the next deadline arrives", so the loop sleeps for exactly as long as it needs to and no longer. That is the only meaningful difference between this toy loop and the real one.

async def is def with one extra property: the function returns a coroutine object instead of running its body. The coroutine object is the same shape as a generator. It pauses at await.

await x is yield from x.__await__(). It is yield from with a different name and a slightly tighter contract.

asyncio.run(main()) is the same as while ready: loop, with proper exception handling, signal handling, and the selector mentioned above.

The asyncio source code is more than 30 lines long, but the extra lines cover corner cases (cancellation propagation, exception groups, the lost-task trap), not new mechanisms. The mechanism is what you just built.

Where to go next

If you want to go deeper, my book Asyncio from Ground Up opens with the same toy loop, then takes the lid off at the bytecode level (__await__, coro.send, what a coroutine frame holds), then introduces the real asyncio API as a polished version of what you already have.

The book is on Leanpub: https://leanpub.com/author/book/asyncio/home. More about my work at www.riteshmodi.com.

The technology is small. The runtime fits on a single screen of code. The barrier was the order in which it was taught to you.