Building a Character-Level Bigram Language Model from Scratch with PyTorch

TL;DR: I built the simplest possible neural language model — a character-level bigram — from scratch using PyTorch. No transformers, no attention, no abstractions. Just 79 characters, a counting matrix, and multinomial sampling.

---

The Problem

Every time I opened a tutorial about LLMs, it started with "Here's how attention works" or "Let's build a mini GPT."

But I wanted to understand the foundation. What happens before attention? How does text even become numbers? How does a model learn to predict the next token without any neural network at all?

So I went all the way down to the simplest possible language model: the bigram.

What's a Bigram Model?

A bigram model predicts the next character based only on the current character. That's it. No context window of thousands of tokens — just "given character X, what's the most likely character to follow?"

It's the Hello World of language modeling.

The Implementation

Step 1: Raw Text → Character Vocabulary

with open("Oz.txt", 'r', encoding='utf-8') as f:
    text = f.read()

chars = sorted(set(text))
print(len(chars))  # 79 unique characters

79 unique characters: letters (a-z, A-Z), digits, punctuation, spaces, and newlines. A tiny vocabulary compared to GPT-4's ~100K token BPE vocabulary.

Step 2: Character Encoding

string_to_int = {ch: i for i, ch in enumerate(chars)}
int_to_string = {i: ch for i, ch in enumerate(chars)}

encoder = lambda s: [string_to_int[c] for c in s]
decoder = lambda l: ''.join([int_to_string[i] for i in l])

data = torch.tensor(encoder(text), dtype=torch.long)
print(data.shape)  # torch.Size([231881])

Every character maps to an integer 0-78. The entire text of "The Wonderful Wizard of Oz" becomes a 231,881-element tensor.

Step 3: Train/Val Split

n = int(0.8 * len(data))
train_data = data[:n]
val_data = data[n:]

Standard 80/20 split. Simple and effective.

Step 4: Input-Target Pairs

For next-token prediction, we need (input, target) pairs where the target is the next character after the input sequence:

block_size = 8
x = train_data[:block_size]
y = train_data[1:block_size+1]

for t in range(block_size):
    context = x[:t+1]
    target = y[t]
    print(f"when input is {context} the target: {target}")
# when input is tensor([0]) the target: 0
# when input is tensor([0, 0]) the target: 43
# when input is tensor([0, 0, 43]) the target: 67
# ...

This is the core pattern of autoregressive language modeling: given a sequence, predict what comes next. Every modern LLM — from GPT to LLaMA to Mistral — uses this same fundamental approach. The only difference is scale.

Step 5: GPU vs CPU Benchmark

Before building the actual bigram probability matrix, I benchmarked PyTorch's matrix multiplication on CUDA vs NumPy on CPU:

torch_rand1 = torch.rand(100, 100, 100, 100).to(device)
torch_rand2 = torch.rand(100, 100, 100, 100).to(device)

# CUDA: 0.017 seconds
# NumPy CPU: 0.134 seconds

8x speedup on GPU — even for random matrix multiplication. Hardware acceleration is not optional at scale.

What's Next?

A bigram model is just the start. The next step is adding an embedding layer, then a Transformer block, then scaling up. But I wanted to build from first principles — understanding the foundation before adding complexity.

Key Takeaways

1. Language modeling starts with counting — the bigram probability matrix is just normalized counts
2. Character-level tokenization is simple but powerful: 79 tokens cover an entire novel
3. The input-target prediction pattern is universal — from bigrams to GPT-4
4. GPU acceleration matters even at toy scale (8x speedup on random matrices)
5. Building from scratch teaches you what frameworks abstract away

Resources

• Code: github.com/HENI-MOHAMED
• Dataset: The Wonderful Wizard of Oz (public domain)

---

Built with PyTorch 2.x on a local machine with CUDA support. Full notebook available on request.

What's a fundamental ML concept you've built from scratch recently? I'd love to hear what foundations you explored.