Building a 200M parameter LLM from scratch in PyTorch

gulding — Thu, 03 Sep 2026 09:20:30 +0000

It's really easy to spin up Unsloth and fine-tune Llama 3 in an afternoon. I wanted to see what happens when you don't do that.

I wanted to write the BPE tokenizer, implement RoPE, code the SwiGLU feed-forward blocks, and write the training loop in pure PyTorch from absolute scratch.

The result is EmsyAI (V4), a 196M parameter model I trained on a consumer GPU.

The Setup

Active Parameters: 196.7M (roughly 180M in the transformer blocks, the rest in the ~16k vocab embeddings)
Training Tokens: 1.96 Billion
Context Window: 4,096 tokens
Hidden Dimension: 1,024
Attention: GQA (16 Query / 4 KV)
FFN Dimension: 2,816

Pretraining

I trained this for 15,000 steps using mixed-precision FP16. Pretraining is terrifying because you're constantly waiting for the loss to suddenly spike to NaN and ruin hours of compute.

Somehow, the curve actually held. It decayed down to a validation perplexity of 5.86.

Watching the raw output go from random token garbage at step 100 to recognizable Python syntax at step 10,000 is a bizarrely satisfying experience.

HumanEval and Data Contamination

After pretraining, I instruction-tuned the base model with a 2.3M parameter LoRA adapter on the CodeAlpaca dataset. Then I ran it against OpenAI's HumanEval benchmark.

It scored exactly 0.0%.

That stings for a second, but it's the empirically expected baseline. A model this small, trained on just 2 billion tokens, isn't going to solve multi-step algorithmic puzzles.

The interesting part was looking at the training data itself. Using the standard 13-gram exact-match threshold from the GPT-3 paper, I audited CodeAlpaca and found 1,644 sequences that leaked HumanEval test logic.

The fact that EmsyAI still scored 0% despite seeing those leaked solutions during fine-tuning doesn't prove it's a generalization genius. It just means the model—and specifically the tiny 2.3M LoRA adapter—simply lacked the capacity to rote-memorize those sequences verbatim. The contamination didn't help it cheat, because it just couldn't remember the answers anyway.

What broke (and the plan for V5)

Scaling this architecture up to 196M exposed a lot of edge cases that I completely missed at the 88M scale.

Attention logit explosions
At hidden dim 1024, the attention logits would occasionally spike wildly. Standard architectures don't normalize queries and keys before the dot product, which becomes a huge stability liability at scale. I originally thought recent releases like Qwen 2.5 and Gemma 2 fixed this directly, but that was wrong—Gemma 2 used attention-logit soft-capping. It wasn't until OLMo 2, and later Gemma 3 (inspired by Meta's Chameleon) and Qwen 3, that QK-Norm was widely credited for stabilizing training. V5 will implement QK-Norm to stop these spikes for good.

Tokenizer byte collisions
I wrote the BPE tokenizer from scratch, but I messed up the fallback byte mappings. Bytes 0-3 currently overlap with my special control tokens (<|endoftext|>, etc.). It didn't crash the training, but it's a silent bug that degrades performance. V5 is getting a total tokenizer rebuild, and I'll be doubling the vocab from 16k to 32k.

Cross-document attention poisoning
Right now, my dataloader blindly concatenates text files to hit the 4096 context length. The model spends a lot of compute attending across completely unrelated documents that just happen to share the same training window. I need to write document-aware packing with proper attention masking.

Try it locally

I exported the final weights to GGUF, so you can run it in Ollama if you want to see a 196M model try (and fail) to write FizzBuzz.

git clone https://github.com/gulding/EmsyAI.git
cd EmsyAI
huggingface-cli download gulding/EmsyAI emsyai-v4-instruct-f32.gguf --local-dir .
ollama create emsyai-v4 -f Modelfile
ollama run emsyai-v4