📰 Originally published on Securityelites — AI Red Team Education — the canonical, fully-updated version of this article.
🗣️ LLM BASICS FOR BEGINNERS FREE
Day 2 of 5 · 40% complete
There’s a number I think about whenever someone asks me how LLMs got so good at sounding human. GPT-3, released in 2020, was trained on about 570 gigabytes of text. That’s roughly 300 billion words. If you read for 12 hours a day, every day, it would take you about 34,000 years to read that much. GPT-4 was trained on even more.
That scale is why LLMs are impressive. But the scale also creates problems nobody tells beginners about. When you train on everything ever written on the internet, you train on the good, the bad, the outdated, and the false. You train on biased opinions, wrong facts, old information, and content written by people who had no idea what they were talking about.
Today I’m going to explain how LLMs actually learn — the training process, what tokens are, what parameters are, and why the knowledge cutoff means your LLM might be living in the past. By the end you’ll understand why an LLM’s quality depends entirely on what went into it.
🎯 What You’ll Learn in Day 2
What training data is and why it’s the most important ingredient
What tokens are — explained simply with real examples
What parameters are (in plain English, no maths)
Why the knowledge cutoff means your LLM is frozen in time
Why messy training data creates an LLM with blind spots and biases
⏱ 20 min read · 3 exercises · Just a browser needed
📋 Before You Start:
- Completed Day 1: What Is a Large Language Model?
- Remember: an LLM predicts the next word — it’s not thinking, it’s pattern-matching
- Remember: ChatGPT is a product; the LLM is the engine inside it
How LLMs Learn From Text — Day 2 of 5
- Training Data — The Raw Ingredient
- Tokens — What an LLM Actually Reads
- Parameters — Where the Learning Lives
- The Training Process — How It Goes From Nothing to Brilliant
- The Knowledge Cutoff — Your LLM Is Frozen in Time
- Why Data Quality Is Everything
- Questions and Answers
Day 1 explained what an LLM is and what it does. Today we go inside the learning process. Understanding this makes Day 3 (why LLMs make mistakes) make complete sense — because almost every LLM failure traces back to something about how it learned. Also: our email breach checker is a great example of an AI-powered security tool — the kind of tool that uses trained models to recognise patterns. Understanding training is understanding how these tools work too.
Training Data — The Raw Ingredient
Every time I explain LLMs to someone new, I start with training data. Not tokens. Not parameters. Training data. Because everything — the good outputs, the bad outputs, the biases, the blind spots — traces back to this one thing. Training data is the raw ingredient. Get it right and you get a brilliant LLM. Get it wrong and you get a confidently wrong one.
Before an LLM can do anything, it needs to learn. And to learn, it needs examples. Billions and billions of examples of human text. That collection of examples is called training data.
What goes into LLM training data? The honest answer is: most of the publicly available internet. Web pages. Wikipedia articles. Books. News articles. Code repositories. Reddit threads. Academic papers. Customer reviews. Subtitles from TV shows and films. Forum posts. Recipes. Legal documents. Technical manuals.
I think of training data as the LLM’s entire education. A human child learns language by hearing and reading examples over years. An LLM learns language by processing billions of text examples in weeks of compute time. The human child ends up knowing language shaped by their family, teachers, and community. The LLM ends up knowing language shaped by whatever was most common on the internet.
Three things about training data that really matter:
Volume: More text means more patterns learned. GPT-4-scale models are trained on trillions of words. That staggering volume is why they can discuss almost any topic — because almost every topic appeared somewhere in the training data.
Diversity: The text needs to cover many topics, styles, and languages for the LLM to generalise. An LLM trained only on English Wikipedia would struggle with casual conversation, poetry, and any language other than English. Good training data covers huge variation.
Quality: Garbage in, garbage out. If the training data contains lots of wrong information, outdated information, or biased content, the LLM learns those wrong, outdated, and biased patterns. The internet contains a lot of each. We’ll look at this on Day 3.
Tokens — What an LLM Actually Reads
Here’s something I find most people are surprised to learn: an LLM doesn’t actually read words. It reads tokens. A token is a chunk of text — usually 3 to 4 characters in English, roughly three-quarters of a word on average.
📖 Read the complete guide on Securityelites — AI Red Team Education
This article continues with deeper technical detail, screenshots, code samples, and an interactive lab walk-through. Read the full article on Securityelites — AI Red Team Education →
This article was originally written and published by the Securityelites — AI Red Team Education team. For more cybersecurity tutorials, ethical hacking guides, and CTF walk-throughs, visit Securityelites — AI Red Team Education.
Top comments (0)