When you read a ChatGPT answer, it feels like a mind at work. Sentences arrive shaped like reasoning. The impression is strong — and it is wrong.
Under the interface there is no reasoning. There is a strict mathematical pipeline whose only job is to calculate the probability of the next token — the next unit of text — one token at a time, through roughly 175 billion parameters. What you are watching is not a thought forming. It is a ranked list being computed, over and over again, extremely fast.
🎬 Full video: https://youtu.be/zWNakhjsUp0
Here is the full path of one piece of text through the machine.
1. Tokenization: text becomes numbers
Your text is sliced into subword units called tokens. Each token maps to an integer ID from a fixed vocabulary of about 50,000: "the" might be 914, "cat" might be 3542. The model never sees letters or words — only numbers.
2. Embeddings: meaning becomes geometry
A single number cannot hold meaning, so each token ID is expanded into a vector — an array of thousands of numbers (768 in GPT-2; 12,288 in GPT-3). Words with similar meanings end up close together in this space. Language has been translated into coordinates.
3. Positional embeddings: where the word sits
The pipeline processes everything in parallel, so it has no natural sense of word order. A second vector, encoding each token's position, is added to the token vector. The result carries both the "what" and the "where" of every word.
4. Attention: context rewrites every word
Attention lets each token look back at the tokens before it and weigh their relevance. Multiple attention heads do this in parallel for different kinds of relationships, pulling information from relevant earlier words directly into the current token's vector. After attention, no word means quite what it meant alone.
5. Feed-forward layers: the pattern engine, 96 times
The re-weighted vectors pass through a dense layer of weights that cross-references the features attention extracted. In GPT-3, the attention-plus-feed-forward block repeats 96 times. This is the bulk of the 175 billion parameters — a pattern-matching engine refining the vector until it encodes the cumulative structure of the whole prompt. And because the design is stateless, the entire cascade is re-run from scratch for every single token generated.
6. Softmax: from vectors to a ranked list
The final vector is decoded against all 50,000 vocabulary tokens by the softmax function, which turns raw scores into probabilities that sum to exactly 100%. The peaks of that distribution are the tokens the network calculates are most likely to come next.
7. Temperature: engineered randomness
If the system always picked the top probability, the text would loop and flatten. A parameter called temperature rescales the distribution: at 0 the choice is deterministic; around 0.8 the distribution flattens enough that lower-ranked tokens sometimes win. That calculated randomness is what reads to us as creativity.
The loop that writes every word
One token is chosen, appended to the text, and the entire pipeline fires again. Every word you have ever seen ChatGPT write cost a full pass through 175 billion parameters.
Why this matters
Once you see the machine, you use it differently. You stop asking it to "think" and start engineering inputs for a prediction engine: clearer context, better constraints, sharper examples. The tool does not change. Your results do.
Paragon AI MAG tracks revolutionary AI research and explains what it means — before it changes everything. Subscribe to get the next breakdown first: https://paragonaimag.substack.com
Top comments (0)