"My cat cant read this"
Held my laptop up, my cat stared at the screen for two seconds, then walked off like I'd personally offended her. Zero understanding. Fair enough, she's a cat.
But here's the thing that actually gets interesting, I can't fully explain how I read either. I'm doing it right now, obviously, but no neuroscientist on the planet can fully explain how my brain turns these squiggles into "oh, a cat joke." That one's still unsolved.
ChatGPT though, we can actually try to understand, not every step of it start to finish but can have an over all idea, no mystery here. So let's go through it properly, follow one sentence on its entire journey, from raw text to a reply showing up on our screen.
You type this into ChatGPT:
"My cat cant read this"
...and it immediately starts judging you(pun intended)
So how did it know? Let's actually go underneath the chat window, because that's the whole point of this post, not what ChatGPT is, but what it's actually doing to your sentence in the half second before it replies.
Underneath, it's a model doing exactly one job, look at some text, guess what word comes next. That's genuinely it. There's no separate grammar checker bolted on, no dictionary it's flipping through. Catching your missing apostrophe, understanding what you meant, replying like a smug know it all, all of that comes out of "guess the next word," just done at a scale that's hard to picture. Let's watch it happen on your actual sentence.
Step 1: Chopping the sentence up (Tokenization)
Computers can't read English, not even a little. All they work with is numbers. So before anything else happens, your sentence gets cut into small pieces called tokens , and each piece gets a number.
My cat cant read this → [My] [cat] [can] [t] [read] [this]
"cant" gets split weird, into [can] and [t]. That's because "cant" isn't actually a real word, it's missing its apostrophe, so the tokenizer, which only knows the chunks it was trained on(its vocabulary), does its best and breaks it apart. Every token maps to a fixed number, and that mapping doesn't change based on our intended meaning.
Step 2: Vector Embeddings
A number alone tells you nothing. 47 doesn't mean anything about "cat". So each token's number gets swapped for something bigger, a long list of numbers called an embedding , which works like a personality profile for that word. Words with related meaning end up with similar looking profiles. "cat" and "dog" sit close together numerically. "cat" and "computer" don't.
These profiles aren't invented fresh for our sentence. They're premade. Learned once during training, and just sitting there in a giant table afterward we can visualize it as words positioned in 3d space such that the distance between words represent how closely they are related. When our sentence comes in, the model isn't building a personality for "cat" on the spot, it's pulling "cat"'s profile off the shelf, already fully formed from having seen the word a billion times before during training.
Step 3: Remembering the order (Positional Encoding)
Small problem, the next step processes every token at once, together, not one after another. Which is fast, but it also means word order gets lost by default. "My cat cant read this" and "this cant read my cat" would otherwise look the same to it.
So before that step, each token gets a small stamp added to its profile, basically marking "I was word 1," "I was word 3," and so on, so the order survives even though everything's being processed together.
Step 4: Where the actual understanding happens (Self Attention)
Quick reminder before this part, by now every token isn't a word anymore, it's the number profile from Step 2, already stamped with its position from Step 3. That stamp matters here because attention checks every token against every other token all at once, not in order, so without it, the model would have no way to tell "My cat cant read this" apart from "this cant read my cat", same words, same profiles, totally different meaning.
When "cant" "looks at" every other word, it's doing a direct number comparison, checking how closely its numbers align with "cat"'s, "read"'s, and "My"'s. The closer the alignment, the more of that word's numbers get mixed into "cant". These alignments come from patterns the model picked up during training, but here's the honest part, we can't actually read those numbers and say "this is the grammar bit" or "this is the meaning bit." Nobody can point at them and narrate what each one means in plain English. What we can say is what happens mechanically, closely aligned words get blended in heavily, weakly aligned ones barely at all, and this runs for every token at once, all comparing against each other simultaneously. That's also why it doesn't lose track in long sentences, word 2 and word 200 can compare directly in one step, instead of that information passing hand to hand through every word in between.
Step 5: A private pass (Feed Forward Layer)
After the mixing in Step 4, each token's profile is just a blend of other tokens' numbers. This step takes that blend and reworks it through another formula, also learned during training, using only what this one token already has, no borrowing from others this time. As for what this reworking actually captures, same honest answer as before, we can't point at it and say in plain words what it's doing. What we can say is the shape of it, Steps 4 and 5 together repeat many times, stacked on top of each other, and across all those rounds each token's profile gradually shifts from a generic word into a word shaped by this specific sentence. The exact thing each round contributes isn't something anyone can narrate, but the end result is that every token's numbers end up reflecting the whole sentence around it, not just the word on its own.
Step 6: Actually picking a word (Softmax)
After all those rounds of Steps 4 and 5, the model has one final set of numbers for the position where the next word should go. Now it has to turn that into an actual word. It compares those numbers against every word in its vocabulary, tens of thousands of them, and produces a raw score for each one, how well each word fits given everything so far. Those raw scores are messy and hard to work with, some could even be negative, so they get run through softmax, which just squashes them all into clean percentages that add up to 100%. Something like "You" at 38%, "That" at 20%, "Well" at 9%, and so on down the list.
Then it picks a word from that list, usually leaning toward the higher percentages, adds it to the sentence, and runs the entire thing again from the start, tokenizing, embedding, comparing, reworking, all of it, to figure out the next word after that. One word at a time, which is why you watch the reply type itself out instead of appearing all at once. It's not being revealed to you from somewhere, it's being built, live, one word at a time, right in front of you.
Quick recap since this is the part that trips people up, the token list, the embedding profiles, and all the internal formulas are fixed. Learned once during training, frozen after that. None of it changes while you are chatting, no matter what you type. And to answer the obvious question, no, the model isn't learning from your conversation as you go. Your chats may still be collected and used to help train future versions, depending on the service and your settings, but even then that training happens later, offline, not while you type. It does improve over time, but only when the company trains and releases a newer version, and the version you're talking to right now stays exactly the same its whole life until it's swapped for a new one. What's fresh every single time is the computation itself, your specific words getting run through those frozen numbers in a way that's never happened before, because your exact may sentence have never existed before either.
Top comments (0)