I'm continuing my Forward Deployed Engineering and Generative AI learning journey.
In my previous lecture, I learned an important engineering lesson:
Don't start with the technology. Start with the problem.
This lecture moved one level deeper.
I wanted to understand what actually happens inside an LLM when I type a prompt and it starts generating a response.
Before studying this, it was easy to think about an LLM like this:
Prompt
↓
LLM
↓
Answer
That's useful at the application level, but it hides almost everything happening inside the model.
The mental model I have now is very different.
It starts with a surprisingly simple idea
At the generation level, an autoregressive LLM repeatedly predicts the next token.
Suppose we provide:
The capital of India is
The model doesn't need to magically generate the complete sentence in one step.
Instead, it estimates which token should come next.
Conceptually:
The capital of India is
↓
Delhi
Then the new sequence becomes:
The capital of India is Delhi
The model predicts again.
And again.
And again.
So the basic generation loop is:
Context
↓
Predict next token
↓
Select token
↓
Append token
↓
Predict again
That simple idea helped me understand many of the other concepts in the lecture.
But what exactly is a token?
This was another important correction to my mental model.
A token isn't necessarily a complete word.
Depending on the tokenizer, a token can represent:
- A complete word
- Part of a word
- Punctuation
- A symbol
- Another frequently occurring text fragment
So:
Token ≠ Word
This matters because LLMs operate on tokens rather than directly operating on human concepts like "sentences" or "paragraphs."
Text has to become numbers
Neural networks operate mathematically.
So the text needs to be converted into numerical representations.
Conceptually:
Text
↓
Tokens
↓
Token IDs
↓
Embeddings
Token IDs identify tokens.
But an ID by itself doesn't represent meaning.
The model therefore uses embeddings: learned numerical vectors that can be processed by the neural network.
This distinction was useful:
Token ID
→ Which token is this?
Embedding
→ Numerical representation used for processing
Then I ran into a problem: order matters
Consider:
Dog bites man.
and:
Man bites dog.
Almost the same words.
Completely different meaning.
So the model needs information about position.
It needs to understand not only:
What token is this?
but also:
Where does this token occur?
That gives us the idea of position information.
Then context makes everything more interesting
Consider the word:
bank
In:
I deposited money at the bank.
it has one meaning.
In:
We sat on the bank of the river.
it has another.
The token itself hasn't changed.
The context has.
This leads to an important idea:
A token shouldn't always be interpreted independently of the other tokens around it.
And this is where attention becomes important.
Attention: which other tokens matter?
Consider:
The animal didn't cross the street because it was too tired.
To interpret it, other tokens in the sentence matter.
Attention provides a mechanism for determining which other tokens should influence the current token's representation.
At a high level, I think of it as:
Current token
↓
Which other tokens are relevant?
↓
Gather useful information
↓
Update representation
The lecture introduces Query, Key and Value as the three learned representations used in this mechanism.
The search analogy helped me:
Query
→ What am I looking for?
Key
→ What kind of information do I contain?
Value
→ What information can I provide?
They aren't literally English questions inside the model. They are mathematical vectors.
But the analogy helped me understand their roles.
Attention isn't the Transformer
Another misconception I had to correct:
Transformer = Attention
Not quite.
Attention is an important component of a Transformer.
A Transformer contains repeated processing blocks.
Conceptually:
Input
↓
Transformer Block
↓
Transformer Block
↓
Transformer Block
↓
...
↓
Final Representation
Each layer can progressively refine the representations.
So instead of imagining the model as one giant operation, I can think about it as repeated transformations of the information.
Finally: how does the model choose the next token?
After the Transformer processes the context, the model produces scores for possible next tokens.
These are called logits.
Then Softmax converts those scores into probabilities.
So:
Context
↓
Transformer
↓
Logits
↓
Softmax
↓
Probability distribution
For example:
Delhi 92%
Kolkata 2%
Punjab 2%
Mumbai 1%
...
Now we have another question:
Which token should actually be selected?
That's where decoding comes in.
The model and the decoder are different things
This distinction was particularly useful to me.
The model produces:
Probability distribution
The decoding strategy decides:
Which token gets selected?
For example, greedy decoding chooses the highest-probability token.
Sampling can choose among possible tokens according to their probabilities.
Temperature changes how sharp or flat the distribution becomes before sampling.
So the pipeline becomes:
MODEL
↓
Probability Distribution
↓
DECODING STRATEGY
↓
Selected Token
And then the whole process starts again
This is probably the most important mental model I took from the lecture.
The model selects one token.
That token is appended to the sequence.
The updated sequence becomes the context for the next prediction.
Then the process repeats.
Prompt
↓
Predict
↓
Select
↓
Append
↓
Predict
↓
Select
↓
Append
↓
...
This is autoregressive generation.
And suddenly, streaming text from an LLM makes more sense too.
The application can expose generated tokens or chunks as they become available instead of waiting for the entire response to be generated.
My mental model now
Before:
Prompt → LLM → Answer
After this lecture:
Text
↓
Tokens
↓
Token IDs
↓
Embeddings + Position
↓
Transformer + Attention
↓
Contextual Representations
↓
Logits
↓
Probabilities
↓
Decoding
↓
Next Token
↓
Append
↓
Repeat
That is a much more useful mental model for me as an engineer.
It also gives me a better foundation for understanding the AI systems I'll eventually build around these models.
What I'm taking forward
The biggest lesson from this lecture isn't that I memorized definitions for:
- Tokens
- Embeddings
- Attention
- Transformers
- Logits
- Softmax
- Temperature
It's that these concepts fit together into one generation process.
An LLM repeatedly uses context to predict what token should come next.
Everything else helps make that process possible and useful.
I'm continuing to study Forward Deployed Engineering and Generative AI alongside System Design.
The next step is to keep moving from:
understanding the model → understanding AI applications → understanding production AI systems.
Top comments (0)