Large Language Models have quickly become part of everyday software development.
We ask them to explain code, debug errors, generate tests, write Python scripts, summarize documentation, or help us understand an unfamiliar codebase.
Within seconds, we get a response that can feel surprisingly natural.
But what actually happens during those few seconds?
Suppose you type:
What is a build system?
The model doesn't simply search through a database for a stored answer, and it doesn't generate the entire response in one shot.
At the heart of an autoregressive LLM is a deceptively simple task:
Given the tokens I've seen so far, what token should come next?
Getting to that prediction, however, involves several layers of computation.
At a high level:
Prompt
↓
Tokens
↓
Embeddings
↓
Transformer
↓
Logits
↓
Next Token
↓
Repeat
Let's follow that journey.
1. Everything Starts With the Prompt
Consider:
What is a build system?
Humans immediately recognize the words and their meaning.
A neural network needs numbers.
Before the model can process the question, the text passes through a tokenizer.
2. Tokenization: Breaking Text Into Pieces
A tokenizer divides text into smaller units called tokens.
Conceptually, our prompt might become:
["What", " is", " a", " build", " system", "?"]
This is only an illustration. Actual tokenization depends on the tokenizer used by the model.
A token isn't necessarily a complete word. It might represent:
- a complete word
- part of a word
- punctuation
- whitespace combined with text
- a number
- part of an identifier
- a programming-language symbol
Each token is mapped to an integer called a token ID.
Conceptually:
["What", " is", " a", " build", " system", "?"]
↓
[3923, 374, 264, 1975, 1887, 30]
The IDs above are illustrative.
The important part is the transformation:
Human-readable text has become a sequence of numbers the model can process.
But token IDs themselves don't capture useful semantic relationships.
The number 1975, for example, doesn't inherently explain what build means.
That's where embeddings enter the picture.
3. Embeddings: Turning Tokens Into Vectors
Each token is mapped to an embedding—a high-dimensional vector of numbers.
Conceptually:
build → [0.12, -0.47, 0.83, 0.21, ...]
These useful representations are learned during model training rather than manually assigned by engineers.
Embeddings give the neural network much richer representations to work with.
But language depends heavily on context.
Consider:
The developer restarted the build because it failed.
What does it refer to?
A human reader can connect it with the build.
The model needs a mechanism for modeling relationships between tokens as well.
That's where self-attention becomes important.
4. Self-Attention: Which Other Tokens Matter?
Self-attention is one of the core ideas behind the Transformer architecture.
Instead of treating every token independently, attention allows each token's representation to incorporate information from other relevant tokens in the context.
A useful mental model is that each token is asking:
Which other tokens should influence my representation right now?
Under the hood, tokens are projected into three important vectors:
- Query (Q)
- Key (K)
- Value (V)
An intuitive way to think about them is:
Query: What information am I looking for?
Key: What information might I match with?
Value: What information should I contribute if I'm relevant?
The standard scaled dot-product attention operation is:
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V
We'll explore the mathematics more deeply in a later article.
For now, the key idea is:
Attention allows the representation of a token to depend on its surrounding context.
Transformers also use multiple attention heads, allowing different kinds of relationships to be modeled in parallel.
5. Through the Transformer
Self-attention is part of the larger Transformer architecture.
Token representations pass through a stack of Transformer layers.
A simplified view:
Token Embeddings
↓
Self-Attention
↓
Feed-Forward Network
↓
Next Transformer Layer
↓
Self-Attention
↓
Feed-Forward Network
↓
...
Real Transformer blocks also contain mechanisms such as residual connections and normalization.
As the representations move through these layers, they become increasingly contextualized.
The representation associated with build, for example, is no longer simply a static representation of that token.
It now incorporates information about how build is being used in this particular context.
Eventually, the model reaches the central question:
What token should come next?
6. From Transformer Output to Logits
After processing the context, the model produces scores for possible next tokens.
These scores are called logits.
Imagine a tiny example:
| Possible Token | Logit |
|---|---|
| A | 7.2 |
| The | 5.8 |
| Build | 3.4 |
| Software | 2.7 |
| Banana | -4.1 |
A real model may have a vocabulary containing tens or hundreds of thousands of tokens.
That means the model produces a large vector of scores at every generation step.
Higher logits generally indicate more plausible next tokens.
These scores can then be converted into a probability distribution using softmax.
Conceptually:
| Token | Probability |
|---|---|
| A | 55% |
| The | 25% |
| Build | 10% |
| Software | 7% |
| Everything else | 3% |
Again, these numbers are illustrative.
Now the model has possible continuations and their relative likelihoods.
7. Choosing the Next Token
How the next token is chosen depends on the decoding strategy.
Generation can be influenced by techniques such as:
- temperature
- top-k sampling
- top-p sampling
- greedy or other decoding strategies
Suppose the model selects:
A
The sequence now becomes:
What is a build system? A
The model isn't finished.
It predicts again.
8. The Answer Grows One Token at a Time
The model now has:
What is a build system? A
It predicts another token.
Perhaps:
build
Now:
What is a build system? A build
Then:
system
Then:
is
Then:
a
The process continues until a stopping condition is reached.
Eventually, we might get:
A build system is software that automates the process of compiling, linking, testing, and packaging software.
This is autoregressive generation.
At each step:
Current Context
↓
Transformer
↓
Next-Token Distribution
↓
Select Token
↓
Append Token
↓
Repeat
This is one of the most important ideas for understanding LLMs:
The answer isn't normally generated all at once. It grows token by token.
9. What Happens When an LLM Generates Code?
The same basic process applies when an LLM generates software.
Suppose we ask:
Write a C++ function that returns the larger of two integers.
The model might produce:
int maxValue(int a, int b) {
return a > b ? a : b;
}
From our perspective, it looks as though the model understood the requirement and produced a complete C++ function.
But the fundamental generation mechanism hasn't changed.
The output is still produced as a sequence of tokens.
Conceptually:
int
→ maxValue
→ (
→ int
→ a
→ ,
→ int
→ b
→ )
→ {
→ return
→ a
→ >
→ b
→ ?
→ a
→ :
→ b
→ ;
→ }
The actual token boundaries depend on the tokenizer.
What's interesting is how the existing context changes what becomes likely next.
After:
int maxValue(
tokens associated with a C++ function signature become plausible.
Later, after:
return a > b
the prompt, previously generated code, and patterns learned during training make certain continuations more plausible than unrelated ones.
The model doesn't suddenly switch to a dedicated C++ code-generation engine.
The same fundamental loop continues:
Prompt + Generated Code
↓
Transformer
↓
Next-Token Distribution
↓
Select Token
↓
Append Token
↓
Repeat
The same principle applies to Python, Rust, Java, SQL, shell scripts, and other programming languages.
Of course, generating a small function is much easier than generating or modifying a large software component.
For larger tasks, the model may need to maintain relationships involving:
- function signatures
- types
- variable names
- APIs
- control flow
- previously generated code
- programming-language syntax
- user requirements
across a much larger context.
That raises an interesting question:
If an LLM generates code token by token, how can it produce syntactically and sometimes logically consistent programs?
That's a topic worth exploring separately.
For now, the important takeaway is:
Natural language and source code look very different to us, but to an LLM they ultimately become sequences of tokens to be modeled and predicted.
10. Doesn't Repeating the Transformer Get Expensive?
Yes.
Suppose the model has already processed a long prompt and generated hundreds of tokens.
Recalculating all of the attention information for every previous token from scratch for each new token would involve substantial redundant computation.
One important optimization is the KV cache.
During self-attention, the model computes Key and Value representations for tokens it has already processed.
During autoregressive decoding, those representations can be cached and reused.
Conceptually:
Without KV Cache
Recompute previous K/V
↓
Process next token
↓
Generate
↓
Repeat
With caching:
With KV Cache
Reuse previous K/V
↓
Compute K/V for new token
↓
Generate
↓
Repeat
This reduces redundant computation during generation, although the growing cache also consumes memory as the context becomes longer.
KV caching deserves its own deeper discussion later.
11. From Prompt to Prediction
We can now put the complete journey together:
User Prompt
↓
Tokenization
↓
Token IDs
↓
Embeddings
↓
Transformer Layers
↓
Self-Attention
↓
Contextual Representations
↓
Logits
↓
Next-Token Distribution
↓
Token Selection
↓
Append Token
↓
Repeat
↓
Final Response
From the user's perspective, the interaction looks incredibly simple:
Ask a question → Get an answer
Behind that interaction is a repeated sequence of numerical transformations and next-token predictions.
12. Is an LLM Really "Just a Next-Token Predictor"?
You've probably heard this description:
LLMs are just next-token predictors.
There is an important truth behind it.
Next-token prediction is central to the training objective of many autoregressive LLMs, and autoregressive generation produces output by repeatedly predicting subsequent tokens.
But the word just hides most of what makes these systems interesting.
Predicting what follows:
The sky is...
may seem straightforward.
Now consider asking the same model to:
- explain an algorithm
- generate a C++ implementation
- translate Python into Rust
- summarize a technical document
- analyze an error message
- reason about unfamiliar code
The output still emerges token by token.
But producing useful next-token distributions requires a sophisticated learned representation of the context.
The simple question is:
What should come next?
The complicated part is the enormous learned system answering it.
Transformer layers, attention mechanisms, embeddings, billions of learned parameters, and highly optimized inference infrastructure all contribute to those predictions.
Then the model does it again.
And again.
And again.
Final Thoughts
An LLM can seem mysterious when all we see is:
Prompt → Answer
But once we look inside, the process becomes easier to reason about.
Text becomes tokens.
Tokens become numerical representations.
Transformers use attention to build contextual representations.
Those representations produce scores for possible next tokens.
A token is selected.
Then the process repeats.
That basic loop is the foundation for much of what we experience when interacting with modern language models.
Once these pieces become familiar, an LLM starts looking less like a mysterious black box and more like something engineers are used to understanding:
A very large system built from smaller, understandable components.
What's Next?
We've followed an LLM from the moment a prompt enters the system to the point where a response emerges, one token at a time.
But we moved quickly through the very first step: tokenization.
How does an LLM decide where one token ends and another begins? Why can a single word become multiple tokens? And what happens when the input is source code rather than ordinary English?
In the next article in this LLM Internals series, we'll take a closer look at:
Tokenization: How LLMs Break Text and Source Code Into Tokens
Once we understand that first transformation, we'll be ready to move deeper into what happens inside the model.
Top comments (0)