DEV Community: Adham Hewala

Transformers Explained: Encoder, Decoder, and How GPT Generates Text

Adham Hewala — Sat, 05 Sep 2026 11:27:36 +0000

The Transformer diagram looks simple until you try to follow what actually happens to one token inside it.

You see embeddings, attention, Q, K, V, Add & Norm, FFN, another attention block, logits, and finally some probabilities.

Then you look at GPT and notice something even more confusing:

Where is the Encoder?

The original Transformer has an Encoder and a Decoder.

GPT doesn't.

Yet GPT is built on the Transformer architecture.

So rather than treating these as separate topics, it makes more sense to follow the architecture from the bottom up and see how the pieces connect.

1. The Original Transformer: Encoder + Decoder

The Transformer was introduced in the 2017 paper "Attention Is All You Need."

The original architecture was designed for sequence-to-sequence tasks such as translation.

At a high level:

Input sequence
      ↓
   Encoder
      ↓
Encoded representation
      ↓
   Decoder
      ↓
Output sequence

For example:

"I love programming."
          ↓
       Encoder
          ↓
   contextual representation
          ↓
       Decoder
          ↓
"J'aime programmer."

The Encoder's job is to process the input and build useful contextual representations.

The Decoder's job is to use those representations to generate the output.

This distinction matters because modern Transformer models don't all use both sides.

There are three common configurations:

Encoder-only

Used primarily for understanding or representation.

Example: BERT

Encoder–Decoder

Used for transforming one sequence into another.

Examples: the original Transformer, T5

Decoder-only

Used primarily for autoregressive generation.

Examples: GPT-style models

So "Transformer" describes the architecture family, not one fixed model design.

2. Text Becomes Tokens, Then Vectors

Before the Encoder or Decoder can do anything, text has to become numerical data.

Suppose the input is:

"Translate this sentence into French."

A tokenizer breaks it into tokens:

["Translate", " this", " sentence", " into", " French", "."]

The exact tokenization depends on the tokenizer.

A token can be a complete word, part of a word, punctuation, or another frequently occurring piece of text.

Those tokens are then mapped to integer IDs:

[18472, 351, 9281, 417, 6321, 13]

But the IDs themselves don't contain meaning.

They're simply indexes into the model's vocabulary.

So we need another step:

Token ID
   ↓
Embedding
   ↓
Vector

A token such as "cat" might become a vector like:

[0.21, -0.73, 0.42, 0.18, ...]

In a real model, that vector can contain hundreds or thousands of values.

This is the point where the model stops dealing with discrete symbols and starts working with continuous numerical representations.

3. Position Matters

There's another problem.

A Transformer needs to know the order of the tokens.

Consider:

Dog bites man.

and:

Man bites dog.

The same basic words are present, but the meaning is obviously different.

Attention alone doesn't inherently tell the model that one token came before another.

That's why Transformers use positional information.

In the original Transformer, this was done with sinusoidal positional encodings.

Modern architectures often use other approaches, such as Rotary Positional Embeddings (RoPE).

The exact implementation can vary, but the purpose is the same:

Give the model information about where tokens occur and how their positions relate to one another.

Conceptually:

Token embedding
      +
Position information
      ↓
Position-aware representation

Now the sequence is ready to enter the Transformer blocks.

4. Self-Attention

Now we get to the part that made Transformers such a big deal.

Consider:

"The animal didn't cross the road because it was tired."

When the model processes "it", information about "animal" may be useful.

The model needs a way to decide which other tokens matter to the current token.

That's what self-attention does.

A useful mental model is:

Each token can look at the other tokens in the sequence and determine which ones are relevant to its current representation.

And this is where Query, Key, and Value come in.

5. Query, Key, and Value

The names sound more complicated than they really are.

Suppose the input representations are stored in a matrix:

The model uses three learned weight matrices:

W_Q
W_K
W_V

to calculate:

Q = XW_Q
K = XW_K
V = XW_V

So Q, K, and V aren't three magical objects attached to every word.

They're learned projections of the current token representations.

A useful intuition is:

Query

What information am I looking for?

Key

What information do I contain?

Value

What information should I provide if I'm relevant?

That gives us a way to compare a token with the rest of the sequence.

6. Attention Scores

Now we need to measure how strongly tokens should interact.

This starts with:

The dot product between a Query and a Key gives us a compatibility score.

A larger score means the two representations are more aligned.

A smaller score means they are less aligned.

You can think of it conceptually as:

Current token
     ↓
   Query
     ↓
compare with
     ↓
Keys from other tokens
     ↓
attention scores

Suppose a token produces scores like:

The      → 0.2
animal   → 2.4
road     → 0.7
tired    → 1.1

At this stage, these are just raw scores.

We still need to turn them into useful weights.

7. Scaling Before Softmax

The full scaled dot-product attention equation is:

That division by:

isn't there for decoration.

As the dimensionality of the vectors grows, the dot products can also grow in magnitude.

Very large values can make Softmax extremely sharp, which can make optimization difficult.

Scaling keeps the scores in a more useful numerical range before Softmax is applied.

So the pipeline is:

QKᵀ
 ↓
Scale
 ↓
Softmax
 ↓
Attention weights

Each step exists for a reason.

8. Softmax Turns Scores Into Weights

Softmax converts the attention scores into a normalized distribution.

For example:

The      → 0.06
animal   → 0.60
road     → 0.11
tired    → 0.23

The values sum to approximately 1.

Now we can use them as weights when combining the Value vectors.

So the model is effectively giving different amounts of influence to different tokens:

The      → small influence
animal   → large influence
road     → small influence
tired    → moderate influence

This is a useful intuition, although we shouldn't take it too literally as the model "thinking" about words.

9. The Attention Output

Once we have the attention weights, we use them to calculate a weighted combination of the Value vectors.

Conceptually:

Attention weights
        ×
Value vectors
        ↓
New representation

If "animal" has a large attention weight, its Value contributes more to the resulting representation.

The result is a new representation that contains information gathered from the surrounding tokens.

That is the important idea behind self-attention:

A token can build a new representation using information from other tokens.

10. Multi-Head Attention

One attention mechanism gives the model one learned way to look at relationships.

Transformers use Multi-Head Attention, which runs several attention mechanisms in parallel.

Conceptually:

                 Input
                   │
       ┌───────────┼───────────┐
       ↓           ↓           ↓
     Head 1      Head 2      Head 3   ...
       ↓           ↓           ↓
       └───────────┼───────────┘
                   ↓
              Concatenate
                   ↓
               Projection

Each head has its own learned projections.

We don't manually tell one head:

"You are responsible for grammar."

and another:

"You are responsible for pronouns."

The model learns useful patterns during training.

Different heads can capture different relationships in the sequence.

Afterward, the outputs are concatenated and passed through a learned output projection.

11. What Does Projection Mean?

You'll see the word projection constantly in Transformer implementations.

In this context, it usually means applying a learned linear transformation.

For example:

 X
 ↓
W_Q
 ↓
 Q

is a projection.

Likewise, after multiple attention heads:

Head outputs
      ↓
Concatenate
      ↓
Output projection
      ↓
Updated representation

And later:

Hidden representation
      ↓
Vocabulary projection
      ↓
Logits

So "projection" sounds more exotic than it really is.

A lot of Transformer computation is ultimately learned matrix multiplication.

12. Residual Connections

Attention produces a transformed representation.

But the Transformer doesn't simply throw away the original input.

It uses residual connections.

Mathematically:

The original representation is added back to the transformed representation.

Why?

Because Transformers can contain many layers.

Residual connections give information and gradients a shorter path through the network, which makes deep networks much easier to optimize.

A useful intuition is:

Keep what we already have, then add what this layer learned.

13. Layer Normalization and Add & Norm

You'll often see Transformer diagrams containing:

Add & Norm

This is shorthand for a residual addition together with Layer Normalization.

A simplified flow is:

Input
  │
  ├──→ Attention ──┐
  │                ↓
  └──────────────→ Add
                    ↓
                   Norm

Layer Normalization helps keep the numerical activations in a more stable range as the representation passes through many transformations.

Pre-LN vs Post-LN

The exact placement of LayerNorm can differ between Transformer architectures. Two common arrangements are Post-LN and Pre-LN.

Post-LN

In Post-LN, the sublayer runs first, the residual connection is added, and LayerNorm is applied afterward:

x → Attention → Add → LayerNorm

Or more explicitly:

x + Attention(x)
        ↓
    LayerNorm

Pre-LN

In Pre-LN, LayerNorm is applied before the sublayer:

x → LayerNorm → Attention → Add

Or:

x
↓
LayerNorm
↓
Attention
↓
+ x

The same ordering applies to the FFN (Feed-Forward Network) sublayer as well.

The distinction matters because the placement of normalization affects how easily the Transformer can be optimized. Pre-LN is common in many modern Transformer architectures because it tends to make optimization easier, especially for deeper models.

So when you see:

Attention → Add → Norm

versus:

Norm → Attention → Add

you're not looking at two completely different ideas.
They're simply two different ways of arranging the same basic components.

The important part is understanding the roles:

Residual connection → preserve a direct path through the network.

Layer normalization → stabilize the representations.

14. The Feed-Forward Network

Attention isn't the entire Transformer block.

After attention, the representation goes through a Feed-Forward Network (FFN).

A simplified version looks like:

Representation
      ↓
Linear transformation
      ↓
Non-linear activation
      ↓
Linear transformation
      ↓
New representation

Mathematically, something like:

A useful distinction is:

Attention lets tokens exchange information.

FFN transforms each token's representation using learned nonlinear transformations.

So conceptually:

Attention asks, "What information from elsewhere matters?"

FFN asks, "Given that information, how should this representation change?"

15. One Transformer Block

Now we can finally put the pieces together.

A simplified Transformer block looks like:

Input
  ↓
Self-Attention
  ↓
Residual + Norm
  ↓
FFN
  ↓
Residual + Norm
  ↓
Output

The exact ordering varies between implementations, but the major components remain recognizable.

And one block isn't enough.

16. Stacking Transformer Blocks

A Transformer stacks many blocks:

Input
  ↓
Transformer Block 1
  ↓
Transformer Block 2
  ↓
Transformer Block 3
  ↓
...
  ↓
Transformer Block N

The representation is repeatedly transformed.

It's tempting to say:

"The first layer learns grammar, the next layer learns facts, and the next layer learns reasoning."

That's too simplistic.

The model's learned information is distributed across many layers and parameters.

A better way to think about it is:

Each layer repeatedly transforms the representations, allowing increasingly useful and contextual patterns to emerge.

17. The Encoder

Now we can look at the Encoder itself.

In the original Encoder–Decoder Transformer, the Encoder is a stack of Transformer blocks using self-attention without a causal restriction.

If the input is:

"The cat is sitting on the mat."

the Encoder can allow every input token to attend to the other input tokens.

Conceptually:

"The" ↔ "cat" ↔ "is" ↔ "sitting" ↔ ...

This is often described as bidirectional self-attention.

The Encoder can see the complete input because its job isn't to generate the output token by token.

Its job is to build rich contextual representations of the input.

So the Encoder is roughly:

Input tokens
     ↓
Embeddings + positional information
     ↓
Transformer blocks
     ↓
Contextual representations

18. The Decoder

The Decoder has a different problem.

It is responsible for generating the output sequence.

And generation is autoregressive.

That means the Decoder can't simply look at everything in the target sequence, because future tokens don't exist yet during generation.

So the Decoder uses masked self-attention.

Suppose we're generating:

The cat sat on the mat

When predicting "mat", the model must not be allowed to use "mat" itself or anything after it.

The attention pattern is therefore causal:

      T1  T2  T3  T4
T1    ✓   ✗   ✗   ✗
T2    ✓   ✓   ✗   ✗
T3    ✓   ✓   ✓   ✗
T4    ✓   ✓   ✓   ✓

Each token can attend to itself and previous tokens, but not future tokens.

That's why this is called causal masking.

19. The Decoder Has Two Different Attention Mechanisms

This is the part that makes the Encoder–Decoder architecture much easier to understand.

The Decoder needs to use:

Its own previously generated tokens.
The information produced by the Encoder.

The first is handled by:

Masked Self-Attention

The second is handled by:

Cross-Attention

So a simplified Decoder block looks like:

Previous output tokens
          ↓
Masked Self-Attention
          ↓
Cross-Attention ← Encoder output
          ↓
FFN
          ↓
Output representation

These two attention mechanisms have different jobs.

20. Self-Attention vs Cross-Attention

For self-attention, Q, K, and V come from the same sequence.

In Encoder self-attention:

Q ← Encoder representations
K ← Encoder representations
V ← Encoder representations

In Decoder self-attention:

Q ← Decoder representations
K ← Decoder representations
V ← Decoder representations

The Decoder's self-attention is masked, so it can only use previous positions.

Cross-attention is different:

Q ← Decoder representation

K ← Encoder output
V ← Encoder output

That means the Decoder is effectively asking:

"Given what I have generated so far, which parts of the encoded input should influence my next prediction?"

This is the bridge between the Encoder and Decoder.

21. Translation Makes the Difference Obvious

Take:

"I love programming."

and translate it into French.

The Encoder reads the whole English sentence:

"I"   "love"   "programming"

and produces contextual representations.

The Decoder then starts with a special beginning-of-sequence token and begins generating:

<START>
   ↓
J'aime
   ↓
programmer
   ↓
<END>

At every step, the Decoder can use two sources of information.

Its previously generated tokens:

masked self-attention

and the original English input:

cross-attention

So while generating "programmer", the Decoder isn't working from the word "programmer" out of nowhere.

It can use the Encoder's representation of the original input.

That's the purpose of the Encoder–Decoder design.

22. Training With Teacher Forcing

Now we need to separate training from generation.

During training, we already know the correct output.

Suppose the target is:

"I love programming."

The Decoder can be trained with the correct previous tokens:

<BOS>          → I
<BOS> I        → love
<BOS> I love   → programming

This is called teacher forcing.

Instead of feeding the model its own prediction at every step during training, we give it the correct previous token and ask it to predict the next one.

That makes training much more manageable.

Without teacher forcing, one early incorrect prediction could become part of the next input and cause the errors to compound.

23. Training: Prediction, Loss, and Backpropagation

During training, the model makes a prediction for the next token.

We already know what the correct token should have been.

So we compare the prediction with the target using a loss function such as cross-entropy.

Suppose the correct token is:

Paris

but the model predicts:

Paris   → 0.20
London  → 0.30
Rome    → 0.10
...

The model should increase the probability of "Paris".

Cross-entropy gives us a way to measure how bad the prediction was.

For a single target token, a simplified form is:

So:

P(correct) = 0.90
→ small loss

P(correct) = 0.01
→ large loss

That loss is then used by backpropagation to calculate gradients.

The gradients tell the optimizer how the model's parameters should change.

Conceptually:

Prediction
    ↓
Loss
    ↓
Backpropagation
    ↓
Gradients
    ↓
Update parameters

This happens through the entire network.

The goal is to make future predictions better.

24. Training vs Inference

This distinction is worth keeping clear.

Training

Input + target
      ↓
Prediction
      ↓
Cross-entropy loss
      ↓
Backpropagation
      ↓
Update parameters

The weights change.

Inference

Input
  ↓
Transformer
  ↓
Logits
  ↓
Probabilities
  ↓
Choose token
  ↓
Append token
  ↓
Repeat

The learned weights are being used.

The model isn't learning new weights from your prompt during normal inference.

It's using what it already learned.

25. From Hidden Representation to Logits

After the Decoder processes the current context, we still don't have a word or token.

We have a hidden representation.

The model applies a final learned projection into the vocabulary space.

Conceptually:

Hidden representation
        ↓
Vocabulary projection
        ↓
Logits

If the vocabulary contains 100,000 tokens, the model can produce roughly 100,000 logits for the next token.

For example:

"the"        → 8.2
"program"    → 7.4
"Python"     → 6.9
"banana"     → 0.7

These are raw scores.

They're not probabilities yet.

26. Logits Become Probabilities

We apply Softmax:

Now the scores become a probability distribution.

Conceptually:

"the"        → 0.42
"program"    → 0.28
"Python"     → 0.19
"banana"     → 0.01
...

The model has now produced a probability distribution over possible next tokens.

Then the decoding process chooses one.

And this takes us directly to autoregressive generation.

27. Autoregressive Generation

Suppose GPT receives:

"Python is"

It predicts a probability distribution for the next token.

Maybe the selected token is:

" a"

Now the context becomes:

"Python is a"

The model runs again.

Maybe it predicts:

" programming"

Now:

"Python is a programming"

And so on.

Conceptually:

Current context
      ↓
Transformer
      ↓
Logits
      ↓
Probabilities
      ↓
Choose next token
      ↓
Append token
      ↓
New context
      ↓
Transformer again
      ↓
Repeat

This is autoregressive generation.

The model doesn't generate the entire paragraph in one step.

It generates one token, adds it to the context, and predicts the next token.

28. Temperature and Sampling

One final part of the generation process is sampling.

Temperature is not part of the Transformer architecture itself.

It is part of the decoding process.

Suppose the model produces logits:

A → 8.0
B → 7.0
C → 5.0

Changing the temperature changes how concentrated the resulting distribution becomes.

Lower temperature generally produces a sharper, more predictable distribution.

Higher temperature generally produces a flatter distribution and allows more variation.

Conceptually:

Low temperature
      ↓
Sharper distribution
      ↓
More predictable choices


High temperature
      ↓
Flatter distribution
      ↓
More variation

So temperature doesn't teach the model anything.

It changes how we sample from the model's predictions.

29. Where GPT Fits

Now the original question becomes much easier to answer.

The original Transformer was:

Encoder
   ↓
Decoder

GPT-style models are:

Decoder-only

That means GPT does not have a separate Encoder stack.

It also does not need the Encoder-to-Decoder cross-attention used in the classic Encoder–Decoder Transformer.

Instead, the prompt goes directly through a stack of decoder-style Transformer blocks using causal self-attention.

Conceptually:

Prompt
  ↓
Tokenization
  ↓
Embeddings + positional information
  ↓
Causal Self-Attention
  ↓
FFN
  ↓
Transformer block
  ↓
Transformer block
  ↓
...
  ↓
Logits
  ↓
Probabilities
  ↓
Next token

Then that token is appended to the sequence and the process repeats.

30. GPT vs the Original Encoder–Decoder Transformer

The difference can be summarized very simply.

Original Transformer

Input
  ↓
Encoder
  ↓
Encoder representations
  ↓
Decoder
  ↓
Output

The Decoder uses:

masked self-attention
cross-attention to the Encoder
FFN

GPT-style Transformer

Prompt
  ↓
Decoder-style blocks
  ↓
Logits
  ↓
Next token

It uses:

causal self-attention
FFN
residual connections
normalization
stacked Transformer blocks

but there is no separate Encoder and no Encoder-to-Decoder cross-attention.

That's why GPT is called decoder-only.

31. The Three Transformer Configurations

At this point, the three major configurations are easier to remember.

Encoder-only

Input
  ↓
Encoder
  ↓
Representation

Mental model:

Understand the input.

BERT is a well-known example.

Encoder–Decoder

Input
  ↓
Encoder
  ↓
Representation
  ↓
Decoder
  ↓
Output

Mental model:

Transform one sequence into another.

T5 is a well-known example.

Decoder-only

Prompt
  ↓
Decoder-style blocks
  ↓
Next token
  ↓
Next token
  ↓
Next token
  ↓
...

Mental model:

Continue the sequence.

GPT-style models belong here.

These are not the only possible ways to use Transformers, but they're three important architectural patterns to recognize.

32. The Complete Picture

At this point, the pieces fit together into one pipeline.

For a classic Encoder–Decoder Transformer:

Text
 ↓
Tokens
 ↓
Embeddings
 ↓
Positional Information
 ↓
Encoder
 ├── Self-Attention
 ├── Add + Norm
 ├── FFN
 └── repeated × N
 ↓
Encoded representation
 ↓
Decoder
 ├── Masked Self-Attention
 ├── Cross-Attention
 ├── FFN
 └── repeated × N
 ↓
Projection
 ↓
Logits
 ↓
Softmax
 ↓
Probabilities
 ↓
Next token
 ↓
Repeat

And for GPT-style decoder-only models:

Text
 ↓
Tokens
 ↓
Embeddings
 ↓
Positional Information
 ↓
Causal Self-Attention
 ↓
FFN
 ↓
Transformer Block × N
 ↓
Projection
 ↓
Logits
 ↓
Softmax / Decoding
 ↓
Next token
 ↓
Append token
 ↓
Repeat

The architecture becomes much easier to reason about once each component has a specific job.

33. The Part That Matters When Reading Transformer Diagrams

There are a lot of names to remember:

Q, K, V

Attention

Softmax

Masking

Multi-Head Attention

Projection

Residual Connections

Layer Normalization

FFN

Cross-Attention

Logits

Cross-Entropy

Backpropagation

Teacher Forcing

Autoregressive Generation

At first, they look like unrelated pieces of terminology.

They aren't.

They form a pipeline.

Tokenization gives us discrete pieces of text.

Embeddings turn those pieces into vectors.

Positional information tells the model where those tokens occur.

Q, K, and V create learned views of those representations.

Attention allows tokens to exchange information.

Masking controls which tokens are allowed to interact during generation.

Multi-Head Attention gives the model multiple learned ways to inspect relationships.

Residual Connections and Layer Normalization help those transformations work across deep networks.

FFNs further transform the resulting representations.

Encoder layers build contextual representations of an input.

Decoder layers use previous output tokens to generate the next ones.

Cross-Attention lets an Encoder–Decoder model connect those two sides.

Projection maps the hidden representation into the vocabulary space.

Logits give a score to every possible next token.

Softmax converts those scores into probabilities.

Cross-Entropy measures how well the model predicted the correct token during training.

Backpropagation computes how the parameters should change.

Teacher Forcing makes supervised sequence training practical.

And finally:

Autoregressive generation turns one predicted token into the context for the next prediction.

That's the Transformer.

Not one mysterious box.

Not a collection of AI buzzwords.

A sequence of numerical transformations where each component has a specific role.

And once that structure is clear, GPT stops looking like something completely different from the original Transformer.

It becomes what it actually is:

a decoder-only Transformer architecture built around causal self-attention and autoregressive next-token prediction.

LLMs Finally Made Sense to Me: A Software Engineer’s Mental Model

Adham Hewala — Sat, 15 Aug 2026 19:49:51 +0000

A few weeks ago, I started seriously learning how Large Language Models actually work.

Not how to use ChatGPT.

Not how to write better prompts.

Not how to call an LLM API from Python.

I mean what actually happens after I press Enter.

At first, it felt like opening a completely different field of computer science.

I kept running into terms like:

tokens, embeddings, attention, transformers, logits, softmax, temperature, context windows, fine-tuning, RLHF, RAG, vector databases, tool calling, agents, mixture of experts...

I understood each term individually.

But I didn't understand how they connected.

And I think that's the hardest part of getting into AI today.

There is no shortage of explanations for individual concepts.

What's missing, at least for me, was a mental model that connected everything together.

So I decided to build one.

This article is my attempt to explain it from the perspective of a software engineer who is learning LLMs, without starting with complicated research papers or assuming you're already an ML expert.

Why should software engineers understand this?

There's a practical reason I wanted to learn this.

Look at software engineering job descriptions today and you'll increasingly see things like:

LLM integration
RAG
Embeddings
Vector databases
AI agents
Prompt engineering
Generative AI
Tool calling
AI/ML APIs

AI is becoming another layer of the software stack.

And I don't think every software engineer needs to become an ML researcher.

But I do think we should understand what we're actually building with.

If I'm integrating an LLM into a backend service, I don't necessarily need to know every detail of the model's training infrastructure.

But I should understand enough to answer questions like:

What exactly is a token?

Why does the model have a context window?

What is an embedding?

Why does attention matter?

Why does the model sometimes produce different answers?

Why would I need RAG?

What actually happens when an LLM calls a tool?

Once those questions start making sense, the rest of the ecosystem becomes much easier to navigate.

Let's start with the simplest possible question

What happens when I send text to an LLM?

Suppose I send:

"Explain Docker to me."

As a human, I see a sentence.

The model doesn't.

The first thing that happens is tokenization.

1. Text → Tokens

Neural networks don't operate directly on strings like:

"Explain Docker to me."

They operate on numerical representations.

So the text is broken into tokens.

A token isn't necessarily a word.

Depending on the tokenizer, it could be:

a whole word
part of a word
punctuation
whitespace
or another small piece of text

For example, conceptually:

"unbelievable"
        ↓
["un", "believ", "able"]

The exact split depends on the tokenizer.

Each token is then mapped to an integer ID:

["un", "believ", "able"]
        ↓
[1254, 8271, 394]

Now we have something a neural network can process.

Why do we tokenize this way?

Because language is messy.

If we created one token for every possible word, we'd need an enormous vocabulary.

Consider:

run
runs
running
runner
runners
rerunning
...

A subword tokenizer can reuse pieces of words instead of storing every possible variation as a completely independent token.

One famous approach is Byte Pair Encoding (BPE), which builds a vocabulary by repeatedly merging frequently occurring sequences.

So the first mental model is:

LLMs don't read text directly. They process token representations.

2. Context Windows: The Model Can't See Everything

Now imagine sending a huge document to an LLM.

There is a limit to how much information can participate in a single model invocation.

That's the context window.

You might see models advertised with context sizes such as:

8K
32K
128K
1M+

These refer to the amount of tokenized context the model can handle.

Think of it as the model's working context for a particular inference request.

This is important because it explains why applications often need techniques such as:

chunking
summarization
retrieval
conversation memory
RAG

If you understand context windows, a lot of modern LLM application architecture starts making more sense.

3. Token IDs → Vectors

We now have something like:

[1254, 8271, 394]

But these integers are just IDs.

There's nothing inherently meaningful about the number 8271.

So we need another transformation.

Each token ID is mapped to a vector of floating-point numbers.

Something like:

8271
 ↓
[0.12, -0.43, 0.87, 0.04, ...]

This is an embedding.

And this is one of the concepts that completely changed how I think about machine learning.

Instead of representing information as a label, we represent it as a position in a high-dimensional mathematical space.

The model can learn useful relationships between these representations.

4. Embeddings Are More Than "Meaning = A Vector"

You'll often hear:

"Embeddings represent the meaning of words."

That's a useful beginner intuition, but it's an oversimplification.

Modern neural representations are distributed and contextual.

There isn't necessarily one dimension that means:

"This number represents happiness."

Instead, meaning and useful linguistic patterns are distributed across many dimensions.

The model learns these representations because they're useful for predicting and processing language.

And this idea existed long before today's LLMs.

5. Word2Vec: An Early Glimpse of What Was Possible

One of the famous milestones in NLP was Word2Vec, introduced in 2013.

It demonstrated that words appearing in similar contexts could develop similar vector representations.

One famous example is:

King - Man + Woman ≈ Queen

The important part isn't that this equation is some universal law of language.

The important part is what it demonstrated:

Relationships between linguistic concepts can emerge in vector space.

That idea became incredibly important for modern NLP.

6. So How Does the Model Understand Context?

This is where we reach the Transformer.

In 2017, researchers published the paper:

Attention Is All You Need

It introduced the Transformer architecture.

Transformers eventually became the foundation of modern LLMs.

And the most famous part of the Transformer is:

Attention

Consider these two sentences:

I went to the bank to deposit money.

and:

I sat beside the bank of the river.

The word bank is the same.

But its meaning depends heavily on the surrounding words.

This is where attention becomes important.

7. Attention: Which Tokens Matter Right Now?

A simplified way to think about attention is:

Each token can look at other tokens and determine which ones are relevant to its current representation.

For example:

"The animal didn't cross the road because it was tired."

To understand "it", the model needs to consider other parts of the sentence.

Attention provides a mechanism for modeling those relationships.

Under the hood, this involves:

Query (Q)

Key (K)

Value (V)

and a calculation commonly expressed as:

You don't need to memorize this equation to understand the big picture.

The important idea is:

Attention allows the model to dynamically determine which parts of the context should influence each token's representation.

And that's a huge part of why Transformers work so well with language.

8. A Transformer Is More Than Attention

A Transformer block isn't just attention.

A simplified view looks something like:

Input
  ↓
Self-Attention
  ↓
Feed-Forward Network
  ↓
Output

Real Transformer architectures also include things like:

residual connections
normalization
positional information
masking, depending on the architecture

And an LLM doesn't have just one Transformer block.

It stacks many of them.

The output of one layer becomes the input to the next.

Layer after layer, the model transforms the representation.

9. Then Comes the Part That Surprised Me

After all of that processing, the model doesn't output:

"Hello, here is your answer."

Instead, it produces scores for possible next tokens.

These raw scores are called logits.

Imagine the model has a vocabulary containing thousands of possible tokens.

For the next token, it might produce something conceptually like:

"the"       → 8.2
"container" → 7.4
"Docker"    → 6.9
"database"  → 4.1
"banana"    → 0.7
...

Those aren't probabilities yet.

They're logits.

10. Logits → Probabilities

To turn those scores into a probability distribution, we can apply Softmax.

Now we might have:

"the"       → 0.42
"container" → 0.28
"Docker"    → 0.19
"database"  → 0.08
"banana"    → 0.01

The exact values are illustrative, but the idea is important.

The model has effectively produced a probability distribution over possible next tokens.

And now comes something even more important:

The model chooses a token.

11. The Model Generates One Token at a Time

This was probably the most important thing for me to understand.

When ChatGPT gives you a paragraph, the model isn't necessarily generating the whole paragraph in one shot.

It's generating tokens sequentially.

Conceptually:

Input:
"Explain Docker"

        ↓

"Explain Docker is"

        ↓

"Explain Docker is a"

        ↓

"Explain Docker is a platform"

        ↓

"Explain Docker is a platform for"

        ↓

...

Each generated token becomes part of the context used to generate the next token.

This is called autoregressive generation.

So the basic inference loop is:

Tokenize
   ↓
Represent tokens
   ↓
Transformer
   ↓
Logits
   ↓
Probabilities
   ↓
Choose next token
   ↓
Append token
   ↓
Repeat

Over and over.

Until the generation stops.

12. Why Doesn't It Always Give the Same Answer?

If the model always selected the highest-probability token, its output would be much more deterministic.

Instead, generation can involve sampling.

Two parameters you'll often encounter are:

Temperature

Temperature controls how concentrated or spread out the probability distribution becomes.

Lower temperature generally means:

More predictable output.

Higher temperature generally means:

More variation.

Top-p

Top-p, or nucleus sampling, limits sampling to a subset of tokens whose cumulative probability reaches a specified threshold.

So:

Temperature changes the shape of the distribution.

Top-p limits the candidate pool.

This is one reason the same prompt can produce different answers.

13. But Where Did the Model Learn All of This?

This brings us to training.

And there's an important distinction:

An LLM doesn't start life as a helpful chatbot.

A simplified training lifecycle looks something like:

Massive Dataset
      ↓
Pretraining
      ↓
Base Model
      ↓
Supervised Fine-Tuning
      ↓
Preference / Alignment Training
      ↓
Assistant

Let's unpack that.

14. Pretraining: Learn to Predict What Comes Next

The basic objective during language-model pretraining is surprisingly simple:

Predict the next token.

Given:

"The capital of France is"

the model should assign high probability to:

"Paris"

Do this over an enormous amount of data, with enormous models and compute, and something interesting happens.

The model doesn't just memorize a list of sentences.

It learns statistical structures and representations that support language prediction.

It can learn patterns involving:

syntax
programming
facts
concepts
styles
relationships
reasoning-like patterns

The result is a base language model.

But a base model isn't necessarily optimized to be a great assistant.

15. Supervised Fine-Tuning

Now imagine giving the model examples like:

User:
What is an API?

Assistant:
An API is...

and many other high-quality examples.

The model can be trained to follow this conversational format.

This is commonly called:

SFT — Supervised Fine-Tuning.

It helps transform the raw language model into something much closer to an assistant.

16. Preference Training and RLHF

But there's another problem.

Suppose the model gives three answers.

One is:

correct
clear
safe
helpful

Another is technically correct but confusing.

The third is simply wrong.

We need a way to teach the model which behaviors humans prefer.

One historically important approach is:

RLHF — Reinforcement Learning from Human Feedback.

Humans provide preference signals, and the training process uses those signals to encourage desirable behavior.

Modern models use a broader range of techniques for alignment and preference optimization, but the central idea remains:

The model isn't only trained to predict text. It is also optimized to produce useful behavior.

17. Now We Reach the Part Software Engineers Care About

Understanding how the model works is useful.

But the really exciting part is what happens when we put an LLM inside a software system.

This is where terms like:

RAG

Vector databases

Tool calling

Agents

start appearing.

And suddenly, all the concepts we've discussed begin connecting.

18. Embeddings + Vector Databases + RAG

Imagine you're building an internal company assistant.

You want it to answer questions about:

company policies
technical documentation
product manuals
internal knowledge
recent documents

You can't simply expect the model to magically know your private documents.

And retraining the entire model every time a document changes isn't practical.

So we can use Retrieval-Augmented Generation (RAG).

A simplified pipeline:

Documents
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Database

Then when a user asks:

"What is our vacation policy?"

the system can do:

Question
   ↓
Embedding
   ↓
Vector Search
   ↓
Relevant Documents
   ↓
LLM + Retrieved Context
   ↓
Answer

Now the model isn't responsible for remembering everything.

Your application retrieves the relevant information and gives it to the model as context.

This is a perfect example of where traditional software engineering and LLMs meet.

19. Tool Calling: When the LLM Needs Software

An LLM can generate text.

But your application can do much more.

Your backend can:

query a database
call an API
perform calculations
search documents
send an email
create an order
retrieve weather information
interact with another service

So why not let the model decide when one of these capabilities is needed?

That's where tool calling comes in.

A simplified flow:

User
 ↓
LLM
 ↓
"I need the weather tool"
 ↓
Tool Call
 ↓
Your Backend
 ↓
Weather API
 ↓
Tool Result
 ↓
LLM
 ↓
Final Response

The LLM isn't directly performing the API request.

Your application executes the tool.

The model receives the result and continues.

This distinction is extremely important.

The LLM becomes a reasoning and language interface around actual software capabilities.

20. And That's Where Agents Come From

Once you combine:

LLM + tools + state + an execution loop

you can start building systems that behave more like agents.

For example:

Goal
 ↓
LLM decides what to do
 ↓
Call Tool A
 ↓
Observe Result
 ↓
Decide next action
 ↓
Call Tool B
 ↓
Observe Result
 ↓
Final Answer

This is why I think it's more useful to understand the underlying pieces than to simply memorize:

"Agent = AI that does things."

The interesting engineering questions are:

What tools does it have?
How does it select them?
What state does it maintain?
How do we control its actions?
What happens when a tool fails?
How do we prevent unwanted actions?
How do we evaluate it?

Now we're back in familiar software engineering territory.

21. Mixture of Experts

Another term you'll encounter when exploring modern LLM architectures is:

Mixture of Experts (MoE).

The basic idea is that instead of having every part of a huge network process every token, the model can contain multiple expert components.

A router decides which experts should process a token.

Conceptually:

                  ┌── Expert A
                  │
Token → Router ───┼── Expert B
                  │
                  └── Expert C

Only a subset of experts may be activated for each token.

This can make it possible to build models with very large total parameter counts while controlling the amount of computation used per token.

Again, the important thing isn't memorizing the architecture.

It's understanding why the technique exists.

22. What About Models That "Think"?

This is another area where terminology can become confusing.

You may hear:

reasoning models
thinking models
chain-of-thought
extended thinking

The core intuition is that some tasks benefit from giving the model additional computation before producing the final answer.

Because these models generate sequentially, intermediate reasoning or computation can provide additional steps for solving a difficult problem.

But there's an important distinction:

"Chain-of-thought" does not simply mean showing the user the model's private reasoning.

Modern systems may use internal reasoning processes that are not exposed verbatim.

From an engineering perspective, the important idea is:

Inference can involve additional computation, not just one immediate prediction.

23. How Do We Know an LLM Is Actually Good?

Here's where my software-engineering brain immediately kicks in.

We test it.

You wouldn't deploy a backend service and say:

"It seems to work."

You'd write tests.

You'd monitor it.

You'd measure performance.

The same philosophy applies to LLM applications.

Models can be evaluated against benchmarks and task-specific datasets.

There are also evaluation frameworks and "evaluation harnesses" that automate running models against collections of standardized tests.

And for production applications, the most important evaluation is often even more specific:

Does the model perform well on the tasks my users actually care about?

A model can perform extremely well on a benchmark and still perform badly in your specific application.

That's why evaluation is becoming a major part of AI engineering.

24. The Mental Model That Finally Made Everything Click

After going through all of this, I now visualize an LLM roughly like this:

                    USER INPUT
                        │
                        ▼
                   TOKENIZATION
                        │
                        ▼
                    TOKEN IDs
                        │
                        ▼
                    EMBEDDINGS
                        │
                        ▼
                TRANSFORMER LAYERS
              ┌─────────┴─────────┐
              ▼                   ▼
          ATTENTION               FFN
              │                   │
              └─────────┬─────────┘
                        ▼
                     LOGITS
                        │
                        ▼
                     SOFTMAX
                        │
                        ▼
                  PROBABILITIES
                        │
                        ▼
                    SAMPLING
                        │
                        ▼
                   NEXT TOKEN
                        │
                        └──────────────┐
                                       │
                                       ▼
                              REPEAT THE LOOP

And around the model, we can build:

                  ┌──────────────┐
                  │     LLM      │
                  └──────┬───────┘
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
            RAG        Tools       Memory
             │           │
             ▼           ▼
       Vector DBs      APIs

That's the mental model I'm keeping.

25. The Part I Find Most Exciting

Here's what surprised me most.

The more I learned about LLMs, the less they felt like something completely separate from software engineering.

They started looking like another component in a software architecture.

Think about a traditional backend:

Frontend
   ↓
  API
   ↓
Business Logic
   ↓
Database
   ↓
External Services

Now imagine adding an LLM:

Frontend
   ↓
API
   ↓
LLM
   ├── RAG → Vector Database
   ├── Tools → External APIs
   ├── Memory → Database
   └── Guardrails / Evaluation

Suddenly, the skills you already have become relevant.

APIs.

Authentication.

Databases.

Caching.

Queues.

Observability.

Error handling.

Testing.

Cloud infrastructure.

Distributed systems.

The LLM is only one piece.

And I think that's a very important realization for software engineers trying to enter AI.

26. You Don't Need to Become an AI Researcher to Start

If you're a software engineer who wants to get into AI, I don't think the first step should be:

"I need to learn every mathematical detail of neural networks."

And I don't think it should be:

"I'll just learn how to call OpenAI's API."

There is a middle ground.

Understand the architecture first.

Understand the vocabulary.

Understand the flow.

Then start building.

For example, build a simple application that:

Accepts a user's question.
Calls an LLM.
Stores conversations.
Retrieves relevant documents.
Uses embeddings.
Performs vector search.
Calls external tools.
Evaluates the generated answers.

At that point, concepts that looked abstract start becoming practical.

27. If I Had to Reduce Everything to One Diagram

This is probably the diagram I'd keep if I forgot everything else:

                 ┌────────────────────┐
                 │      YOUR APP      │
                 └─────────┬──────────┘
                           │
                           ▼
                    ┌─────────────┐
                    │    INPUT    │
                    └──────┬──────┘
                           │
                           ▼
                       TOKENIZER
                           │
                           ▼
                        TOKENS
                           │
                           ▼
                      EMBEDDINGS
                           │
                           ▼
                  ┌─────────────────┐
                  │   TRANSFORMER   │
                  │                 │
                  │ Attention + FFN │
                  │       × N       │
                  └────────┬────────┘
                           │
                           ▼
                        LOGITS
                           │
                           ▼
                        SOFTMAX
                           │
                           ▼
                      NEXT TOKEN
                           │
                           ▼
                     REPEAT / LOOP
                           │
                           ▼
                       RESPONSE

And around that core:

       RAG ───────> External Knowledge
       Tools ─────> External Actions
       Memory ────> Persistent State
       Evaluation → Quality Measurement

That, for me, is the foundation.

Final Thoughts

I'm still learning.

There are many things I haven't covered here: positional encodings, KV caching, quantization, distributed training, inference optimization, LoRA, multimodal models, RL techniques, model architectures, serving infrastructure, and much more.

But I don't think you need to understand everything before you start building.

What helped me most was going one layer at a time:

Text → Tokens → Vectors → Attention → Transformer → Logits → Probabilities → Next Token

Then:

Pretraining → Fine-tuning → Alignment

And finally:

LLM → RAG → Tools → Agents → Production AI Systems

Once you see those connections, the endless stream of AI terminology becomes much less intimidating.

You stop seeing "RAG", "embeddings", "tool calling", and "agents" as completely separate technologies.

You start seeing them as pieces of a larger system.

And perhaps that's the most useful lesson I've taken from learning LLMs so far:

You don't need to understand every detail of AI before you start building with it.

But you should understand the foundations well enough to know what is happening underneath the abstraction.

That's the point where AI stops feeling like magic.

And starts feeling like engineering.

If you're a software engineer starting your own journey into LLMs, I hope this mental model saves you some of the confusion I had when I started.

There's a lot more to learn.

But now I finally feel like I know where the pieces belong.

Pokémon Info Retriever: A Fun and Educational Project

Adham Hewala — Mon, 28 Oct 2024 17:15:27 +0000

As a passionate software developer, I've embarked on an exciting journey to create a Pokémon Info Retriever application. This project combines multiple technologies to provide users with a seamless way to access detailed Pokémon information using the PokeAPI. In this post, I will share the development process, the technologies I used, and some lessons learned along the way.

Project Overview

The Pokémon Info Retriever consists of three main components:

FastAPI Backend: This is the core of the application, responsible for handling requests and retrieving Pokémon data from the PokeAPI.
Python Desktop GUI: A user-friendly desktop application that allows users to input a Pokémon's name and receive its information.
HTML Web Interface: A simple web interface that allows users to access the Pokémon data through a browser.

Key Technologies Used

FastAPI: A modern web framework for building APIs with Python 3. It allows for rapid development and easy handling of asynchronous requests. FastAPI's automatic generation of OpenAPI documentation is a significant advantage.
PokeAPI: A RESTful API that provides access to a vast amount of Pokémon data, including abilities, types, and stats.
Tkinter: The standard GUI toolkit for Python, used to create the desktop application. It's lightweight and allows for quick development of simple user interfaces.
HTML/CSS: Used for the web interface, enabling access to Pokémon data through any web browser.

Development Process

Step 1: Setting Up the FastAPI Backend
I started by setting up the FastAPI backend, which involved defining routes for fetching Pokémon data. The backend handles requests and interacts with the PokeAPI to retrieve relevant information based on user input.

Step 2: Creating the Desktop GUI
Using Tkinter, I designed a simple yet effective GUI. The GUI prompts users to enter a Pokémon name and displays relevant data upon clicking the retrieve button.

Step 3: Building the HTML Interface
The HTML template serves as an accessible front-end option, allowing users to interact with the API directly through their browsers.

Challenges Faced

Error Handling: One of the challenges was ensuring proper error handling when users entered invalid Pokémon names. I implemented comprehensive exception handling to provide user-friendly error messages.
Cross-Origin Resource Sharing (CORS): Configuring CORS in FastAPI was essential to allow requests from the front-end applications. I learned how to set this up efficiently.

Future Improvements

I plan to enhance the application further by:

Adding more detailed Pokémon stats, including evolutions and habitats.
Implementing user authentication for personalized experiences.
Improving the GUI design for better aesthetics and user experience.

Useful Links

You can find more about the Pokémon data and endpoints in the PokeAPI Documentation.
For detailed information on how to get started with FastAPI, check out the FastAPI Documentation.
The complete code for the Pokémon Info Retriever project can be found in my GitHub repository: GitHub - Pokémon Info Retriever.

Conclusion

This project was a fantastic learning experience, allowing me to delve into FastAPI, GUI development, and API integration. I encourage anyone interested in similar projects to give it a try!