DEV Community: zeromathai

Discriminative vs Generative Models: What Are You Actually Learning?

zeromathai — Wed, 02 Sep 2026 05:33:44 +0000

When a supervised model maps an input to an output, it is easy to assume that every model is learning essentially the same relationship. In practice, two models can solve the same prediction problem while modeling very different probability relationships.

A Discriminative Model learns the relationship needed to predict the output directly. A Generative Model instead models how the input is distributed for each possible output and uses that probability structure to infer the prediction.

The most useful question is therefore not simply, ""What does the model predict?"" but ""What probability relationship does it learn directly?""

Start with the supervised learning objective

At a high level, supervised learning asks us to learn a mapping from an observed input $x$ to a target output $y$ :

f : x \mapsto y

Both discriminative and generative approaches aim to solve this mapping problem. The difference is the probability relationship they model in order to reach the prediction.

A compact mental model is:

Discriminative
x
│
├── learn p(y | x)
│
└── choose y directly

Generative
y
│
├── learn p(x | y)
├── combine with p(y)
│
└── infer the best y for x

The prediction target is the same, but the learned structure is different.

Discriminative Models: learn the conditional relationship directly

A Discriminative Model focuses on the conditional probability of an output given an input:

p (y ∣ x)

For classification, prediction can be written as:

f (x) = ar g y max p (y ∣ x)

The calculation flow is direct:

input x
   ↓
estimate p(y | x)
   ↓
compare candidate outputs
   ↓
select the most probable y

The model focuses on the relationship needed to distinguish possible outputs from the observed input. In the input space, this leads naturally to the idea of a decision boundary: regions of the space are associated with different outputs according to the learned predictive relationship.

Support Vector Machine is a representative discriminative example in this comparison. Its role here is best understood through the decision structure it learns between outputs rather than as a model of the overall probability distribution of the data.

Generative Models: model the input distribution for each output

A Generative Model approaches the same supervised problem from the opposite conditional direction. Instead of directly modeling $p (y ∣ x)$ , it models:

p (x ∣ y)

This tells us how likely an input $x$ is under a particular output $y$ . The model then combines this conditional probability with the prior probability of the output, $p (y)$ .

Prediction can therefore be written as:

f (x) = ar g y max p (x ∣ y) p (y)

The calculation flow becomes:

candidate y
   ↓
evaluate p(x | y)
   ↓
combine with p(y)
   ↓
compare p(x | y)p(y)
   ↓
select the best y

Bayes rule shows how this connects back to the conditional probability used by the discriminative view:

p (y ∣ x) = \frac{p ( x ∣ y ) p ( y )}{p ( x )}

For a fixed input $x$ , the denominator $p (x)$ is shared across all candidate outputs. When the goal is only to determine which output has the highest posterior probability, that common denominator does not affect the comparison.

This is why a Generative Model can use $p (x ∣ y)$ together with $p (y)$ to choose the appropriate output without directly modeling $p (y ∣ x)$ .

Gaussian Mixture and Bayes Nets are representative generative examples in this comparison. The important distinction is that this approach models the probability distribution describing how inputs relate to possible outputs.

The comparison that matters

It is tempting to reduce the distinction to ""classification versus generation,"" but that misses the central point. In this supervised-learning setting, both approaches can predict $y$ from $x$ .

What differs is what they learn directly.

	Discriminative Model	Generative Model
Prediction goal	Predict $y$ from $x$	Predict $y$ from $x$
Directly modeled relationship	$p (y ∣ x)$	$p (x ∣ y)$
Additional probability used	—	$p (y)$
Prediction rule	$ar g max_{y} p (y ∣ x)$	$ar g max_{y} p (x ∣ y) p (y)$
Useful structural view	Decision boundary	Probability distribution
Representative example	Support Vector Machine	Gaussian Mixture, Bayes Nets

For developers, this distinction is useful when reading a model definition or implementation. If the model directly represents the relationship from $x$ to $y$ , the focus is discriminative. If it models how $x$ behaves under each $y$ and combines that with $p (y)$ , the focus is generative.

Takeaway

Discriminative and generative models can solve the same supervised prediction problem through different probabilistic views.

A Discriminative Model directly models:

p (y ∣ x)

A Generative Model models:

p (x ∣ y)

and combines it with $p (y)$ to infer the appropriate output.

The key distinction is not simply classification vs generation. It is which probability relationship the model learns directly and uses for prediction.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/discriminative-vs-generative-models-en/

Generative Modeling: From Data Distributions to Deep Generative Models

zeromathai — Wed, 02 Sep 2026 00:13:36 +0000

If you approach generative models as ""networks that create images,"" the field quickly turns into a collection of disconnected architectures.

A more useful developer mental model starts one level lower:

What probability structure could have produced the data, and how can we represent, learn, and infer that structure without making the computation impossible?

That question connects autoregressive models, VAEs, flow-based models, GANs, and diffusion models. Their architectures look very different, but they all respond to the same underlying tension: high-dimensional data distributions are difficult to represent, learn, normalize, sample from, and reason about.

Generative modeling can therefore be organized around three interacting problems:

Representation: How do we represent a complex high-dimensional joint distribution?
Learning: How do we make the model distribution approach the data distribution?
Inference: Given an observation, how do we reason backward about hidden variables or the process that generated it?

Once these three pieces are connected, the major families of deep generative models become much easier to understand.

From prediction to distribution learning

A discriminative model usually begins with a prediction problem. Given an input $x$ , predict the most likely output $y$ :

f (x) = y ar g max p (y ∣ x)

The model focuses directly on the conditional relationship required for prediction.

A generative model asks a broader question. Instead of learning only the path from $x$ to $y$ , it models the probability structure from which the data arises.

For class-conditional modeling, for example, we can model $p (x ∣ y)$ together with the prior $p (y)$ and recover the posterior using Bayes' rule:

p (y ∣ x) = \frac{p ( x ∣ y ) p ( y )}{p ( x )}

In unsupervised generative modeling, the target becomes the data distribution itself. We assume the training samples come from some unknown distribution:

x_{1}, x_{2}, \dots, x_{N} \sim i.i.d. p_{data} (x)

The model then constructs a parameterized distribution intended to approximate it.

This perspective connects tasks that can otherwise seem separate:

x_{train} \sim p_{data} (x), x_{new} \sim p_{model} (x)

Density estimation asks how probable an observation is under the model. Generation draws new samples from the learned model distribution. Unsupervised representation learning can capture latent structure shared by the data while learning that distribution.

All three begin with the same underlying goal: learning the probability structure of the data.

Why high-dimensional data makes this hard

Real data usually lives in a high-dimensional space. An image, for example, is described by many variables, while valid observations occupy only a limited and complicated region among all possible combinations.

So the first problem is not simply ""use a neural network."" It is:

How do we represent the joint distribution of many random variables compactly enough to compute with, while keeping enough flexibility to describe the data?

This connects naturally to inverse problems.

A forward problem moves from a cause $x$ to an observation $y$ . An inverse problem starts from an observed $y$ and asks which possible $x$ could have produced it. Conditional generation extends the same idea by generating possible $x$ values consistent with a given condition.

Forward:
cause x  ──────────>  observation y

Inverse:
possible cause x  <──────────  observation y

Conditional generation:
condition y  ──────────>  possible samples x

Recovering a high-resolution image from a low-resolution one, estimating color from grayscale data, reconstructing missing regions, or recovering a CT slice from a sparse-view sinogram all share this structure.

The difficulty is that the inverse may not be unique. A single observation can be consistent with multiple possible causes. Inverse problems are therefore not simply about running a function backward; they require reasoning over possible causes that are compatible with the observation.

Latent variables turn inversion into posterior inference

Suppose a model contains an observed variable $x$ and a hidden variable $z$ .

The generative direction models the relationship between $z$ and $x$ . Once $x$ has been observed, the reverse question is which values of $z$ could have produced it.

That is posterior inference:

p (z ∣ x) = \frac{p ( z , x )}{p ( x )}

Posterior inference can require both computing the posterior distribution and calculating expectations under it. The E-step of the EM algorithm is a direct example:

Q (θ, θ^{old}) = z \sum p (z ∣ x; θ^{old}) lo g p (x, z; θ)

Here the posterior over the unobserved $z$ is used to compute the expected complete-data log-likelihood.

As the latent space grows and interactions among hidden variables become more complicated, direct posterior computation can become intractable. Increasing model expressivity therefore leads naturally to another question: how can the required inference remain computationally feasible?

more expressive representation
        ↓
more complicated latent structure
        ↓
harder posterior inference
        ↓
need for approximate inference

When the posterior is intractable

Two major approaches are MCMC and Variational Inference. Both approximate a posterior that is difficult to compute directly, but the approximation arises in different ways.

MCMC: approximate with samples

MCMC constructs an ergodic Markov chain whose stationary distribution is the target posterior. Samples generated by running the chain are then used to approximate the posterior and the expectations we need.

target posterior
      ↓
construct Markov chain
      ↓
run the chain
      ↓
collect samples
      ↓
approximate posterior expectations

The practical approximation comes from finite computation. In real problems, only a limited amount of time is available for generating samples, and exploring the relevant probability regions can be expensive in high-dimensional or complex models.

MCMC therefore has a finite-time sampling approximation in practice.

Variational Inference: approximate with optimization

Variational Inference makes a different tradeoff. Instead of sampling directly from the posterior, it defines a tractable family of distributions $Q$ and searches within that family for a distribution close to the true posterior:

q^{*} (z) = q (z) \in Q ar g min D_{KL}! [q (z) ∥ p (z ∣ x)]

This turns posterior inference into an optimization problem.

The tractability comes from restricting the search space. If the true posterior cannot be represented by the chosen variational family, more optimization time alone cannot eliminate the mismatch.

The contrast is useful:

MCMC approximation source:
finite-time sampling

Variational Inference approximation source:
restricted distribution family

This distinction becomes concrete in a VAE. Rather than computing a complicated latent posterior directly, the model uses a variational distribution, with the Encoder acting as a Recognition Network that constructs the approximation. The Decoder generates observations from the latent variable.

An abstract inference problem has now become part of a trainable model architecture.

The real bottleneck: flexibility versus tractability

Representation and computation collide when we try to model realistic high-dimensional distributions.

A simple probability model can be easy to normalize and evaluate but may lack enough flexibility to describe complex data. A deep neural network can represent much richer functions, but its output is not automatically a valid probability density.

A probability density must be nonnegative and integrate to one. One way to construct such a density is:

p_{θ} (x) = \frac{e ^{f_{θ} (x)}}{Z _{θ}}

where

Z_{θ} = \int e^{f_{θ} (x)} d x

Exponentiation gives a positive quantity, while the normalizing constant $Z_{θ}$ ensures that the density integrates to one.

For a complex neural function over a high-dimensional space, however, computing that integral can itself become intractable.

simple probability model
    → tractable
    → limited flexibility

deep neural representation
    → flexible
    → normalization or inference may become intractable

This is the Tractability-Flexibility Tradeoff.

It provides a useful way to compare deep generative models. Rather than asking only which network architecture a model uses, ask which difficult probability computation its structure makes manageable.

Learning a model distribution

In likelihood-based generative modeling, learning can be viewed as reducing the difference between the data distribution and the model distribution:

D_{KL}! [p_{data} (x) ∥ p_{model} (x; θ)] E x \sim p data! [lo g \frac{p _{data} ( x )}{p _{model} ( x ; θ )}]

Expanding this expression separates a term that depends only on the data distribution from a term containing the model log probability. The data-only term is independent of $θ$ , so it does not change as the model parameters are optimized.

The learning objective therefore connects to increasing the model's log probability on observed data. With a finite training set, this gives the progression:

minimize KL divergence
        ↓
minimize cross-entropy
        ↓
minimize negative log-likelihood

The challenge is to make $p_{model} (x; θ)$ expressive enough for high-dimensional data while keeping the required probability computations feasible.

This is where the major generative model families diverge.

Autoregressive models: factorize the joint distribution

An autoregressive model handles a high-dimensional joint distribution by decomposing it into sequential conditional distributions:

p (x) = i = 1 \prod n p (x_{i} ∣ x_{1}, \dots, x_{i - 1})

Instead of directly modeling one complicated joint distribution, the model works with a product of conditional probabilities.

The key structural idea is factorization. The model represents the high-dimensional joint distribution through a sequence of conditional distributions that can be modeled in a computationally manageable form.

VAE: latent variables meet approximate inference

A latent-variable model can express the observed distribution as:

p (x) = \int p (z) p (x ∣ z) d z

This provides a latent space for representing hidden structure, but it also introduces the posterior problem $p (z ∣ x)$ .

A VAE connects the pieces this way:

observed x
   ↓
Encoder / Recognition Network
   ↓
variational approximation to posterior over z
   ↓
latent z
   ↓
Decoder / Generative Network
   ↓
generated or reconstructed x

Training connects this architecture to Variational Inference through maximizing the ELBO:

latent-variable representation
        ↓
intractable posterior
        ↓
variational approximation
        ↓
Encoder
        ↓
ELBO-based learning

This is why a VAE is more than an Encoder followed by a Decoder. Its structure turns posterior intractability and variational approximation into a trainable generative model.

Flow-based models: use invertibility to preserve tractability

Flow-based models start from a simple distribution and apply a sequence of invertible transformations to reach a more complex target distribution.

Invertibility is the critical structural constraint. Because the transformations can be reversed, probability density can be tracked through them, allowing the model to train with NLL.

architectural restriction
        ↓
invertible transformation
        ↓
density remains computable
        ↓
likelihood-based training becomes possible

The constraint is part of the solution to the tractability problem: it introduces a structural restriction while keeping density evaluation possible.

GAN: learn the generation process instead of explicit density

GANs make a different choice. Instead of directly computing an explicit density function or normalizing constant, they learn the generation process itself.

The Generator tries to produce samples resembling real data, while the Discriminator tries to distinguish generated samples from real ones. The two networks train with opposing objectives in a minimax problem.

latent input
    ↓
Generator
    ↓
generated sample ──┐
                   ├──> Discriminator
real sample ───────┘

GANs use this adversarial minimax process to construct an implicit generative model rather than directly evaluating an explicit model density.

Diffusion: connect noise and data through many steps

Diffusion models take another route. Rather than directly handling the transformation between a simple distribution and a complex data distribution in one step, they connect the two through a sequence of noise levels.

The forward process gradually adds Gaussian noise to real data:

x_{0} \sim q (x), x_{T} \to N (0, I) as T \to \infty

As the process continues, the original data structure gradually disappears and approaches a simple isotropic Gaussian distribution.

Generation runs in the opposite direction:

data
  ↓
add noise
  ↓
add more noise
  ↓
...
  ↓
Gaussian-like noise

generation:

Gaussian noise
  ↓
learned reverse step
  ↓
learned reverse step
  ↓
...
  ↓
data-like sample

Instead of using one direct transformation, diffusion models learn a reverse process that progressively reconstructs data structure from noise.

The score view of diffusion

Score-based generative models can describe this process using the score function:

s (x) = \nabla_{x} lo g p (x)

The score indicates the direction in which the log probability increases.

Two representative approaches are SMLD and DDPM.

SMLD, or Score Matching with Langevin Dynamics, estimates the score at multiple noise scales and uses Langevin Dynamics to sample while the noise scale decreases.

DDPM, or Denoising Diffusion Probabilistic Model, learns a probabilistic reverse model for the steps of the forward noise-corruption process.

Their emphasis can be summarized as:

SMLD:
score estimation
      +
Langevin Dynamics

DDPM:
forward corruption steps
      +
learned reverse probability model

The details differ, but both approaches connect a simple noise distribution to a complex data distribution through multiple noise levels or steps.

That iterative structure also creates a practical tradeoff. Because generation requires multiple diffusion steps, sampling speed can become an important limitation.

One probability problem, different structural answers

The major generative model families are easier to compare when viewed as different computational responses to the same high-dimensional distribution problem:

Autoregressive models factorize a joint distribution into sequential conditional distributions.
VAEs introduce latent variables and use Variational Inference to approximate a difficult posterior while training through the ELBO.
Flow-based models impose invertibility so density evaluation and NLL-based learning remain possible.
GANs avoid direct explicit-density evaluation and learn the generation process through adversarial minimax training.
Diffusion models transform generation into a sequence of noise-corruption and reverse-process steps.

The normalizing constant fits into the same picture. Energy-Based Models work with energy-based probability distributions in which the normalizing constant must be handled or approximated. Autoregressive models, flow-based models, and VAEs use computationally manageable structural constraints or model constructions. GANs avoid direct computation of explicit density and its normalizing constant by learning the generation process, while score-based diffusion uses the score rather than directly computing the normalizing constant.

Different architectures make different choices, but the underlying problem is the same: preserve enough flexibility to represent complex data while keeping generative modeling computationally feasible.

A developer mental model for generative architectures

When you encounter a generative architecture, start with three questions rather than its layer diagram.

Representation: What distribution or latent structure is the model trying to represent?

Learning: What objective reduces the difference between the model and the observed data distribution?

Inference: What hidden variables or generating processes must be inferred from observations, and can those quantities be computed directly?

Then look at the structural choice the model uses to keep the relevant computations manageable:

high-dimensional data distribution
          ↓
representation must be expressive
          ↓
probability or posterior computation becomes difficult
          ↓
choose a computational strategy
          ↓
factorize
approximate
restrict
avoid explicit density
or learn a multi-step reverse process

No single family removes every tradeoff. In the comparison considered here, VAEs and flow-based models can show weaknesses in generation quality, GANs can face difficulties with generation diversity, and diffusion models can pay for their iterative generation process with slower sampling.

So the useful comparison is not simply ""Which model is better?""

It is:

Which computational problem does the model solve through its structural choice, and what does that choice cost in quality, diversity, or speed?

Takeaway

Generative modeling is not fundamentally about producing realistic-looking samples. It is about learning an unknown high-dimensional data distribution while keeping Representation, Learning, and Inference computationally manageable.

Autoregressive models, VAEs, flow-based models, GANs, and diffusion models look different because they make different structural choices around the same underlying problem. Once you identify whether an architecture factorizes a distribution, approximates an intractable quantity, imposes a structural constraint, avoids explicit density evaluation, or learns a multi-step reverse process, the landscape of deep generative models becomes much easier to reason about.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/gan-en/

How Modern Transformer Blocks Work — From RMSNorm to MoE

zeromathai — Mon, 29 Jun 2026 10:42:05 +0000

The original Transformer idea is still alive.

But modern LLM blocks are not just the 2017 Transformer copied and scaled.

They are engineered for deeper training, longer context, cheaper inference, and larger capacity.

That is why components like RMSNorm, GQA, RoPE, SwiGLU, and MoE matter.

Core Idea

A modern Transformer block still follows the same basic pattern:

Attention updates token relationships.

The Feed-Forward Network transforms each token representation.

Residual connections keep information flowing.

But modern LLMs changed the details.

Those details are not cosmetic.

They make large-scale training and inference practical.

The Key Structure

A typical modern Transformer block looks like this:

Input

→ RMSNorm or Pre-Layer Normalization

→ Self-Attention with GQA and RoPE

→ Residual Connection

→ RMSNorm or Pre-Layer Normalization

→ Feed-Forward Network with SwiGLU or MoE

→ Residual Connection

More compactly:

Modern Transformer Block = stable normalization + efficient attention + stronger FFN + residual flow

Each component solves a real scaling problem.

Pre-LN improves deep training stability.

GQA reduces KV Cache memory.

RoPE injects position into attention.

SwiGLU improves FFN expressiveness.

MoE increases capacity without activating all parameters.

Pseudo-code View

A simplified modern block looks like this:

def transformer_block(x):
    h = rms_norm(x)

    attn = grouped_query_attention(
        q=apply_rope(query(h)),
        k=apply_rope(key(h)),
        v=value(h)
    )

    x = x + attn

    h = rms_norm(x)

    ffn = swiglu_ffn(h)

    x = x + ffn

    return x

With MoE, the FFN part can become:

h = rms_norm(x)

selected_experts = router(h)

ffn = run_top_k_experts(h, selected_experts)

x = x + ffn

The pattern stays simple.

Normalize.

Transform.

Add back.

Repeat.

Concrete Example

Imagine the model processes this token:

"bank"

The attention block helps decide whether “bank” means:

a financial institution

or the side of a river

RoPE helps the model understand token order and distance.

GQA helps attention run with a smaller KV Cache.

The FFN then transforms the contextual representation.

If the model uses MoE, the router may send this token to experts specialized for finance, geography, or general language.

That is the intuition.

Modern Transformer blocks are not just bigger.

They are more selective, stable, and hardware-aware.

Pre-LN vs Post-LN

The original Transformer commonly used Post-LN.

Post-LN:

x = LayerNorm(x + Sublayer(x))

Modern LLMs often use Pre-LN.

Pre-LN:

x = x + Sublayer(LayerNorm(x))

The difference looks small.

But it matters.

Pre-LN normalizes before the sublayer.

That helps gradients flow through deep Transformer stacks.

When a model has dozens or hundreds of layers, this becomes critical.

Pre-LN is not just a formatting choice.

It is a training stability choice.

RMSNorm

RMSNorm is a simpler normalization method.

LayerNorm recenters and rescales.

RMSNorm mainly rescales using the root mean square.

The RMS is:

RMS(h) = sqrt((1 / n) * Σ hᵢ²)

Then the normalized vector is:

h_norm = h / (RMS(h) + ε) * g

Why use it?

It keeps activation scale stable.

It removes some computation compared with LayerNorm.

It works well in large LLMs.

Example:

h = [3, 4]

RMS(h) = sqrt((9 + 16) / 2) ≈ 3.54

Normalized h ≈ [0.85, 1.13]

The key idea:

RMSNorm stabilizes scale without doing more than necessary.

Attention Block: GQA + RoPE

Modern attention is often not plain Multi-Head Attention.

It usually combines memory-aware attention with positional encoding.

Grouped-Query Attention reduces KV Cache size.

Rotary Positional Embedding injects position into Query and Key.

The attention flow becomes:

Input

→ Q, K, V projection

→ Apply RoPE to Q and K

→ Share K/V by groups using GQA

→ Compute attention

→ Output projection

This matters for inference.

Long-context generation is often limited by KV Cache memory.

GQA reduces that pressure.

RoPE keeps position information inside attention without adding a large position table.

SwiGLU

The Feed-Forward Network is not just a simple MLP anymore.

Many modern LLMs use SwiGLU.

SwiGLU is a gated activation.

One path carries information.

Another path controls how much passes through.

A simplified formula:

SwiGLU(x) = (W₁x) * Swish(W₂x)

Example:

W₁x = 4

Swish(W₂x) = 0.5

Output = 2

The gate decides how much information moves forward.

That gives the FFN more control than a plain activation.

Mixture of Experts

Mixture of Experts increases model capacity without activating every parameter for every token.

Instead of one FFN, the model has multiple expert networks.

A router chooses which experts handle each token.

Example router output:

Expert 1 = 0.45

Expert 2 = 0.19

Expert 3 = 0.05

Expert 4 = 0.31

With Top-2 routing:

Expert 1 and Expert 4 are selected.

Only those experts run.

This is why MoE is called sparse.

The model may have many parameters.

But each token uses only a small subset.

Dense FFN vs MoE

Dense FFN:

every token uses the same FFN
all FFN parameters are active
simpler to train and serve
compute grows directly with FFN size

MoE:

each token is routed to selected experts
only part of the model activates
increases total capacity efficiently
adds routing and load-balancing complexity

The key difference:

Dense FFN = same compute path for every token

MoE = conditional compute path per token

MoE is powerful.

But it is not free.

It introduces routing instability, expert imbalance, and distributed communication overhead.

Multi-Token Prediction

Standard language modeling predicts one next token.

At position t:

predict token t + 1

Multi-Token Prediction trains the model to predict multiple future tokens.

At position t:

predict token t + 1, t + 2, t + 3 ...

This gives more learning signals from the same representation.

Standard training:

one position → one supervision signal

MTP training:

one position → multiple supervision signals

This can improve sample efficiency.

In some systems, it can also support faster generation ideas.

Naive vs Modern View

Naive view:

Transformer block = attention + FFN

Modern view:

Transformer block = stable normalization + efficient attention + gated FFN + sparse scaling

Naive block:

attention
ffn

Modern block:

rmsnorm
rope
gqa
residual
rmsnorm
swiglu or moe
residual

This matters because modern LLM performance is not just about parameter count.

It is about architecture details that make those parameters trainable and deployable.

Implementation Perspective

When reading modern LLM code, look for these patterns:

self.input_layernorm = RMSNorm(...)

self.self_attn = Attention(..., rope=True, num_key_value_heads=...)

self.post_attention_layernorm = RMSNorm(...)

self.mlp = SwiGLU(...) or MoE(...)

The key clue for GQA is:

number of query heads > number of key-value heads

The key clue for RoPE is:

position is applied to Q and K before attention

The key clue for MoE is:

router logits decide which experts run

These details tell you what kind of Transformer block you are actually looking at.

Important Conditions and Limits

Pre-LN improves stability, but the whole optimization setup still matters.

RMSNorm is efficient, but it does not replace good initialization or training design.

GQA reduces KV Cache memory, but may trade off some attention flexibility.

RoPE works well for long contexts, but very long extrapolation may still need scaling techniques.

SwiGLU improves FFN behavior, but increases FFN structure complexity.

MoE increases capacity, but adds routing and system complexity.

Modern Transformer design is a trade-off system.

Every upgrade solves one bottleneck and introduces another design choice.

Why This Matters Again

Modern LLMs are not just large neural networks.

They are carefully engineered stacks.

If you understand the block, you can better understand:

why inference needs KV Cache optimization
why RoPE appears in attention code
why RMSNorm replaces LayerNorm
why GQA changes memory usage
why MoE models can be huge but still sparse

This is the difference between using LLMs and understanding how they scale.

Takeaway

Modern Transformer blocks preserve the original Transformer idea.

But they upgrade almost every practical detail.

The shortest version:

Modern Transformer Block = Pre-LN/RMSNorm + GQA/RoPE Attention + SwiGLU/MoE FFN + Residual Connections

If Self-Attention is the core idea, the modern block is the production-grade version of that idea.

It is built for depth, context length, inference memory, and scalable capacity.

Discussion

When reading modern LLM architecture, which component feels most important to understand first?

RMSNorm, RoPE, GQA, SwiGLU, or MoE?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/modern-transformer-blocks-llm-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

Why Positional Embeddings Matter — APE, RPE, and RoPE Explained for Developers

zeromathai — Fri, 26 Jun 2026 15:01:50 +0000

Self-Attention can compare every token with every other token.

But there is a catch.

By itself, it does not know the order of tokens.

That is a serious problem because “dog bites man” and “man bites dog” use the same words but mean completely different things.

Core Idea

A Transformer needs two kinds of information:

what the token is

where the token is

Token embeddings provide the “what.”

Positional embeddings provide the “where.”

This matters because attention without position is order-blind.

It can compare tokens, but it does not naturally know which token came first.

The Key Structure

A simple positional embedding flow looks like this:

Token Embedding + Positional Information → Input Representation

For Absolute Positional Embedding:

E = X + P

Where:

X = token embedding

P = positional embedding

E = final input representation

More compactly:

Transformer input = meaning vector + position signal

Different positional methods change how the position signal is injected.

Pseudo-code View

Basic positional injection:

tokens = tokenize(text)

x = embedding(tokens)

position = positional_embedding(token_positions)

input_representation = x + position

For attention-based position methods:

q = project_query(x)

k = project_key(x)

q = apply_position(q)

k = apply_position(k)

attention_scores = q @ k.T

APE usually modifies the input embedding.

RPE usually modifies the attention score.

RoPE usually modifies Query and Key.

That difference is the whole story.

Concrete Example

Compare these two sentences:

dog bites man

man bites dog

The token set is the same:

dog, bites, man

But the order changes the meaning.

Without positional information, Self-Attention sees token relationships but has no built-in sequence order.

With positional information, each token representation includes location.

So “dog” at position 1 is different from “dog” at position 3.

This is why positional encoding is not optional.

It is required for language understanding.

APE: Absolute Positional Embedding

Absolute Positional Embedding assigns a vector to each position index.

Position 1 has one vector.

Position 2 has another vector.

Position 3 has another vector.

Then the model adds that position vector to the token embedding.

Example:

Token embedding:

X = [0.2, 0.5]

Position embedding:

P = [0.1, -0.2]

Final representation:

E = [0.3, 0.3]

APE is easy to understand.

It says:

this token is at this exact position

Why APE Is Useful

APE is simple.

It is easy to implement.

It works well when sequence lengths stay close to what the model saw during training.

Implementation-wise, it is just:

x = token_embedding + position_embedding

That makes it cheap and clean.

But the simplicity has a cost.

APE treats position as a fixed index.

If the model sees much longer inputs than it was trained on, unseen positions can become unreliable.

That makes APE weaker for long-context extrapolation.

RPE: Relative Positional Embedding

Relative Positional Embedding focuses on distance.

Instead of asking:

What position is this token at?

It asks:

How far apart are these two tokens?

This is often more natural for language.

A subject and verb may appear at different absolute positions.

But their relative distance and direction still matter.

A simplified RPE attention score looks like this:

Aᵢⱼ = (QᵢKⱼᵀ + Rᵢ₋ⱼ) / √d

Rᵢ₋ⱼ represents the relative position between token i and token j.

This means position directly affects attention.

Concrete RPE Example

Suppose:

QᵢKⱼᵀ = 12

Rᵢ₋ⱼ = 4

√d = 4

Then:

Aᵢⱼ = (12 + 4) / 4 = 4

Without the relative term:

Aᵢⱼ = 12 / 4 = 3

So the distance relationship increased the attention score.

That is the intuition.

RPE lets the model say:

This token is more relevant because of where it is relative to me.

RoPE: Rotary Positional Embedding

Rotary Positional Embedding takes a different path.

It does not add a position vector to the input.

It rotates Query and Key vectors based on position.

The core idea:

position becomes rotation

A 2D rotation matrix looks like this:

Rθ = [[cosθ, -sinθ], [sinθ, cosθ]]

If you rotate [1, 0] by 90 degrees:

[1, 0] → [0, 1]

RoPE applies this idea across Query and Key dimensions.

Different positions get different rotations.

Then attention scores naturally include relative position.

Why RoPE Works Well

RoPE uses absolute position to rotate Q and K.

But when Q and K are compared, the score depends on their relative position difference.

The key relationship is:

(RθⁱQ)ᵀ(RθʲK) = QᵀRθʲ⁻ⁱK

This means the attention score contains j - i.

That is the relative distance.

So RoPE gives you a useful combination:

absolute-position injection + relative-position behavior

This is why RoPE became popular in modern LLMs.

APE vs RPE vs RoPE

APE:

adds position vectors to token embeddings
simple and cheap
good for fixed or known sequence lengths
weaker for long-context extrapolation

RPE:

adds relative distance information to attention scores
directly models token-to-token distance
flexible for variable lengths
can complicate attention implementation

RoPE:

rotates Query and Key vectors by position
makes relative distance appear inside attention
memory-efficient
works well with modern long-context LLMs

The key difference:

APE = where am I?

RPE = how far are we?

RoPE = rotate Q/K so distance appears in attention

Implementation Perspective

If you are reading Transformer code, look at where position enters the model.

APE usually appears near the embedding layer:

x = token_embedding + position_embedding

RPE usually appears inside attention score computation:

scores = q @ k.T + relative_position_bias

RoPE usually appears after Q and K projection:

q = apply_rope(q, positions)

k = apply_rope(k, positions)

scores = q @ k.T

This is the developer shortcut.

Find the injection point.

Then you know which positional method the model uses.

Naive vs Practical View

Naive view:

Positional embedding just tells the model token order.

Practical view:

Positional design affects long-context behavior, caching, memory, and attention quality.

Naive mindset:

add positions
run attention

Practical mindset:

choose how position enters attention
consider context length
consider extrapolation
consider KV Cache compatibility
consider implementation complexity

This matters because positional encoding is not a small detail.

It changes how the model behaves when the context becomes long.

Why This Matters Again

Short inputs can hide positional weaknesses.

Long-context models expose them.

If positional information does not extrapolate well, the model may become unstable outside its training length.

This is why modern LLMs care so much about RoPE variants and long-context scaling.

The position method affects whether a model can reliably handle long prompts, code files, documents, and conversations.

Important Conditions and Limits

APE is easy but tied to absolute indices.

RPE is expressive but can complicate attention computation.

RoPE is efficient and practical, but still needs careful scaling for very long contexts.

Also:

Positional embeddings do not create reasoning by themselves.

They only give attention a way to use order.

The model still needs training to learn useful patterns.

Takeaway

Self-Attention needs positional information because it is order-blind by default.

APE adds absolute position to embeddings.

RPE adds relative distance to attention scores.

RoPE rotates Query and Key vectors so relative position appears naturally.

The shortest version:

Positional Embedding = the order signal that makes attention understand sequence structure

If you understand where position enters the model, you understand the difference between APE, RPE, and RoPE.

Discussion

When learning Transformer internals, which positional method feels most intuitive to you?

APE, RPE, or RoPE?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/advanced-positional-embeddings-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

Why KV Cache Matters — How MQA, GQA, and MLA Make LLM Inference Faster

zeromathai — Thu, 25 Jun 2026 14:15:58 +0000

LLMs generate text one token at a time.

That sounds simple.

But without KV Cache, every new token would repeat a lot of old work.

That is why inference optimization starts with keys and values.

Core Idea

KV Cache stores previously computed Key and Value tensors.

During generation, the model only needs to compute the new token’s Query, Key, and Value.

Then the new Query attends to cached Keys and Values.

This matters because autoregressive generation repeats the same context again and again.

KV Cache removes a huge amount of duplicated computation.

The Key Structure

Autoregressive generation:

Prompt tokens

→ compute K/V

→ store K/V in cache

→ generate next token

→ append new K/V

→ repeat

More compactly:

KV Cache = reuse past K/V + compute only new K/V

But there is a trade-off.

KV Cache reduces recomputation.

It does not remove attention cost.

And as context length grows, the cache itself becomes large.

Pseudo-code View

Without KV Cache:

context = prompt_tokens

while not finished:
    Q, K, V = compute_qkv(context)

    output = attention(Q, K, V)

    next_token = sample(output)

    context.append(next_token)

With KV Cache:

context = prompt_tokens

K_cache, V_cache = compute_and_store_kv(context)

while not finished:
    q_new, k_new, v_new = compute_qkv(new_token)

    K_cache.append(k_new)
    V_cache.append(v_new)

    output = attention(q_new, K_cache, V_cache)

    next_token = sample(output)

The optimized version avoids recomputing K and V for old tokens.

That is the main speedup.

Concrete Example

Prompt:

Dear

The model generates:

Sarah

Next context:

Dear Sarah

Without KV Cache:

The model recomputes K/V for “Dear” again.

With KV Cache:

The model reuses the cached K/V for “Dear.”

It only computes new K/V for “Sarah.”

Now extend this to a 10,000-token conversation.

Recomputing old tokens becomes wasteful.

Caching becomes essential.

What KV Cache Reduces

KV Cache reduces repeated computation.

Specifically:

past Key computation
past Value computation
repeated projection work for old tokens

But it does not eliminate everything.

The new Query still attends to cached Keys and Values.

So longer context still costs more.

This matters in production.

A long chat can become memory-heavy even if generation is optimized.

The New Bottleneck

KV Cache speeds up inference.

But it also creates a memory problem.

For every layer, every token stores Key and Value tensors.

Longer context means larger cache.

More users mean more cache memory.

More heads mean more K/V tensors.

So the bottleneck shifts:

Before KV Cache:

recompute cost

After KV Cache:

memory cost

This is why MQA, GQA, and MLA exist.

MHA vs MQA vs GQA vs MLA

The main difference is how Key and Value tensors are stored.

Standard Multi-Head Attention:

Each head has its own K/V.

Multi-Query Attention:

All heads share one K/V.

Grouped-Query Attention:

Groups of heads share K/V.

Multi-Head Latent Attention:

K/V information is stored in compressed latent form.

The goal is the same:

reduce KV Cache size while preserving useful attention behavior.

Multi-Head Attention

In standard Multi-Head Attention, each head has separate Query, Key, and Value projections.

If there are 8 heads:

8 heads → 8 K/V pairs

This is expressive.

Each head can learn its own representation.

But it is expensive during inference.

More heads mean larger cache.

So MHA gives quality and flexibility.

But it pays with memory.

Multi-Query Attention

Multi-Query Attention keeps different Queries for each head.

But all heads share the same Key and Value.

If there are 8 heads:

8 query heads → 1 shared K/V pair

This sharply reduces cache size.

It is memory-efficient.

But there is a trade-off.

Because all heads share K/V, head diversity can decrease.

So MQA is fast and compact.

But it may lose some expressiveness.

Grouped-Query Attention

Grouped-Query Attention is the compromise.

Instead of one shared K/V for all heads, it divides heads into groups.

Each group shares one K/V pair.

Example:

8 heads

2 groups

→ 2 K/V pairs

This sits between MHA and MQA.

MHA stores 8 K/V pairs.

MQA stores 1 K/V pair.

GQA stores a configurable middle ground.

That makes GQA practical for modern LLM inference.

Multi-Head Latent Attention

Multi-Head Latent Attention goes further.

Instead of storing full K/V tensors directly, it stores compressed latent representations.

Then it reconstructs or projects the needed information during attention.

The idea is:

store less

recover enough

This is especially useful for long-context inference.

Because when context length grows, KV Cache grows with it.

MLA attacks the memory problem at the representation level.

Comparison Table

MHA:

separate K/V per head
high expressiveness
large KV Cache

MQA:

one shared K/V for all heads
smallest shared-KV cache
possible quality trade-off

GQA:

shared K/V per head group
balanced memory and quality
common practical compromise

MLA:

compressed latent K/V
strong cache reduction
useful for long-context models

Implementation Perspective

In real inference systems, KV Cache is not just a model detail.

It affects:

latency
GPU memory
batch size
max context length
serving cost

A model with a smaller KV Cache can serve longer contexts or more users on the same hardware.

That is why shared K/V designs matter.

They are not just architecture theory.

They directly affect deployment.

Naive vs Practical View

Naive view:

LLM inference = run the model repeatedly

Practical view:

LLM inference = manage cached states efficiently

Naive generation:

recompute all token states every step

Optimized generation:

cache past K/V
compute only new token states
reduce K/V storage with MQA, GQA, or MLA

This is one of the biggest differences between understanding Transformers conceptually and running them efficiently.

Important Conditions and Limits

KV Cache does not make attention free.

The new Query still attends over cached tokens.

Long context still increases memory and latency.

MQA reduces memory but may reduce head diversity.

GQA balances memory and quality.

MLA reduces cache size through compression, but adds architectural complexity.

So the real design question is:

How much memory can we save without hurting generation quality too much?

Why This Matters Again

Long-context models are useful only if inference is practical.

A model that supports huge context but cannot fit enough cache in GPU memory is hard to serve.

KV Cache makes autoregressive generation faster.

MQA, GQA, and MLA make KV Cache more scalable.

That is why modern LLM architecture spends so much effort on shared or compressed Key-Value attention.

Takeaway

KV Cache reuses past Keys and Values.

MQA shares K/V across all heads.

GQA shares K/V within groups.

MLA compresses K/V into latent representations.

The shortest version:

KV optimization = faster generation + smaller memory footprint

If attention is the engine, KV Cache is the memory system that keeps generation practical.

Discussion

When optimizing LLM inference, which bottleneck do you usually notice first?

Latency, GPU memory, context length, or serving cost?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/kv-cache-shared-key-value-attention-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

Why Attention Becomes the Bottleneck — And How Efficient Attention Fixes It

zeromathai — Wed, 24 Jun 2026 14:23:42 +0000

Your model got smarter.

But suddenly it got slower.

Why does increasing context length explode compute?

Because attention is O(n²).

And that becomes the real bottleneck in modern LLMs.

Core Idea

Attention compares every token with every other token.

That is powerful.

But it is expensive.

Efficient Attention methods try to answer one question:

How do we keep useful context while reducing cost?

This matters because long-context LLMs are useless if they are too slow or too expensive.

The Key Structure

Full Attention cost:

Attention Cost = O(n²)

Meaning:

n tokens → n × n comparisons

Example:

1,000 tokens → 1M comparisons

10,000 tokens → 100M comparisons

10× longer input → 100× more work

That is the bottleneck.

More compactly:

Attention = full connectivity + quadratic cost

Efficient Attention = reduce connections or optimize computation

Pseudo-code View

Full attention:

for i in tokens:
    for j in tokens:
        score[i][j] = dot(Q[i], K[j])

Efficient attention idea:

restrict or optimize comparisons

for i in tokens:
    for j in selected_tokens:
        score[i][j] = dot(Q[i], K[j])

Or:

compute same attention
but optimize memory access

Two strategies:

reduce what you compute
optimize how you compute

Concrete Example

Imagine reading a 10,000-token document.

Full Attention:

Every word looks at every other word.

That is like comparing every sentence to every sentence.

Local Attention:

Each word looks only at nearby words.

Like reading paragraph by paragraph.

Sparse Attention:

Each word looks at selected words.

Like focusing on keywords and headings.

FlashAttention:

Still reads everything.

But does it efficiently by avoiding unnecessary memory movement.

Different methods.

Same goal:

Reduce cost without losing important context.

Full Attention vs Efficient Attention

Full Attention:

connects every token to every token
captures long-range dependencies
expensive in compute and memory

Efficient Attention:

reduces connections or optimizes execution
scales to longer sequences
trades off some flexibility for efficiency

The key difference:

Full = maximum connectivity

Efficient = selective or optimized connectivity

Local Attention

Local Attention limits attention to a window.

Example:

Each token attends to last 128 tokens.

Cost becomes:

O(n × window)

Instead of O(n²)

This works because:

Nearby context often matters most.

But limitation:

Long-range dependencies can be missed.

Sparse Attention

Sparse Attention generalizes Local Attention.

Instead of full connections:

Use structured patterns.

Examples:

local windows
strided attention
global tokens
block patterns

This reduces cost while keeping some long-range connections.

But trade-off:

Too sparse → lose important relationships

So many models mix:

full attention + sparse attention layers

FlashAttention

FlashAttention does NOT change attention logic.

It changes how attention is computed.

Problem:

Attention is often memory-bound.

GPU spends time moving data, not computing.

FlashAttention solution:

compute attention in blocks
keep data in fast SRAM
avoid storing large intermediate matrices

Instead of:

store full attention matrix → read again

It does:

compute on-the-fly → minimize memory movement

Key idea:

Optimize IO, not just math

Naive vs Optimized View

Naive view:

Attention cost = math operations

Optimized view:

Attention cost = math + memory movement

Naive:

compute QK^T
store matrix
apply softmax

Optimized (FlashAttention):

compute in chunks
avoid large memory writes
reuse data efficiently

This is why FlashAttention speeds up real systems.

Not by changing theory.

But by fixing hardware inefficiency.

Why This Matters (Again)

Early:

Attention made Transformers powerful.

Now:

Attention limits how far they can scale.

If you cannot optimize attention:

context stays short
inference becomes slow
cost explodes

Efficient attention enables:

longer context windows
faster inference
lower GPU cost
production-scale LLM systems

Important Conditions and Limits

Local Attention:

fast
but weak for long-range dependencies

Sparse Attention:

flexible
but pattern design matters

FlashAttention:

exact attention
but requires hardware-aware implementation

Also:

Even optimized attention still grows with sequence length.

There is no free lunch.

Only better trade-offs.

Takeaway

Attention is the core of Transformers.

But it is also the bottleneck.

Full Attention = powerful but expensive

Efficient Attention = scalable but selective or optimized

The shortest version:

Efficient Attention = reduce connections OR optimize memory access

If you understand that, you understand why modern LLM engineering focuses so much on attention optimization.

Discussion

When working with long-context models, which matters more to you?

Accuracy from full attention or efficiency from optimized attention?

Originally published at zeromathai.com
Original article: https://zeromathai.com/en/efficient-attention-flashattention-sparse-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

How Transformer Decoders Generate Text — From Causal Masking to Decoding

zeromathai — Tue, 23 Jun 2026 14:43:46 +0000

A Transformer Decoder does not generate a sentence all at once.

It predicts one token.

Then it feeds that token back and predicts the next one.

That simple loop is the core of modern LLM generation.

Core Idea

A Transformer Decoder is built for autoregressive generation.

That means:

previous tokens → next token prediction → repeat

The Decoder creates hidden representations.

The LM Head converts those representations into vocabulary scores.

A decoding strategy chooses the actual next token.

This matters because generation quality is not only about the model.

It also depends on how tokens are selected.

The Key Structure

A simplified generation pipeline looks like this:

Input Context

→ Decoder Layers

→ Hidden State

→ LM Head

→ Logits

→ Softmax

→ Decoding Strategy

→ Next Token

More compactly:

Text Generation = decoder representation + vocabulary scoring + token selection

The Decoder answers:

What should the next representation be?

The LM Head answers:

Which vocabulary tokens are likely?

The decoding strategy answers:

Which token should we actually output?

Pseudo-code View

Autoregressive decoding looks like this:

context = prompt_tokens

while not stop:
    hidden = decoder(context)

    logits = lm_head(hidden[-1])

    probs = softmax(logits / temperature)

    next_token = decode(probs)

    context.append(next_token)

The key loop is:

predict → append → repeat

This is why LLM inference is sequential.

Even if training can be parallelized, generation still produces tokens one step at a time.

Transformer Decoder Structure

A Transformer Decoder layer usually contains:

Masked Self-Attention
Cross-Attention
Feed-Forward Network

Masked Self-Attention lets the Decoder look only at previous tokens.

Cross-Attention lets it look at Encoder outputs when an input sequence exists.

The Feed-Forward Network transforms each token representation.

For decoder-only LLMs, Cross-Attention is usually removed.

The model only continues from the current context.

Causal Masking

The Decoder must not cheat.

When predicting token 5, it cannot look at token 6.

That is the role of the causal mask.

The generation probability can be written as:

P(y₁, y₂, ..., yₜ | x) = Π P(yₜ | y₁, ..., yₜ₋₁, x)

Each token depends only on previous output tokens and the input.

This is important.

Without causal masking, the model could see future answers during training.

Then it would fail during real generation.

Concrete Example

Target sentence:

I love you

During training, the Decoder input is shifted right:

Input:

I love

Target:

I love you

So the model learns:

→ I

I → love

I love → you

At inference time, there is no target sentence.

The model must use its own previous output.

That is why errors can accumulate during generation.

Teacher Forcing

Teacher forcing is used during training.

Instead of feeding the model’s wrong prediction back into the next step, we feed the correct previous token.

This makes training more stable.

Training:

input = correct previous tokens

Inference:

input = model-generated previous tokens

This difference matters.

A model can behave well during training but drift during generation.

That is why decoding strategy and evaluation matter in real systems.

LM Head and Logits

The Decoder outputs hidden vectors.

But hidden vectors are not tokens.

The LM Head maps a hidden vector to vocabulary-sized scores.

These scores are called logits.

If the vocabulary size is 50,000, the LM Head outputs 50,000 scores.

Each score corresponds to one possible next token.

Logits are not probabilities yet.

Softmax converts them into probabilities.

The pipeline is:

hidden state → logits → probabilities → selected token

Temperature Scaling

Temperature controls how sharp or flat the probability distribution becomes.

The formula is:

pᵢ(τ) = exp(zᵢ / τ) / Σ exp(zⱼ / τ)

Lower temperature:

sharper distribution
more deterministic output
less randomness

Higher temperature:

flatter distribution
more diverse output
more randomness

Example:

With logits [2, 1, 0]:

temperature = 0.5 makes the top token much stronger.

temperature = 2 makes lower-ranked tokens more likely.

This matters in practice.

Temperature is one of the simplest ways to control creativity.

What Decoding Means

Decoding means selecting the next token from probabilities.

The model gives a distribution.

The decoding algorithm makes a choice.

That choice affects:

correctness
creativity
repetition
diversity
determinism
latency

So decoding is not a small detail.

It is part of the generation behavior.

Greedy Decoding

Greedy decoding always chooses the most likely token.

If probabilities are:

A = 0.70

B = 0.20

C = 0.10

Greedy always picks A.

It is simple and fast.

But it can be repetitive.

It can also choose a locally good token that leads to a worse full sentence.

Beam Search

Beam search keeps multiple candidate sequences.

Instead of only keeping the best next token, it keeps the best k paths.

If beam size = 3, the model tracks three candidate continuations.

This can improve structured generation.

But it can also reduce diversity.

When k = 1, beam search becomes greedy decoding.

Top-k Sampling

Top-k sampling keeps only the k most likely tokens.

Then it samples from that smaller set.

Example:

k = 3

Only the top 3 tokens can be selected.

This prevents the model from choosing extremely unlikely tokens.

But it still allows some randomness.

Top-k is useful when you want controlled diversity.

Top-p Sampling

Top-p sampling is also called nucleus sampling.

Instead of keeping a fixed number of tokens, it keeps the smallest set whose cumulative probability exceeds p.

Example:

Token probabilities:

honeycomb = 0.45

gingerbread = 0.20

donut = 0.12

cupcake = 0.04

If p = 0.6:

honeycomb + gingerbread = 0.65

So only those two tokens enter the sampling set.

Top-p adapts to the confidence of the model.

That makes it more flexible than fixed Top-k.

Deterministic vs Stochastic Decoding

Deterministic decoding:

greedy decoding
beam search
same input usually gives same output
useful for predictable tasks

Stochastic decoding:

Top-k sampling
Top-p sampling
can generate different outputs
useful for creative tasks

The difference is simple:

Deterministic = choose the best-looking path

Stochastic = sample from likely paths

For coding tasks, deterministic settings are often useful.

For brainstorming, stochastic settings are often better.

Encoder-Decoder vs Decoder-Only Models

Encoder-Decoder models use both input understanding and output generation.

They are useful for tasks like translation.

The Encoder reads the source sequence.

The Decoder generates the target sequence.

Decoder-only models use only the generation stack.

They predict the next token from the previous context.

Most GPT-style LLMs are decoder-only.

The architecture is simpler for open-ended text generation.

Implementation Perspective

In real inference code, generation is not just:

model(prompt)

It is closer to:

tokenize prompt

run decoder

get logits from LM Head

apply temperature

filter with top-k or top-p

sample or choose token

append token

repeat

This matters because small decoding changes can produce very different outputs.

A model can feel precise, boring, creative, unstable, or repetitive depending on decoding settings.

The model gives probabilities.

Your decoding pipeline turns those probabilities into behavior.

Naive vs Practical View

Naive view:

LLM = text in, text out

Practical view:

LLM = token loop + logits + decoding policy

Naive mindset:

ask model
receive answer

Practical mindset:

manage context
control temperature
choose decoding strategy
stop generation correctly
handle repetition
optimize inference cost

This is why developers need to understand the Decoder.

Generation is a system, not a single function call.

Important Conditions and Limits

Decoder generation is sequential.

Each new token depends on previous tokens.

That can make inference slow.

Causal masking is required to prevent future-token leakage.

Teacher forcing helps training, but inference uses the model’s own predictions.

Decoding strategy changes output behavior.

Temperature, Top-k, and Top-p are not cosmetic options.

They directly shape the generated text.

Takeaway

The Transformer Decoder generates text by predicting one token at a time.

Masked Self-Attention prevents future-token access.

The LM Head converts hidden states into vocabulary logits.

Softmax turns logits into probabilities.

Decoding chooses the actual next token.

The shortest version is:

Decoder generation = causal attention + LM Head + decoding loop

If you understand that loop, you understand how LLMs actually produce text.

Discussion

When tuning LLM output, which setting do you usually adjust first?

Temperature, Top-k, Top-p, or the prompt itself?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/transformer-decoder-lm-head-decoding-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

Why Multi-Head Attention Needs Position, Residuals, and Normalization

zeromathai — Mon, 22 Jun 2026 14:29:20 +0000

Self-Attention is powerful.

But by itself, it has three problems.

It needs multiple views, it needs word order, and it needs stable training.

That is why Multi-Head Attention, Positional Encoding, and Add & Norm exist.

Core Idea

A Transformer block is not just attention.

Attention computes token relationships.

Multi-Head Attention makes those relationships richer.

Positional Encoding tells the model where tokens are.

Add & Norm keeps deep Transformer blocks trainable.

This matters because modern LLMs are deep.

Without these support structures, attention alone is not enough.

The Key Structure

A simplified Transformer encoder block looks like this:

Input

→ Positional Information

→ Multi-Head Attention

→ Add & Norm

→ Feed-Forward Network

→ Add & Norm

→ Output

More compactly:

Transformer Block = attention + position + residual flow + normalization

Each part solves a specific problem.

Multi-Head Attention solves the “single view” problem.

Positional Encoding solves the “no order” problem.

Add & Norm solves the “deep training stability” problem.

Implementation View

At a high level, the block works like this:

tokens = tokenize(text)

x = embedding(tokens)

x = x + positional_encoding

attention_output = multi_head_attention(x)

x = layer_norm(x + attention_output)

ffn_output = feed_forward(x)

output = layer_norm(x + ffn_output)

In modern Pre-LN style, the order often changes:

attention_output = multi_head_attention(layer_norm(x))

x = x + attention_output

ffn_output = feed_forward(layer_norm(x))

output = x + ffn_output

The idea is the same.

Keep the original signal flowing.

Normalize activations.

Let attention and FFN update the representation.

Multi-Head Attention

Single attention gives one relationship map.

But language has many relationship types.

A token may need to track:

nearby words
subject-verb structure
semantic similarity
long-distance references
coreference

One attention head cannot easily capture all of these at once.

Multi-Head Attention fixes this by running several attention heads in parallel.

Each head gets its own learned Q, K, and V projections.

So each head can learn a different representation subspace.

The formula is:

MultiHead(Q, K, V) = Concat(head₁, ..., headₕ)Wᴼ

In plain English:

Run attention multiple ways.

Concatenate the results.

Project them back into one vector space.

Concrete Example

Take this sentence:

The animal did not cross the street because it was tired.

What does “it” refer to?

One attention head may focus on “animal.”

Another may focus on “tired.”

Another may track the structure around “because.”

This is useful because the model does not need one attention map to explain everything.

Different heads can specialize.

That is why Multi-Head Attention matters.

It gives the model multiple ways to read the same sentence.

Single-Head vs Multi-Head Attention

Single-head attention:

uses one attention distribution
sees token relationships from one perspective
is simpler
can mix different patterns together

Multi-head attention:

uses multiple attention distributions
views tokens through different learned projections
captures diverse relationships
recombines the results afterward

The key difference:

Single-head = one view of context

Multi-head = multiple views of context

This is not just repeated computation.

It is structured parallel interpretation.

Why Positional Encoding Is Needed

Self-Attention compares tokens at the same time.

That is great for parallelism.

But it creates a problem.

Attention alone does not know token order.

Consider:

dog bites man

man bites dog

Same words.

Different meaning.

Without position information, the model does not naturally know which token came first.

So Transformers inject position into token representations.

The basic structure is:

Input Representation = Token Embedding + Positional Encoding

This matters because word order changes meaning.

A language model must know both:

what the token is

where the token is

Sinusoidal Positional Encoding

The original Transformer used fixed sine and cosine patterns.

Even dimensions use sine.

Odd dimensions use cosine.

The idea is:

different positions get different wave patterns.

A simplified view:

PE(pos, 2i) = sin(pos / 10000^(2i / d_model))

PE(pos, 2i + 1) = cos(pos / 10000^(2i / d_model))

This is not just a position ID.

It creates smooth position signals across dimensions.

The model can use these signals to reason about position and distance.

APE, RPE, and RoPE

There are several ways to inject position.

Absolute Positional Embedding:

assigns a position vector to each absolute index
position 1 has one vector
position 2 has another vector

Relative Positional Embedding:

focuses on distance between tokens
useful when relative position matters more than absolute index

Rotary Positional Embedding:

rotates Query and Key vectors using position
makes relative position work naturally inside attention
commonly used in modern LLMs

The shared goal is simple:

Give attention a way to understand order.

Add & Norm

A Transformer block also needs stability.

That is where Add & Norm comes in.

Add means residual connection.

Norm means layer normalization.

The classic Post-LN formula is:

Output = LayerNorm(x + Sublayer(x))

The residual connection preserves the original input.

The sublayer adds a learned update.

LayerNorm keeps the representation stable.

This is important because Transformers stack many layers.

Without residual paths, information can degrade.

Without normalization, training can become unstable.

Residual Connection Intuition

A sublayer should not have to rebuild everything from scratch.

It should only need to learn an update.

Instead of:

new_output = sublayer(x)

Use:

new_output = x + sublayer(x)

This is a huge difference.

The original representation can pass forward directly.

The sublayer only adds useful changes.

That makes deep networks easier to train.

Layer Normalization Intuition

LayerNorm normalizes each token representation.

It works across the feature dimensions of a token.

Not across the sequence.

That means each token vector is stabilized independently.

In practice, this helps keep activation values in a manageable range.

This matters when many Transformer blocks are stacked.

Small instability in one layer can grow across dozens or hundreds of layers.

Pre-LN vs Post-LN

Post-LN:

x = LayerNorm(x + Sublayer(x))

Pre-LN:

x = x + Sublayer(LayerNorm(x))

The original Transformer used Post-LN.

Many modern large Transformers use Pre-LN.

Why?

Pre-LN often improves training stability in deep models.

The difference is placement.

Post-LN normalizes after the residual addition.

Pre-LN normalizes before the sublayer.

Both use residual connections.

But their training behavior can be different.

Naive vs Practical View

Naive view:

Transformer block = attention layer

Practical view:

Transformer block = attention + position + residuals + normalization + FFN

Naive implementation mindset:

run attention
return output

Practical implementation mindset:

add position
run multi-head attention
preserve input through residuals
normalize representations
apply feed-forward updates
repeat safely across many layers

This is why implementation details matter.

The architecture works because these parts support each other.

Important Conditions and Limits

Multi-Head Attention is powerful, but more heads are not always better.

Too many heads can increase cost.

Some heads may become redundant.

Positional Encoding is necessary because attention is order-agnostic by default.

But different positional methods behave differently in long-context settings.

Add & Norm improves stability.

But the exact Pre-LN or Post-LN choice affects optimization.

So these are not decorative components.

They are architectural decisions.

Takeaway

Multi-Head Attention gives the model multiple views of token relationships.

Positional Encoding gives attention a sense of order.

Add & Norm keeps deep Transformer blocks stable.

The shortest version is:

Transformer Block = multi-view attention + position signal + stable residual updates

If Self-Attention is the engine, these components are the systems that make the engine usable at scale.

Discussion

When reading Transformer architecture, which part feels most important to understand first?

Multi-Head Attention, Positional Encoding, or Add & Norm?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/multi-head-attention-positional-encoding-add-norm-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

How Self-Attention Works — QKV, Softmax, and Matrix Computation

zeromathai — Thu, 18 Jun 2026 14:19:01 +0000

Self-Attention is not just “looking at important words.”

It is a matrix operation.

And that is exactly why Transformers scale.

Core Idea

Self-Attention lets each token compare itself with every other token in the same sequence.

Each token asks:

Which other tokens are useful for updating my representation?

This matters because meaning is contextual.

A token should not stay as a static embedding.

It should become a representation shaped by the sentence around it.

The Key Structure

Self-Attention follows this pipeline:

Input Embeddings

→ Query, Key, Value Projection

→ Similarity Scores

→ Scaling

→ Softmax Weights

→ Weighted Sum of Values

→ Contextual Token Output

More compactly:

Self-Attention = matching + weighting + information mixing

The full formula is:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

This equation looks dense.

But the idea is simple:

Compare tokens.

Convert scores into weights.

Use weights to mix information.

Pseudo-code View

At a high level, Self-Attention works like this:

X = token_embeddings

Q = X @ W_Q
K = X @ W_K
V = X @ W_V

scores = Q @ K.T

scaled_scores = scores / sqrt(d_k)

weights = softmax(scaled_scores)

output = weights @ V

That is the core computation.

In real Transformer implementations, this is done for all tokens at once.

Not token by token.

That is why the matrix form matters.

Concrete Example

Take this sentence:

I love you

When updating the token “love”, Self-Attention compares it with:

I

love

you

The token “love” may strongly attend to “I” and “you”.

So its representation becomes more contextual.

It no longer means only the word “love.”

It becomes something closer to:

love as an action between I and you

That is why Self-Attention is powerful.

It turns isolated token vectors into relationship-aware vectors.

QKV Intuition

Each token is projected into three roles:

Query, Key, and Value.

Query:

What am I looking for?

Key:

What do I contain that others can match against?

Value:

What information do I pass forward if selected?

Search analogy:

Query = search request

Key = searchable index

Value = retrieved content

This separation is important.

The model can learn different spaces for matching and information transfer.

Step 1: Generate Q, K, and V

Given input embeddings X:

Q = XW_Q

K = XW_K

V = XW_V

W_Q, W_K, and W_V are learned matrices.

They are trained with the model.

This means QKV is not manually designed.

The model learns how to project tokens into attention roles.

Implementation-wise, this is just matrix multiplication.

Conceptually, it creates three different views of the same token.

Step 2: Compute Attention Scores

The model compares Query and Key vectors.

For one token:

score = q · k

A larger dot product means stronger similarity.

Example:

q₁ · k₁ = 112

q₁ · k₂ = 96

The first key matches more strongly.

But these are still raw scores.

They are not probabilities yet.

Step 3: Scale and Apply Softmax

Dot products can become large when vector dimensions grow.

Large scores can make Softmax too sharp.

That can make training unstable.

So Self-Attention scales the scores:

score = (q · k) / √dₖ

Then Softmax converts scores into weights.

Example:

scores = [14, 12]

softmax(scores) ≈ [0.88, 0.12]

Now the model has attention weights.

These weights say how much each token should contribute.

This matters in practice.

Without scaling, attention can collapse too aggressively onto one token.

Step 4: Weighted Sum of Values

The final output is a weighted sum of Value vectors.

z = Σ αᵢvᵢ

Example:

values = [10, 20]

weights = [0.88, 0.12]

output = 0.88 × 10 + 0.12 × 20 = 11.2

The first value contributes more.

The second value contributes less.

That is the basic meaning of attention output.

It is not a simple average.

It is selective information mixing.

Self-Attention vs Cross-Attention

Self-Attention:

Query, Key, and Value come from the same sequence
models relationships inside one sequence
used in Transformer encoders and decoders

Cross-Attention:

Query comes from the decoder
Key and Value come from the encoder
models relationships between two sequences
used in encoder-decoder models

In short:

Self-Attention = inside the same sequence

Cross-Attention = between different sequences

This difference matters when reading Transformer code.

If Q, K, and V come from the same tensor, it is Self-Attention.

If Q comes from one tensor and K/V come from another, it is Cross-Attention.

Naive vs Matrix View

Naive view:

Each token compares with every other token one by one.

Matrix view:

All token relationships are computed at once.

Naive logic:

for token_i in tokens:
    for token_j in tokens:
        compute_similarity(token_i, token_j)

Matrix logic:

scores = Q @ K.T

That single matrix multiplication computes all pairwise token scores.

This is why Transformers are GPU-friendly.

They replace sequential loops with dense linear algebra.

Why Matrix Computation Matters

The attention matrix contains token-to-token relationships.

If the sequence length is n, the score matrix is n × n.

Each row means:

How much one token attends to every token.

Each column means:

How much that token is attended to by others.

This structure is powerful.

But it also creates a cost problem.

Full Self-Attention grows roughly with O(n²).

Longer context means more computation and memory.

So the same design that makes attention expressive also makes it expensive.

That is why efficient attention methods exist.

Important Conditions and Limits

Self-Attention needs positional information.

By itself, attention compares token content.

It does not automatically know token order.

Self-Attention also gets expensive as sequence length grows.

For short and medium sequences, full attention is powerful.

For very long sequences, memory and compute become major constraints.

Another important point:

Attention weights are not always perfect explanations.

They show how information is mixed.

But they should not always be treated as human-level reasoning traces.

Implementation Perspective

In real models, QKV projection is often implemented as one combined linear layer.

Instead of computing three separate matrix multiplications:

Q = XW_Q

K = XW_K

V = XW_V

Implementations often compute:

QKV = XW_QKV

Then split the result into Q, K, and V.

This is faster and cleaner.

The math stays the same.

The implementation is optimized.

That is the developer mindset:

Understand the formula.

Then recognize the optimized tensor layout in code.

Takeaway

Self-Attention is the core operation behind Transformers.

It works by projecting tokens into Q, K, and V.

Q and K compute relevance.

Softmax turns relevance into weights.

Weights mix V into contextual outputs.

The shortest version is:

Self-Attention = compare tokens → weight information → update representations

If you understand QKᵀ and weighted Values, you understand the heart of Transformer computation.

Discussion

When reading Transformer code, which part feels most confusing?

QKV projection, Softmax attention weights, or the final matrix multiplication with V?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/self-attention-qkv-matrix-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

How Attention Actually Works — From Next-Token Prediction to QKV Intuition

zeromathai — Wed, 17 Jun 2026 15:38:00 +0000

A language model does not “write sentences.”

It predicts the next token. One step at a time.

So the real question is:

How does it decide what matters right now?

That is why attention exists.

Core Idea

A Language Model = next-token probability estimator.

Given previous tokens, it predicts the next token.

Attention = mechanism that decides which past tokens matter more.

This is critical.

Because not all context is equally useful.

The Key Structure

Language Modeling can be reduced to:

P(x₁, x₂, ..., xₜ) = Π P(xₜ | x₁...xₜ₋₁)

And attention adds:

weighted context selection

More concretely:

Language Model = context + weighting + prediction

Without attention:

All context is compressed.

With attention:

Context is dynamically re-weighted at every step.

Pseudo-code View

Autoregressive generation:

context = ["I", "love"]

while not finished:
    probs = model(context)
    next_token = sample(probs)

    context.append(next_token)

Attention inside the model:

for each token t:
    score = compare(query_t, keys)

    weights = softmax(score)

    output_t = sum(weights * values)

That is the core loop.

Predict → append → repeat.

Concrete Example

Input:

"I love"

Possible next tokens:

you, it, this, pizza

The model assigns probabilities:

you → 0.6

it → 0.2

this → 0.1

pizza → 0.1

Why does “you” win?

Because attention focuses on relationships in context.

“I” + “love” → strong pattern → “you”

Now extend:

"I love you because"

The model must now decide:

What does “because” relate to?

Attention allows it to re-evaluate the entire context.

Not just the last token.

Why Attention Is Needed

Old Seq2Seq models:

compress entire input into one vector
lose information as sequence grows

Attention fixes this:

keeps all token representations
dynamically selects relevant ones

This matters because:

Long sentences break fixed representations.

Attention removes that bottleneck.

QKV Intuition

Attention uses three vectors:

Query, Key, Value

Think like search:

Query = what I want

Key = what each token offers

Value = the actual information

Flow:

compare Query with Keys
compute similarity scores
normalize with softmax
combine Values using weights

That is how context is selected.

The Core Formula

Attention is:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

Meaning:

match Query with Keys
turn matches into probabilities
use those probabilities to mix Values

Result:

Each token becomes context-aware.

Cross Attention and Context Vector

In encoder-decoder models:

Decoder does not rely only on its own tokens.

It looks at Encoder outputs.

Context vector:

c = Σ (attention_weight × encoder_hidden_state)

This is dynamic.

At every step, the model recomputes what matters.

Not a fixed summary.

A moving focus.

Naive vs Real View

Naive view:

Language model = next word generator

Real view:

Language model = dynamic context weighting system

Naive:

predict next token

Real:

compute attention
reweight context
then predict token

That difference is everything.

It explains why Transformers outperform older models.

Important Constraints

Attention is powerful, but not free.

cost grows with sequence length
requires memory for all tokens
depends on good tokenization
still generates sequentially at inference

Also:

Attention does not understand meaning by itself.

It only learns patterns from data.

So quality depends on training.

Why This Matters (Again)

Early:

Without attention → information bottleneck

Now:

With attention → full context + selective focus

This is why modern LLMs work.

Not because they “know language.”

But because they efficiently manage context.

Takeaway

Language Model = next-token prediction.

Attention = context selection.

QKV = mechanism for selecting information.

If you remember one thing:

Attention lets a model decide what to look at before predicting what to say.

That is the core of modern LLMs.

Discussion

When you think about LLM behavior, do you see it more as:

a probability engine or a context selection system?

Originally published at zeromathai.com
Original article: https://zeromathai.com/en/attention-language-modeling-basics-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

How Transformer Architecture Works — Encoder, Decoder, Tokens, and Context

zeromathai — Tue, 16 Jun 2026 15:14:10 +0000

Transformers changed NLP because they stopped treating text as a simple left-to-right chain.

Instead of reading one token at a time, they compare tokens directly.

That shift made modern language models faster, more scalable, and better at understanding context.

Core Idea

A Transformer is a sequence-to-sequence architecture.

It maps an input sequence to an output sequence.

For example:

English sentence → Korean sentence

Question → Answer

Document → Summary

But the key idea is not “replace one word with another word.”

The key idea is:

Transformers build contextual token representations first.

Then they generate or transform output from those representations.

That is why the architecture matters.

It gives the model a structured way to understand relationships inside text.

The Key Structure

A simplified Transformer flow looks like this:

Input Text

→ Tokens

→ Word Embeddings

→ Encoder

→ Contextual Representations

→ Decoder

→ Output Tokens

More compactly:

Transformer = tokenization + embeddings + attention + encoder-decoder structure

The model first converts raw text into tokens.

Then each token becomes a vector.

Then attention updates each vector based on relationships with other tokens.

The Encoder understands the input.

The Decoder generates the output.

Implementation View

At a high level, the architecture works like this:

split input text into tokens

convert tokens into embedding vectors

pass embeddings through encoder layers

for each encoder layer:
    compute self-attention

    mix information across tokens

    apply feed-forward transformation

    produce contextual token representations

pass previous output tokens into decoder

for each decoder layer:
    apply masked self-attention

    attend to encoder output with cross-attention

    apply feed-forward transformation

    predict the next output token

This structure is practical because attention can be computed with matrix operations.

That makes Transformers much more GPU-friendly than step-by-step recurrent models.

This is one of the biggest reasons Transformers scaled so well.

Concrete Example

Take this sentence:

I love you.

An RNN reads it step by step:

I → love → you

A Transformer can compare all tokens directly.

When processing “love”, it can look at both “I” and “you” at the same time.

So “love” is not treated as an isolated word.

It becomes a contextual representation.

The model learns:

Who loves?

Who is loved?

Which tokens are related?

This matters because language is not just a sequence of words.

Language is a structure of relationships.

Sequence-to-Sequence View

A Transformer can be understood as a sequence-to-sequence model.

It receives one sequence.

It produces another sequence.

Examples:

translation
summarization
question answering
text generation
code generation

The input and output lengths do not need to match.

That is important.

A short sentence can become a long explanation.

A long document can become a short summary.

The model is not copying token positions.

It is transforming meaning.

RNN vs Transformer

This comparison explains why Transformers became dominant.

RNN:

processes tokens one by one
keeps information in a hidden state
naturally handles order
is hard to parallelize
can struggle with long-range dependencies

Transformer:

processes tokens in parallel
compares tokens directly
uses attention instead of recurrence
scales better on GPUs
models long-distance relationships more directly

The difference is simple:

RNN = memory through sequence steps

Transformer = relationships through attention

This is why Transformers are not just “faster RNNs.”

They represent sequence information in a different way.

Encoder-Decoder Architecture

The original Transformer uses an Encoder-Decoder structure.

The Encoder reads the input sequence.

The Decoder generates the output sequence.

Encoder:

receives input tokens
applies self-attention
builds contextual representations
outputs one vector per token

Decoder:

receives previously generated tokens
uses masked self-attention
attends to encoder output
predicts the next token

The Encoder answers:

What does the input mean?

The Decoder answers:

What should be generated next?

Transformer Encoder

The Transformer Encoder is a stack of repeated encoder layers.

Each layer has two main parts:

Self-Attention
Feed-Forward Network

Self-Attention lets each token look at other tokens in the same input.

The Feed-Forward Network transforms each token representation independently.

A simplified encoder layer looks like this:

Input

→ Self-Attention

→ Feed-Forward Network

→ Contextual Output

The important part is that every token representation becomes context-aware.

A word is no longer just a word vector.

It becomes a word vector shaped by the sentence around it.

Word Embedding, Tokens, and Vocabulary

A Transformer does not understand raw text directly.

It first splits text into tokens.

A token can be:

a word
a subword
a character-like unit
a special symbol

The full set of possible tokens is called the vocabulary.

Each token is mapped to a vector through an embedding layer.

The flow looks like this:

Raw text

→ Tokens

→ Token IDs

→ Embedding vectors

For example:

"I love you"

→ ["I", "love", "you"]

→ [token_id_1, token_id_2, token_id_3]

→ [vector_1, vector_2, vector_3]

This matters in practice.

When building with LLMs, tokenization affects cost, context length, latency, and output behavior.

So tokens are not just preprocessing details.

They are part of the model interface.

Transformer Decoder

The Transformer Decoder generates output tokens.

It has three main components:

Masked Self-Attention
Cross-Attention
Feed-Forward Network

Masked Self-Attention prevents the model from seeing future tokens.

This is required for autoregressive generation.

When predicting the next token, the model can only use previous tokens.

The flow looks like this:

Previous output tokens

→ Masked Self-Attention

→ Cross-Attention with Encoder Output

→ Feed-Forward Network

→ Next Token Prediction

This is how the model generates text step by step.

It predicts one token.

Then it appends that token.

Then it predicts the next token.

Cross-Attention

Cross-Attention connects the Decoder to the Encoder.

The Decoder asks:

Which part of the input should I focus on right now?

This is especially useful in translation.

The output word order may be different from the input word order.

A phrase in one language may correspond to several words in another language.

Cross-Attention helps the Decoder align output generation with the encoded input.

Without Cross-Attention, the Decoder would generate mainly from its own previous tokens.

With Cross-Attention, it can reference the input meaning directly.

Context Length

Context length means:

How many tokens the model can process at once.

A longer context allows the model to use more information.

This is useful for:

long documents
long conversations
code files
retrieval-augmented generation
summarization

But longer context is not free.

Attention compares tokens with other tokens.

So computational cost grows quickly as the sequence gets longer.

This is why context length is both powerful and expensive.

In real systems, context length affects memory usage, latency, and price.

Naive vs Practical View

Naive view:

A Transformer is a model that takes text and returns text.

Practical developer view:

A Transformer is a token-processing system with attention, context limits, and generation constraints.

Naive mindset:

input text
get output text

Practical mindset:

tokenize input

manage context length

understand attention cost

choose decoding strategy

optimize inference

control output quality

This matters because production AI systems are not only about model accuracy.

They are also about speed, memory, cost, and reliability.

Important Conditions and Limits

Transformers are powerful, but they have important constraints.

They need tokenization before processing text.

They need positional information because attention alone does not know order.

They can become expensive with long context.

Decoder generation is sequential during inference.

Context length limits how much information the model can use at once.

These limits explain why modern LLM engineering focuses so much on:

efficient attention
KV Cache
long-context optimization
better tokenization
inference speed
memory reduction

The architecture is elegant.

But scaling it requires engineering.

Transformer vs Traditional Seq2Seq

Traditional Seq2Seq:

often uses RNN-based Encoder and Decoder
compresses input into hidden states
processes sequence step by step
may lose information in long sequences

Transformer Seq2Seq:

uses attention-based Encoder and Decoder
keeps contextual representations for all tokens
supports parallel computation
models token relationships directly

The key difference:

Traditional Seq2Seq compresses through recurrence.

Transformer Seq2Seq connects through attention.

That is why Transformers became the foundation for modern NLP systems.

Takeaway

A Transformer works by turning tokens into contextual representations.

The Encoder understands the input.

The Decoder generates the output.

Self-Attention models relationships inside a sequence.

Cross-Attention connects generated output to encoded input.

Context length controls how much information the model can use.

If you remember one structure, remember this:

Text → Tokens → Embeddings → Attention → Contextual Representations → Output

That is the backbone of Transformer architecture.

Discussion

When learning Transformers, which part helped you understand the architecture fastest?

The Encoder-Decoder structure, Self-Attention, tokenization, or the generation loop?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/transformer-architecture-core-components-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

How Transformers Work — From Self-Attention to Modern LLM Architecture

zeromathai — Mon, 15 Jun 2026 15:12:47 +0000

Transformers changed AI because they stopped reading sequences one token at a time.

Instead of moving step by step like an RNN, a Transformer compares tokens directly.

That one design shift made modern LLMs possible.

Core Idea

A Transformer is a neural network architecture built around attention.

It looks at a sequence of tokens and learns how those tokens relate to each other.

This matters because language is contextual.

A word is not understood alone.

It is understood through its relationship with surrounding words.

That is why Self-Attention became the core mechanism.

The Key Structure

A simplified Transformer flow looks like this:

Tokens → Embeddings → Positional Information → Self-Attention → Feed-Forward Network → Output

More compactly:

Transformer = token representations + attention + position + stacked blocks

The model first converts text into token vectors.

Then it injects position information.

Then each Transformer block updates the token representations using attention and feed-forward layers.

Implementation View

At a high level, a Transformer processes text like this:

split text into tokens

convert tokens into embeddings

add positional information

for each Transformer block:
    compute Self-Attention

    mix token information

    apply feed-forward transformation

    keep stable flow with residual connections and normalization

produce contextual token representations

For decoder-based LLMs, generation continues like this:

predict next token

append generated token

reuse cached keys and values

repeat until stopping condition

This is why Transformers are practical for large-scale generation.

They can learn relationships across many tokens.

And with caching, they can generate efficiently.

Concrete Example

Take this sentence:

The animal did not cross the street because it was tired.

What does “it” refer to?

A simple left-to-right model may struggle if long context matters.

Self-Attention lets the token “it” compare itself with other tokens like “animal” and “street.”

The model can assign stronger attention to the token that best explains the meaning.

That is the intuition.

Attention lets tokens ask:

Which other tokens matter for understanding me?

RNN vs Transformer

This comparison explains why Transformers became so important.

RNN:

processes tokens step by step
carries information through hidden state
naturally captures order
is harder to parallelize
can struggle with long-range dependencies

Transformer:

processes tokens in parallel
compares tokens directly through attention
needs positional information for order
scales well on GPUs
handles long-range relationships more flexibly

So the Transformer was not just faster.

It changed how sequence relationships are represented.

RNNs remember through recurrence.

Transformers relate through attention.

Self-Attention

Self-Attention computes relationships between tokens in the same sequence.

Each token creates three vectors:

Query
Key
Value

The intuition is simple:

Query = what this token is looking for

Key = what each token offers for matching

Value = information to retrieve if the match is strong

The core formula is:

Attention(Q, K, V) = softmax((QK^T) / sqrt(d_k))V

This means:

compare queries and keys
turn scores into weights
use those weights to combine values

That is how each token becomes context-aware.

Multi-Head Attention

One attention calculation is useful.

But one view is not enough.

Multi-Head Attention runs several attention heads in parallel.

Each head can focus on a different type of relationship.

One head may track syntax.

Another may track semantic similarity.

Another may track long-distance references.

Then the outputs are combined into one representation.

This makes attention richer than a single similarity calculation.

Why Positional Encoding Is Needed

Self-Attention does not automatically know token order.

If you only give it a bag of token embeddings, the model needs another signal to know which token came first.

That is why positional information is added.

Common positional methods include:

Absolute Positional Embedding
Relative Positional Embedding
Rotary Positional Embedding

APE gives each position its own vector.

RPE focuses on relative distance between tokens.

RoPE rotates query and key vectors based on position, making relative position work naturally inside attention.

This is why RoPE became common in modern LLMs.

Encoder, Decoder, and LLMs

The original Transformer used an Encoder-Decoder structure.

Encoder:

reads the input
builds contextual representations
works well for understanding tasks

Decoder:

generates output tokens
uses causal masking
works well for autoregressive generation

Encoder-Decoder:

connects input understanding with output generation
useful for translation-style tasks

Modern GPT-style LLMs are mostly decoder-based.

They generate text one token at a time.

The decoder predicts the next token, appends it, and repeats.

Decoding Strategies

Once the model produces logits, it needs to choose the next token.

Different decoding strategies create different behavior.

Greedy decoding:

chooses the most likely token
simple and deterministic
can be repetitive

Beam search:

keeps multiple candidate sequences
useful for structured generation
can still feel less diverse

Top-k sampling:

samples from the top k likely tokens
adds diversity

Top-p sampling:

samples from the smallest probability mass above a threshold
adapts the candidate set dynamically

So generation quality is not only about the model.

It also depends on decoding.

The Efficiency Problem

Full Attention is powerful but expensive.

If the sequence length is n, attention has roughly O(n^2) cost.

That means longer context becomes expensive quickly.

This is why efficient attention matters.

Local Attention reduces the view to nearby tokens.

Sparse Attention computes only selected attention links.

FlashAttention keeps the formula but improves GPU memory access.

The key idea:

Do less unnecessary work, or move data more efficiently.

Both make longer context more practical.

KV Cache

Autoregressive generation has another problem.

When generating one token at a time, the model repeatedly needs past key and value tensors.

KV Cache stores those tensors.

So the model does not recompute them from scratch at every step.

The flow looks like this:

Generated tokens → cached keys and values → new query attends to cache → next token

This makes inference faster.

But it creates a memory problem.

Longer context means a larger KV Cache.

That is why modern LLMs use techniques like:

Multi-Query Attention
Grouped-Query Attention
Multi-Head Latent Attention

These methods reduce the memory cost of storing key-value information.

Modern Transformer Blocks

Modern LLMs still use the Transformer idea.

But the block has evolved.

A typical modern block looks like this:

Input
→ RMSNorm or Pre-Layer Normalization
→ Self-Attention with GQA and RoPE
→ Residual Connection
→ RMSNorm or Pre-Layer Normalization
→ Feed-Forward Network with SwiGLU or Mixture of Experts
→ Residual Connection

Important upgrades include:

RMSNorm for simpler normalization
RoPE for positional representation
GQA for efficient inference
SwiGLU for stronger feed-forward layers
MoE for sparse expert-based scaling

So today’s Transformer is not exactly the 2017 Transformer copied directly.

It is an evolved architecture family.

Transformer vs Modern LLM Architecture

Original Transformer:

encoder-decoder structure
standard multi-head attention
sinusoidal positional encoding
layer normalization
dense feed-forward layers

Modern LLM architecture:

often decoder-only
causal self-attention
RoPE
RMSNorm
GQA or related KV-sharing methods
SwiGLU
sometimes Mixture of Experts
KV Cache for inference

The core idea stayed the same.

The engineering changed dramatically.

Recommended Learning Order

If Transformer architecture feels too large, learn it in this order:

Attention Mechanism
Self-Attention
QKV Computation
Multi-Head Attention
Positional Encoding
Encoder-Decoder Architecture
Transformer Decoder
KV Cache
Efficient Attention
Modern Transformer Block

This order works because you first understand the relationship mechanism.

Then you understand generation.

Then you understand why modern LLMs needed efficiency upgrades.

Takeaway

The Transformer is the architecture language of modern LLMs.

The shortest version is:

Transformer = attention + position + stacked blocks + efficient generation

Self-Attention computes token relationships.

Positional encoding injects order.

The decoder generates tokens.

KV Cache makes autoregressive inference practical.

Modern upgrades like RoPE, RMSNorm, GQA, SwiGLU, and MoE make the architecture scalable.

If you remember one idea, remember this:

Transformers work by turning a sequence into a set of contextual relationships, then refining those relationships through stacked attention-based blocks.

Discussion

When learning Transformers, do you find it easier to start from the attention formula, the decoder generation loop, or the modern LLM block structure?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/transformer-architecture-overview-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai