DEV Community: zeromathai

Diffusion Models: From Noise Corruption to Reverse Generation

zeromathai — Fri, 18 Sep 2026 09:31:21 +0000

A diffusion model does not try to solve generation from a complex data distribution in one step.

Instead, it defines a Forward Diffusion Process that gradually corrupts real data with Gaussian noise, then learns a Reverse Diffusion Process that moves in the opposite direction, from noise back toward data.

The core idea is straightforward:

data
  |
  | add Gaussian noise step by step
  v
intermediate noisy states
  |
  v
Gaussian noise

Gaussian noise
  |
  | learned reverse transitions
  v
less noisy states
  |
  v
generated data

Define a manageable path from data to noise, then learn how to travel back along that path.

Turn a Hard Generation Problem Into Many Smaller Steps

High-dimensional data such as images follows a complicated distribution. Diffusion Models avoid tackling that generation problem directly by introducing a gradual corruption process.

Start with a real sample $x_{0}$ . As Gaussian noise is added over multiple steps, we obtain

x_{0} \to x_{1} \to x_{2} \to \dots \to x_{T}

As $t$ increases, the structure of the original sample becomes weaker while noise becomes more dominant. In the theoretical limit of an infinitely long process,

T \to \infty, x_{T} \sim N (0, I)

the final state approaches an isotropic Gaussian distribution with mean zero and identity covariance.

This changes the generation problem in an important way. The original data distribution may be highly complex, while sampling from a Gaussian distribution is straightforward. Once the path from data to noise has been defined, generation becomes the problem of learning how to move from that Gaussian endpoint back toward the data distribution.

Forward Diffusion: Gradually Corrupt the Data

The Forward Diffusion Process begins with a data sample $x_{0}$ and progressively adds Gaussian noise.

Each step is a stochastic transition. The transition from $x_{t - 1}$ to $x_{t}$ can be written as

q (x_{t} ∣ x_{t - 1})

where $x_{t - 1}$ is the previous state and $x_{t}$ is the next, noisier state.

Forward diffusion is therefore not one large corruption operation. It is a sequence of probabilistic transitions:

original data
   |
   v
slightly noisy state
   |
   v
more noisy state
   |
   v
...
   |
   v
near-Gaussian noise

As these transitions are repeated, more of the original data structure disappears. With enough steps, the process moves a complicated data sample toward a much simpler noise distribution, which then becomes the starting point for generation.

Reverse Diffusion: Learn the Way Back

Generation follows the opposite direction:

x_{T} \to x_{T - 1} \to \dots \to x_{1} \to x_{0}

But reverse diffusion is not obtained by simply flipping the arrows.

The forward process specifies how noise is added. The reverse distribution required for generation must instead be learned from data.

A single reverse transition is represented as

p_{θ} (x_{t - 1} ∣ x_{t})

Given a noisy state $x_{t}$ , the model learns a probabilistic transition toward the previous, less corrupted state $x_{t - 1}$ .

During generation, this learned transition is applied repeatedly across the chain. Sampling starts from Gaussian noise, moves through progressively less noisy states, and eventually reaches a data-like sample.

The model therefore decomposes generation into many smaller probabilistic reverse steps.

Forward and Reverse Processes at a Glance

Process	Direction	Role
Forward Diffusion	$x_{0} \to x_{T}$	Gradually corrupt data with Gaussian noise
Reverse Diffusion	$x_{T} \to x_{0}$	Generate data by reversing the corruption process
Forward transition	$q (x_{t} ∣ x_{t - 1})$	Defines the next noisy state
Reverse transition	$p_{θ} (x_{t - 1} ∣ x_{t})$	Models movement toward a less noisy state

The important distinction is that the forward corruption process is defined, while the reverse process required for generation is learned. Because generation follows that reverse process step by step, diffusion is inherently built around a sequence of stochastic states.

The Score-Based View

The same corruption-and-reversal idea also appears in Score-based Generative Models.

A central quantity is the Score Function:

s (x) = \nabla_{x} lo g p (x)

Here, $p (x)$ is the probability density and $\nabla_{x}$ is the gradient with respect to $x$ .

The score does not give the density value itself. Instead, it describes the direction in input space in which the log-density increases.

Intuitively, it indicates how a sample should move toward a region of higher data density.

SMLD: Scores Across Multiple Noise Scales

SMLD, or Score Matching with Langevin Dynamics, estimates the Score Function at multiple Noise Scales.

The model learns score information for data corrupted at different levels of noise. During generation, Langevin Dynamics is used while moving from higher Noise Scales toward lower ones.

Conceptually:

high noise
   |
   | Score Function + Langevin Dynamics
   v
lower noise
   |
   | Score Function + Langevin Dynamics
   v
lower noise
   |
   v
data-like sample

At each Noise Scale, the score provides directional information about how the current sample should move relative to the underlying data distribution.

DDPM: Learn Reverse Probabilistic Transitions

A Denoising Diffusion Probabilistic Model (DDPM) approaches the same corruption-and-reversal problem as a sequence of probabilistic steps.

The forward process progressively adds noise to training data. The model then learns a probabilistic process that reverses each corruption step.

The high-level structure is:

Forward:
data -> progressively corrupted states -> noise

Reverse:
noise -> learned probabilistic transitions -> data

The known functional form of the Reverse Distribution makes the learning problem analytically tractable while retaining a flexible generative process.

The central idea is the structured multi-step reversal: DDPM learns how to undo the forward corruption process one probabilistic step at a time.

How SMLD and DDPM Connect

SMLD and DDPM use different training formulations, but they share the same broad structure.

Both progressively corrupt data and learn how to reverse that corruption across multiple Noise Scales. Generation then starts from noise and moves progressively toward the data distribution.

The connection becomes especially clear when considering a continuous state space with continuous $t$ . In that setting, the DDPM training objective can be interpreted as implicitly computing the score at each Noise Scale.

This is why SMLD and DDPM can be understood within the broader framework of Score-based Generative Models, rather than as completely unrelated generation principles.

What to Look for in an Implementation

The most useful implementation-level distinction is simple: the forward corruption process is defined, the neural network learns the reverse-process information, and generation repeatedly applies the learned reverse transitions.

In short:

data to noise is defined; noise to data is learned.

Why Sampling Speed Is the Main Trade-Off

The same sequential structure that defines diffusion generation also creates its main limitation.

Sampling requires following a long reverse Markov Chain:

x_{T} \to x_{T - 1} \to \dots \to x_{1} \to x_{0}

Because those reverse transitions are applied step by step, a long diffusion chain directly affects sampling speed.

The representative comparison with other major Deep Generative Model families is:

Model	Representative limitation
GAN	Training can be unstable, and generated diversity can be limited
VAE	Sample quality limitations and dependence on a surrogate loss
Flow-based Model	Requires specialized architectures for reversible transformations
Diffusion Model	Requires sequential sampling through a long diffusion chain

The source also summarizes the quality-diversity-speed trade-off as follows:

Model	Quality	Diversity	Speed
VAE	×	○	○
GAN	○	×	○
Flow-based Model	×	○	○
Diffusion Model	○	○	×

This is not an absolute ranking of every implementation. It represents the characteristic trade-off emphasized for each model family.

In this comparison, Diffusion Models are strong in quality and diversity, while their sequential reverse sampling process makes speed the main weakness.

Takeaway

A Diffusion Model is best understood as a learned reversal of progressive noise corruption.

Forward Diffusion defines a stochastic path from real data toward Gaussian noise. Reverse Diffusion learns the probabilistic transitions needed to move back toward data, while Score-based methods describe the same broader problem through the direction in which log-density increases.

SMLD and DDPM differ in formulation, but both fit the same high-level picture: progressively corrupt the data, then learn how to reverse that corruption.

The essential implementation mental model is equally compact:

data to noise is defined; noise to data is learned.

That sequential reverse process is both the core of diffusion-based generation and the source of its sampling-speed trade-off.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/diffusion-models-course-en/

Deep Generative Models: Four Ways to Make Complex Distributions Learnable

zeromathai — Thu, 17 Sep 2026 10:30:43 +0000

When developers first encounter deep generative models, it is easy to reduce them to one idea: models that generate new data.

That is only the visible outcome. The deeper problem is deciding how to represent a complex data distribution in a form that a model can compute and learn.

High-dimensional data such as images and audio can follow extremely complex distributions. A single simple probability model is not enough to represent them directly, so different generative model families restructure the problem in different ways.

An Autoregressive Model factorizes a joint distribution into conditional distributions. A Variational Autoencoder (VAE) introduces latent variables. A Flow-based Model starts from a simple distribution and transforms it through invertible mappings. A Generative Adversarial Network (GAN) takes a different route and learns a generation process without directly evaluating an explicit probability density.

A useful mental model is:

Deep generative modeling is not just about producing samples. It is about turning a difficult distribution-modeling problem into something the model can compute and learn.

Two Broad Families

One useful distinction is between Likelihood-based and Likelihood-Free approaches.

Likelihood-based models include:

Autoregressive Models
Variational Autoencoders
Flow-based Models
Diffusion Models
Score-based Models

GAN is a representative Likelihood-Free model.

This distinction matters because it reveals the structural choice behind each model family. A complex distribution can be decomposed into conditional distributions, represented indirectly through latent variables, transformed from a simpler distribution, or learned without directly computing its probability density.

Diffusion Models and Score-based Models also belong to the Likelihood-based side of this classification. Their detailed mechanisms, including Forward Diffusion, Reverse Diffusion, and Score Matching, are outside the scope here.

From KL Divergence to NLL

For Likelihood-based modeling, a natural goal is to make the model distribution close to the data distribution.

Let the true data distribution be $p_{data}$ and the model distribution parameterized by $θ$ be $p_{model}$ .

We can express that goal by minimizing KL Divergence:

θ min D_{KL} [p_{data} (x) ∣ p_{model} (x; θ)]

Expanding the KL Divergence gives:

D_{KL} [p_{data} ∣ p_{model}] E x \sim p data [lo g p_{data} (x)] E x \sim p data [lo g p_{model} (x; θ)]

The first term depends only on the data distribution. If $p_{data}$ is fixed, changing $θ$ does not affect it.

The model-dependent part is therefore:

θ min E x \sim p data [- lo g p_{model} (x; θ)]

This is the Cross-Entropy term between the data distribution and the model distribution.

In practice, we do not know the full underlying data distribution, so the expectation is approximated using training samples. The resulting objective is Negative Log-Likelihood (NLL) minimization.

The optimization flow is:

Minimize KL Divergence
        ↓
Remove the entropy term independent of θ
        ↓
Minimize Cross-Entropy
        ↓
Minimize NLL on training data

The intuition is simple: assign high probability to regions where real data occurs.

This connection is the basic picture when the model provides a way to work explicitly with Likelihood. When $p (x)$ becomes difficult to optimize directly, the model needs a different structure.

That is where the major generative model families begin to diverge.

Strategy 1: Factorize the Distribution

An Autoregressive Model starts with a straightforward idea: if the full joint distribution is difficult to model at once, break it into a sequence of conditional distributions.

The Probability Chain Rule gives an exact factorization:

p (x) = i = 1 \prod n p (x_{i} ∣ x_{1}, \dots, x_{i - 1})

This is not an approximation introduced for convenience. It is an exact decomposition of the joint distribution.

The calculation proceeds through conditional terms such as $p (x_{1})$ , then $p (x_{2} ∣ x_{1})$ , then $p (x_{3} ∣ x_{1}, x_{2})$ , continuing until $p (x_{n} ∣ x_{1}, \dots, x_{n - 1})$ .

That changes the modeling problem. Instead of asking how to represent one complicated $p (x)$ directly, we ask how well the network can model each conditional distribution.

GPT, RNN, PixelRNN, PixelCNN, WaveNet, NADE, and MADE use different data and network structures, but they share this conditional factorization principle.

From an implementation perspective, the important thing to inspect is the ordering and conditioning structure. Each prediction must depend on the variables that precede it in the chosen factorization.

Strategy 2: Introduce a Latent Variable

A VAE restructures the problem differently.

Instead of factorizing the observed variables sequentially, it introduces an unobserved latent variable $z$ and models the observed data $x$ through that latent state.

The data distribution is written as:

p (x) = \int p (z) p (x ∣ z) d z

Here, $z$ is the latent variable, $p (z)$ defines its distribution, and $p (x ∣ z)$ describes how an observation is generated from a particular latent state.

The structural flow is:

Latent state
     ↓
Conditional generation
     ↓
Observed data

In probabilistic terms, the Decoder models $p (x ∣ z)$ .

The VAE also needs a way to infer which latent states are plausible for a given observation. Its Encoder, also called the Recognition Network, does not simply map $x$ to one fixed latent value. It constructs an approximate distribution over $z$ given $x$ .

The Decoder, or Generative Network, models the conditional distribution of $x$ given $z$ .

The difficulty is that the exact Posterior over $z$ given an observation can be hard to compute. VAE addresses this with Variational Inference and trains by maximizing an ELBO.

The important point here is not the full ELBO derivation. It is the structural move: VAE combines latent-variable modeling with approximate inference so that a difficult $p (x)$ becomes a learnable problem.

Strategy 3: Transform a Simple Distribution

Normalizing Flow takes another approach.

Where an Autoregressive Model decomposes a complex $p (x)$ into conditional distributions, a Flow-based Model begins with a simple distribution and changes its shape through a sequence of transformations.

The structure is:

Simple Distribution
        ↓
Invertible Transformation
        ↓
Invertible Transformation
        ↓
        ...
        ↓
Target Distribution

The key requirement is invertibility.

It is not enough to map values from a simple distribution into a more complicated one. The transformation must also allow the model to trace a transformed value back to where it came from.

That property keeps changes in probability density computable through the sequence of transformations.

This requirement also constrains the model architecture. A Flow-based Model cannot freely use arbitrary transformations if they violate invertibility.

The benefit is that the final data variable $x$ has an explicit probability density. Its Likelihood can therefore be computed, and training can directly optimize NLL on the observed data.

A useful implementation-level mental model is:

Preserve invertibility so that probability density remains computable while the distribution becomes progressively more complex.

Saying that a Flow-based Model models the data distribution does not mean the learned distribution becomes identical to the true data distribution. It means the architecture defines an explicit model distribution $p (x)$ whose Likelihood can be computed and optimized.

Strategy 4: Learn Without Direct Density Evaluation

GAN changes the setup more substantially.

Instead of putting the explicit value of $p (x)$ at the center of training, GAN introduces two competing models:

a Generator, which produces samples resembling real data,
a Discriminator, which tries to distinguish real samples from generated ones.

Their opposing objectives form a Minimax Problem.

The structural view is:

Generator
    ↓
Generated samples
    ↓
Discriminator
 ↙           ↘
Real      Generated
data        data

As the Generator produces more convincing samples, the Discriminator has a harder time separating generated data from real data. As the Discriminator improves, the Generator is pushed to produce better samples.

The important difference from an Autoregressive Model or a Flow-based Model is that GAN training does not center on directly computing an explicit $p (x)$ for each observation.

Instead, the interaction between the Generator and Discriminator provides the learning signal that pushes the Generator toward producing samples similar to those from the real data distribution.

That is why GAN is treated as a Likelihood-Free approach in this comparison.

The Structural Difference That Matters

The most useful way to compare these models is not by memorizing architecture names, but by asking how each one turns a complex distribution into a problem the model can compute and learn.

Model	Structural move	Core training view
Autoregressive Model	Factorize $p (x)$ into conditional distributions	Model each conditional probability
VAE	Introduce latent variable $z$	Approximate inference and optimize ELBO
Flow-based Model	Apply invertible transformations	Explicit density and NLL
GAN	Avoid direct explicit density evaluation	Generator–Discriminator Minimax learning

These approaches begin with the same fundamental difficulty: complex high-dimensional data distributions are hard to model directly.

What changes is the structure used to make learning possible.

Takeaway

When reading the implementation of a generative model, do not start with the layer names.

Start with the probability structure. Check whether the model factorizes the distribution, introduces latent variables and approximate inference, preserves density through invertible transformations, or learns through an adversarial objective without directly evaluating an explicit density.

Once that structural choice is clear, the architecture, objective, and training procedure become much easier to understand as parts of the same generative-modeling problem.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/deep-generative-models-course-en/

Normalizing Constants: Turning Neural Network Scores into Probability Distributions

zeromathai — Tue, 15 Sep 2026 08:39:31 +0000

A neural network can assign a real-valued score to an input, but that score is not automatically a probability density. It may be negative, and even after making it positive, the resulting density still needs to integrate to 1 over the entire input space.

That second requirement is where the Normalizing Constant comes in.

For a Deep Generative Model, the core mental model is:

neural network score → exponentiation → normalization → valid probability density

Evaluating the network for one input can be straightforward. The harder part is often computing the quantity needed to normalize its outputs over the entire input space.

From Neural Network Scores to a Probability Density

Suppose a neural network defines a real-valued function $f_{θ} (x)$ .

A generic neural network does not constrain this output to satisfy the requirements of a probability density. In particular, $f_{θ} (x)$ may be negative, so its raw value cannot be interpreted directly as a density.

Exponentiating the output solves the non-negativity problem:

e^{f_{θ} (x)}

This quantity is always positive. But positivity alone is not enough. For a continuous probability density, we also need

\int p_{θ} (x) d x = 1.

To enforce this condition, introduce a Normalizing Constant $Z_{θ}$ :

p_{θ} (x) = \frac{e ^{f_{θ} (x)}}{Z _{θ}} .

The numerator provides a positive, unnormalized value for each input. The denominator rescales those values so that the resulting density integrates to 1.

The construction is therefore:

input $x$ → network score $f_{θ} (x)$ → positive value $e^{f_{θ} (x)}$ → divide by $Z_{θ}$ → probability density $p_{θ} (x)$

The important distinction is that $e^{f_{θ} (x)}$ is positive but still unnormalized. It becomes a valid probability density only after normalization.

What the Normalizing Constant Does

To determine the required scale, integrate the unnormalized values over the entire input space:

Z_{θ} = \int e^{f_{θ} (x)} d x

This gives the total amount represented by the unnormalized model. Using that quantity as the denominator gives

\int p_{θ} (x) d x = \frac{1}{Z _{θ}} \int e^{f_{θ} (x)} d x = 1

So $Z_{θ}$ is not an arbitrary scaling parameter. It is exactly the quantity required to turn the unnormalized function into a valid probability density.

This also reveals an important computational distinction:

Local computation: evaluate $f_{θ} (x)$ for a particular input $x$ .
Global computation: integrate $e^{f_{θ} (x)}$ over the entire input space to obtain $Z_{θ}$ .

Evaluating the network at one input and integrating the function it defines over the entire input space are fundamentally different tasks. That difference is where the normalization problem begins.

When Normalization Is Tractable

Normalizing constants are not always difficult to compute. For a probability distribution with a simple, known structure, the normalization factor may have an explicit form.

Consider a multivariate Gaussian with identity covariance:

p_{μ} (x) = \frac{1}{( 2 π ) ^{d /2}} e^{- \frac{∥ x - μ ∥ ^{2}}{2}}

Its normalization factor is determined explicitly by the dimension $d$ :

\frac{1}{( 2 π ) ^{d /2}}

This is what makes the Gaussian case useful as a contrast. Its known distributional structure allows the normalization factor to be computed directly.

A more flexible neural-network probability model can represent a more complicated function, but the corresponding normalization integral may no longer be easy to compute.

Model structure	Expressivity	Normalization
Structured distribution such as the Gaussian above	Constrained by a known distributional form	Explicitly computable
Complex neural-network probability model	Can represent more flexible functions	May require an intractable integral

Normalization itself is therefore not the problem. The difficulty appears when the function being normalized becomes sufficiently complex.

Why $Z_{θ}$ Becomes Difficult

For one input, the network computes

f_{θ} (x) .

To obtain the Normalizing Constant, however, we need

Z_{θ} = \int e^{f_{θ} (x)} d x .

The first operation evaluates the model at a particular $x$ . The second depends on the behavior of the model across the entire input space.

When $f_{θ} (x)$ is a complicated neural-network function, integrating it over all possible inputs may become intractable. If the input space is also very high-dimensional, as with images, the problem becomes even harder.

The key point is therefore more precise than simply saying that high-dimensional data is difficult. The central challenge is having to integrate an arbitrarily complex neural-network function over the full input space.

The Tractability-Flexibility Tradeoff

The Normalizing Constant exposes a central tension in probabilistic modeling.

A simpler model can make probability calculations easier, but its ability to represent complex data distributions may be limited. A more expressive model can represent more flexible distributions, but the required probability calculations may become harder.

The tradeoff can be summarized as:

more structural restriction → easier probability computation → less flexibility

versus

more model flexibility → more difficult probability computation → potentially intractable normalization

This is the Tractability-Flexibility Tradeoff.

Deep Generative Modeling therefore has to consider both sides of the problem: how expressive the model is and whether the probability computations implied by that model can actually be carried out.

The Normalizing Constant sits directly at that intersection.

How Generative Models Handle the Normalization Problem

When $Z_{θ}$ is difficult to compute directly, different Generative Modeling approaches make different choices.

Approximate the Normalizing Constant: Energy-Based Models approach the difficult Normalizing Constant through approximation.
Restrict the model structure: Autoregressive Models, Flow-based Models, and Variational Autoencoders impose structural constraints to make the required probability-modeling computations manageable.
Model the generation process instead: GANs focus on the generation process rather than directly computing the Normalizing Constant of an explicit probability density.
Bypass direct computation of the Normalizing Constant: Score-based Diffusion Generative Models take an approach that does not require computing the Normalizing Constant directly.

A Restricted Neural Network Model does not simply mean a weak or low-capacity network. The restriction is structural: the model is designed so that the required probabilistic computations remain manageable.

From this perspective, the differences among Deep Generative Models are not only architectural. They also reflect different ways of handling the computational problem created by the Normalizing Constant.

What to Look for in a Probabilistic Model

When examining a probabilistic generative model, trace how it turns a model output into a valid probability distribution.

If the model begins with an unnormalized score, ask what makes that score non-negative and what makes the resulting density integrate to 1. In the formulation used here, that means identifying

Z_{θ} = \int e^{f_{θ} (x)} d x .

Then ask whether $Z_{θ}$ can actually be computed.

If it cannot, the model must take another approach: approximate the Normalizing Constant, constrain the model structure so the required computations become manageable, focus on the generation process instead of an explicit normalized density, or bypass direct computation of the Normalizing Constant.

That choice is a useful way to understand why different Generative Models adopt different structures.

Takeaway

A neural network score does not automatically define a probability density. Exponentiation makes the score positive,

e^{f_{θ} (x)},

but normalization requires

Z_{θ} = \int e^{f_{θ} (x)} d x,

which gives

p_{θ} (x) = \frac{e ^{f_{θ} (x)}}{Z _{θ}} .

For simple structured distributions, the normalization factor can be computed explicitly. For flexible neural-network probability models, computing it over the entire input space may become intractable.

That is the central idea: greater model flexibility can make probability computation harder, and different Generative Models make different structural choices to handle that tradeoff.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/normalizing-constant-course-en/

Modeling Data Distributions: The Core Idea Behind Generative Models

zeromathai — Sat, 12 Sep 2026 04:16:32 +0000

A generative model is not trying to memorize a dataset. Its real target is the probability structure that could have produced the observed data.

That distinction is the foundation of distribution modeling: we can observe finite samples, but we do not know the analytical form of the Data Distribution behind them. Generative modeling therefore introduces a parameterized Model Distribution and learns its parameters from those samples.

The challenge becomes much harder with high-dimensional data. A useful probability model needs enough flexibility to represent complex structure while remaining tractable enough to evaluate and train. This Tractability-Flexibility Tradeoff is what eventually motivates more expressive Deep Generative Models.

Start With Samples, Not the Distribution

Suppose a training dataset contains $N$ observations drawn independently from an unknown Data Distribution:

x_{1}, x_{2}, \dots, x_{N} \sim i.i.d. p_{data} (x)

This equation does not mean that we know the analytical form of $p_{data} (x)$ . We only observe the samples $x_{1}, \dots, x_{N}$ and assume that some underlying probability distribution generated them.

For an image dataset, the image files are directly observable. What we do not observe is the probability structure that determines which kinds of images are common, which are rare, and which structures repeatedly appear across different images.

The basic relationship is:

unknown Data Distribution
          |
          v
   observed samples

Training data therefore provides evidence about the underlying distribution rather than direct access to it. Collecting many samples helps reveal recurring statistical patterns and shared structure that can be used to estimate what lies behind the observations.

Because the exact Data Distribution is unknown, however, we cannot directly use it to evaluate probabilities or draw new samples. We need a separate probability model that can be learned from the observed data.

Data Distribution vs. Model Distribution

Generative modeling introduces a trainable Model Distribution:

p_{model} (x; θ) \approx p_{data} (x)

The two distributions have different roles:

Data Distribution $p_{data} (x)$ is the unknown distribution assumed to have generated the training data.
Model Distribution $p_{model} (x; θ)$ is a parameterized probability distribution whose parameters are learned from the observed samples.

The workflow is not to discover the exact Data Distribution first and then copy it. We never directly observe $p_{data} (x)$ . Instead, we choose a probability model capable of representing the data structure and adjust its parameters through training.

Conceptually:

observed i.i.d. samples
          |
          v
parameterized Model Distribution
          |
       training
          |
          v
approximation of the unknown
Data Distribution

The relationship

p_{model} (x; θ) \approx p_{data} (x)

is not a specific training objective. It expresses the goal: the learned Model Distribution should approximate the unknown distribution behind the observations.

Once learned, the same Model Distribution supports two important directions of computation:

data ---------> probability
     evaluation

distribution -> data
      sampling

Probability Evaluation asks how plausible a given data point is under the learned distribution. Sampling moves in the opposite direction by drawing a new data point from that distribution. Both operations depend on the same learned probability structure.

One Distribution, Three Useful Views

Learning a Data Distribution means more than generating realistic-looking samples. The same modeled distribution can be viewed through Generation, Density Estimation, and Unsupervised Representation Learning.

1. Generation

Generation draws a new sample from the learned Model Distribution:

x_{new} \sim p_{model} (x; θ)

If the model has learned a distribution of dog images, a new sample should exhibit the kinds of structures found in that data. The goal is not to reproduce a stored training image, but to generate a new data point consistent with the learned distribution.

2. Density Estimation

Density Estimation looks at the same distribution from the probability-evaluation side. Given a new input, the question is how plausible that input is under the learned distribution.

An input that matches structures commonly found in the training data can receive higher probability, while one that differs substantially from the learned data structure can receive lower probability. This perspective also connects naturally to Anomaly Detection, where inputs that deviate strongly from the learned distribution are of interest.

3. Unsupervised Representation Learning

Distribution modeling also requires capturing structures that recur across observations. Two dog images may have very different pixel values while still sharing repeated patterns such as ears, faces, tails, and bodies.

Learning these recurring patterns can be viewed as Feature Learning. In high-dimensional spaces, learning how data is structurally organized also connects distribution modeling to Manifold Learning.

Generation, Density Estimation, and Representation Learning are therefore not unrelated capabilities. They are different ways of looking at the same learned distribution: where the data tends to exist, what structure it shares, and what new samples are consistent with that structure.

Why High-Dimensional Data Changes the Problem

The basic idea becomes much harder when the number of variables grows.

Images, for example, occupy a very high-dimensional input space. More importantly, producing individually plausible pixel values is not enough to produce a coherent image. Pixels forming an eye must relate to nearby pixels, the eyes must relate to the face, and the face must relate to the rest of the object.

A useful Model Distribution must therefore represent not only individual variable values but also complicated dependencies among many variables.

That creates a fundamental tension:

simple model
    |
    +-- easier probability computation and training
    |
    `-- limited ability to represent complex structure

more flexible model
    |
    +-- richer representation of data structure
    |
    `-- probability computation and training
        may become harder

This is the Tractability-Flexibility Tradeoff.

Tractability vs. Flexibility

Tractability describes whether probability calculations and model training remain practically manageable. Flexibility describes how richly the model can represent complex forms of the Data Distribution.

High-dimensional data often requires substantial flexibility because the underlying distribution may contain complicated dependencies. At the same time, a probability model must remain tractable enough to use and train.

A simple isotropic Gaussian illustrates the tradeoff:

p_{μ} (x) = \frac{1}{( 2 π ) ^{d /2}} exp (- \frac{∥ x - μ ∥ ^{2}}{2})

Here, $x$ is the data vector, $μ$ is the mean vector at the center of the distribution, and $d$ is the dimensionality of the data space.

The important point is not the Gaussian itself, but what its simplicity gives us. Its probability structure is easy to work with, so it offers high tractability. That same simplicity, however, limits the range of complex high-dimensional distributions it can represent.

A simple distribution may therefore be computationally convenient while lacking enough flexibility to capture the structures found in real data.

Model property	Benefit	Limitation
Simple distribution	High tractability	Limited flexibility
More flexible distribution	Richer data structure	Harder probability computation and training

Generative modeling has to operate between these two requirements. If the model is too simple, it cannot capture the structure of the Data Distribution. If flexibility is increased without preserving tractability, probability computation and training can become difficult.

The goal is not simply to increase the number of parameters. The model must be expressive enough to represent complex high-dimensional structure while remaining usable as a probability model.

Why This Leads to Deep Generative Models

When high-dimensional Data Distributions are too complex for simple probability models, greater Model Expressivity becomes necessary.

Deep Neural Networks can represent complex relationships through multiple layers. Using that expressive capacity inside a Model Distribution provides a way to model structures that simpler probability distributions cannot represent adequately.

The target remains:

p_{θ} (x) \approx p_{data} (x)

This relationship is not a particular optimization objective. It states the goal of using a more expressive model to approximate the unknown Data Distribution.

The conceptual progression is:

observed high-dimensional data
              |
              v
complex unknown Data Distribution
              |
      requires flexibility
              |
              v
    more expressive model

The motivation for Deep Generative Models follows directly from this problem. We observe only finite i.i.d. samples, yet the unknown distribution behind high-dimensional data can contain complicated structure and dependencies.

Simple probability models may preserve tractability but lack sufficient flexibility. Deeper models provide the additional expressivity needed to approximate those more complex distributions.

Takeaway

Generative modeling begins with a simple asymmetry: we can observe samples, but we cannot directly observe the probability distribution that generated them.

We therefore construct a parameterized Model Distribution and train it to approximate the unknown Data Distribution:

p_{model} (x; θ) \approx p_{data} (x)

Once that distribution is learned, Generation, Density Estimation, and Representation Learning become different perspectives on the same modeled structure.

The difficult part is high-dimensional data. Complex dependencies require flexibility, while probability evaluation and training require tractability. The Tractability-Flexibility Tradeoff is the key connection between simple probability distributions and the motivation for Deep Generative Models.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/data-distribution-modeling-course-en/

Approximate Inference: Sampling the Posterior vs Optimizing an Approximation

zeromathai — Thu, 10 Sep 2026 04:54:39 +0000

In a simple probabilistic model, posterior inference sounds straightforward: observe some data, compute the posterior distribution over hidden variables, and use it to answer the question you care about.

In complex models, that computation is often the hard part. The latent space may be high-dimensional, the posterior may have a complicated shape, and expectations under that posterior may be difficult to compute analytically.

The practical question becomes:

If the exact posterior is difficult to compute, how can we turn inference into something tractable?

Approximate Inference gives us two major answers:

MCMC: approach the posterior through sampling.
Variational Inference: approximate the posterior with a distribution found through optimization.

Both address the same posterior-inference problem, but they make it computationally manageable in different ways.

The core mental model

Suppose $x$ is observed data and $z$ represents hidden variables. Posterior inference is concerned with $p (z ∣ x)$ and, when needed, expectations under that distribution.

The problem is not simply defining the posterior. It is computing with it.

When multiple hidden variables interact, their joint states determine the shape of the posterior. As those interactions become more complex, directly representing the exact posterior or calculating expectations under it becomes increasingly difficult.

Approximate inference changes the computational problem instead of insisting on an exact closed-form calculation.

Exact posterior is difficult to compute
                |
        Approximate Inference
          /             \
         /               \
      MCMC                 VI
       |                   |
   Sampling           Optimization
       |                   |
Markov Chain       Variational Family
       |                   |
Posterior-like      Best approximate
samples             distribution

The key distinction is simple: MCMC approaches the posterior through samples, while VI searches for a tractable distribution that can stand in for it.

MCMC: represent the posterior with samples

Markov Chain Monte Carlo starts from the fact that directly obtaining the samples we want from a complicated posterior may be difficult.

Instead of calculating the posterior itself in closed form, MCMC constructs a stochastic process that approaches that posterior over time.

The goal is to build an ergodic Markov Chain over the latent-variable space whose Stationary Distribution is the target posterior.

The computational flow is:

Construct an ergodic Markov Chain
whose stationary distribution
is the target posterior
                |
                v
Run the chain
                |
                v
Collect states as samples
                |
                v
Estimate posterior quantities empirically

Ergodicity matters because the chain should not remain trapped indefinitely in a single starting state. As it progresses, the chain must be able to explore the state space needed to approach its target Stationary Distribution.

Once the chain has approached that distribution, collected states can be used to approximate properties of the posterior. Metropolis-Hastings and Gibbs Sampling are representative examples of this MCMC approach.

The useful implementation-level mental model is that the samples become the practical representation of the posterior. You do not need a convenient closed-form expression for the entire distribution if you can obtain samples that reflect it well enough to estimate the quantities you need.

That flexibility comes with computational cost. MCMC may require many repeated transitions, and it can be difficult to determine whether the collected samples are sufficiently independent. For complex models or large problems, repeated sampling can therefore become expensive.

This also shows where approximation enters MCMC. The method does not approximate the target posterior by choosing a simpler distribution family. With sufficient computation, it can approach the exact result.

In practice, however, processor time is finite, so only a limited number of samples can be collected.

For MCMC, finite computation is the practical source of approximation.

Why high-dimensional sampling is difficult

The role of the Markov Chain becomes easier to understand when we look at sampling in high-dimensional spaces.

Not every part of a large state space matters equally. The full space may be enormous, while useful samples can be concentrated in regions to which the target distribution assigns high probability.

Randomly choosing points from the entire space is therefore an inefficient way to find the regions that matter.

MCMC approaches this problem sequentially. Starting from some state, the chain repeatedly moves to another state. It first needs to reach a High-Probability Region and can then continue exploring regions with substantial probability under the target distribution.

The mental model is not:

Search the entire high-dimensional space at once

It is closer to:

current state
     |
     v
next state
     |
     v
next state
     |
     v
high-probability region
     |
     v
continue exploring and sampling

Random-walk behavior provides useful intuition here. The point is not that MCMC learns a separate manifold.

The important idea is that finding probability-concentrated regions is itself difficult in high dimensions, and a Markov Chain provides a way to move toward and explore those regions.

That is why chain construction and sample collection are central to the method.

Variational Inference: turn inference into optimization

Variational Inference takes the same posterior-inference problem and makes it tractable in a different way.

Instead of repeatedly collecting samples that reflect the exact posterior, VI approximates the difficult posterior with a probability distribution that is easier to work with.

This turns inference into an optimization problem.

The first step is to define a Variational Family $Q$ , a set of candidate distributions that are allowed to approximate the exact posterior.

Each $q (z) \in Q$ is a candidate approximation to $p (z ∣ x)$ . To keep the approximation tractable, the family may use a Factorized Approximation or a Parametric Form such as a Gaussian.

The next step is to select the candidate that is closest to the exact posterior.

VI expresses this as:

q^{*} (z) = q (z) \in Q argmin D_{KL} (q (z) ∣ p (z ∣ x))

Here, $Q$ is the Variational Family, $q (z)$ is a candidate approximate distribution, $p (z ∣ x)$ is the Exact Posterior, and $q^{*} (z)$ is the optimal approximation within $Q$ according to KL Divergence.

The KL Divergence being minimized is:

D_{KL} (q (z) ∣ p (z ∣ x)) = E_{z \sim q (z)} [lo g \frac{q ( z )}{p ( z ∣ x )}]

The two equations play different roles.

The first defines the optimization problem: choose the candidate distribution in $Q$ that minimizes the divergence from the exact posterior.

The second defines the quantity being minimized: the KL Divergence expressed as an expectation under $q (z)$ .

The overall calculation flow is:

Start with the difficult Exact Posterior $p (z ∣ x)$ .
Choose a Variational Family $Q$ .
Consider candidate distributions $q (z) \in Q$ .
Measure their KL Divergence from $p (z ∣ x)$ .
Optimize within $Q$ .
Obtain the optimal approximation $q^{*} (z)$ .
Use $q^{*} (z)$ in place of the Exact Posterior for subsequent inference calculations.

This is the core idea behind Inference as Optimization.

MCMC explores the posterior through a sequence of samples. VI searches through a restricted space of probability distributions.

Where VI's approximation comes from

Restricting the search space is what makes VI easier to compute, but that restriction also creates its main limitation.

Suppose the true posterior cannot be represented well by the chosen Variational Family $Q$ . Optimization can still find the best possible $q^{*} (z)$ within that family, but even a perfectly solved optimization problem cannot produce a distribution outside the family that was allowed from the start.

This is where VI differs fundamentally from MCMC.

For VI, the approximation is not simply:

We stopped computing too early.

It is:

We deliberately restricted
what the approximate posterior
is allowed to look like.

Factorized or parametric assumptions can make inference tractable, but they also limit which posterior shapes can be represented.

For VI, the restricted analytical approximation is itself the source of approximation.

MCMC vs VI: the distinction that matters

Both methods address the same problem: posterior inference when direct exact computation is difficult.

Their computational structures, however, are different.

	MCMC	Variational Inference
Core approach	Sampling	Optimization
Posterior representation	Collected samples	Approximate distribution $q (z)$
Main structure	Markov Chain	Variational Family $Q$
Target	Stationary Distribution is the posterior	Minimize KL Divergence to the posterior
Practical source of approximation	Finite computation	Restricted approximation family
Computational characteristic	Repeated sampling can be costly	Some VI methods can scale relatively well to large applications

When reading a probabilistic model or inference implementation, the most useful question is:

Where has the original posterior-inference problem been moved?

With MCMC, it has been moved into the behavior of a Markov Chain. You construct the chain so that its Stationary Distribution is the target posterior, run it, collect samples, and estimate posterior quantities empirically.

With VI, it has been moved into an optimization problem. You define $Q$ , search within that family, and use $q^{*} (z)$ as the tractable replacement for $p (z ∣ x)$ .

That structural difference is the most important mental model to carry into equations, models, and implementations.

Takeaway

Approximate Inference is a way to make difficult Posterior Inference computationally manageable.

MCMC says:

Do not compute the entire posterior directly. Construct a Markov Chain whose Stationary Distribution is the posterior, then use samples from that chain.

Variational Inference says:

Do not compute the difficult posterior directly. Search for a tractable distribution that approximates it.

The crucial difference is where approximation enters the computation. In MCMC, practical approximation comes from finite sampling and finite computation. In VI, approximation comes from restricting the posterior to a chosen Variational Family.

Both are computational answers to the same question: what should we do when the posterior we want is too difficult to compute exactly?

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/approximate-inference-course-en/

Posterior Inference: From Joint Distributions to the Inference Bottleneck

zeromathai — Tue, 08 Sep 2026 06:40:17 +0000

A probabilistic model can describe more than the data you observe. It can also include hidden variables that capture structure you cannot observe directly.

But defining that model is only the beginning. Once an observation $x$ is available, the practical question changes:

Given this $x$ , what does the model imply about the hidden variable $z$ ?

That is the central problem of Posterior Inference.

The notation is compact, but the computation is not always easy. High-dimensional latent spaces, complex posterior distributions, and interactions among hidden variables can make both the posterior itself and expectations under that posterior difficult to compute.

Start with the Joint Distribution

Suppose a probabilistic model contains an observed variable $x$ and a hidden or latent variable $z$ .

The model does not treat them as unrelated quantities. Instead, it represents their probabilistic relationship through a Joint Distribution:

p (z, x)

This joint distribution describes how the observed data and the hidden variable fit together inside a single probability structure.

Once $x$ is observed, however, the question becomes conditional. We are no longer asking only how $x$ and $z$ relate in general. We want to know how the possible values of $z$ are distributed given the particular observation $x$ .

That conditional distribution is the posterior.

Posterior Distribution: Conditioning on Observed Data

The Posterior Distribution is

p (z ∣ x) = \frac{p ( z , x )}{p ( x )}

The numerator $p (z, x)$ contains the probabilistic relationship between the latent variable and the observation. The denominator $p (x)$ normalizes those values so that the result becomes a conditional probability distribution over $z$ .

The distinction is important:

The joint distribution $p (z, x)$ describes the probability structure of the model.
The posterior distribution $p (z ∣ x)$ tells us what that structure implies about $z$ after $x$ has been observed.

In that sense, the posterior connects the model with actual data.

Posterior Inference Has Two Closely Related Tasks

Posterior Inference is not limited to obtaining the posterior distribution itself.

One task is to compute

p (z ∣ x)

which answers the question: how should probability be distributed across possible latent states after observing $x$ ?

The other task is to compute an expectation under that posterior. In this case, the posterior is used to average a quantity required for later analysis or learning.

This distinction matters because some learning procedures use the posterior inside another computation. The EM Algorithm is a representative example.

EM Shows Why Posterior Expectations Matter

Consider a model with a latent variable $z$ .

Because $z$ is unobserved, the Complete-Data Log-Likelihood cannot be used directly as though both $x$ and $z$ were known. The Expectation-Maximization (EM) Algorithm instead uses the posterior distribution under the current model parameters to compute its expectation over the possible latent states:

Q (θ, θ^{o l d}) ≜ z \sum p (z ∣ x; θ^{o l d}) lo g p (x, z; θ)

Here, $θ^{o l d}$ denotes the current model parameters, while $θ$ denotes the parameters being evaluated.

The calculation can be understood in three steps:

Use $θ^{o l d}$ to obtain the posterior $p (z ∣ x; θ^{o l d})$ .
Use those posterior probabilities as weights for $lo g p (x, z; θ)$ .
Sum over $z$ to obtain $Q (θ, θ^{o l d})$ .

For each possible latent state, the Complete-Data Log-Likelihood is weighted by that state's posterior probability.

So $Q (θ, θ^{o l d})$ is not merely an arbitrary weighted sum. It is the Complete-Data Log-Likelihood averaged under the posterior distribution of the latent variable.

This gives a useful implementation-level mental model: Posterior Inference determines how latent states are weighted, and learning uses those weights to construct the quantity it needs.

Posterior Inference is therefore connected directly to model learning, not only to analyzing a probabilistic model.

Where Exact Inference Becomes Difficult

The conceptual process seems simple: define $p (z, x)$ , observe $x$ , obtain $p (z ∣ x)$ , and then compute any required posterior expectation.

The difficulty is that the last two computations are not always tractable.

High-Dimensional Latent Spaces

As the dimensionality of the latent space increases, the space of possible latent states becomes harder to handle directly.

The challenge is not merely that $z$ is hidden. A high-dimensional latent variable can create a large state space that must be considered during inference.

Complex Posterior Distributions

The posterior itself may also have a complicated form.

When that happens, required expectations under $p (z ∣ x)$ may not be available through an analytical calculation.

This creates two related computational questions:

Can $p (z ∣ x)$ be computed exactly?
Can the required expectation under $p (z ∣ x)$ be computed exactly?

Difficulty in either calculation can make inference a bottleneck.

Why Deep Models Add More Difficulty

Deep Learning can add another source of difficulty through interactions among hidden variables.

When hidden variables appear across multiple layers, the state of one variable can be affected by relationships involving other variables and layers. The resulting posterior structure must reflect those interactions.

The inference problem therefore becomes difficult for several reasons at once:

the latent space may be high-dimensional,
the posterior distribution may have a complex form,
posterior expectations may not have an analytical solution,
interactions among hidden variables across layers can further complicate the posterior.

This is why the Challenge of Inference cannot be explained simply by saying that latent variables are unobserved.

The real difficulty comes from the structure of the inference problem. As the latent structure becomes more complicated, exact computation of the posterior and its expectations can become difficult enough to limit model learning and use.

Modeling and Inference Are Different Problems

A useful mental model is to separate the probability model from the inference problem.

The modeling question is:

What joint probability structure describes the observed and hidden variables?

That structure is represented by

p (z, x)

The inference question is:

Given an actual observation $x$ , what does that model imply about the hidden variable $z$ ?

That target is

p (z ∣ x)

A later learning or analysis step may then require a quantity computed under that posterior, such as

Q (θ, θ^{o l d}) ≜ z \sum p (z ∣ x; θ^{o l d}) lo g p (x, z; θ)

Defining a probabilistic model does not automatically mean that every quantity implied by that model can be computed easily.

The probability structure may be well defined while exact inference remains computationally difficult.

Why Approximate Posterior Inference Becomes Necessary

If exact Posterior Inference were always tractable, no additional solution would be needed.

But complex probabilistic models cannot always rely on exact computation of the posterior distribution or the required posterior expectations.

That is where Approximate Posterior Inference becomes necessary.

The important point here is not a particular approximation method. It is the reason approximation is needed in the first place.

As latent structure becomes more complicated, exact posterior computation becomes harder. A complex posterior can also make the expectations required for learning difficult to evaluate. When those computations become impractical, inference itself becomes a computational limitation.

Approximate Posterior Inference is the next step for dealing with that limitation.

What to Look for in a Probabilistic Model

When reading a latent-variable model, separate three pieces.

First, identify the joint model:

p (z, x)

Then identify the posterior inference target:

p (z ∣ x)

Finally, determine whether the algorithm needs the posterior itself or a quantity computed under it, such as the posterior expectation used by EM:

Q (θ, θ^{o l d}) ≜ z \sum p (z ∣ x; θ^{o l d}) lo g p (x, z; θ)

That makes the practical question much clearer:

Can the required posterior computation be performed exactly, or has inference itself become the computational problem?

Takeaway

Posterior Inference connects a probabilistic model with observed data and asks what the model implies about its hidden variables after that data is available.

The conceptual progression is straightforward: start from the joint distribution $p (z, x)$ , condition on the observed $x$ to obtain $p (z ∣ x)$ , and then use that posterior directly or compute expectations under it when learning requires them.

The difficulty is computational. High-dimensional latent spaces, complex posterior distributions, and interactions among hidden variables can make exact Posterior Inference difficult.

That computational limitation is what motivates Approximate Posterior Inference.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/posterior-inference-course-en/

Inverse Problems: Why Predicting Backward Is Harder Than It Looks

zeromathai — Sun, 06 Sep 2026 05:26:16 +0000

Most machine learning examples follow a familiar direction: start with an input and predict an output.

Inverse problems ask the opposite question. You observe the result first, then try to infer the input, parameter, or underlying state that could have produced it.

The important catch is that this is often not a simple matter of running a function backward. One observation can be consistent with multiple possible causes, so the problem may require reasoning about a distribution of plausible solutions rather than recovering one unique answer.

That is the core connection between inverse problems and generative modeling.

Forward Problems vs. Inverse Problems

A Forward Problem starts with an input and predicts the corresponding output.

Supervised classification is a typical example:

input x
   |
   v
model
   |
   v
output y

Probabilistically, the quantity of interest is:

p (y ∣ x)

Given $x$ , we want to predict or characterize $y$ .

An Inverse Problem starts from the observed result and asks what input or state could have produced it:

observed result y
        |
        v
infer possible x

The corresponding conditional distribution is:

p (x ∣ y)

Given $y$ , we want to reason about possible values of $x$ .

The difference is deeper than swapping the order of the variables. In a forward problem, the relationship between an input and its result can be used to compute an output. In an inverse problem, one observed result may correspond to several possible inputs.

That makes the inverse direction fundamentally ambiguous in many cases.

Forward computation asks what an input produces. Inverse inference asks which inputs could explain what we observed.

Why the Inverse Direction Is Different

Consider a process that transforms detailed data into a more limited observation:

possible original data
        |
        v
  forward process
        |
        v
 limited observation

The inverse task goes in the opposite direction:

limited observation
        |
        v
    inference
        |
        v
possible original data

The challenge is that the observation may no longer contain enough information to identify one original state.

A low-resolution image, for example, may not preserve every detail from its high-resolution source. Several high-resolution images can therefore be consistent with the same low-resolution observation.

The inverse problem is better described by a conditional distribution:

P (high-resolution ∣ low-resolution)

The goal is not simply to undo an operation. It is to reason about the possible high-resolution images that are consistent with the observed low-resolution image.

Common Examples of Inverse Problems

The same structure appears in several image reconstruction and transformation tasks.

Low Resolution to High Resolution

Given a low-resolution image, infer possible high-resolution images:

P (high-resolution ∣ low-resolution)

Masked Observation to Full Image

Given partial image information, infer possible complete images:

P (full image ∣ mask)

Grayscale to Color

Given a grayscale image, infer possible color images:

P (color image ∣ gray image)

Stroke Painting to Image

Given a stroke painting, infer possible images:

P (image ∣ stroke painting)

Sparse-View CT Observation to Cross-Sectional Image

Given a sparse-view sinogram, infer possible CT cross-sectional images:

P (cross-sectional image ∣ sparse-view sinogram)

These examples differ in their data and observation process, but they share the same basic structure:

limited or transformed observation
              |
              v
     conditional inference
              |
              v
       more complete data

This is also why inverse problems are closely related to Conditional Generation. The observation acts as a condition, and the model must represent data that is consistent with that condition.

The important idea here is not a particular generation method. It is the reverse reasoning structure: infer possible data or inputs from an observed result.

Why Not Just Learn an Inverse Function?

Because an inverse may not be unique.

Suppose several inputs produce the same observation:

x1 ----\
x2 -----+----> y
x3 ----/

If only $y$ is available, there may be no way to determine which original $x$ produced it.

This is one reason inverse problems can be ill-posed. When one observation corresponds to several possible inputs, the observation alone may not determine a unique solution.

Additional information or regularization may therefore be needed.

A useful mental model is:

Naive view:
y -> inverse function -> x

Better view:
y -> distribution over possible x

The second view keeps the ambiguity visible instead of forcing a single answer where the observation may not justify one.

High-Dimensional Data Makes the Problem Harder

Non-uniqueness is not the only challenge.

Images and similar data are high-dimensional. They can be represented by many variables, but real data is not randomly spread across every possible point in that high-dimensional space.

A useful way to think about this is through manifold structure. Real data exists within a complex structure rather than being distributed arbitrarily across the entire space.

That means inverse inference cannot simply treat every possible high-dimensional configuration as equally meaningful. A model must be able to represent the complex structure in which real data exists and reason about possible data associated with the observation.

Latent Variables Add Another Layer

Observed data may also fail to reveal the underlying structure through which it was generated.

In those cases, Latent Variable Inference becomes important.

Conceptually:

latent structure
      |
      v
generated data
      |
      v
observation

Looking at the process in reverse means reasoning from the observation toward structure that may not be directly visible in the raw data.

This is another reason inverse problems are not simply algebraic inversion. The problem may also require inference about latent structure behind the observation.

Three Core Challenges in Generative Modeling

Inverse problems lead naturally into three broader questions that arise when modeling complex data with generative models:

Representation
     |
     v
Learning
     |
     v
Inference

Representation: How Do We Model the Joint Distribution?

High-dimensional data can contain many interacting random variables.

The representation problem asks how to model their Joint Distribution.

When data contains a very large number of variables, directly handling all of their interactions becomes difficult. A generative model therefore needs a way to represent a complex probability distribution efficiently.

For a developer, the practical question is straightforward:

What kind of probability structure can this model represent?

That representation determines how the model describes the structure of complex data.

Learning: How Do We Model the Data Distribution from Samples?

The underlying data distribution can be written as:

p_{data} (x)

Training data can be viewed as samples drawn from this distribution.

The model defines its own parameterized distribution:

p_{θ} (x)

Learning means using observed samples to build a model distribution that describes the probability structure of the real data.

Conceptually:

samples from real data
        |
        v
     learning
        |
        v
model distribution p_theta(x)

This leads to a central learning question:

How should the real data distribution and the model distribution be compared?

The underlying distribution $p_{data} (x)$ is not directly available as a complete probability distribution. The model instead learns from samples and uses $p_{θ} (x)$ to describe the probability structure behind those samples.

Inference: What Structure Explains the Observation?

The third challenge is inference.

If generation moves from underlying structure toward observable data, inference looks in the reverse direction: from raw observations toward the higher-level structure or features that explain them.

This perspective is closely related to Inverse Graphics:

high-level structure
        |
        v
   generated data

Generation: top -> bottom
Inference:  bottom -> top

A generative model is therefore not only about representing or generating data. It also connects to understanding what structure could explain an observation.

What to Look for in an Inverse-Problem System

When reading a model or system that solves an inverse problem, focus on a few questions:

What is observed? Identify the condition $y$ .
What must be inferred? Identify the unknown $x$ or latent structure.
Is the solution unique? If not, think in terms of multiple plausible solutions rather than a single deterministic inverse.
What constrains the solution? Look for additional information, regularization, or learned data structure.
How is high-dimensional data represented? The model must account for the complex structure of real data.
What distribution is being learned? Distinguish $p_{data} (x)$ from the model distribution $p_{θ} (x)$ .
What does inference recover? Determine whether the system is reasoning about possible data, latent structure, or high-level features.

This framing helps clarify the role of representation, learning, and inference in an inverse-problem setting.

Takeaway

A Forward Problem predicts an output from a known input:

p (y ∣ x)

An Inverse Problem starts from an observation and reasons about possible inputs:

p (x ∣ y)

The hard part is that the inverse direction may not have one unique answer.

That is why inverse problems naturally lead to probabilistic modeling. The model may need to represent multiple possible solutions, account for complex high-dimensional data structure, and infer latent structure that is not directly visible in the observation.

In generative modeling, this connects to three core tasks: representing complex probability distributions, learning them from observed samples, and using the learned structure for inference.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/inverse-problems-course-en/

Generative Modeling: Learning the Distribution Behind the Data

zeromathai — Thu, 03 Sep 2026 22:59:49 +0000

Generative Modeling starts with a simple shift in perspective: instead of memorizing individual observations, learn the probabilistic structure that produced them.

If a model can represent that structure, it can use the learned distribution to evaluate data and generate new samples that follow the same underlying patterns.

That gives us the core mental model:

Learn the distribution behind the data, then use that distribution for estimation or generation.

From Observed Data to a Model Distribution

Suppose a dataset contains many dog images.

Storing those images is not enough to create a new dog image that was never part of the training set. To generate new samples, the model needs to capture structure shared across the observed examples and represent that structure as a probability distribution.

We can think of the training data as samples drawn from an unknown Data Distribution:

training data \sim p_{data} (x)

The exact form of $p_{data} (x)$ is not known in advance. We only observe samples from it.

Generative Modeling therefore learns a Model Distribution, $p_{model} (x)$ , from those observations. New samples can then be drawn from the learned distribution:

generated samples \sim p_{model} (x)

Here, $p_{data} (x)$ represents the distribution followed by real data, while $p_{model} (x)$ represents the distribution learned by the model.

The goal is for the Model Distribution to represent the Data Distribution well enough to support both probability evaluation and new sample generation.

Density Estimation and Sample Generation Share the Same Foundation

A learned Model Distribution can be used in two closely related ways.

Density Estimation

Density Estimation focuses on learning a distribution that can evaluate probabilities for data.

If the model captures the structure of real data well, samples similar to real data can receive higher probability, while samples that differ from that structure can receive lower probability.

The important point is that the model is not simply remembering particular training examples. It is representing the broader probabilistic structure shared by the data.

Sample Generation

Sample Generation uses the learned Model Distribution to draw new samples.

The generated samples do not need to be identical to anything in the training set. What matters is that they follow the structure represented by $p_{model} (x)$ .

The overall flow is:

Observed training data
        |
        v
Learn the Model Distribution
        |
        v
p_model(x)
   /          \
  v            v
Density      Sample
Estimation   Generation

Density Estimation and Sample Generation are therefore two ways of using the same learned distribution.

For a developer, the key structural idea is to identify the distribution being modeled and then see how that learned structure is used for probability evaluation or generation.

Why Random Sampling Is Not Enough

Images, text, and audio are represented in high-dimensional spaces. But not every point in those spaces corresponds to meaningful real data.

Consider images. If every pixel value is selected uniformly at random, the result is unlikely to resemble a real image.

The same applies to text. Randomly choosing characters rarely produces meaningful language.

The issue becomes clearer when we think about the full high-dimensional space:

High-dimensional data space
+--------------------------------------+
|                                      |
|          enormous set of             |
|        possible configurations       |
|                                      |
|             +----------+             |
|             | real data|             |
|             | structure|             |
|             +----------+             |
|                                      |
+--------------------------------------+

The number of possible images or text sequences can be enormous, while realistic data occupies only a negligible portion of that space.

So generation is not simply a matter of selecting values from a large space. The model has to represent the restricted structure where real data actually exists.

The Real Challenge in High Dimensions

The difficulty is not only that the data lives in a high-dimensional space. The model must represent the complex structure formed by real observations inside that space.

Realistic images are not arbitrary collections of pixel values. Meaningful text is not an arbitrary sequence of characters.

Real data exists in limited, structured regions rather than being spread uniformly across all possible configurations.

This perspective connects naturally to the kind of data structure considered in Manifold Learning.

A naive picture of generation would be:

Generation
    =
choose values from a large space

A better mental model is:

Generation
    =
learn where realistic data is structured
    +
sample according to that learned structure

Representing complex high-dimensional distributions requires sufficient model capacity. Simple models may not be able to capture this complexity, which leads naturally toward Deep Generative Models for representing more complex data distributions.

Why Distribution Learning Matters Beyond Direct Generation

Generative Modeling matters for more than producing new samples. Once a model has learned structure in the data distribution, that structure can also support other learning problems.

In reinforcement learning, possible futures can be simulated and used for Planning.

In Continual Learning, Generative Replay can reuse previously learned information through generated data.

Distribution learning can also help when part of the data is unobserved. The learned structure can support predictions about missing parts, connecting Generative Modeling with Imputation.

Another important connection is Latent Representation learning. A generative model can capture features shared across different observations in a latent form, and those representations can then serve as general features for other learning problems.

Across these cases, the common idea is that structure learned from the data can be reused beyond direct sample generation.

From Learned Distributions to Real Generation Tasks

The distribution perspective becomes especially important when the output must resemble realistic data.

Examples include:

Image Super-Resolution
Audio Super-Resolution
Art Creation
X-to-X Translation

These problems look different, but they share a common requirement: the generated output must reflect the structure learned from real data.

In Super-Resolution, the model takes a lower-resolution input and produces a higher-resolution result. Producing additional values is not enough; the result must follow the learned structure of realistic data.

X-to-X Translation transforms one data representation into another while producing an output consistent with the structure learned from data.

Art Creation follows the same general principle: the model uses learned data structure to produce a new result rather than simply reproducing an observed sample.

The common flow is:

Real data
   |
   v
Learn its probabilistic structure
   |
   v
Model Distribution
   |
   +--> evaluate data probability
   |
   +--> generate new samples

From an implementation perspective, the central structure is straightforward: determine what distribution is being modeled, understand how the model represents the structured region occupied by high-dimensional data, and see how that learned distribution is used for estimation or generation.

Takeaway

Generative Modeling is fundamentally about learning the distribution behind observed data.

Training examples are sampled from $p_{data} (x)$ , while generated samples come from the learned $p_{model} (x)$ .

Density Estimation and Sample Generation share that learned distribution as their common foundation.

The difficult part is that realistic images, text, audio, and other high-dimensional data occupy only a small, structured portion of the full space of possible values. A generative model must capture that structure rather than sample randomly from the full space.

Learn the structure of the data distribution, then use that structure to evaluate or generate data.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/generative-modeling-course-en/

Discriminative vs Generative Models: What Are You Actually Learning?

zeromathai — Wed, 02 Sep 2026 05:33:44 +0000

When a supervised model maps an input to an output, it is easy to assume that every model is learning essentially the same relationship. In practice, two models can solve the same prediction problem while modeling very different probability relationships.

A Discriminative Model learns the relationship needed to predict the output directly. A Generative Model instead models how the input is distributed for each possible output and uses that probability structure to infer the prediction.

The most useful question is therefore not simply, ""What does the model predict?"" but ""What probability relationship does it learn directly?""

Start with the supervised learning objective

At a high level, supervised learning asks us to learn a mapping from an observed input $x$ to a target output $y$ :

f : x \mapsto y

Both discriminative and generative approaches aim to solve this mapping problem. The difference is the probability relationship they model in order to reach the prediction.

A compact mental model is:

Discriminative
x
│
├── learn p(y | x)
│
└── choose y directly

Generative
y
│
├── learn p(x | y)
├── combine with p(y)
│
└── infer the best y for x

The prediction target is the same, but the learned structure is different.

Discriminative Models: learn the conditional relationship directly

A Discriminative Model focuses on the conditional probability of an output given an input:

p (y ∣ x)

For classification, prediction can be written as:

f (x) = ar g y max p (y ∣ x)

The calculation flow is direct:

input x
   ↓
estimate p(y | x)
   ↓
compare candidate outputs
   ↓
select the most probable y

The model focuses on the relationship needed to distinguish possible outputs from the observed input. In the input space, this leads naturally to the idea of a decision boundary: regions of the space are associated with different outputs according to the learned predictive relationship.

Support Vector Machine is a representative discriminative example in this comparison. Its role here is best understood through the decision structure it learns between outputs rather than as a model of the overall probability distribution of the data.

Generative Models: model the input distribution for each output

A Generative Model approaches the same supervised problem from the opposite conditional direction. Instead of directly modeling $p (y ∣ x)$ , it models:

p (x ∣ y)

This tells us how likely an input $x$ is under a particular output $y$ . The model then combines this conditional probability with the prior probability of the output, $p (y)$ .

Prediction can therefore be written as:

f (x) = ar g y max p (x ∣ y) p (y)

The calculation flow becomes:

candidate y
   ↓
evaluate p(x | y)
   ↓
combine with p(y)
   ↓
compare p(x | y)p(y)
   ↓
select the best y

Bayes rule shows how this connects back to the conditional probability used by the discriminative view:

p (y ∣ x) = \frac{p ( x ∣ y ) p ( y )}{p ( x )}

For a fixed input $x$ , the denominator $p (x)$ is shared across all candidate outputs. When the goal is only to determine which output has the highest posterior probability, that common denominator does not affect the comparison.

This is why a Generative Model can use $p (x ∣ y)$ together with $p (y)$ to choose the appropriate output without directly modeling $p (y ∣ x)$ .

Gaussian Mixture and Bayes Nets are representative generative examples in this comparison. The important distinction is that this approach models the probability distribution describing how inputs relate to possible outputs.

The comparison that matters

It is tempting to reduce the distinction to ""classification versus generation,"" but that misses the central point. In this supervised-learning setting, both approaches can predict $y$ from $x$ .

What differs is what they learn directly.

	Discriminative Model	Generative Model
Prediction goal	Predict $y$ from $x$	Predict $y$ from $x$
Directly modeled relationship	$p (y ∣ x)$	$p (x ∣ y)$
Additional probability used	—	$p (y)$
Prediction rule	$ar g max_{y} p (y ∣ x)$	$ar g max_{y} p (x ∣ y) p (y)$
Useful structural view	Decision boundary	Probability distribution
Representative example	Support Vector Machine	Gaussian Mixture, Bayes Nets

For developers, this distinction is useful when reading a model definition or implementation. If the model directly represents the relationship from $x$ to $y$ , the focus is discriminative. If it models how $x$ behaves under each $y$ and combines that with $p (y)$ , the focus is generative.

Takeaway

Discriminative and generative models can solve the same supervised prediction problem through different probabilistic views.

A Discriminative Model directly models:

p (y ∣ x)

A Generative Model models:

p (x ∣ y)

and combines it with $p (y)$ to infer the appropriate output.

The key distinction is not simply classification vs generation. It is which probability relationship the model learns directly and uses for prediction.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/discriminative-vs-generative-models-course-en/

Generative Modeling: From Data Distributions to Deep Generative Models

zeromathai — Wed, 02 Sep 2026 00:13:36 +0000

If you approach generative models as ""networks that create images,"" the field quickly turns into a collection of disconnected architectures.

A more useful developer mental model starts one level lower:

What probability structure could have produced the data, and how can we represent, learn, and infer that structure without making the computation impossible?

That question connects autoregressive models, VAEs, flow-based models, GANs, and diffusion models. Their architectures look very different, but they all respond to the same underlying tension: high-dimensional data distributions are difficult to represent, learn, normalize, sample from, and reason about.

Generative modeling can therefore be organized around three interacting problems:

Representation: How do we represent a complex high-dimensional joint distribution?
Learning: How do we make the model distribution approach the data distribution?
Inference: Given an observation, how do we reason backward about hidden variables or the process that generated it?

Once these three pieces are connected, the major families of deep generative models become much easier to understand.

From prediction to distribution learning

A discriminative model usually begins with a prediction problem. Given an input $x$ , predict the most likely output $y$ :

f (x) = y ar g max p (y ∣ x)

The model focuses directly on the conditional relationship required for prediction.

A generative model asks a broader question. Instead of learning only the path from $x$ to $y$ , it models the probability structure from which the data arises.

For class-conditional modeling, for example, we can model $p (x ∣ y)$ together with the prior $p (y)$ and recover the posterior using Bayes' rule:

p (y ∣ x) = \frac{p ( x ∣ y ) p ( y )}{p ( x )}

In unsupervised generative modeling, the target becomes the data distribution itself. We assume the training samples come from some unknown distribution:

x_{1}, x_{2}, \dots, x_{N} \sim i.i.d. p_{data} (x)

The model then constructs a parameterized distribution intended to approximate it.

This perspective connects tasks that can otherwise seem separate:

x_{train} \sim p_{data} (x), x_{new} \sim p_{model} (x)

Density estimation asks how probable an observation is under the model. Generation draws new samples from the learned model distribution. Unsupervised representation learning can capture latent structure shared by the data while learning that distribution.

All three begin with the same underlying goal: learning the probability structure of the data.

Why high-dimensional data makes this hard

Real data usually lives in a high-dimensional space. An image, for example, is described by many variables, while valid observations occupy only a limited and complicated region among all possible combinations.

So the first problem is not simply ""use a neural network."" It is:

How do we represent the joint distribution of many random variables compactly enough to compute with, while keeping enough flexibility to describe the data?

This connects naturally to inverse problems.

A forward problem moves from a cause $x$ to an observation $y$ . An inverse problem starts from an observed $y$ and asks which possible $x$ could have produced it. Conditional generation extends the same idea by generating possible $x$ values consistent with a given condition.

Forward:
cause x  ──────────>  observation y

Inverse:
possible cause x  <──────────  observation y

Conditional generation:
condition y  ──────────>  possible samples x

Recovering a high-resolution image from a low-resolution one, estimating color from grayscale data, reconstructing missing regions, or recovering a CT slice from a sparse-view sinogram all share this structure.

The difficulty is that the inverse may not be unique. A single observation can be consistent with multiple possible causes. Inverse problems are therefore not simply about running a function backward; they require reasoning over possible causes that are compatible with the observation.

Latent variables turn inversion into posterior inference

Suppose a model contains an observed variable $x$ and a hidden variable $z$ .

The generative direction models the relationship between $z$ and $x$ . Once $x$ has been observed, the reverse question is which values of $z$ could have produced it.

That is posterior inference:

p (z ∣ x) = \frac{p ( z , x )}{p ( x )}

Posterior inference can require both computing the posterior distribution and calculating expectations under it. The E-step of the EM algorithm is a direct example:

Q (θ, θ^{old}) = z \sum p (z ∣ x; θ^{old}) lo g p (x, z; θ)

Here the posterior over the unobserved $z$ is used to compute the expected complete-data log-likelihood.

As the latent space grows and interactions among hidden variables become more complicated, direct posterior computation can become intractable. Increasing model expressivity therefore leads naturally to another question: how can the required inference remain computationally feasible?

more expressive representation
        ↓
more complicated latent structure
        ↓
harder posterior inference
        ↓
need for approximate inference

When the posterior is intractable

Two major approaches are MCMC and Variational Inference. Both approximate a posterior that is difficult to compute directly, but the approximation arises in different ways.

MCMC: approximate with samples

MCMC constructs an ergodic Markov chain whose stationary distribution is the target posterior. Samples generated by running the chain are then used to approximate the posterior and the expectations we need.

target posterior
      ↓
construct Markov chain
      ↓
run the chain
      ↓
collect samples
      ↓
approximate posterior expectations

The practical approximation comes from finite computation. In real problems, only a limited amount of time is available for generating samples, and exploring the relevant probability regions can be expensive in high-dimensional or complex models.

MCMC therefore has a finite-time sampling approximation in practice.

Variational Inference: approximate with optimization

Variational Inference makes a different tradeoff. Instead of sampling directly from the posterior, it defines a tractable family of distributions $Q$ and searches within that family for a distribution close to the true posterior:

q^{*} (z) = q (z) \in Q ar g min D_{KL}! [q (z) ∥ p (z ∣ x)]

This turns posterior inference into an optimization problem.

The tractability comes from restricting the search space. If the true posterior cannot be represented by the chosen variational family, more optimization time alone cannot eliminate the mismatch.

The contrast is useful:

MCMC approximation source:
finite-time sampling

Variational Inference approximation source:
restricted distribution family

This distinction becomes concrete in a VAE. Rather than computing a complicated latent posterior directly, the model uses a variational distribution, with the Encoder acting as a Recognition Network that constructs the approximation. The Decoder generates observations from the latent variable.

An abstract inference problem has now become part of a trainable model architecture.

The real bottleneck: flexibility versus tractability

Representation and computation collide when we try to model realistic high-dimensional distributions.

A simple probability model can be easy to normalize and evaluate but may lack enough flexibility to describe complex data. A deep neural network can represent much richer functions, but its output is not automatically a valid probability density.

A probability density must be nonnegative and integrate to one. One way to construct such a density is:

p_{θ} (x) = \frac{e ^{f_{θ} (x)}}{Z _{θ}}

where

Z_{θ} = \int e^{f_{θ} (x)} d x

Exponentiation gives a positive quantity, while the normalizing constant $Z_{θ}$ ensures that the density integrates to one.

For a complex neural function over a high-dimensional space, however, computing that integral can itself become intractable.

simple probability model
    → tractable
    → limited flexibility

deep neural representation
    → flexible
    → normalization or inference may become intractable

This is the Tractability-Flexibility Tradeoff.

It provides a useful way to compare deep generative models. Rather than asking only which network architecture a model uses, ask which difficult probability computation its structure makes manageable.

Learning a model distribution

In likelihood-based generative modeling, learning can be viewed as reducing the difference between the data distribution and the model distribution:

D_{KL}! [p_{data} (x) ∥ p_{model} (x; θ)] E x \sim p data! [lo g \frac{p _{data} ( x )}{p _{model} ( x ; θ )}]

Expanding this expression separates a term that depends only on the data distribution from a term containing the model log probability. The data-only term is independent of $θ$ , so it does not change as the model parameters are optimized.

The learning objective therefore connects to increasing the model's log probability on observed data. With a finite training set, this gives the progression:

minimize KL divergence
        ↓
minimize cross-entropy
        ↓
minimize negative log-likelihood

The challenge is to make $p_{model} (x; θ)$ expressive enough for high-dimensional data while keeping the required probability computations feasible.

This is where the major generative model families diverge.

Autoregressive models: factorize the joint distribution

An autoregressive model handles a high-dimensional joint distribution by decomposing it into sequential conditional distributions:

p (x) = i = 1 \prod n p (x_{i} ∣ x_{1}, \dots, x_{i - 1})

Instead of directly modeling one complicated joint distribution, the model works with a product of conditional probabilities.

The key structural idea is factorization. The model represents the high-dimensional joint distribution through a sequence of conditional distributions that can be modeled in a computationally manageable form.

VAE: latent variables meet approximate inference

A latent-variable model can express the observed distribution as:

p (x) = \int p (z) p (x ∣ z) d z

This provides a latent space for representing hidden structure, but it also introduces the posterior problem $p (z ∣ x)$ .

A VAE connects the pieces this way:

observed x
   ↓
Encoder / Recognition Network
   ↓
variational approximation to posterior over z
   ↓
latent z
   ↓
Decoder / Generative Network
   ↓
generated or reconstructed x

Training connects this architecture to Variational Inference through maximizing the ELBO:

latent-variable representation
        ↓
intractable posterior
        ↓
variational approximation
        ↓
Encoder
        ↓
ELBO-based learning

This is why a VAE is more than an Encoder followed by a Decoder. Its structure turns posterior intractability and variational approximation into a trainable generative model.

Flow-based models: use invertibility to preserve tractability

Flow-based models start from a simple distribution and apply a sequence of invertible transformations to reach a more complex target distribution.

Invertibility is the critical structural constraint. Because the transformations can be reversed, probability density can be tracked through them, allowing the model to train with NLL.

architectural restriction
        ↓
invertible transformation
        ↓
density remains computable
        ↓
likelihood-based training becomes possible

The constraint is part of the solution to the tractability problem: it introduces a structural restriction while keeping density evaluation possible.

GAN: learn the generation process instead of explicit density

GANs make a different choice. Instead of directly computing an explicit density function or normalizing constant, they learn the generation process itself.

The Generator tries to produce samples resembling real data, while the Discriminator tries to distinguish generated samples from real ones. The two networks train with opposing objectives in a minimax problem.

latent input
    ↓
Generator
    ↓
generated sample ──┐
                   ├──> Discriminator
real sample ───────┘

GANs use this adversarial minimax process to construct an implicit generative model rather than directly evaluating an explicit model density.

Diffusion: connect noise and data through many steps

Diffusion models take another route. Rather than directly handling the transformation between a simple distribution and a complex data distribution in one step, they connect the two through a sequence of noise levels.

The forward process gradually adds Gaussian noise to real data:

x_{0} \sim q (x), x_{T} \to N (0, I) as T \to \infty

As the process continues, the original data structure gradually disappears and approaches a simple isotropic Gaussian distribution.

Generation runs in the opposite direction:

data
  ↓
add noise
  ↓
add more noise
  ↓
...
  ↓
Gaussian-like noise

generation:

Gaussian noise
  ↓
learned reverse step
  ↓
learned reverse step
  ↓
...
  ↓
data-like sample

Instead of using one direct transformation, diffusion models learn a reverse process that progressively reconstructs data structure from noise.

The score view of diffusion

Score-based generative models can describe this process using the score function:

s (x) = \nabla_{x} lo g p (x)

The score indicates the direction in which the log probability increases.

Two representative approaches are SMLD and DDPM.

SMLD, or Score Matching with Langevin Dynamics, estimates the score at multiple noise scales and uses Langevin Dynamics to sample while the noise scale decreases.

DDPM, or Denoising Diffusion Probabilistic Model, learns a probabilistic reverse model for the steps of the forward noise-corruption process.

Their emphasis can be summarized as:

SMLD:
score estimation
      +
Langevin Dynamics

DDPM:
forward corruption steps
      +
learned reverse probability model

The details differ, but both approaches connect a simple noise distribution to a complex data distribution through multiple noise levels or steps.

That iterative structure also creates a practical tradeoff. Because generation requires multiple diffusion steps, sampling speed can become an important limitation.

One probability problem, different structural answers

The major generative model families are easier to compare when viewed as different computational responses to the same high-dimensional distribution problem:

Autoregressive models factorize a joint distribution into sequential conditional distributions.
VAEs introduce latent variables and use Variational Inference to approximate a difficult posterior while training through the ELBO.
Flow-based models impose invertibility so density evaluation and NLL-based learning remain possible.
GANs avoid direct explicit-density evaluation and learn the generation process through adversarial minimax training.
Diffusion models transform generation into a sequence of noise-corruption and reverse-process steps.

The normalizing constant fits into the same picture. Energy-Based Models work with energy-based probability distributions in which the normalizing constant must be handled or approximated. Autoregressive models, flow-based models, and VAEs use computationally manageable structural constraints or model constructions. GANs avoid direct computation of explicit density and its normalizing constant by learning the generation process, while score-based diffusion uses the score rather than directly computing the normalizing constant.

Different architectures make different choices, but the underlying problem is the same: preserve enough flexibility to represent complex data while keeping generative modeling computationally feasible.

A developer mental model for generative architectures

When you encounter a generative architecture, start with three questions rather than its layer diagram.

Representation: What distribution or latent structure is the model trying to represent?

Learning: What objective reduces the difference between the model and the observed data distribution?

Inference: What hidden variables or generating processes must be inferred from observations, and can those quantities be computed directly?

Then look at the structural choice the model uses to keep the relevant computations manageable:

high-dimensional data distribution
          ↓
representation must be expressive
          ↓
probability or posterior computation becomes difficult
          ↓
choose a computational strategy
          ↓
factorize
approximate
restrict
avoid explicit density
or learn a multi-step reverse process

No single family removes every tradeoff. In the comparison considered here, VAEs and flow-based models can show weaknesses in generation quality, GANs can face difficulties with generation diversity, and diffusion models can pay for their iterative generation process with slower sampling.

So the useful comparison is not simply ""Which model is better?""

It is:

Which computational problem does the model solve through its structural choice, and what does that choice cost in quality, diversity, or speed?

Takeaway

Generative modeling is not fundamentally about producing realistic-looking samples. It is about learning an unknown high-dimensional data distribution while keeping Representation, Learning, and Inference computationally manageable.

Autoregressive models, VAEs, flow-based models, GANs, and diffusion models look different because they make different structural choices around the same underlying problem. Once you identify whether an architecture factorizes a distribution, approximates an intractable quantity, imposes a structural constraint, avoids explicit density evaluation, or learns a multi-step reverse process, the landscape of deep generative models becomes much easier to reason about.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/generative-model-overview-course-en/

How Modern Transformer Blocks Work — From RMSNorm to MoE

zeromathai — Mon, 29 Jun 2026 10:42:05 +0000

The original Transformer idea is still alive.

But modern LLM blocks are not just the 2017 Transformer copied and scaled.

They are engineered for deeper training, longer context, cheaper inference, and larger capacity.

That is why components like RMSNorm, GQA, RoPE, SwiGLU, and MoE matter.

Core Idea

A modern Transformer block still follows the same basic pattern:

Attention updates token relationships.

The Feed-Forward Network transforms each token representation.

Residual connections keep information flowing.

But modern LLMs changed the details.

Those details are not cosmetic.

They make large-scale training and inference practical.

The Key Structure

A typical modern Transformer block looks like this:

Input

→ RMSNorm or Pre-Layer Normalization

→ Self-Attention with GQA and RoPE

→ Residual Connection

→ RMSNorm or Pre-Layer Normalization

→ Feed-Forward Network with SwiGLU or MoE

→ Residual Connection

More compactly:

Modern Transformer Block = stable normalization + efficient attention + stronger FFN + residual flow

Each component solves a real scaling problem.

Pre-LN improves deep training stability.

GQA reduces KV Cache memory.

RoPE injects position into attention.

SwiGLU improves FFN expressiveness.

MoE increases capacity without activating all parameters.

Pseudo-code View

A simplified modern block looks like this:

def transformer_block(x):
    h = rms_norm(x)

    attn = grouped_query_attention(
        q=apply_rope(query(h)),
        k=apply_rope(key(h)),
        v=value(h)
    )

    x = x + attn

    h = rms_norm(x)

    ffn = swiglu_ffn(h)

    x = x + ffn

    return x

With MoE, the FFN part can become:

h = rms_norm(x)

selected_experts = router(h)

ffn = run_top_k_experts(h, selected_experts)

x = x + ffn

The pattern stays simple.

Normalize.

Transform.

Add back.

Repeat.

Concrete Example

Imagine the model processes this token:

"bank"

The attention block helps decide whether “bank” means:

a financial institution

or the side of a river

RoPE helps the model understand token order and distance.

GQA helps attention run with a smaller KV Cache.

The FFN then transforms the contextual representation.

If the model uses MoE, the router may send this token to experts specialized for finance, geography, or general language.

That is the intuition.

Modern Transformer blocks are not just bigger.

They are more selective, stable, and hardware-aware.

Pre-LN vs Post-LN

The original Transformer commonly used Post-LN.

Post-LN:

x = LayerNorm(x + Sublayer(x))

Modern LLMs often use Pre-LN.

Pre-LN:

x = x + Sublayer(LayerNorm(x))

The difference looks small.

But it matters.

Pre-LN normalizes before the sublayer.

That helps gradients flow through deep Transformer stacks.

When a model has dozens or hundreds of layers, this becomes critical.

Pre-LN is not just a formatting choice.

It is a training stability choice.

RMSNorm

RMSNorm is a simpler normalization method.

LayerNorm recenters and rescales.

RMSNorm mainly rescales using the root mean square.

The RMS is:

RMS(h) = sqrt((1 / n) * Σ hᵢ²)

Then the normalized vector is:

h_norm = h / (RMS(h) + ε) * g

Why use it?

It keeps activation scale stable.

It removes some computation compared with LayerNorm.

It works well in large LLMs.

Example:

h = [3, 4]

RMS(h) = sqrt((9 + 16) / 2) ≈ 3.54

Normalized h ≈ [0.85, 1.13]

The key idea:

RMSNorm stabilizes scale without doing more than necessary.

Attention Block: GQA + RoPE

Modern attention is often not plain Multi-Head Attention.

It usually combines memory-aware attention with positional encoding.

Grouped-Query Attention reduces KV Cache size.

Rotary Positional Embedding injects position into Query and Key.

The attention flow becomes:

Input

→ Q, K, V projection

→ Apply RoPE to Q and K

→ Share K/V by groups using GQA

→ Compute attention

→ Output projection

This matters for inference.

Long-context generation is often limited by KV Cache memory.

GQA reduces that pressure.

RoPE keeps position information inside attention without adding a large position table.

SwiGLU

The Feed-Forward Network is not just a simple MLP anymore.

Many modern LLMs use SwiGLU.

SwiGLU is a gated activation.

One path carries information.

Another path controls how much passes through.

A simplified formula:

SwiGLU(x) = (W₁x) * Swish(W₂x)

Example:

W₁x = 4

Swish(W₂x) = 0.5

Output = 2

The gate decides how much information moves forward.

That gives the FFN more control than a plain activation.

Mixture of Experts

Mixture of Experts increases model capacity without activating every parameter for every token.

Instead of one FFN, the model has multiple expert networks.

A router chooses which experts handle each token.

Example router output:

Expert 1 = 0.45

Expert 2 = 0.19

Expert 3 = 0.05

Expert 4 = 0.31

With Top-2 routing:

Expert 1 and Expert 4 are selected.

Only those experts run.

This is why MoE is called sparse.

The model may have many parameters.

But each token uses only a small subset.

Dense FFN vs MoE

Dense FFN:

every token uses the same FFN
all FFN parameters are active
simpler to train and serve
compute grows directly with FFN size

MoE:

each token is routed to selected experts
only part of the model activates
increases total capacity efficiently
adds routing and load-balancing complexity

The key difference:

Dense FFN = same compute path for every token

MoE = conditional compute path per token

MoE is powerful.

But it is not free.

It introduces routing instability, expert imbalance, and distributed communication overhead.

Multi-Token Prediction

Standard language modeling predicts one next token.

At position t:

predict token t + 1

Multi-Token Prediction trains the model to predict multiple future tokens.

At position t:

predict token t + 1, t + 2, t + 3 ...

This gives more learning signals from the same representation.

Standard training:

one position → one supervision signal

MTP training:

one position → multiple supervision signals

This can improve sample efficiency.

In some systems, it can also support faster generation ideas.

Naive vs Modern View

Naive view:

Transformer block = attention + FFN

Modern view:

Transformer block = stable normalization + efficient attention + gated FFN + sparse scaling

Naive block:

attention
ffn

Modern block:

rmsnorm
rope
gqa
residual
rmsnorm
swiglu or moe
residual

This matters because modern LLM performance is not just about parameter count.

It is about architecture details that make those parameters trainable and deployable.

Implementation Perspective

When reading modern LLM code, look for these patterns:

self.input_layernorm = RMSNorm(...)

self.self_attn = Attention(..., rope=True, num_key_value_heads=...)

self.post_attention_layernorm = RMSNorm(...)

self.mlp = SwiGLU(...) or MoE(...)

The key clue for GQA is:

number of query heads > number of key-value heads

The key clue for RoPE is:

position is applied to Q and K before attention

The key clue for MoE is:

router logits decide which experts run

These details tell you what kind of Transformer block you are actually looking at.

Important Conditions and Limits

Pre-LN improves stability, but the whole optimization setup still matters.

RMSNorm is efficient, but it does not replace good initialization or training design.

GQA reduces KV Cache memory, but may trade off some attention flexibility.

RoPE works well for long contexts, but very long extrapolation may still need scaling techniques.

SwiGLU improves FFN behavior, but increases FFN structure complexity.

MoE increases capacity, but adds routing and system complexity.

Modern Transformer design is a trade-off system.

Every upgrade solves one bottleneck and introduces another design choice.

Why This Matters Again

Modern LLMs are not just large neural networks.

They are carefully engineered stacks.

If you understand the block, you can better understand:

why inference needs KV Cache optimization
why RoPE appears in attention code
why RMSNorm replaces LayerNorm
why GQA changes memory usage
why MoE models can be huge but still sparse

This is the difference between using LLMs and understanding how they scale.

Takeaway

Modern Transformer blocks preserve the original Transformer idea.

But they upgrade almost every practical detail.

The shortest version:

Modern Transformer Block = Pre-LN/RMSNorm + GQA/RoPE Attention + SwiGLU/MoE FFN + Residual Connections

If Self-Attention is the core idea, the modern block is the production-grade version of that idea.

It is built for depth, context length, inference memory, and scalable capacity.

Discussion

When reading modern LLM architecture, which component feels most important to understand first?

RMSNorm, RoPE, GQA, SwiGLU, or MoE?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/modern-transformer-blocks-llm-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

Why Positional Embeddings Matter — APE, RPE, and RoPE Explained for Developers

zeromathai — Fri, 26 Jun 2026 15:01:50 +0000

Self-Attention can compare every token with every other token.

But there is a catch.

By itself, it does not know the order of tokens.

That is a serious problem because “dog bites man” and “man bites dog” use the same words but mean completely different things.

Core Idea

A Transformer needs two kinds of information:

what the token is

where the token is

Token embeddings provide the “what.”

Positional embeddings provide the “where.”

This matters because attention without position is order-blind.

It can compare tokens, but it does not naturally know which token came first.

The Key Structure

A simple positional embedding flow looks like this:

Token Embedding + Positional Information → Input Representation

For Absolute Positional Embedding:

E = X + P

Where:

X = token embedding

P = positional embedding

E = final input representation

More compactly:

Transformer input = meaning vector + position signal

Different positional methods change how the position signal is injected.

Pseudo-code View

Basic positional injection:

tokens = tokenize(text)

x = embedding(tokens)

position = positional_embedding(token_positions)

input_representation = x + position

For attention-based position methods:

q = project_query(x)

k = project_key(x)

q = apply_position(q)

k = apply_position(k)

attention_scores = q @ k.T

APE usually modifies the input embedding.

RPE usually modifies the attention score.

RoPE usually modifies Query and Key.

That difference is the whole story.

Concrete Example

Compare these two sentences:

dog bites man

man bites dog

The token set is the same:

dog, bites, man

But the order changes the meaning.

Without positional information, Self-Attention sees token relationships but has no built-in sequence order.

With positional information, each token representation includes location.

So “dog” at position 1 is different from “dog” at position 3.

This is why positional encoding is not optional.

It is required for language understanding.

APE: Absolute Positional Embedding

Absolute Positional Embedding assigns a vector to each position index.

Position 1 has one vector.

Position 2 has another vector.

Position 3 has another vector.

Then the model adds that position vector to the token embedding.

Example:

Token embedding:

X = [0.2, 0.5]

Position embedding:

P = [0.1, -0.2]

Final representation:

E = [0.3, 0.3]

APE is easy to understand.

It says:

this token is at this exact position

Why APE Is Useful

APE is simple.

It is easy to implement.

It works well when sequence lengths stay close to what the model saw during training.

Implementation-wise, it is just:

x = token_embedding + position_embedding

That makes it cheap and clean.

But the simplicity has a cost.

APE treats position as a fixed index.

If the model sees much longer inputs than it was trained on, unseen positions can become unreliable.

That makes APE weaker for long-context extrapolation.

RPE: Relative Positional Embedding

Relative Positional Embedding focuses on distance.

Instead of asking:

What position is this token at?

It asks:

How far apart are these two tokens?

This is often more natural for language.

A subject and verb may appear at different absolute positions.

But their relative distance and direction still matter.

A simplified RPE attention score looks like this:

Aᵢⱼ = (QᵢKⱼᵀ + Rᵢ₋ⱼ) / √d

Rᵢ₋ⱼ represents the relative position between token i and token j.

This means position directly affects attention.

Concrete RPE Example

Suppose:

QᵢKⱼᵀ = 12

Rᵢ₋ⱼ = 4

√d = 4

Then:

Aᵢⱼ = (12 + 4) / 4 = 4

Without the relative term:

Aᵢⱼ = 12 / 4 = 3

So the distance relationship increased the attention score.

That is the intuition.

RPE lets the model say:

This token is more relevant because of where it is relative to me.

RoPE: Rotary Positional Embedding

Rotary Positional Embedding takes a different path.

It does not add a position vector to the input.

It rotates Query and Key vectors based on position.

The core idea:

position becomes rotation

A 2D rotation matrix looks like this:

Rθ = [[cosθ, -sinθ], [sinθ, cosθ]]

If you rotate [1, 0] by 90 degrees:

[1, 0] → [0, 1]

RoPE applies this idea across Query and Key dimensions.

Different positions get different rotations.

Then attention scores naturally include relative position.

Why RoPE Works Well

RoPE uses absolute position to rotate Q and K.

But when Q and K are compared, the score depends on their relative position difference.

The key relationship is:

(RθⁱQ)ᵀ(RθʲK) = QᵀRθʲ⁻ⁱK

This means the attention score contains j - i.

That is the relative distance.

So RoPE gives you a useful combination:

absolute-position injection + relative-position behavior

This is why RoPE became popular in modern LLMs.

APE vs RPE vs RoPE

APE:

adds position vectors to token embeddings
simple and cheap
good for fixed or known sequence lengths
weaker for long-context extrapolation

RPE:

adds relative distance information to attention scores
directly models token-to-token distance
flexible for variable lengths
can complicate attention implementation

RoPE:

rotates Query and Key vectors by position
makes relative distance appear inside attention
memory-efficient
works well with modern long-context LLMs

The key difference:

APE = where am I?

RPE = how far are we?

RoPE = rotate Q/K so distance appears in attention

Implementation Perspective

If you are reading Transformer code, look at where position enters the model.

APE usually appears near the embedding layer:

x = token_embedding + position_embedding

RPE usually appears inside attention score computation:

scores = q @ k.T + relative_position_bias

RoPE usually appears after Q and K projection:

q = apply_rope(q, positions)

k = apply_rope(k, positions)

scores = q @ k.T

This is the developer shortcut.

Find the injection point.

Then you know which positional method the model uses.

Naive vs Practical View

Naive view:

Positional embedding just tells the model token order.

Practical view:

Positional design affects long-context behavior, caching, memory, and attention quality.

Naive mindset:

add positions
run attention

Practical mindset:

choose how position enters attention
consider context length
consider extrapolation
consider KV Cache compatibility
consider implementation complexity

This matters because positional encoding is not a small detail.

It changes how the model behaves when the context becomes long.

Why This Matters Again

Short inputs can hide positional weaknesses.

Long-context models expose them.

If positional information does not extrapolate well, the model may become unstable outside its training length.

This is why modern LLMs care so much about RoPE variants and long-context scaling.

The position method affects whether a model can reliably handle long prompts, code files, documents, and conversations.

Important Conditions and Limits

APE is easy but tied to absolute indices.

RPE is expressive but can complicate attention computation.

RoPE is efficient and practical, but still needs careful scaling for very long contexts.

Also:

Positional embeddings do not create reasoning by themselves.

They only give attention a way to use order.

The model still needs training to learn useful patterns.

Takeaway

Self-Attention needs positional information because it is order-blind by default.

APE adds absolute position to embeddings.

RPE adds relative distance to attention scores.

RoPE rotates Query and Key vectors so relative position appears naturally.

The shortest version:

Positional Embedding = the order signal that makes attention understand sequence structure

If you understand where position enters the model, you understand the difference between APE, RPE, and RoPE.

Discussion

When learning Transformer internals, which positional method feels most intuitive to you?

APE, RPE, or RoPE?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/advanced-positional-embeddings-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai

DEV Community: zeromathai

Diffusion Models: From Noise Corruption to Reverse Generation

Turn a Hard Generation Problem Into Many Smaller Steps

Forward Diffusion: Gradually Corrupt the Data

Reverse Diffusion: Learn the Way Back

Forward and Reverse Processes at a Glance

The Score-Based View

SMLD: Scores Across Multiple Noise Scales

DDPM: Learn Reverse Probabilistic Transitions

How SMLD and DDPM Connect

What to Look for in an Implementation

Why Sampling Speed Is the Main Trade-Off

Takeaway

Deep Generative Models: Four Ways to Make Complex Distributions Learnable

Two Broad Families

From KL Divergence to NLL

Strategy 1: Factorize the Distribution

Strategy 2: Introduce a Latent Variable

Strategy 3: Transform a Simple Distribution

Strategy 4: Learn Without Direct Density Evaluation

The Structural Difference That Matters

Takeaway

Normalizing Constants: Turning Neural Network Scores into Probability Distributions

From Neural Network Scores to a Probability Density

What the Normalizing Constant Does

When Normalization Is Tractable

Why Zθ​ Becomes Difficult

The Tractability-Flexibility Tradeoff

How Generative Models Handle the Normalization Problem

What to Look for in a Probabilistic Model

Takeaway

Modeling Data Distributions: The Core Idea Behind Generative Models

Start With Samples, Not the Distribution

Data Distribution vs. Model Distribution

One Distribution, Three Useful Views

1. Generation

2. Density Estimation

3. Unsupervised Representation Learning

Why High-Dimensional Data Changes the Problem

Tractability vs. Flexibility

Why This Leads to Deep Generative Models

Takeaway

Approximate Inference: Sampling the Posterior vs Optimizing an Approximation

The core mental model

MCMC: represent the posterior with samples

Why high-dimensional sampling is difficult

Variational Inference: turn inference into optimization

Where VI's approximation comes from

MCMC vs VI: the distinction that matters

Takeaway

Posterior Inference: From Joint Distributions to the Inference Bottleneck

Start with the Joint Distribution

Posterior Distribution: Conditioning on Observed Data

Posterior Inference Has Two Closely Related Tasks

EM Shows Why Posterior Expectations Matter

Where Exact Inference Becomes Difficult

High-Dimensional Latent Spaces

Complex Posterior Distributions

Why Deep Models Add More Difficulty

Modeling and Inference Are Different Problems

Why Approximate Posterior Inference Becomes Necessary

What to Look for in a Probabilistic Model

Takeaway

Inverse Problems: Why Predicting Backward Is Harder Than It Looks

Forward Problems vs. Inverse Problems

Why the Inverse Direction Is Different

Common Examples of Inverse Problems

Low Resolution to High Resolution

Masked Observation to Full Image

Grayscale to Color

Stroke Painting to Image

Sparse-View CT Observation to Cross-Sectional Image

Why Not Just Learn an Inverse Function?

High-Dimensional Data Makes the Problem Harder

Latent Variables Add Another Layer

Three Core Challenges in Generative Modeling

Representation: How Do We Model the Joint Distribution?

Learning: How Do We Model the Data Distribution from Samples?

Inference: What Structure Explains the Observation?

What to Look for in an Inverse-Problem System

Why $Z_{θ}$ Becomes Difficult