Normalizing Constants: Turning Neural Network Scores into Probability Distributions

A neural network can assign a real-valued score to an input, but that score is not automatically a probability density. It may be negative, and even after making it positive, the resulting density still needs to integrate to 1 over the entire input space.

That second requirement is where the Normalizing Constant comes in.

For a Deep Generative Model, the core mental model is:

neural network score → exponentiation → normalization → valid probability density

Evaluating the network for one input can be straightforward. The harder part is often computing the quantity needed to normalize its outputs over the entire input space.

From Neural Network Scores to a Probability Density

Suppose a neural network defines a real-valued function $f_\theta(x)$ .

A generic neural network does not constrain this output to satisfy the requirements of a probability density. In particular, $f_\theta(x)$ may be negative, so its raw value cannot be interpreted directly as a density.

Exponentiating the output solves the non-negativity problem:

$$e^{f_\theta(x)}$$

This quantity is always positive. But positivity alone is not enough. For a continuous probability density, we also need

$$\int p_\theta(x)\,dx = 1.$$

To enforce this condition, introduce a Normalizing Constant $Z_\theta$ :

$$p_\theta(x)=\frac{e^{f_\theta(x)}}{Z_\theta}.$$

The numerator provides a positive, unnormalized value for each input. The denominator rescales those values so that the resulting density integrates to 1.

The construction is therefore:

input $x$ → network score $f_\theta(x)$ → positive value $e^{f_\theta(x)}$ → divide by $Z_\theta$ → probability density $p_\theta(x)$

The important distinction is that $e^{f_\theta(x)}$ is positive but still unnormalized. It becomes a valid probability density only after normalization.

What the Normalizing Constant Does

To determine the required scale, integrate the unnormalized values over the entire input space:

$$Z_\theta=\int e^{f_\theta(x)}dx$$

This gives the total amount represented by the unnormalized model. Using that quantity as the denominator gives

$$\int p_\theta(x)dx=\frac{1}{Z_\theta}\int e^{f_\theta(x)}dx=1$$

So $Z_\theta$ is not an arbitrary scaling parameter. It is exactly the quantity required to turn the unnormalized function into a valid probability density.

This also reveals an important computational distinction:

• Local computation: evaluate $f_\theta(x)$ for a particular input $x$ .
• Global computation: integrate $e^{f_\theta(x)}$ over the entire input space to obtain $Z_\theta$ .

Evaluating the network at one input and integrating the function it defines over the entire input space are fundamentally different tasks. That difference is where the normalization problem begins.

When Normalization Is Tractable

Normalizing constants are not always difficult to compute. For a probability distribution with a simple, known structure, the normalization factor may have an explicit form.

Consider a multivariate Gaussian with identity covariance:

$$p_\mu(x)=\frac{1}{(2\pi)^{d/2}}e^{-\frac{\lVert x-\mu\rVert^2}{2}}$$

Its normalization factor is determined explicitly by the dimension $d$ :

$$\frac{1}{(2\pi)^{d/2}}$$

This is what makes the Gaussian case useful as a contrast. Its known distributional structure allows the normalization factor to be computed directly.

A more flexible neural-network probability model can represent a more complicated function, but the corresponding normalization integral may no longer be easy to compute.

Model structure	Expressivity	Normalization
Structured distribution such as the Gaussian above	Constrained by a known distributional form	Explicitly computable
Complex neural-network probability model	Can represent more flexible functions	May require an intractable integral

Normalization itself is therefore not the problem. The difficulty appears when the function being normalized becomes sufficiently complex.

Why $Z_\theta$ Becomes Difficult

For one input, the network computes

$$f_\theta(x).$$

To obtain the Normalizing Constant, however, we need

$$Z_\theta=\int e^{f_\theta(x)}dx.$$

The first operation evaluates the model at a particular $x$ . The second depends on the behavior of the model across the entire input space.

When $f_\theta(x)$ is a complicated neural-network function, integrating it over all possible inputs may become intractable. If the input space is also very high-dimensional, as with images, the problem becomes even harder.

The key point is therefore more precise than simply saying that high-dimensional data is difficult. The central challenge is having to integrate an arbitrarily complex neural-network function over the full input space.

The Tractability-Flexibility Tradeoff

The Normalizing Constant exposes a central tension in probabilistic modeling.

A simpler model can make probability calculations easier, but its ability to represent complex data distributions may be limited. A more expressive model can represent more flexible distributions, but the required probability calculations may become harder.

The tradeoff can be summarized as:

more structural restriction → easier probability computation → less flexibility

versus

more model flexibility → more difficult probability computation → potentially intractable normalization

This is the Tractability-Flexibility Tradeoff.

Deep Generative Modeling therefore has to consider both sides of the problem: how expressive the model is and whether the probability computations implied by that model can actually be carried out.

The Normalizing Constant sits directly at that intersection.

How Generative Models Handle the Normalization Problem

When $Z_\theta$ is difficult to compute directly, different Generative Modeling approaches make different choices.

• Approximate the Normalizing Constant: Energy-Based Models approach the difficult Normalizing Constant through approximation.
• Restrict the model structure: Autoregressive Models, Flow-based Models, and Variational Autoencoders impose structural constraints to make the required probability-modeling computations manageable.
• Model the generation process instead: GANs focus on the generation process rather than directly computing the Normalizing Constant of an explicit probability density.
• Bypass direct computation of the Normalizing Constant: Score-based Diffusion Generative Models take an approach that does not require computing the Normalizing Constant directly.

A Restricted Neural Network Model does not simply mean a weak or low-capacity network. The restriction is structural: the model is designed so that the required probabilistic computations remain manageable.

From this perspective, the differences among Deep Generative Models are not only architectural. They also reflect different ways of handling the computational problem created by the Normalizing Constant.

What to Look for in a Probabilistic Model

When examining a probabilistic generative model, trace how it turns a model output into a valid probability distribution.

If the model begins with an unnormalized score, ask what makes that score non-negative and what makes the resulting density integrate to 1. In the formulation used here, that means identifying

$$Z_\theta=\int e^{f_\theta(x)}dx.$$

Then ask whether $Z_\theta$ can actually be computed.

If it cannot, the model must take another approach: approximate the Normalizing Constant, constrain the model structure so the required computations become manageable, focus on the generation process instead of an explicit normalized density, or bypass direct computation of the Normalizing Constant.

That choice is a useful way to understand why different Generative Models adopt different structures.

Takeaway

A neural network score does not automatically define a probability density. Exponentiation makes the score positive,

$$e^{f_\theta(x)},$$

but normalization requires

$$Z_\theta=\int e^{f_\theta(x)}dx,$$

which gives

$$p_\theta(x)=\frac{e^{f_\theta(x)}}{Z_\theta}.$$

For simple structured distributions, the normalization factor can be computed explicitly. For flexible neural-network probability models, computing it over the entire input space may become intractable.

That is the central idea: greater model flexibility can make probability computation harder, and different Generative Models make different structural choices to handle that tradeoff.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/normalizing-constant-course-en/