Inverse Problems: Why Predicting Backward Is Harder Than It Looks

Most machine learning examples follow a familiar direction: start with an input and predict an output.

Inverse problems ask the opposite question. You observe the result first, then try to infer the input, parameter, or underlying state that could have produced it.

The important catch is that this is often not a simple matter of running a function backward. One observation can be consistent with multiple possible causes, so the problem may require reasoning about a distribution of plausible solutions rather than recovering one unique answer.

That is the core connection between inverse problems and generative modeling.

Forward Problems vs. Inverse Problems

A Forward Problem starts with an input and predicts the corresponding output.

Supervised classification is a typical example:

input x
   |
   v
model
   |
   v
output y

Probabilistically, the quantity of interest is:

$$p(y|x)$$

Given $x$ , we want to predict or characterize $y$ .

An Inverse Problem starts from the observed result and asks what input or state could have produced it:

observed result y
        |
        v
infer possible x

The corresponding conditional distribution is:

$$p(x|y)$$

Given $y$ , we want to reason about possible values of $x$ .

The difference is deeper than swapping the order of the variables. In a forward problem, the relationship between an input and its result can be used to compute an output. In an inverse problem, one observed result may correspond to several possible inputs.

That makes the inverse direction fundamentally ambiguous in many cases.

Forward computation asks what an input produces. Inverse inference asks which inputs could explain what we observed.

Why the Inverse Direction Is Different

Consider a process that transforms detailed data into a more limited observation:

possible original data
        |
        v
  forward process
        |
        v
 limited observation

The inverse task goes in the opposite direction:

limited observation
        |
        v
    inference
        |
        v
possible original data

The challenge is that the observation may no longer contain enough information to identify one original state.

A low-resolution image, for example, may not preserve every detail from its high-resolution source. Several high-resolution images can therefore be consistent with the same low-resolution observation.

The inverse problem is better described by a conditional distribution:

$$P(\text{high-resolution}|\text{low-resolution})$$

The goal is not simply to undo an operation. It is to reason about the possible high-resolution images that are consistent with the observed low-resolution image.

Common Examples of Inverse Problems

The same structure appears in several image reconstruction and transformation tasks.

Low Resolution to High Resolution

Given a low-resolution image, infer possible high-resolution images:

$$P(\text{high-resolution}|\text{low-resolution})$$

Masked Observation to Full Image

Given partial image information, infer possible complete images:

$$P(\text{full image}|\text{mask})$$

Grayscale to Color

Given a grayscale image, infer possible color images:

$$P(\text{color image}|\text{gray image})$$

Stroke Painting to Image

Given a stroke painting, infer possible images:

$$P(\text{image}|\text{stroke painting})$$

Sparse-View CT Observation to Cross-Sectional Image

Given a sparse-view sinogram, infer possible CT cross-sectional images:

$$P(\text{cross-sectional image}|\text{sparse-view sinogram})$$

These examples differ in their data and observation process, but they share the same basic structure:

limited or transformed observation
              |
              v
     conditional inference
              |
              v
       more complete data

This is also why inverse problems are closely related to Conditional Generation. The observation acts as a condition, and the model must represent data that is consistent with that condition.

The important idea here is not a particular generation method. It is the reverse reasoning structure: infer possible data or inputs from an observed result.

Why Not Just Learn an Inverse Function?

Because an inverse may not be unique.

Suppose several inputs produce the same observation:

x1 ----\
x2 -----+----> y
x3 ----/

If only $y$ is available, there may be no way to determine which original $x$ produced it.

This is one reason inverse problems can be ill-posed. When one observation corresponds to several possible inputs, the observation alone may not determine a unique solution.

Additional information or regularization may therefore be needed.

A useful mental model is:

Naive view:
y -> inverse function -> x

Better view:
y -> distribution over possible x

The second view keeps the ambiguity visible instead of forcing a single answer where the observation may not justify one.

High-Dimensional Data Makes the Problem Harder

Non-uniqueness is not the only challenge.

Images and similar data are high-dimensional. They can be represented by many variables, but real data is not randomly spread across every possible point in that high-dimensional space.

A useful way to think about this is through manifold structure. Real data exists within a complex structure rather than being distributed arbitrarily across the entire space.

That means inverse inference cannot simply treat every possible high-dimensional configuration as equally meaningful. A model must be able to represent the complex structure in which real data exists and reason about possible data associated with the observation.

Latent Variables Add Another Layer

Observed data may also fail to reveal the underlying structure through which it was generated.

In those cases, Latent Variable Inference becomes important.

Conceptually:

latent structure
      |
      v
generated data
      |
      v
observation

Looking at the process in reverse means reasoning from the observation toward structure that may not be directly visible in the raw data.

This is another reason inverse problems are not simply algebraic inversion. The problem may also require inference about latent structure behind the observation.

Three Core Challenges in Generative Modeling

Inverse problems lead naturally into three broader questions that arise when modeling complex data with generative models:

Representation
     |
     v
Learning
     |
     v
Inference

Representation: How Do We Model the Joint Distribution?

High-dimensional data can contain many interacting random variables.

The representation problem asks how to model their Joint Distribution.

When data contains a very large number of variables, directly handling all of their interactions becomes difficult. A generative model therefore needs a way to represent a complex probability distribution efficiently.

For a developer, the practical question is straightforward:

What kind of probability structure can this model represent?

That representation determines how the model describes the structure of complex data.

Learning: How Do We Model the Data Distribution from Samples?

The underlying data distribution can be written as:

$$p_{\mathrm{data}}(x)$$

Training data can be viewed as samples drawn from this distribution.

The model defines its own parameterized distribution:

$$p_\theta(x)$$

Learning means using observed samples to build a model distribution that describes the probability structure of the real data.

Conceptually:

samples from real data
        |
        v
     learning
        |
        v
model distribution p_theta(x)

This leads to a central learning question:

How should the real data distribution and the model distribution be compared?

The underlying distribution $p_{\mathrm{data}}(x)$ is not directly available as a complete probability distribution. The model instead learns from samples and uses $p_\theta(x)$ to describe the probability structure behind those samples.

Inference: What Structure Explains the Observation?

The third challenge is inference.

If generation moves from underlying structure toward observable data, inference looks in the reverse direction: from raw observations toward the higher-level structure or features that explain them.

This perspective is closely related to Inverse Graphics:

high-level structure
        |
        v
   generated data

Generation: top -> bottom
Inference:  bottom -> top

A generative model is therefore not only about representing or generating data. It also connects to understanding what structure could explain an observation.

What to Look for in an Inverse-Problem System

When reading a model or system that solves an inverse problem, focus on a few questions:

• What is observed? Identify the condition $y$ .
• What must be inferred? Identify the unknown $x$ or latent structure.
• Is the solution unique? If not, think in terms of multiple plausible solutions rather than a single deterministic inverse.
• What constrains the solution? Look for additional information, regularization, or learned data structure.
• How is high-dimensional data represented? The model must account for the complex structure of real data.
• What distribution is being learned? Distinguish $p_{\mathrm{data}}(x)$ from the model distribution $p_\theta(x)$ .
• What does inference recover? Determine whether the system is reasoning about possible data, latent structure, or high-level features.

This framing helps clarify the role of representation, learning, and inference in an inverse-problem setting.

Takeaway

A Forward Problem predicts an output from a known input:

$$p(y|x)$$

An Inverse Problem starts from an observation and reasons about possible inputs:

$$p(x|y)$$

The hard part is that the inverse direction may not have one unique answer.

That is why inverse problems naturally lead to probabilistic modeling. The model may need to represent multiple possible solutions, account for complex high-dimensional data structure, and infer latent structure that is not directly visible in the observation.

In generative modeling, this connects to three core tasks: representing complex probability distributions, learning them from observed samples, and using the learned structure for inference.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/inverse-problems-en/