Discriminative vs Generative Models: What Are You Actually Learning?

When a supervised model maps an input to an output, it is easy to assume that every model is learning essentially the same relationship. In practice, two models can solve the same prediction problem while modeling very different probability relationships.

A Discriminative Model learns the relationship needed to predict the output directly. A Generative Model instead models how the input is distributed for each possible output and uses that probability structure to infer the prediction.

The most useful question is therefore not simply, ""What does the model predict?"" but ""What probability relationship does it learn directly?""

Start with the supervised learning objective

At a high level, supervised learning asks us to learn a mapping from an observed input $x$ to a target output $y$ :

$$f:x\mapsto y$$

Both discriminative and generative approaches aim to solve this mapping problem. The difference is the probability relationship they model in order to reach the prediction.

A compact mental model is:

Discriminative
x
│
├── learn p(y | x)
│
└── choose y directly

Generative
y
│
├── learn p(x | y)
├── combine with p(y)
│
└── infer the best y for x

The prediction target is the same, but the learned structure is different.

Discriminative Models: learn the conditional relationship directly

A Discriminative Model focuses on the conditional probability of an output given an input:

$$p(y\mid x)$$

For classification, prediction can be written as:

$$f(x)=\arg\max_y p(y\mid x)$$

The calculation flow is direct:

input x
   ↓
estimate p(y | x)
   ↓
compare candidate outputs
   ↓
select the most probable y

The model focuses on the relationship needed to distinguish possible outputs from the observed input. In the input space, this leads naturally to the idea of a decision boundary: regions of the space are associated with different outputs according to the learned predictive relationship.

Support Vector Machine is a representative discriminative example in this comparison. Its role here is best understood through the decision structure it learns between outputs rather than as a model of the overall probability distribution of the data.

Generative Models: model the input distribution for each output

A Generative Model approaches the same supervised problem from the opposite conditional direction. Instead of directly modeling $p(y\mid x)$ , it models:

$$p(x\mid y)$$

This tells us how likely an input $x$ is under a particular output $y$ . The model then combines this conditional probability with the prior probability of the output, $p(y)$ .

Prediction can therefore be written as:

$$f(x)=\arg\max_y p(x\mid y)p(y)$$

The calculation flow becomes:

candidate y
   ↓
evaluate p(x | y)
   ↓
combine with p(y)
   ↓
compare p(x | y)p(y)
   ↓
select the best y

Bayes rule shows how this connects back to the conditional probability used by the discriminative view:

$$p(y\mid x)=\frac{p(x\mid y)p(y)}{p(x)}$$

For a fixed input $x$ , the denominator $p(x)$ is shared across all candidate outputs. When the goal is only to determine which output has the highest posterior probability, that common denominator does not affect the comparison.

This is why a Generative Model can use $p(x\mid y)$ together with $p(y)$ to choose the appropriate output without directly modeling $p(y\mid x)$ .

Gaussian Mixture and Bayes Nets are representative generative examples in this comparison. The important distinction is that this approach models the probability distribution describing how inputs relate to possible outputs.

The comparison that matters

It is tempting to reduce the distinction to ""classification versus generation,"" but that misses the central point. In this supervised-learning setting, both approaches can predict $y$ from $x$ .

What differs is what they learn directly.

	Discriminative Model	Generative Model
Prediction goal	Predict $y$ from $x$	Predict $y$ from $x$
Directly modeled relationship	$p(y\mid x)$	$p(x\mid y)$
Additional probability used	—	$p(y)$
Prediction rule	$\arg\max_y p(y\mid x)$	$\arg\max_y p(x\mid y)p(y)$
Useful structural view	Decision boundary	Probability distribution
Representative example	Support Vector Machine	Gaussian Mixture, Bayes Nets

For developers, this distinction is useful when reading a model definition or implementation. If the model directly represents the relationship from $x$ to $y$ , the focus is discriminative. If it models how $x$ behaves under each $y$ and combines that with $p(y)$ , the focus is generative.

Takeaway

Discriminative and generative models can solve the same supervised prediction problem through different probabilistic views.

A Discriminative Model directly models:

$$p(y\mid x)$$

A Generative Model models:

$$p(x\mid y)$$

and combines it with $p(y)$ to infer the appropriate output.

The key distinction is not simply classification vs generation. It is which probability relationship the model learns directly and uses for prediction.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/discriminative-vs-generative-models-en/