DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

LDA vs PCA: the projection that separates your classes is rarely the one with the most variance

#ai #machinelearning #datascience #python

PCA is the reflex when someone says "reduce the dimensions." But PCA never looks at your labels, and there is an extremely common situation where that blindness makes it point in exactly the wrong direction. I built an interactive demo where you drag two clouds of points around and watch two projection axes — PCA's and LDA's — recompute live, and it makes the failure, and the fix, impossible to miss.

The setup that breaks PCA

Picture two elongated classes, each stretched into a long cigar shape, sitting side by side. The direction of biggest total spread runs along the length of those cigars — and that direction has nothing to do with telling the classes apart. PCA, which maximises variance, happily picks it. Collapse the points onto that axis and the two classes pile into one blurred hump.

LDA (Linear Discriminant Analysis) uses the labels. It looks for the axis where the class means are far apart relative to the spread inside each class — which here is the direction cutting across the cigars, nearly perpendicular to what PCA chose.

The Fisher criterion

LDA maximises a ratio of two "scatters":

J(w) = (wᵀ S_b w) / (wᵀ S_w w)
       between-class scatter  /  within-class scatter
Enter fullscreen mode Exit fullscreen mode

S_b is the signal — how far the class means sit from the grand mean. S_w is the noise — how spread each class is around its own mean. Maximising signal-over-noise is the whole idea. The two scatter matrices are the from-scratch heart of it:

Sw = np.zeros((d, d))                       # within-class scatter
for c in classes:
    Xc = X[y == c] - means[c]
    Sw += Xc.T @ Xc                         # sum of (x-mu_c)(x-mu_c)^T

Sb = np.zeros((d, d))                       # between-class scatter
for c in classes:
    n_c  = (y == c).sum()
    diff = (means[c] - overall).reshape(d, 1)
    Sb  += n_c * (diff @ diff.T)            # n_c (mu_c - mu)(mu_c - mu)^T
Enter fullscreen mode Exit fullscreen mode

Solving for the axis

Maximising the Fisher criterion is a generalized eigenvalue problem — the best axes are the top eigenvectors of S_w⁻¹ S_b. For two classes it collapses to a tidy closed form, and that one-liner is exactly what runs live in the demo:

# two-class closed form (what the page draws)
w = np.linalg.inv(Sw) @ (means[1] - means[0])
w /= np.linalg.norm(w)

# general multi-class: top eigenvectors of Sw^-1 Sb
eigvals, eigvecs = np.linalg.eig(np.linalg.inv(Sw) @ Sb)
order = np.argsort(eigvals.real)[::-1]
W = eigvecs[:, order[:n_components]].real
Enter fullscreen mode Exit fullscreen mode

You classify a new point by projecting it and comparing to the midpoint of the projected class means — the decision threshold you see in the demo.

LDA is both a classifier and a reducer

In scikit-learn, LDA is a fast, closed-form classifier and a supervised dimensionality reducer. As a reducer it is the supervised cousin of PCA: same "keep a few axes" idea, but it keeps the axes that discriminate rather than the ones that merely have the most variance.

X_lda = LinearDiscriminantAnalysis(n_components=1).fit_transform(X, y)  # uses y
X_pca = PCA(n_components=1).fit_transform(X)                            # ignores y
# On elongated classes X_lda separates the labels; X_pca often does not.
Enter fullscreen mode Exit fullscreen mode

One thing to remember: transform gives you at most C − 1 axes for C classes, because that is all the between-class scatter can span.

Know the assumptions

LDA assumes each class is Gaussian with a shared covariance, which is why its decision boundary is linear. That is a feature when it holds and a trap when it does not:

  • LDA — Gaussian classes, shared covariance → linear boundary, supervised.
  • QDA — per-class covariance → curved boundary, when class shapes genuinely differ.
  • PCA — no labels, keep variance → unsupervised compression.

And when S_w is near-singular because you have few samples, don't invert it raw — reach for shrinkage (solver="lsqr", shrinkage="auto").

The whole point lives in the gap between the two accuracies. When within-class elongation dwarfs class separation, PCA veers off along the stretch and its 1-D projection scores badly, while LDA divides out that stretch and lands on the separating axis. Given even a hopeless, overlapping problem, LDA still finds the least-bad direction — which is more than variance alone can promise.

Drag the class separation and elongation sliders and watch both axes, the projections, and the accuracies recompute in real time:
https://dev48v.infy.uk/ml/day35-lda.html

Top comments (0)