DEV Community: mohit modi

Understanding Principal Component Analysis

mohit modi — Tue, 01 Sep 2026 03:52:54 +0000

mohit modi

Sep 1

What PCA is actually doing to your data

#discuss #machinelearning #datascience #learning

5 min read

Understanding Principal Component Analysis

mohit modi — Tue, 01 Sep 2026 03:52:54 +0000

mohit modi

Sep 1

What PCA is actually doing to your data

#discuss #machinelearning #datascience #learning

5 min read

What PCA is actually doing to your data

mohit modi — Tue, 01 Sep 2026 03:31:17 +0000

Most explanations of PCA stop at "it reduces dimensions." That is true, and it is roughly as useful as saying a regression "fits a line." It tells you what happens without telling you what the method is doing or when it will fail you.

Here is the whole idea in one sentence: PCA finds the directions your data varies in most, by taking the eigenvectors of its covariance matrix.

Everything else is detail. But that sentence only helps if you know what a covariance matrix is and what an eigenvector does, so let's build it up — and if the prior question is why you'd reduce dimensions at all, start there.

Start with the covariance matrix

If you centre your data — subtract the mean from every column — the covariance matrix is:

Sigma = 1/(n-1).X.X_t

For p features you get a pxp matrix. The diagonal holds each feature's variance. Everything off the diagonal holds the covariance between a pair of features: how much they move together.

That matrix is the entire input to PCA. Not the raw data — the covariance structure. Which means PCA only ever sees how your features vary and co-vary, and is blind to anything else about them.

What the eigenvectors are doing

For a square matrix Sigma, an eigenvector v is a direction that the matrix does not rotate:

Sigma v = lambda v

Multiplying by Sigma only stretches v, by a factor of lambda. Every other direction gets turned.

When Sigma is a covariance matrix, those special directions have a concrete meaning: they are the axes along which the data varies independently, and lambda is how much variance lies along each one.

So sorting the eigenvectors by their eigenvalues sorts the directions of your data from "most spread out" to "least." The first eigenvector is the first principal component. It is the single direction that captures more of your data's variance than any other.

That is the whole method. Compute the covariance matrix, take its eigenvectors, order them by eigenvalue, keep the top few. The full derivation is worth walking through once.

Why the components are uncorrelated

A covariance matrix is symmetric, and symmetric matrices have orthogonal eigenvectors. Every principal component is at right angles to every other.

Orthogonal directions have zero covariance. So your new features are, by construction, completely uncorrelated with each other — a property none of your original features are likely to have.

Hold onto that. It is the source of PCA's most underrated use, which we'll come to.

Choosing how many components to keep

Each eigenvalue is the variance along its component, so the fraction of total variance a component explains is just its share of the sum:

explained variance ratio (i) = lambda(i)/ sigma of lambda(j)

Plot the cumulative version and keep however many components clear the bar you care about — 90%, 95%, whatever the application justifies. There is no correct threshold, which is worth saying plainly rather than pretending an elbow plot settles it. More on choosing the number of components.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)   # not optional — see below
pca = PCA().fit(X_scaled)

print(pca.explained_variance_ratio_.cumsum())

PCA and feature selection

This is where terminology causes real confusion, so let's be exact.

PCA is not feature selection. It is feature extraction.

Selection keeps a subset of your original columns. You start with 50 features, you end with 12 of the original 50, and you can still say "this model uses income and tenure."
Extraction builds new features out of the old ones. PC1 is not one of your columns; it is a weighted blend of all of them — something like 0.4xIncome + 0.3xtenure - 0.2xage + ..

So PCA never selects anything. It replaces your feature set with a smaller, rotated one. If someone tells you they "used PCA for feature selection," they mean one of the three things below.

1. Dimensionality reduction before modelling

The standard use. Fifty correlated features become ten components carrying 95% of the variance, and the model trains on those. It serves the same purpose as selection — fewer inputs, less overfitting, faster training — while doing something mathematically different.

2. Reading the loadings to inform real selection

Each component has loadings: the weight of every original feature within it. If one feature dominates the top components, that is evidence it carries a lot of the variance, and you might keep it in a genuine selection step.

This is legitimate but weaker than it looks. High loading means high variance contribution, not high predictive value — and those are not the same thing. See interpreting component loadings.

3. Removing multicollinearity

The one people underuse. Because the components are orthogonal, they have zero correlation with each other by construction. Regressing on principal components instead of raw features makes multicollinearity structurally impossible.

If you have ever watched a VIF climb past 10 and had to decide which of two near-identical predictors to drop, PCA sidesteps that decision entirely — it is one of the few honest answers to that violated regression assumption, rather than a coin toss between two predictors.

Advantages

No multicollinearity, guaranteed by orthogonality — not something you check for afterwards.
Fewer inputs means less overfitting and faster training, particularly when p approaches n.
Noise reduction. Low-variance components are often mostly measurement noise; dropping them can improve signal.
It keeps information that selection throws away. Dropping a column discards everything in it. PCA compresses all fifty features into ten components, so a feature's contribution survives even when it isn't individually important.
It's unsupervised, so it can run before you have labels, and it cannot leak your target into the transformation.

The costs, which are not small

You lose interpretability. PC1 is a blend of everything. You can no longer say "a year of tenure is worth £400." For anything that needs explaining to a regulator, a clinician, or a stakeholder, that alone can rule PCA out.

PCA maximises variance, not predictive power. It never sees your target. The direction your data varies in most is not necessarily the direction that predicts y — a low-variance component can easily be the one that matters, and dropping it because it explains 2% of variance can quietly cost you the model.

It only finds linear structure. Data on a curved manifold won't be captured well. That's what kernel PCA, t-SNE and UMAP are for.

The mistake almost everyone makes once

Standardise your features first.

PCA maximises variance, and variance has units. A feature measured in rupees will have a variance millions of times larger than one measured in years — so the first principal component will point almost exactly along the rupee axis, and you will have discovered nothing except which column has the biggest numbers.

Scaling puts every feature on equal footing so the components reflect structure rather than measurement units. The only time you skip it is when your features are already in the same units and their relative scales are meaningful — the rest of the preprocessing PCA expects is short but non-negotiable.

Where to go from here

PCA is the point where linear algebra stops being abstract. Eigenvectors go from a homework exercise to the thing deciding which ten of your fifty columns survive.

If any of the pieces felt shaky, fix them before you use this in anger — the covariance structure and the eigen-decomposition do all the work here, and everything else is bookkeeping:

Principal Component Analysis — the full topic, free to read
Understanding covariance — the matrix PCA actually consumes
Understanding correlation — why orthogonal means uncorrelated
Linear regression assumptions — the multicollinearity problem PCA dissolves

Correlation Is Pairwise. Multicollinearity Isn't.

mohit modi — Sat, 22 Aug 2026 04:58:52 +0000

Almost every regression project I've seen in the last decade starts the same way. Load the data, df.corr(), plot the heatmap, scan for red squares, drop one variable from every pair above 0.8. Then we move on, feeling like we've handled multicollinearity.

We haven't. We've handled the version of it that happens to show up in pairs.

The three-line example that should end the ritual

Take two independent features and construct a third as their sum:

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
x1 = rng.normal(size=10_000)
x2 = rng.normal(size=10_000)
x3 = x1 + x2

pd.DataFrame({"x1": x1, "x2": x2, "x3": x3}).corr().round(3)

The correlation matrix shows roughly 0.7 between x1 and x3, roughly 0.7 between x2 and x3, and approximately 0 between x1 and x2.

Every pair passes a 0.8 threshold. Every pair passes a 0.75 threshold. And yet x3 is perfectly determined by the other two. The design matrix is singular. There is no unique solution for the coefficients — your software will either return garbage or silently regularise its way out of the problem.

The heatmap saw nothing, because there was nothing to see pairwise. Correlation is a two-variable statistic. Multicollinearity is a property of the entire design matrix. Those are different questions, and the heatmap only answers the first one.

This isn't a contrived edge case. It's every ratio you've engineered, every "total" column that's the sum of its parts, every set of channel spends that add up to a budget, every one-hot encoding you forgot to drop a level from. In practice, near-dependence across three or four features is far more common than a single scary pair — and it is exactly what the heatmap is blind to.

What VIF actually measures

The Variance Inflation Factor asks the right question. For each feature, regress it on all the other features and take:

VIF = 1/(1 - R^2)

That R^2 is multivariate by construction. In the example above, regressing x3 on x1 and x2 gives R^2 = 1, so the VIF is infinite. The heatmap said fine; the VIF says the model is unidentifiable.

The name is literal, which is the part people miss. A VIF of 10 means the variance of that coefficient estimate is ten times what it would be if the feature were orthogonal to the others — so the standard error is sqrt(10) ~ 3.2 times larger. That's the whole mechanism. Your coefficient isn't wrong on average; it's just so unstable that it's useless. Resample the data and it swings, sometimes across zero.

from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm

X = sm.add_constant(pd.DataFrame({"x1": x1, "x2": x2, "x3": x3}))
[variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

A note on the thresholds, in the spirit of this whole post: VIF > 5 and VIF > 10 are conventions, not findings. They're no more principled than the 0.8 on the heatmap. Use them to rank features by severity, not as a pass/fail gate.

The part nobody says out loud: this is an inference problem, not a prediction problem

If you only care about predictions, multicollinearity mostly doesn't matter. Correlated features don't bias the fitted values, don't inflate test error, and don't need to be removed. A gradient boosting model on collinear features will predict perfectly well.

Multicollinearity only hurts when you need to read the coefficients — when the model is going to answer "how much should we spend on this channel" or "what happens if we raise price by 5%." That's where unstable coefficients become bad decisions.

So the first question isn't "which features are correlated." It's "am I predicting or explaining?" Half the arguments about multicollinearity are people answering different questions and not realising it.

Why I sometimes drop a variable that correlates strongly with the target

Building Marketing Mix Models, I started the way everyone does: rank variables by correlation with sales, keep the strong ones. It worked, in the sense that the model fit beautifully. It also quietly pushed every media driver out of the model.

The pattern was consistent. Distribution, seasonality, holiday flags — these sat at 0.8 and above. The media variables I actually needed to report on sat between 0.4 and 0.6. Ranked on correlation, the media never stood a chance.

The problem is that in any real market, promotions, media and seasonal demand all move together. December sales are up. So is TV spend, so is display, so is the holiday flag. Every one of those variables is competing to explain the same peak, and the one that wins the competition is the one most tightly coupled to it — which is almost always the variable you can't control.

So I dropped variables with high correlation to the target, deliberately, and accepted a worse fit. Because a marketing mix model whose coefficients say "sales are seasonal" is not a model. Nobody can act on it. The entire point of the exercise was the media coefficients, and the high-correlation variables were crowding out the only part anyone was going to read.

This is the inference-versus-prediction distinction from earlier, arriving with a budget attached. If I were forecasting sales, keeping seasonality and dropping media would be the right call — better fit, better forecast, done. But the model existed to answer "what should we spend on this channel next quarter," and that question is answered by coefficients, not by fit.

What's left is a judgement call that no diagnostic makes for you: balancing model fit against business expectation and explainability. The correlation matrix can't help here. It ranks variables by their relationship to the target and has nothing at all to say about which relationships you need to be reportable.

The general principle I've settled on: feature selection is a question about what the model is for, not just what's in the data. A variable that improves fit but destabilises the coefficient you're going to act on is a bad trade, regardless of what the heatmap says.

The mirror-image mistake: dropping features with low target correlation

While we're dismantling this ritual, the other half of it deserves the same treatment. People keep the features that correlate strongly with the target and cut the ones that don't.

But a feature can have near-zero correlation with the target and still be one of the most important variables in the model. These are suppressor variables: they don't predict the target directly, they explain away noise in another predictor, sharpening its signal. Cut it on a univariate screen and the model gets worse.

The pattern is the same error in both directions — using a pairwise statistic to make a multivariate decision.

Where eigenvalues come in

If you want the honest, single-number version of "is my design matrix in trouble," it's the condition number: the ratio of the largest to the smallest singular value of your scaled feature matrix. Values above ~30 are the usual flag.

That's not a coincidence — it's the same underlying idea. Near-dependence among your features means the matrix X^T X has eigenvalues close to zero, and inverting something with a near-zero eigenvalue is what blows the coefficient variances up in the first place. VIF and the condition number are two views of one geometric fact: your features don't span the space you think they do.

What I'd actually do instead

Decide first whether you're predicting or explaining. If predicting, most of this is moot.
Skip the heatmap as a decision tool. Keep it for exploration — it's genuinely useful for spotting data errors and leakage.
Compute VIF on the full feature set. Rank, don't threshold.
Check the condition number as a whole-matrix sanity check.
For anything with a high VIF, ask why it's dependent. Engineered ratio? Sum of components? Dummy trap? The fix is usually structural, not deletion.
Only then decide what to drop — and decide it against what the model is for.

None of this is more work than making the heatmap. It's just less satisfying, because there's no red square to point at.

Want the underlying concepts, properly? Explore Correlation & Covariance, Linear Regression, and Eigenvalues in the Bitelrn Open Library — no sign-up.