Filter methods score each column against the target on its own, before any model exists. They are fast, they are model-agnostic, and they share one specific failure that is worth understanding before you trust a ranking to decide what gets built.
Why screen before fitting at all
Three legitimate reasons, and one bad one.
The scale reason: with 20,000 candidate columns, a wrapper method that fits a model per subset is not going to finish. A filter is a single pass and gives you a shortlist to spend real compute on.
The leakage reason: a filter ranking is the first place a leaked column announces itself, because a feature that already knows the answer sits at the top with a suspicious margin over everything else. That screen is the first step in finding a feature that already knows the answer.
The cost reason: some columns are expensive to compute or to acquire at serving time. Knowing that a column has essentially no marginal association with the target is grounds for not building the pipeline that produces it.
The bad reason is using a filter ranking to select the final feature set. Filters score columns one at a time and models use them together, and that gap is not small — it is the subject of the last section here.
Correlation, and its blind spot
Pearson correlation measures linear association only. It is exactly zero for a perfectly deterministic non-monotone relationship: if y = x² and x is symmetric around zero, the correlation is 0 while the relationship is total. Spearman fixes the nonlinear-but-monotone case by correlating ranks, so it catches an exponential or a logarithm, and it is still zero on the parabola.
For a binary target and a continuous feature, the point-biserial correlation is what you want, and it is algebraically identical to running Pearson with the target coded 0/1. For a categorical feature against a binary target, correlation does not apply; use a chi-squared test of independence or mutual information.
Correlation also says nothing about how much of the target a feature explains once you have the others. Two columns each correlated 0.4 with the target may be carrying the same 0.4, or two different ones, and the statistic is identical in both cases. That is the difference between a marginal and a partial association, and no univariate screen can report the second.
The practical caveat is that correlation is not robust. A handful of extreme values can create or destroy one. Compute Spearman alongside Pearson and treat a large gap between them as a flag to look at the scatterplot, not as a result.
Mutual information
Mutual information measures how much knowing the feature reduces uncertainty about the target, in nats or bits. It is zero if and only if the two are statistically independent, and it makes no assumption about the functional form — it catches the parabola, the U shape, and the “high in two disconnected ranges” pattern that both correlations miss.
The price is estimation. Mutual information between a continuous feature and a discrete target has to be estimated from a finite sample, and scikit-learn’s mutual_info_classif uses a nearest-neighbour estimator based on the work of Kraskov and colleagues, in the variant Ross published for the mixed continuous-discrete case. Its n_neighbors parameter defaults to 3; a small value gives a low-bias, high-variance estimate and a larger one smooths it. The scikit-learn documentation for mutual_info_classif names the estimators and the papers.
Two consequences of it being an estimate. It is stochastic — pass random_state or the ranking moves between runs, because the estimator adds small noise to break ties in continuous data. And it is biased upward for high-cardinality discrete features: a column with a unique value per row has maximal mutual information with anything, which is how an id column tops a ranking it has no business being in.
The three ranked side by side
Four features against a binary target. linear_signal is a plain monotone predictor. quadratic_signal predicts strongly but symmetrically, so it is high risk at both extremes. noise is nothing. row_id is a unique integer per row.
import numpy as np, pandas as pd
from scipy.stats import pearsonr, spearmanr
from sklearn.feature_selection import mutual_info_classif, f_classif
rng = np.random.default_rng(0)
n = 20_000
linear = rng.normal(size=n)
quad = rng.normal(size=n)
noise = rng.normal(size=n)
row_id = np.arange(n)
logit = 0.9 * linear + 1.4 * (quad ** 2 - 1)
y = rng.binomial(1, 1 / (1 + np.exp(-logit)))
X = pd.DataFrame({
"linear_signal": linear,
"quadratic_signal": quad,
"noise": noise,
"row_id": row_id,
})
rank = pd.DataFrame({
"pearson": [abs(pearsonr(X[c], y)[0]) for c in X],
"spearman": [abs(spearmanr(X[c], y)[0]) for c in X],
"f_stat": f_classif(X, y)[0],
"mutual_info": mutual_info_classif(X, y, random_state=0),
}, index=X.columns)
print(rank.round(4))
Read the table by column, not by row. Pearson and Spearman put quadratic_signal near zero — indistinguishable from noise — because its association is symmetric and both statistics measure direction. The F statistic from f_classif is testing whether the feature’s mean differs between classes, so it fails the same way for the same reason. Mutual information ranks it above linear_signal, which is the correct answer given the coefficients used to generate the target.
And row_id is the cautionary column. Its correlations are near zero, correctly, but its mutual information estimate is inflated by cardinality. Drop identifier-like columns before ranking, or the screen you built to find leakage will manufacture some.
What every univariate screen misses
All of the above score one column against the target in isolation. Two things live outside that frame.
Pure interactions. Let y = a XOR b for two balanced binary features. Each of a and b is exactly independent of y: correlation zero, mutual information zero, F statistic zero. Together they determine it completely. Every filter method ranks both last, and a model given both gets perfect accuracy. This is not a contrived edge case — “this promotion works for this segment and backfires for that one” is the same shape.
Redundancy between high scorers. Take the top 50 by mutual information and you may have taken forty copies of one signal. A filter ranking says nothing about overlap between features, which is what collinearity diagnostics are for. Minimum-redundancy-maximum-relevance approaches exist precisely to penalise a candidate for its similarity to what has already been selected.
So the honest use of a filter ranking is as a triage and a leak alarm, not as a selection method. When the goal is genuinely to choose a feature subset, a method that evaluates features jointly — recursive elimination, permutation importance on a fitted model, or an L1 penalty — is the right instrument, and automated feature selection covers the trade-offs between them.
Top comments (0)