DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Elastic Net: why blending L1 and L2 beats lasso on correlated features

#ai #machinelearning #datascience #python

Regularization is usually taught as a two-horse race: ridge or lasso, L2 or L1. But there is a case that trips up both, and it is extremely common in real data — features that are correlated with each other. I built an interactive demo where coordinate descent recomputes eight coefficients live as you drag two sliders, and it makes the failure of pure lasso, and the fix, impossible to miss.

The setup that breaks lasso

The demo dataset has eight standardized features. Two of them (x₁, x₂) are correlated twins — about 0.98 correlation — that jointly carry one real signal. Two more (x₃, x₄) are independent real signal. The last four are pure noise, true coefficient zero. A good model should keep the signal, share the twins fairly, and zero the noise.

Watch what each method does:

  • Ordinary least squares gives all eight a weight, noise included, and splits the twins more or less at random.
  • Ridge (L2) shrinks everything smoothly and shares weight fairly between the twins, but never drives anything to exactly zero — no feature selection.
  • Lasso (L1) zeros features for a nice sparse model, but faced with the two correlated twins it keeps one at random and discards the other. That pick flips with tiny data changes — an unstable model that changes its story when the data wiggles.
  • Elastic Net adds both penalties: the L1 part zeros the noise (sparsity), and the L2 part keeps the correlated pair together (the grouping effect).

The two knobs

The objective is least-squares loss plus a blended penalty:

minimize over w:
  (1 / 2n) * ||y - Xw||^2                      <- fit
  + alpha * l1_ratio       * ||w||_1           <- L1: sparsity
  + 0.5 * alpha * (1 - l1_ratio) * ||w||_2^2   <- L2: shrink + grouping
Enter fullscreen mode Exit fullscreen mode

alpha scales the whole penalty (0 = OLS); l1_ratio mixes it. Set l1_ratio=1 and you have lasso; set it to 0 and you have ridge. Those are the two sliders in the demo — and the endpoints really do reproduce plain lasso and plain ridge exactly.

One thing that catches people: standardize first. L1 and L2 penalize the size of each coefficient, so a feature in millimetres gets a bigger coefficient and a harsher penalty than the same feature in metres. Put everything on a common scale inside a pipeline so the scaler refits per CV fold and never leaks.

Soft-thresholding is what makes the zeros

Elastic Net has no closed form because of the L1 kink, so the solver cycles one coefficient at a time — coordinate descent. For each feature it computes a partial residual correlation, applies soft-thresholding, then divides by the L2 term. That soft-threshold operator is the thing that manufactures exact zeros:

def soft_threshold(x, g):          # the L1 operator -> makes exact zeros
    return np.sign(x) * max(abs(x) - g, 0.0)

def elastic_net(X, y, alpha, l1_ratio, iters=500):
    X = (X - X.mean(0)) / X.std(0)          # standardize
    y = y - y.mean()
    n, p = X.shape
    w = np.zeros(p); r = y - X @ w          # residual
    thr   = alpha * l1_ratio                # L1 shrink amount
    denom = 1.0 + alpha * (1 - l1_ratio)    # L2 divisor (col var = 1)
    for _ in range(iters):
        for j in range(p):
            rho = (X[:, j] @ (r + X[:, j] * w[j])) / n   # add j back in
            new = soft_threshold(rho, thr) / denom
            r += X[:, j] * (w[j] - new)     # keep residual in sync
            w[j] = new
    return w                                # zeros = dropped features
Enter fullscreen mode Exit fullscreen mode

This is the same algorithm scikit-learn uses, and the same one running live in the browser demo — every bar you drag is a genuine solve, no libraries.

Reading the grouping effect

The demo tracks a "twin gap" — |w₁ − w₂|. Under lasso it is large (one twin gets everything, the other is zeroed); under ridge and elastic net it is small (the shared signal is split fairly). Small gap means a stable model. On the held-out test set here, elastic net beats pure lasso precisely because it does not throw away half a correlated pair.

When to reach for it

Elastic Net is not always the answer. Reach for it when you have many, correlated features and want sparsity you can trust. If nothing is correlated, plain lasso is simpler. If you want to keep every feature and only tame their size, ridge is enough. And do not hand-tune the knobs — let ElasticNetCV sweep a grid of l1_ratio values and an automatic alpha path with k-fold CV and keep the best.

Bagging cuts variance, boosting cuts bias, and elastic net cuts the specific pain of correlated features that lasso handles badly.

Drag the two sliders and watch the coefficients, the regularization path, and the twin gap recompute in real time:
https://dev48v.infy.uk/ml/day34-elastic-net.html

Top comments (0)