In machine learning tasks, we often face the challenge of overfitting. When a model is too complex or uses very little data, it can overfit on the data. In order to prevent this we have a few methods, such as regularizing the model with added penalties on its weights , or dimensionality reduction. Among these, we’ll be looking at regularization in this post.
Regularization is the process of preventing overfitting of a model by adding a mathematical penalty or constraint during the training process.
Before we discuss regularization, it is important to note that we always standardize the data before performing regularization, fitting the scaler on the training set only. This prevents the model from unfairly penalizing variables that are on a larger scale.
Lasso or L1 regularization , which stands for Least Absolute Shrinkage and Selection Operator , penalizes the absolute weights of each feature and shrinks some weights to exactly 0 , thereby automatically dropping irrelevant features in a dataset that may have many features.
Ridge or L2 regularization doesn’t shrink any feature weights to exactly 0, unlike Lasso. It penalizes the squared weights, which shrinks all the weights towards zero and retains all the features. When features are highly correlated, it shares the weight between them instead of dropping any.
Elastic Net is a blend of both Lasso and Ridge regularization methods. It uses a ratio of how much of Lasso and Ridge to use during training, which is defined by the user and is usually tuned with cross-validation. So not only do we get to drop the features with the weakest weights, but we also get to keep groups of highly correlated features together.
Lasso (L1-Regularization) on NYC Airbnb price prediction:
Predicting the per-night price of a property using listing details, availability, review metrics and location features such as neighbourhood_group, neighbourhood and room_type.
When we try to use this dataset and one-hot encode a column like neighbourhood (which has 200+ distinct areas ), we end up creating too many binary dummy columns. Many of these neighbourhoods either have very few listings or aren’t different enough to shift the price away from the baseline average of their neighbourhood_group.
In such a scenario, where you suspect that some features are irrelevant and only a few are strongly related to the target , we use Lasso. This means that for listings in a rare neighbourhood, Lasso automatically drops their dummies and keeps only the locations and room types that genuinely drive price differences.
Ridge (L2-penalised logistic regression) for heart disease classification:
Heart disease classification, where the target column is a binary class, with 13 clinical features like age, restingBloodPressure, cholesterol, maximumHeartRate, oldpeak and chestPain, across 303 patient records.
Unlike in the Airbnb dataset, here every single feature is a medical indicator and none of them are considered random noise. However, this dataset has just about 300 patient records , and some physiological markers (such as age, maximumHeartRate and exerciseInducedAngina) clinically overlap. Without regularization, a standard logistic regression model can assign unstable weights to individual features based on a few extreme patient records.
Ridge preserves the collective diagnostic signal across all 13 clinical variables while preventing any single biomarker from dominating the prediction on a small sample.
Elastic Net for housing price prediction
Predicting price using structural dimensions (area, bedrooms, bathrooms, stories, parking) alongside categorical amenities (mainroad, guestroom, basement, hotwaterheating, airconditioning, prefarea, furnishingstatus).
This dataset combines two modelling challenges: grouped multicollinearity and sparse amenities. Larger houses tend to have more area, more bedrooms and bathrooms, and more stories (floors). A pure Lasso tends to pick just one significant feature from a group of correlated features (like keeping area and dropping bedrooms and stories), risking throwing away useful information. Features like hotwaterheating (a binary feature where the majority of records are ‘no’) or furnishingstatus may add little value once major features like area and airconditioning are considered.
Elastic Net’s Ridge component groups the correlated features (area, bedrooms, bathrooms and stories) so they share weights smoothly rather than one being picked over the others, while the Lasso component simultaneously drops any weak or redundant feature weights to 0.
Summary table to compare Lasso, Ridge and Elastic Net regularization
Along with these regularization methods, what other methods do you use to prevent overfitting?
This was Day 7 of my 30 Days of Data Science writing challenge. Writing this article was a good challenge to recollect where I’ve used these regularization methods before and understand why they were used.
If you’re a beginner in ML and would like to learn more about regularization and other ML concepts, I’d recommend this book: An Introduction to Statistical Learning (ISL). It has free online editions in Python (ISLP) and R (ISLR), with its own packages and datasets for practice.


Top comments (0)