Machine learning often starts with a simple question: Can we predict a number from data?
That’s where regression comes in. It’s one of the most fundamental techniques in data science, and understanding it opens the door to more advanced models.
What is Regression?
Regression is about predicting a continuous target variable (like salary, house price, or temperature) based on one or more input features.
Simple Linear Regression: Predicts using one feature.
Example: Predicting house price from square footage.Multiple Linear Regression: Uses several features.
Example: Predicting salary from years of experience, education level, and certifications.
The general form looks like this:
General form of a multiple linear regression model:
y = β0 + β1*x1 + β2*x2 + ... + βn*xn + ε
Where:
y - target (the value we want to predict)
x1..xn - features (input variables)
β - coefficients (weights learned by the model)
ε - error term (the part the model cannot explain)
Why Regularization?
Regression models can sometimes overfit — they learn the training data too well, including noise, and fail to generalize to new data.
This happens especially when you have many features or multicollinearity (features highly correlated with each other).
Regularization is a technique to prevent overfitting by adding a penalty to the size of the coefficients.
Types of Regularization
1. Ridge Regression
- Adds a penalty proportional to the square of coefficients.
- Encourages smaller, more stable coefficients.
- Formula:
Ridge Regression Loss Function:
Loss = RSS + λ * Σ(β_j^2)
Where:
RSS - Residual Sum of Squares (error between predictions and actual values)
λ - Regularization parameter (controls penalty strength)
β_j - Coefficients of the regression model
2. Lasso Regression
- Adds a penalty proportional to the absolute value of coefficients.
- Can shrink some coefficients to zero, effectively performing feature selection.
- Formula:
Lasso Regression Loss Function:
Loss = RSS + λ * Σ|β_j|
Where:
RSS - Residual Sum of Squares (error between predictions and actual values)
λ - Regularization parameter (controls penalty strength)
β_j - Coefficients of the regression model
Note:
The absolute value penalty (|β_j|) can shrink some coefficients exactly to zero.
- This makes Lasso useful for feature selection as well as preventing overfitting.
3. Elastic Net
- Combines Ridge and Lasso penalties.
- Useful when you want both stability and feature selection.
Choosing Between Them
- Use Ridge when you have many correlated features.
- Use Lasso when you suspect only a few features are truly important.
- Use Elastic Net when you want a balance of both.
Practical Example in Python
python
from sklearn.linear_model import LinearRegression, Ridge, Lasso
# Simple regression
lin_reg = LinearRegression().fit(X_train, y_train)
# Ridge regression
ridge_reg = Ridge(alpha=1.0).fit(X_train, y_train)
# Lasso regression
lasso_reg = Lasso(alpha=0.1).fit(X_train, y_train)
Top comments (1)
Really nice work