Part 1: Linear Regression — What the Model Actually Does
Linear regression is one of the simplest predictive models in machine learning. Its job is straightforward: learn the relationship between a set of input features and a continuous target by fitting the best possible line through the data. If you have many features, that "line" becomes a hyperplane, but the underlying idea stays the same.
For multiple features, the model can be written as:
ŷ = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ
Where ŷ is the predicted value, β₀ is the intercept, and β₁ through βₙ are coefficients that quantify how much each feature contributes to the prediction.
How Coefficients Are Found: Ordinary Least Squares
The model finds the coefficients that minimise the Residual Sum of Squares (RSS):
RSS = Σ(yᵢ - ŷᵢ)²
Each term (yᵢ - ŷᵢ) is the residual — the difference between the actual value and the model's prediction for observation i. Squaring serves two purposes: it prevents positive and negative errors from cancelling each other out, and it penalises large errors more heavily than small ones.
This approach is called Ordinary Least Squares (OLS). It has a closed-form analytical solution, meaning the optimal coefficients can be calculated directly rather than estimated iteratively. This makes OLS computationally efficient and, importantly, it produces the Best Linear Unbiased Estimator (BLUE) when the five classical regression assumptions hold.
The Five Assumptions
Linearity. The relationship between predictors and target must be approximately linear. A model cannot capture a curve by drawing a straight line.
Independence of errors. Residuals for different observations should not be correlated. This assumption is most frequently violated in time series data, where prediction errors in one period predict errors in the next.
No multicollinearity. Predictor variables should not be highly correlated with each other. When they are, the model cannot reliably separate their individual effects. Coefficients become unstable — small changes in the data produce wildly different estimates.
No influential outliers. A single extreme observation can pull the regression line toward it, distorting coefficients for all other rows. The model minimises squared errors, so large errors have disproportionate influence.
Normality of residuals. Errors should be approximately normally distributed. This matters primarily for hypothesis testing on coefficients — calculating p-values and confidence intervals. It is less critical for prediction.
Detecting Multicollinearity: VIF
The Variance Inflation Factor (VIF) quantifies how much the variance of a coefficient is inflated due to multicollinearity:
VIF = 1 / (1 - R²ⱼ)
Where R²ⱼ is the R² obtained when predicting feature j from all other features. If feature j can be predicted well from its peers, it is nearly redundant, and VIF will be high. A VIF above 10 is a serious signal; above 5 warrants attention.
from statsmodels.stats.outliers_influence import variance_inflation_factor
X_with_constant = X_numeric.copy()
X_with_constant['constant'] = 1
vif_data["VIF"] = [variance_inflation_factor(X_with_constant.values, i)
for i in range(len(X_numeric.columns))]
Part 2: Evaluating Model Performance
Training a model is only half the work. The other half is understanding how well it performs — and, critically, whether that performance will hold on data the model has never seen.
The Four Core Metrics
Mean Absolute Error (MAE) is the average of the absolute prediction errors. It is easy to interpret — an MAE of 5,000 means predictions are off by approximately 5,000 units on average. It treats all errors equally.
Mean Squared Error (MSE) squares the errors before averaging. This penalises large errors disproportionately. The units are squared, making direct interpretation difficult.
Root Mean Squared Error (RMSE) is the square root of MSE, returning the metric to the original units. RMSE will always be equal to or larger than MAE. When the gap is large, outlier predictions exist. RMSE is appropriate when large errors are especially costly.
R² (Coefficient of Determination) measures the proportion of variance in the target variable explained by the model:
R² = 1 - (SS_residual / SS_total)
An R² of 0.96 means the model explains 96% of the variation in the target. The remaining 4% is attributable to factors outside the model or genuine randomness.
A critical limitation of R²: adding any predictor — even a completely irrelevant one — will never decrease R². A model with shoe size as a feature will have equal or higher R² than one without it. This makes R² unreliable for comparing models with different numbers of features.
Adjusted R² corrects for this by applying a penalty for adding predictors that do not genuinely improve the model. It decreases when an irrelevant predictor is added and increases when a useful one is. Use Adjusted R² whenever comparing models with different feature counts.
The Train-Test Split
Performance metrics calculated on training data are meaningless as evaluation tools — the model has already seen that data and optimised itself against it. True evaluation requires testing on data the model has never encountered.
The standard approach is to hold out a portion of data before training:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
test_size=0.2 reserves 20% for testing. random_state=42 makes the split reproducible. You train on X_train and y_train, then evaluate predictions on X_test against y_test — data the model never touched during training.
Part 3: The Overfitting Problem
On a salary prediction project using the job_salary_prediction_dataset, my plain linear regression model produced:
- Training R²: 0.9634
- Test R²: 0.9635
- Gap: 0.0001
This is a well-behaved model. The near-identical scores on training and test data indicate the model learned genuine patterns rather than memorising the training data's noise.
But this result is not always guaranteed. The gap between training and test performance tells the real story.
Overfitting Defined
Overfitting occurs when a model learns the training data too specifically — including its noise, outliers, and dataset-specific quirks — and fails to generalise to new observations. It manifests as high training performance and substantially lower test performance.
At the coefficient level, overfitting produces very large values. The model assigns extreme weights to certain features to minimise training error, at the expense of generalisation.
Underfitting is the opposite failure: the model is too simple to capture the underlying pattern and performs poorly on both training and test data.
The tension between the two is formalised as the bias-variance tradeoff:
- High bias (underfitting): the model consistently misses the true pattern. Rigid, makes systematic errors.
- High variance (overfitting): the model is too sensitive to the specific training data. Small changes in training data produce large changes in predictions.
Reducing bias typically increases variance, and vice versa. The goal is to find the model complexity that balances both.
Part 4: Regularization — Adding a Penalty
Regularization addresses overfitting by modifying the cost function. Instead of minimising only the prediction error, the model simultaneously minimises prediction error and a penalty on the size of the coefficients. Large coefficients are discouraged, producing a simpler, more generalisable model.
Three regularization techniques are widely used in practice.
Lasso Regression (L1)
Cost = RSS + α × Σ|βⱼ|
Lasso adds a penalty proportional to the sum of absolute values of the coefficients. The parameter α controls the strength of the penalty: α = 0 reduces to plain linear regression, and increasing α progressively shrinks coefficients.
The critical property of Lasso — a consequence of the geometry of the L1 penalty — is that it can push coefficients to exactly zero. When a coefficient reaches zero, that feature is removed from the model entirely. Lasso performs automatic feature selection.
This makes Lasso valuable in high-dimensional settings where many features are suspected to be irrelevant. In the salary prediction project, running Lasso with α = 1 zeroed out industry_Education, identifying it as a non-contributor to salary prediction. Increasing α to 1000 zeroed out 23 features — but at the cost of dropping the R² from 0.9635 to 0.7719. This is the bias-variance tradeoff made visible: the stronger the penalty, the simpler the model, and beyond a point, too simple to capture genuine patterns.
from sklearn.linear_model import Lasso
lasso = Lasso(alpha=1)
lasso.fit(X_train_encoded, y_train)
lasso_pred = lasso.predict(X_test_encoded)
Use Lasso when: you suspect many features are irrelevant and want the model to identify and remove them automatically.
Limitation: when two features are highly correlated, Lasso tends to arbitrarily select one and zero out the other, which may not reflect the true underlying structure.
Ridge Regression (L2)
Cost = RSS + α × Σβⱼ²
Ridge uses the sum of squared coefficients as its penalty. The structure is nearly identical to Lasso, but the geometry of squaring means Ridge shrinks coefficients close to zero without ever reaching it exactly. Every feature remains in the model — only with reduced influence.
This behaviour makes Ridge more appropriate when all features are expected to contribute something meaningful, and particularly when predictors are correlated. Ridge distributes the weight between correlated features more gracefully than Lasso's winner-take-all approach.
from sklearn.linear_model import Ridge
ridge = Ridge(alpha=1)
ridge.fit(X_train_encoded, y_train)
ridge_pred = ridge.predict(X_test_encoded)
Use Ridge when: all features are genuinely relevant, or when multicollinearity between predictors is a concern.
Elastic Net (L1 + L2)
Cost = RSS + λ[α × Σ|βⱼ| + ((1-α)/2) × Σβⱼ²]
Elastic Net combines both penalties. The parameter l1_ratio (α in the formula above) controls the blend:
-
l1_ratio = 1.0→ pure Lasso -
l1_ratio = 0.0→ pure Ridge -
l1_ratio = 0.5→ equal mix
This flexibility resolves the individual weaknesses of each: Elastic Net can zero out genuinely irrelevant features while handling correlated features more gracefully than Lasso alone.
from sklearn.linear_model import ElasticNet
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5)
elastic.fit(X_train_encoded, y_train)
Use Elastic Net when: you are uncertain whether Lasso or Ridge is more appropriate. It is a reliable default in high-dimensional problems.
Part 5: When to Use Each Approach
The choice of regularization is not arbitrary — it should follow from what you know about your data and the problem structure.
| Technique | Penalty | Zeros coefficients? | Best suited for |
|---|---|---|---|
| Plain OLS | None | No | No overfitting present |
| Lasso (L1) | Absolute values | Yes | Many irrelevant features |
| Ridge (L2) | Squared values | No | All features relevant; high correlation |
| Elastic Net | Both | Yes | Uncertain — safe default |
A practical workflow: always start with plain linear regression and evaluate the training-to-test gap. If the gap is small, regularization adds little value. If the gap is large, overfitting is present — try Lasso first if feature selection is desirable, Ridge if all features are believed to contribute, and Elastic Net when the situation is ambiguous.
The strength of regularization (α) is a hyperparameter — it is not learned from data but set by the analyst. Finding the optimal value requires cross-validation, a technique that evaluates model performance across multiple held-out subsets of the training data to identify the α that generalises best.
Closing Thoughts
Regression is deceptively simple to implement and genuinely difficult to implement well. The gap between running LinearRegression().fit(X_train, y_train) and building a model you can trust for decision-making lies in understanding what the model is doing, what assumptions it relies on, and where it will fail.
Regularization is not a correction applied when something goes wrong — it is a deliberate modelling choice that encodes prior knowledge about the problem: that coefficients should be small, that some features are likely irrelevant, or that the training data is noisy. Making that choice thoughtfully, rather than defaulting to whatever produces the highest R², is the difference between fitting data and building models that work.
Top comments (0)