Supervised machine learning regression predicts a continuous target variable from input features. This guide walks you through the core concepts, popular algorithms, and a hands‑on Python workflow you can start today. By the end you’ll have a working model and know how to iterate for better results.
What is regression?
Regression is a type of supervised learning where the goal is to estimate a numeric value — like house price, sales forecast, or temperature — based on historical data. Unlike classification, which outputs a category, regression outputs a real number. The training data consists of feature columns (independent variables) and a target column (dependent variable). The model learns the relationship so it can predict the target for new, unseen feature sets.
Common regression algorithms
Choosing the right algorithm depends on data size, linearity, and interpretability needs. Here are the most widely used ones:
- Linear Regression – assumes a straight‑line relationship; fast, easy to interpret.
- Ridge & Lasso Regression – add regularisation to handle multicollinearity and perform feature selection.
- Decision Tree Regressor – captures non‑linear patterns; prone to over‑fitting if not pruned.
- Random Forest Regressor – ensembles many trees for better generalisation.
- Gradient Boosting Regressors (XGBoost, LightGBM, CatBoost) – state‑of‑the‑art for tabular data, often winning competitions.
- Support Vector Regression (SVR) – works well with small‑to‑medium datasets and non‑linear kernels.
Setting up your Python environment
-
Install Anaconda or Miniconda – gives you a clean
condaenvironment. - Create a new env:
conda create -n reg-ml python=3.11and activate it. - Install core packages:
conda install numpy pandas scikit-learn matplotlib seaborn jupyterlab
- (Optional) For boosting libraries:
pip install xgboost lightgbm catboost
- Launch Jupyter:
jupyter laband start a new notebook.
Building a simple linear regression model
Below is a minimal end‑to‑end example using the California housing dataset (built‑in to scikit‑learn).
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
# Load data
data = fetch_california_housing(as_frame=True)
df = data.frame
X = df.drop('MedHouseVal', axis=1)
y = df['MedHouseVal']
# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train
model = LinearRegression()
model.fit(X_train, y_train)
# Predict
y_pred = model.predict(X_test)
# Evaluate
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'MSE: {mse:.4f}, R²: {r2:.4f}')
Run the cell; you’ll see a baseline MSE and R². This is your starting point.
Evaluating model performance
Beyond MSE and R², consider:
- Mean Absolute Error (MAE) – less sensitive to outliers.
- Residual plots – check for patterns indicating non‑linearity or heteroscedasticity.
-
Cross‑validation –
cross_val_scorewithcv=5gives a more robust estimate.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_squared_error')
print('CV MSE:', -scores.mean())
If residuals show curvature, a linear model is insufficient — move to non‑linear algorithms.
Improving your model
-
Feature engineering – create interaction terms, log‑transform skewed features, or add polynomial features (
PolynomialFeatures). -
Regularisation – switch to Ridge (
alpha=1.0) or Lasso (alpha=0.01) to shrink coefficients. -
Tree‑based models – try
RandomForestRegressor(n_estimators=300, random_state=42)orXGBRegressor. -
Hyperparameter tuning – use
GridSearchCVorRandomizedSearchCVon key parameters (e.g.,max_depth,learning_rate). - Ensemble stacking – combine predictions from multiple models for a final meta‑learner.
Each step should be validated with cross‑validation to avoid over‑fitting.
Next steps
- Deepen the math – linear algebra and calculus underpin gradient‑based optimisation. The Mathematics for Data Science: Linear Algebra & Calculus (Beginner) course covers the essentials.
- Master the fundamentals – if you want a focused, hands‑on regression track, start with the Supervised Machine Learning: Regression Algorithms with Python (Beginner) course.
- Strengthen linear regression intuition – the Mastering Linear Regression: Theory & Implementation (Beginner) course dives deep into assumptions, diagnostics, and regularisation.
- Explore ensembles – once comfortable, move to the Ensemble ML Techniques & Applications (Intermediate) course for boosting, bagging, and stacking strategies.
- Practice on real projects – grab open datasets (Kaggle, UCI, government portals) and apply the full pipeline: cleaning, EDA, modelling, evaluation, and deployment.
Keep iterating, document your experiments, and build a portfolio notebook for each project. Consistent practice turns theory into job‑ready skills.
Originally published at NerdIT. NerdIT offers free online courses in AI, data science, Python and more, with verified certificates — browse the courses.
Top comments (0)