DEV Community

Cover image for Supervised Machine Learning: Regression Algorithms with Python – A Practical How‑To Guide
siddharth parakh
siddharth parakh

Posted on Originally published at nerdit.in

Supervised Machine Learning: Regression Algorithms with Python – A Practical How‑To Guide

Supervised machine learning regression predicts a continuous target variable from input features. This guide walks you through the core concepts, popular algorithms, and a hands‑on Python workflow you can start today. By the end you’ll have a working model and know how to iterate for better results.

What is regression?

Regression is a type of supervised learning where the goal is to estimate a numeric value — like house price, sales forecast, or temperature — based on historical data. Unlike classification, which outputs a category, regression outputs a real number. The training data consists of feature columns (independent variables) and a target column (dependent variable). The model learns the relationship so it can predict the target for new, unseen feature sets.

Common regression algorithms

Choosing the right algorithm depends on data size, linearity, and interpretability needs. Here are the most widely used ones:

  • Linear Regression – assumes a straight‑line relationship; fast, easy to interpret.
  • Ridge & Lasso Regression – add regularisation to handle multicollinearity and perform feature selection.
  • Decision Tree Regressor – captures non‑linear patterns; prone to over‑fitting if not pruned.
  • Random Forest Regressor – ensembles many trees for better generalisation.
  • Gradient Boosting Regressors (XGBoost, LightGBM, CatBoost) – state‑of‑the‑art for tabular data, often winning competitions.
  • Support Vector Regression (SVR) – works well with small‑to‑medium datasets and non‑linear kernels.

Setting up your Python environment

  1. Install Anaconda or Miniconda – gives you a clean conda environment.
  2. Create a new env: conda create -n reg-ml python=3.11 and activate it.
  3. Install core packages:
   conda install numpy pandas scikit-learn matplotlib seaborn jupyterlab
Enter fullscreen mode Exit fullscreen mode
  1. (Optional) For boosting libraries:
   pip install xgboost lightgbm catboost
Enter fullscreen mode Exit fullscreen mode
  1. Launch Jupyter: jupyter lab and start a new notebook.

Building a simple linear regression model

Below is a minimal end‑to‑end example using the California housing dataset (built‑in to scikit‑learn).

import pandas as pd
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Load data
data = fetch_california_housing(as_frame=True)
df = data.frame
X = df.drop('MedHouseVal', axis=1)
y = df['MedHouseVal']

# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train
model = LinearRegression()
model.fit(X_train, y_train)

# Predict
y_pred = model.predict(X_test)

# Evaluate
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'MSE: {mse:.4f}, R²: {r2:.4f}')
Enter fullscreen mode Exit fullscreen mode

Run the cell; you’ll see a baseline MSE and R². This is your starting point.

Evaluating model performance

Beyond MSE and R², consider:

  • Mean Absolute Error (MAE) – less sensitive to outliers.
  • Residual plots – check for patterns indicating non‑linearity or heteroscedasticity.
  • Cross‑validation – cross_val_score with cv=5 gives a more robust estimate.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_squared_error')
print('CV MSE:', -scores.mean())
Enter fullscreen mode Exit fullscreen mode

If residuals show curvature, a linear model is insufficient — move to non‑linear algorithms.

Improving your model

  1. Feature engineering – create interaction terms, log‑transform skewed features, or add polynomial features (PolynomialFeatures).
  2. Regularisation – switch to Ridge (alpha=1.0) or Lasso (alpha=0.01) to shrink coefficients.
  3. Tree‑based models – try RandomForestRegressor(n_estimators=300, random_state=42) or XGBRegressor.
  4. Hyperparameter tuning – use GridSearchCV or RandomizedSearchCV on key parameters (e.g., max_depth, learning_rate).
  5. Ensemble stacking – combine predictions from multiple models for a final meta‑learner.

Each step should be validated with cross‑validation to avoid over‑fitting.

Next steps

Keep iterating, document your experiments, and build a portfolio notebook for each project. Consistent practice turns theory into job‑ready skills.


Originally published at NerdIT. NerdIT offers free online courses in AI, data science, Python and more, with verified certificates — browse the courses.

Top comments (0)