Viswa M

Posted on Mar 27

Building a Linear Regression Model from Scratch with Gradient Descent in Python

#linearregression #gradientdescent #python #machinelearning

Overview

Title

Gradient Descent Linear Regression in Python

Meta Description

Learn how to build a linear regression model from scratch using gradient descent in Python. Step‑by‑step code, math, and practical tips.

1. Introduction

Linear regression is usually the first model you build when learning machine learning. It introduces the essential concepts of parameters, loss, gradients, and optimisation in the simplest setting: a straight‑line fit.

In this post we’ll walk through a compact Python script that learns a line from five data points using gradient descent. We’ll explain the maths, step through the code, and predict a new value. By the end you’ll understand why the parameters change and how to tweak the algorithm for your own data.

2. What the program does in a nutshell

The script trains a linear model

to minimise mean‑squared error between predictions $\hat{y}$ and the true outputs $y$ .

Starting from $m=0$ , $b=0$ , it repeatedly updates these two numbers until the loss stops improving, then prints the learned slope, intercept, and a prediction for a new input.

3. Code Implementation

import numpy as np
from tqdm import tqdm

# Data
X = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])

# Initial parameters
m = 0          # slope
b = 0          # intercept

# Hyper‑parameters
lr = 0.01      # learning rate
epochs = 1000  # number of iterations

n = len(X)     # number of training samples

# Gradient descent loop
for _ in tqdm(range(epochs)):
    y_hat = m * X + b

    dm = (-2 / n) * np.sum(X * (y - y_hat))   # gradient wrt m
    db = (-2 / n) * np.sum(y - y_hat)         # gradient wrt b

    m -= lr * dm
    b -= lr * db

print("Slope:", m)
print("Intercept:", b)

# Prediction for a new input
input_value = 6
pred = m * input_value + b
print("Predicted y for x=6:", pred)

4. Step‑by‑step walk‑through

Imports – numpy for vector maths; tqdm for a progress bar.
Data – X (inputs) and y (targets).
Parameters – start with m = 0, b = 0.
Hyper‑parameters – learning rate (lr) controls the step size; epochs limits how many updates we perform.
Training loop
1. Compute predictions: y_hat = m * X + b.
2. Compute gradients:
  - dm = (-2 / n) * np.sum(X * (y - y_hat)) (slope).
  - db = (-2 / n) * np.sum(y - y_hat) (intercept).
3. Update parameters: move a fraction (lr) of the negative gradient.
After training – print the final m and b, then predict $y$ for $x = 6$ .

5. Key concepts (with maths)

5.1 Linear regression model

$m$ – slope (rate of change).
$b$ – intercept (value at $x=0$ ).

5.2 Loss function (mean‑squared error)

5.3 Gradient of the MSE

These match the dm and db formulas in the code.

5.4 Gradient descent update

With learning rate $\alpha$ :

Repeating moves the parameters toward the minimum of the loss surface.

6. Example section

Here’s what the first few iterations look like (values rounded for clarity):

Step	$m$	$b$	MSE
0	0.000	0.000	12.80
1	0.040	0.240	8.23
2	0.070	0.360	6.46
…	…	…	…

After 1 000 epochs the algorithm converges to approximately

Slope: 0.60
Intercept: 1.20

Predicting for $x = 6$ gives

so the program outputs “Predicted y for x=6: 4.8”.

7. Quick sanity checks

Setting	Observation
Small learning rate (`lr = 0.0001`)	Converges slowly; more epochs needed.
Large learning rate (`lr = 1`)	Updates overshoot the optimum; loss may diverge.

Choosing a suitable lr and epochs is a standard practice for any optimisation problem.

8. Take‑away

This code implements ordinary least‑squares regression with gradient descent.
Gradient descent is a generic optimisation routine that underpins logistic regression, neural networks, and more.
Understanding the update equations clarifies why the parameters evolve and how to troubleshoot training.

Feel free to experiment:

Swap in a new dataset.
Try different learning rates or epoch counts.
Normalise your inputs or add a bias term.

Happy coding and keep building!

Slug: linear-regression-gradient-descent-python

SEO Title: Gradient Descent Linear Regression in Python

Meta Description: Learn how to build a linear regression model from scratch using gradient descent in Python. Step‑by‑step code, math, and practical tips.

Keywords: linear regression, gradient descent, python, machine learning, data science, supervised learning, mean squared error, model training, algorithm explanation

Tags: linearregression, gradientdescent, python, machinelearning, dataanalysis, codingtutorial, algorithm, mse, supervisedlearning

DEV Community