DEV Community: Idighekere Udo

Building Linear Regression From Scratch in Python: From Matrix Math to Real Predictions

Idighekere Udo — Sat, 31 Oct 2026 01:48:24 +0000

When learning machine learning, it's always easy to rely on scikit-learn to build models, but importing these algorithms and estimators makes it difficult for one to explain what is happening with these numbers and data under the hood. Building a linear regression from scratch using just numpy forces you to meet matrix dimension, cost functions, learning rate directly.

What is Linear Regression

Linear Regression is a supervised learning machine learning algorithm that models a relationship between the features/input ( $X$ ) and the target/output ( $y$ ) by choosing optimal weights/parameters. So we will understand in simple terms how scikit-learn handles this by implementing it from scratch using the Ordinary Least Square (Normal Equation) and iterative Gradient Descent Approach through vectorized matrix operations.

I am not going to go through the whole mathematics behind it in this article, if you are interested, you can find my note here on github.

Designing a learning algorithm

I am gonna define symbols i will be using through out the article.

$X$ = input/features
$y$ = prediction/target/output
$θ$ = weight/parameter.
$n$ = number of features
$m$ = number of rows
$J (θ)$ = cost function/error function

You have your training data -> you pass it through a learning algorithm (Linear Regresssor) -> you get an hypothesis -> pass an input through the hypothesis -> get an output

This hypothesis is just a function with weights. The learning algorithm learns the weights for each feature and by this it will use to make predictions.

Since our weights in our hypothesis determines the prediction we get

How do we choose a good weight/parameter for our algorithm?

We choose $θ$ s that when used, we get predictions $y^$ close to our actual values $y$ .
We calculate our error = $y^$ - $y$ and try to the least possible value of the error. If the actaul value is 100 and our model's output is 80. The error is -20 and we can know that our model predicted 20 less than the actual.

Instead of just looking at the error for a single row, we need a way to measure the total error across all rows in our dataset. To do this, we square the differences so that positive and negative errors don't cancel each other out, and we sum them up using the Mean Squared Error (MSE) formula:

J (θ) = \frac{1}{2} i = 1 \sum m (y^^{(i)} - y^{(i)})^{2}

(Note: The $\frac{1}{2}$ is just a math convenience that makes the calculus cleaner when we take derivatives later).

Preparing the Data and the Intercept Column

Before feeding data into our algorithm, we load our dataset (a small CSV with roughly 30 samples). Because our hypothesis requires a y-intercept ( $θ_{0}$ ), we have to append a column of ones to our feature matrix $X$ using np.hstack so matrix multiplication can compute the intercept for every row simultaneously:

import numpy as np

# Suppose X_raw is our feature matrix from our ~30 sample CSV
m = X_raw.shape[0]
X = np.hstack((np.ones((m, 1)), X_raw))  # Appending column of 1s

How Gradient Descent Finds the Best Weights

Now that we know how to calculate our error, how do we actually find the optimal weights $(θ)$ that give us the smallest possible error? This is where Gradient Descent comes in.

Imagine you are standing at the very top of a foggy mountain, and your goal is to walk down to the lowest possible valley (the lowest elevation). That lowest elevation represents the minimum of our cost function—where our error is as small as it can possibly get.

Since it's foggy and you can't see the whole map, what do you do? You look around your feet, feel the steepness of the slope, and take a step downward in the direction that goes down the steepest. You repeat this process over and over until you can't step down any further.That is exactly how Gradient Descent works:

Start Randomly: We initialize our weights $(θ)$ with random values (or zeros). At this starting point, our model's predictions are terrible, and our cost (the elevation on our mountain) is massive.
Calculate the Slope (Gradient): We calculate the partial derivative of the cost function with respect to each feature's weight. This tells us the slope of the hill at our current position.
Take a Step: We update our weights by moving in the opposite direction of the slope to reduce the cost.

The Learning Rate ( $α$ )

*How big of a step do we take as we walk down the mountain? *

That is controlled by a hyperparameter called the learning rate ($\alpha$).

If your learning rate is too big, you take giant leaps down the mountain. Instead of landing smoothly in the valley, you overshoot, bounce back and forth, or launch your weights straight into infinity.
If your learning rate is too small, you take microscopic steps. Your model will survive, but it might take tens of thousands of epochs just to inch its way down the slope.Each step of the update rule looks like this:

θ_{j} := θ_{j} - α \frac{\partial}{\partial θ _{j}} J (θ)

Computing Cost and Gradient Descent in Code

Now that we have our cost function, we can write the Python functions using NumPy to calculate the cost and update our weights iteratively without using any slow for loops.

Here is how the cost function looks in code:

def compute_cost(X, y, theta):
  predictions = X @ theta
  errors = predictions - y
  return 0.5 * np.sum(errors ** 2)

And here is the gradient descent step that calculates how to shift our weights $(θ)$ to lower that cost:

def gradient_descent_step(theta, X, y, alpha=0.001):
  predictions = X @ theta # @ stands for dot product operation
  errors = predictions - y
  # X.T @ errors calculates the sum across all rows for every feature simultaneously
  gradient = X.T @ errors
  return theta - alpha * gradient

Running the Training Loop

With our functions ready, we can initialize our hyperparameters, set up our training loop to run across thousands of epochs(how much iteration), and track our best cost and weights:

EPOCHS = 200000
LEARNING_RATE = 1e-3 #0.001
np.random.seed(42) # for reproducibilty and make the random initializion of theta always same

best_theta = None
best_epoch = 0
best_cost = float("inf")

theta = np.random.randn(n, 1)
cost_history = []

for epoch in range(EPOCHS):
  theta = gradient_descent_step(theta, X_train, y_train, LEARNING_RATE)
  cost = compute_cost(X_train, y_train, theta)
  cost_history.append(cost)

  # Check for improvement
  if cost < best_cost:
    best_cost = cost
    best_epoch = epoch
    best_theta = theta

print("Initial Cost:", cost_history[0])
print(
    f"Final Cost: {cost_history[-1]} (Best Cost: {best_cost} at epoch"
    f" {best_epoch})"
)

When I ran this loop, my initial cost started massive at over 1.5 billion, but as gradient descent iterated, it dropped down to 337 million, finding its optimal weights right around epoch 6,128.

(Note: Alternatively, for linear regression, you can also solve for weights in a single closed-form step using the Normal Equation: $θ = (X^{T} X)^{- 1} X^{T} y$ , but iterative gradient descent gives us a clear look at how models learn dynamically!)

def normal_equation(X,y):
    return np.linalg.inv(X.T@X) @ X.T @ y
one_shot_weight = normal_equation(X_train,y_train)
print(one_shot_weight)

Moving From Scripts to Classes (Scikit-Learn Style)

When learning machine learning, procedural code is great for seeing the steps, but professional libraries wrap everything into reusable classes.

Just like how scikit-learn separates its models, we can structure our code into two distinct estimators:

LinearRegression: Uses the closed-form Normal Equation to find weights in a single shot (matching scikit-learn's LinearRegression).
BGDRegressor: Uses iterative Batch Gradient Descent (similar to how scikit-learn's SGDRegressor iteratively minimizes cost, though using the entire batch of data per step).

Here is how we implement both approaches as clean, object-oriented classes that automatically handle intercept columns and training loops:

1. The Normal Equation Class (Closed-Form)

import numpy as np

class LinearRegression:
    def __init__(self):
        self.theta = None
        self.best_theta = None

    def _compute_cost(self, X,y, theta):
        """
        X: feature matrix: shape (m x n)
        y: target values: shape (m x 1 or m,)
        theta: model weights: shape (n x 1)
        """

        #calcualting the hypothesis h(x) = X . theta
        predictions = X @ theta
        #calculating hte error:
        errors = predictions - y
        return 0.5 * np.sum(errors ** 2)

    def _normal_equation(self, X, y):
        return np.linalg.inv(X.T@X) @ X.T @ y


    def fit(self, X_train_raw,y_train):
        # automatcially handles the intercept column of ones so that the user won't have to

        m = X_train_raw.shape[0] # number of rows
        print(X_train_raw.shape)

        X_train = np.hstack((np.ones((m,1)),X_train_raw))

        n = X_train.shape[1] # number of features

        one_shot_weight = self._normal_equation(X_train,y_train)
        self.theta = one_shot_weight

        return self

    def predict(self,X_test_raw):
        if not self.theta.any():
            raise ValueError("This model has not been fitted yet! Call .fit() first.")

        m = X_test_raw.shape[0]
        X_test = np.hstack((np.ones((m, 1)), X_test_raw))

        return X_test @ self.theta

2. The Batch Gradient Descent Class (Iterative)


class BGDRegressor:
    def __init__(self,learning_rate=1e-4,epochs=20000,random_state=42):
        self.learning_rate = learning_rate
        self.epochs = epochs
        self.theta = None
        self.best_theta = None
        self.best_cost = float("inf")
        self.best_epoch = 0
        self.cost_history = [ ]
        self.seed=random_state

    def _compute_cost(self, X,y, theta):
        """
        X: feature matrix: shape (m x n)
        y: target values: shape (m x 1 or m,)
        theta: model weights: shape (n x 1)
        """

        #calcualting the hypothesis h(x) = X . theta
        predictions = X @ theta
        #calculating hte error:
        errors = predictions - y
        return 0.5 * np.sum(errors ** 2)

    def _normal_equation(self, X, y):
        return np.linalg.inv(X.T@X) @ X.T @ y

    def _batch_gradient_descent_step(self, X,y):
        """
        X: feature matrix: shape (m x n)
        y: target values: shape (m x 1 or m,)
        """

        predictions = X @ self.theta
        errors = predictions - y

        # Gradient calculation
        gradient = X.T @ errors

        return self.theta - (self.learning_rate * gradient) 


    def fit(self, X_train_raw,y_train):
        # automatcially handles the intercept column of ones so that the user won't have to

        rng = np.random.default_rng(self.seed)
        m = X_train_raw.shape[0] # number of rows
        X_train = np.hstack((np.ones((m,1)),X_train_raw))

        self.theta = rng.random(size=(X_train.shape[1],1))

        n = X_train.shape[1] # number of features


        for epoch in range(self.epochs):
            self.theta = self._batch_gradient_descent_step(X_train,y_train )

            cost = self._compute_cost(X_train, y_train, self.theta)
            self.cost_history.append(cost)

        # Track best parameters
            if cost < self.best_cost:
                self.best_cost = cost
                self.best_epoch = epoch
                self.best_theta = self.theta.copy()

        print(f"Training Complete! Best Cost: {self.best_cost:.4f} at Epoch {self.best_epoch}")
        return self



    def predict(self,X_test_raw):
        if not self.theta.any():
            raise ValueError("This model has not been fitted yet! Call .fit() first.")

        m = X_test_raw.shape[0]
        X_test = np.hstack((np.ones((m, 1)), X_test_raw))

        return X_test @ self.best_theta

Using them is just like your favorite libraries:

# Train via Normal Equation
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
preds_lr = lr_model.predict(X_test)

# Train via Batch Gradient Descent
bgd_model = BGDRegressor(learning_rate=1e-3, epochs=20000)
bgd_model.fit(X_train, y_train)
preds_bgd = bgd_model.predict(X_test)

Link to Jupyter Notebook File

Where It Broke (Mistakes and Fixes)

The Learning Rate Trap: When I first tried a learning rate of 0.01, my model didn't gently roll down the mountain, it violently overshot and threw overflow warnings and my predictions were NaN. Dropping it down to 0.001 was what finally kept the weights stable.
Forgetting the Intercept Column: Early on, my matrix multiplications kept throwing shape mismatch errors until I remembered to explicitly use np.hstack to add a column of ones so that $θ_{0}$ had a feature to multiply against.

What I'd Tell My Past Self

Always check your shapes: Print out X.shape and theta.shape before debugging matrix multiplication errors.
Watch your initial cost: If your first epoch's cost is in the trillions or throws NaN, stop the script immediately, your learning rate is too aggressive.
Vectorization over loops: Let NumPy handle the dot products (@) instead of writing slow Python for loops over your rows.

If you want to dive deeper into the full mathematical derivations and proofs behind these equations, you can check out my complete notes stored in my public Obsidian vault on GitHub.

Resources I Used

Stanford CS229 Lecture Notes (Andrew Ng & Tengyu Ma)
Gemini Notebook (pka Notebook LM) + Gemini
NumPy Official Documentation for Matrix Multiplication (@) and random methods (default_rng)

Why I want to Be a Front-End Developer and How HNG Will Help me Achieve My Goals

Idighekere Udo — Sun, 02 Feb 2025 16:02:28 +0000

"A journey of a thousand miles begins with a single step". My journey began years back when I was introduced to Google's Blogger (blogspot.com) for creating blog articles, after experimenting for few months I transitioned to using Wordpress and also purchasing my first domain and having a site. The issue I've always had then was trying to configure a theme to make the site to fit my taste. I decided to learn HTML and CSS where I was now able to inspect my Wordpress site grab the classes and ids and make the changes to the styling and applying it to Theme Custom Stylings.

I still felt a bit of limitation while using it because I had no knowledge of PHP which would have enabled me to create custom Wordpress themes to my taste. I decided to learn Javascript to complement the knowledge I had. I was now able to build interactive user interfaces just by writing codes.
I proceeded to learning React JS, a framework of Javascript and built some projects with it. On joining HNG11 in 2024, I got to learn NextJS and Typescript and using Firebase which enabled me to create my first fullstack project.

I chose frontend development because I liked creating intuitive user interfaces and experience. Learning React Native has also enabled me to use my knowledge of Javascript and React to build cross-platform applications.

Joining HNG will enable me to work on amazing real-life projects, collaborating with backend and designers, staying up to date with the evolving technologies in the ecosystem. The internship will also help me position myself for future job opportunities.

Tailwind vs CSS

Idighekere Udo — Thu, 27 Jun 2024 22:40:00 +0000

It can be so tiring to write multiple lines of CSS styling in our web apps. One of the challenges I always face in using HTML and CSS is usually coming up with generic class names for my elements, but Tailwind CSS or Bootstrap comes in as an aid. The debate between which is best always springs up amongst developers and my peers at HNG usually try to convince me that Bootstrap is best, but I still take Tailwind CSS.

This era of web development has evolved that frameworks and libraries are being utilized by developers to make codes much simplified and easier to be used. After the reading this article, you should be able to: know what the frameworks are: get the differences & similarities between Tailwind CSS and Bootstrap; know which one to go for; and actually get know how and where to get started using them.

When should I start using CSS Frameworks?

As a beginner, you can start using CSS frameworks when you know all the CSS required to build a good-looking user interface.
You should be able to use CSS to build responsive layouts that support different screen sizes, some basic flexible layouts, use grid layouts etc.
You shouldn't be in a haste to learn a CSS framework when you don't have a solid foundation of CSS.

Tailwind 🆚 Bootstrap

If you have checked the official documentation of the above frameworks, you will get to know more about them.
If you haven't, I got you.

Tailwind CSS
Tailwind CSS is a utility-first CSS framework for rapidly building modern websites without ever leaving your HTML. It is packed with classes flex, pt-4, text-center and rotate-90 etc which simply means you don't have to think of class names for your html elements and not also writing long css stylings

Without Tailwind

//HTML
<button class="subscribe-button">Subscribe </button>

//CSS
.subscribe-button {
   padding-right: 1rem;
   padding-left: 1rem;
}

With tailwind

//HTML
<button class="px-4">Subscribe </button>

You can see that writing just px-4 in the class attribute in Tailwindcss is equivalent to writing the class name and padding-right: 1rem; & padding-left: 1rem; in CSS

Bootstrap

Bootstrap is a UI (User Interface) Toolkit, which means it has pre-built components for buttons, modal, accordian, navbar, footers etc.

<button type="button" class="btn btn-primary">Primary</button>
<button type="button" class="btn btn-secondary">Secondary</button>
<button type="button" class="btn btn-success">Success</button>
<button type="button" class="btn btn-danger">Danger</button>
<button type="button" class="btn btn-warning">Warning</button>
<button type="button" class="btn btn-info">Info</button>
<button type="button" class="btn btn-light">Light</button>
<button type="button" class="btn btn-dark">Dark</button>

<button type="button" class="btn btn-link">Link</button>

Result.

You can see that you don't need to write much styling for a button you will just need to specify what kind of button it is.

Tailwind & Bootstrap Differences

Application created using Tailwind are always flexible and unique whereas that of Bootstrap are generally identical because it already has in-built templates for websites.
Tailwind offers unique utility-first classes while Bootstrap offers ready-made themes and templates.

Tailwind & Bootstrap Similarities

Less CSS codes
The usage of the both frameworks makes the developer not to worry about class names for the elements.
It saves the developer's time trying to write css properties.
You might have to always run back to the documentation to get the setup procedure and correct class name for what you want to achieve.

Which one should I choose?

There's no better answer to which is better or the best,it all lies on developer's preference.
If you'll need a website ready within a short period of time, Bootstrap is the best fit as it has pre-built components like sidebars, buttons, navbars, footers, cards, accordian, alert, spinner etc.
If you'll need your site to look unique and flexible (being able to design it to your taste), then Tailwind is the best fit.
Most developers often say that Tailwindcss makes code ugly to be read because of the numerous utility classes you will have to write to achieve your design, so if you happen to agree with them, then you should consider using Bootstrap.

Installation

Both frameworks require certain procedures to be followed before you can start using them. You'll have to head over to Tailwind CSS doc & Bootstrap doc for the setup procedure. I wrote this article to onboarding you to the frameworks.

Conclusion

In this article, you have learnt when you can start using any of the framework, how they differ, what they share in common, which one to choose.
Enroll as a frontend developer at HNG to gain more experience.