Test

Tuvshin — Thu, 04 Jun 2026 14:59:01 +0000

+++
date = '2026-06-02T04:12:15+09:00'
draft = false
title = 'Test 00'
math = 'true'
+++

Introduction to Linear Regression

Linear Regression is a foundational machine learning algorithm used to model the relationship between a dependent scalar variable $y$ and one or more explanatory variables $X$.

Mathematical Formulation

The goal of linear regression is to find a linear function that predicts the dependent variable $y$ given the input features $\mathbf{x}$.

The linear model is expressed as:

$$y = \mathbf{w}^T \mathbf{x} + b$$

Where:

$\mathbf{w}$ is the weight vector (parameters).
$\mathbf{x}$ is the feature vector.
$b$ is the bias term (intercept).

The Cost Function (Mean Squared Error)

To find the optimal parameters $\mathbf{w}$ and $b$, we minimize the Mean Squared Error (MSE), which measures the average squared difference between actual values $y_i$ and predicted values $\hat{y}_i$:

$$J(\mathbf{w}, b) = \frac{1}{2m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right)^2$$

Where $m$ is the total number of training examples.

Parameter Optimization: Gradient Descent

We optimize the cost function using Gradient Descent. The parameters are updated iteratively by moving in the opposite direction of the gradient.

The gradient updates for weights and bias are calculated using the partial derivatives of the cost function:

$$\frac{\partial J}{\partial \mathbf{w}} = \frac{1}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right) \mathbf{x}^{(i)}$$

$$\frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right)$$

Update Rules:

$$\mathbf{w} := \mathbf{w} - \alpha \frac{\partial J}{\partial \mathbf{w}}$$

$$b := b - \alpha \frac{\partial J}{\partial b}$$

Where $\alpha$ is the learning rate.

Python Implementation

Below is a clean NumPy implementation of Linear Regression built from scratch using gradient descent.

import numpy as np

class LinearRegressionGradientDescent:
    def __init__(self, learning_rate=0.01, iterations=1000):
        self.lr = learning_rate
        self.iterations = iterations
        self.weights = None
        self.bias = None

    def fit(self, X, y):
        num_samples, num_features = X.shape
        # Initialize parameters to zeros
        self.weights = np.zeros(num_features)
        self.bias = 0.0

        # Gradient Descent Loop
        for _ in range(self.iterations):
            # 1. Compute predicted values
            y_predicted = np.dot(X, self.weights) + self.bias

            # 2. Compute gradients
            dw = (1 / num_samples) * np.dot(X.T, (y_predicted - y))
            db = (1 / num_samples) * np.sum(y_predicted - y)

            # 3. Update parameters
            self.weights -= self.lr * dw
            self.bias -= self.lr * db

    def predict(self, X):
        return np.dot(X, self.weights) + self.bias

# Example Execution
if __name__ == "__main__":
    # Generate simple synthetic data
    X = np.array([[1], [2], [3], [4], [5]])
    y = np.array([2, 4, 5, 4, 5])

    # Train model
    model = LinearRegressionGradientDescent(learning_rate=0.01, iterations=500)
    model.fit(X, y)

    # Test predictions
    predictions = model.predict(X)
    print("Trained Weights:", model.weights)
    print("Trained Bias:", model.bias)
    print("Predictions:", predictions)