Transforming Simplicity: Adapting Linear Regression to Capture Complex Non-Linear Phenomena with NumPy

In the field of Machine Learning, non-linear phenomena are ubiquitous. Often, people think that linear regression models cannot capture these complexities. However, with a few adjustments, linear regressions can be surprisingly effective in modeling non-linear relationships. In this article, we will explore how to adapt a linear regression model for non-linear phenomena using Python and NumPy.

Linear Regression Model
Linear regression seeks to find the best line (or hyperplane in higher dimensions) that minimizes the error between predictions and actual values. The equation of a simple linear regression model is:

For multivariate models, this equation becomes:

Transforming Variables to Capture Non-Linearity
To adapt this model to non-linear phenomena, we can transform the input variables using non-linear functions. For example, we can include quadratic or cubic terms:

These transformations allow the linear model to learn complex non-linear relationships.

Cost Function
The cost function for linear regression is usually the mean squared error (MSE):

Gradient Descent
To minimize the cost function, we use the gradient descent algorithm. The parameter updates are as follows:

where 𝛼 is the learning rate.

Implementation in Python with NumPy
Here is a simple implementation in Python:

import numpy as np
from sklearn.datasets import make_regression
import matplotlib.pyplot as plt

x, y = make_regression(n_samples=100, n_features=1, noise=10)
y = y + abs(y/2)
plt.scatter(x, y)

X = np.hstack((x, np.ones(x.shape)))
X = np.hstack((x**2, X))

theta = np.random.randn(3, 1)
theta

def cost_function(X, y, theta):
    m = len(y)
    return 1/(2*m) * np.sum((model(X, theta) - y)**2)

def grad(X, y, theta):
    m = len(y)
    return 1/m * X.T.dot(model(X, theta) - y)
def gradient_descent(X, y, theta, learning_rate, n_iter):
    cost_history = np.zeros(n_iter)
    for i in range(0, n_iter):
        theta = theta - learning_rate * grad(X, y, theta)
        cost_history[i] = cost_function(X, y, theta)

    return theta, cost_history

n_iter = 1000
learning_rate = 0.01

theta_final, cost_history = gradient_descent(X, y, theta, learning_rate, n_iter)

predict = model(X, theta_final)

plt.scatter(x[:, 0], y)
plt.scatter(x[:, 0], predict, c='r')

By transforming the input variables, we can use linear regression models to capture non-linear relationships. This flexibility, combined with techniques such as gradient descent, allows linear models to handle much more complex phenomena than they would normally. NumPy provides a solid foundation for implementing these concepts in Python, making non-linear modeling more accessible.