What is linear regression?
Linear regression from the term regression analysis, is a technique used to model and analyze the relationship between a dependent variable (the outcome you want to predict) and one or more independent variables (the factors influencing the outcome).
linear regression predicts and defines relationships between continuous variables.
What is a continuous variable?
This is a number that can take on any value within a given range, including decimals and fractions.
Count of items: 10 books, 100 Sales, 200 Students
Measures: 20.004Kg, 120Km, 20.2Pounds
Problem it solves
1.Business Problem
With the power of linear regression we can help a business:
- Predict sales and come up with the stategies to improve on them.
- Understand relationship between their sales and factors like:
- Count of items
- day of the week
- discount levels.
- Estimate customer lifetime value.
2.Health Care Problem
Linear regression in the field of health care can help:
- Predict the weight of a patient
- Understand the relationship between a patient's weight and:
- Sugar levels.
- Genes
Intution behind linear regression
Take an example of a Retail shop, where The price of an item like milk depends on its quantity:
PLotting it would look like:
- On the x-axis (horizontal axis), enter the milk quantity.E.g:
1L, 500Ml, 3L - On the y- axis (Vertical axis), enter the price of the milk. The points won't fit in a straight line but will be scattered in the chat.
- Now take a ruler and try drawing a straight line on the points in such a way that it passes at the centre of the data. This is the line of best fit.
The model adjusts only two things:
- Intercept — where the line starts on the Y-axis.
- Slope — how steep the line is.
These three steps are what linear regression is trying to do.
Terms in linear regression
From the term linear in analysis, linear regression is a relationship that forms a straight line:
y = mx + c
An increase in a unit in x leads to a unit increase in y
Dependent variable (x)
This is a variable or factor that influences the outcome.
Independent variable (y)
It is the outcome to predict.
Slope (m)
The number that shows how much your output (y) changes when your input (x) goes up by one unit.
Intercept (c)
is the expected value of your output (y) when your input (x) is exactly zero. It is the starting point where your regression line crosses the vertical Y-axis.
Fit
Describes how well your straight line matches the actual data points. A good fit means your line passes very close to most of your data points, making your predictions highly accurate.
Residuals
These are mistakes made by the model. Which is gotten by subtracting the actual values and the predicted values
An analogy to the terms
The data provided below is a supermarket sales data:
| Order No | Order Date | Customer Name | Ship Date | Retail Price (USD) | Order Quantity | Tax (USD) | Total (USD) |
|---|---|---|---|---|---|---|---|
| 1001 | 1/1/2024 | John Smith | 1/3/2024 | 49.99 | 2 | 9.998 | 109.978 |
| 1002 | 1/1/2024 | Jane Doe | 1/4/2024 | 29.99 | 1 | 2.999 | 32.989 |
| 1003 | 1/2/2024 | Michael Johnson | 1/7/2024 | 99.99 | 3 | 29.997 | 329.967 |
| 1004 | 1/2/2024 | Emily Brown | 1/3/2024 | 19.99 | 4 | 7.996 | 87.956 |
| 1005 | 1/3/2024 | David Wilson | 1/8/2024 | 149.99 | 1 | 14.999 | 164.989 |
We are supposed to predict the future total price making it our dependent variable (y). We should consider factors like Retail Price (USD), Tax (USD) which are our independent variables (x).
After making our pediction based on (x), we can plot the predicted total price against the actual total price. By plotting a straight line through the data points. A good fit means our line passes very close to most of your data points, making your predictions highly accurate.
If the line of best fit shows a Positive slope then an increase in (x) leads to an increase in (y) but if it shows a negative slope then an increase in (x) leads to a decreases in (y) and vise versa.
If the line of best fit does not touch all the data points the the model has some errors which are the residuals.
Assumptions in linear regression
Linear regression relies on five key assumptions to ensure accurate, reliable, and interpretable results.
1. Linearity
The relationship between the dependent variable (y) and the independent variables (X) must be linear.
2. Homoscedasticity (Equal Variance)
The variance of the error terms should be constant across all values of the independent variables. If the errors spread out or bunch together as X increases, it is called heteroscedasticity, which leads to unreliable standard errors.
3. No multicollinearity
In multiple linear regession, independent variables are should not to have high corelation with another. High multicollinearity makes it difficult to isolate the individual effect of each predictor on the dependent variable.
4. Normality of residuals
The residauls should be normally distributed around the mean zero. This ensures the p-value and the confidence interval are reliable for hypothesis testing.
5. Independence of Errors
The residuals must be independent of one another. This is especially crucial for time-series data, where consecutive observations should not influence each other (no autocorrelation)
How to check for the assumptions
Linearity
Scatterplots or residual vs. fitted plots. If you see a curved pattern, a non-linear model or data transformation is required.
Homoscedasticity (Equal Variance)
Residual vs. fitted values plot. Ideally, the points should be randomly scattered without any "cone" or "funnel" shape.
No multicollinearity
Correlation matrices or Variance Inflation Factor (VIF) scores. A VIF value over 5 to 10 typically indicates an issue.
Normality of residuals
Normal Q-Q plots, histograms of residuals, or statistical tests like the Shapiro-Wilk test.
Independence of Errors
Plotting residuals against time/index or using the Durbin-Watson test.
Hypothesis testing in linear regression
Hypothesis testing in linear regression determines if there is a statistically significant relationship between the predictor variables and the response variable. It primarily uses t-tests for individual coefficients (to see if they are zero) and an F-test for the overall model's validity.
t-test implimentation
Null Hypothesis states that the variable has no linear effect on the response.
Alternative Hypothesis states that the variable has a significant linear effect on the response.
f-test implimentation
Null Hypothesis states that none of the predictors are useful.
Alternative Hypothesis states that at least one predictor is significantly associated with the response.
Simple linear regression
Simple linear regression method used to predict the value of one variable based on the value of a different variable.
Y = β₀ + β₁X
Where:
- Y (Dependent Variable): The main thing you are trying to predict or understand.
- X (Independent Variable): The input data you are using to make the prediction.
- β₀ (Intercept): The starting value of Y when X is zero.
- β₁ (Slope): How much Y changes for every single step up in X.
Use case
From the data we discussed above, the only difference we make is that we only concider one factor like Order Quantity to predict the Total (USD).
Multiple linear regression
It is method used to predict the value of one variable based on two or more different input variables.
Y = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ
plaintext
Where:
- Y: The main thing you are trying to predict (the output).
- β₀ (Intercept): The starting baseline value when all your inputs are zero.
- X₁, X₂, X₃: Your different input variables.
- β₁, β₂, β₃ (Slopes): The individual impact of each input. For example, β₁ tells you how much Y changes when X₁ goes up, assuming all other inputs stay exactly the same.
Use case
We should consider factors like Retail Price (USD), Tax (USD) and Order Quantity to predict the Total (USD).
Building our first linear regression model in Python
1. Installing the required libraries
pip install numpy pandas matplotlib seaborn scikit-learn
2. Importing the python libraries
# Importing pandas for data analysis
import pandas as pd
# Importing numpy for array manipulation
import numpy as np
# Import seabon and matplotlib for visaulization
import matplotlib.pyplot as plt
import seaborn as sns
# Importing train_test_split for extracting train and test data
from sklearn.model_selection import train_test_split
# Creating LinearRegression for model creation and continous variables prediction
from sklearn.linear_model import LinearRegression
# Importing mean_absolute_error, mean_squared_error, r2_score for model evaluation
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
3. Reading our excel file as a pandas dataftame
# Reading our excel file as a pandas dataftame
dataframe = pd.read_excel('DATA/Supermarket-Sales-Sample-Data.xlsx', skiprows=7)
# Displaying the first 5 rows in the dataframe
dataframe.head()
| Order No | Order Date | Customer Name | Ship Date | Retail Price (USD) | Order Quantity | Tax (USD) | Total (USD) |
|---|---|---|---|---|---|---|---|
| 1001 | 1/1/2024 | John Smith | 1/3/2024 | 49.99 | 2 | 9.998 | 109.978 |
| 1002 | 1/1/2024 | Jane Doe | 1/4/2024 | 29.99 | 1 | 2.999 | 32.989 |
| 1003 | 1/2/2024 | Michael Johnson | 1/7/2024 | 99.99 | 3 | 29.997 | 329.967 |
| 1004 | 1/2/2024 | Emily Brown | 1/3/2024 | 19.99 | 4 | 7.996 | 87.956 |
| 1005 | 1/3/2024 | David Wilson | 1/8/2024 | 149.99 | 1 | 14.999 | 164.989 |
4. Showing the correlation between columns
# Calculating the correlation betwee numerical columns
corr = dataframe.corr()
# Displaying the correlation using a heatmap
sns.heatmap(data=corr, cmap='coolwarm')
The correlation gave the following insights:
- Order number had no relationship with other columns
- Retail Price had a negative relationship with Order Quantity but a postive one with the Tax and Total columns
- Order Quantity had a negative relationship with Retail Price, Tax and Total columns
- Tax had a negative relationship with Order Quantity but a postive one with the Retail Price and Total columns
- Total had a negative relationship with Order Quantity but a postive one with the Tax and Retail Price columns
5. Selecting columns fit for modelling
Before training the predictive model, unnecessary columns are removed to improve model performance and reduce noise. Since machine learning algorithms require numerical inputs, categorical date columns are transformed into numerical features using one-hot encoding.The preprocessing steps include:
- Removing columns that do not contribute to prediction.
- Preparing the dataset for model training.
# Dropping the unwanted colums during modeling and prediction
dataframe = dataframe.drop(columns=['Order No', 'Customer Name', 'Order Date', 'Ship Date'])
# Creating the predictors and the independent variables for modelling
X = dataframe.drop('Total (USD)', axis=1)
y = dataframe['Total (USD)']
python
6. Splitting the data for training and testing
The cleaned dataset is divided into predictor variables (X) and the target variable (y). The data is then split into training and testing sets using a 70:30 ratio.
# Splitting our dataset for trainig the model and testing it's perfomance
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=.7, random_state=42)
7. Building the Linear Regression Model
A Linear Regression model is trained using the training data, after which predictions are generated on the test set. These predictions will later be evaluated to determine how well the model estimates supermarket sales.
# assigning our LinearRegression model
model = LinearRegression()
# Training the model using X_train and y_train to help it uderstand the predictors and predicted values
model.fit(X_train, y_train)
# Performing prediction on the X_test to get predicted variables
y_pred = model.predict(X_test)
y_pred
array([142.989, 109.978, 142.989, 164.989, 219.989, 219.989, 164.945,
164.989, 219.989, 164.967, 142.989, 219.989, 142.989, 175.978,
131.967, 164.967, 142.989, 131.967, 164.967, 175.978, 153.978])
8. Performing model evaluation
To determine the effectiveness of the Linear Regression model, several evaluation metrics are calculated:
- Mean Absolute Error (MAE): Measures the average prediction error.
- Root Mean Squared Error (RMSE): Penalizes larger prediction errors and indicates the model's overall accuracy.
- R-squared (R²): Represents the proportion of variance in the target variable explained by the model.
Together, these metrics provide a comprehensive assessment of the model's predictive performance.
# 1. Mean Absolute Error (MAE)
mae = mean_absolute_error(y_test, y_pred)
# 2. Root Mean Squared Error (RMSE)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
# 3. R-squared Score (R²)
r2 = r2_score(y_test, y_pred)
print(f"Mean Absolute Error: {mae:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")
print(f"R-squared Score: {r2:.2f}")
Mean Absolute Error: 0.00
Root Mean Squared Error: 0.00
R-squared Score: 1.00
With uniform transaction math in clean sample data, errors will compute to 0.00 and R² will equal a perfect 1.00, validating that the model successfully extracted the exact pricing rules
9. Visualizing Model Predictions
A scatter plot is used to compare the actual sales values against the model's predicted values.
A 45-degree reference line is included to represent perfect predictions.The closer the data points lie to this reference line, the better the model's predictive accuracy.
This visualization provides an intuitive way to assess how well the Linear Regression model performs on unseen data.
# Defining the plot's configuration
plt.figure(figsize=(8, 6))
sns.set_theme(style="whitegrid")
# 1. Plot actual answers vs predicted answers
sns.scatterplot(x=y_test, y=y_pred, color='purple', alpha=0.6, label='Predictions')
# 2. Draw a perfect 45-degree reference line
# (If your model is perfect, all dots will sit exactly on this line)
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()],
color='orange', linewidth=2.5, linestyle='--', label='Perfect Fit')
plt.title('Actual Values vs. Predicted Values', fontsize=14, pad=15)
plt.xlabel('Actual True Values', fontsize=12)
plt.ylabel('Model Predictions', fontsize=12)
plt.legend()
plt.show()
To sum up
Top comments (0)