DEV Community: Kelvin Muthama (Tush)

Understanding Decision Trees

Kelvin Muthama (Tush) — Tue, 21 Jul 2026 15:23:35 +0000

It's on a weekend and you are planning on whether you are supposed to go outside and practice for your football match. You go back to your tracker and check how the weather affected your previous practice days. In your tracker there are weather factors like: [Outlook, Humidity, Wind] which affected whether you practiced or not.
The tracker provides the following:

Day	Outlook	Humidity	Wind	Play?
D1	Sunny	High	Weak	No
D2	Sunny	Normal	Weak	Yes
D3	Rain	High	Strong	No
D4	Overcast	High	Weak	Yes

You decide to make decision for tomorrow's pracrice and you come up with:

To determine whether practice should go ahead, the you evaluate environmental factors in a structured hierarchy, starting with the overall Outlook as its primary root split.

If the forecast is Overcast, the tree leads directly to a Yes outcome, as historical data (Day D4) shows favorable conditions regardless of other variables.
When the forecast indicates Sunny conditions, you branch down to evaluate Humidity: a High humidity level (Day D1) yields a No (canceling practice), whereas Normal humidity (Day D2) results in a Yes.
Conversely, when the outlook predicts Rain, the decision hinges entirely on Wind strength.
A Strong wind condition (Day D3) poses too much of a disruption, triggering a No decision to call off practice.
If the wind is Weak, the path resolves to a Yes, allowing practice to continue safely.

By systematically evaluating conditions through these binary splits, the decision tree provides a clear, reliable rule set for making quick practice decisions based on daily weather inputs.

What is a decision tree?

A decision tree is a supervised machine learning algorithm used for both classification (predicting a category) and regression (predicting a continuous number).
It maps decisions and their potential outcomes in an upside-down, flowchart-like structure that closely mimics human logic.

Anatomy of a decision tree

A decision tree routes a data point from the top down through several key components:
Root Node: The very first node at the top containing the complete dataset. It asks the initial question or evaluates the most important feature.
Internal / Decision Nodes: Sub-nodes that represent subsequent data tests or attributes (e.g., "Is income > $50,000?").
Branches: The paths connecting nodes. They represent the outcomes of a test or choices available (e.g., Yes or No).
Leaf Nodes: The terminal endpoints of the tree. They do not split any further and contain the final prediction or decision.

How a decision tree works

The training and deployment of a decision tree happen in two distinct phases: Phase 1 (Building the Tree) and Phase 2 (Making Predictions).

Phase 1 (Building the Tree)

This phase operates on a recursive binary splitting approach. It works from the top down, splitting data into two branches at a time until the data is organized.

Step 1: Evaluate All Possible Splits

The algorithm looks at every single feature in your dataset.
For each feature, it reviews every possible value to split on (e.g., outlook = 'sunny', outlook = 'rainy').

Step 2: Calculate Metric Purity

For every potential split, it calculates a purity score using Gini Impurity or Entropy.
It measures which split creates the most "pure" resulting groups (where data points are mostly the same class).

Step 3: Select the Best Split

-The single feature and specific value that maximize Information Gain (or minimize Gini Impurity) are chosen.

This feature becomes the Root Node (or current decision node), and the dataset splits into two branches.

Step 4: Repeat Recursively

The algorithm repeats Steps 1–3 for each new branch, treating it as a brand-new mini-dataset.

Step 5: Apply Stopping Criteria

The tree stops growing when it hits a user-defined limit, such as:

All data points in a node belong to the same class (perfect purity).
The tree reaches its maximum allowed depth.
A node contains fewer than the minimum required samples to split.

Phase 2: Making Predictions (Inference)

Once built, the tree is a static set of rules. Predicting the label of a new, unseen data point is incredibly fast and operates like a game of 20 Questions.

                      [ START: Root Node ]
                          Is Outlook?
                         /      |     \
                Overcast/       |Sunny \Rain
                       /        |       \__________
                      v         v                  v
               (Leaf Node)  [ Decision Node ]   (Leaf Node)
                 PLAY: YES     Is Humidity?      PLAY: NO
                             /         \
                        High/           \Normal       
                           /             \
                          v               v
                    (Leaf Node)       (Leaf Node)
                      PLAY: NO         PLAY: YES
                         ▲
                         │
             [ NEW DAY LANDS HERE ]
             Prediction: PLAY = NO

Step 1: Enter at the Root

The new data point enters the tree at the very top Root Node.

Step 2: Evaluate the Condition

The model checks the data point's value against the node's rule (e.g., If the node asks Outlook = Sunny? and the data point's outlook is Sunny, the answer is Yes).

Step 3: Follow the Branch

Based on the answer, the data point travels down the corresponding Yes or No branch to the next node.

Step 4: Repeat Until a Leaf is Reached

The model evaluates the next internal node's condition and continues routing the data down the branches until it can go no further.

Step 5: Assign the Final Value

Once the data point lands in a terminal Leaf Node, the prediction is made:

For Classification: It assigns the majority class of the training samples in that leaf.
For Regression: It assigns the average numerical value of the training samples in that leaf.

Calculate Metric Purity

Entropy

Entropy is a metric that measures the amount of impurity, disorder, or randomness in a dataset.
In information theory, it quantifies how much uncertainty there is in a group of items.

H (S) = - i = 1 \sum c p_{i} lo g_{2} (p_{i})

Where:
(p_{i}): The proportion (probability) of data points belonging to class (i).
Scale:

(0.0) (Perfectly Pure): All samples belong to a single class (e.g., a node with only "Yes" entries).There is zero uncertainty.
(1.0) (Maximum Disorder): The classes are split perfectly down the middle in a two-class system (e.g., 50% "Yes" and 50% "No").

Gini Impurity

Gini Impurity measures the probability that a randomly chosen element from a node would be incorrectly labeled if it were randomly classified according to the distribution of targets in that node.

Gini = 1 - i = 1 \sum c (p_{i})^{2}

Scale:

(0.0) (Perfectly Pure): Every single element belongs to one class. The chance of misclassification is zero.
(0.5) (Maximum Disorder): In a two-class system, the data is split 50/50. You have a 50% chance of guessing the wrong label.

Information Gain

Information Gain is the metric used to decide which feature to split on. It measures the reduction in entropy (or disorder) after a dataset is split based on a specific attribute.

Information Gain (S, A) = H (S) - v \in Values (A) \sum \frac{∣ S _{v} ∣}{∣ S ∣} H (S_{v})

H (S) : Baseline entropy of the parent dataset before the split.

v \in Values (A) \sum \frac{∣ S _{v} ∣}{∣ S ∣} H (S_{v}) : Weighted average entropy of all child nodes created by the split.

Goal: The decision tree algorithm calculates the Information Gain for every available feature and selects the feature that yields the highest value, as it cleans up the data the fastest.

Mathematical operarion in a decision tree

To find the absolute best starting point (the root node) for your dataset, we must calculate the Metric Purity for the entire dataset and evaluate potential splits. We will use Entropy and Information Gain to find the most optimal split.
Here is the step-by-step mathematical calculation:

Keeping in mind

Total Samples ((S)): 4 days (D1, D2, D3, D4)
Target Classes ("Play?"): 2 Yes, 2 No

1. Calculate Base Entropy (Total Dataset)

Before making any splits, we calculate the initial disorder ((H)) of the 4 days using the entropy formula:

H (S) = - p_{yes} lo g_{2} (p_{yes}) - p_{no} lo g_{2} (p_{no})

Since 2 out of 4 are "Yes"

(p_{yes} = \frac{2}{4} = 0.5)

and 2 out of 4 are "No"

(p_{no} = \frac{2}{4} = 0.5) :

H (S) = - (0.5 \times lo g_{2} (0.5)) - (0.5 \times lo g_{2} (0.5))

H (S) = - (- 0.5) - (- 0.5)

H (S) = 1.0

The base entropy is (1.0), meaning the dataset is perfectly mixed and completely impure.

2. Evaluate Split on "Outlook"

The feature Outlook splits our 4 days into three distinct branches: Sunny (2 days), Overcast (1 day), and Rain (1 day).

Calculate Entropy for Each Branch:

Sunny Branch (D1, D2): Contains 1 "No" (D1) and 1 "Yes" (D2).

H (Sunny) = - (\frac{1}{2} lo g_{2} \frac{1}{2}) - (\frac{1}{2} lo g_{2} \frac{1}{2}) = 1.0

Overcast Branch (D4): Contains 1 "Yes". It is perfectly pure.

H (Overcast) = - (1 lo g_{2} 1) - (0) = 0.0

Rain Branch (D3): Contains 1 "No". It is perfectly pure.

H (Rain) = - (0) - (1 lo g_{2} 1) = 0.0

Calculate Remainder Entropy (Weighted Average):

Remainder (Outlook) = \frac{2}{4} (1.0) + \frac{1}{4} (0.0) + \frac{1}{4} (0.0) = 0.5

Calculate Information Gain:

Information Gain = Base Entropy - Remainder

IG (Outlook) = 1.0 - 0.5 = 0.50

3. 3. Evaluate Split on "Humidity"

The feature Humidity splits our 4 days into two branches: High (3 days) and Normal (1 day).

Calculate Entropy for Each Branch:

High Branch (D1, D3, D4): Contains 2 "No" (D1, D3) and 1 "Yes" (D4).

H (High) = - (\frac{1}{3} lo g_{2} \frac{1}{3}) - (\frac{2}{3} lo g_{2} \frac{2}{3}) \approx 0.918

Normal Branch (D2): Contains 1 "Yes". Perfectly pure.(H(\text{Normal})=0.0)

H (Normal) = 0.0

Calculate Remainder Entropy (Weighted Average):

Remainder (Humidity) = \frac{3}{4} (0.918) + \frac{1}{4} (0.0) \approx 0.689

Calculate Information Gain:

IG (Humidity) = 1.0 - 0.689 = 0.311

4. Evaluate Split on "Wind"

The feature Wind splits our 4 days into two branches: Weak (3 days) and Strong (1 day).

Calculate Entropy for Each Branch:

Weak Branch (D1, D2, D4): Contains 2 "Yes" (D2, D4) and 1 "No" (D1).

H (Weak) = - (\frac{2}{3} lo g_{2} \frac{2}{3}) - (\frac{1}{3} lo g_{2} \frac{1}{3}) \approx 0.918

Strong Branch (D3): Contains 1 "No". Perfectly pure

H (Strong) = 0.0

Calculate Remainder Entropy (Weighted Average):

Remainder (Wind) = \frac{3}{4} (0.918) + \frac{1}{4} (0.0) \approx 0.689

Calculate Information Gain:

IG (Wind) = 1.0 - 0.689 = 0.311

Feature	Base Entropy	Remainder Entropy	Information Gain (Purity Score)
Outlook	1.0	0.500	0.500 (Winner)
Humidity	1.0	0.689	0.311
Wind	1.0	0.689	0.311

Outlook provides the absolute highest Information Gain (0.500). Therefore, the algorithm chooses Outlook as the Root Node for Phase 1 because it cleans up the disorder of the dataset faster than any other feature.

Understanding Logistic Regression

Kelvin Muthama (Tush) — Mon, 20 Jul 2026 10:21:35 +0000

Imagine you own a red wine company. You need a reliable way to ensure every bottle meets high quality standards before shipping. Instead of waiting for human tasters, you use a machine learning tool called logistic regression. This statistical model is perfect for predicting a binary choice like good or bad. It looks at chemical measurements to calculate the exact probability that a wine batch will succeed or fail.
Your dataset serves as the training manual for this model. You feed it inputs like volatile acidity, alcohol levels, and pH to predict the final quality label.
For example, the model learns that high volatile acidity shifts the probability toward a bad rating. By running new batches through this logistic regression formula, you can instantly flag flawed barrels in the lab, adjust your ingredients early, and protect your brand's reputation.

What is classification?

Classification is a machine learning process that automatically sorts data into distinct, predefined categories or labels.
It takes input features (like the chemical measurements of a wine) and maps them to a specific group based on patterns learned from historical data.

Core Types of Classification

Binary Classification: Sorting data into exactly two groups (e.g., Good vs. Bad wine, Spam vs. Not Spam email).
Multiclass Classification: Sorting data into three or more distinct groups (e.g., classifying a wine's grape type as Merlot, Cabernet, or Syrah).
Multilabel Classification: Assigning multiple labels to a single piece of data (e.g., tagging a wine bottle as both "Award-Winning" and "Organic").

Common Classification Algorithms

Logistic Regression: Predicts probabilities for binary groups using an S-curve.
Decision Trees: Uses a flowchart-like tree structure of yes/no questions to split data.
Random Forest: Combines multiple decision trees together to make a more accurate, democratic decision.
Support Vector Machines (SVM): Finds the optimal boundary line that widely separates different categories.

What is logistic regression?

Logistic regression is a statistical and machine learning model used to predict a categorical outcome with a fixed number of choices, most commonly a binary outcome like yes/no, true/false, or good/bad.

Why Linear Regression fails?

Linear regression fails when the true relationship in the data cannot be accurately represented by a straight line, or when its strict mathematical assumptions are violated:

When there is multicolinearity.
When there is heteroscedasticity.
When there is non-linearity.
When the independent variable is categorical.

The Sigmoid Function

The Sigmoid Function is a mathematical formula that squashes any real-numbered value into a strict range between 0 and 1.
In machine learning, it is the core engine behind logistic regression, acting as the bridge that converts raw numerical calculations into clean, usable probabilities.

S (z) = \frac{1}{1 + e ^{- z}}

(z): The raw input value (the score calculated from your data features).
(e): Euler's constant (roughly 2.718).
(S(z)): The output probability (always between 0 and 1).

When our logistic regression model evaluates a batch of red wine, it multiplies the chemical inputs (like volatile acidity and alcohol) by specific weights to get a single raw score ((z)).
If a batch is high in vinegar-like volatile acid, the model calculates a negative score e.g. (z = -4).
Passing (-4) into the sigmoid function yields an output close to 0.02 (a 2% chance of being good).
Because 2% is below your 50% quality threshold, the system automatically classifies the batch as bad.

How predictions are made

Let's look at exactly how a trained logistic regression model evaluates a new batch of red wine in your cellar.
Imagine our model learned three specific things during its training phase:
Baseline Bias (β₀): -2.0 (The starting point)
Alcohol Weight (β₁): +1.5 (Higher alcohol strongly increases quality)
Volatile Acidity Weight (β₂): -4.0 (Higher volatile acidity strongly decreases quality)
Here is the step-by-step pipeline the model uses to predict if a new batch is Good (1) or Bad (0).

Step 1: Calculate the Raw Score (z)
A new barrel comes into the lab with the following chemical measurements:

Alcohol (x₁): 12.0%
Volatile Acidity (x₂): 0.3 g/dm³ The model Plugs these numbers into the linear log-odds equation: $z = - 2.0 + (1.5 \cdot 12.0) + (- 4.0 \cdot 0.3)$ $z = - 2.0 + 18.0 - 1.2$ $z = 14.8$ The raw score is 14.8. Because this number is highly positive, it indicates a strong lean toward a good classification.

Step 2: Convert to a Probability (P)
Next, the model squashes that raw score of 14.8 through the Sigmoid Function to turn it into an exact percentage between 0 and 1:

P = \frac{1}{1 + e ^{- z}}

P = \frac{1}{1 + e ^{- 14.8}}

P = \frac{1}{1 + 0.00000037}

P \approx 0.9999996

The model calculates a 99.9% probability that this specific wine batch is high quality.

Step 3: Apply the Quality Threshold
Finally, the model compares the probability against your standard decision boundary threshold of 0.50 (50%):Is 0.999 ≥ 0.50? Yes.
The model instantly outputs the final hard classification label: Good.
This barrel is cleared for bottling and shipping.

Hands on logistic regession using python

Import necessary libraries

import pandas as pd
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (classification_report, roc_auc_score)
from statsmodels.stats.outliers_influence import variance_inflation_factor

Reading the data

# Reading our csv file as a dataframe.
dataframe = pd.read_csv('DATA/wine.csv')
# displaying the first five rows of the dataset
dataframe.head()

fixed acidity	volatile acidity	citric acid	residual sugar	chlorides	free sulfur dioxide	total sulfur dioxide	density	pH	sulphates	alcohol	quality
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	bad
7.8	0.88	0.00	2.6	0.098	25.0	67.0	0.9968	3.20	0.68	9.8	bad
7.8	0.76	0.04	2.3	0.092	15.0	54.0	0.9970	3.26	0.65	9.8	bad
11.2	0.28	0.56	1.9	0.075	17.0	60.0	0.9980	3.16	0.58	9.8	good
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	bad

Data information and summary

dataframe.info()

#	Column	Non-Null Count	Dtype
0	fixed acidity	1599 non-null	float64
1	volatile acidity	1599 non-null	float64
2	citric acid	1599 non-null	float64
3	residual sugar	1599 non-null	float64
4	chlorides	1599 non-null	float64
5	free sulfur dioxide	1599 non-null	float64
6	total sulfur dioxide	1599 non-null	float64
7	density	1599 non-null	float64
8	pH	1599 non-null	float64
9	sulphates	1599 non-null	float64
10	alcohol	1599 non-null	float64
11	quality	1599 non-null	object

dataframe.describe()

	fixed acidity	volatile acidity	citric acid	residual sugar	chlorides	free sulfur dioxide	total sulfur dioxide	density	pH	sulphates	alcohol	quality
count	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000
mean	8.319637	0.527821	0.270976	2.538806	0.087467	15.874922	46.467792	0.996747	3.311113	0.658149	10.422983	0.534709
std	1.741096	0.179060	0.194801	1.409928	0.047065	10.460157	32.895324	0.001887	0.154386	0.169507	1.065668	0.498950
min	4.600000	0.120000	0.000000	0.900000	0.012000	1.000000	6.000000	0.990070	2.740000	0.330000	8.400000	0.000000
25%	7.100000	0.390000	0.090000	1.900000	0.070000	7.000000	22.000000	0.995600	3.210000	0.550000	9.500000	0.000000
50%	7.900000	0.520000	0.260000	2.200000	0.079000	14.000000	38.000000	0.996750	3.310000	0.620000	10.200000	1.000000
75%	9.200000	0.640000	0.420000	2.600000	0.090000	21.000000	62.000000	0.997835	3.400000	0.730000	11.100000	1.000000
max	15.900000	1.580000	1.000000	15.500000	0.611000	72.000000	289.000000	1.003690	4.010000	2.000000	14.900000	1.000000

correlation = dataframe.corr()
sns.heatmap(correlation, cmap= 'coolwarm')

Preparing the data

# Converting the quality column to zeros and ones
dataframe['quality'] = dataframe['quality'].map({'good': 1, 'bad': 0})
dataframe.head()

fixed acidity	volatile acidity	citric acid	residual sugar	chlorides	free sulfur dioxide	total sulfur dioxide	density	pH	sulphates	alcohol	quality
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	0
7.8	0.88	0.00	2.6	0.098	25.0	67.0	0.9968	3.20	0.68	9.8	0
7.8	0.76	0.04	2.3	0.092	15.0	54.0	0.9970	3.26	0.65	9.8	0
11.2	0.28	0.56	1.9	0.075	17.0	60.0	0.9980	3.16	0.58	9.8	1
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	0

# Splitting the data to for the independent variable (X) and dependent variable (y)
X = dataframe.iloc[:,0:11]
y = dataframe.iloc[:, -1]

# Performing train test split
X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = .8, random_state = 42)

# Standardizing our data to a mea of 0 and a variance of 1
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Build the model in Python

# Defining the model
model = LogisticRegression()

# Fittinng the X train and y train for the model to learn
model.fit(X_train_scaled, y_train)

# making predictions based on the X test. 
y_pred = model.predict(X_test_scaled)
y_pred

array([0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0], dtype=int64)

Evaluate performance

print(roc_auc_score(y_test,y_pred))

0.7410555093307976

print(classification_report(y_test,y_pred))

	precision	recall	f1-score	support
0	0.69	0.74	0.72	141
1	0.79	0.74	0.76	179
accuracy			0.74	320
macro avg	0.74	0.74	0.74	320
weighted avg	0.74	0.74	0.74	320

Visualizations

# Generate predictions and matrix
y_pred = model.predict(X_test_scaled)
cm = confusion_matrix(y_test, y_pred)

# Plot Heatmap
plt.figure(figsize=(6, 5))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', 
            xticklabels=['Bad Wine (0)', 'Good Wine (1)'], 
            yticklabels=['Bad Wine (0)', 'Good Wine (1)'])
plt.title('Confusion Matrix for Wine Quality')
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.show()

Evaluation metrics used to measure the performance of logistic regression model.

Accuracy

The overall correctness of the model. It calculates the ratio of correct predictions (both positive and negative) to total predictions. Use this when classes are balanced.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Precision

Indicates the quality of the model's positive predictions. It answers: "Out of all the instances the model predicted as positive, how many were actually correct?" Use this when false positives are costly.

Precision = TP / (TP + FP)

Recall

Measures the model's ability to identify all positive instances. It answers: "Out of all the actual positive instances, how many did the model find?" Use this when false negatives are costly

Recall = TP / (TP + FN)

F1 Score

The harmonic mean of precision and recall. It balances both metrics, which is crucial when dealing with imbalanced datasets where one class outnumbers the other.

F1 Score = 2 * (Precision * Recall) / (Precision + Recall)

Advantages & Limitations

Advantages of Logistic Regression

Highly Interpretable: You can easily see exactly how much each ingredient (like alcohol or volatile acidity) impacts the final classification by looking at the model's weights.
Outputs Probabilities: It does not just give a blind "good" or "bad" answer; it provides a precise confidence percentage (e.g., a 72% chance of being a good wine).
Computationally Efficient: The model trains incredibly fast and requires very little computing power, making it highly scalable for production lines.
No Assumptions About Feature Distributions: Unlike linear regression, it does not require input features to be normally distributed or have equal variance.
Low Risk of Overfitting: In clean, simple datasets, it is less prone to memorizing noise compared to more complex algorithms like deep neural networks.

Limitations of Logistic Regression

Assumes Linearity in the Log-Odds: It relies on a straight-line boundary to split data. If your high-quality wines require a highly complex, curved combination of traits, this model will underfit.
Vulnerable to Multicollinearity: If your chemical features highly correlate with each other (such as fixed acidity and pH closely tracking together), the model's weights become highly unstable and difficult to interpret.
Requires Independent Observations: The data points must be independent. For example, if you sample multiple times from the exact same wine barrel, it can artificially skew the model's confidence.
Struggles with Complex Data Patterns: It cannot easily handle text, images, or deeply complex data structures without extensive manual feature engineering.
Highly Sensitive to Outliers: Extreme anomalies or mislabeled lab readings can heavily pull the decision boundary line, leading to incorrect classifications for normal batches.

Real-world applications

Logistic regression is one of the most widely deployed machine learning algorithms because businesses need to make binary decisions based on risk and probability.
Here is how different industries use it to make automated, real-world choices:

1. Finance & Banking

Credit Risk Assessment: Banks feed a borrower's credit score, income, and debt levels into a model to predict whether they will default (1) or pay back (0) a loan.
Credit Card Fraud Detection: Payment processors analyze purchase location, transaction size, and timing to instantly flag a charge as fraudulent or legitimate.

2. Healthcare & Medicine

Disease Diagnosis: Doctors input a patient's biometrics (blood pressure, cholesterol, age) to determine the probability of a patient having a specific condition, such as classifying a tumor as malignant or benign.
Patient Readmission Risk: Hospitals predict whether a discharged patient has a high probability of being readmitted within 30 days, allowing them to allocate preventative home-care resources.

3. Marketing & E-Commerce

Customer Churn Prediction: Telecom and software companies analyze usage drops or customer service complaints to identify users at risk of canceling their subscription.
Click-Through Rate (CTR) Optimization: Digital advertising platforms predict the exact probability that a specific user will click an ad or scroll past it, ensuring ads are only shown to relevant audiences.
Conclusion

[Boost]

Kelvin Muthama (Tush) — Sun, 12 Jul 2026 21:34:07 +0000

Faith Njenga

Jul 12

Commitment Issues: Tracking Changes, Romantic Promises, and Surviving the Git Staging Crate

#git #github #analytics #dataengineering

12 min read

Understanding Linear Regression

Kelvin Muthama (Tush) — Sun, 12 Jul 2026 21:32:37 +0000

What is linear regression?

Linear regression from the term regression analysis, is a technique used to model and analyze the relationship between a dependent variable (the outcome you want to predict) and one or more independent variables (the factors influencing the outcome).
linear regression predicts and defines relationships between continuous variables.
What is a continuous variable?
This is a number that can take on any value within a given range, including decimals and fractions.
Count of items: 10 books, 100 Sales, 200 Students
Measures: 20.004Kg, 120Km, 20.2Pounds

Problem it solves

1.Business Problem

With the power of linear regression we can help a business:

Predict sales and come up with the stategies to improve on them.
Understand relationship between their sales and factors like:
- Count of items
- day of the week
- discount levels.
Estimate customer lifetime value.

2.Health Care Problem

Linear regression in the field of health care can help:

Predict the weight of a patient
Understand the relationship between a patient's weight and:
- Sugar levels.
- Genes

Intution behind linear regression

Take an example of a Retail shop, where The price of an item like milk depends on its quantity:
PLotting it would look like:

On the x-axis (horizontal axis), enter the milk quantity.E.g: 1L, 500Ml, 3L
On the y- axis (Vertical axis), enter the price of the milk. The points won't fit in a straight line but will be scattered in the chat.
Now take a ruler and try drawing a straight line on the points in such a way that it passes at the centre of the data. This is the line of best fit.

The model adjusts only two things:

Intercept — where the line starts on the Y-axis.
Slope — how steep the line is.

These three steps are what linear regression is trying to do.

Terms in linear regression

From the term linear in analysis, linear regression is a relationship that forms a straight line:

y = mx + c

An increase in a unit in x leads to a unit increase in y

Dependent variable (x)
This is a variable or factor that influences the outcome.
Independent variable (y)
It is the outcome to predict.
Slope (m)
The number that shows how much your output (y) changes when your input (x) goes up by one unit.
Intercept (c)
is the expected value of your output (y) when your input (x) is exactly zero. It is the starting point where your regression line crosses the vertical Y-axis.
Fit
Describes how well your straight line matches the actual data points. A good fit means your line passes very close to most of your data points, making your predictions highly accurate.
Residuals
These are mistakes made by the model. Which is gotten by subtracting the actual values and the predicted values

An analogy to the terms

The data provided below is a supermarket sales data:

Order No	Order Date	Customer Name	Ship Date	Retail Price (USD)	Order Quantity	Tax (USD)	Total (USD)
1001	1/1/2024	John Smith	1/3/2024	49.99	2	9.998	109.978
1002	1/1/2024	Jane Doe	1/4/2024	29.99	1	2.999	32.989
1003	1/2/2024	Michael Johnson	1/7/2024	99.99	3	29.997	329.967
1004	1/2/2024	Emily Brown	1/3/2024	19.99	4	7.996	87.956
1005	1/3/2024	David Wilson	1/8/2024	149.99	1	14.999	164.989

We are supposed to predict the future total price making it our dependent variable (y). We should consider factors like Retail Price (USD), Tax (USD) which are our independent variables (x).

After making our pediction based on (x), we can plot the predicted total price against the actual total price. By plotting a straight line through the data points. A good fit means our line passes very close to most of your data points, making your predictions highly accurate.

If the line of best fit shows a Positive slope then an increase in (x) leads to an increase in (y) but if it shows a negative slope then an increase in (x) leads to a decreases in (y) and vise versa.

If the line of best fit does not touch all the data points the the model has some errors which are the residuals.

Assumptions in linear regression

Linear regression relies on five key assumptions to ensure accurate, reliable, and interpretable results.

1. Linearity

The relationship between the dependent variable (y) and the independent variables (X) must be linear.

2. Homoscedasticity (Equal Variance)

The variance of the error terms should be constant across all values of the independent variables. If the errors spread out or bunch together as X increases, it is called heteroscedasticity, which leads to unreliable standard errors.

3. No multicollinearity

In multiple linear regession, independent variables are should not to have high corelation with another. High multicollinearity makes it difficult to isolate the individual effect of each predictor on the dependent variable.

4. Normality of residuals

The residauls should be normally distributed around the mean zero. This ensures the p-value and the confidence interval are reliable for hypothesis testing.

5. Independence of Errors

The residuals must be independent of one another. This is especially crucial for time-series data, where consecutive observations should not influence each other (no autocorrelation)

How to check for the assumptions

Linearity

Scatterplots or residual vs. fitted plots. If you see a curved pattern, a non-linear model or data transformation is required.

Homoscedasticity (Equal Variance)

Residual vs. fitted values plot. Ideally, the points should be randomly scattered without any "cone" or "funnel" shape.

No multicollinearity

Correlation matrices or Variance Inflation Factor (VIF) scores. A VIF value over 5 to 10 typically indicates an issue.

Normality of residuals

Normal Q-Q plots, histograms of residuals, or statistical tests like the Shapiro-Wilk test.

Independence of Errors

Plotting residuals against time/index or using the Durbin-Watson test.

Hypothesis testing in linear regression

Hypothesis testing in linear regression determines if there is a statistically significant relationship between the predictor variables and the response variable. It primarily uses t-tests for individual coefficients (to see if they are zero) and an F-test for the overall model's validity.

t-test implimentation

Null Hypothesis states that the variable has no linear effect on the response.
Alternative Hypothesis states that the variable has a significant linear effect on the response.

f-test implimentation

Null Hypothesis states that none of the predictors are useful.
Alternative Hypothesis states that at least one predictor is significantly associated with the response.

Simple linear regression

Simple linear regression method used to predict the value of one variable based on the value of a different variable.

Y = β₀ + β₁X

Where:

Y (Dependent Variable): The main thing you are trying to predict or understand.
X (Independent Variable): The input data you are using to make the prediction.
β₀ (Intercept): The starting value of Y when X is zero.
β₁ (Slope): How much Y changes for every single step up in X.

Use case

From the data we discussed above, the only difference we make is that we only concider one factor like Order Quantity to predict the Total (USD).

Multiple linear regression

It is method used to predict the value of one variable based on two or more different input variables.

Y = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ

plaintext

Where:

Y: The main thing you are trying to predict (the output).
β₀ (Intercept): The starting baseline value when all your inputs are zero.
X₁, X₂, X₃: Your different input variables.
β₁, β₂, β₃ (Slopes): The individual impact of each input. For example, β₁ tells you how much Y changes when X₁ goes up, assuming all other inputs stay exactly the same.

Use case

We should consider factors like Retail Price (USD), Tax (USD) and Order Quantity to predict the Total (USD).

Building our first linear regression model in Python

1. Installing the required libraries

pip install numpy pandas matplotlib seaborn scikit-learn

2. Importing the python libraries

# Importing pandas for data analysis
import pandas as pd

# Importing numpy for array manipulation
import numpy as np

# Import seabon and matplotlib for visaulization
import matplotlib.pyplot as plt
import seaborn as sns

# Importing train_test_split for extracting train and test data
from sklearn.model_selection import train_test_split

# Creating LinearRegression for model creation and continous variables prediction
from sklearn.linear_model import LinearRegression

# Importing mean_absolute_error, mean_squared_error, r2_score for model evaluation
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

3. Reading our excel file as a pandas dataftame

# Reading our excel file as a pandas dataftame
dataframe = pd.read_excel('DATA/Supermarket-Sales-Sample-Data.xlsx', skiprows=7)
# Displaying the first 5 rows in the dataframe
dataframe.head()

Order No	Order Date	Customer Name	Ship Date	Retail Price (USD)	Order Quantity	Tax (USD)	Total (USD)
1001	1/1/2024	John Smith	1/3/2024	49.99	2	9.998	109.978
1002	1/1/2024	Jane Doe	1/4/2024	29.99	1	2.999	32.989
1003	1/2/2024	Michael Johnson	1/7/2024	99.99	3	29.997	329.967
1004	1/2/2024	Emily Brown	1/3/2024	19.99	4	7.996	87.956
1005	1/3/2024	David Wilson	1/8/2024	149.99	1	14.999	164.989

4. Showing the correlation between columns

# Calculating the correlation betwee numerical columns
corr = dataframe.corr()
# Displaying the correlation using a heatmap
sns.heatmap(data=corr, cmap='coolwarm')

The correlation gave the following insights:

Order number had no relationship with other columns
Retail Price had a negative relationship with Order Quantity but a postive one with the Tax and Total columns
Order Quantity had a negative relationship with Retail Price, Tax and Total columns
Tax had a negative relationship with Order Quantity but a postive one with the Retail Price and Total columns
Total had a negative relationship with Order Quantity but a postive one with the Tax and Retail Price columns

5. Selecting columns fit for modelling

Before training the predictive model, unnecessary columns are removed to improve model performance and reduce noise. Since machine learning algorithms require numerical inputs, categorical date columns are transformed into numerical features using one-hot encoding.The preprocessing steps include:

Removing columns that do not contribute to prediction.
Preparing the dataset for model training.

# Dropping the unwanted colums during modeling and prediction
dataframe = dataframe.drop(columns=['Order No', 'Customer Name', 'Order Date', 'Ship Date'])

# Creating the predictors and the independent variables for modelling
X = dataframe.drop('Total (USD)', axis=1)
y = dataframe['Total (USD)']

python

6. Splitting the data for training and testing

The cleaned dataset is divided into predictor variables (X) and the target variable (y). The data is then split into training and testing sets using a 70:30 ratio.

# Splitting our dataset for trainig the model and testing it's perfomance
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=.7, random_state=42)

7. Building the Linear Regression Model

A Linear Regression model is trained using the training data, after which predictions are generated on the test set. These predictions will later be evaluated to determine how well the model estimates supermarket sales.

# assigning our LinearRegression model
model = LinearRegression()

# Training the model using X_train and y_train to help it uderstand the predictors and predicted values
model.fit(X_train, y_train)

# Performing prediction on the X_test to get predicted variables
y_pred = model.predict(X_test)
y_pred

array([142.989, 109.978, 142.989, 164.989, 219.989, 219.989, 164.945, 164.989, 219.989, 164.967, 142.989, 219.989, 142.989, 175.978, 131.967, 164.967, 142.989, 131.967, 164.967, 175.978, 153.978])

8. Performing model evaluation

To determine the effectiveness of the Linear Regression model, several evaluation metrics are calculated:

Mean Absolute Error (MAE): Measures the average prediction error.
Root Mean Squared Error (RMSE): Penalizes larger prediction errors and indicates the model's overall accuracy.
R-squared (R²): Represents the proportion of variance in the target variable explained by the model.

Together, these metrics provide a comprehensive assessment of the model's predictive performance.

# 1. Mean Absolute Error (MAE)
mae = mean_absolute_error(y_test, y_pred)

# 2. Root Mean Squared Error (RMSE)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))

# 3. R-squared Score (R²)
r2 = r2_score(y_test, y_pred)

print(f"Mean Absolute Error: {mae:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")
print(f"R-squared Score: {r2:.2f}")

Mean Absolute Error: 0.00 Root Mean Squared Error: 0.00 R-squared Score: 1.00
With uniform transaction math in clean sample data, errors will compute to 0.00 and R² will equal a perfect 1.00, validating that the model successfully extracted the exact pricing rules

9. Visualizing Model Predictions

A scatter plot is used to compare the actual sales values against the model's predicted values.
A 45-degree reference line is included to represent perfect predictions.The closer the data points lie to this reference line, the better the model's predictive accuracy.
This visualization provides an intuitive way to assess how well the Linear Regression model performs on unseen data.

# Defining the plot's configuration
plt.figure(figsize=(8, 6))
sns.set_theme(style="whitegrid")

# 1. Plot actual answers vs predicted answers
sns.scatterplot(x=y_test, y=y_pred, color='purple', alpha=0.6, label='Predictions')

# 2. Draw a perfect 45-degree reference line
# (If your model is perfect, all dots will sit exactly on this line)
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 
         color='orange', linewidth=2.5, linestyle='--', label='Perfect Fit')

plt.title('Actual Values vs. Predicted Values', fontsize=14, pad=15)
plt.xlabel('Actual True Values', fontsize=12)
plt.ylabel('Model Predictions', fontsize=12)
plt.legend()
plt.show()