Understanding Logistic Regression

Imagine you own a red wine company. You need a reliable way to ensure every bottle meets high quality standards before shipping. Instead of waiting for human tasters, you use a machine learning tool called logistic regression. This statistical model is perfect for predicting a binary choice like good or bad. It looks at chemical measurements to calculate the exact probability that a wine batch will succeed or fail.
Your dataset serves as the training manual for this model. You feed it inputs like volatile acidity, alcohol levels, and pH to predict the final quality label.
For example, the model learns that high volatile acidity shifts the probability toward a bad rating. By running new batches through this logistic regression formula, you can instantly flag flawed barrels in the lab, adjust your ingredients early, and protect your brand's reputation.

What is classification?

Classification is a machine learning process that automatically sorts data into distinct, predefined categories or labels.
It takes input features (like the chemical measurements of a wine) and maps them to a specific group based on patterns learned from historical data.

Core Types of Classification

Binary Classification: Sorting data into exactly two groups (e.g., Good vs. Bad wine, Spam vs. Not Spam email).
Multiclass Classification: Sorting data into three or more distinct groups (e.g., classifying a wine's grape type as Merlot, Cabernet, or Syrah).
Multilabel Classification: Assigning multiple labels to a single piece of data (e.g., tagging a wine bottle as both "Award-Winning" and "Organic").

Common Classification Algorithms

Logistic Regression: Predicts probabilities for binary groups using an S-curve.
Decision Trees: Uses a flowchart-like tree structure of yes/no questions to split data.
Random Forest: Combines multiple decision trees together to make a more accurate, democratic decision.
Support Vector Machines (SVM): Finds the optimal boundary line that widely separates different categories.

What is logistic regression?

Logistic regression is a statistical and machine learning model used to predict a categorical outcome with a fixed number of choices, most commonly a binary outcome like yes/no, true/false, or good/bad.

Why Linear Regression fails?

Linear regression fails when the true relationship in the data cannot be accurately represented by a straight line, or when its strict mathematical assumptions are violated:

• When there is multicolinearity.
• When there is heteroscedasticity.
• When there is non-linearity.
• When the independent variable is categorical.

The Sigmoid Function

The Sigmoid Function is a mathematical formula that squashes any real-numbered value into a strict range between 0 and 1.
In machine learning, it is the core engine behind logistic regression, acting as the bridge that converts raw numerical calculations into clean, usable probabilities.

$$S(z) = \frac{1}{1 + e^{-z}}$$

(z): The raw input value (the score calculated from your data features).
(e): Euler's constant (roughly 2.718).
(S(z)): The output probability (always between 0 and 1).

When our logistic regression model evaluates a batch of red wine, it multiplies the chemical inputs (like volatile acidity and alcohol) by specific weights to get a single raw score ((z)).
If a batch is high in vinegar-like volatile acid, the model calculates a negative score e.g. (z = -4).
Passing (-4) into the sigmoid function yields an output close to 0.02 (a 2% chance of being good).
Because 2% is below your 50% quality threshold, the system automatically classifies the batch as bad.

How predictions are made

Let's look at exactly how a trained logistic regression model evaluates a new batch of red wine in your cellar.
Imagine our model learned three specific things during its training phase:
Baseline Bias (β₀): -2.0 (The starting point)
Alcohol Weight (β₁): +1.5 (Higher alcohol strongly increases quality)
Volatile Acidity Weight (β₂): -4.0 (Higher volatile acidity strongly decreases quality)
Here is the step-by-step pipeline the model uses to predict if a new batch is Good (1) or Bad (0).

Step 1: Calculate the Raw Score (z)
A new barrel comes into the lab with the following chemical measurements:

• Alcohol (x₁): 12.0%
• Volatile Acidity (x₂): 0.3 g/dm³ The model Plugs these numbers into the linear log-odds equation:
  $$\begin{aligned} z &= -2.0 + (1.5 \cdot 12.0) + (-4.0 \cdot 0.3) \ \end{aligned}$$
  $$\begin{aligned} z &= -2.0 + 18.0 - 1.2 \ \end{aligned}$$
  $$\begin{aligned} z &= 14.8 \end{aligned}$$
  The raw score is 14.8. Because this number is highly positive, it indicates a strong lean toward a good classification.

Step 2: Convert to a Probability (P)
Next, the model squashes that raw score of 14.8 through the Sigmoid Function to turn it into an exact percentage between 0 and 1:

$$\begin{aligned} P &= \frac{1}{1 + e^{-z}} \ \end{aligned}$$

$$\begin{aligned} P &= \frac{1}{1 + e^{-14.8}} \ \end{aligned}$$

$$\begin{aligned} P &= \frac{1}{1 + 0.00000037} \ \end{aligned}$$

$$\begin{aligned} P &\approx 0.9999996 \end{aligned}$$

The model calculates a 99.9% probability that this specific wine batch is high quality.

Step 3: Apply the Quality Threshold
Finally, the model compares the probability against your standard decision boundary threshold of 0.50 (50%):Is 0.999 ≥ 0.50? Yes.
The model instantly outputs the final hard classification label: Good.
This barrel is cleared for bottling and shipping.

Hands on logistic regession using python

Import necessary libraries

import pandas as pd
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (classification_report, roc_auc_score)
from statsmodels.stats.outliers_influence import variance_inflation_factor

Reading the data

# Reading our csv file as a dataframe.
dataframe = pd.read_csv('DATA/wine.csv')
# displaying the first five rows of the dataset
dataframe.head()

fixed acidity	volatile acidity	citric acid	residual sugar	chlorides	free sulfur dioxide	total sulfur dioxide	density	pH	sulphates	alcohol	quality
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	bad
7.8	0.88	0.00	2.6	0.098	25.0	67.0	0.9968	3.20	0.68	9.8	bad
7.8	0.76	0.04	2.3	0.092	15.0	54.0	0.9970	3.26	0.65	9.8	bad
11.2	0.28	0.56	1.9	0.075	17.0	60.0	0.9980	3.16	0.58	9.8	good
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	bad

Data information and summary

dataframe.info()

#	Column	Non-Null Count	Dtype
0	fixed acidity	1599 non-null	float64
1	volatile acidity	1599 non-null	float64
2	citric acid	1599 non-null	float64
3	residual sugar	1599 non-null	float64
4	chlorides	1599 non-null	float64
5	free sulfur dioxide	1599 non-null	float64
6	total sulfur dioxide	1599 non-null	float64
7	density	1599 non-null	float64
8	pH	1599 non-null	float64
9	sulphates	1599 non-null	float64
10	alcohol	1599 non-null	float64
11	quality	1599 non-null	object

dataframe.describe()

	fixed acidity	volatile acidity	citric acid	residual sugar	chlorides	free sulfur dioxide	total sulfur dioxide	density	pH	sulphates	alcohol	quality
count	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000	1599.000000
mean	8.319637	0.527821	0.270976	2.538806	0.087467	15.874922	46.467792	0.996747	3.311113	0.658149	10.422983	0.534709
std	1.741096	0.179060	0.194801	1.409928	0.047065	10.460157	32.895324	0.001887	0.154386	0.169507	1.065668	0.498950
min	4.600000	0.120000	0.000000	0.900000	0.012000	1.000000	6.000000	0.990070	2.740000	0.330000	8.400000	0.000000
25%	7.100000	0.390000	0.090000	1.900000	0.070000	7.000000	22.000000	0.995600	3.210000	0.550000	9.500000	0.000000
50%	7.900000	0.520000	0.260000	2.200000	0.079000	14.000000	38.000000	0.996750	3.310000	0.620000	10.200000	1.000000
75%	9.200000	0.640000	0.420000	2.600000	0.090000	21.000000	62.000000	0.997835	3.400000	0.730000	11.100000	1.000000
max	15.900000	1.580000	1.000000	15.500000	0.611000	72.000000	289.000000	1.003690	4.010000	2.000000	14.900000	1.000000

correlation = dataframe.corr()
sns.heatmap(correlation, cmap= 'coolwarm')

Preparing the data

# Converting the quality column to zeros and ones
dataframe['quality'] = dataframe['quality'].map({'good': 1, 'bad': 0})
dataframe.head()

fixed acidity	volatile acidity	citric acid	residual sugar	chlorides	free sulfur dioxide	total sulfur dioxide	density	pH	sulphates	alcohol	quality
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	0
7.8	0.88	0.00	2.6	0.098	25.0	67.0	0.9968	3.20	0.68	9.8	0
7.8	0.76	0.04	2.3	0.092	15.0	54.0	0.9970	3.26	0.65	9.8	0
11.2	0.28	0.56	1.9	0.075	17.0	60.0	0.9980	3.16	0.58	9.8	1
7.4	0.70	0.00	1.9	0.076	11.0	34.0	0.9978	3.51	0.56	9.4	0

# Splitting the data to for the independent variable (X) and dependent variable (y)
X = dataframe.iloc[:,0:11]
y = dataframe.iloc[:, -1]

# Performing train test split
X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = .8, random_state = 42)

# Standardizing our data to a mea of 0 and a variance of 1
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Build the model in Python

# Defining the model
model = LogisticRegression()

# Fittinng the X train and y train for the model to learn
model.fit(X_train_scaled, y_train)

# making predictions based on the X test. 
y_pred = model.predict(X_test_scaled)
y_pred

array([0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0,
1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0,
1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0,
1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1,
1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0,
1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0,
0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1,
1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1,
1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0,
0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0,
0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0,
1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1,
1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1,
0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1,
1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0], dtype=int64)

Evaluate performance

print(roc_auc_score(y_test,y_pred))

0.7410555093307976

print(classification_report(y_test,y_pred))

	precision	recall	f1-score	support
0	0.69	0.74	0.72	141
1	0.79	0.74	0.76	179
accuracy			0.74	320
macro avg	0.74	0.74	0.74	320
weighted avg	0.74	0.74	0.74	320

Visualizations

# Generate predictions and matrix
y_pred = model.predict(X_test_scaled)
cm = confusion_matrix(y_test, y_pred)

# Plot Heatmap
plt.figure(figsize=(6, 5))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', 
            xticklabels=['Bad Wine (0)', 'Good Wine (1)'], 
            yticklabels=['Bad Wine (0)', 'Good Wine (1)'])
plt.title('Confusion Matrix for Wine Quality')
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.show()

Evaluation metrics used to measure the performance of logistic regression model.

Accuracy

The overall correctness of the model. It calculates the ratio of correct predictions (both positive and negative) to total predictions. Use this when classes are balanced.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Precision

Indicates the quality of the model's positive predictions. It answers: "Out of all the instances the model predicted as positive, how many were actually correct?" Use this when false positives are costly.

Precision = TP / (TP + FP)

Recall

Measures the model's ability to identify all positive instances. It answers: "Out of all the actual positive instances, how many did the model find?" Use this when false negatives are costly

Recall = TP / (TP + FN)

F1 Score

The harmonic mean of precision and recall. It balances both metrics, which is crucial when dealing with imbalanced datasets where one class outnumbers the other.

F1 Score = 2 * (Precision * Recall) / (Precision + Recall)

Advantages & Limitations

Advantages of Logistic Regression

Highly Interpretable: You can easily see exactly how much each ingredient (like alcohol or volatile acidity) impacts the final classification by looking at the model's weights.
Outputs Probabilities: It does not just give a blind "good" or "bad" answer; it provides a precise confidence percentage (e.g., a 72% chance of being a good wine).
Computationally Efficient: The model trains incredibly fast and requires very little computing power, making it highly scalable for production lines.
No Assumptions About Feature Distributions: Unlike linear regression, it does not require input features to be normally distributed or have equal variance.
Low Risk of Overfitting: In clean, simple datasets, it is less prone to memorizing noise compared to more complex algorithms like deep neural networks.

Limitations of Logistic Regression

Assumes Linearity in the Log-Odds: It relies on a straight-line boundary to split data. If your high-quality wines require a highly complex, curved combination of traits, this model will underfit.
Vulnerable to Multicollinearity: If your chemical features highly correlate with each other (such as fixed acidity and pH closely tracking together), the model's weights become highly unstable and difficult to interpret.
Requires Independent Observations: The data points must be independent. For example, if you sample multiple times from the exact same wine barrel, it can artificially skew the model's confidence.
Struggles with Complex Data Patterns: It cannot easily handle text, images, or deeply complex data structures without extensive manual feature engineering.
Highly Sensitive to Outliers: Extreme anomalies or mislabeled lab readings can heavily pull the decision boundary line, leading to incorrect classifications for normal batches.

Real-world applications

Logistic regression is one of the most widely deployed machine learning algorithms because businesses need to make binary decisions based on risk and probability.
Here is how different industries use it to make automated, real-world choices:

1. Finance & Banking

Credit Risk Assessment: Banks feed a borrower's credit score, income, and debt levels into a model to predict whether they will default (1) or pay back (0) a loan.
Credit Card Fraud Detection: Payment processors analyze purchase location, transaction size, and timing to instantly flag a charge as fraudulent or legitimate.

2. Healthcare & Medicine

Disease Diagnosis: Doctors input a patient's biometrics (blood pressure, cholesterol, age) to determine the probability of a patient having a specific condition, such as classifying a tumor as malignant or benign.
Patient Readmission Risk: Hospitals predict whether a discharged patient has a high probability of being readmitted within 30 days, allowing them to allocate preventative home-care resources.

3. Marketing & E-Commerce

Customer Churn Prediction: Telecom and software companies analyze usage drops or customer service complaints to identify users at risk of canceling their subscription.
Click-Through Rate (CTR) Optimization: Digital advertising platforms predict the exact probability that a specific user will click an ad or scroll past it, ensuring ads are only shown to relevant audiences.
Conclusion