DEV Community: Abhijeet Pratap Singh

Decision Trees (Supervised Learning)

Abhijeet Pratap Singh — Wed, 01 Jul 2026 21:48:10 +0000

1. The Problem It Solves

Many real-world problems don't follow a straight-line relationship.

People don't make decisions by gradually increasing or decreasing something. Instead, they often make decisions based on conditions.

For example:

Will this customer upgrade?
Is this transaction fraudulent?
Should this loan be approved?
Will this machine fail?
Is this email spam?

The answer usually depends on a series of if-else rules, not a mathematical equation.

For example:

If monthly spending is greater than $500 and
Login frequency is less than twice a week and
Support tickets are increasing

then the customer is likely to churn.

Decision Trees are designed to discover these kinds of rules automatically.

Instead of fitting a line like Linear or Logistic Regression, they keep asking questions that split the data into smaller and more similar groups.

2. Core Intuition

Imagine you're playing 20 Questions.

You're trying to guess whether a customer will upgrade their subscription.

Instead of making one big guess, you ask simple Yes/No questions.

For example:

Does the customer have more than 20 seats?

If yes...

Ask another question.

Are API calls greater than 500 per day?

If yes...

Ask another question.

Has the account been active in the last week?

Eventually, you reach a point where almost every customer in that group behaves the same way.

That final group becomes a Leaf Node.

Whenever a new customer arrives, you simply walk them through the same set of questions until they reach a leaf.

The prediction is based on the majority of training examples that ended up there.

3. How the Algorithm Works

Decision Trees are built one split at a time.

At every node, the algorithm asks:

"Which question separates the data the best?"

It tries every feature.

Then every possible split point.

The split that creates the cleanest separation is chosen.

This process repeats until the stopping criteria are met.

4. Measuring Node Purity

To decide whether a split is good, the algorithm measures how "mixed" the classes are inside each node.

One of the most common metrics is Gini Impurity.

Where:

pᵢ = probability of class i
C = total number of classes

Interpretation:

Gini = 0 → Every sample belongs to one class (perfectly pure)
Higher values → Classes are mixed together

The goal is to make every leaf node as pure as possible.

5. Information Gain

Every possible split is evaluated.

The algorithm calculates how much impurity decreases after making that split.

This decrease is called Information Gain.

The split with the highest Information Gain becomes the next branch in the tree.

Then the entire process repeats recursively for each child node.

6. When Does the Tree Stop Growing?

If left alone, a Decision Tree keeps splitting until every training example has its own leaf.

That almost always leads to overfitting.

To prevent this, we usually limit tree growth using parameters like:

max_depth
min_samples_split
min_samples_leaf
max_leaf_nodes

These regularization settings help the tree generalize to unseen data instead of memorizing the training set.

7. When Should You Use Decision Trees?

Decision Trees work well when:

Relationships are non-linear.
Data contains many conditional rules.
Features are a mix of numerical and categorical values.
Interpretability is important.
You don't want extensive preprocessing.

Typical applications include:

Customer churn prediction
Credit approval
Fraud detection
Medical diagnosis
Product recommendation
Customer segmentation
Risk assessment

8. Advantages

Decision Trees have several practical benefits.

No feature scaling required.
Handles numerical and categorical data.
Learns non-linear relationships automatically.
Easy to visualize and explain.
Captures feature interactions naturally.
Works well even with missing values (depending on implementation).

9. When It Starts Breaking Down

Decision Trees are powerful, but they have some important weaknesses.

Overfitting

The biggest problem.

If the tree grows without limits, it starts memorizing the training data instead of learning real patterns.

This usually results in poor performance on new data.

High Variance

Decision Trees are unstable.

A small change in the training data can completely change the structure of the tree.

Two trees trained on almost identical datasets may look very different.

Greedy Decisions

The algorithm always chooses the best split right now.

It never looks ahead.

That means an early decision can prevent the tree from finding a better overall structure later.

Bias Toward Features with Many Split Points

Continuous numerical features often have many possible split locations.

Without proper controls, the algorithm may favor these features even when they aren't the most meaningful.

10. Python Implementation

import numpy as np
import pandas as pd

from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import export_text

from sklearn.metrics import accuracy_score

# Generate sample data
np.random.seed(42)

seat_count = np.random.uniform(1, 100, 100)
api_calls = np.random.uniform(10, 1000, 100)

# Business rule
upgraded = (
    (seat_count > 20) &
    (api_calls > 500)
).astype(int)

df = pd.DataFrame({
    "Seat_Count": seat_count,
    "API_Calls": api_calls,
    "Upgraded": upgraded
})

X = df[["Seat_Count", "API_Calls"]]
y = df["Upgraded"]

# Train Decision Tree
model = DecisionTreeClassifier(
    max_depth=3,
    random_state=42
)

model.fit(X, y)

# Predictions
predictions = model.predict(X)

print(
    "Accuracy:",
    accuracy_score(y, predictions)
)

print("\nDecision Rules\n")

print(
    export_text(
        model,
        feature_names=[
            "Seat_Count",
            "API_Calls"
        ]
    )
)

11. How to Evaluate the Model

Accuracy

Measures the percentage of correct predictions.

Useful when classes are balanced.

Precision

How many predicted positives were actually positive.

Recall

How many actual positive cases were correctly identified.

F1 Score

Balances Precision and Recall.

Useful for imbalanced datasets.

Tree Depth

A deeper tree isn't always better.

Very deep trees usually indicate overfitting.

Feature Importance

Decision Trees automatically estimate how useful each feature was during training.

This helps explain which variables influenced predictions the most.

12. Real-World Engineering Notes

Here are a few things you'll notice in production:

Decision Trees are one of the easiest ML models to explain to non-technical teams.
They require very little preprocessing.
Always limit tree growth using max_depth or min_samples_leaf.
A single Decision Tree rarely gives the best performance.
Most production systems use ensembles like Random Forest or Gradient Boosting because they reduce overfitting and improve accuracy.
Think of a Decision Tree as the building block for many of today's strongest machine learning algorithms.

13. Key Takeaways

Decision Trees solve classification and regression problems using a series of if-else rules.
They automatically discover non-linear relationships in data.
The algorithm chooses splits that maximize Information Gain and reduce impurity.
Easy to understand, visualize, and explain.
Requires little preprocessing and no feature scaling.
Can overfit easily if not regularized.
Forms the foundation of Random Forests, Extra Trees, XGBoost, LightGBM, and many other ensemble methods.

Logistic Regression (Supervised Family)

Abhijeet Pratap Singh — Wed, 01 Jul 2026 21:31:00 +0000

1. The Problem It Solves

Logistic Regression is used when the outcome is a category rather than a number.

Most commonly, it's used for binary classification, where the answer is either Yes or No, True or False, or 1 or 0.

Typical business problems include:

Will a customer churn?
Is this transaction fraudulent?
Will a customer click an ad?
Will a loan default?
Is an email spam?
Will a machine fail in the next 24 hours?

Unlike Linear Regression, we're not trying to predict a continuous value.

Instead, we're predicting the probability that an event belongs to a particular class.

For example:

A customer may have an 82% probability of churning.

The business can then decide whether that probability is high enough to trigger an intervention.

2. Core Intuition

Imagine you're trying to predict whether a customer will cancel their subscription.

Suppose the only feature you have is how many times they opened your app this month.

If you use a straight line like Linear Regression, the predictions quickly become unrealistic.

A very active customer might end up with a -20% chance of churn.

A completely inactive customer could end up with 140%.

Probabilities obviously can't work like that.

To fix this, Logistic Regression takes the linear equation and passes it through a mathematical function called the Sigmoid Function.

Instead of producing a straight line, it creates an S-shaped curve.

No matter how large or small the input becomes, the output always stays between 0 and 1.

That makes it perfect for probability estimation.

3. The Mathematical Model

The model first calculates a linear score.

Instead of using that score directly, it passes it through the Sigmoid function.

Where:

z = linear score
p̂ = predicted probability

The final output is always between 0 and 1.

For example:

0.08  → Very unlikely
0.32  → Low risk
0.65  → Moderate risk
0.94  → Very high probability

Businesses can then choose a decision threshold.

For example:

Probability ≥ 0.50 → Predict Churn
Probability < 0.50 → Predict Renewal

That threshold doesn't have to be 0.5.

Fraud detection systems often use much lower thresholds to catch more suspicious transactions.

4. What Is the Model Optimizing?

Linear Regression minimizes squared error.

That doesn't work well for classification.

Instead, Logistic Regression minimizes Log Loss (also called Binary Cross Entropy).

Log Loss heavily penalizes predictions that are both wrong and confident.

For example:

Actual class = Fraud

Prediction = 0.99 Legitimate

This receives a much larger penalty than predicting 0.55.

That's exactly what we want.

A model should never be extremely confident when it's wrong.

5. How the Model Learns

Unlike Linear Regression, there isn't a direct mathematical formula that instantly finds the best coefficients.

Instead, Logistic Regression learns gradually.

It starts with random weights.

It makes predictions.

Measures the error.

Then adjusts the coefficients a little.

This repeats thousands of times until the Log Loss stops improving.

Gradient Descent is one of the most common optimization methods used during this process.

6. Decision Boundary

Eventually, the model needs to convert probabilities into class labels.

This is done using a decision threshold.

For example:

Predicted Probability = 0.81

Threshold = 0.50

Prediction = Churn

Changing the threshold changes how conservative the model becomes.

Lower thresholds increase recall.

Higher thresholds increase precision.

Choosing the right threshold depends on the business problem.

7. When Should You Use Logistic Regression?

Logistic Regression works well when:

The target is binary.
The classes are reasonably separable.
You need probability estimates.
You want a fast, interpretable model.
The relationship between features and the log-odds is roughly linear.

Common applications include:

Customer churn prediction
Fraud detection
Medical diagnosis
Email spam detection
Credit approval
Employee attrition prediction
Marketing campaign response prediction

8. Core Assumptions

Independent Observations

Each training example should be independent.

Linear Relationship in Log-Odds

The features should have a roughly linear relationship with the log-odds, not necessarily with the probability itself.

No High Multicollinearity

Features shouldn't contain nearly identical information.

Highly correlated variables make the coefficients unstable.

Limited Influence of Extreme Outliers

Extreme feature values can heavily influence the learned coefficients.

9. When It Starts Breaking Down

Logistic Regression isn't designed for every classification problem.

It struggles when:

Class boundaries are highly non-linear.
Features interact in complex ways.
Classes overlap heavily.
There are many irrelevant features.
One feature perfectly separates both classes (Perfect Separation).

For example:

Suppose every customer with more than five support tickets always churns.

The coefficient for that feature can grow toward infinity, making the model unstable.

10. Python Implementation

import numpy as np
import pandas as pd

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    roc_auc_score
)

# Generate sample customer activity
np.random.seed(42)

app_opens = np.concatenate([
    np.random.normal(25, 5, 50),
    np.random.normal(4, 2, 50)
])

churned = np.concatenate([
    np.zeros(50),
    np.ones(50)
])

df = pd.DataFrame({
    "App_Opens": app_opens,
    "Churned": churned
})

X = df[["App_Opens"]]
y = df["Churned"]

# Train model
model = LogisticRegression(
    solver="lbfgs",
    random_state=42
)

model.fit(X, y)

# Predictions
df["Probability"] = model.predict_proba(X)[:, 1]
df["Prediction"] = model.predict(X)

print(f"Intercept : {model.intercept_[0]:.4f}")
print(f"Coefficient : {model.coef_[0][0]:.4f}")

print(classification_report(
    y,
    df["Prediction"]
))

print(
    "ROC AUC:",
    roc_auc_score(
        y,
        df["Probability"]
    )
)

11. How to Evaluate the Model

Accuracy

The percentage of correct predictions.

Works well only when classes are balanced.

Precision

Out of everything predicted as positive,

how many were actually positive?

Useful when false positives are expensive.

Recall

Out of all actual positive cases,

how many did the model find?

Useful when missing a positive case is costly.

F1 Score

Balances Precision and Recall.

A good overall metric for imbalanced datasets.

ROC-AUC

Measures how well the model separates the two classes across every possible threshold.

1.0 → Perfect classifier
0.5 → Random guessing

Higher is better.

12. Real-World Engineering Notes

Some practical lessons you'll run into:

Always look at predicted probabilities, not just class labels.
Adjust the decision threshold based on business needs instead of blindly using 0.5.
Scale numerical features when using gradient-based optimization.
Logistic Regression is often the strongest baseline classifier before trying tree-based models.
Highly imbalanced datasets usually need class weighting or resampling techniques.
Don't rely on accuracy alone—Precision, Recall, F1 Score, and ROC-AUC usually tell a much better story.

13. Key Takeaways

Logistic Regression predicts probabilities for binary classification problems.
It converts a linear model into probabilities using the Sigmoid function.
It learns by minimizing Log Loss instead of squared error.
Fast to train, easy to interpret, and widely used in production.
Produces probability scores rather than just Yes/No predictions.
Works best when class boundaries are reasonably linear.
A great baseline classifier before moving to Decision Trees, Random Forests, XGBoost, or Neural Networks.

Linear Regression (Supervised Learning)

Abhijeet Pratap Singh — Tue, 30 Jun 2026 20:37:03 +0000

1. The Problem It Solves

Linear Regression is one of the simplest and most widely used machine learning algorithms for predicting continuous numeric values.

Whenever your target is a number rather than a category, Linear Regression is usually the first model worth trying.

Some common examples include:

Predicting monthly cloud infrastructure costs
Estimating customer lifetime value (CLV)
Forecasting next month's sales
Predicting electricity consumption
Estimating delivery times
Predicting marketing leads based on ad spend

The idea is simple.

Given a set of input features, the model learns the relationship between them and predicts a numeric output.

For example, suppose a SaaS company wants to estimate a customer's next monthly usage bill.

The inputs could be:

Active seats
API requests
Storage usage
Historical consumption

The output would be a single number:

Predicted Monthly Bill

2. Core Intuition

Imagine plotting every house in a city.

The horizontal axis represents the size of the house.

The vertical axis represents its selling price.

Every house becomes a point on the graph.

The points won't line up perfectly. They'll be scattered everywhere.

Now imagine placing a long ruler across those points.

You slowly rotate it and move it up or down until it passes through the center of the data as closely as possible.

That's exactly what Linear Regression is trying to do.

The model adjusts only two things:

Intercept — where the line starts on the Y-axis.
Slope — how steep the line is.

Its goal is to find the line that produces the smallest overall prediction error.

3. The Mathematical Model

Linear Regression assumes that the relationship between the input variables (X) and the target (y) can be represented using a straight line.

Where:

ŷ = predicted value
β₀ = intercept
β₁ ... βₙ = feature coefficients
x₁ ... xₙ = input features

Every coefficient tells us how much the prediction changes when that feature increases by one unit.

For example:

Suppose the learned equation becomes:

Predicted Leads = 50 + 0.08 × Marketing Spend

That means every extra $1 spent on marketing increases the expected leads by 0.08, assuming everything else stays the same.

This interpretability is one of the biggest reasons Linear Regression is still widely used in business.

4. What Is the Model Optimizing?

Not every line fits the data equally well.

Some lines pass too high.

Others pass too low.

Linear Regression measures the difference between the actual value and the predicted value.

These differences are called Residual Errors.

Instead of simply adding those errors together (which would cancel positive and negative values), the model squares every error before adding them.

This gives us the Sum of Squared Residuals (SSR).

The smaller this value becomes, the better the fitted line.

The entire training process is simply trying to minimize this error.

5. How the Model Learns

There are two common ways to calculate the coefficients.

Method 1 — Normal Equation

For smaller datasets, Linear Regression has a direct mathematical solution.

Instead of learning gradually, it computes the best coefficients in one step.

Advantages:

Exact solution
No learning rate
No iterations

Limitations:

Computationally expensive for very large datasets
Requires matrix inversion

Method 2 — Gradient Descent

For larger datasets, calculating the exact solution becomes expensive.

Instead, the model starts with random coefficients.

It then repeatedly measures the prediction error and slightly adjusts the coefficients in the direction that reduces the loss.

Each update moves the model closer to the minimum error.

Where:

α = learning rate
∂J/∂β = gradient of the loss function

The process repeats until the error stops improving.

6. When Should You Use Linear Regression?

Linear Regression works well when:

The target is continuous.
The relationship is approximately linear.
You need an interpretable model.
Training speed matters.
You need a strong baseline before trying more advanced algorithms.

Typical applications include:

Revenue prediction
Cost estimation
Demand forecasting
Capacity planning
Financial modeling
Energy consumption forecasting

7. Core Assumptions

Linear Regression relies on several assumptions.

Linearity

The relationship between inputs and output should roughly follow a straight line.

Independence

Observations should not influence one another.

Homoscedasticity

Residual errors should have roughly constant variance across all prediction levels.

No Multicollinearity

Input variables should not be highly correlated with each other.

For example:

Using both:

Age in Years
Birth Year

creates redundant information and makes coefficients unstable.

Normally Distributed Residuals (mainly for statistical inference)

Residual errors should be approximately normally distributed if confidence intervals or hypothesis testing are important.

8. When It Starts Breaking Down

Linear Regression is powerful, but only under the right conditions.

It struggles when:

The relationship is curved rather than linear.
A few extreme outliers dominate the data.
Important variables are missing.
Input features are highly correlated.
The variance changes dramatically across different prediction ranges.

A common example is stock prices.

Markets rarely move in a straight line, so Linear Regression usually performs poorly without additional feature engineering.

9. Python Implementation

import numpy as np
import pandas as pd

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Generate sample data
np.random.seed(42)

marketing_spend = np.random.uniform(500, 10000, 100)

leads_generated = (
    50 +
    0.08 * marketing_spend +
    np.random.normal(0, 50, 100)
)

df = pd.DataFrame({
    "Marketing_Spend": marketing_spend,
    "Leads_Generated": leads_generated
})

X = df[["Marketing_Spend"]]
y = df["Leads_Generated"]

# Train model
model = LinearRegression()
model.fit(X, y)

# Predictions
df["Predicted_Leads"] = model.predict(X)

# Evaluation
rmse = np.sqrt(
    mean_squared_error(y, df["Predicted_Leads"])
)

r2 = r2_score(y, df["Predicted_Leads"])

print(f"Intercept : {model.intercept_:.4f}")
print(f"Coefficient : {model.coef_[0]:.4f}")
print(f"RMSE : {rmse:.4f}")
print(f"R² Score : {r2:.4f}")

10. How to Evaluate the Model

RMSE (Root Mean Squared Error)

Measures the average prediction error.

Lower is better.

R² Score

Measures how much variance the model explains.

1.0 → Perfect predictions
0.8 → Explains 80% of the variance
0.0 → No better than predicting the average

11. Real-World Engineering Notes

Some lessons you'll quickly learn in production:

Linear Regression should almost always be your first baseline model.
Feature engineering usually improves accuracy more than changing algorithms.
Always inspect residual plots before trusting the predictions.
Remove or investigate extreme outliers before training.
Scale isn't required for ordinary Linear Regression, but becomes important when using Gradient Descent or regularized variants like Ridge and Lasso.
Just because the R² score is high doesn't mean the assumptions are satisfied.

12. Key Takeaways

One of the simplest and most interpretable machine learning algorithms.
Predicts continuous numeric values using a linear relationship.
Finds the best-fitting line by minimizing squared prediction errors.
Extremely fast to train and easy to explain to business stakeholders.
Works best when relationships are approximately linear.
Struggles with non-linear patterns, outliers, and multicollinearity.
A great baseline model before moving to more advanced algorithms like Decision Trees, Random Forests, or Gradient Boosting.