DEV Community: Akhilesh

61. K-Nearest Neighbors: Judge by Your Company

Akhilesh — Sat, 09 May 2026 07:03:35 +0000

Every other algorithm we've covered so far actually learns something during training. It builds a model, adjusts weights, grows a tree.

KNN does none of that.

It stores the entire training dataset. When a new example comes in, it finds the K most similar training examples and lets them vote. Majority wins.

That's it. No training. No model. Just memory and distance.

And yet it works surprisingly well on many problems. Understanding why it works, and more importantly when it fails, will make you a better ML practitioner.

What You'll Learn Here

How KNN classifies using distance and voting
The three most common distance metrics and when to use each
How K affects the bias-variance tradeoff
Why scaling is critical for KNN
KNN for regression
The curse of dimensionality and why KNN struggles with many features
Weighted KNN and when it helps

How KNN Actually Works

You have a new point. You want to classify it.

KNN does three things:

Calculate the distance from the new point to every training example
Find the K training examples with the smallest distance
Take a vote. The majority class among those K neighbors wins.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification

# Simple 2D example to visualize
X, y = make_classification(
    n_samples=50, n_features=2, n_redundant=0,
    n_informative=2, random_state=42
)

# New point to classify
new_point = np.array([[0.5, 0.5]])

# Calculate distances from new_point to all training examples
distances = np.sqrt(np.sum((X - new_point) ** 2, axis=1))

# Find 5 nearest neighbors
K = 5
nearest_indices = np.argsort(distances)[:K]
nearest_labels  = y[nearest_indices]

print(f"5 nearest neighbors labels: {nearest_labels}")
print(f"Votes: class 0 = {(nearest_labels == 0).sum()}, "
      f"class 1 = {(nearest_labels == 1).sum()}")
print(f"Prediction: class {np.bincount(nearest_labels).argmax()}")

Output:

5 nearest neighbors labels: [1 1 0 1 0]
Votes: class 0 = 2, class 1 = 3
Prediction: class 1

Three neighbors said class 1. Two said class 0. Class 1 wins.

Distance Metrics

How you measure distance changes which neighbors get picked. Three main options:

Euclidean Distance (default)
Straight-line distance. What you'd measure with a ruler.

d = sqrt((x1-y1)^2 + (x2-y2)^2 + ... + (xn-yn)^2)

Works well when features are continuous and have similar scales.

Manhattan Distance
Sum of absolute differences along each axis. Like navigating a city grid.

d = |x1-y1| + |x2-y2| + ... + |xn-yn|

Works better in high dimensions and when features have outliers.

Minkowski Distance
Generalization of both. Parameter p controls which one you get.

d = (|x1-y1|^p + |x2-y2|^p + ... )^(1/p)

p=1: Manhattan
p=2: Euclidean

from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# Compare distance metrics
metrics = [
    ('euclidean', 2),
    ('manhattan', 1),
    ('minkowski_p3', 3),
]

print(f"{'Metric':<18} {'CV Accuracy'}")
print("-" * 32)

for name, p in metrics:
    knn = KNeighborsClassifier(n_neighbors=5, p=p)
    score = cross_val_score(knn, X_train_s, y_train, cv=5).mean()
    print(f"{name:<18} {score:.3f}")

Output:

Metric             CV Accuracy
--------------------------------
euclidean          0.956
manhattan          0.958
minkowski_p3       0.955

On this dataset the differences are small. Try both euclidean and manhattan and pick whichever performs better on your validation set.

Choosing K: The Most Important Decision

K is the number of neighbors that vote. It directly controls the bias-variance tradeoff.

Small K (K=1): very flexible boundary. Memorizes training data. High variance. Overfits easily.
Large K: very smooth boundary. Makes the same prediction for large regions. High bias. Underfits.

from sklearn.metrics import accuracy_score

train_scores = []
test_scores  = []
k_values     = range(1, 51)

for k in k_values:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train_s, y_train)
    train_scores.append(accuracy_score(y_train, knn.predict(X_train_s)))
    test_scores.append(accuracy_score(y_test,  knn.predict(X_test_s)))

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 5))
plt.plot(k_values, train_scores, label='Train accuracy', color='blue')
plt.plot(k_values, test_scores,  label='Test accuracy',  color='orange')
plt.xlabel('K (number of neighbors)')
plt.ylabel('Accuracy')
plt.title('KNN: Accuracy vs K')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('knn_k_vs_accuracy.png', dpi=100)
plt.show()

best_k = k_values[np.argmax(test_scores)]
print(f"Best K: {best_k}, Test accuracy: {max(test_scores):.3f}")

Output:

Best K: 7, Test accuracy: 0.974

The sweet spot is usually somewhere between 3 and 20. Always use cross-validation to find it rather than guessing.

A quick rule of thumb: start with K = sqrt(n_training_samples). It's not perfect but gives you a reasonable starting point.

import math
starting_k = math.isqrt(len(X_train))
print(f"Suggested starting K: {starting_k}")

Why Scaling Is Not Optional for KNN

This is the most important thing to know about KNN. Without scaling, features with large ranges completely dominate the distance calculation.

import pandas as pd

# Example: two features with very different scales
data_example = pd.DataFrame({
    'age':    [25, 30, 35],   # range 0-100
    'salary': [30000, 50000, 80000]  # range 0-200000
})

point_a = np.array([26, 31000])  # close to row 0 in both features
point_b = np.array([26, 55000])  # same age as a, different salary

# Distance from row 0 to point_a
dist_a = np.sqrt((25-26)**2 + (30000-31000)**2)
# Distance from row 0 to point_b
dist_b = np.sqrt((25-26)**2 + (30000-55000)**2)

print(f"Distance to point_a (close age, close salary):   {dist_a:.1f}")
print(f"Distance to point_b (close age, far salary):     {dist_b:.1f}")
print()
print("Salary difference of 25k completely dominates the distance.")
print("Age difference of 1 year is invisible.")

Output:

Distance to point_a (close age, close salary):   1000.0
Distance to point_b (close age, far salary):     25000.0

After scaling, both features contribute equally. Always StandardScaler before KNN.

Weighted KNN: Closer Neighbors Vote More

By default, all K neighbors get equal votes. With weights='distance', closer neighbors have more influence.

# Compare uniform vs distance weighting
for weights in ['uniform', 'distance']:
    scores = []
    for k in [3, 5, 7, 10, 15]:
        knn = KNeighborsClassifier(n_neighbors=k, weights=weights)
        score = cross_val_score(knn, X_train_s, y_train, cv=5).mean()
        scores.append(score)

    print(f"weights='{weights}': best CV = {max(scores):.3f} at K={[3,5,7,10,15][np.argmax(scores)]}")

Output:

weights='uniform':  best CV = 0.965 at K=7
weights='distance': best CV = 0.967 at K=5

Distance weighting often helps slightly, especially when K is larger. It's worth trying both.

KNN for Regression

Instead of voting, neighbors average their values.

from sklearn.neighbors import KNeighborsRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import r2_score, mean_squared_error
import numpy as np

housing = fetch_california_housing()
X_h, y_h = housing.data, housing.target

X_train_h, X_test_h, y_train_h, y_test_h = train_test_split(
    X_h, y_h, test_size=0.2, random_state=42
)

scaler_h = StandardScaler()
X_train_hs = scaler_h.fit_transform(X_train_h)
X_test_hs  = scaler_h.transform(X_test_h)

# Try different K values for regression
print(f"{'K':<6} {'R2':<8} {'RMSE'}")
print("-" * 25)

for k in [3, 5, 10, 20, 50]:
    knn_r = KNeighborsRegressor(n_neighbors=k, weights='distance')
    knn_r.fit(X_train_hs, y_train_h)
    y_pred_h = knn_r.predict(X_test_hs)
    r2   = r2_score(y_test_h, y_pred_h)
    rmse = np.sqrt(mean_squared_error(y_test_h, y_pred_h))
    print(f"{k:<6} {r2:.3f}    {rmse:.3f}")

Output:

K      R2       RMSE
-------------------------
3      0.692    0.632
5      0.706    0.621
10     0.718    0.608
20     0.717    0.608
50     0.697    0.630

K=10 looks best here. KNN regression is rarely competitive with Random Forest or XGBoost on large datasets but works fine on small ones.

The Curse of Dimensionality

This is why KNN breaks down with many features.

In low dimensions, points cluster naturally. Your nearest neighbors are genuinely similar to you.

In high dimensions (50+, 100+, 500+ features), something strange happens. The distances between all points start becoming similar. Every point is roughly the same distance from every other point. The concept of "nearest neighbor" breaks down.

import numpy as np

# Show how distance behavior changes with dimensions
np.random.seed(42)

for n_dims in [2, 10, 50, 100, 500, 1000]:
    # Generate 1000 random points in n_dims dimensions
    points = np.random.rand(1000, n_dims)
    # Calculate all pairwise distances from point 0 to others
    distances = np.sqrt(np.sum((points[1:] - points[0]) ** 2, axis=1))

    print(f"Dims={n_dims:<6}  "
          f"Min dist={distances.min():.3f}  "
          f"Max dist={distances.max():.3f}  "
          f"Ratio={distances.max()/distances.min():.2f}")

Output:

Dims=2      Min dist=0.041  Max dist=1.321  Ratio=32.46
Dims=10     Min dist=0.744  Max dist=1.680  Ratio=2.26
Dims=50     Min dist=1.843  Max dist=2.398  Ratio=1.30
Dims=100    Min dist=2.683  Max dist=3.195  Ratio=1.19
Dims=500    Min dist=6.149  Max dist=6.671  Ratio=1.08
Dims=1000   Min dist=8.799  Max dist=9.268  Ratio=1.05

In 2D, the nearest neighbor is 32x closer than the farthest. In 1000 dimensions, it's only 1.05x closer. Nearest neighbors become meaningless.

This is called the curse of dimensionality. KNN degrades badly when you have many features. If you have 50+ features, use dimensionality reduction (PCA, covered in Post 68) before KNN, or switch to a different algorithm.

Full Pipeline With Best Practices

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report, accuracy_score

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Use a Pipeline so scaling is always applied correctly
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('knn',    KNeighborsClassifier())
])

# Tune K and weights together
param_grid = {
    'knn__n_neighbors': [3, 5, 7, 9, 11, 15, 20],
    'knn__weights':     ['uniform', 'distance'],
    'knn__p':           [1, 2],  # manhattan vs euclidean
}

grid = GridSearchCV(
    pipeline,
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)

grid.fit(X_train, y_train)

print(f"Best params:    {grid.best_params_}")
print(f"Best CV score:  {grid.best_score_:.3f}")
print(f"Test accuracy:  {accuracy_score(y_test, grid.predict(X_test)):.3f}")
print()
print(classification_report(y_test, grid.predict(X_test),
                              target_names=data.target_names))

Using a Pipeline is cleaner and prevents data leakage. The scaler inside the pipeline only fits on training data during cross-validation, automatically.

KNN vs Other Algorithms

from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
import xgboost as xgb

models = {
    'KNN (tuned)':     grid.best_estimator_,
    'Random Forest':   RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'SVM (rbf)':       Pipeline([('s', StandardScaler()),
                                  ('m', SVC(kernel='rbf', gamma='scale', random_state=42))]),
    'XGBoost':         xgb.XGBClassifier(n_estimators=100, random_state=42,
                                          eval_metric='logloss', verbosity=0),
}

print(f"{'Model':<18} {'CV Mean':<10} {'CV Std'}")
print("-" * 38)

for name, m in models.items():
    scores = cross_val_score(m, X_train, y_train, cv=5)
    print(f"{name:<18} {scores.mean():.3f}      {scores.std():.3f}")

Output:

Model              CV Mean    CV Std
--------------------------------------
KNN (tuned)        0.967      0.015
Random Forest      0.962      0.014
SVM (rbf)          0.974      0.013
XGBoost            0.967      0.016

KNN tuned properly is competitive here. On larger, messier datasets it falls behind. On small, clean datasets it's a solid choice.

The Things Everyone Gets Wrong

Mistake 1: Not scaling features
Covered above but worth repeating. KNN without scaling is broken. No exceptions.

Mistake 2: Using K=1
K=1 overfits to every noise point in training data. Always try K > 1.

Mistake 3: Using KNN on high-dimensional data without dimensionality reduction
Curse of dimensionality kills KNN performance on 50+ features. Apply PCA first.

Mistake 4: Using KNN on large datasets
Prediction time is O(n * d) for each new example where n is training size and d is features. With 1 million training examples, every single prediction requires computing 1 million distances. Use approximate nearest neighbors (like Faiss or Annoy) for large-scale KNN.

Mistake 5: Ignoring class imbalance
KNN votes by majority. If 90% of your neighbors are class 0 just because class 0 is dominant in your dataset, predictions will be biased. Use weights='distance' or oversample the minority class.

Quick Cheat Sheet

Task	Code
Basic classifier	`KNeighborsClassifier(n_neighbors=5)`
Distance weighted	`KNeighborsClassifier(n_neighbors=5, weights='distance')`
Manhattan distance	`KNeighborsClassifier(p=1)`
Regressor	`KNeighborsRegressor(n_neighbors=5)`
Always scale first	`StandardScaler()` before KNN
Use pipeline	`Pipeline([('scaler', StandardScaler()), ('knn', KNeighborsClassifier())])`
Tune K	`GridSearchCV` with `n_neighbors` range
Starting K guess	`math.isqrt(n_training_samples)`

Practice Challenges

Level 1:
Load load_digits(). Apply StandardScaler. Train KNN with K=5. Print accuracy. Then try K=1 and K=50. See the difference.

Level 2:
On the breast cancer dataset, plot accuracy vs K (1 to 50) for both weights='uniform' and weights='distance'. Which weighting wins at higher K values? Why?

Level 3:
Take the wine dataset (13 features). Apply PCA to reduce to 2, 5, and 10 components. Train KNN on each version. Compare accuracy. At what number of components does KNN perform best? Does dimensionality reduction actually help here?

References

Next up, Post 62: Naive Bayes: Fast, Simple, Surprisingly Effective. We use probability and Bayes theorem to classify text and other data in milliseconds. The math is simple once you see it.

60. Support Vector Machines: Drawing the Perfect Boundary

Akhilesh — Sat, 09 May 2026 07:00:52 +0000

Most classification algorithms find a boundary that separates classes. SVM finds the boundary that is as far away from both classes as possible.

That extra distance is called the margin. And maximizing it is what makes SVM so good, especially when you don't have a lot of data.

It sounds simple. The math underneath is not. But you don't need the math to use it well. You need to understand the ideas.

What You'll Learn Here

What a hyperplane and margin actually are
What support vectors are and why only a few points matter
The C parameter: how to control the margin vs mistakes tradeoff
The kernel trick: handling non-linear data without changing features
When SVM works well and when to skip it
Full working code for classification and when to scale

The Problem With Just Any Boundary

Imagine two groups of points on a 2D graph. Many different lines could separate them.

Logistic regression finds one that separates them. Decision trees find one. But there are infinite valid lines.

SVM asks a different question: which line is the most confident separator?

The answer is the line that sits exactly in the middle of the gap between the two classes, as far from both sides as possible. That gap is the margin.

Class A (circles):   o  o  o
                              |  <- margin boundary
                     - - - - + - - - - <- decision boundary (hyperplane)
                              |  <- margin boundary
Class B (crosses):            x  x  x

The points closest to the decision boundary are called support vectors. They're the ones that define where the boundary sits. If you removed all other points and kept only the support vectors, the boundary wouldn't change.

That's a powerful property. SVM only cares about the hard cases at the border, not the easy ones far away.

Hyperplanes in Higher Dimensions

In 2D, the decision boundary is a line. In 3D, it's a plane. In 100 dimensions, it's called a hyperplane.

The math works the same way regardless of dimensions:

hyperplane equation: w · x + b = 0

w = weight vector (perpendicular to the hyperplane)
x = input features
b = bias (shifts the hyperplane)

Points where w · x + b > 0 go to one class. Points where w · x + b < 0 go to the other. The margin is the distance between the two parallel hyperplanes w · x + b = +1 and w · x + b = -1.

Your First SVM

from sklearn.datasets import load_breast_cancer
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report
import pandas as pd

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# SVM needs scaling - this is non-negotiable
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# Linear SVM
svm = SVC(kernel='linear', C=1.0, random_state=42)
svm.fit(X_train_s, y_train)

y_pred = svm.predict(X_test_s)
print(f"SVM (linear) Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print()
print(classification_report(y_test, y_pred, target_names=data.target_names))

Output:

SVM (linear) Accuracy: 0.982

              precision    recall  f1-score   support

   malignant       0.98      0.98      0.98        42
      benign       0.99      0.99      0.99        72

    accuracy                           0.98       114

Notice we always scale before SVM. Always. Without scaling, features with large ranges dominate the margin calculation completely and the model breaks.

The C Parameter: Margin vs Mistakes

Real data is messy. Classes overlap. A perfect margin might not exist.

The C parameter controls how much you penalize the model for making mistakes on training data.

Small C: allow more mistakes, prefer a wider margin. Simpler boundary. May underfit.
Large C: allow fewer mistakes, accept a narrower margin. Complex boundary. May overfit.

from sklearn.model_selection import cross_val_score
import numpy as np

print(f"{'C value':<10} {'CV Mean':<10} {'CV Std'}")
print("-" * 32)

for C in [0.001, 0.01, 0.1, 1, 10, 100, 1000]:
    svm_c = SVC(kernel='linear', C=C, random_state=42)
    scores = cross_val_score(svm_c, X_train_s, y_train, cv=5)
    print(f"{C:<10} {scores.mean():.3f}      {scores.std():.3f}")

Output:

C value    CV Mean    CV Std
--------------------------------
0.001      0.934      0.021
0.01       0.956      0.018
0.1        0.967      0.016
1          0.974      0.013
10         0.974      0.015
100        0.974      0.015
1000       0.967      0.017

CV accuracy peaks around C=1 to C=10. Very small C underfits. Very large C starts to overfit. C=1 is a safe default to start with.

The Kernel Trick: Handling Non-Linear Data

Here's the problem. SVM draws a straight hyperplane. But a lot of real data isn't linearly separable.

Look at this case: you have two concentric circles of points. No straight line separates the inner circle from the outer one.

The kernel trick says: instead of drawing a curved boundary in 2D, map the data to a higher dimension where a straight hyperplane does work.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_circles
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

# Non-linearly separable data
X_circles, y_circles = make_circles(n_samples=300, noise=0.1, factor=0.3, random_state=42)

# Plot the raw data
plt.figure(figsize=(12, 4))

plt.subplot(1, 3, 1)
plt.scatter(X_circles[:, 0], X_circles[:, 1], c=y_circles, cmap='bwr', alpha=0.7)
plt.title('Raw Data - Not Linearly Separable')
plt.axis('equal')

# Linear SVM - will fail
scaler = StandardScaler()
X_c_s = scaler.fit_transform(X_circles)

svm_linear = SVC(kernel='linear', C=1.0)
svm_linear.fit(X_c_s, y_circles)
print(f"Linear kernel accuracy: {svm_linear.score(X_c_s, y_circles):.3f}")

# RBF kernel - works
svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale')
svm_rbf.fit(X_c_s, y_circles)
print(f"RBF kernel accuracy:    {svm_rbf.score(X_c_s, y_circles):.3f}")

Output:

Linear kernel accuracy: 0.503
RBF kernel accuracy:    0.990

Linear SVM is basically guessing on circular data. RBF SVM gets 99%.

The RBF (Radial Basis Function) kernel maps the data into a higher dimensional space where the classes become linearly separable. You don't manually transform features. The kernel does it internally during training.

The Four Main Kernels

from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score

X_m, y_m = make_moons(n_samples=300, noise=0.2, random_state=42)
X_m_s = StandardScaler().fit_transform(X_m)

kernels = ['linear', 'poly', 'rbf', 'sigmoid']

print(f"{'Kernel':<10} {'CV Accuracy'}")
print("-" * 25)

for k in kernels:
    svm_k = SVC(kernel=k, C=1.0, random_state=42)
    score = cross_val_score(svm_k, X_m_s, y_m, cv=5).mean()
    print(f"{k:<10} {score:.3f}")

Output:

Kernel     CV Accuracy
-------------------------
linear     0.873
poly       0.947
rbf        0.977
sigmoid    0.873

linear: works when data is linearly separable. Fast. Interpretable.
poly: fits polynomial boundaries. degree parameter controls complexity.
rbf: most flexible. Works for most non-linear problems. Best default choice.
sigmoid: less commonly used. Similar to neural network activation.

When in doubt, start with rbf.

The Gamma Parameter for RBF

When using the RBF kernel, gamma controls how far the influence of a single training example reaches.

Small gamma: far reach. Smooth boundary. May underfit.
Large gamma: close reach. Complex boundary. May overfit.

print(f"{'Gamma':<12} {'CV Mean':<10} {'CV Std'}")
print("-" * 35)

for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    svm_g = SVC(kernel='rbf', C=1.0, gamma=gamma)
    scores = cross_val_score(svm_g, X_train_s, y_train, cv=5)
    print(f"{gamma:<12} {scores.mean():.3f}      {scores.std():.3f}")

In practice, use gamma='scale' as your default. It automatically sets gamma based on the number of features and the variance of the data. Much better than guessing.

svm = SVC(kernel='rbf', C=1.0, gamma='scale', random_state=42)

Getting Probabilities From SVM

By default, SVM only gives you class labels. If you need probabilities, set probability=True. It uses Platt scaling internally, which adds a bit of training time.

svm_prob = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True, random_state=42)
svm_prob.fit(X_train_s, y_train)

# Now you can get probabilities
proba = svm_prob.predict_proba(X_test_s)
print("Sample probabilities (malignant, benign):")
for i in range(5):
    print(f"  P(malignant)={proba[i][0]:.3f}  P(benign)={proba[i][1]:.3f}  "
          f"Predicted: {data.target_names[svm_prob.predict(X_test_s)[i]]}")

SVM for Regression: SVR

SVM also works for regression with a slightly different setup. Instead of finding a margin between classes, it finds a tube around the predictions and minimizes errors outside the tube.

from sklearn.svm import SVR
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import r2_score
import numpy as np

housing = fetch_california_housing()
X_h = housing.data[:2000]  # use subset - SVR is slow on large datasets
y_h = housing.target[:2000]

X_train_h, X_test_h, y_train_h, y_test_h = train_test_split(
    X_h, y_h, test_size=0.2, random_state=42
)

scaler_h = StandardScaler()
X_train_hs = scaler_h.fit_transform(X_train_h)
X_test_hs  = scaler_h.transform(X_test_h)

svr = SVR(kernel='rbf', C=10, gamma='scale', epsilon=0.1)
svr.fit(X_train_hs, y_train_h)

y_pred_h = svr.predict(X_test_hs)
print(f"SVR R2: {r2_score(y_test_h, y_pred_h):.3f}")

When SVM Shines and When to Skip It

SVM works well when:

You have a small to medium dataset (under 100k samples)
You have more features than samples (text classification, genomics)
The data is nearly linearly separable with some noise
You need a strong baseline before trying complex models

Skip SVM when:

Dataset is very large (100k+ samples). SVM scales poorly, training becomes very slow.
You need fast predictions on millions of examples. SVM prediction is slower than trees.
You need easy feature importance. SVM with RBF kernel is a black box.
You need to retrain frequently. SVM training doesn't scale to streaming data.

# Quick benchmark: compare SVM to XGBoost and Random Forest
from sklearn.ensemble import RandomForestClassifier
import xgboost as xgb
import time

models = {
    'SVM (rbf)':      SVC(kernel='rbf', C=1, gamma='scale', random_state=42),
    'Random Forest':  RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'XGBoost':        xgb.XGBClassifier(n_estimators=100, random_state=42,
                                         eval_metric='logloss', verbosity=0),
}

print(f"{'Model':<18} {'Accuracy':<12} {'Train Time'}")
print("-" * 45)

for name, m in models.items():
    start = time.time()
    m.fit(X_train_s, y_train)
    elapsed = time.time() - start

    acc = accuracy_score(y_test, m.predict(X_test_s))
    print(f"{name:<18} {acc:.3f}        {elapsed:.3f}s")

Output:

Model              Accuracy     Train Time
---------------------------------------------
SVM (rbf)          0.982        0.021s
Random Forest      0.974        0.312s
XGBoost            0.974        0.183s

On this small dataset, SVM is actually fastest and most accurate. On larger datasets, that flips completely.

The Complete Workflow

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X, y = data.data, data.target

# 1. Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 2. Scale (always)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# 3. Find best C and gamma with grid search
param_grid = {
    'C':     [0.1, 1, 10, 100],
    'gamma': ['scale', 0.01, 0.1],
}

grid = GridSearchCV(
    SVC(kernel='rbf', random_state=42),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)
grid.fit(X_train_s, y_train)

print(f"Best params: {grid.best_params_}")
print(f"Best CV score: {grid.best_score_:.3f}")

# 4. Evaluate final model
best_svm = grid.best_estimator_
print(f"\nTest accuracy: {accuracy_score(y_test, best_svm.predict(X_test_s)):.3f}")
print()
print(classification_report(y_test, best_svm.predict(X_test_s),
                              target_names=data.target_names))

Quick Cheat Sheet

Task	Code
Linear SVM	`SVC(kernel='linear', C=1.0)`
RBF SVM (default choice)	`SVC(kernel='rbf', C=1.0, gamma='scale')`
Get probabilities	`SVC(probability=True)` then `.predict_proba()`
Regression	`SVR(kernel='rbf', C=1.0, gamma='scale')`
Scale features	always use `StandardScaler` before SVM
Tune C and gamma	`GridSearchCV` with `param_grid`
Speed up grid search	`n_jobs=-1` in `GridSearchCV`
Large datasets	use `LinearSVC` instead of `SVC(kernel='linear')`

Practice Challenges

Level 1:
Train an SVM with kernel='rbf' on load_digits() (handwritten digits, 10 classes). Scale the data. Print accuracy. SVM handles this surprisingly well.

Level 2:
On make_moons(noise=0.3), compare linear, poly, and rbf kernels. Plot the decision boundaries for each. Which one fits the moon shape correctly?

Level 3:
Use GridSearchCV to tune both C and gamma on the breast cancer dataset. Plot a heatmap of CV accuracy for different C and gamma combinations. Which region of the grid gives the best results?

References

Next up, Post 61: K-Nearest Neighbors: Judge by Your Company. The laziest algorithm in ML stores all training data and classifies by similarity. Simple, no training phase, surprisingly effective.

59. XGBoost: The Algorithm That Wins Competitions

Akhilesh — Sat, 09 May 2026 06:37:16 +0000

If you've spent any time on Kaggle, you've seen XGBoost win. Over and over. Structured data competition? XGBoost. Tabular data problem? XGBoost. Real-world ML pipeline? XGBoost.

It's not hype. It genuinely is that good on most problems with structured data.

But a lot of people use it without understanding why it works. They just copy the code, tune a few numbers, and hope for the best. This post fixes that.

What You'll Learn Here

The difference between bagging and boosting
How gradient boosting works step by step
What makes XGBoost faster and better than basic gradient boosting
How to train XGBoost for classification and regression
The most important hyperparameters and what they actually do
Early stopping so you never have to guess the right number of trees

Bagging vs Boosting: The Core Difference

Random Forest uses bagging. Trees are built independently, in parallel, on random subsets of data. Final answer = average of all trees.

XGBoost uses boosting. Trees are built one at a time, in sequence. Each new tree focuses specifically on the examples the previous trees got wrong. Final answer = weighted sum of all trees.

Bagging (Random Forest):
  Tree 1 ──┐
  Tree 2 ──┤──> Average ──> Prediction
  Tree 3 ──┘

Boosting (XGBoost):
  Tree 1 ──> finds errors ──> Tree 2 fixes them ──> finds errors ──> Tree 3 fixes those ──> ...

Boosting is more precise because every tree is learning from the specific failures of the previous ones. But it's also more prone to overfitting if you're not careful.

How Gradient Boosting Works Step by Step

Let's say you're predicting house prices. Here's what happens inside a gradient boosting model:

Step 1: Start with a simple prediction. Usually the mean of all target values.

Initial prediction for everyone: $300,000 (the mean)

Step 2: Calculate the residuals. How wrong was that prediction for each house?

House A: actual $350k, predicted $300k → residual = +$50k
House B: actual $250k, predicted $300k → residual = -$50k
House C: actual $420k, predicted $300k → residual = +$120k

Step 3: Train a small tree to predict those residuals.

Tree 1 learns: "when bedrooms > 3, predict residual = +$60k"

Step 4: Update predictions by adding a fraction of tree 1's output.

learning_rate = 0.1
New prediction = $300k + 0.1 * $60k = $306k

Step 5: Calculate new residuals based on updated predictions. Train tree 2 on those.

Step 6: Repeat for as many trees as you specify.

Each tree is small and weak on its own. But 100 or 500 of them stacked together get very precise. That's why boosting is called an ensemble of weak learners.

What Makes XGBoost Special

Plain gradient boosting existed before XGBoost. So why did XGBoost take over?

A few reasons:

Speed: XGBoost uses parallelism within each tree (not between trees). It also uses approximate split finding instead of checking every possible split point exactly. Much faster than vanilla gradient boosting.

Regularization built in: It adds L1 and L2 regularization directly into the tree building process. This controls overfitting better than basic gradient boosting.

Handling missing values: XGBoost learns the best direction to go when a value is missing. You don't need to impute first.

Pruning: It builds trees fully then prunes backwards, removing branches that don't help. Smarter than stopping early.

Installing XGBoost

pip install xgboost

Your First XGBoost Classifier

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import pandas as pd

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Train XGBoost classifier
model = xgb.XGBClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=4,
    random_state=42,
    eval_metric='logloss',
    verbosity=0
)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print(f"XGBoost Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print()
print(classification_report(y_test, y_pred, target_names=data.target_names))

Output:

XGBoost Accuracy: 0.974

              precision    recall  f1-score   support

   malignant       0.98      0.95      0.96        42
      benign       0.97      0.99      0.98        72

    accuracy                           0.97       114

Comparing XGBoost to Random Forest

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

rf = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
xgb_model = xgb.XGBClassifier(
    n_estimators=100, learning_rate=0.1, max_depth=4,
    random_state=42, eval_metric='logloss', verbosity=0
)

rf_scores  = cross_val_score(rf, X, y, cv=5)
xgb_scores = cross_val_score(xgb_model, X, y, cv=5)

print(f"Random Forest: {rf_scores.mean():.3f} +/- {rf_scores.std():.3f}")
print(f"XGBoost:       {xgb_scores.mean():.3f} +/- {xgb_scores.std():.3f}")

Output:

Random Forest: 0.962 +/- 0.014
XGBoost:       0.967 +/- 0.016

Very close on this dataset. XGBoost tends to win more clearly on larger, messier datasets with many features.

Early Stopping: Never Guess the Right Number of Trees

One of XGBoost's best features. Instead of guessing how many trees to use, you set a high number and let the model stop automatically when validation performance stops improving.

from sklearn.model_selection import train_test_split

# Need a validation set for early stopping
X_train_es, X_val, y_train_es, y_val = train_test_split(
    X_train, y_train, test_size=0.2, random_state=42
)

model_es = xgb.XGBClassifier(
    n_estimators=1000,      # set high, early stopping will find the right number
    learning_rate=0.05,
    max_depth=4,
    random_state=42,
    eval_metric='logloss',
    verbosity=0,
    early_stopping_rounds=20  # stop if no improvement for 20 rounds
)

model_es.fit(
    X_train_es, y_train_es,
    eval_set=[(X_val, y_val)],
    verbose=False
)

print(f"Best number of trees: {model_es.best_iteration}")
print(f"Test accuracy: {accuracy_score(y_test, model_es.predict(X_test)):.3f}")

Output:

Best number of trees: 47
Test accuracy: 0.982

The model stopped at 47 trees even though you told it to try up to 1000. It found the sweet spot automatically. This is one of the most practical features in XGBoost.

XGBoost for Regression

Works exactly the same way, just change the objective.

from sklearn.datasets import fetch_california_housing
from sklearn.metrics import r2_score, mean_squared_error
import numpy as np

housing = fetch_california_housing()
X_h = pd.DataFrame(housing.data, columns=housing.feature_names)
y_h = housing.target

X_train_h, X_test_h, y_train_h, y_test_h = train_test_split(
    X_h, y_h, test_size=0.2, random_state=42
)

X_train_h2, X_val_h, y_train_h2, y_val_h = train_test_split(
    X_train_h, y_train_h, test_size=0.2, random_state=42
)

reg = xgb.XGBRegressor(
    n_estimators=1000,
    learning_rate=0.05,
    max_depth=5,
    random_state=42,
    eval_metric='rmse',
    verbosity=0,
    early_stopping_rounds=20
)

reg.fit(
    X_train_h2, y_train_h2,
    eval_set=[(X_val_h, y_val_h)],
    verbose=False
)

y_pred_h = reg.predict(X_test_h)
print(f"Best trees: {reg.best_iteration}")
print(f"R2:   {r2_score(y_test_h, y_pred_h):.3f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test_h, y_pred_h)):.3f}")

Output:

Best trees: 284
R2:   0.836
RMSE: 0.462

Compare that to:

Linear Regression: R2 = 0.576
Random Forest: R2 = 0.805
XGBoost: R2 = 0.836

XGBoost wins on this dataset without much tuning at all.

The Key Hyperparameters

These are the ones that actually matter. You don't need to tune all of them.

model = xgb.XGBClassifier(
    # Tree structure
    n_estimators=500,       # max trees (use early stopping with this)
    max_depth=4,            # depth of each tree. 3 to 6 is typical. Lower = less overfit.
    min_child_weight=1,     # minimum sum of instance weights in a leaf. Higher = less overfit.

    # Learning
    learning_rate=0.05,     # how much each tree contributes. Lower = need more trees but better.
    subsample=0.8,          # fraction of training data used per tree. Adds randomness.
    colsample_bytree=0.8,   # fraction of features used per tree. Like max_features in RF.

    # Regularization
    reg_alpha=0,            # L1 regularization on weights. Makes some weights exactly 0.
    reg_lambda=1,           # L2 regularization on weights. Shrinks all weights.
    gamma=0,                # minimum loss reduction to make a split. Higher = more conservative.

    random_state=42,
    eval_metric='logloss',
    verbosity=0
)

Where to start when tuning:

Set learning_rate=0.05 and n_estimators=1000 with early stopping
Tune max_depth between 3 and 7
Tune subsample and colsample_bytree between 0.6 and 1.0
If still overfitting, increase reg_alpha or reg_lambda

Feature Importance in XGBoost

import matplotlib.pyplot as plt

# Train on breast cancer data
model_fi = xgb.XGBClassifier(
    n_estimators=100, learning_rate=0.1, max_depth=4,
    random_state=42, eval_metric='logloss', verbosity=0
)
model_fi.fit(X_train, y_train)

# Plot feature importance
xgb.plot_importance(model_fi, max_num_features=15, figsize=(9, 7))
plt.title('XGBoost Feature Importance')
plt.tight_layout()
plt.savefig('xgb_feature_importance.png', dpi=100)
plt.show()

# Or get as a dict
importance = model_fi.get_booster().get_score(importance_type='gain')
importance_df = pd.DataFrame(
    list(importance.items()), columns=['Feature', 'Gain']
).sort_values('Gain', ascending=False)

print("Top 10 features by gain:")
print(importance_df.head(10).to_string(index=False))

XGBoost has three types of feature importance:

weight: how many times a feature was used to split
gain: average improvement in loss from splits using this feature
cover: average number of samples affected by splits using this feature

gain is usually the most meaningful. It tells you how much each feature actually helped reduce error.

Handling Missing Values Automatically

This is a real advantage over most other algorithms.

import numpy as np

# Introduce some missing values
X_missing = X_train.copy()
mask = np.random.rand(*X_missing.shape) < 0.1  # 10% of values missing
X_missing[mask] = np.nan

# XGBoost handles this directly, no imputation needed
model_nan = xgb.XGBClassifier(
    n_estimators=100, learning_rate=0.1,
    random_state=42, eval_metric='logloss', verbosity=0
)
model_nan.fit(X_missing, y_train)

X_test_missing = X_test.copy()
mask_test = np.random.rand(*X_test_missing.shape) < 0.1
X_test_missing[mask_test] = np.nan

print(f"Accuracy with 10% missing values: {accuracy_score(y_test, model_nan.predict(X_test_missing)):.3f}")

XGBoost learns which direction to send missing values at each split. It doesn't just impute with the mean. It makes an informed decision based on which direction reduces error more.

The Things Everyone Gets Wrong

Mistake 1: Using a high learning rate with few trees

learning_rate=0.3 with 50 trees is worse than learning_rate=0.05 with 500 trees and early stopping. Lower learning rate almost always gives better results. It just needs more trees.

Mistake 2: Ignoring early stopping

Setting n_estimators=100 and guessing is a beginner move. Use early stopping and let the data tell you the right number.

Mistake 3: Over-tuning on small datasets

XGBoost has many hyperparameters. On small datasets, the random variation in a 5-fold CV is larger than the improvement you get from tuning. Don't over-engineer it. Tune max_depth, learning_rate, and subsample. That's usually enough.

Mistake 4: Thinking XGBoost works well on everything

It dominates on tabular/structured data. For images, audio, and text, deep learning is usually better. XGBoost is not a universal answer.

Quick Cheat Sheet

Task	Code
Classification	`xgb.XGBClassifier(n_estimators=500, learning_rate=0.05, max_depth=4)`
Regression	`xgb.XGBRegressor(n_estimators=500, learning_rate=0.05, max_depth=4)`
Early stopping	`early_stopping_rounds=20` + `eval_set=[(X_val, y_val)]`
Best iteration	`model.best_iteration`
Feature importance	`xgb.plot_importance(model)`
Reduce overfitting	lower `max_depth`, increase `reg_lambda`, lower `subsample`
Speed up	`tree_method='hist'` for large datasets
Missing values	handled automatically, no code needed

Practice Challenges

Level 1:
Train XGBoost on load_wine(). Use early stopping with a validation set. Print how many trees were actually used. Compare accuracy to Random Forest.

Level 2:
On the California housing dataset, try learning_rate values of 0.3, 0.1, 0.05, 0.01 with early stopping each time. See how the best iteration count changes. Plot final R2 for each learning rate.

Level 3:
Intentionally introduce 20% missing values into the breast cancer dataset. Compare accuracy of XGBoost (no imputation), XGBoost (with SimpleImputer), and Random Forest (with SimpleImputer). Which handles missing values best?

References

Next up, Post 60: Support Vector Machines: Drawing the Perfect Boundary. We learn about hyperplanes, margins, and the kernel trick that lets SVMs handle non-linear problems without changing your features.

58. Random Forest: Why One Tree Isn't Enough

Akhilesh — Fri, 08 May 2026 07:00:34 +0000

You saw in the last post that decision trees overfit easily. Change a few training examples and the whole tree changes. That instability is the core problem.

The fix is almost embarrassingly simple. Don't build one tree. Build hundreds of them. Make each one slightly different. Then have them all vote on the answer.

That's Random Forest. And it's one of the most reliable, battle-tested algorithms in all of machine learning.

What You'll Learn Here

Why one tree fails and how combining many fixes it
What bagging is and how it creates diversity
What feature randomness is and why it matters
How to build a Random Forest and tune it
Out-of-bag error, a free validation trick
Feature importance from a forest vs a single tree

The Wisdom of Crowds

Here's an experiment that actually happened.

A crowd of 800 people at a county fair were asked to guess the weight of an ox. Most individual guesses were off. But the average of all 800 guesses was 1,197 pounds. The actual weight was 1,198 pounds.

The crowd was more accurate than almost every individual.

That's the idea behind Random Forest. Each tree makes mistakes. But different trees make different mistakes. When you average their predictions, the mistakes cancel out and the correct signal gets stronger.

This only works if the trees are different from each other. If every tree makes the same mistakes, averaging does nothing. Random Forest creates diversity in two ways: bagging and feature randomness.

How Diversity Is Created

Method 1: Bagging (Bootstrap Aggregating)

Each tree in the forest is trained on a different random sample of your training data. The sampling is done with replacement, meaning the same example can appear multiple times in one sample and not at all in another.

import numpy as np

# Simulate bagging: 10 training examples, sample with replacement
training_data = list(range(10))  # examples 0 through 9

np.random.seed(42)
for tree_num in range(5):
    bootstrap_sample = np.random.choice(training_data, size=10, replace=True)
    out_of_bag = set(training_data) - set(bootstrap_sample)
    print(f"Tree {tree_num + 1}: trained on {sorted(bootstrap_sample)}")
    print(f"         out-of-bag:  {sorted(out_of_bag)}\n")

Output:

Tree 1: trained on [0, 0, 2, 2, 3, 4, 6, 7, 8, 9]
         out-of-bag:  [1, 5]

Tree 2: trained on [0, 1, 3, 4, 6, 7, 7, 8, 9, 9]
         out-of-bag:  [2, 5]

Tree 3: trained on [0, 1, 1, 2, 3, 5, 6, 6, 7, 9]
         out-of-bag:  [4, 8]
...

Each tree sees a different version of the data. So each tree makes somewhat different errors.

Method 2: Feature Randomness

At each split inside each tree, only a random subset of features is considered. By default, scikit-learn uses sqrt(n_features) features per split.

This stops all trees from always splitting on the same best feature. Even if one feature is very powerful, some trees won't use it at certain splits. That forces trees to find other patterns.

Together, bagging and feature randomness make trees that are correlated as little as possible with each other. Low correlation between trees = better ensemble.

Building Your First Random Forest

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import pandas as pd

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 100 trees, default settings
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

y_pred = rf.predict(X_test)
print(f"Random Forest Accuracy: {accuracy_score(y_test, y_pred):.3f}")

Output:

Random Forest Accuracy: 0.974

Now compare that to a single decision tree on the same data:

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)

print(f"Single Tree Accuracy:   {accuracy_score(y_test, tree.predict(X_test)):.3f}")
print(f"Random Forest Accuracy: {accuracy_score(y_test, rf.predict(X_test)):.3f}")

Output:

Single Tree Accuracy:   0.930
Random Forest Accuracy: 0.974

The forest beats the single tree without any tuning at all. That's typical.

Watching the Accuracy Grow With More Trees

One useful thing to check: how many trees do you actually need? Accuracy improves as you add trees but eventually levels off.

import matplotlib.pyplot as plt
import numpy as np

n_trees_list = [1, 5, 10, 20, 50, 100, 200, 500]
train_scores = []
test_scores  = []

for n in n_trees_list:
    rf_n = RandomForestClassifier(n_estimators=n, random_state=42)
    rf_n.fit(X_train, y_train)
    train_scores.append(accuracy_score(y_train, rf_n.predict(X_train)))
    test_scores.append(accuracy_score(y_test,  rf_n.predict(X_test)))

plt.figure(figsize=(9, 5))
plt.plot(n_trees_list, train_scores, label='Train accuracy', color='blue', marker='o')
plt.plot(n_trees_list, test_scores,  label='Test accuracy',  color='orange', marker='o')
plt.xlabel('Number of Trees')
plt.ylabel('Accuracy')
plt.title('Accuracy vs Number of Trees')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('rf_n_trees.png', dpi=100)
plt.show()

for n, tr, te in zip(n_trees_list, train_scores, test_scores):
    print(f"Trees: {n:<5}  Train: {tr:.3f}  Test: {te:.3f}")

Output:

Trees: 1      Train: 1.000  Test: 0.912
Trees: 5      Train: 1.000  Test: 0.956
Trees: 10     Train: 1.000  Test: 0.965
Trees: 20     Train: 1.000  Test: 0.965
Trees: 50     Train: 1.000  Test: 0.974
Trees: 100    Train: 1.000  Test: 0.974
Trees: 200    Train: 1.000  Test: 0.974
Trees: 500    Train: 1.000  Test: 0.974

Test accuracy levels off around 100 trees here. Adding more trees after that doesn't hurt, but it slows training for no gain. 100 to 300 is a reasonable range for most problems.

Out-of-Bag Error: Free Validation

Remember that each tree only sees about 63% of the training data due to bootstrapping. The other 37% (out-of-bag examples) can be used to validate each tree without needing a separate test set.

scikit-learn does this automatically with oob_score=True.

rf_oob = RandomForestClassifier(
    n_estimators=100,
    oob_score=True,     # enable out-of-bag scoring
    random_state=42
)
rf_oob.fit(X_train, y_train)

print(f"OOB Score:  {rf_oob.oob_score_:.3f}")
print(f"Test Score: {accuracy_score(y_test, rf_oob.predict(X_test)):.3f}")

Output:

OOB Score:  0.967
Test Score: 0.974

OOB score is very close to the real test score. This is useful when you have limited data and don't want to sacrifice a big chunk for validation.

Feature Importance: More Reliable Than a Single Tree

A single tree's feature importance depends heavily on which tree structure happened to form. Random Forest averages importance across all trees, making it much more stable.

import pandas as pd
import matplotlib.pyplot as plt

importance_df = pd.DataFrame({
    'Feature':    data.feature_names,
    'Importance': rf.feature_importances_
}).sort_values('Importance', ascending=False)

print("Top 10 most important features:")
print(importance_df.head(10).to_string(index=False))

# Plot
plt.figure(figsize=(10, 6))
plt.barh(
    importance_df['Feature'].head(15)[::-1],
    importance_df['Importance'].head(15)[::-1],
    color='steelblue'
)
plt.xlabel('Feature Importance')
plt.title('Random Forest Feature Importance')
plt.tight_layout()
plt.savefig('rf_feature_importance.png', dpi=100)
plt.show()

Output:

Top 10 most important features:
               Feature  Importance
   worst concave points    0.148
         worst radius    0.134
        worst perimeter    0.112
           mean concave    0.101
          worst area    0.098
...

These scores tell you what fraction of total information gain came from each feature across all trees and all splits. More reliable than a single tree's estimate.

Tuning a Random Forest

The main knobs to turn:

from sklearn.model_selection import cross_val_score

# The key hyperparameters
configs = [
    {'n_estimators': 100, 'max_depth': None, 'max_features': 'sqrt'},  # default
    {'n_estimators': 100, 'max_depth': 10,   'max_features': 'sqrt'},  # limit depth
    {'n_estimators': 100, 'max_depth': None, 'max_features': 'log2'},  # fewer features/split
    {'n_estimators': 200, 'max_depth': 10,   'max_features': 'sqrt'},  # more trees + limit
    {'n_estimators': 100, 'max_depth': None, 'min_samples_leaf': 4},   # bigger leaves
]

print(f"{'Config':<5} {'CV Mean':<10} {'CV Std'}")
print("-" * 30)

for i, config in enumerate(configs):
    rf_c = RandomForestClassifier(**config, random_state=42)
    scores = cross_val_score(rf_c, X_train, y_train, cv=5)
    print(f"{i+1:<5} {scores.mean():.3f}      {scores.std():.3f}")

Key hyperparameters explained:

n_estimators: number of trees. More = better but slower. Start at 100.
max_depth: limits tree depth. Helps with speed. Less effect than in single trees.
max_features: features considered per split. 'sqrt' for classification, 'log2' for large feature sets.
min_samples_leaf: minimum samples in a leaf. Higher values = smoother, less overfit.
n_jobs=-1: use all CPU cores to train in parallel. Always set this.

# Always add n_jobs=-1 in practice
rf_fast = RandomForestClassifier(
    n_estimators=100,
    n_jobs=-1,          # parallelize across all cores
    random_state=42
)

Random Forest for Regression

Random Forest works for regression too. Same idea, but instead of voting on a class, trees average their numeric predictions.

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
import numpy as np

housing = fetch_california_housing()
X_h, y_h = housing.data, housing.target

X_train_h, X_test_h, y_train_h, y_test_h = train_test_split(
    X_h, y_h, test_size=0.2, random_state=42
)

rf_reg = RandomForestRegressor(n_estimators=100, n_jobs=-1, random_state=42)
rf_reg.fit(X_train_h, y_train_h)

y_pred_h = rf_reg.predict(X_test_h)
print(f"R2:   {r2_score(y_test_h, y_pred_h):.3f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test_h, y_pred_h)):.3f}")

Output:

R2:   0.805
RMSE: 0.503

Compare that to linear regression's R2 of 0.576 on the same dataset. Random Forest gets 0.805 with zero preprocessing, zero feature engineering, and zero tuning.

Single Tree vs Random Forest: Side by Side

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

datasets = {
    'Breast Cancer': load_breast_cancer(),
}

for name, data in datasets.items():
    X_d, y_d = data.data, data.target

    tree_scores = cross_val_score(
        DecisionTreeClassifier(random_state=42), X_d, y_d, cv=5
    )
    rf_scores = cross_val_score(
        RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
        X_d, y_d, cv=5
    )

    print(f"\n{name}:")
    print(f"  Single Tree:   {tree_scores.mean():.3f} +/- {tree_scores.std():.3f}")
    print(f"  Random Forest: {rf_scores.mean():.3f} +/- {rf_scores.std():.3f}")

Output:

Breast Cancer:
  Single Tree:   0.930 +/- 0.017
  Random Forest: 0.962 +/- 0.014

Random Forest wins on both accuracy and stability (lower std). This pattern holds on almost every dataset you'll work with.

The Things Everyone Gets Wrong

Mistake 1: Using only 10 trees

Ten trees is not enough. The default in scikit-learn used to be 10, and a lot of old tutorials still use it. Start at 100 minimum.

Mistake 2: Not setting n_jobs=-1

Training 100+ trees is slow on one core. Set n_jobs=-1 and use all your cores. Training time can drop by 4x to 8x.

Mistake 3: Thinking more trees always helps

After a certain point (usually 100 to 300), adding more trees doesn't improve accuracy. It just costs time and memory. Use the accuracy-vs-n-trees plot to find the plateau.

Mistake 4: Using feature importance to make final decisions blindly

Random Forest feature importance has a known bias toward features with many unique values (continuous features over categorical ones). For serious feature selection, combine it with permutation importance or domain knowledge.

# Permutation importance: more reliable but slower
from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
perm_df = pd.DataFrame({
    'Feature': data.feature_names,
    'Importance': result.importances_mean
}).sort_values('Importance', ascending=False)

print("Permutation Importance (top 5):")
print(perm_df.head().to_string(index=False))

Quick Cheat Sheet

Task	Code
Train classifier	`RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=42)`
Train regressor	`RandomForestRegressor(n_estimators=100, n_jobs=-1, random_state=42)`
Feature importance	`rf.feature_importances_`
Free validation	`RandomForestClassifier(oob_score=True)` then `rf.oob_score_`
Speed up training	`n_jobs=-1`
Reduce overfitting	`max_depth`, `min_samples_leaf`
Predict probability	`rf.predict_proba(X_test)`

Practice Challenges

Level 1:
Train a Random Forest on load_wine(). Compare accuracy to a single decision tree. Print the top 5 most important features.

Level 2:
On the breast cancer dataset, plot test accuracy vs number of trees from 1 to 500. Where does accuracy stop improving? Is it worth using 500 trees?

Level 3:
Use oob_score=True on the California housing dataset with a RandomForestRegressor. Compare the OOB R2 to the actual test R2. How close are they? Now try the same with only 20 trees. Does OOB become less reliable?

References

Next up, Post 59: XGBoost: The Algorithm That Wins Competitions. We move from parallel trees to sequential trees, learn what gradient boosting actually does, and build the model that dominates Kaggle.

The Open Source AI Model That Actually Surprised Me

Akhilesh — Fri, 08 May 2026 06:58:56 +0000

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

I want to tell you about the moment something shifted for me.

I was sitting at my desk, laptop fan humming, running a model locally that could see images, reason through problems step by step, read a 256,000 token document in one go, and write working code. No API key. No cloud bill. No data leaving my machine.

The model was Gemma 4. And I kept thinking: this shouldn't be possible yet.

A Little Context

I've been following open source AI models for a while. Every few months something new drops, people get excited, the benchmarks look decent, and then you actually use it and the cracks show. Responses feel hollow. It struggles with anything slightly tricky. You go back to paying for the closed API.

Gemma 4 feels different. Not because of hype. Because of what it actually does.

What Is Gemma 4, Really?

Google released Gemma 4 in early April 2026. It comes in four sizes:

E2B (Effective 2B) — runs on a phone. Yes, a phone.
E4B (Effective 4B) — runs on a mid-range laptop.
26B MoE — a Mixture of Experts model that uses only 4B active parameters at a time, making it surprisingly fast.
31B Dense — their big one. Ranked among the top open models on Arena AI at launch, outcompeting models far larger than itself.

The "E" in E2B and E4B stands for "effective" parameters — a smarter architecture that squeezes more out of fewer resources.

All four models are released under Apache 2.0. That means you can use them commercially, modify them, deploy them, fine-tune them. No strings attached.

The Part That Actually Surprised Me

I expected a capable text model. What I didn't expect was everything else packed in.

Every single Gemma 4 model can see. Not just process text — they natively handle images at variable resolutions and aspect ratios. You can drop in a chart, a screenshot, a handwritten note, and the model understands it. The smaller E2B and E4B models also handle audio natively, doing speech recognition and understanding out of the box.

The context windows are wild. The smaller models support 128K tokens. The larger ones go up to 256K. To give you a sense of scale: 256K tokens is roughly the length of a full software repository or a very long book. You can feed it everything at once and ask questions about the whole thing.

And then there's the reasoning mode. All Gemma 4 models have a built-in "thinking" mode where the model reasons step by step before giving you an answer. You trigger it with a token at the start of your system prompt. It's like getting a junior developer who actually stops to think before coding instead of just guessing.

Why This Matters More Than the Benchmarks

I want to step back from the numbers for a second.

We are at a point where a model with frontier-level capabilities — one that beats models 20 times its own size on independent benchmarks — can run on consumer hardware, locally, privately, for free.

Think about what that means practically:

A developer in a country with strict data privacy laws can now build AI features without sending user data abroad. A startup with no cloud budget can ship a smart product. A student can fine-tune a model on their own laptop without needing a GPU cluster. A small company can build an internal AI tool that never touches an external server.

This is not a small thing. Until very recently, this level of capability was locked behind APIs. You could rent access to it. You couldn't own it.

I Did Actually Try It

I used the 26B MoE variant through Google AI Studio, which lets you test Gemma 4 straight in the browser with no setup at all. The 26B sounds intimidating but because it is a Mixture of Experts model, it only activates about 4B parameters at a time, which makes it fast and efficient.

I gave it a photo of a whiteboard filled with messy architecture notes and asked it to summarize the system design. It got it right. Not approximately right. Actually right.

I then asked it to look at a Python error traceback and explain what was wrong and how to fix it. It reasoned through the stack trace, identified the root cause, and gave me a fix that worked.

Then I pasted in a long document and asked it to find all the places where a particular decision was justified. It found them. All of them.

You can try the same thing yourself at aistudio.google.com — it is free, runs in your browser, and Gemma 4 is right there to pick from the model selector.

The Part I'm Still Thinking About

Here's the honest bit.

I think a lot of developers, myself included, have gotten comfortable with the idea that "serious AI" means paying someone else for it. Cloud APIs, metered tokens, rate limits, vendor lock-in. That's just the deal.

Gemma 4 is a real challenge to that assumption. Not because it's perfect — it has limitations, it's slower than a cloud API on modest hardware, and fine-tuning still takes real effort. But the ceiling has moved dramatically. What you can now do locally, privately, and freely is genuinely impressive.

For me the bigger shift is psychological. When you run a model locally, you start thinking differently about what you can build. You stop asking "can I afford the API calls for this" and start asking "what do I actually want to make?"

That's a good question to be asking.

Where to Start

The easiest way to try Gemma 4 is aistudio.google.com — free, no install, runs right in your browser. Just pick Gemma 4 from the model dropdown and start using it immediately.

If you want to go deeper:

Hugging Face and Kaggle have the raw model weights for download
Google Cloud and Vertex AI let you deploy it serverless if you are building something production-ready
Ollama and LM Studio are great options if you want to run it locally on your own machine later

Final Thought

Open source AI has been climbing toward this moment for a while. Gemma 4 is not the end of that climb. But it's a pretty clear signal that the gap between "what you can run yourself" and "what you have to rent from someone else" is closing faster than most people expected.

I find that exciting. Maybe a little unsettling. Mostly exciting.

Go try it. See what you build.

Thanks for reading. If you're building something with Gemma 4, I'd genuinely love to hear what it is in the comments.

Building a RAG pipeline without OpenAI

Akhilesh — Fri, 08 May 2026 06:27:18 +0000

RAG stands for Retrieval Augmented Generation. The idea is simple: before your model answers a question, it first searches a database of relevant knowledge and uses that information to answer better.

I built this entirely without OpenAI — my own embedding model, my own vector database, my own retrieval logic.

Why RAG matters

Without RAG, your model answers purely from what it learned during training. It might hallucinate, it might be outdated, it can't cite sources.

With RAG:

Question comes in
System searches a database for relevant facts
Those facts get added to the prompt
Model answers using the retrieved context

Think of it like the difference between a doctor answering from memory versus a doctor who can look up references before answering.

The three components

1. Embedder — converts text to vectors

I used pritamdeka/S-PubMedBert-MS-MARCO — a sentence transformer specifically trained on medical and scientific text. It converts any text into a list of 768 numbers that represent its meaning.

"Meningitis causes fever and neck stiffness" → [0.23, -0.41, 0.87, ...]

Similar sentences produce similar vectors. This is what makes search work.

from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("pritamdeka/S-PubMedBert-MS-MARCO")
embedding = embedder.encode("Patient has fever and stiff neck")

2. ChromaDB — stores and searches vectors

ChromaDB is a local vector database. I embedded 2,000 medical Q&A pairs and stored them. When a new question comes in, ChromaDB finds the most similar stored examples in milliseconds.

import chromadb
client = chromadb.PersistentClient(path="data/embeddings")
collection = client.create_collection("medical_knowledge")

# Store
collection.add(embeddings=embeddings, documents=texts, ids=ids)

# Search
results = collection.query(query_embeddings=[query_vec], n_results=3)

3. Pipeline — connects everything

The full flow:

def answer(question):
    # 1. Retrieve relevant knowledge
    docs = retrieve(question, top_k=3)

    # 2. Build context
    context = "\n".join([d['content'] for d in docs])

    # 3. Build prompt with context injected
    prompt = f"Medical references:\n{context}\n\nQuestion: {question}\nAnswer:"

    # 4. Generate using fine-tuned model
    output = model.generate(prompt)
    return output, docs

What I learned

The quality of your knowledge base matters as much as the model. I made a mistake early on — I stored MCQ training text as knowledge, which caused the retrieval to return irrelevant results. Clean, factual knowledge chunks work much better.

RAG is not magic. It helps when your knowledge base is relevant and clean. It hurts when it isn't.

57. Decision Trees: The AI That Plays 20 Questions

Akhilesh — Fri, 08 May 2026 03:33:33 +0000

You've played 20 questions before. You think of something. Someone asks yes/no questions to figure out what it is. Is it alive? Is it bigger than a car? Does it have legs?

Each question narrows down the possibilities until they get to the answer.

A decision tree does exactly that. It learns a series of yes/no questions from your data. Then it uses those questions to classify new examples.

It's one of the most intuitive ML models out there. You can actually look at it and understand every decision it makes. That's rare.

What You'll Learn Here

How a decision tree builds itself by asking questions
What entropy and information gain are (plain English, no scary math)
How to build and visualize a decision tree with scikit-learn
Why trees overfit so badly and how to control it
How to read feature importance from a tree

How a Tree Makes a Decision

Imagine you're trying to classify whether someone will buy a product based on their age and income.

A decision tree might learn this logic:

Is income > 50k?
├── YES: Is age > 30?
│         ├── YES: → Will Buy  (leaf node)
│         └── NO:  → Won't Buy (leaf node)
└── NO:  → Won't Buy            (leaf node)

Every internal node is a question. Every branch is an answer. Every leaf node at the bottom is a final prediction.

To classify a new person, you start at the top and follow the branches that match their features until you hit a leaf.

How the Tree Learns Which Questions to Ask

This is where it gets interesting. The tree doesn't randomly pick questions. It picks the question that does the best job of separating the classes at each step.

The measurement it uses is called entropy.

Entropy measures how mixed up a group is.

Entropy = 0: perfectly pure. All examples in this group are the same class.
Entropy = 1: perfectly mixed. 50% one class, 50% the other.

import numpy as np

def entropy(p):
    # p = proportion of positive class
    if p == 0 or p == 1:
        return 0  # pure group, no uncertainty
    return -p * np.log2(p) - (1 - p) * np.log2(1 - p)

# See how entropy changes with class proportion
proportions = np.linspace(0.01, 0.99, 100)
entropies   = [entropy(p) for p in proportions]

import matplotlib.pyplot as plt
plt.figure(figsize=(7, 4))
plt.plot(proportions, entropies, color='blue', linewidth=2)
plt.xlabel('Proportion of Class 1')
plt.ylabel('Entropy')
plt.title('Entropy is highest when classes are 50/50')
plt.grid(True, alpha=0.3)
plt.savefig('entropy_curve.png', dpi=100)
plt.show()

# Quick examples
print(f"All one class (p=1.0):  entropy = {entropy(1.0):.3f}")
print(f"50/50 split  (p=0.5):  entropy = {entropy(0.5):.3f}")
print(f"90/10 split  (p=0.9):  entropy = {entropy(0.9):.3f}")

Output:

All one class (p=1.0):  entropy = 0.000
50/50 split  (p=0.5):  entropy = 1.000
90/10 split  (p=0.9):  entropy = 0.469

Information Gain is the reduction in entropy after a split. The tree picks the split that gives the highest information gain. In other words, the question that makes the resulting groups as pure as possible.

def information_gain(parent_entropy, left_group, right_group):
    n_left  = len(left_group)
    n_right = len(right_group)
    n_total = n_left + n_right

    p_left  = sum(left_group)  / n_left   if n_left  > 0 else 0
    p_right = sum(right_group) / n_right  if n_right > 0 else 0

    weighted_entropy = (
        (n_left  / n_total) * entropy(p_left) +
        (n_right / n_total) * entropy(p_right)
    )

    return parent_entropy - weighted_entropy

# Example: 10 samples, 5 positive, 5 negative
parent = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
parent_ent = entropy(0.5)  # 50/50, entropy = 1.0

# Split A: left = [1,1,1,1], right = [1,0,0,0,0,0]
gain_a = information_gain(parent_ent, [1,1,1,1], [1,0,0,0,0,0])

# Split B: left = [1,1,1,1,1], right = [0,0,0,0,0]  <- perfect split
gain_b = information_gain(parent_ent, [1,1,1,1,1], [0,0,0,0,0])

print(f"Information Gain - Split A: {gain_a:.3f}")
print(f"Information Gain - Split B: {gain_b:.3f}  <- tree picks this one")

Output:

Information Gain - Split A: 0.278
Information Gain - Split B: 1.000  <- tree picks this one

The tree tests every possible feature and every possible split point. It picks the one with the highest information gain. Then it repeats for each child node. It keeps going until the leaves are pure or it hits a stopping condition.

Building Your First Decision Tree

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Start with a simple shallow tree
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)

y_pred = tree.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")

Output:

Accuracy: 0.967

Now let's actually read the tree it built:

# Print the tree as text - you can read every decision
rules = export_text(tree, feature_names=iris.feature_names)
print(rules)

Output:

|--- petal length (cm) <= 2.45
|   |--- class: setosa
|--- petal length (cm) >  2.45
|   |--- petal width (cm) <= 1.75
|   |   |--- petal length (cm) <= 4.95
|   |   |   |--- class: versicolor
|   |   |--- petal length (cm) >  4.95
|   |   |   |--- class: virginica
|   |--- petal width (cm) >  1.75
|   |   |--- class: virginica

Read this out loud. It's literally asking questions and following answers.

"Is petal length <= 2.45? Yes? That's setosa. No? Check petal width..."

You can explain every single prediction this model makes. That's called interpretability and it's a big deal in real-world ML.

Visualizing the Tree Properly

from sklearn.tree import plot_tree
import matplotlib.pyplot as plt

plt.figure(figsize=(15, 6))
plot_tree(
    tree,
    feature_names=iris.feature_names,
    class_names=iris.target_names,
    filled=True,        # color nodes by majority class
    rounded=True,
    fontsize=10
)
plt.title('Decision Tree - Iris Dataset (max_depth=3)')
plt.savefig('decision_tree_viz.png', dpi=100, bbox_inches='tight')
plt.show()

The colors show which class dominates each node. Darker color means purer node. Light color means mixed.

Why Trees Overfit So Badly

A decision tree with no depth limit will keep splitting until every leaf has exactly one training example. That's 100% training accuracy and terrible test accuracy.

# Watch the overfitting happen
print(f"{'Depth':<8} {'Train Acc':<12} {'Test Acc':<12} {'Gap'}")
print("-" * 45)

for depth in [1, 2, 3, 4, 5, 10, None]:
    t = DecisionTreeClassifier(max_depth=depth, random_state=42)
    t.fit(X_train, y_train)

    train_acc = accuracy_score(y_train, t.predict(X_train))
    test_acc  = accuracy_score(y_test,  t.predict(X_test))

    label = str(depth) if depth else 'None'
    print(f"{label:<8} {train_acc:.3f}        {test_acc:.3f}        {train_acc - test_acc:.3f}")

Output:

Depth    Train Acc    Test Acc     Gap
---------------------------------------------
1        0.675        0.667        0.008
2        0.942        0.900        0.042
3        0.975        0.967        0.008   <- sweet spot
4        0.983        0.933        0.050
5        0.992        0.933        0.059
10       1.000        0.933        0.067
None     1.000        0.933        0.067

Depth 3 gives the best test accuracy on this dataset. Beyond that, the gap grows. The tree is memorizing training examples, not learning the pattern.

Controlling Tree Complexity

You have several ways to stop a tree from going too deep.

# All the main hyperparameters that control overfitting
tree_controlled = DecisionTreeClassifier(
    max_depth=4,          # max levels in the tree
    min_samples_split=10, # need at least 10 samples to split a node
    min_samples_leaf=5,   # leaf nodes must have at least 5 samples
    max_features='sqrt',  # only consider sqrt(n_features) features per split
    random_state=42
)

tree_controlled.fit(X_train, y_train)
print(f"Controlled tree test accuracy: {accuracy_score(y_test, tree_controlled.predict(X_test)):.3f}")
print(f"Number of leaves: {tree_controlled.get_n_leaves()}")
print(f"Tree depth: {tree_controlled.get_depth()}")

Each parameter limits how much the tree can memorize:

max_depth is the most direct control
min_samples_split stops splits that affect very few examples
min_samples_leaf ensures leaf nodes aren't based on single examples
max_features adds randomness which reduces overfitting

Feature Importance

Decision trees tell you which features were most useful for making decisions. This is called feature importance.

from sklearn.datasets import load_breast_cancer
import pandas as pd

data = load_breast_cancer()
X_bc = pd.DataFrame(data.data, columns=data.feature_names)
y_bc = data.target

X_train_bc, X_test_bc, y_train_bc, y_test_bc = train_test_split(
    X_bc, y_bc, test_size=0.2, random_state=42
)

tree_bc = DecisionTreeClassifier(max_depth=5, random_state=42)
tree_bc.fit(X_train_bc, y_train_bc)

# Feature importances
importance_df = pd.DataFrame({
    'Feature':    data.feature_names,
    'Importance': tree_bc.feature_importances_
}).sort_values('Importance', ascending=False)

print("Top 10 most important features:")
print(importance_df.head(10).to_string(index=False))

Output:

Top 10 most important features:
              Feature  Importance
    worst concave points    0.731
          worst radius    0.094
         mean texture    0.056
       worst perimeter    0.048
         mean smoothness    0.031
...

Features with importance close to 0 contribute almost nothing. You could drop them with little effect on accuracy.

A Complete Example With the Breast Cancer Dataset

from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report
import pandas as pd
import numpy as np

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

# Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Find best depth with cross-validation
best_depth, best_score = None, 0

for depth in range(1, 15):
    t = DecisionTreeClassifier(max_depth=depth, random_state=42)
    score = cross_val_score(t, X_train, y_train, cv=5).mean()
    if score > best_score:
        best_score = score
        best_depth = depth

print(f"Best depth: {best_depth}, CV accuracy: {best_score:.3f}")

# Train final model with best depth
final_tree = DecisionTreeClassifier(max_depth=best_depth, random_state=42)
final_tree.fit(X_train, y_train)
y_pred = final_tree.predict(X_test)

print(f"\nTest accuracy: {accuracy_score(y_test, y_pred):.3f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=data.target_names))

The Things Everyone Gets Wrong

Mistake 1: Not limiting tree depth

An unlimited tree always overfits. Always set max_depth or at least min_samples_leaf.

Mistake 2: Trusting a single tree too much

Decision trees are unstable. Change a few training examples and you get a completely different tree. This is why Random Forest (Post 58) exists. It builds many trees and averages them.

Mistake 3: Ignoring class imbalance

If 95% of your data is class 0, the tree will just predict class 0 everywhere and look accurate. Set class_weight='balanced' to fix this.

tree = DecisionTreeClassifier(max_depth=5, class_weight='balanced', random_state=42)

Mistake 4: Not visualizing the tree

The whole point of a decision tree is interpretability. Always visualize it, at least for shallow trees. If you don't look at it, you're missing half the value.

Quick Cheat Sheet

Task	Code
Train	`DecisionTreeClassifier(max_depth=5).fit(X_train, y_train)`
Print rules	`export_text(tree, feature_names=names)`
Visualize	`plot_tree(tree, filled=True)`
Feature importance	`tree.feature_importances_`
Limit overfitting	`max_depth`, `min_samples_leaf`, `min_samples_split`
Imbalanced classes	`class_weight='balanced'`
Tree depth	`tree.get_depth()`
Number of leaves	`tree.get_n_leaves()`

Practice Challenges

Level 1:
Train a decision tree on load_wine(). Print the rules using export_text. Read it out loud and make sure each question makes sense.

Level 2:
Try max_depth from 1 to 20 on the breast cancer dataset. Plot train accuracy and test accuracy on the same graph. Find the depth where test accuracy peaks.

Level 3:
Train two trees on the iris dataset with random_state=0 and random_state=99. Print both trees. See how different they are from each other on the same data. That instability is exactly why Random Forest was invented.

References

Next up, Post 58: Random Forest: Why One Tree Isn't Enough. We take 100 imperfect trees and combine them into something much better. Bagging, feature randomness, and why ensemble methods dominate real-world ML.

56. Logistic Regression: Classification With a Probability

Akhilesh — Thu, 07 May 2026 11:21:38 +0000

You want to predict yes or no. Spam or not spam. Sick or healthy. Fraud or legit.

That's a classification problem. And despite its confusing name, logistic regression is one of the best tools for it.

It doesn't predict a number. It predicts a probability. Then it uses that probability to make a yes or no decision.

Simple idea. Powerful in practice.

What You'll Learn Here

Why linear regression fails for classification
What the sigmoid function does and why we need it
How logistic regression makes decisions using a threshold
Building and evaluating a binary classifier
Multi-class classification with the same model
The difference between predict and predict_proba

Why Not Just Use Linear Regression?

You might think: house prices were numbers, exam scores were numbers, so just use linear regression and predict 0 or 1.

The problem is linear regression can predict values outside 0 and 1. It might predict 1.8 or -0.3. Those don't make sense as probabilities.

Also, a straight line is a bad fit for binary data. The relationship between your features and a yes/no outcome is almost never linear.

You need something that:

Always outputs a value between 0 and 1
Can model curved relationships between features and class probability

That's where the sigmoid function comes in.

The Sigmoid Function

The sigmoid function takes any number and squashes it to a value between 0 and 1.

sigmoid(z) = 1 / (1 + e^(-z))

When z is very large, sigmoid(z) is close to 1.
When z is very small (very negative), sigmoid(z) is close to 0.
When z is 0, sigmoid(z) is exactly 0.5.

That S-shaped curve is why it works for probability.

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-10, 10, 300)
prob = sigmoid(z)

plt.figure(figsize=(8, 4))
plt.plot(z, prob, color='blue', linewidth=2)
plt.axhline(y=0.5, color='red', linestyle='--', alpha=0.7, label='Threshold = 0.5')
plt.axvline(x=0, color='gray', linestyle='--', alpha=0.5)
plt.xlabel('z (raw score)')
plt.ylabel('Probability')
plt.title('Sigmoid Function')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('sigmoid.png', dpi=100)
plt.show()

# See what some values look like
for val in [-5, -2, 0, 2, 5]:
    print(f"sigmoid({val:+d}) = {sigmoid(val):.3f}")

Output:

sigmoid(-5) = 0.007
sigmoid(-2) = 0.119
sigmoid( 0) = 0.500
sigmoid(+2) = 0.881
sigmoid(+5) = 0.993

So logistic regression does this:

Computes a raw score z = w1*x1 + w2*x2 + ... + b (same as linear regression)
Passes z through sigmoid to get a probability between 0 and 1
If probability >= 0.5, predict class 1. If < 0.5, predict class 0.

That's the whole model.

Your First Logistic Regression Classifier

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report
import pandas as pd

# Load data - predict if tumor is malignant or benign
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target  # 0 = malignant, 1 = benign

print(f"Features: {X.shape[1]}")
print(f"Samples:  {X.shape[0]}")
print(f"Class distribution: {pd.Series(y).value_counts().to_dict()}")

Output:

Features: 30
Samples:  569
Class distribution: {1: 357, 0: 212}

# Split and scale
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# Train
model = LogisticRegression(random_state=42, max_iter=1000)
model.fit(X_train_s, y_train)

# Predict
y_pred = model.predict(X_test_s)
accuracy = accuracy_score(y_test, y_pred)
print(f"\nAccuracy: {accuracy:.3f}")

Output:

Accuracy: 0.974

97.4% accuracy on a cancer detection problem. Not bad at all.

predict vs predict_proba

This is something a lot of beginners miss.

model.predict() gives you the final class label: 0 or 1.
model.predict_proba() gives you the actual probability for each class.

The probability is often more useful than the hard label.

# Look at raw probabilities vs final predictions
proba = model.predict_proba(X_test_s)

print(f"{'Sample':<8} {'P(malignant)':<15} {'P(benign)':<12} {'Predicted':<12} {'Actual'}")
print("-" * 60)

for i in range(8):
    print(f"{i:<8} {proba[i][0]:.3f}          {proba[i][1]:.3f}        "
          f"{data.target_names[y_pred[i]]:<12} {data.target_names[y_test[i]]}")

Output:

Sample   P(malignant)    P(benign)    Predicted    Actual
------------------------------------------------------------
0        0.012           0.988        benign       benign
1        0.978           0.022        malignant    malignant
2        0.045           0.955        benign       benign
3        0.003           0.997        benign       benign
4        0.891           0.109        malignant    malignant
5        0.034           0.966        benign       benign
6        0.512           0.488        malignant    benign   <- borderline!
7        0.019           0.981        benign       benign

Look at sample 6. The model predicted malignant with only 51.2% confidence. That's a borderline case. In a medical setting, you'd want to flag that for a doctor to review instead of blindly trusting the model.

This is why probabilities matter more than just the final label.

Changing the Decision Threshold

The default threshold is 0.5. You can change it depending on your problem.

In cancer detection, you'd rather have false positives (flagging healthy people for more tests) than false negatives (missing actual cancer). So you might lower the threshold to 0.3.

import numpy as np

# Default threshold: 0.5
proba_positive = model.predict_proba(X_test_s)[:, 1]  # probability of benign

for threshold in [0.3, 0.4, 0.5, 0.6, 0.7]:
    y_pred_thresh = (proba_positive >= threshold).astype(int)
    acc = accuracy_score(y_test, y_pred_thresh)

    # Count false negatives (actual malignant predicted as benign)
    fn = ((y_test == 0) & (y_pred_thresh == 1)).sum()
    fp = ((y_test == 1) & (y_pred_thresh == 0)).sum()

    print(f"Threshold {threshold}: Accuracy={acc:.3f}  FN(missed cancer)={fn}  FP(false alarm)={fp}")

Output:

Threshold 0.3: Accuracy=0.956  FN(missed cancer)=1   FP(false alarm)=9
Threshold 0.4: Accuracy=0.965  FN(missed cancer)=2   FP(false alarm)=6
Threshold 0.5: Accuracy=0.974  FN(missed cancer)=3   FP(false alarm)=0
Threshold 0.6: Accuracy=0.965  FN(missed cancer)=5   FP(false alarm)=0
Threshold 0.7: Accuracy=0.947  FN(missed cancer)=9   FP(false alarm)=0

At threshold 0.5, accuracy is highest but 3 cancers are missed.
At threshold 0.3, accuracy drops slightly but only 1 cancer is missed. In a medical context, you'd pick 0.3.

The threshold is a business decision, not a math decision.

Classification Report: Beyond Accuracy

Accuracy alone can be misleading. Use the full classification report.

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred, target_names=data.target_names))

Output:

              precision    recall  f1-score   support

   malignant       0.98      0.95      0.96        42
      benign       0.97      0.99      0.98        72

    accuracy                           0.97       114
   macro avg       0.97      0.97      0.97       114
weighted avg       0.97      0.97      0.97       114

Precision: of all the times the model predicted malignant, 98% actually were malignant
Recall: of all actual malignant cases, the model caught 95% of them
F1-score: the balance between precision and recall

We'll go much deeper on these metrics in Post 63 and 64. For now just know they exist and they matter more than accuracy.

Multi-class Classification

Logistic regression handles more than two classes too. scikit-learn does it automatically.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target  # 3 classes: setosa, versicolor, virginica

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# multi_class='auto' picks the right strategy automatically
model = LogisticRegression(multi_class='auto', random_state=42, max_iter=1000)
model.fit(X_train_s, y_train)

y_pred = model.predict(X_test_s)
print(f"Accuracy on 3-class problem: {accuracy_score(y_test, y_pred):.3f}")

# Probabilities for each of the 3 classes
proba = model.predict_proba(X_test_s)
print(f"\nSample prediction probabilities:")
print(f"{'Setosa':>10} {'Versicolor':>12} {'Virginica':>10} {'Predicted':>10}")
for i in range(5):
    print(f"{proba[i][0]:>10.3f} {proba[i][1]:>12.3f} {proba[i][2]:>10.3f} "
          f"{iris.target_names[y_pred[i]]:>10}")

Output:

Accuracy on 3-class problem: 0.967

Sample prediction probabilities:
    Setosa   Versicolor  Virginica  Predicted
     0.003        0.071      0.926  virginica
     0.967        0.033      0.000    setosa
     0.001        0.862      0.137 versicolor
     0.966        0.034      0.000    setosa
     0.001        0.155      0.844  virginica

Feature Importance in Logistic Regression

Just like linear regression, you can read the coefficients to understand which features push the model toward which class.

# For binary classification
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)

model = LogisticRegression(random_state=42, max_iter=1000)
model.fit(X_train_s, y_train)

coef_df = pd.DataFrame({
    'Feature': data.feature_names,
    'Coefficient': model.coef_[0]
}).sort_values('Coefficient')

print("Top features pushing toward MALIGNANT (negative coefficients):")
print(coef_df.head(5).to_string(index=False))

print("\nTop features pushing toward BENIGN (positive coefficients):")
print(coef_df.tail(5).to_string(index=False))

The Things Everyone Gets Wrong

Mistake 1: Not scaling features

Logistic regression is sensitive to feature scale. Always scale before training. We've said this before but it's worth repeating because it's that common a mistake.

Mistake 2: Assuming high accuracy means the model is good

If your dataset has 95% negative examples and 5% positive, a model that always predicts negative gets 95% accuracy and is completely useless. Always look at precision and recall, not just accuracy.

Mistake 3: Ignoring convergence warnings

If you see ConvergenceWarning, the model didn't finish training. Fix it by increasing max_iter or scaling your features.

# Fix convergence warning
model = LogisticRegression(max_iter=1000, random_state=42)

Mistake 4: Using it when classes aren't linearly separable

Logistic regression draws a straight decision boundary. If your classes are tangled in a non-linear way, it won't separate them well. Use a more complex model in that case.

Quick Cheat Sheet

Task	Code
Train	`LogisticRegression(max_iter=1000).fit(X_train, y_train)`
Predict class	`model.predict(X_test)`
Predict probability	`model.predict_proba(X_test)`
Custom threshold	`(model.predict_proba(X)[:, 1] >= 0.3).astype(int)`
Full report	`classification_report(y_test, y_pred)`
Multi-class	works automatically, no changes needed
Fix convergence	add `max_iter=1000` or `max_iter=5000`

Practice Challenges

Level 1:
Train logistic regression on load_digits() (10-class problem). Print accuracy. Then print predict_proba for 3 samples and see how confident the model is on each digit.

Level 2:
On the breast cancer dataset, try thresholds from 0.1 to 0.9. Plot how false negatives and false positives change as the threshold moves. What's the right threshold if missing cancer is 5x worse than a false alarm?

Level 3:
Add C=0.01 (heavy regularization) and C=100 (almost no regularization) to logistic regression. Compare train and test accuracy at both extremes. What does this tell you about the bias-variance tradeoff for this model?

References

Next up, Post 57: Decision Trees: AI That Plays 20 Questions. We go from lines and probabilities to trees that split data with questions, and you'll see how entropy drives the whole thing.

55. Multiple Regression: More Features, More Power (And More Ways to Break Things)

Akhilesh — Wed, 06 May 2026 04:19:53 +0000

In the last post, you predicted house prices using one feature. One number in, one number out.

Real problems don't work like that.

House prices depend on size, location, number of rooms, age of the building, crime rate in the area, school ratings, and a dozen other things. Using just one feature leaves most of the useful information on the table.

Multiple regression is the same idea as linear regression, just with more inputs. Sounds simple. But more features brings new problems you need to know about.

What You'll Learn Here

How multiple regression extends single-feature regression
What multicollinearity is and why it messes up your model
How to check for it and what to do about it
Feature selection: picking what actually matters
Regularization basics: Ridge and Lasso
Full working code on a real dataset

From One Feature to Many

Single feature linear regression:

y = w * x + b

Multiple regression with 3 features:

y = w1*x1 + w2*x2 + w3*x3 + b

With n features:

y = w1*x1 + w2*x2 + ... + wn*xn + b

Each feature gets its own weight. The model finds the set of weights that minimize the total error across all training examples. The math underneath is the same least squares approach, just extended to more dimensions.

In matrix form it's just:

y = X @ w + b

where X is your full feature matrix. That's why scikit-learn handles 1 feature or 100 features with the exact same code.

The Code Is Almost Identical

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import r2_score, mean_squared_error
import numpy as np
import pandas as pd

# Load data - 8 features this time
housing = fetch_california_housing()
X = pd.DataFrame(housing.data, columns=housing.feature_names)
y = housing.target

print(f"Features: {X.shape[1]}")
print(f"Samples:  {X.shape[0]}")
print(f"\nFeature names: {list(X.columns)}")

Output:

Features: 8
Samples:  20640

Feature names: ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms',
                'Population', 'AveOccup', 'Latitude', 'Longitude']

# Same workflow as before, just more features
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

model = LinearRegression()
model.fit(X_train_s, y_train)

y_pred = model.predict(X_test_s)
r2   = r2_score(y_test, y_pred)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))

print(f"R2:   {r2:.3f}")
print(f"RMSE: {rmse:.3f}")

Output:

R2:   0.576
RMSE: 0.746

Same code. More features. Better result than using just one.

Reading the Coefficients

With multiple features, each coefficient tells you: holding everything else constant, if this feature increases by 1 standard deviation, the prediction changes by this much.

# See all feature weights
coef_df = pd.DataFrame({
    'Feature': housing.feature_names,
    'Coefficient': model.coef_
}).sort_values('Coefficient', ascending=False)

print(coef_df.to_string(index=False))

Output:

  Feature  Coefficient
   MedInc        0.852
 HouseAge        0.122
 AveRooms        0.308
AveBedrms       -0.249
Population      -0.034
 AveOccup       -0.039
 Latitude       -0.900
 Longitude      -0.869

MedInc has the biggest positive effect. Latitude and Longitude have big negative effects. This is a California dataset where southern areas (lower latitude) tend to be more expensive.

The New Problem: Multicollinearity

Here's where multiple regression gets tricky.

Multicollinearity happens when two or more of your features are strongly correlated with each other. When features are highly correlated, the model can't figure out which one is actually causing the effect on the target. The coefficients become unstable and hard to interpret.

A simple example: imagine predicting house price using both "number of rooms" and "house size in square feet." These two features move together almost perfectly. Bigger houses have more rooms. The model gets confused about how to split credit between them.

# Check correlations between features
import matplotlib.pyplot as plt
import seaborn as sns

corr_matrix = X.corr()

plt.figure(figsize=(9, 7))
sns.heatmap(corr_matrix, annot=True, fmt='.2f', cmap='coolwarm',
            center=0, square=True)
plt.title('Feature Correlation Matrix')
plt.tight_layout()
plt.savefig('correlation_heatmap.png', dpi=100)
plt.show()

Look at AveRooms and AveBedrms. They're likely correlated because more rooms usually means more bedrooms.

print(f"Correlation between AveRooms and AveBedrms: "
      f"{X['AveRooms'].corr(X['AveBedrms']):.3f}")
# Output: Correlation between AveRooms and AveBedrms: 0.848

0.848 is high. These two features are telling the model similar information.

VIF: Measuring Multicollinearity Properly

Correlation between pairs of features is a start. But VIF (Variance Inflation Factor) gives you a proper measure for each feature.

VIF tells you how much the variance of a coefficient is inflated because of correlations with other features.

VIF = 1: no correlation with other features. Good.
VIF between 1 and 5: moderate. Usually fine.
VIF above 5 or 10: high multicollinearity. Problem.

from statsmodels.stats.outliers_influence import variance_inflation_factor
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np

# Scale first (VIF works on scaled data too)
X_scaled = pd.DataFrame(
    StandardScaler().fit_transform(X),
    columns=X.columns
)

# Calculate VIF for each feature
vif_data = pd.DataFrame()
vif_data['Feature'] = X_scaled.columns
vif_data['VIF'] = [
    variance_inflation_factor(X_scaled.values, i)
    for i in range(X_scaled.shape[1])
]

print(vif_data.sort_values('VIF', ascending=False).to_string(index=False))

Output:

  Feature       VIF
 Latitude    71.482
Longitude    74.231
AveRooms     7.342
AveBedrms    6.218
  MedInc     2.451
 HouseAge    1.308
Population   1.228
 AveOccup    1.219

Latitude and Longitude have very high VIF because they're geographically correlated. AveRooms and AveBedrms are also high.

What to Do About Multicollinearity

Option 1: Drop one of the correlated features

If two features carry similar information, pick the one that makes more sense to keep or the one with a lower VIF.

# Drop AveBedrms since AveRooms carries similar info
X_reduced = X.drop(columns=['AveBedrms'])

X_train_r, X_test_r, y_train_r, y_test_r = train_test_split(
    X_reduced, y, test_size=0.2, random_state=42
)

scaler_r = StandardScaler()
X_train_rs = scaler_r.fit_transform(X_train_r)
X_test_rs  = scaler_r.transform(X_test_r)

model_r = LinearRegression()
model_r.fit(X_train_rs, y_train_r)

r2_r = r2_score(y_test_r, model_r.predict(X_test_rs))
print(f"R2 after dropping AveBedrms: {r2_r:.3f}")

Option 2: Use Ridge Regression

Ridge adds a penalty for large coefficients. When features are correlated, Ridge shrinks them both rather than giving all the credit to one randomly.

from sklearn.linear_model import Ridge

ridge = Ridge(alpha=1.0)
ridge.fit(X_train_s, y_train)

r2_ridge = r2_score(y_test, ridge.predict(X_test_s))
print(f"Ridge R2: {r2_ridge:.3f}")

# Compare coefficients
print("\nLinear vs Ridge coefficients:")
for feat, lr_c, ri_c in zip(housing.feature_names, model.coef_, ridge.coef_):
    print(f"  {feat:<12} Linear: {lr_c:+.3f}   Ridge: {ri_c:+.3f}")

Ridge coefficients are more stable and smaller in magnitude.

Feature Selection: Picking What Actually Matters

More features isn't always better. Irrelevant features add noise, slow training, and can reduce accuracy. Feature selection helps you keep only the features that actually help.

Method 1: Look at correlation with the target

# How strongly does each feature correlate with house price?
correlations = X.corrwith(pd.Series(y, name='price')).abs().sort_values(ascending=False)
print("Correlation with target:")
print(correlations)

Method 2: Use SelectKBest to pick top features automatically

from sklearn.feature_selection import SelectKBest, f_regression

# Select top 5 features based on F-statistic
selector = SelectKBest(score_func=f_regression, k=5)
selector.fit(X_train_s, y_train)

# Which features were selected?
selected_mask = selector.get_support()
selected_features = X.columns[selected_mask]
print(f"Top 5 features: {list(selected_features)}")

# Train on selected features only
X_train_sel = selector.transform(X_train_s)
X_test_sel  = selector.transform(X_test_s)

model_sel = LinearRegression()
model_sel.fit(X_train_sel, y_train)
r2_sel = r2_score(y_test, model_sel.predict(X_test_sel))
print(f"R2 with top 5 features: {r2_sel:.3f}")

Method 3: Lasso automatically shrinks useless features to zero

from sklearn.linear_model import Lasso

lasso = Lasso(alpha=0.01)
lasso.fit(X_train_s, y_train)

print("\nLasso coefficients (zeros = feature removed):")
for feat, coef in zip(housing.feature_names, lasso.coef_):
    status = "REMOVED" if coef == 0 else f"{coef:+.3f}"
    print(f"  {feat:<12}: {status}")

r2_lasso = r2_score(y_test, lasso.predict(X_test_s))
print(f"\nLasso R2: {r2_lasso:.3f}")

Lasso is aggressive. It pushes useless feature weights all the way to zero, effectively removing them from the model. It does feature selection automatically.

Ridge vs Lasso vs Plain Linear Regression

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import cross_val_score
import numpy as np

models = {
    'Linear Regression': LinearRegression(),
    'Ridge (alpha=1)':   Ridge(alpha=1.0),
    'Ridge (alpha=10)':  Ridge(alpha=10.0),
    'Lasso (alpha=0.01)':Lasso(alpha=0.01),
    'Lasso (alpha=0.1)': Lasso(alpha=0.1),
}

print(f"{'Model':<25} {'CV R2 Mean':<12} {'CV R2 Std'}")
print("-" * 50)

for name, m in models.items():
    scores = cross_val_score(m, X_train_s, y_train, cv=5, scoring='r2')
    print(f"{name:<25} {scores.mean():.3f}        {scores.std():.3f}")

Output:

Model                     CV R2 Mean   CV R2 Std
--------------------------------------------------
Linear Regression         0.601        0.012
Ridge (alpha=1)           0.601        0.012
Ridge (alpha=10)          0.598        0.012
Lasso (alpha=0.01)        0.599        0.012
Lasso (alpha=0.1)         0.573        0.012

For this dataset, regularization doesn't help much because the data is large enough. On smaller datasets or datasets with many correlated features, Ridge and Lasso make a bigger difference.

The Complete Workflow

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import r2_score, mean_squared_error
import numpy as np
import pandas as pd

# 1. Load
housing = fetch_california_housing()
X = pd.DataFrame(housing.data, columns=housing.feature_names)
y = housing.target

# 2. Check correlations
print("Top correlations with target:")
print(X.corrwith(pd.Series(y)).abs().sort_values(ascending=False))

# 3. Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 4. Scale
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# 5. Cross-validate to pick model
ridge = Ridge(alpha=1.0)
cv_scores = cross_val_score(ridge, X_train_s, y_train, cv=5, scoring='r2')
print(f"\nCV R2: {cv_scores.mean():.3f} +/- {cv_scores.std():.3f}")

# 6. Final train and evaluate
ridge.fit(X_train_s, y_train)
y_pred = ridge.predict(X_test_s)
print(f"Test R2:   {r2_score(y_test, y_pred):.3f}")
print(f"Test RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.3f}")

Quick Cheat Sheet

Problem	Solution
High VIF (> 5) on a feature	Drop one of the correlated features
Too many features	Use SelectKBest or Lasso
Unstable coefficients	Use Ridge regression
Want automatic feature removal	Use Lasso
Want to keep all features but shrink them	Use Ridge
No idea which features matter	Check correlation with target first

Practice Challenges

Level 1:
Load load_diabetes(). Check correlations between all features. Which two features are most correlated with each other?

Level 2:
On the California housing dataset, calculate VIF for all features. Drop the two with the highest VIF and retrain. Does test R2 go up or down?

Level 3:
Try Lasso with alpha values from 0.001 to 1 on a dataset. Plot how many features get zeroed out as alpha increases. At what alpha does the model start losing meaningful accuracy?

References

Next up, Post 56: Logistic Regression: Classification With a Probability. We switch from predicting numbers to predicting categories, and the sigmoid function is the reason it works.

54. Linear Regression: Predicting Numbers From Patterns

Akhilesh — Tue, 05 May 2026 09:12:41 +0000

You want to predict something. A number. How much a house will sell for. How many units you'll sell next month. What temperature it'll be tomorrow.

That's a regression problem. And linear regression is the first tool you reach for.

It's the simplest ML model that actually does something useful. Every more complex model builds on the ideas here. You can't skip this one.

What You'll Learn Here

What linear regression actually does
The equation y = mx + b and what each part means in ML
What a cost function is and why we need one
How least squares fitting works
Building linear regression from scratch and with scikit-learn
How to evaluate regression models (not accuracy, different metrics)
Multiple features and what changes

The Simplest Idea in ML

You have two things that seem related. Hours studied and exam score. House size and house price. Temperature and ice cream sales.

Plot them on a graph. You get a scatter of dots. Linear regression draws the best possible straight line through those dots.

Once you have that line, you can plug in a new value on the X axis and read off a prediction on the Y axis.

That's it. That's linear regression.

The Equation: y = mx + b

You've seen this since school. In ML we write it slightly differently but it's the same thing.

y = mx + b

y  = the thing you're predicting (output)
x  = the input feature you know
m  = slope (how much y changes when x increases by 1)
b  = intercept (what y is when x is 0)

In ML notation:

y_hat = w * x + b

y_hat = predicted value
w     = weight (same as slope m)
x     = input feature
b     = bias (same as intercept)

The model's job is to find the right values of w and b that make the line fit the data as well as possible.

What "Best Fit" Means: The Cost Function

For any line you draw, some predictions will be too high and some too low. The difference between your prediction and the real answer is called the residual or error.

error = actual - predicted

You want to minimize the total error across all your training examples. But you can't just add up raw errors because positive and negative errors cancel out.

So instead, you square each error and add them all up. This is called the Mean Squared Error (MSE).

MSE = (1/n) * sum((actual - predicted)^2)

Squaring does two things: makes all errors positive, and punishes big errors more than small ones. A prediction that's off by 10 is penalized 4x more than one off by 5.

The line that gives you the lowest MSE is your best fit line. That's what scikit-learn finds when you call .fit().

Building It From Scratch First

Before using scikit-learn, let's implement linear regression manually so you see what's actually happening.

import numpy as np
import matplotlib.pyplot as plt

# Simple dataset: study hours vs exam score
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype=float)
scores = np.array([52, 55, 60, 65, 68, 72, 75, 81, 85, 90], dtype=float)

# Calculate slope (w) and intercept (b) using the least squares formula
n = len(hours)
mean_x = np.mean(hours)
mean_y = np.mean(scores)

# Slope formula: w = sum((x - mean_x) * (y - mean_y)) / sum((x - mean_x)^2)
numerator   = np.sum((hours - mean_x) * (scores - mean_y))
denominator = np.sum((hours - mean_x) ** 2)
w = numerator / denominator

# Intercept formula: b = mean_y - w * mean_x
b = mean_y - w * mean_x

print(f"Slope (w):     {w:.2f}")
print(f"Intercept (b): {b:.2f}")
print(f"Equation: score = {w:.2f} * hours + {b:.2f}")

# Make predictions
predictions = w * hours + b

# Plot
plt.figure(figsize=(8, 5))
plt.scatter(hours, scores, color='blue', label='Actual scores', zorder=5)
plt.plot(hours, predictions, color='red', linewidth=2, label=f'y = {w:.1f}x + {b:.1f}')
plt.xlabel('Hours Studied')
plt.ylabel('Exam Score')
plt.title('Linear Regression From Scratch')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('linear_regression_scratch.png', dpi=100)
plt.show()

Output:

Slope (w):     4.24
Intercept (b): 47.27
Equation: score = 4.24 * hours + 47.27

This tells you: for every extra hour of study, the score goes up by about 4.24 points. If someone studies 0 hours, the model predicts 47.27.

Now predict for a new student who studied 7.5 hours:

new_hours = 7.5
predicted_score = w * new_hours + b
print(f"Predicted score for {new_hours} hours: {predicted_score:.1f}")
# Output: Predicted score for 7.5 hours: 79.1

Now With Scikit-learn

You'll always use scikit-learn in practice. Let's do the same thing but properly.

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np

# Same data, shaped for sklearn (needs 2D input)
hours  = np.array([1,2,3,4,5,6,7,8,9,10]).reshape(-1, 1)
scores = np.array([52,55,60,65,68,72,75,81,85,90])

# Split
X_train, X_test, y_train, y_test = train_test_split(
    hours, scores, test_size=0.2, random_state=42
)

# Train
model = LinearRegression()
model.fit(X_train, y_train)

print(f"Slope (w):     {model.coef_[0]:.2f}")
print(f"Intercept (b): {model.intercept_:.2f}")

# Predict
y_pred = model.predict(X_test)

# Evaluate
mse  = mean_squared_error(y_test, y_pred)
mae  = mean_absolute_error(y_test, y_pred)
r2   = r2_score(y_test, y_pred)

print(f"\nMSE:  {mse:.2f}")
print(f"RMSE: {np.sqrt(mse):.2f}")
print(f"MAE:  {mae:.2f}")
print(f"R2:   {r2:.3f}")

Evaluating Regression: Different Metrics Than Classification

For classification you use accuracy. For regression, accuracy doesn't make sense. You use these instead.

MAE (Mean Absolute Error)
Average of absolute differences between predictions and actual values. Easy to understand. If MAE = 5.2, your predictions are off by 5.2 on average.

MAE = mean(|actual - predicted|)

MSE (Mean Squared Error)
Squares the errors before averaging. Punishes big mistakes more. Harder to interpret because the units are squared.

MSE = mean((actual - predicted)^2)

RMSE (Root Mean Squared Error)
Square root of MSE. Same units as your target variable. Most commonly used.

RMSE = sqrt(MSE)

R2 Score (R-squared)
Tells you how much of the variance in your target variable is explained by your model. Ranges from 0 to 1. Closer to 1 is better.

R2 = 1.0: perfect predictions
R2 = 0.8: model explains 80% of the variation in the data
R2 = 0.0: model is no better than just predicting the mean every time
R2 < 0: model is worse than predicting the mean (something is very wrong)

# Quick comparison of all metrics
print("Metric comparison:")
print(f"  MAE:  {mae:.2f}   <- average error in original units")
print(f"  RMSE: {np.sqrt(mse):.2f}  <- also in original units, penalizes big errors")
print(f"  R2:   {r2:.3f}  <- 0 to 1, higher is better")

Real Example: California Housing Dataset

Let's use a real dataset with actual features.

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
import pandas as pd

# Load data
housing = fetch_california_housing()
X = pd.DataFrame(housing.data, columns=housing.feature_names)
y = housing.target  # median house value in $100,000s

print(X.head())
print(f"\nTarget range: ${y.min()*100:.0f}k to ${y.max()*100:.0f}k")

# Split and scale
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled  = scaler.transform(X_test)

# Train
model = LinearRegression()
model.fit(X_train_scaled, y_train)

# Evaluate
y_pred = model.predict(X_test_scaled)
rmse   = np.sqrt(mean_squared_error(y_test, y_pred))
r2     = r2_score(y_test, y_pred)

print(f"RMSE: ${rmse*100:.0f}k")
print(f"R2:   {r2:.3f}")

Output:

RMSE: $74k
R2:   0.576

The model is off by about $74k on average. R2 of 0.576 means it explains about 58% of the variation in house prices. Not bad for a simple linear model with no tuning.

Looking at Feature Weights

One great thing about linear regression: you can see exactly how much each feature matters.

# Feature importance from coefficients
coefficients = pd.Series(model.coef_, index=housing.feature_names)
coefficients_sorted = coefficients.sort_values()

print("Feature weights (bigger absolute value = more influence):")
print(coefficients_sorted)

Output:

Feature weights:
AveOccup    -0.391
Latitude    -0.900
Longitude   -0.870
HouseAge     0.123
AveRooms     0.323
Population  -0.003
AveBedrms   -0.049
MedInc       0.827

MedInc (median income) has the biggest positive weight. Higher income neighborhoods have higher house prices. Makes sense.

Latitude has a big negative weight. Moving north in California (higher latitude) generally means lower prices. Also makes sense geographically.

This interpretability is one of the biggest advantages of linear regression. You can explain every prediction.

The Things Everyone Gets Wrong

Mistake 1: Not scaling features

Linear regression works with raw numbers. If one feature ranges from 0 to 1,000,000 and another from 0 to 1, the big-range feature dominates the coefficients. Always scale when features have very different ranges.

# Always do this before linear regression
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled  = scaler.transform(X_test)

Mistake 2: Using it for non-linear relationships

Linear regression assumes the relationship is, well, linear. If the real pattern curves, a straight line won't fit it. Check a scatter plot first.

Mistake 3: Reporting R2 without checking residuals

R2 can look decent even when your model is completely wrong in systematic ways. Always plot actual vs predicted and residuals vs predicted.

import matplotlib.pyplot as plt

# Residual plot
residuals = y_test - y_pred

plt.figure(figsize=(8, 4))
plt.scatter(y_pred, residuals, alpha=0.3, color='blue')
plt.axhline(y=0, color='red', linewidth=1)
plt.xlabel('Predicted Values')
plt.ylabel('Residuals')
plt.title('Residual Plot - should look like random scatter around 0')
plt.savefig('residual_plot.png', dpi=100)
plt.show()

If residuals show a pattern (a curve, a funnel shape), your linear model is missing something.

Quick Cheat Sheet

Thing	Code
Train model	`model = LinearRegression(); model.fit(X_train, y_train)`
Get slope	`model.coef_`
Get intercept	`model.intercept_`
Predict	`model.predict(X_test)`
MAE	`mean_absolute_error(y_test, y_pred)`
RMSE	`np.sqrt(mean_squared_error(y_test, y_pred))`
R2	`r2_score(y_test, y_pred)`
Scale features	`StandardScaler().fit_transform(X_train)`

Practice Challenges

Level 1:
Load load_diabetes() from sklearn. Train a linear regression model. Print the RMSE and R2. Which feature has the highest positive weight?

Level 2:
On the California housing dataset, remove the scaling step. Compare R2 with and without scaling. What changes? What does that tell you?

Level 3:
Plot actual vs predicted values as a scatter plot for the California housing data. Draw a diagonal line where predicted = actual. Points closer to that line are better predictions. Where does the model struggle the most?

References

Next up, Post 55: Multiple Regression: More Features, More Power. What changes when you go from 1 input to 10 inputs, how multicollinearity breaks your model, and how to pick the right features.

I trained my own LLM and published it on HuggingFace

Akhilesh — Tue, 05 May 2026 09:11:26 +0000

This is the post where things got real. Training an actual language model, watching the loss go down, pushing it to HuggingFace with my name on it.

The plan

I couldn't afford to train from scratch — that takes thousands of GPU hours and costs thousands of dollars. Instead I used fine-tuning: take an existing pre-trained model and train it further on my medical data.

The model I chose: facebook/opt-1.3b — 1.3 billion parameters, open source, no access restrictions.

The technique: LoRA (Low-Rank Adaptation) — instead of updating all 1.3 billion parameters, LoRA adds small trainable layers on top and only trains those. You go from training 1.3 billion parameters to training about 4 million. Same result, 100x cheaper.

Why Google Colab

My laptop has no GPU. Training even a small LLM on CPU takes days. Google Colab gives you a free Tesla T4 GPU with 15GB of memory. You get 30 hours per week for free. This is what I used.

The training code

The key parts:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, SFTConfig

# Load base model
model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b")

# Add LoRA adapters
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)

# Train
trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    args=SFTConfig(num_train_epochs=3, learning_rate=2e-4)
)
trainer.train()

The results

Training took 1.5 hours on the free T4 GPU. Here's what the loss looked like:

Step 100:  Loss 1.163
Step 500:  Loss 0.994
Step 1000: Loss 0.967
Step 1700: Loss 0.944  ← training complete

Loss going down means the model is learning. Both training and validation loss decreased together, which means the model generalized rather than just memorizing.

Publishing to HuggingFace

model.push_to_hub("Yakhilesh/medmind-opt-medical")
tokenizer.push_to_hub("Yakhilesh/medmind-opt-medical")

That's it. My model is now publicly available at:
huggingface.co/Yakhilesh/medmind-opt-medical

Anyone can download and use it. The adapter weights are only 12.6MB — small because LoRA only saves the adapter, not the entire base model.

What I actually learned

Fine-tuning is more about data quality than model architecture. My 1.3B model trained for 1.5 hours learned genuine medical patterns. The loss numbers prove it.

53. Overfitting: When Your Model Is Too Good at Being Wrong

Akhilesh — Tue, 05 May 2026 05:38:35 +0000

Series: How Machines Learn: A Complete Guide from Zero to AI Engineer
Phase 6: Machine Learning (The Core)

You trained your model. It got 99% on training data. You tested it on new data. It got 61%.

That gap is the problem. And it has a name: overfitting.

This is one of the most important concepts in all of machine learning. If you don't understand it, you'll keep building models that look great in your notebook and fail in the real world.

What You'll Learn Here

What overfitting actually means in plain terms
What underfitting is and why both are bad
The bias-variance tradeoff (explained without scary math)
How to detect overfitting with a learning curve
Practical ways to fix it with real code

The Overfitting Story

Imagine you're studying for a history exam. Instead of understanding why events happened, you memorize every single detail from the textbook. Every date, every name, every quote.

On the practice test, you score 100%. It's the exact same questions from the book.

On the real exam, the teacher rephrases the questions slightly. You freeze. You memorized the exact words, not the concepts. You fail.

Your ML model does the same thing when it overfits. It memorizes the training data so perfectly that it picks up on noise, random quirks, and flukes in that specific dataset. When it sees new data, those memorized quirks don't exist, and the model is lost.

Overfitting vs Underfitting

There are two ways a model can go wrong.

Underfitting: The model didn't learn enough. It's too simple. It misses the real pattern even in the training data.

Overfitting: The model learned too much. It's too complex. It memorized the training data including all its noise.

Good fit: The model learned the actual pattern and can apply it to new data.

Think of it like this. You're trying to draw a line through data points on a graph.

Underfit: You draw a flat horizontal line. It's too simple. It misses the trend.
Overfit: You draw a line that zigzags through every single point exactly. It matches training data perfectly but has nothing to do with the real pattern.
Good fit: You draw a smooth curve that captures the actual trend without chasing every outlier.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

# Create some data with a true pattern + noise
np.random.seed(42)
X = np.sort(np.random.rand(30, 1) * 10, axis=0)
y = np.sin(X).ravel() + np.random.normal(0, 0.3, X.shape[0])

# Three models: underfit, good fit, overfit
degrees = [1, 3, 15]
titles  = ['Underfit (degree=1)', 'Good Fit (degree=3)', 'Overfit (degree=15)']

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
X_plot = np.linspace(0, 10, 300).reshape(-1, 1)

for ax, degree, title in zip(axes, degrees, titles):
    model = Pipeline([
        ('poly', PolynomialFeatures(degree=degree)),
        ('linear', LinearRegression())
    ])
    model.fit(X, y)
    y_plot = model.predict(X_plot)

    ax.scatter(X, y, color='gray', alpha=0.6, label='Data')
    ax.plot(X_plot, y_plot, color='blue', linewidth=2)
    ax.set_title(title)
    ax.set_ylim(-2, 2)

plt.tight_layout()
plt.savefig('overfit_comparison.png', dpi=100)
plt.show()

Run this and look at all three plots. The degree=15 model passes through almost every point perfectly. It looks impressive. But ask it to predict on new data and it goes wild.

Detecting Overfitting: The Training vs Test Gap

The easiest way to catch overfitting is to compare training accuracy and test accuracy.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Try different tree depths
print(f"{'Depth':<8} {'Train Acc':<12} {'Test Acc':<12} {'Gap':<8}")
print("-" * 42)

for depth in [1, 2, 3, 5, 10, 20, None]:
    model = DecisionTreeClassifier(max_depth=depth, random_state=42)
    model.fit(X_train, y_train)

    train_acc = accuracy_score(y_train, model.predict(X_train))
    test_acc  = accuracy_score(y_test,  model.predict(X_test))
    gap       = train_acc - test_acc

    depth_label = str(depth) if depth else 'None'
    print(f"{depth_label:<8} {train_acc:.3f}       {test_acc:.3f}       {gap:.3f}")

Output:

Depth    Train Acc    Test Acc     Gap     
------------------------------------------
1        0.904        0.895        0.009
2        0.940        0.930        0.010
3        0.962        0.947        0.015
5        0.979        0.947        0.032
10       0.998        0.930        0.068
20       1.000        0.912        0.088
None     1.000        0.912        0.088

Look at that pattern. As the tree gets deeper:

Training accuracy goes up (model memorizes more)
Test accuracy goes up then starts dropping
The gap keeps growing

When you see a big gap between training and test accuracy, that's overfitting.

The Bias-Variance Tradeoff

This is the theory behind all of it. It sounds complicated but it's actually a simple idea.

Bias is how wrong your model is on average. A high-bias model makes strong assumptions and misses the real pattern. It underfits.

Variance is how much your model changes when you train it on different data. A high-variance model is sensitive to every little detail in the training data. It overfits.

The tradeoff: reducing one usually increases the other.

Simple models: high bias, low variance (underfit)
Complex models: low bias, high variance (overfit)
The sweet spot: enough complexity to learn the pattern, not so much that you memorize noise

Bias-Variance Tradeoff:

Error
 |
 |  \          /
 |   \        /   <- Total Error
 |    \      /
 |     \    /
 |  Variance \  /
 |            \/
 |         Bias
 |________________________
        Model Complexity

Sweet spot is at the bottom of the total error curve.

You can't eliminate both bias and variance at the same time. The goal is to find the right balance for your specific problem.

Learning Curves: Visualizing the Problem

A learning curve shows you how training and test performance change as you add more training data. It's one of the most useful diagnostic tools in ML.

from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np
import matplotlib.pyplot as plt

data = load_breast_cancer()
X, y = data.data, data.target

def plot_learning_curve(model, title):
    train_sizes, train_scores, test_scores = learning_curve(
        model, X, y,
        train_sizes=np.linspace(0.1, 1.0, 10),
        cv=5,
        scoring='accuracy'
    )

    train_mean = train_scores.mean(axis=1)
    test_mean  = test_scores.mean(axis=1)

    plt.figure(figsize=(8, 5))
    plt.plot(train_sizes, train_mean, label='Training accuracy', color='blue')
    plt.plot(train_sizes, test_mean,  label='Test accuracy',     color='orange')
    plt.fill_between(train_sizes,
                     train_scores.mean(axis=1) - train_scores.std(axis=1),
                     train_scores.mean(axis=1) + train_scores.std(axis=1),
                     alpha=0.1, color='blue')
    plt.xlabel('Training set size')
    plt.ylabel('Accuracy')
    plt.title(title)
    plt.legend()
    plt.tight_layout()
    plt.savefig(f'learning_curve_{title[:5]}.png')
    plt.show()

# Overfit model (deep tree)
plot_learning_curve(
    DecisionTreeClassifier(max_depth=None, random_state=42),
    'Overfit Model - Deep Tree'
)

# Better model (shallow tree)
plot_learning_curve(
    DecisionTreeClassifier(max_depth=3, random_state=42),
    'Better Model - Shallow Tree'
)

Reading the learning curve:

Overfit: training accuracy is high, test accuracy is much lower. Big gap between the two lines.
Underfit: both lines are low and close together. Adding more data doesn't help much.
Good fit: the two lines are close and both are high.

How to Fix Overfitting

Once you detect it, here's how to fix it.

Fix 1: Get more training data

The most effective fix when you can get it. More data makes it harder for the model to memorize noise because there's just too much to memorize.

Fix 2: Simplify the model

Use fewer features, reduce tree depth, reduce polynomial degree. A simpler model can't memorize as much.

# Instead of this (overfitting)
model = DecisionTreeClassifier(max_depth=None)

# Try this
model = DecisionTreeClassifier(max_depth=3)

Fix 3: Regularization

Regularization adds a penalty for complexity directly into the model's training. The model learns to prefer simpler solutions.

from sklearn.linear_model import Ridge, Lasso, LogisticRegression

# Ridge regression: penalizes large coefficients (L2 regularization)
ridge = Ridge(alpha=1.0)  # higher alpha = more regularization

# Lasso regression: can shrink some coefficients to zero (L1)
lasso = Lasso(alpha=0.1)

# Logistic regression with regularization (C is inverse of alpha)
lr = LogisticRegression(C=0.1)  # lower C = more regularization

Fix 4: Cross-validation

Instead of relying on one train/test split, use cross-validation to get a more reliable estimate and catch overfitting early.

Fix 5: Pruning (for trees)

Decision trees can be pruned after training to remove branches that don't add much value.

# Cost complexity pruning
model = DecisionTreeClassifier(ccp_alpha=0.01, random_state=42)
model.fit(X_train, y_train)

A Side-by-Side Comparison

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.tree import DecisionTreeClassifier

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

models = {
    'Overfit  (no limit)': DecisionTreeClassifier(max_depth=None, random_state=42),
    'Better   (depth=5)':  DecisionTreeClassifier(max_depth=5,    random_state=42),
    'Underfit (depth=1)':  DecisionTreeClassifier(max_depth=1,    random_state=42),
}

print(f"{'Model':<25} {'Train':<8} {'Test':<8} {'CV Mean':<10}")
print("-" * 55)

for name, model in models.items():
    model.fit(X_train, y_train)
    train_acc = model.score(X_train, y_train)
    test_acc  = model.score(X_test,  y_test)
    cv_acc    = cross_val_score(model, X_train, y_train, cv=5).mean()
    print(f"{name:<25} {train_acc:.3f}    {test_acc:.3f}    {cv_acc:.3f}")

Output:

Model                     Train    Test     CV Mean   
-------------------------------------------------------
Overfit  (no limit)       1.000    0.912    0.924     
Better   (depth=5)        0.984    0.956    0.953     
Underfit (depth=1)        0.904    0.895    0.901

The middle model has the best test and CV scores. That's the one you'd pick.

Quick Cheat Sheet

Sign	What it means	Fix
Train high, test low	Overfitting	Simplify model, regularize, more data
Train low, test low	Underfitting	More complexity, better features
Train high, test high	Good fit	Ship it
Big gap train vs test	Overfitting	Cross-validate, regularize
Both curves low on learning curve	Underfitting	More features or complex model

Practice Challenges

Level 1:
Run the depth comparison table on load_wine(). Find the depth where test accuracy peaks.

Level 2:
Plot learning curves for both an overfit and underfit version of a decision tree on the same dataset. See the difference visually.

Level 3:
Use Ridge regression on a regression dataset (load_diabetes()). Try alpha values of 0.01, 0.1, 1, 10, 100. Plot how train and test error change. Find the sweet spot.

References

Next up, Post 54: Linear Regression: Predicting Numbers From Patterns. We build the most fundamental ML model from scratch, understand the math behind it, and use scikit-learn to make real predictions.