Sachin Patel

Posted on Aug 5 • Originally published at techreactlearning.blogspot.com

Machine Learning in the Garage: Understanding Decision Trees and Random Forests

#ai #datascience #machinelearning #python

Originally published at Programming Tech Lab.

Welcome Back to the Garage: From Straight Lines to Flowcharts

In previous guides, we explored Linear Regression (predicting car prices along a straight trendline) and Logistic Regression (predicting binary outcomes like engine failure using a probability S-curve).

However, real-world diagnostic problems are rarely linear. A car's value doesn't drop steadily if it has a rare trim, and an engine won't fail based on high temperature alone unless oil pressure is also critically low.

When decision-making depends on a series of condition checks—if this, then that—we move away from mathematical curves and enter the world of Decision Trees and Random Forests.

What is a Decision Tree? (The Senior Mechanic's Checklist)

Imagine bringing a car with a starting issue to an experienced auto mechanic. The mechanic doesn't run a complex formula in their head; they follow a mental flowchart:

Does the engine turn over when you turn the key?
- No: Check the battery voltage.
  - Battery Voltage < 12V: Replace Battery.
  - Battery Voltage ≥ 12V: Check Starter Motor.
- Yes: Check the fuel delivery system.
  - Fuel Pressure Low: Replace Fuel Pump.
  - Fuel Pressure Normal: Inspect Spark Plugs.

This step-by-step diagnostic process is exactly how a Decision Tree operates.

Core Anatomy of a Decision Tree:

Root Node: The initial starting question (e.g., Engine turns over?).
Internal Nodes (Branches): Follow-up conditional split points based on features (e.g., Battery Voltage < 12V).
Leaf Nodes: The final decision or outcome prediction (e.g., Replace Battery).

How the Tree Decides Where to Split: Impurity & Gini Index

How does an algorithm automatically choose which question to ask first?

When training a Decision Tree, the goal at every step is to separate mixed data into pure categories. The algorithm evaluates candidate feature splits using metrics like Gini Impurity or Entropy.

High Impurity: A node contains an equal mix of healthy engines and failing engines (50/50 split).
Zero Impurity (Pure Leaf): A node contains only failing engines (100% pure outcome).

The Decision Tree tests every available feature and selects the split point that maximizes the reduction in impurity (known as Information Gain).

The Flaw of Single Decision Trees: Overfitting

While Decision Trees are easy to understand, a single tree has a major drawback: it easily overfits the data.

If a single mechanic inspects 1,000 cars, they might memorize tiny peculiarities specific to those exact vehicles—like "If a blue 2012 sedan makes a clicking sound on Tuesdays, replace the alternator."

When a single tree grows too deep, it memorizes noise in the training set instead of learning general patterns. As a result, performance drops significantly on new, unseen data.

Enter the Random Forest: An Ensemble of Experts

To fix the overfitting problem of a single tree, we use a Random Forest.

Instead of relying on one mechanic, imagine getting a joint evaluation from a panel of 100 independent mechanics:

Each mechanic receives a randomly selected sample of past car repair records (Bootstrap Sampling / Bagging).
Each mechanic evaluates a randomly selected subset of available diagnostic features (Feature Randomness).
Each mechanic builds their own individual Decision Tree.
When a new car arrives, all 100 mechanics vote on the diagnosis. The majority vote wins (Ensemble Prediction).

By combining predictions from dozens or hundreds of slightly different trees, individual errors cancel out—yielding far higher accuracy and stability than any single tree could achieve alone.

Quick Implementation (Python / Scikit-Learn)

Here is how you can train both a Decision Tree and a Random Forest classifier in Python:

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Telemetry Data: [Battery Voltage (V), Fuel Pressure (PSI), Oil Level (L)]
X = np.array([
    [10.5, 45, 4.0],
    [12.6, 12, 4.2],
    [12.4, 48, 1.5],
    [10.2, 15, 1.2],
    [12.8, 50, 4.5],
    [12.5, 46, 4.1]
])

# Labels: 0 = Healthy Engine, 1 = Mechanical Issue
y = np.array([1, 1, 1, 1, 0, 0])

# Split into train/test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42
)

# 1. Single Decision Tree
tree_model = DecisionTreeClassifier(max_depth=3, random_state=42)
tree_model.fit(X_train, y_train)
tree_preds = tree_model.predict(X_test)

# 2. Random Forest Ensemble
forest_model = RandomForestClassifier(n_estimators=100, random_state=42)
forest_model.fit(X_train, y_train)
forest_preds = forest_model.predict(X_test)

print(f"Decision Tree Accuracy: {accuracy_score(y_test, tree_preds):.2f}")
print(f"Random Forest Accuracy: {accuracy_score(y_test, forest_preds):.2f}")

Real-World Applications

Automotive & IoT Fault Diagnosis: Real-time sensor monitoring systems evaluate multiple threshold branches simultaneously to flag failing hardware before breakdown occurs.
Credit Risk Assessment: Banks evaluate loan applications using decision ensembles to assess creditworthiness based on income, debt-to-income ratio, and payment history.
Medical Triage & Diagnostics: Emergency rooms use clinical flowchart trees to prioritize patient treatment based on vital sign thresholds.

Frequently Asked Questions (FAQ)

Q1: How do you prevent a single Decision Tree from overfitting?

Answer: You can apply Pruning techniques or restrict hyperparameters like setting a maximum tree depth (max_depth), requiring a minimum number of samples per leaf (min_samples_leaf), or setting a maximum limit on leaf nodes.

Q2: When should I use a Decision Tree instead of a Random Forest?

Answer: Use a Decision Tree when absolute interpretability and model visual presentation are critical (e.g., explaining a decision step-by-step to non-technical stakeholders). Use a Random Forest when higher predictive accuracy and generalizability are required.

Q3: Does feature scaling (normalization) matter for trees?

Answer: No. Decision Trees evaluate features independently at each split point based on ordering rather than magnitude, making them invariant to monotonic feature transformations or feature scaling.

This article was originally published on Programming Tech Lab.

DEV Community