Scikit-Learn, Pipeline Fundamentals: A Titanic Survival Prediction Guide

#machinelearning #datascience #tutorial #python

Machine learning workflows often suffer from code duplication, data leakage, and hyperparameter tuning complexities. Scikit-Learn's Pipeline and ColumnTransformer modules simplify this by combining feature preprocessing and model estimation into an integrated, reproducible object.

This guide demonstrates a complete predictive modeling workflow using the classic Titanic: Machine Learning from Disaster dataset, covering exploratory data analysis, pipeline-based feature engineering, model selection, and cross-validation.

1. Environment Setup & Data Loading

Begin by importing the necessary libraries for data manipulation, visualization, preprocessing, modeling, and evaluation.

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# Model selection & evaluation
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.metrics import roc_auc_score, classification_report

# Preprocessing & Pipelines
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

# Classifiers
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from xgboost import XGBClassifier

# Load dataset
titanic_df = pd.read_csv("data/train.csv")

2. Exploratory Data Analysis & Data Sanitation

A quick inspection of the dataset structure reveals the feature types, missing values, and potential columns to drop.

def basic_checks(df):
    print("Shape:", df.shape)
    print("Missing Values:\n", df.isnull().sum())
    print("Data Types:\n", df.dtypes)

basic_checks(titanic_df)

Missing Value Analysis

Age: Missing 19.87% of data (177 missing values).
Embarked: Missing 0.22% of data (2 missing values).
Cabin: Missing 77.10% of data (687 missing values).

Due to the extreme proportion of missing data, the Cabin column is dropped.

titanic_df = titanic_df.drop(columns=['Cabin'])

3. Train-Test Split & Data Partitioning

Identifiers such as PassengerId, Name, and Ticket carry no predictive power for survival and are excluded from the feature set $X$. We split the data using a 80/20 train-test split stratified by the target column Survived to maintain class proportions.

# Separate features and target
X = titanic_df.drop(columns=['PassengerId', 'Name', 'Ticket', 'Survived'])
y = titanic_df['Survived']

# Categorize feature types
numeric_columns = ['Age', 'SibSp', 'Parch', 'Fare']
categorical_columns = ['Pclass', 'Sex', 'Embarked']

# Stratified split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

4. Building Scikit-Learn Preprocessing Pipelines

To avoid data leakage (such as calculating the mean or standard deviation on the entire dataset before splitting), transformations must be learned only on the training set. Using Pipeline and ColumnTransformer ensures these transformations are safely applied during cross-validation.

# 1. Numerical Pipeline: Impute missing values with median, then scale
numerical_pipeline = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# 2. Categorical Pipeline: Impute missing values with mode, then One-Hot Encode
categorical_pipeline = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

# 3. Combine Preprocessing Steps
preprocessor = ColumnTransformer(transformers=[
    ('num', numerical_pipeline, numeric_columns),
    ('cat', categorical_pipeline, categorical_columns)
])

5. Model Candidate Comparison & Stratified Cross-Validation

Evaluate multiple classification algorithms using 5-Fold Stratified Cross-Validation evaluated on the $F_1$-score metric.

# Define candidate models
models = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'K-Nearest Neighbors': KNeighborsClassifier(n_neighbors=7, metric='euclidean'),
    'Random Forest': RandomForestClassifier(n_estimators=300, max_depth=15, random_state=42),
    'Support Vector Machine': SVC(kernel='rbf', probability=True, random_state=42),
    'Gradient Boosting': GradientBoostingClassifier(n_estimators=300, max_depth=15, random_state=42)
}

# Cross-Validation setup
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = {}

for name, model in models.items():
    # Chain preprocessing and model estimation into a single pipeline
    full_pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor),
        ('model', model)
    ])

    cv_scores = cross_val_score(full_pipeline, X_train, y_train, cv=cv, scoring='f1', n_jobs=-1)
    results[name] = {
        'Mean F1 Score': cv_scores.mean(),
        'Std Dev': cv_scores.std()
    }

# Display results in a table
results_df = pd.DataFrame(results).T.sort_values(by='Mean F1 Score', ascending=False)

Model Performance Metrics

Model Classifier	Mean $F_1$ Score	Standard Deviation
Support Vector Machine (SVM)	0.7447	0.0311
Random Forest	0.7374	0.0200
Logistic Regression	0.7301	0.0217
K-Nearest Neighbors	0.7194	0.0229
Gradient Boosting	0.7002	0.0233

6. Key Takeaways & Best Practices

Preventing Data Leakage: Wrapping feature engineering steps inside a Pipeline guarantees that statistics (e.g., mean/median for imputation, standard deviations for scaling) are calculated strictly on training folds during cross-validation.
Handling Mixed Data Types: ColumnTransformer cleanly routes numerical features to continuous scaling modules and categorical variables to encoding blocks.
Model Verdict: The Support Vector Machine (SVM) achieved the highest mean $F_1$-score ($0.7447$), with Random Forest displaying the highest stability across folds with the lowest variance ($\sigma = 0.0200$).

DEV Community