Scikit-learn Pipeline - Fundamentals.

Introduction:

Datasets usually contain errors, making them messy and unsuitable for analysis or manipulation. This means data preprocessing must be conducted, i.e cleaning, scaling, and encoding of your numerical and categorical columns iteratively for each batch of data you process before you can analyse or make predictions.

Doing this manually is tedious and error-prone. This is where scikit-learn’s Pipeline comes in. This pipeline saves these data-processing steps and a final model into a single object/class, so the entire process, from raw data to prediction, is executed, tuned, and can be reused with multiple datasets using a single command.

Benefits of using a pipeline:

• Prevents data leakage:** preprocessing (like scaling) is learned only from training data, not test data.
• Cleaner code: combines several steps into one object instead of scattered function calls.
• Reusability: the same transformations are automatically applied to new data.
• **Easier tuning: **works directly with tools like GridSearchCV to tune preprocessing and model parameters together.

The Basic Flow of a Scikit Pipeline:

A pipeline is basically a sequence of steps: every step except the last must be a transformer.

Transformers include scalers, imputers, and encoders, denoted with methods like .fit() and .transform().

It is important to note that you should only call .fit_transform on the training data.

The last step is training and fitting a model with .fit() and using the model to make predictions with a .predict() method.

Example 1: Simple scaler and classifier pipeline:

Below is a pipeline that uses a StandardScaler() to scale numeric features, then trains a Logistic Regression classifier model.

Import important libraries:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

Load and split data

bc_data = load_breast_cancer()

Split your dataset into training and test subsets.

X_train, X_test, y_train, y_test = train_test_split(
bc_data.data, bc_data.target, test_size=0.2, random_state=42)

Build the pipeline: scale features, then classify

pipeline = Pipeline(steps=[
("scaler", StandardScaler()),
("classifier", LogisticRegression(max_iter=5000))
])

Fit the entire pipeline (scaling + training) in one call

pipeline.fit(X_train, y_train)

Predict using the same pipeline (scaling is applied automatically)

predictions = pipeline.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))

• Notice that we never manually scaled X_test — the pipeline applied the scaling learned from X_train automatically. This is exactly what prevents data leakage.

Example 2: complex pipeline:

• Numeric and categorical columns are processed, missing values are handled, the model is fitted, and predictions are made in a single pipeline.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

df = pd.DataFrame({
"age": [25, 32, None, 40],
"income": [50000, 60000, 55000, None],
"city": ["Nairobi", "Lagos", "Nairobi", "Accra"],
"purchased": [1, 0, 1, 0]
})

X, y = df.drop("purchased", axis=1), df["purchased"]
numeric_features = ["age", "income"]
categorical_features = ["city"]

• Fill missing numeric values, then scale and fill missing categories, then one-hot encode the categorical columns

numeric_transformer = Pipeline(steps=[
("imputer", SimpleImputer(strategy="mean")),
("scaler", StandardScaler())
])
categorical_transformer = Pipeline(steps=[
("imputer", SimpleImputer(strategy="most_frequent")),
("encoder", OneHotEncoder(handle_unknown="ignore"))
])

• Combine both pipelines using a columnTransformer and fit it to the full pipeline

preprocessor = ColumnTransformer(transformers=[
("num", numeric_transformer, numeric_features),
("cat", categorical_transformer, categorical_features)
])
full_pipeline = Pipeline(steps=[
("preprocessor", preprocessor),
("classifier", RandomForestClassifier(random_state=42))
])
full_pipeline.fit(X, y)

Common Mistakes with Sk-learn pipelines:

• Fitting transformers on the whole dataset before splitting into train/test — leaks information and inflates performance.

• Forgetting 9handle_unknown="ignore") in OneHotEncoder, causing errors when new categories appear in test data.

A scikit-learn Pipeline bundles preprocessing and modeling steps into a single, reusable object. Combined with ColumnTransformer, pipelines handle realistic datasets with mixed numeric and categorical data in a structured way.