DEV Community: Mercy Moraa

Building ML Pipelines with Python: From Data to Insights

Mercy Moraa — Sun, 07 Jun 2026 01:32:13 +0000

In machine learning, writing a script that trains a model on a clean dataset is only a fraction of the work. The real challenge lies in building a system that can reliably ingest raw data, transform it, train a model, and serve predictions in production.

When code is written as a series of disconnected Jupyter Notebook cells, it inevitably becomes brittle, difficult to test, and prone to data leakage. The solution is to transition from isolated scripts to structured Machine Learning Pipelines. A pipeline automates the workflow, ensures reproducibility, and bridges the gap between data science and software engineering.

Let us build an end-to-end Machine Learning pipeline using pure Python and the industry-standard scikit-learn framework.

What is an ML Pipeline?

An ML pipeline binds data preprocessing steps and model execution into a single, cohesive software element.

Instead of manually applying transformations to your training data and remembering to apply those exact same transformations to your testing data, the pipeline executes the sequence automatically. This design completely eliminates data leakage—a common error where information from outside the training dataset is accidentally used to train the model, leading to overly optimistic but invalid evaluation metrics.

Step 1: Setting Up the Environment

To follow along, initialize a clean workspace and install the core data science libraries:

pip install numpy pandas scikit-learn

Step 2: Designing the Complete Pipeline Code

We will build a pipeline that handles a realistic, messy dataset containing both numerical features (which need scaling) and categorical features (which need encoding), followed by a classification model.

Create a file named pipeline.py and implement the following structure:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# 1. Simulate a realistic raw dataset
def generate_mock_data():
    np.random.seed(42)
    n_samples = 1000

    data = {
        'age': np.random.choice([np.nan, 25, 34, 45, 52, 61], size=n_samples),
        'income': np.random.choice([50000, 80000, 120000, np.nan], size=n_samples),
        'department': np.random.choice(['Sales', 'Engineering', 'Marketing', None], size=n_samples),
        'purchased': np.random.choice([0, 1], size=n_samples, p=[0.7, 0.3])
    }

    df = pd.DataFrame(data)
    # Replace the string "None" with proper NaN values so the imputer can detect them
    df['department'] = df['department'].replace('None', np.nan)
    return df

def main():
    # Load raw data
    df = generate_mock_data()

    # Separate features (X) and target label (y)
    X = df.drop(columns=['purchased'])
    y = df['purchased']

    # Split into train and test sets before any preprocessing occurs
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )

    # 2. Define feature groups
    numeric_features = ['age', 'income']
    categorical_features = ['department']

    # 3. Create sub-transformers for different data types
    numeric_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='median')),   # Fill missing values with median
        ('scaler', StandardScaler())                     # Scale values to standard normal distribution
    ])

    categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='most_frequent')),  # Fill missing text with mode
        ('encoder', OneHotEncoder(handle_unknown='ignore'))     # Convert text strings to numeric vectors
    ])

    # 4. Combine transformers using ColumnTransformer
    preprocessor = ColumnTransformer(
        transformers=[
            ('num', numeric_transformer, numeric_features),
            ('cat', categorical_transformer, categorical_features)
        ]
    )

    # 5. Build the master pipeline (Preprocessing + Model Estimator)
    clf_pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor),
        ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
    ])

    # 6. Train the entire pipeline with a single call
    # Transformations are fitted strictly on training data
    clf_pipeline.fit(X_train, y_train)

    # 7. Evaluate performance
    # Test data is passed through the pre-fit transformations automatically
    predictions = clf_pipeline.predict(X_test)

    print("--- Model Performance Metrics ---")
    print(classification_report(y_test, predictions))

if __name__ == '__main__':
    main()

Step 3: Unpacking the Architectural Choices

ColumnTransformer

Real-world data is heterogeneous. Your code needs to treat numerical values differently than strings. The ColumnTransformer lets you isolate specific columns and apply dedicated processing sub-pipelines to them in parallel, before stitching them back together into a unified matrix for the machine learning algorithm.

Streamlined Inference

Notice the execution phase: clf_pipeline.fit(X_train, y_train) handles the entire transformation and training sequence. When it is time to make a prediction on new, raw data, you simply call clf_pipeline.predict(X_new). You do not need to repeat the code for scaling or filling empty data blocks; the pipeline remembers the mathematical rules established during the training step.

Step 4: Production Deployment Strategy

Once your pipeline compiles and performs well, it needs to move out of your local development environment.

Model Serialization

To save the entire trained pipeline—including both the data preprocessing weights and the model parameters—use joblib. It is the recommended serialization tool in the scikit-learn ecosystem as it handles large numpy arrays more efficiently than pickle:

import joblib

# Persist the entire trained pipeline object to disk
joblib.dump(clf_pipeline, 'ml_pipeline.pkl')

To load the pipeline later for inference:

# Load the pipeline in a production script or notebook
clf_pipeline = joblib.load('ml_pipeline.pkl')

# Predict on brand new raw data
predictions = clf_pipeline.predict(new_data)

Serving via API

In a production deployment, an API service layer (such as FastAPI) loads this single serialized file into memory on startup. When a user submits raw data via a JSON endpoint, the raw payload is converted directly into a Pandas DataFrame and passed straight to .predict(). This structural cleanliness guarantees that data transformations in production match your training conditions exactly.

Summary Principles for Clean ML Engineering

Split First, Transform Second: Always execute train_test_split before configuring transforms. If you calculate the mean or median of a column using the entire dataset, your model is subtly cheating by seeing data from the test set.
Handle Missing Values Correctly: Ensure that placeholder strings like "None" or "NaN" are converted to proper np.nan values so that SimpleImputer can detect and fill them appropriately. Failing to do this treats them as valid categories and introduces noise into your model.
Handle Unknown Labels: When configuring categorical encoders, always include parameters like handle_unknown='ignore'. This prevents your API from crashing if a user inputs a completely new category in production that wasn't present during training.
Version Your Artifacts: Treat your pipeline binary file like source code. If the data schemas or hyperparameters change, tag the exported file version cleanly so you can easily rollback if production anomalies occur.

Transitioning from raw scripting blocks to structured object-oriented pipelines makes your code reliable, clean, and immediately ready for modern deployment architectures.

Happy coding!

Getting Started with Go: A Beginner's Journey

Mercy Moraa — Sun, 07 Jun 2026 01:14:00 +0000

If you told me a few months ago that I would be obsessed with pointers, memory allocation, and low-level strings manipulation, I probably would have laughed. But here I am, deep into a rigorous technical software development program, and completely falling in love with Go (Golang).

Go was designed by Google to be simple, blazing fast, and highly concurrent. While it handles heavy backend architectures globally, it is also one of the most rewarding languages to learn as a beginner, no matter where you are in the world.

In this article, I want to share my raw insights from an intense tech training environment, unpack Go's fundamental layout, and help you build your very first Command Line Interface (CLI) application using nothing but the Go Standard Library. No cheat frameworks. No heavy external dependencies. Just raw logic.

Why Go?

When you start learning Go in a strict environment, you quickly realize it doesn't hold your hand like Python, but it isn't as terrifyingly manual as C. It sits in the perfect sweet spot:

Strict Compilation: If you import a package or declare a variable and don't use it, your code will not compile. This forces you to write clean, minimal code from day one.
Blazing Fast: It compiles directly to machine code, which means it runs incredibly fast.
Standard Library Power: Go’s built-in tools (net/http, os, strconv) are so powerful that you rarely need to download external packages to build production-grade tools.

Step 1: Setting Up Your Workspace

Before writing code, initialize your project workspace. Open your terminal and run:

mkdir go-beginner-cli
cd go-beginner-cli
go mod init go-beginner-cli

This creates a go.mod file, which manages your application's path and dependencies.

Step 2: Understanding the Base Structure

Every execution file in Go follows a strict anatomy. Create a main.go file and inspect this template:

package main

import (
    "fmt"
)

func main() {
    fmt.Println("Hello, World!")
}

The Breakdown:

package main: Tells the Go compiler that this file should compile as an executable program rather than a shared utility library.
import: Brings in standard tools. Here, "fmt" (Format) handles console input and output.
func main(): The ultimate entry point. When you run your program, execution starts exactly here.

To execute it, run:

go run main.go

Step 3: Building Your First App ; A Dynamic CLI Case Converter

Let’s move past "Hello World" and build a practical CLI application. We will write a tool that accepts a string argument from the terminal and modifies it based on what the user wants (Uppercase or Lowercase).

We will use the native os package to capture terminal input arguments (os.Args).

Replace the contents of main.go with the following code:

package main

import (
    "fmt"
    "os"
    "strings"
)

func main() {
    // os.Args[0] is always the program name itself.
    // We expect: program_name, string_to_modify, and mode (up/low)
    if len(os.Args) < 3 {
        fmt.Println("Usage: go run . [text] [up|low]")
        return
    }

    inputText := os.Args[1]
    mode := os.Args[2]

    switch mode {
    case "up":
        result := strings.ToUpper(inputText)
        fmt.Println("Result:", result)
    case "low":
        result := strings.ToLower(inputText)
        fmt.Println("Result:", result)
    default:
        fmt.Println("Unknown mode! Use 'up' for uppercase or 'low' for lowercase.")
    }
}

How to Run Your New CLI Tool:

Test your application directly from your terminal pane by passing arguments:

Test 1: Uppercase Mod

go run . "learning go is awesome" up
# Output: Result: LEARNING GO IS AWESOME

Test 2: Lowercase Mod

go run . "GLOBAL DEVELOPERS" low
# Output: Result: global developers

Key Takeaways from My Learning Journey

Building text manipulation tools and algorithmic blueprints from scratch taught me three vital lessons:

Think in Runes, Not Just Strings: In Go, strings are read-only slices of bytes. If you want to handle text properly without breaking special characters, learn to convert your data into []rune.
Handle Errors Safely: Go doesn't use traditional try/catch exceptions. Functions return data and errors explicitly side-by-side. It forces you to deal with errors immediately.
Keep it Simple: The beauty of Go lies in its minimalism. If your function is getting too complex, break it down into smaller structural files under package main.

Wrap Up

This is just the baseline of what you can accomplish. Once you master string tracking and the basic os filesystem controls, you can transition into building full file processors, custom parsers, or high-performance APIs.

Are you currently learning Go or thinking about diving in? Let’s connect in the comments below! Share your favorite Go optimization tips or ask any questions if you are stuck on your own learning pipeline.

Happy coding!