Amit chakraborty

Posted on Sep 10 Originally published at amitchakraborty.dev

MLOps: models that survive production · 1. Why models fail in production and never in the notebook

#tutorial #aiengineering #mlops #executionstateandtherepl

MLOps: models that survive production · Chapter 1 of 12 · AI Engineering · new chapter every Thursday night

By the end of this chapter: Name the gap between an experiment and a running system.

Why this matters

A machine learning model is the only software artifact where the code can execute perfectly, throw no exceptions, return a valid response, and still be completely wrong.

When you write standard software, a bug usually results in a stack trace or a failed test. When you train a model in a Jupyter notebook, you are working in an environment designed for exploration, not execution. The notebook hides the complexity of state, time, and data distribution. You can achieve 99% accuracy on a static CSV file, but the moment that model is exposed to a production system—where data arrives asynchronously, schemas drift, and memory is stateless—it will fail.

Sometimes it fails loudly, crashing the service because a feature is missing. More often, it fails silently. It returns a prediction based on misaligned data types or leaked state, degrading the user experience while your monitoring dashboards show a 200 OK HTTP status.

If you do not understand the mechanical differences between an experimental notebook and a production runtime, you will deploy systems that require constant manual intervention. You will spend weeks debugging models that "worked on your machine." Naming and understanding this gap is the prerequisite for everything that follows in machine learning operations.

Before you start

To run the code and understand the concepts in this chapter, you must have:

Python 3.10 or higher installed.
scikit-learn==1.3.2, pandas==2.1.4, and numpy==1.26.2 installed in your environment.
An understanding of how to train a basic model using scikit-learn and how to make a prediction.
Familiarity with standard Python execution (running a .py script from the terminal) versus interactive execution (running cells in Jupyter or IPython).

Execution state and the REPL

The most immediate gap between an experiment and a production system is how state is managed.

A notebook is a Read-Eval-Print Loop (REPL). When you execute a cell, any variables, functions, or classes you define are stored in the memory of the running kernel. Because you can execute cells in any order, the state of the kernel rapidly diverges from the top-to-bottom text written on the screen. You might instantiate a dataframe in cell 1, modify it in cell 5, delete cell 5, and then train a model in cell 2. The model trains successfully in your active memory. If you restart the kernel and run the notebook from top to bottom, it will crash.

Production systems do not have a persistent, interactive kernel. They execute linearly. When a model is loaded into a serving API, it must contain entirely self-sufficient instructions for taking an input and producing an output.

This state mismatch creates severe problems during serialization. When you save a model using Python's pickle or joblib, you are not saving the code that generated the model. You are saving the state of the Python object in memory, along with a reference to the class definition and the module it belongs to.

If you define a custom data transformer class in a notebook cell and use it in your model, Python registers that class under the __main__ module. When you serialize the model and attempt to load it in a separate production script, the unpickler looks for that custom class in the serving script's __main__ module. It will not find it. The model will refuse to load with an AttributeError, even if the underlying logic was mathematically sound. Production requires that all dependencies and custom classes be defined in importable, version-controlled modules, completely decoupled from the execution state of an experiment.

The data reality gap

In a notebook, your data is finite, static, and available all at once. You typically load a CSV or query a data warehouse, resulting in a single, well-typed dataframe.

In production, data is infinite, dynamic, and arrives one record at a time. This fundamental difference in how data is shaped and processed is the primary cause of silent model failure.

Consider categorical encoding. A common notebook pattern is to use pandas.get_dummies() to convert text categories into binary columns. When you pass a dataframe with a color column containing "red", "green", and "blue" to get_dummies(), pandas scans the entire column and creates three new columns. You train your model on this matrix.

In production, a single JSON request arrives: {"color": "red"}. If you convert this to a dataframe and pass it to get_dummies(), pandas only sees "red". It creates a matrix with one column. When you pass this matrix to your model, the model expects three columns. It throws a ValueError and crashes. The notebook approach relied on seeing the entire future distribution of data to format the present data.

This applies equally to numerical scaling. If you call StandardScaler.fit_transform() on your entire dataset before splitting it into training and testing sets, you have committed data leakage. The mean and variance used to scale the training data were calculated using information from the test data. The model appears highly accurate. In production, you cannot call fit_transform() on a single incoming request, because the variance of a single number is zero. You must use the exact mean and variance calculated strictly from the training set.

The gap here is that notebooks encourage global transformations on static datasets. Production requires stateful transformations, where the parameters learned during training (the vocabulary of categories, the mean of a column) are saved and applied blindly to new, unseen data.

Environment and dependency drift

A model is not just a matrix of weights; it is a compiled artifact that sits on top of a deep graph of software dependencies.

When you run an experiment, your environment contains specific versions of Python, scikit-learn, numpy, and underlying C or C++ libraries like BLAS or LAPACK. If you train a model on an Apple Silicon Mac using scikit-learn 1.2, and deploy it to a Linux container running scikit-learn 1.3, you are crossing the environment gap.

Machine learning libraries move fast. Internal class structures, default hyperparameters, and optimization routines change between minor versions. If the production environment has a different version of a library than the training environment, one of two things will happen.

The loud failure is a ModuleNotFoundError or a deserialization error, because an internal class the model relies on was renamed or moved.

The silent failure is mathematical drift. If a library updates its underlying matrix multiplication routine, or changes the default handling of missing values in a transformer, your model will successfully load and predict. However, the exact floating-point outputs will differ from your experiment. Over millions of predictions, this environment-induced drift can cost a business significant revenue, and it is nearly impossible to detect without strict dependency locking.

The notebook hides this because the training and the evaluation happen in the exact same process, on the exact same hardware, at the exact same time. Production separates training and inference by time, space, and operating system.

A worked example

This script demonstrates the data reality gap. It simulates an engineer training a model in a notebook using global pandas transformations, and the subsequent failure when that model processes a single production request. It then demonstrates the correct, production-ready approach using a stateful pipeline.

Save this code as gap_demo.py and run it via python gap_demo.py.

import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

def simulate_notebook_failure():
    print("--- Running Notebook Simulation ---")

    # 1. The static dataset available in the notebook
    train_data = pd.DataFrame({
        'user_type': ['new', 'returning', 'guest', 'new'],
        'time_on_page': [12.5, 45.2, 5.0, 15.1],
        'converted': [0, 1, 0, 0]
    })

    # 2. Global transformation (The Notebook Anti-pattern)
    # get_dummies sees all possible values ('new', 'returning', 'guest') 
    # and creates exactly 3 columns for user_type.
    X_train = pd.get_dummies(train_data[['user_type', 'time_on_page']])
    y_train = train_data['converted']

    # 3. Model training
    model = LogisticRegression()
    model.fit(X_train, y_train)
    print(f"Model trained successfully. Expected features: {model.n_features_in_}")
    print(f"Feature names: {list(X_train.columns)}")

    # 4. Production inference
    # A single user request arrives via an API payload
    api_payload = {'user_type': 'returning', 'time_on_page': 50.5}

    # The serving script attempts to apply the same preprocessing
    prod_df = pd.DataFrame([api_payload])
    X_prod = pd.get_dummies(prod_df)

    print(f"\nProduction data prepared. Features present: {X_prod.shape[1]}")
    print(f"Feature names: {list(X_prod.columns)}")

    # 5. The Crash
    try:
        model.predict(X_prod)
    except ValueError as e:
        print(f"\nCRASH! The gap revealed itself:\n{e}")

def simulate_production_success():
    print("\n--- Running Production-Ready Pipeline ---")

    train_data = pd.DataFrame({
        'user_type': ['new', 'returning', 'guest', 'new'],
        'time_on_page': [12.5, 45.2, 5.0, 15.1],
        'converted': [0, 1, 0, 0]
    })

    X_train = train_data[['user_type', 'time_on_page']]
    y_train = train_data['converted']

    # 1. Stateful Preprocessing
    # We define a transformer that will learn the categories during .fit()
    # and apply them blindly during .transform()
    categorical_transformer = OneHotEncoder(handle_unknown='ignore', sparse_output=False)

    preprocessor = ColumnTransformer(
        transformers=[
            ('cat', categorical_transformer, ['user_type'])
        ],
        remainder='passthrough'
    )

    # 2. The Pipeline encapsulates both stateful preprocessing and the model
    pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor),
        ('classifier', LogisticRegression())
    ])

    # 3. Training
    pipeline.fit(X_train, y_train)
    print("Pipeline trained successfully.")

    # 4. Production inference
    api_payload = {'user_type': 'returning', 'time_on_page': 50.5}
    prod_df = pd.DataFrame([api_payload])

    # 5. The Success
    # The pipeline remembers the 3 categories from training, even though 
    # the production payload only contains 1.
    prediction = pipeline.predict(prod_df)
    print(f"Prediction successful: Class {prediction[0]}")

if __name__ == "__main__":
    simulate_notebook_failure()
    simulate_production_success()

Where people get stuck

Symptom: You load a saved model in your API, pass it data, and receive a ValueError: X has N features, but Model is expecting M features.
The fix: You have a data reality gap. Your training script used a global transformation like pd.get_dummies() or dropped columns dynamically based on missing value thresholds. You must rewrite your training code to use stateful transformers (like scikit-learn's OneHotEncoder) and bundle them with the model using a Pipeline.

Symptom: You attempt to load a .pkl or .joblib model file and receive an AttributeError: Can't get attribute 'MyTransformer' on <module '__main__'>.
The fix: You have an execution state gap. You defined a custom class or function in the same notebook where you trained and saved the model. Move the custom class definition into a separate .py file, import it into your notebook to train the model, and import it into your serving script to load the model.

Symptom: The model runs perfectly in production, but the accuracy is vastly lower than the notebook metrics indicated, even on the first day of deployment.
The fix: You likely committed data leakage during training. Check your notebook to see if you applied scaling, imputation, or text vectorization to the entire dataset before calling train_test_split. If the test set influenced the preprocessing parameters, your offline metrics are a lie. Move all preprocessing inside a pipeline that is fitted strictly on the training split.

Your tasks

Audit for hidden state: Open a machine learning notebook you have written in the past. Select "Restart Kernel and Run All Cells". If it fails, identify which cell relied on hidden state (variables modified out of order or deleted cells). Fix the notebook so it executes linearly from top to bottom.
Eradicate global pandas transformations: Find a script or notebook where you used pd.get_dummies() or df.fillna(df.mean()). Rewrite that section to use sklearn.preprocessing.OneHotEncoder and sklearn.impute.SimpleImputer. Ensure you call .fit_transform() on the training data and only .transform() on the test data.
Simulate the environment gap: Train a simple scikit-learn model in a virtual environment running scikit-learn==1.2.2 and save it as a pickle file. Create a new virtual environment, install scikit-learn==1.3.2, and write a script to load the pickle file and make a prediction. Observe the warnings or errors generated by the version mismatch.

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.