Deploying ML Models in Production: What They Don't Teach You

Every course teaches you how to train a model. Almost none teach you how to keep it alive. This is the part of ML that pays the bills — and the part that breaks.

Eighteen months ago I built a demand-forecasting model for a logistics company in Dubai. In offline testing it hit 92% accuracy. The client was thrilled. We deployed it. Within a week, the forecasts were off by 40%, the operations team stopped trusting it, and I learned more about deployment in seven days than I had in seven years of training models.

The model did not change. The world did. And that is the entire lesson this article exists to teach.

The interesting part is what the client said when I showed him the numbers. He did not ask me to make the model smarter. He asked how we would know, in the future, the moment it stopped working. He had been burned before by software that degraded quietly, and he understood that the forecast was only as good as the alarms around it. That question — "how will we know?" — is the real deployment spec. Everything in this article is my answer to it.

If you are currently in the phase of your ML journey where the hard part is a validation curve, this post is your warning shot: training is the easy 20%. The other 80% — packaging, serving, monitoring, retraining, rollback — is what separates a notebook from a product, and it is exactly what they do not teach you.

First, Let's Reframe What "Deployment" Means

When most people hear "deployment," they picture uploading a file to a server. That is like saying a restaurant is "the building." Deployment is a system, and it has six parts that all have to work:

1. Packaging — the model and its environment in a reproducible artifact.
2. Serving — a fast, safe interface that turns requests into predictions.
3. Feature alignment — making sure what the model saw in training is exactly what it sees at serving time.
4. Monitoring — watching for drift, latency, and silent degradation.
5. Retraining — a mechanism to refresh the model when the world moves.
6. Rollback — the ability to un-ship a bad model in minutes, not days.

Every horror story I have ever seen in ML production is a failure of one of these six. My Dubai disaster was a failure of feature alignment and monitoring, in that order.

The Architecture You're Actually Building Toward

Here is the production stack I build toward now, and the components are boring on purpose:

 training data ──▶ training pipeline ──▶ model registry
                                             │
  features (feature store) ◀──┐              ▼
                              │          serving layer
  live data ──▶ feature compute ──▶ prediction API ──▶ your app
                              │              │
                              └──▶ monitoring ──▶ alerting ──▶ retrain trigger

The model registry is the quiet hero. It stores every model version, its metrics, its training data hash, and who approved it. Without it you cannot answer the two most expensive questions in production: "what changed?" and "how do we undo it?"

Packaging: The Dockerfile Nobody Teaches You

The model is not a .pkl file. It is a .pkl file plus a Python version, a library list, and a set of preprocessing steps — and the whole bundle must be reproducible. This is what containers are for. A production-grade image is small, non-root, and pinned to exact versions:

FROM python:3.11-slim

RUN useradd --create-home modeluser
WORKDIR /app

# Pin exact versions — "latest" is how models rot.
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY --chown=modeluser . .
USER modeluser

EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Two details that matter more than they look: python:3.11-slim keeps the image small enough to cold-start fast, and USER modeluser means a compromise in the app does not give an attacker root on your box. Neither appears in any tutorial, and both have saved me real incidents.

Serving: FastAPI in Under Forty Lines

I serve almost everything through FastAPI, and you should too — it is fast, typed, and gives you automatic request docs. The part people get wrong is not the endpoint; it is loading the model once, outside the handler, so you are not paying disk reads per request:

import numpy as np
from fastapi import FastAPI
import joblib
from pydantic import BaseModel

app = FastAPI()

# Load once at startup, not per request.
model = joblib.load("/models/forecast.joblib")
encoder = joblib.load("/models/encoder.joblib")

class ForecastRequest(BaseModel):
    warehouse_id: str
    day_of_week: int
    historical_volume: float

@app.post("/predict")
def predict(req: ForecastRequest) -> dict:
    # Apply the SAME preprocessing as training — this is where skew lives.
    features = encoder.transform([[req.warehouse_id, req.day_of_week, req.historical_volume]])
    value = float(model.predict(features)[0])
    return {"forecast": value, "model_version": "v7"}

The model_version field in the response is not decoration. When the forecast goes wrong, it is how you know which model is wrong.

The Failure That Cost Me a Week: Train-Serve Skew

My Dubai model failed because the training data had a column the live data did not — a feature that recorded whether a shipment was expedited. In training, 30% of rows had it; in production, the field was empty on almost every request because the client's new system stopped populating it. The model silently defaulted to the median and the forecasts drifted 40% off.

This is called train-serve skew, and it is the number one killer of ML deployments. It happens when the features at serving time differ from training: a missing column, a renamed field, a silently imputed value, a different timezone for timestamps. It never shows up in offline validation, because offline you are using the training data.

The fix is a contract, not a wish: assert on every input that the exact columns and ranges you trained on are present, and fail loudly when they are not:

REQUIRED_FEATURES = {"warehouse_id", "day_of_week", "historical_volume"}

@app.post("/predict")
def predict(req: ForecastRequest) -> dict:
    incoming = req.model_dump().keys()
    missing = REQUIRED_FEATURES - set(incoming)
    if missing:
        raise HTTPException(400, f"missing features: {missing}")
    ...

Fail loudly in dev, and your monitoring never has to guess why accuracy dropped. Fail silently, and you get a week of blaming the model.

Monitoring: Watch the Data, Not Just the Metric

Here is the truth nobody wants: you cannot monitor accuracy in production, because you do not have labels until days later. So you monitor the things you can see — and drift is the signal that pays for itself.

Data drift is when the distribution of input features shifts: average order value doubles, a new warehouse opens, a season you never saw arrives. Concept drift is when the relationship between features and outcome changes: same inputs, different result.

A minimal drift check on a single feature takes ten lines and catches most disasters:

import numpy as np
from scipy.stats import ks_2samp

TRAIN_HISTORY = np.load("/models/train_volume_history.npy")

def check_drift(live_window: np.ndarray, alpha: float = 0.05) -> bool:
    stat, p_value = ks_2samp(TRAIN_HISTORY, live_window)
    return p_value < alpha  # True = distribution likely shifted

# Run every hour, alert to Slack/PagerDuty when True.

The alert is the whole point. A drift alert means "investigate now," not "the model is broken" — it means something in the business changed, which is exactly the signal you want the model to adapt to. What you do about it is the retraining loop.

Beyond drift, log a small set of prediction-level metrics on every request — timestamp, model version, feature hashes, prediction value, latency — to a cheap store. You will not read most of them, but the day you need to explain a production incident you will need all of them. There are exactly three questions your logs must answer: what did the model see, which model was it, and how long did it take. Structured JSON logs beat prose every time, because the incident will be investigated at 2 AM by someone who is not you.

The Latency Fork: Online vs Batch

Before you build the serving layer, decide which of the two realities you are in. Online inference returns a prediction in milliseconds on a live request — it needs a warm model, an API, and autoscaling for spikes. Batch inference runs predictions over stored data on a schedule — nightly forecasts, reports, scoring tables — and it can tolerate a slow job instead of a fast API. Half the "production ML is hard" posts I read are really "I built an online service for a batch problem and paid online prices for it." The cheapest deployment decision is often choosing batch: a nightly forecast that takes ten minutes on a CPU is a completely different cost class from a 99th-percentile sub-100-millisecond API that must survive a traffic spike.

When you do need online, plan the autoscaler explicitly: a GPU-backed service that cold-starts a five-gigabyte model under load will time out your users exactly when they need it most. The practical pattern is a minimum pool of warm replicas, an autoscaler on top, and a fast cheap fallback model that serves degraded predictions while the main one warms up. Degraded-but-present beats absent every time.

The Retraining Loop and the Latency Budget

Models rot on a schedule, and the schedule is not "when accuracy drops." It is "when drift says so, or on a calendar." For my forecasting work, I retrain weekly with a rolling window and always run the new model through a shadow phase: serve it in parallel with the old one, compare outputs on live traffic for a day, and only then promote it.

The other budget nobody budgets for is latency. Inference is not free, and it is not uniform. A boosted forest predicts in under a millisecond; a large transformer on a GPU costs 50–200 milliseconds plus cold-start overhead, and a GPU hour is real money. The rule I use: measure the 99th percentile, not the average, because that is what your users feel, and put a latency budget in the deployment contract before the team celebrates a fast demo.

The Honest Cost Table

Let me give you real numbers so you can plan, drawn from the systems I run:

Cost	What it is	Typical monthly bill
Inference compute	CPU for classical models, GPU for deep learning	$40–$400 classical, $500–$3,000 GPU
Cold-start overhead	Loading a large model on first request	part of the above; kills tail latency
Retraining jobs	Scheduled re-runs on a GPU/CPU	$50–$500
Monitoring + logging	Drift checks, prediction storage, alerts	$30–$200
Human on-call	The 2 AM drift page	the real cost — budget for it

A single GPU serving a large model can cost more per month than a mid-size data scientist's salary slice. That is why the "when NOT to deploy ML" section below is not an afterthought — it is the section that saves the most money.

When NOT to Deploy ML

Not everything that looks like a prediction problem needs a model in production. The three cases I turn clients away from:

1. A deterministic rule works. If "reorder when stock < reorder point and lead time > X" captures 95% of the value, a model adds cost and risk for 5% of value. I have deployed exactly one model where a rules engine would have done — and it was a mistake.
2. Labels arrive too late. If you cannot know whether the model was right until next quarter, you are flying blind. Monitoring is impossible, and blind deployment is gambling with someone else's money.
3. The cost of a wrong prediction is catastrophic. If a wrong output starts a payment or denies a person a loan, you need a human review layer first — and that layer, not the model, is your product. The arithmetic is unforgiving: if one wrong prediction costs $10,000 and the model is wrong 1% of the time at ten thousand predictions a day, that is a million dollars of expected loss per day before you add supervision. The human layer is not a cost to be trimmed; it is the only thing that makes the deployment rational.

The best deployment decision is sometimes no deployment. I tell this to every client, and the good ones pay me anyway, because I just saved them a year of infrastructure they did not need.

The Practitioner's Checklist

Before you call a model deployed, run this list:

• [ ] Model is versioned in a registry with metrics and approval recorded
• [ ] Container is small, pinned, and runs as a non-root user
• [ ] Model loads once at startup, not per request
• [ ] Input contract validated on every request — missing features fail loudly
• [ ] Preprocessing at serving time is byte-identical to training time
• [ ] Latency budget set, 99th percentile measured
• [ ] Drift checks on input features, alerting to a real channel
• [ ] Shadow-serving or canary before full promotion
• [ ] Rollback path tested — you have actually reverted a model in staging
• [ ] Someone owns the 2 AM page, and that someone knows the process

What I Would Do Differently in Dubai

If I could redo that deployment, the changes are obvious in hindsight: a feature contract on the input, a drift check on the expedited-shipment column, and a shadow phase before the old model was switched off. Each is five lines of code and a day of setup. Together they would have converted a week-long firefight into a routine Tuesday.

That is the whole point of production ML: the model is the easy part, and it is not even close. The system around the model — packaging, serving, monitoring, retraining, rollback — is where the value is, where the money is, and where the discipline separates engineers who ship from people who demo. Build the system, and the model can be boring. Boring is the highest compliment a deployed model can earn.

---

*Gulshan Yad