The order to build MLOps in when you have one model, not fifty

Sandeep Varma — Sun, 09 Aug 2026 04:30:54 +0000

A five-person team brought me in last month to help ship their first production model, a churn scorer that flags accounts likely to cancel in the next 30 days. Before I looked at a single line of their training code, I found out they had already signed a contract for a feature store.

One model. Zero other models sharing features. That contract was real money going toward a problem they did not have yet.

This kept happening across projects this year, so I want to write down the order I actually build an MLOps stack in for a small team shipping its first model, and why the sequence most vendor content pushes gets it backwards. Feature stores, dedicated model monitoring platforms, and multi-cluster serving setups solve problems that show up once you have several models in production competing for the same data. A team with one model has a much narrower set of problems: get the model out, know when a new version is worse than the old one, and get back to the old one fast if it is.

What the team had when I started

Training happened in a notebook. Whoever trained the latest version emailed a pickle file to whoever was deploying that week. Deploys were a Slack thread: "which version is live right now?" followed by someone SSHing into a box and restarting a process. There was no record of which data a given model version had actually been trained on. Rolling back meant finding the previous pickle file in someone's downloads folder.

None of this is unusual. It is close to the default state for a team whose first job is proving the model is worth building, not building infrastructure around it.

Step one: experiment tracking and a model registry, before anything else

The first thing I set up was MLflow, self-hosted on a single small VM rather than a managed offering, because at this stage the team did not need multi-user scale, just a shared source of truth.

import mlflow

mlflow.set_tracking_uri("http://mlflow.internal:5000")
mlflow.set_experiment("churn-scorer")

with mlflow.start_run():
    mlflow.log_param("model_type", "lightgbm")
    mlflow.log_param("training_data_snapshot", "2026-07-28")
    mlflow.log_metric("auc", 0.891)
    mlflow.sklearn.log_model(model, "model", registered_model_name="churn-scorer")

Every run now logs the data snapshot it trained on, the hyperparameters, and the resulting metrics. The model registry gives each version a number and a stage, so "which version is live" became a query instead of a Slack search. This step took two days and cost nothing beyond the VM. It should come before a feature store, before a monitoring dashboard, before Kubernetes, because none of the later steps mean anything if you cannot answer "what is actually running and what was it trained on."

Step two: a containerized inference service

Next was wrapping the model in a small FastAPI service with one job, and health checks that check the thing that actually matters.

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

HEALTHCHECK --interval=30s --timeout=5s \
  CMD python -c "import requests; requests.get('http://localhost:8000/healthz', timeout=3).raise_for_status()"

CMD ["uvicorn", "app:api", "--host", "0.0.0.0", "--port", "8000"]

The /healthz endpoint loads the registered model from MLflow and runs one prediction against a fixed test row on startup, not just a check that the process is alive. Early on, this service had passed its liveness probe while quietly serving from a stale in-memory model because a deploy had not actually swapped the artifact. A health check that confirms the model loads and predicts catches that class of bug before it reaches users.

Step three: CI/CD that builds and deploys on merge

With a containerized service, CI/CD is mostly plumbing, and there was no reason to reach for anything beyond GitHub Actions.

name: build-and-deploy
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and push image
        run: |
          docker build -t registry.internal/churn-scorer:${{ github.sha }} .
          docker push registry.internal/churn-scorer:${{ github.sha }}
      - name: Update rollout image
        run: |
          kubectl argo rollouts set image churn-scorer \
            churn-scorer=registry.internal/churn-scorer:${{ github.sha }}

This replaced the SSH-and-restart routine entirely. A merge to main now produces a tagged image and updates the rollout, and there is a record of exactly which commit is running.

We added one more gate before the deploy step: a smoke test job that spins up the built image, hits /healthz and /predict with a handful of fixed inputs, and checks the outputs are within an expected range. It runs in about 40 seconds and catches the boring failures, a missing dependency in the image, a schema change in the input payload, before any of it reaches a real rollout. It is not a substitute for the canary. It is there so the canary is not spending its 10 percent traffic window catching mistakes a unit test should have caught for free.

Step four: canary rollout with automatic rollback

This is the step that actually mattered for confidence, because a model can be technically healthy, serving requests fine, and still be quietly worse than the version it replaced. A crash-only health check will not catch that. What catches it is comparing behavior against the previous version before fully committing to the new one.

I used Argo Rollouts for the progressive delivery piece. Its canary strategy lets you define a sequence of steps, each shifting a percentage of traffic to the new version and pausing before continuing.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: churn-scorer
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 30m }
        - setWeight: 50
        - pause: { duration: 30m }
        - setWeight: 100

Argo Rollouts can also query metrics during those pause windows and drive automated promotion or rollback based on what it sees, rather than requiring a person to eyeball a dashboard and decide. We fed it two signals: the service's own error rate, and a drift score computed by Evidently, an open source library that runs statistical tests, Kolmogorov-Smirnov for numeric features and a chi-squared test for categorical ones, comparing the distribution of incoming requests against the training data snapshot logged in MLflow.

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=training_snapshot, current_data=recent_requests)
drift_score = report.as_dict()["metrics"][0]["result"]["dataset_drift"]

If the new version's error rate climbed past a threshold, or the drift score flagged the incoming traffic as meaningfully different from what the model had trained on, the rollout stopped promoting and rolled back to the previous stable version on its own. No page, no 2am dashboard check. The one time this actually fired in the first month, it caught a training data join that had silently dropped a chunk of rows for one customer segment, producing a model that was worse specifically for that segment. A crash-only health check would have shown a perfectly healthy service the whole time.

Picking the rollback thresholds

The first version of the error rate threshold was too tight. We set it at half a percentage point above the current stable version's baseline, and the canary rolled itself back twice in the first week over noise, a couple of slow requests during a deploy window that coincided with a batch job hitting the same database. Widening it to two percentage points above baseline, measured over a rolling five-minute window rather than an instant snapshot, stopped the false rollbacks without losing the ability to catch a genuinely broken version.

The drift threshold took longer to get right. Evidently's default dataset-level drift flag treats a dataset as drifted once a set share of individual features show statistically significant drift, and the default share is tuned for general use rather than this specific model. We narrowed it to the six features the model actually weights most heavily, based on feature importance from the training run logged in MLflow, and ignored drift on the rest. Watching all thirty input columns equally meant a harmless change in a rarely used field could trigger a rollback for a model that did not depend on that field in any meaningful way.

Where the feature store actually fits

None of this required a feature store, because there was one model and one training pipeline. A feature store earns its cost when a second or third model needs the same underlying features, and the team is maintaining that transformation logic in two or three places at once, with training and serving quietly drifting apart because someone updated one copy of the logic and not the other. That is a real, expensive problem. It just was not this team's problem in month one.

What changed

Manual deploys before this work were a three-day process end to end: train, hand off the pickle file, coordinate a deploy window, SSH in, restart, watch logs for a while and hope. After the four steps above, shipping a new model version to production, canary included, took under 30 minutes of wall clock time, most of which was the rollout's own pause windows rather than anything a person had to do. The team went from their first production model to automated canary rollback in about a week and a half of work.

The bigger change was not the speed. It was that a deploy stopped being an event that needed a person watching it. Before, every deploy meant someone staying near a laptop for an hour after pushing, refreshing a log stream. After, the on-call engineer found out about the one rollback that fired from a Slack notification the rollout posted on its own, after it had already happened and already recovered. Nobody had to be watching for it to work.

That also changed what the team's next hire needed to know. The original plan had been to bring on a dedicated ML platform engineer once the model was live, largely to own exactly this kind of deploy babysitting. With the rollback automated, that hire got pushed back a couple of quarters, and the money went toward a second model instead, one that predicts which accounts are worth a retention call rather than just flagging churn risk.

The order, restated

Experiment tracking and a model registry first, because you need to know what is running before you can safely change it. A containerized service with a health check that actually exercises the model, second. CI/CD that removes the human from the deploy path, third. A canary rollout wired to real signals, so a bad version gets caught and reversed automatically, fourth. A feature store, a dedicated monitoring platform, and a multi-cluster serving layer come later, when there are enough models and enough shared data plumbing to justify the cost. Building them first solves a problem you do not have yet, and leaves the problem you do have, shipping this one model safely, unsolved.

DEV Community: Sandeep Varma