You deployed a new model version on Friday afternoon. By Monday morning, your error rate tripled and you have no clean way to roll back because the deployment script overwrote the previous weights. This scenario happens more often than teams admit — not because engineers are careless, but because model versioning is treated as an afterthought until it isn't.
This article covers practical versioning patterns and rollback strategies for production AI systems, with working Python code you can adapt today.
Why model versioning is different from code versioning
Git handles source code well. But a production AI model is more than code — it's a combination of model weights (gigabytes, binary files), hyperparameters, training data version, evaluation metrics at training time, and runtime dependencies.
Git LFS can store weights, but querying "what was the F1 score of the model deployed on Oct 3rd?" requires something more structured. The minimum viable versioning scheme stores a version manifest alongside the model artifact.
import json
import hashlib
import datetime
from pathlib import Path
def save_model_with_manifest(model, save_path: str, metadata: dict):
"""Save model artifact with a version manifest."""
path = Path(save_path)
path.mkdir(parents=True, exist_ok=True)
import pickle
weights_path = path / "model.pkl"
with open(weights_path, "wb") as f:
pickle.dump(model, f)
with open(weights_path, "rb") as f:
artifact_hash = hashlib.sha256(f.read()).hexdigest()
manifest = {
"version": metadata.get("version", "0.0.0"),
"created_at": datetime.datetime.utcnow().isoformat() + "Z",
"artifact_hash": artifact_hash,
"metrics": metadata.get("metrics", {}),
"training_data_hash": metadata.get("training_data_hash"),
"dependencies": metadata.get("dependencies", {}),
"description": metadata.get("description", ""),
}
with open(path / "manifest.json", "w") as f:
json.dump(manifest, f, indent=2)
print(f"Saved model v{manifest['version']} to {path}")
return manifest
# Usage
metadata = {
"version": "1.3.0",
"metrics": {"f1": 0.923, "precision": 0.941, "recall": 0.906},
"training_data_hash": "d3f8a2b1c9e4...",
"dependencies": {"scikit-learn": "1.4.2", "numpy": "1.26.4"},
"description": "Retrained with Q3 data, improved recall on edge cases",
}
This manifest becomes your audit trail. Store it in a database or object storage alongside the weights.
A versioned model registry in Python
A local model registry is the next step — an API to list, load, promote, and retire versions. For production teams, tools like MLflow or BentoML solve this at scale. But understanding the underlying pattern helps when integrating with existing infrastructure.
import json
import shutil
from pathlib import Path
from typing import Optional
class ModelRegistry:
def __init__(self, registry_path: str):
self.root = Path(registry_path)
self.root.mkdir(parents=True, exist_ok=True)
self.index_path = self.root / "registry.json"
self._load_index()
def _load_index(self):
if self.index_path.exists():
with open(self.index_path) as f:
self.index = json.load(f)
else:
self.index = {"models": [], "active_version": None}
def _save_index(self):
with open(self.index_path, "w") as f:
json.dump(self.index, f, indent=2)
def register(self, model_path: str, version: str, metrics: dict):
dest = self.root / version
shutil.copytree(model_path, dest, dirs_exist_ok=True)
entry = {
"version": version,
"path": str(dest),
"metrics": metrics,
"status": "candidate", # candidate -> active -> retired
}
self.index["models"].append(entry)
self._save_index()
print(f"Registered v{version} (status: candidate)")
def promote(self, version: str):
"""Promote a version to active, retiring the current one."""
for m in self.index["models"]:
if m["status"] == "active":
m["status"] = "retired"
if m["version"] == version:
m["status"] = "active"
self.index["active_version"] = version
self._save_index()
print(f"Promoted v{version} to active")
def rollback(self) -> Optional[str]:
"""Reactivate the most recent retired version."""
retired = [m for m in self.index["models"] if m["status"] == "retired"]
if not retired:
print("No retired versions to roll back to")
return None
for m in self.index["models"]:
if m["status"] == "active":
m["status"] = "retired"
previous = retired[-1]
previous["status"] = "active"
self.index["active_version"] = previous["version"]
self._save_index()
print(f"Rolled back to v{previous['version']}")
return previous["version"]
def get_active(self) -> Optional[dict]:
for m in self.index["models"]:
if m["status"] == "active":
return m
return None
The status field (candidate → active → retired) mirrors the promotion workflow most teams already have mentally — but rarely encode explicitly.
Rollback strategies in production
Having versioned artifacts is necessary but not sufficient. The rollback strategy depends on your serving architecture.
Direct model replacement is the simplest and highest risk approach. The serving process loads new weights in-place. Rollback means restarting with the previous artifact path. Works for batch inference or low-traffic APIs where a brief restart is acceptable.
Blue-green deployment runs two identical serving environments: one live, one staging. You promote staging to live by updating a load balancer rule. Rollback is a one-line config change. The tradeoff: double infrastructure cost while both environments run.
Canary deployment routes N% of traffic to the new model version while the rest hits the current stable version. You monitor production metrics and either ramp up or abort. This is the right approach for models where the failure mode is silent — a new model that's technically healthy but produces worse predictions.
For monitoring canaries, instrument your serving layer to tag each response with its model version:
from fastapi import FastAPI, Request
import hashlib
import time
app = FastAPI()
CANARY_VERSION = "v1.4.0"
STABLE_VERSION = "v1.3.0"
CANARY_TRAFFIC_PCT = 10 # send 10% to canary
def get_model_version(request_id: str) -> str:
"""Deterministic routing: same request_id always maps to same version."""
bucket = int(hashlib.md5(request_id.encode()).hexdigest(), 16) % 100
return CANARY_VERSION if bucket < CANARY_TRAFFIC_PCT else STABLE_VERSION
@app.post("/predict")
async def predict(request: Request, payload: dict):
request_id = request.headers.get("X-Request-ID", str(time.time()))
version = get_model_version(request_id)
# result = models[version].predict(payload["features"])
return {
"prediction": "...",
"model_version": version, # Always return this for monitoring
"request_id": request_id,
}
Returning model_version in every response lets you filter your metrics dashboards by version and catch regressions before they affect all traffic. Pair this with a security hardening checklist that covers your inference endpoints — TLS, auth headers, and rate limiting apply to model servers the same way they apply to any production API.
When to automate rollback decisions
Manual rollback decisions are slow. Consider automated rollback when you can define a quantitative threshold — for example, "if error rate on the canary exceeds 5% for more than 10 minutes, revert."
This requires three things: a metric you trust (not just HTTP 5xx — include model-specific errors like empty outputs or latency P99 spikes), a stable baseline from the previous version, and an automated action that calls rollback() or updates the load balancer rule.
The common mistake is rolling back on noisy metrics. If your baseline had a 2% error rate and you trigger at 5%, you'll get false positives during traffic spikes. Use a burn-rate approach: track the error budget consumed in a sliding window rather than a point-in-time rate. This is the same principle behind SLO-based alerting — applied to model quality instead of infrastructure.
The takeaway
Versioning a model means more than tagging the weights file. It means capturing the manifest (metrics, data provenance, dependencies), encoding a promotion workflow (candidate → active → retired), and deciding in advance what rollback looks like for your serving architecture.
The code here is intentionally minimal — enough to understand the pattern, not tied to a specific ML platform. Once you outgrow it, MLflow, BentoML, or a cloud-native registry slot in cleanly because the underlying concepts are the same.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)