Deploying an AI model to production is the easy part. Knowing which version is running, what changed between versions, and how to roll back when outputs silently degrade — that's where most teams discover they have no plan.
This article covers a practical approach to AI model versioning and rollback, with code you can adapt immediately.
Why AI Model Versioning Differs from Code Versioning
In software, a bad commit breaks tests or throws exceptions. A bad model update is subtler: accuracy drifts, latency spikes, a particular input class starts returning nonsense. You often don't know until users complain or a downstream metric tanks.
This asymmetry means versioning an AI model requires more metadata than a git SHA:
- Model weights and architecture: the serialized checkpoint
- Training data hash: ensures reproducibility
- Hyperparameters: learning rate, batch size, epochs
- Evaluation metrics: accuracy, F1, latency p99 at the time of packaging
- Input/output schema: what the model expects and what it returns
- Dependency pinning: CUDA, PyTorch, transformers versions
Miss any of these and you can't reliably reproduce or compare behavior across versions.
A Practical Versioning Scheme
Use semantic versioning adapted for models: MAJOR.MINOR.PATCH.
- MAJOR: architecture change (new backbone, different tokenizer, different task)
- MINOR: significant retraining (new dataset, fine-tuning on new domain)
- PATCH: weight updates from continued training or prompt adjustments
Pair each version with a manifest file stored alongside the weights:
{
"version": "2.3.1",
"model_type": "text-classifier",
"created_at": "2026-09-13T10:00:00Z",
"training_data_sha256": "a3f8c91d...",
"metrics": {
"accuracy": 0.941,
"f1_macro": 0.927,
"latency_p99_ms": 48
},
"input_schema": {"type": "string", "max_tokens": 512},
"dependencies": {
"transformers": "4.40.0",
"torch": "2.3.0"
},
"promoted_by": "engineer@example.com",
"promoted_at": "2026-09-13T11:30:00Z"
}
This manifest is the source of truth. Store it in the same bucket or registry as the weights — never separately.
Implementing a Model Registry in Python
You don't need a heavyweight MLOps platform to get started. A registry backed by object storage and a PostgreSQL table covers most small and mid-size teams.
import hashlib
import json
import datetime
from pathlib import Path
import boto3
import psycopg2
class ModelRegistry:
def __init__(self, bucket: str, db_conn_string: str):
self.s3 = boto3.client("s3")
self.bucket = bucket
self.conn = psycopg2.connect(db_conn_string)
def push(self, model_path: Path, manifest: dict, version: str) -> str:
"""Upload model weights + manifest, register in DB."""
weights_key = f"models/{manifest['model_type']}/{version}/weights.pt"
manifest_key = f"models/{manifest['model_type']}/{version}/manifest.json"
sha = hashlib.sha256(model_path.read_bytes()).hexdigest()
manifest["weights_sha256"] = sha
manifest["version"] = version
self.s3.upload_file(str(model_path), self.bucket, weights_key)
self.s3.put_object(
Bucket=self.bucket,
Key=manifest_key,
Body=json.dumps(manifest, indent=2),
)
with self.conn.cursor() as cur:
cur.execute(
"""
INSERT INTO model_versions
(model_type, version, weights_key, manifest_key, metrics, created_at, status)
VALUES (%s, %s, %s, %s, %s, %s, 'staged')
""",
(
manifest["model_type"],
version,
weights_key,
manifest_key,
json.dumps(manifest.get("metrics", {})),
datetime.datetime.utcnow(),
),
)
self.conn.commit()
return weights_key
def promote(self, model_type: str, version: str) -> None:
"""Mark a version as production-active."""
with self.conn.cursor() as cur:
cur.execute(
"UPDATE model_versions SET status='retired' WHERE model_type=%s AND status='active'",
(model_type,),
)
cur.execute(
"UPDATE model_versions SET status='active', promoted_at=%s WHERE model_type=%s AND version=%s",
(datetime.datetime.utcnow(), model_type, version),
)
self.conn.commit()
def get_active(self, model_type: str) -> dict:
"""Return manifest for the current production version."""
with self.conn.cursor() as cur:
cur.execute(
"SELECT manifest_key FROM model_versions WHERE model_type=%s AND status='active'",
(model_type,),
)
row = cur.fetchone()
if not row:
raise ValueError(f"No active version found for {model_type}")
response = self.s3.get_object(Bucket=self.bucket, Key=row[0])
return json.loads(response["Body"].read())
The status column (staged → active → retired) gives you a clean audit trail. Rollback is explicit: promote the previous version, the current one becomes retired.
Rollback Patterns
Three patterns, ordered from fast to deliberate:
1. Blue/green with registry pointer
Keep two serving instances — blue (current active) and green (previous active). When monitoring detects degradation, flip the active flag in the registry and redeploy. Time to restore: 30–90 seconds depending on your deployment pipeline.
2. Canary with automatic rollback
Route 5–10% of traffic to the new version. Measure proxy metrics — user correction rate, downstream pipeline reject rate, output confidence distribution — against the baseline. If the error rate diverges beyond a threshold, trigger rollback automatically:
def should_rollback(
baseline_error_rate: float,
canary_error_rate: float,
threshold: float = 0.10
) -> bool:
"""Return True if canary error rate exceeds baseline by more than threshold."""
return (canary_error_rate - baseline_error_rate) > threshold
def monitor_canary(
registry: ModelRegistry,
model_type: str,
canary_version: str,
previous_active: str,
check_interval_s: int = 60,
max_checks: int = 30,
) -> bool:
import time
for i in range(max_checks):
baseline = get_error_rate(model_type, "active")
canary = get_error_rate(model_type, canary_version)
if should_rollback(baseline, canary):
print(
f"[rollback] {canary_version} degraded "
f"(baseline={baseline:.3f}, canary={canary:.3f}). "
f"Restoring {previous_active}."
)
registry.promote(model_type, previous_active)
return False
time.sleep(check_interval_s)
# Canary survived — promote it
registry.promote(model_type, canary_version)
return True
3. Shadow mode before promotion
Run the new model in shadow: it receives all requests, produces outputs, but those outputs are discarded. Compare shadow outputs to active model outputs offline using a statistical test (e.g. Wilcoxon signed-rank test on accuracy). Only promote after a statistically significant comparison over at least 1,000 samples. This slows deployment but eliminates surprise regressions.
For teams building AI pipelines in regulated or security-sensitive environments, a hardened deployment checklist — covering canary thresholds, rollback triggers, observability requirements, and access controls — is as important as the code itself. We publish free security hardening checklists covering AI deployment and adjacent topics.
The Takeaway
Model versioning is not a nice-to-have once you have real users. The key points:
- Every model version needs a manifest with metrics, data hash, and dependency pins — not just a filename or timestamp
- Use semantic versioning adapted to the nature of the change (architecture vs. data vs. weight update)
- Build a simple registry before reaching for heavyweight MLOps platforms
- Implement rollback as a first-class operation, tested in staging before you need it in production
- Canary deployment with automatic rollback beats manual hotfixes when availability matters
The failure mode to avoid: discovering your rollback procedure on a Friday evening when the on-call engineer has never run it before.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)