DEV Community

Sherdil IT Academy
Sherdil IT Academy

Posted on Originally published at academy.sherdil.org

I Deployed a Model to Production. The ML Part Was the Easy 20%.

#machinelearning #cloud #devops #career

Training a model that scores well on a validation set takes an afternoon with the right tutorial. Getting that same model to serve real traffic, reliably, without quietly burning through a cloud budget, took me longer than the training did. That gap is not a machine learning problem. It is a cloud engineering checklist, and here is what is actually on it.

Step 1: containerize the model, not the notebook

A notebook is not a deployable artifact. The first real step is packaging inference code into a container with a pinned, minimal dependency set.

# A serving image should be small and boring, not a full training environment
docker build -t model-serve:v1 .
docker run -p 8080:8080 model-serve:v1
curl -X POST localhost:8080/predict -d '{"input": [...]}'
Enter fullscreen mode Exit fullscreen mode

If your image is several gigabytes because it includes the full training toolchain, split it. Training and serving have almost nothing in common as workloads.

Step 2: push it somewhere a scheduler can find it 

# Tag and push to a registry your cluster or managed service can pull from
docker tag model-serve:v1 gcr.io/your-project/model-serve:v1
docker push gcr.io/your-project/model-serve:v1
Enter fullscreen mode Exit fullscreen mode

Step 3: deploy with resource limits set on purpose, not left default

This is the step that separates a working demo from a production service. GPU and memory limits are not optional extras.

# GKE example: a deployment with explicit resource requests/limits
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-serve
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: model-serve
        image: gcr.io/your-project/model-serve:v1
        resources:
          requests: {cpu: "1", memory: "2Gi"}
          limits: {cpu: "2", memory: "4Gi", nvidia.com/gpu: "1"}
EOF
Enter fullscreen mode Exit fullscreen mode

Leave limits unset and one bad request, or one memory leak, can take down every other workload sharing the node. This is the single most common mistake in a first production deployment.

Step 4: put autoscaling on both ends, not just replicas 

kubectl autoscale deployment model-serve --min=2 --max=10 --cpu-percent=70
Enter fullscreen mode Exit fullscreen mode

Scaling replicas up under load is the easy half. The harder half is scaling back down aggressively when traffic drops, since idle GPU time is the line item that quietly wrecks a budget. Check your scale-down settings, not just the scale-up ones.

Step 5: monitor latency and cost as one dashboard, not two 

# Vertex AI example: check endpoint metrics from the CLI
gcloud ai endpoints describe ENDPOINT_ID --region=us-central1
Enter fullscreen mode Exit fullscreen mode

Watch p99 latency and hourly spend side by side. A model that is fast but expensive and a model that is cheap but slow both fail the same way in production: someone eventually turns them off.

Step 6: have a rollback that takes one command 

kubectl rollout undo deployment/model-serve
Enter fullscreen mode Exit fullscreen mode

If rolling back requires a manual process, it will not happen fast enough when it matters. Test the rollback before you need it, not during an incident.

What this list actually is

None of these six steps required a machine learning skill. They required cluster and infrastructure fundamentals applied to a workload that happens to serve a model instead of a web app. That is exactly the gap that is opening up in the market right now: plenty of people can train a model, and a shorter supply of people can be trusted to run one.

If this checklist felt more familiar than the modeling itself, that is worth paying attention to. The roles built around exactly this skill set are growing faster than the general cloud job market right now, and knowing where they actually are matters more than knowing they exist. I mapped out where the current AI-adjacent cloud demand sits in Pakistan, sector by sector, over here: what the cloud computing job market in Pakistan actually looks like right now.

I put the complete picture together, the specific skills that map to each stage of an ML pipeline, realistic salary data for AI-adjacent cloud roles, and where demand is concentrated, over here: AI and Cloud Computing in Pakistan: Why Machine Learning Needs Cloud Engineers.

Which of these six steps have you actually had to debug at 2am? Drop it in the comments. 👇

Top comments (0)