This article was originally published at sivaro.in

Cost Efficient MLOps Practices: What Actually Saves Money

Slug: cost-efficient-mlops-practices-what-actually-saves-money

---

Last March, a fintech client walked into our office in Bangalore with a billing statement. Their monthly AWS bill for ML infrastructure had crossed $2.4M. Not $240K. Two-point-four million dollars. They had 14 model endpoints, a "modest" feature store, and what they called a "simple" training pipeline. I looked at their architecture diagram and felt that particular headache you get when the problem is obvious but the fix requires admitting the design was wrong from day one.

That's the thing nobody tells you when you're choosing your MLOps stack. The tooling decision you make in month one echoes for 36 months. And the "obvious" enterprise choice (yes, you know which one) is almost never the cheapest path to production.

So what are cost efficient MLOps practices? At the simplest level: a set of architectural, operational, and financial decisions that get your models from notebook to production while keeping your cloud bill from becoming a line item the CFO stares at in silence. It's not just "use spot instances." It's a stack of choices: how you orchestrate training, how you serve inference, how you version artifacts, how you scale (or don't scale) when traffic drops at 2 AM.

In this piece, I'm going to compare the major options actually in use as of mid-2026, give you the numbers we've seen in production, and tell you where I'd spend money and where I'd cut it. No vendor sales deck energy here. Just what worked, what didn't, and what costs what.

The Architecture Question Nobody Asks Until It's Too Late

Before you pick a platform, you need to answer a question that feels uncomfortable: what is cost efficient architecture for AI systems, given your traffic pattern?

I know that sounds obvious. But in 70% of the engagements I've seen since 2022, teams picked their MLOps stack first and then shaped their architecture around it. Backwards. They needed a serving layer that handled bursty traffic (think: e-commerce model spiking 40x during sale events, then dropping to near-zero overnight). But they'd already committed to a managed platform with a fixed compute floor. Now you're paying for 40x capacity when you need 1x.

The architecture question has four parts:

• Training compute: GPU hours, how often you retrain, whether you need to retrain at all (more on that later)
• Inference serving: GPU vs CPU, batching strategy, latency requirements
• Data pipeline: storage costs, compute for feature engineering, how long you retain raw data
• Orchestration: pipeline runner, experiment tracking, artifact registry

Each of these has a cost curve that's non-linear. Your inference serving bill might be 60% of total spend if you have high query volume. Your training bill might dominate if you're fine-tuning LLMs weekly. The platform you pick should be cheap at your bottleneck, not the most expensive one.

Comparing the Stacks: What We've Actually Deployed

I've built systems on four broad categories of MLOps infrastructure over the past eight years. Here's the honest breakdown.

Open-Source Self-Hosted: Kubeflow + KServe + MLflow

This is the default recommendation from half the YouTube tutorials, and it's the stack I'd pick if you have a competent infra team (3+ engineers who've actually run Kubernetes in prod) and your traffic is predictable.

Where it wins: No per-node licensing. You own the metal (or at least the VMs). For a team running 20-50 GPU nodes on GCP or AWS, the total cost of ownership is 40-60% lower than equivalent managed services. We ran a 40-node A100 cluster on Kubeflow for a logistics client in 2024. Monthly infra cost: roughly $180K. The same workload on SageMaker would have been $310K+.

Where it hurts: The operational tax is real. You're on-call for your own platform. Kubeflow's UX in 2026 is fine, not great. The KServe deployment YAMLs will make you question your life choices the first two times. And when something breaks at 3 AM in the controller, you're the one debugging.

Real cost: $80K-$150K/month in infra + 2 dedicated platform engineers (salary loaded: $300K+/year in India, $500K+ in US/EU).

Here's a basic KServe deployment we use for CPU-inference workloads (these are 90% of our client's models, and people keep assuming everything needs a GPU):

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detection-v3
  namespace: prod-inference
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 12
    autoscaling:
      targetUtilizationPercentage: 70
    model:
      runtime: onnx
      storage:
        s3:
          endpoint: s3.ap-south-1.amazonaws.com
          bucket: sivaro-models
          key: fraud-v3/model.onnx
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "8"
          memory: "32Gi"

That minReplicas: 1 is where the money lives. One pod. Not ten. You scale up under load, scale down to one when traffic drops. Most teams I've seen set minReplicas: 4 out of "safety" and pay for it every single night.

Cloud-Native Managed: SageMaker, Vertex AI, Azure ML

Let me be blunt: I don't recommend these for cost-sensitive deployments anymore. In 2023, I got it. You save engineering time, you get IAM integration, you get a pretty UI. But the pricing model punishes you at scale.

SageMaker's managed endpoints charge you per-instance-hour whether or not there's traffic. Their Serverless inference helps (you pay per request), but the per-request rate for GPU-backed models is 2-3x what you'd pay running your own vLLM on a spot instance with auto-scaling.

Vertex AI is better. GCP's pricing for A100s is more competitive, and the managed model garden reduces your engineering overhead. But you're still locked into their egress pricing, their storage tiers, their specific GPU SKUs.

When I'd pick a managed platform: You have fewer than 5 ML engineers, you need compliance certifications (HIPAA, SOC2) that come "baked in," and your inference volume is genuinely low (under 10K requests/day). In that case, the $5K-$15K/month for managed endpoints is cheaper than hiring a platform engineer.

The Newer Tier: Modal, Baseten, Anyscale

This is where things got interesting in 2024-2026. These platforms sell "serverless GPU" as a product. You write a Python function, deploy, pay per second of GPU usage.

Modal has been the quiet winner for us. Here's why: the cold start on a T4 is now under 2 seconds (they were at 8-10 seconds in early 2024, and that made it unusable for latency-sensitive work). And the pricing for bursty workloads is genuinely 3-5x cheaper than a reserved SageMaker endpoint.

We moved a recommendation model (transformer-based, runs on a single T4) from a 24/7 SageMaker endpoint to Modal in May 2026. Monthly cost dropped from $14,200 to $3,100. The model gets called during business hours in a 3-hour window. Why pay for 720 hours when you need 90?

import modal

app = modal.App("rec-engine-prod")

@app.function(
    gpu="T4",
    timeout=300,
    secrets=["HF_TOKEN"]
)
@app.cls(
    keep_warm=2,  # Keep 2 instances warm, rest scale to 0
    scaledown_window=120
)
class RecModel:
    @modal.enter()
    def load(self):
        import torch
        self.model = torch.load("/models/rec-v7.pt", map_location="cpu")
        self.model.eval()

    @modal.method()
    def predict(self, user_ids: list, item_features: dict):
        with torch.no_grad():
            scores = self.model(user_ids, item_features)
        return scores.topk(50).values.tolist()

That keep_warm=2 is the key. Two instances stay hot. The rest go to zero. You're paying for 2 instances 24/7 plus burst usage, not 20 instances 24/7.

Baseten is comparable for inference specifically. Their "TensorRT-optimized" containers shave 20-30% off latency vs vanilla PyTorch serving, which means you need fewer GPUs for the same throughput. That's a real cost lever.

Anyscale (Ray-based) is where I'd go if your training workloads are the bottleneck. Distributed training across 100+ GPUs with Ray's fault tolerance is genuinely better than spinning up Kubeflow's distributed training operators. But the per-node-hour pricing adds up fast for long training runs.

How to Implement Autoscaling for Cost Efficient ML Serving

This is the single highest-ROI practice I can point you to. Not the platform. Not the model. Autoscaling.

Most ML serving in production today runs at 15-30% utilization. You provisioned for the peak (Black Friday, the Monday morning batch, the marketing campaign that hits 5x normal traffic) and you're paying for that peak 16/7.

The implementation has three layers:

Layer 1: Horizontal pod scaling (HPA) with the right metric. Most teams scale on CPU utilization. Wrong metric for GPU workloads. You want to scale on either (a) GPU utilization via DCGM exporter, or (b) queue depth / requests per second at the inference gateway. Here's the HPA config we use:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-serving-hpa
  namespace: prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama-70b
  minReplicas: 2
  maxReplicas: 16
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_utilization_dcgm
      target:
        type: AverageUtilization
        averageUtilization: 65
  - type: Pods
    pods:
      metric:
        name: requests_per_second
      target:
        type: AverageValue
        averageValue: "50"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 25
        periodSeconds: 60

That stabilizationWindowSeconds: 300 matters. Without it, you oscillate. A 30-second traffic blip spins up 6 replicas, then 30 seconds later spins them back down. You're burning cold-start GPU memory allocation cycles and, more importantly, paying for transient capacity.

Layer 2: Scale to zero (or one) on off-peak. If your model serves B2B customers in US business hours, why are you running 8 GPU pods at 4 AM India time? Cron-based scaling (KEDA cron scaler) or the keep_warm parameter on Modal/Baseten handles this. We cut a client's serving bill by 38% by simply scheduling scale-down from 02:00 to 07:00 IST.

Layer 3: Spot/preemptible instances for the elastic tier. Your base replicas (the minReplicas) run on on-demand. Your elastic replicas (the ones HPA spins up under load) run on spot. You save 60-70% on the elastic portion. The risk is interruption, but if your base tier can absorb a 2-minute disruption (and it should, because you sized it for baseline load), spot interruptions are a non-event.

# K8s: Label nodes for spot vs on-demand, use nodeAffinity
kubectl label nodes spot-pool instance-type=spot on-demand-pool instance-type=ondemand

# In your deployment spec:
# Base pods (minReplicas) → on-demand via nodeAffinity
# Elastic pods → spot via nodeAffinity on the HPA's scale-up

The Hidden Costs Nobody Puts on the Spreadsheet

I'll save you a quarter of a year of surprises.

Model storage and versioning. You'll keep 30+ versions of a model "just in case." Each version is 2-40GB depending on architecture. Multiply by 5 models in production. Multiply by 12 months. Now you're paying $8K-$20K/month in S3/GCS storage for models nobody loads. We set a 90-day TTL on non-production model versions at SIVARO. Saved one client $4,200/month.

Egress. If your inference endpoints pull model weights or features from a different region or different cloud, egress fees will make you angry. I saw a team paying $12K/month in egress because their feature store was in us-east-1 and their inference was in eu-west-1. "We thought it was a few cents per GB." It wasn't.

The "one more engineer" problem. Every MLOps platform has a hidden cost: the person who owns it. Kubeflow needs a platform engineer. SageMaker needs someone who understands their quirkily-named resource types. Modal is lower-touch, but you still need someone who understands GPU memory allocation and why your batch size of 32 is OOMing. Budget one FTE minimum. That's $150K-$250K/year before the infra bill.

Where I'd Actually Spend Money (and Where I Wouldn't)

After 8 years and roughly 40 production deployments, here's my honest allocation:

Spend on:

• Your serving layer. This is 60-70% of ongoing cost. Get it right. vLLM for LLM inference, ONNX Runtime for traditional ML, Triton if you need multi-framework. Don't use a managed platform for your main serving path if your volume justifies self-hosting.
• GPU capacity for training, but spot GPU capacity. Train on spot. Checkpoint every 10 minutes. If you get preempted, resume. AWS and GCP spot interruption rates are low enough (typically 2-4% weekly) that the savings dwarf the retry cost.
• Observability. You can't optimize what you don't measure. GPU utilization, P99 latency, queue depth, cost-per-inference. If you don't have a dashboard that tells you "this endpoint costs $0.003 per request and 70% of your requests are hitting it," you're flying blind.

Don't spend on:

• A "MLOps platform" with 47 features you'll use 6 of. MLflow is fine for experiment tracking. You don't need a $200K/year enterprise platform that also does data cataloging, feature stores, and model monitoring in one bundle. Pick the best tool per job.
• Over-provisioned "safety margins." I've seen teams provision 5x their expected traffic "in case." You don't need 5x. You need 1.5x with autoscaling that can spin up the other 3.5x in 90 seconds.
• GPU for inference when CPU works fine. 70% of the "ML models" I've seen in production are gradient boosted trees or small linear models that run in 3ms on a CPU. They don't need an A100. They don't need a T4. They need a 4-core CPU and 8GB RAM.

FAQ

What's the cheapest MLOps stack for a startup with 2 engineers and 3 models in production?

Modal for inference (pay per second, scale to zero), GitHub Actions or Prefect for training orchestration, and S3 for model storage. Total monthly infra cost for 3 models serving moderate traffic: $500-$2,000. You don't need Kubeflow. You don't need SageMaker. You need a Python function that gets deployed and a cron job that retrains. Keep it boring.

Does autoscaling actually save money, or is it just complexity?

It saves real money if your traffic has variance (which 95% of workloads do). We measured a 40% reduction in serving costs for a media client after implementing HPA with scale-to-one. The complexity is manageable: one HPA manifest, one DCGM exporter deployment, one Grafana dashboard. The "complexity" argument usually means "we're scared of writing one more YAML file." Write the YAML. Save the $40K/year.

Should I use a feature store, or is that overkill?

Depends. If you have 3 models sharing 20 features and your data pipeline is a nightly batch job, a feature store (Feast, Tecton) adds cost without much benefit. You can just query your data warehouse. But if you have 30+ models, real-time features, and online/offline consistency requirements (your training data must match your serving data), a feature store pays for itself. Feast is open-source and runs on your existing infra. Tecton is $50K+/year but handles the consistency problem for you.

What about LLM serving specifically? The costs are different.

They are, and they're the scariest part of your bill. A 70B parameter model on a single A100 does about 25 tokens/sec. You need 8x A100s for a 70B model at reasonable throughput. That's $4,000/day on-demand. vLLM with continuous batching gets you 3-4x throughput over vanilla HuggingFace generate(), which means you can run the same traffic on 2-3x fewer GPUs. Quantization (AWQ, GPTQ) gets you 2x memory efficiency. If you can get away with a 7B model, do that. The 7B model on a single T4 costs $800/month. The 70B model on 8x A100s costs $115,000/month. The cost efficient mlops practices for LLM serving are: quantize aggressively, use vLLM, autoscale hard, and question whether you need 70B for that use case.

How do I measure cost per inference, and does it matter?

It matters more than you think. You should know, for every endpoint, your cost per 1,000 requests. If your fraud model costs $2 per 1,000 requests and your recommendation model costs $45 per 1,000 requests, but the recommendation model generates 100x the traffic, your inference bill is 90% recommendations. You might not realize that until you look. Set it up: log request counts, divide by monthly GPU cost. Five lines of code in your billing script.

Is it worth migrating from a managed platform to self-hosted?

Run the math on your actual numbers. If your SageMaker bill is $8K/month and you have 5 ML engineers, the migration cost (2-3 months of engineering time) pays back in 6-8 months. If your bill is $3K/month and you have 2 engineers, stay on SageMaker. The migration overhead isn't worth it. The threshold for us is roughly $10K/month in managed serving costs, or $5K/month if you're also doing heavy training. Below that, the managed premium is cheaper than your time.

What's the one practice that gives the most cost reduction for the least effort?

Scale to one (or zero) on off-peak. Seriously. Just that. Set a cron that scales your inference replicas down to 1 (or 0) between 11 PM and 7 AM if your traffic allows it. No new tools. No migration. One KEDA cron trigger or one Modal schedule. We've done this as a "quick win" in 12 different client engagements. Average savings: 25-40% on the serving line item. It takes 30 minutes to implement.

The Bottom Line

Cost efficient MLOps practices aren't about picking the "cheapest" tool. They're about matching your infrastructure to your actual traffic pattern, your actual model complexity, and your actual team size. The $2.4M AWS bill I mentioned at the start? The client didn't need a bigger platform. They needed fewer GPUs, autoscaling that actually turned things off at night, and the courage to delete 30 model versions nobody had loaded since 2023.

You don't need a $500K MLOps platform. You need to know what your models cost per inference, when they're actually being used, and whether a 4-core CPU would do the job that an A100 is sitting idle on.

Start there. The savings compound.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.