This article was originally published at sivaro.in

How to Implement Autoscaling for Cost Efficient ML Serving

Slug: how-to-implement-autoscaling-for-cost-efficient-ml-serving

I watched a client burn $71,400 in a single month on GPU inference. Their actual compute need was about $19,000. The gap wasn't fraud, bad pricing, or a vendor ripoff. It was a min_replicas=4 line someone copy-pasted from a tutorial in March 2024 and nobody ever revisited.

That's the whole game with how to implement autoscaling for cost efficient ML serving. It's rarely about picking the fanciest tool. It's about matching capacity to demand, second by second, without leaving money on the floor during idle hours.

If you're running production inference today — an LLM endpoint, a recommendation model, a vision pipeline — this guide is for you. I'm going to walk through the actual options I've deployed, what they cost, where they break, and how to pick one without a six-month migration. Treat it like a buying guide, because that's what it is. You're buying uptime and buying back your margin at the same time.

What cost efficient architecture for AI systems actually looks like

Most people think cost efficiency means cheaper GPUs. Wrong. Cheaper GPUs with bad scaling is a money fire.

A cost efficient architecture for AI systems has four properties. Predictable baseline capacity, elastic burst capacity, honest metrics for scaling decisions, and a hard ceiling so a bug can't run you dry. That's it. Boring on purpose.

The math that matters is utilization across time, not peak utilization. An H100 at 30% average utilization with autoscaling beats an H100 at 80% peak with static provisioning — because you only pay for the 80% window, not the 24 hours around it.

Here's the framing I use with every client. Ask three questions:

• What's my p50 and p95 request rate over a week?
• How long does a cold start take for my model?
• What's my tolerance for a 2-second p99 spike during burst?

If you can't answer those, no autoscaler will save you. Fix the measurement first.

The autoscaling options, ranked by how they actually behave in production

Let's be specific. There are five architectures you'll realistically choose from in 2026. I've run all five in production and I have opinions.

Kubernetes HPA with custom metrics (the workhorse)

Horizontal Pod Autoscaler reading Prometheus metrics through the custom metrics API. This is what 70% of production ML teams ship. It works. It's also where most cost leaks hide.

You scale on a signal. The obvious signal is requests per second. The better signal is queue depth or GPU utilization, because RPS doesn't tell you if a replica is saturated — a slow model can choke at 3 RPS.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: embedding-server
  minReplicas: 1
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: gpu_utilization
        target:
          type: AverageValue
          averageValue: "65"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60

Two things I want you to notice. minReplicas: 1, not 4. And stabilizationWindowSeconds: 300 on scale-down — a 5-minute window stops the oscillation that wastes money on cold starts.

Where HPA breaks: cold starts. If your model takes 90 seconds to load weights into VRAM, HPA is reactive by design and your p99 will spike every time traffic ramps. You need predictive scaling or pre-warmed pools for that.

KEDA with queue-driven triggers

KEDA scales on external event sources — Kafka lag, SQS depth, Redis list length. For async inference (batch scoring, embedding jobs, document processing), this is the correct choice. Not HPA.

I moved a document intelligence pipeline from HPA-on-CPU to KEDA-on-Kafka-lag in February 2025. Monthly inference compute dropped from $23,800 to $9,100. Same throughput. The CPU signal was simply wrong for the workload — it lagged reality by 40-60 seconds and over-provisioned to compensate.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: embedding-worker
spec:
  scaleTargetRef:
    name: embedding-worker
  minReplicaCount: 0
  maxReplicaCount: 30
  cooldownPeriod: 240
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka:9092
        consumerGroup: embedding-workers
        topic: inference-requests
        lagThreshold: "50"

minReplicaCount: 0 is the magic. For workloads with natural quiet hours, scale-to-zero is the single biggest cost lever you have. Don't dismiss it because "cold starts are scary." Measure them. For a 3B parameter model with a warm container image, cold start is 12-20 seconds. For 70B, 90-180 seconds. The tradeoff is yours to make with data.

Serverless GPU (Modal, Runpod Serverless, Replicate)

Scale-to-zero, pay-per-second. The pitch is beautiful. The reality is a real latency floor.

I benchmarked Modal against a self-managed Kubernetes cluster running an A10G fleet for a client's image generation API in April 2026. Modal's per-second billing looked 40% cheaper on paper. In practice, at their sustained load (11k requests/day spread unevenly), the cold-start tax and per-invocation overhead ate the savings. Self-managed won by 22% — but only because their traffic justified a persistent floor.

Where serverless GPU wins: spiky, unpredictable, low-volume. Prototypes. Internal tools. Anything under ~5,000 requests/day. Above that, the math usually flips.

Predictive / scheduled scaling

Combine HPA with a time-based baseline. If you know your traffic doubles at 9am IST and peaks at 8pm, pre-scale at 8:45am. This is boring and it works.

Kubernetes supports this with KEDA cron triggers or a CronJob that patches replica counts. Datadog and AWS both offer predictive scaling in their managed autoscalers.

We moved a client's LLM endpoint from pure HPA to HPA + a 9am pre-scale. p99 latency at morning ramp dropped from 4.2s to 800ms. No additional cost, because the pre-scaled pods were already within the budget of their peak capacity.

Ray Serve / KServe with built-in autoscaling

If you're already in the Ray ecosystem, Ray Serve's autoscaler is fine. KServe's autoscaler overlays on Knative and works well enough. The value is integration — metrics, model versioning, traffic splitting come for free. The cost is lock-in to that framework's mental model.

If you're starting greenfield in 2026 and don't have a Ray commitment, I wouldn't choose Ray Serve purely for autoscaling. Choose it because you need Ray's distributed compute. Autoscaling is a side benefit.

Cost efficient MLOps practices that make autoscaling work

Autoscaling is only as good as the MLOps hygiene around it. Here's what I've learned the hard way.

Version your scaling config in the same repo as your model. I've seen teams debugging latency issues for a week only to find someone had tweaked targetUtilization in a live cluster three weeks prior. If it's not in Git, it doesn't exist.

Log the scaling decisions, not just the outcome. Every scale-up and scale-down should emit an event with the triggering metric value. Six months later, when your CFO asks why March cost $X, you'll have the answer.

Right-size your resource requests. Kubernetes autoscaling is based on requests, not actual usage. If you request 4 GPUs per pod but use 1.2, you're capped at 30% efficiency regardless of what the autoscaler does. I've seen this alone account for 40% waste in VC-backed startups.

Use spot or preemptible instances for burst capacity. Not for baseline. For burst. A mixed node pool — on-demand floor plus spot above it — cuts cost 50-60% on the elastic portion. The catch: spot interruption. Handle it with graceful pod eviction and a 30-second drain timeout. AWS, GCP, and Azure all publish current spot interruption rates. Don't guess.

Instrument the cold start. Every cold start is a metric. If it takes 45 seconds, your stabilization window needs to be at least double that, or you'll thrash.

A reference implementation: cost efficient ML serving in practice

Let me show you a real pattern. This is the architecture I deployed for a fintech client in June 2026 — a fraud-detection model serving 400 RPS at peak, 40 RPS at trough, with a 15-second p99 SLA.

Three layers:

1. A baseline pool of 2 replicas on reserved instances. Always on. Handles trough and absorbs cold-start protection.
2. HPA scaling from 2 to 12 on GPU utilization, target 60%.
3. KEDA cron trigger pre-scaling to 6 replicas at 8:45am IST and 5:30pm IST, matching their two traffic peaks.

# Prometheus recording rule feeding the HPA
# File: fraud-detection-metrics.yaml
groups:
  - name: inference_metrics
    interval: 15s
    rules:
      - record: inference:gpu_utilization:avg
        expr: |
          avg by (pod) (
            rate(nvidia_gpu_utilization_gpu[1m])
          )
      - record: inference:queue_depth:p95
        expr: |
          histogram_quantile(0.95,
            sum by (le) (rate(inference_queue_duration_seconds_bucket[2m]))
          )

Then the HPA:

metrics:
  - type: Pods
    pods:
      metric:
        name: inference:gpu_utilization:avg
      target:
        type: AverageValue
        averageValue: "60"
  - type: External
    external:
      metric:
        name: inference:queue_depth:p95
        selector:
          matchLabels:
            queue: fraud-inference
      target:
        type: Value
        value: "10"

Two metrics, and HPA takes the max of the two recommended replica counts. This gives you responsiveness (queue depth) and safety (GPU utilization) at once.

Results after 90 days: monthly compute cost fell from $34,200 to $14,900. p99 latency stayed under 11 seconds. Same model, same traffic, different scaling policy.

What most teams get wrong about scale-down

Everyone obsesses over scale-up. Scale-down is where the money lives.

The default HPA scale-down policy is 100% every 15 seconds. That's insane for ML workloads. You'll delete a warm replica, traffic resumes, you scale back up, cold start kills p99. Repeat forever. It's a money printer in reverse.

Set a stabilization window of at least 3x your cold-start time. For a 90-second model load, that's 270 seconds minimum. I use 300 by default.

Also: don't scale down below a floor you've chosen deliberately. A single always-on replica as a warm anchor is often worth the $400/month when it saves you from a 60-second p99 during the first morning request.

Contrarian take: a lot of teams should have a higher floor than they currently do. If you're serving a customer-facing endpoint with an SLA under 5 seconds, running at minReplicas=1 is asking for trouble. The floor isn't waste — it's insurance.

Choosing your tool: a decision framework

Print this. Tape it to your monitor.

• Under 5k requests/day, unpredictable traffic → Serverless GPU (Modal, Runpod Serverless). Accept the cold-start tax.
• Async batch / queue-driven → KEDA. Always KEDA. Nothing else comes close.
• Synchronous API, 10k-500k requests/day → HPA on GPU utilization + queue depth, plus cron pre-scaling for known peaks.
• Multi-model serving, need traffic splitting → KServe or Ray Serve.
• Mixed workloads, existing K8s investment → HPA + KEDA + node autoscaler (Karpenter on AWS, GKE Autopilot elsewhere).

The mistake I see most: teams pick the tool that sounds coolest. The tool matters less than whether your scaling signal matches your bottleneck. RPS is almost never the right signal. Queue depth, GPU utilization, memory pressure — those correlate with reality.

FAQ

Q: What's the biggest single cost lever in ML serving autoscaling?
Baseline right-sizing. Fix your resource requests before you tune anything else. Comparing actual usage to requested resources takes 30 minutes and often reveals 30-50% waste.

Q: Is scale-to-zero ever safe for production?
Yes, for async workloads with SLA tolerance above 60 seconds. No, for synchronous user-facing endpoints unless cold start is under 10 seconds — and at that point you've probably invested enough in caching to keep one replica warm anyway.

Q: Kubernetes or serverless — which one for a startup?
Serverless until you're spending more than $8k/month on inference. Then self-manage. The crossover is real but it's later than most people think, and premature optimization here is a distraction.

Q: How do I set the HPA target utilization value?
Start at 60% for GPU-bound workloads and 70% for CPU-bound. Watch for oscillation — if you see replicas bouncing every 2 minutes, either raise the target or extend the stabilization window. Don't tune both at once.

Q: Does autoscaling help LLM serving specifically?
Yes, but LLM cold starts are brutal. 70B parameter models take 90-180 seconds to load. Budget for pre-warmed pools or accept that autoscaling helps cost far more than it helps latency at that scale.

Q: What metrics should I alert on for autoscaling?
Three: p99 latency, replica count hitting max, and scale-down events that get immediately reversed (thrashing). The third is the one that predicts cost blowups.

Q: Can I autoscale across multiple model versions?
Yes, and you should during canary deploys. Split traffic with KServe or an Istio virtual service, scale each version independently, shift load based on business metrics, not just latency.

The honest tradeoffs

Autoscaling isn't free. It's operational complexity traded for cost savings. Every scaling rule is a bug waiting to happen at 3am. Every stabilization window you tune is a service that might spike if you got it wrong.

I've had autoscalers fail in production. A misconfigured max_replicas once let a staging load test hammer production for 40 minutes before someone noticed the $4,800 bill forming. The fix wasn't better autoscaling — it was a hard budget cap on the cloud account. Layer your defenses.

But the alternative — static provisioning — is worse. You either over-provision and bleed cash, or under-provision and page your on-call at every traffic spike. Neither is a business. Neither is a career.

When I look at cost efficient MLOps practices across the dozens of teams I've worked with, the pattern is consistent. Teams that treat autoscaling as a first-class engineering concern — with tests, monitoring, and a Git history — save 40-65% on inference compute. Teams that bolt it on after the fact get the complexity without the savings.

You know which one you want to be. The question is how to implement autoscaling for cost efficient ML serving without cutting corners on the boring parts. Answer: you don't. Do the boring parts. Save the money.

If you want to talk through your specific workload — LLM, vision, tabular, whatever — the decision usually takes 30 minutes of honest questions. That's cheaper than a month of over-provisioning. Every time.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.