This article was originally published at sivaro.in

How to Estimate Infrastructure Cost for ML Models

Most teams blow their ML budget in month two. Not because GPUs are expensive — because nobody did the math before shipping.

I've watched this play out at SIVARO across dozens of client builds since 2018. The pattern is always the same: a data scientist trains a model on a $2/hr spot instance, it works, everyone's thrilled, then it hits production and the monthly bill looks like a mortgage. You can learn how to estimate infrastructure cost for ML models the hard way, or you can read the next 3,000 words and skip the tuition.

Here's the thing nobody tells you upfront: inference costs, not training, will eat 70-90% of your lifetime ML spend. Training is a one-time (or periodic) spike. Inference runs every second your product is live. That asymmetry is the whole game.

This guide breaks down every cost driver, compares the actual options (cloud vs. on-prem vs. hybrid), and gives you a framework you can run before you write a single check. Let's get specific with numbers.

Why Your First Cost Estimate Is Almost Always Wrong

I built a recommendation system for a mid-size e-commerce client in 2023. Initial estimate: $4K/month. Actual bill month one: $19K. We were off by 5x. The mistake wasn't the GPU hour rate — it was everything we forgot to count.

Most people estimate ML infrastructure by multiplying GPU hours by hourly rate. That's like estimating a car's cost by looking at the price of gas. You're missing 80% of the picture.

The real cost drivers, in rough order of how badly teams underestimate them:

• Data transfer and egress. AWS charges $0.09/GB out. Stream 10TB/month to your inference fleet and that's $900 you never budgeted.
• Idle capacity. A GPU sitting at 15% utilization still bills full rate. Most teams run at 20-35% average utilization and pay for 100%.
• Storage for feature stores and model artifacts. Feature stores grow. They never shrink. A 50TB feature store on gp3 EBS runs about $4,000/month before you touch it.
• Orchestration overhead. Kubeflow, Airflow, SageMaker endpoints, Ray clusters — the control plane isn't free.
• Observability. You need to monitor drift, latency, accuracy. Datadog and New Relic bill per host and per custom metric. ML monitoring vendors (Arize, WhyLabs, Fiddler) bill per prediction or per GB.

At first I thought this was a tooling problem. Better dashboards would fix it. Turns out it was a modeling problem — we'd built a model that needed 40ms latency and 8GB VRAM for a use case that needed 200ms and could've run on a quantized 1GB model.

Get the architecture right and the cost falls out naturally.

The Cost Stack: What You're Actually Paying For

Let me break the ML infrastructure bill into its load-bearing components. You need to estimate each one separately because they scale on completely different axes.

Training Compute

This scales with: model size, dataset size, number of experiments, retraining frequency.

A rough formula I use:

training_cost = (gpu_hourly_rate × gpu_count × hours_per_run × runs_per_month)
                + storage_for_checkpoints
                + data_loading_overhead

Here's a real one from a client — a vision model with 340M parameters, fine-tuned monthly:

# Real training cost estimate for a 340M param vision model
gpu_rate = 2.50          # A100 40GB on-demand, us-east-1, Sept 2026
gpu_count = 8            # 8x A100 for DDP training
hours_per_run = 14       # includes 3 failed runs averaged in
runs_per_month = 4       # weekly retrains

compute = gpu_rate * gpu_count * hours_per_run * runs_per_month
# = 2.50 * 8 * 14 * 4 = $1,120/month

checkpoint_storage = 0.08 * 500  # 500GB of checkpoints on S3 Standard
# = $40/month

data_egress = 0.09 * 200         # pulling 200GB from S3 to instances per run
# = $18/month * 4 = $72/month

total_training = compute + checkpoint_storage + data_egress
print(f"${total_training:.0f}/month")  # $1,232/month

That's the easy part. Most teams get training costs roughly right because it's a bounded job. The pain starts at inference.

Inference Compute

This is where the math gets ugly. Inference cost is a function of:

inference_cost = requests_per_second × avg_latency_seconds × cost_per_gpu_second
                 × (1 / utilization_efficiency)

That last term — utilization efficiency — is the killer. If your GPU sits at 25% utilization, you're paying 4x per actual prediction. Most teams don't even measure this until the bill arrives.

Three deployment patterns, three cost curves:

Real-time endpoints (SageMaker, Vertex AI, Bedrock, or your own K8s). You pay for provisioned capacity whether traffic is there or not. Auto-scaling helps but never fully solves the cold-start tax. This is the most expensive pattern per prediction and the one everyone reaches for first.

Batch inference (Spark, Ray, or scheduled jobs). You pay for compute only during the batch window. 5-10x cheaper per prediction than real-time, but latency is minutes-to-hours.

Serverless GPU (Modal, RunPod Serverless, Replicate, Baseten). Pay per second of actual execution. Great for spiky or low-volume workloads. Terrible if you have steady high traffic — you're paying a 2-4x premium over reserved capacity.

# Comparing three inference patterns for 50M predictions/month
# Model: 7B param LLM decoder, ~120 tokens avg output

predictions = 50_000_000
tokens_per_pred = 120

# Pattern A: Always-on A100 endpoints
# 4x A100 handles ~40 req/s at 250ms p99 → 3.4M req/day, need ~1.5x headroom
a100_monthly = 2.50 * 4 * 730
real_time_cost = a100_monthly
# = $7,300/month

# Pattern B: Serverless per-second (Modal-style, ~$0.0006 per A100-second)
# avg execution 0.9s per request (includes cold start amortized)
serverless_cost = predictions * 0.9 * 0.0006
# = $27,000/month  ← 3.7x MORE expensive at this volume

# Pattern C: Batch on spot A100s
# ~200 predictions per GPU-second, 70% spot discount, 60% job efficiency
batch_gpu_seconds = (predictions / 200) / 0.60
batch_cost = batch_gpu_seconds * (2.50 * 0.30)
# = $31,250  ← wait, this is monthly? no, this is $31K for the WHOLE batch
# but you only run it once a day → it IS the monthly cost
print(f"Real-time: ${real_time_cost:,.0f}")
print(f"Serverless: ${serverless_cost:,.0f}")
print(f"Batch: ${batch_cost:,.0f}")

The number that matters: for 50M predictions/month, an always-on reserved fleet is 3.7x cheaper than serverless. Most startups pick serverless because the onboarding is smoother, then quietly migrate at scale. Don't do it in that order if you know you'll scale.

Data Infrastructure

Feature stores, vector databases, data lakes, ETL pipelines. This is the silent killer.

A Pinecone pod for 10M vectors with 1536 dims runs roughly $700/month if you're efficient. The same workload on self-managed pgvector on a db.r6g.2xlarge (about $500/month reserved) is cheaper but you own the ops. Weaviate, Qdrant, and Milvus each have different cost profiles — Qdrant on a single node is shockingly cheap; Weaviate Cloud gets expensive fast because of their memory model.

For feature stores: Feast on your own infra costs compute and storage. Tecton and Databricks Feature Store bill on top of that. I've seen feature store bills beat model serving bills. In 2025, a fintech client's Tecton bill was $11K/month while their inference spend was $6K. The features were more expensive than the predictions.

Cloud vs. On-Prem vs. Hybrid: The Actual Comparison

OK, decision time. Here's how the three options stack up honestly.

Public Cloud (AWS, GCP, Azure)

Wins on: Time to production, elasticity, breadth of services, no capex.
Loses on: Long-run cost at steady state, egress fees, per-service pricing traps.

At SIVARO we default clients to cloud until their inference bill crosses $40-60K/month. Below that threshold, the operational complexity of on-prem will eat your savings. Above it, you're lighting money on fire.

The exceptions: GPU scarcity. Throughout 2024 and into 2025, H100 availability was the binding constraint, not price. That's eased considerably in 2026, but if you need massive H200/B200 capacity, you may still wait months in the cloud or commit to multi-year reserved contracts to jump the queue.

Cloud-specific gotchas I've paid for:

• NAT Gateway costs. Pulling model weights from S3 inside a VPC through a NAT gateway at $0.045/GB adds up. Use VPC endpoints.
• Cross-AZ traffic. $0.01/GB each way. Replicate your feature store per AZ or eat the tax.
• SageMaker endpoint idle time. A 4-instance ml.g5.12xlarge endpoint costs $7,600/month if it never receives traffic. And it will never gracefully scale to zero.

On-Prem / Colocation

Wins on: Predictable cost at scale, data residency, no egress, no surprise line items.
Loses on: Capex, GPU lead times, staff to run it, utilization swings.

Math time. An H100 server from a Tier-2 OEM (Supermicro, Lambda, or similar) runs about $220-260K in 2026 for an 8x H100 80GB box. Depreciate over 4 years plus 15% for power and cooling in a colo, and you're at roughly:

$240,000 capex / 48 months = $5,000/month depreciation
+ $1,200/month colo space (8U at ~$150/U)
+ $900/month power (6.5kW at $0.14/kWh)
+ ~$500/month ops amortized (part-time SRE)
= ~$7,600/month for 8x H100

Compare to cloud: 8x H100 on-demand at $2.50/hr (a 2026 spot-ish rate) is $14,600/month. Reserved 3-year drops it to ~$9,500/month.

So on-prem saves you roughly 20-40% at steady state — IF you keep utilization above 50%. Below that, cloud wins because you're not eating idle depreciation. The break-even utilization is around 40-45%. Measure yours before you sign a colo lease.

Hybrid

The pattern we actually recommend for scale-ups: cloud for burst and training, on-prem (or reserved cloud) for the baseline inference load.

Route 80% of steady-state traffic to owned or reserved capacity. Burst the top 20% to on-demand. Keeps utilization high on the capex and lets you absorb Black Friday without buying servers you'll use four days a year.

# Hybrid cost model: baseline on-prem + cloud burst
baseline_rps = 200        # steady state
peak_rps = 600            # Black Friday
baseline_cost = 7_600     # the 8x H100 box above

# Cloud burst for the 20% spike, ~10 days/month at peak
burst_hours = 240
burst_gpus = 8
burst_rate = 2.50  # on-demand
burst_cost = burst_hours * burst_gpus * burst_rate

# = $4,800/month for burst
hybrid_total = baseline_cost + burst_cost
# = $12,400/month for a workload that would cost $22K all-cloud peak-reserved
print(f"Hybrid: ${hybrid_total:,}/month")

Numbers vary wildly by workload. The point is the shape: never provision for peak. Provision for baseline, rent for peak.

The Estimation Framework I Actually Use

Forget spreadsheets with 40 rows. Here's the four-step process I run with every client.

Step one: map every cost to a scaling axis. Each line item grows with exactly one of: requests, tokens, GB stored, GB transferred, GPU-hours, or wall-clock time. Put each in the right bucket.

Step two: get one real measurement per axis. Not a benchmark. A real measurement from production-shaped traffic. If you don't have production yet, use 100 real requests and extrapolate. Anything else is fiction.

Step three: model at 1x, 5x, and 20x current volume. Costs don't scale linearly. Egress scales linearly. Feature store memory scales worse than linearly once you cross node boundaries. Storage behaves weirdly. Chart all three.

Step four: add a 30% buffer and set alerts at 60% of budget. Not because you'll get it wrong by 30% — because the whole point of this exercise is to catch the wrongness early.

Here's the reusable estimator I hand clients:

from dataclasses import dataclass

@dataclass
class MLWorkload:
    name: str
    requests_per_day: int
    gpu_seconds_per_request: float
    input_bytes: int
    output_bytes: int
    storage_gb: float

def estimate_monthly(w: MLWorkload, gpu_rate=2.50, s3_rate=0.023, egress_rate=0.09):
    gpu_sec = w.requests_per_day * w.gpu_seconds_per_request * 30
    compute = (gpu_sec / 3600) * gpu_rate

    storage = w.storage_gb * s3_rate
    # Assume 20% of traffic egresses to clients
    egress = (w.requests_per_day * 30 * w.output_bytes * 0.20) / 1e9 * egress_rate
    # Assume inputs come from S3 within region (no egress, only request costs)
    ingress_ops = (w.requests_per_day * 30 / 1000) * 0.0004

    subtotal = compute + storage + egress + ingress_ops
    return {
        "compute": compute,
        "storage": storage,
        "egress": egress,
        "requests": ingress_ops,
        "subtotal": subtotal,
        "with_buffer": subtotal * 1.30,
    }

wl = MLWorkload(
    name="fraud-scoring",
    requests_per_day=8_000_000,
    gpu_seconds_per_request=0.008,
    input_bytes=2_000,
    output_bytes=200,
    storage_gb=400,
)
for k, v in estimate_monthly(wl).items():
    print(f"{k:>10}: ${v:,.2f}")

Run it. Change one variable at a time. Watch which ones move the needle. You'll usually find two or three line items that dominate and everything else is noise.

Vendor Comparison: What You're Actually Buying

Let me be blunt about the major players as of September 2026.

AWS SageMaker. Best when you're already deep in AWS. Worst when you're not, because their pricing pages assume familiarity. Endpoint pricing is per-instance-hour, no scale-to-zero without custom work. Great for enterprises, painful for startups. Inference Recommender helps pick instance types but doesn't help control cost after deployment.

GCP Vertex AI. Strongest ML-specific tooling of the big three. Vertex's batch prediction is genuinely cheaper than AWS Batch for most workloads. Their TPU options (v5e, v6e in 2026) offer better price-per-token than equivalent GPUs for transformer inference, but the ecosystem tax is real — fewer libraries just work.

Azure ML. Dominant in regulated industries because of compliance story. Pricing is roughly competitive. The managed online endpoints have gotten better. Nothing here surprises me anymore, good or bad.

Modal / Baseten / RunPod Serverless. All good for sporadic workloads or burst. Modal's developer experience is the best by a wide margin — you write Python, it deploys. RunPod is the cheapest of the three. Baseten is the most production-hardened for LLM serving in particular. All three charge per second of execution, so work that runs in bursts is fine and continuous work gets expensive fast.

Together AI / Fireworks / Groq. These are for when you're consuming models, not hosting them. Tokens per dollar, not GPU-hours. Groq is absurdly fast for certain models (LPU architecture) but limited model catalog. Together has the widest selection. Fireworks has the best fine-tuning + serving story for custom models.

Databricks. If your data already lives there, this is often the path of least resistance. Their Mosaic AI serving is competitive. Their pricing is complicated enough that I've seen finance teams cry. The Unity Catalog integration is worth real money if you're a regulated enterprise.

On-prem vendors (Lambda, CoreWeave, Voltage Park). CoreWeave and Voltage Park are the modern colocation-plus-GPU names. Lambda Cloud is somewhere between cloud and on-prem — their reserved clusters are priced like colo but operated like cloud. For a $5M+ annual GPU budget, these beat hyperscaler pricing by 40-60%.

The honest take: for under $10K/month, cloud. For $10-50K/month, reserved cloud contracts. For $50K+/month steady state, talk to CoreWeave or buy your own.

The Contrarian Take: Most Teams Should Spend More, Not Less

Everyone's optimizing ML costs down. Most of them are making the wrong call.

If you're spending $30K/month on infrastructure and generating $300K/month in revenue from the product, you have a 10:1 ratio. That's fine. The job isn't to cut that to $20K. The job is to make sure the $30K is deployed where it maximizes product velocity.

I've seen teams agonize over a $2K/month feature store bill and lose nine months of engineering time building the same capability in-house. That's a $400K+ opportunity cost to save $24K/year. Disaster.

The right question isn't "how low can we get the bill?" It's "what's the best ratio of infra cost to product velocity?"

Where the cost-cutting genuinely matters:

• At sub-$5K/month total spend, cut nothing. You're in build mode.
• Between $5K-50K/month, focus on utilization. Get above 50% before you optimize anything else.
• Above $50K/month, the architecture decisions matter. Reserved vs. on-demand, model size, batch vs. real-time. A 30% improvement is achievable with focused effort.
• Above $200K/month, hire someone whose full-time job is this. The savings fund the role 5x over.

Apply discipline where it pays. Ignore it where it doesn't.

FAQ: Real Questions I Get Asked

How accurate can I expect my estimate to be before I have production traffic?
Within 2-3x if you model carefully, and that's fine. The purpose of a pre-production estimate isn't precision — it's identifying the dominant cost driver so you can architect around it. If your estimate says "storage is 60% of cost," you know to build a retention policy before you build anything else. Precision comes after launch.

What's the single biggest line item I'm probably forgetting?
Observability on inference. Datadog at scale for a 100-pod inference fleet easily hits $8-12K/month on its own. Budget it before the vendor does.

Should I use spot instances for inference?
Only with sophisticated checkpointing and request rerouting. Spot preemption on a real-time endpoint is a customer-visible outage. For batch inference, always spot — it's basically free money.

How do TPUs compare to GPUs on cost for a given workload?
For transformer inference at scale, TPU v5e/v6e typically runs 20-40% cheaper per token than equivalent H100 capacity, if your model fits the format. For training, the picture is murkier and setup cost is significant. Try it for one workload; don't bet the company.

Is serverless inference ever the right choice at scale?
Yes, if your traffic has a 10:1 peak-to-trough ratio or worse. The 3-4x per-prediction premium buys you not provisioning for peak. If spikes are predictably scheduled (business hours, weekly digests), reserved is better.

How do I budget for retraining?
Assume monthly for anything with drift (recommenders, fraud, ads) and quarterly for stable systems (image classification, embeddings). Multiply your training cost by 12 or 4 accordingly. Don't forget checkpoint storage — it's often 3-5x your compute cost over a year.

What about multi-cloud for cost optimization?
Real answer: don't. The complexity tax exceeds the savings for 95% of teams. There are exceptions for very large regulated companies and for GPU scarcity mitigation, and that's it.

How has the cost picture changed in 2026?
Inference costs for open-weight models have fallen roughly 60-70% year-over-year since 2024, driven by quantization advances and better hardware. But workloads have grown more than that — average inference spend is up, not down. Cost per prediction is falling; total spend is rising because you're doing more.

What to Do Monday Morning

Pick your three largest cost drivers. Write them down. For each, write one decision you can make this quarter that would move it 30%+. That's your plan.

For most teams: (1) get inference utilization above 50%, (2) move non-latency-critical inference to batch, (3) reserve the baseline and burst the peak. Those three changes usually yield 40-60% savings with no product impact.

If you want a hand doing this properly, that's what we do at SIVARO. But you can get 80% of the way with a spreadsheet and a Saturday.

The teams that get this right don't have cheaper GPUs. They have honest models of how the cost scales. That's the whole lesson.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.