I compared the real cost of running LLMs on AWS - here's when each option makes sense

#aws #llm #devops #finops

AWS gives you three ways to run LLM inference in production. I've deployed all three for clients and the decision always comes down to the same variables: volume, team size, and how much you value your weekends.

Here's the short version.

The three paths

Bedrock — Fully managed, pay-per-token. You call an API, you get tokens back. No GPUs, no cold starts, no 3am pages about OOM pods.

SageMaker Endpoints - Semi-managed. You bring your model (or a fine-tuned one), deploy it on dedicated instances, and handle autoscaling. Pay per hour whether you're serving requests or not.

Self-hosted on EKS — Full control. vLLM or TGI on GPU spot instances with Karpenter. Cheapest per token at scale, most operational overhead.

The cost crossover that matters

This is the table I keep coming back to with every client:

Volume	Bedrock (Haiku)	SageMaker (g5.xlarge)	EKS (g5.xlarge spot)
1K req/day	~$36/mo ✓	~$1,015/mo	~$674/mo
50K req/day	~$1,800/mo	~$1,015/mo	~$674/mo ✓
500K req/day	~$18,000/mo	~$6,090/mo	~$2,022/mo ✓

The crossover point where self-hosting beats Bedrock: 10,000–20,000 requests/day.

Below that, Bedrock wins on simplicity alone. Above it, you're leaving serious money on the table.

The hidden cost nobody models upfront

Teams prototype on Bedrock (smart move — it's the fastest path to production). But the cost curve isn't linear. At 10K requests/day it's cheap. At 50K it's "we need to talk to finance." At 500K it's a rearchitecture project.

The mistake is not choosing Bedrock at low volume. The mistake is not planning the exit path before you need it.

Quick decision framework

You should pick...	When...
Bedrock	No ML infra team, <50K req/day, need frontier models (Claude, Llama)
SageMaker	Fine-tuned models, predictable traffic, need dedicated VPC
EKS self-hosted	>100K req/day, open-source models, dedicated platform team

What I actually recommend

Use a hybrid. Most production systems I've deployed use:

Bedrock for complex reasoning and customer-facing chat (low volume, high quality)
Self-hosted vLLM on EKS for classification, extraction, and embeddings (high volume, cost-sensitive)

Route by task complexity. Best of both worlds.