Originally published on TechSaaS Cloud
Originally published on TechSaaS Cloud
Self-Hosted LLMs vs API: Real Cost Comparison at Production Scale
The numbers nobody shares when pitching "just use the API" or "just self-host it."
The $4,200/Month Wake-Up Call
We ran OpenAI's GPT-4 API for 9 months straight. $4,200/month, predictable billing, zero operational overhead. The CFO loved it. The engineering team loved it. Everyone was happy.
Then usage crossed 100,000 requests per day and the economics flipped overnight.
This isn't a theoretical exercise. We're sharing the actual cost model we built when deciding whether to migrate inference workloads to self-hosted infrastructure — and the framework we now use for every AI infrastructure decision.
The Cost Matrix: API vs Self-Hosted at Three Scales
At 10,000 Requests/Day
| Cost Category | OpenAI API | Self-Hosted (Llama 3 70B) |
|---|---|---|
| Compute | ~$1,400/mo | ~$2,100/mo (A100 amortized) |
| Ops/MLOps staff | $0 | ~$700/mo (fractional) |
| Monitoring/infra | $0 | ~$200/mo |
| Total | ~$1,400/mo | ~$3,000/mo |
Verdict: API wins by 2x. At this scale, the operational burden of self-hosting destroys any compute savings. You need GPU procurement, model serving infrastructure (vLLM or TensorRT-LLM), monitoring, and someone who knows what they're doing. For a 10-person startup, this is a distraction.
At 100,000 Requests/Day
| Cost Category | OpenAI API | Self-Hosted Cluster |
|---|---|---|
| Compute | ~$14,000/mo | ~$4,800/mo (3x A100s) |
| Ops/MLOps staff | $0 | ~$1,200/mo (dedicated) |
| Monitoring/serving | $0 | ~$400/mo |
| Total | ~$14,000/mo | ~$6,400/mo |
Verdict: Self-hosted wins by 2.2x. The break-even point sits around 55,000-65,000 requests/day depending on your model choice and token length. This is where the conversation gets interesting.
At 1,000,000 Requests/Day
| Cost Category | OpenAI API | Self-Hosted Fleet |
|---|---|---|
| Compute | ~$140,000/mo | ~$22,000/mo (12x A100s) |
| MLOps team (2 FTEs) | $0 | ~$12,000/mo |
| Infrastructure | $0 | ~$3,000/mo |
| Total | ~$140,000/mo | ~$37,000/mo |
Verdict: Self-hosted wins by 3.8x. At this scale, the API cost is existential. Companies like Zoho figured this out years ago — their entire AI stack runs on self-hosted infrastructure across their Chennai and Austin data centers.
What the Spreadsheet Doesn't Capture
1. Latency Control
Our self-hosted p99 latency: 180ms, consistent. OpenAI API p99: anywhere from 200ms to 2,400ms depending on their load. For real-time applications — chatbots, code completion, search ranking — this variance kills user experience.
One of our fintech clients in London had an SLA requirement of sub-300ms for their compliance checking pipeline. The API couldn't guarantee it. Self-hosted could.
2. Data Residency and GDPR
For European clients, this is often the deciding factor before cost even enters the conversation. Running inference on EU-hosted servers with no data leaving the jurisdiction simplifies compliance dramatically.
German companies especially care about this — Bundesamt für Sicherheit in der Informationstechnik (BSI) guidelines are strict. Indian companies building for European markets (think Freshworks, Razorpay) face the same calculus.
With API providers, you need a Data Processing Agreement, legal review of their data retention policies, and ongoing compliance monitoring. Self-hosted? Your data never leaves your VPC.
3. Model Customization — The Real Unlock
This is where self-hosting pays dividends that don't show up in cost comparisons. We fine-tuned Llama 3 on domain-specific data and saw a 12% improvement on our eval benchmarks compared to GPT-4 for our specific use case.
The fine-tuning itself cost ~$800 in compute. The ongoing inference is cheaper because the fine-tuned 70B model outperforms GPT-4 for our domain, meaning fewer retry loops and shorter prompt chains.
4. Hidden Self-Hosting Costs Nobody Budgets For
Here's where teams get burned:
- MLOps talent: €80,000-120,000/year in Germany, ₹25-40 lakh in India, $150,000-200,000 in the US. You need at least one person who understands GPU orchestration, model serving, and inference optimization.
- GPU procurement: Still 8-12 weeks lead time for A100s. H100s are worse. Plan ahead or use cloud GPU providers as a bridge.
- Model serving infrastructure: vLLM, TensorRT-LLM, or NVIDIA Triton. Each has trade-offs. Expect 2-4 weeks of setup and tuning.
- Monitoring: Your existing Prometheus/Grafana stack needs GPU metrics, token throughput dashboards, and model quality monitoring. Budget 40-60 hours of engineering time.
- Failover: What happens when your GPU node dies at 3am? You need either redundancy or an API fallback — which means maintaining both stacks.
The Framework We Use Now
After running both approaches for over a year, here's our decision tree:
Under 50K requests/day → API always. The operational simplicity isn't worth sacrificing. Spend your engineering time on product, not GPU orchestration.
50K-100K requests/day → Hybrid. Route simple, high-volume tasks (classification, extraction, summarization) to self-hosted models. Keep complex reasoning tasks on GPT-4/Claude API. This is where most growing companies should be.
Over 100K requests/day → Self-hosted primary, API fallback. Build the team, invest in the infrastructure, but always maintain API access for burst capacity and failover.
Data residency requirements → Self-hosted regardless of scale. If your data cannot leave a specific jurisdiction, the cost comparison is secondary. Budget for it from day one.
The Singapore Factor
For APAC companies routing through Singapore, there's an additional wrinkle: cloud GPU availability in the region is still limited compared to US/EU. AWS ap-southeast-1 has A100 instances but availability is spotty. Companies like Grab and Sea Group have been building their own GPU clusters for this reason.
If you're an Indian startup serving Southeast Asian markets, consider colocation in Singapore with your own hardware. The upfront cost is higher, but the latency and availability improvements pay for themselves within 6 months at scale.
Mistakes We Made During Our Migration
We want to be transparent about what went wrong, because these are the mistakes we see other teams repeat.
Mistake 1: Underestimating cold-start latency. Our self-hosted Llama 3 70B model takes 45 seconds to load into GPU memory. When our primary node crashed at 2am and the failover kicked in, users experienced 45 seconds of downtime while the model loaded. API providers handle this transparently — you never see their cold starts. We fixed this by keeping a warm standby model loaded on a secondary node, but that doubled our GPU cost for the failover capacity.
Mistake 2: Ignoring token-length variance. Our cost model assumed average token usage. In reality, 15% of our requests were 4x longer than average (complex reasoning tasks with long context windows). These heavy requests consumed disproportionate GPU time and threw off our capacity planning. We now route by estimated token length: short requests to self-hosted, long-context requests to API.
Mistake 3: Not accounting for model updates. OpenAI ships model improvements continuously — you get better outputs for the same price without doing anything. Self-hosted models are frozen in time unless you actively retrain and deploy new versions. We budgeted $0 for ongoing model evaluation and retraining. The real cost is ~$2,000/quarter for fine-tuning updates plus 20 engineering hours for evaluation and deployment.
Mistake 4: Building monitoring from scratch. We spent 3 weeks building custom Grafana dashboards for GPU utilization, token throughput, and model quality metrics. We should have started with vLLM's built-in Prometheus metrics and only customized what we needed. The same monitoring principles we cover in our CI/CD pipeline optimization guide apply here — start with what exists, customize incrementally.
Frequently Asked Questions
Q: Can I use cloud GPU providers (Lambda Labs, RunPod, CoreWeave) instead of buying hardware?
Yes, and we recommend this as the starting point. Cloud GPUs let you test self-hosting economics without the 8-12 week procurement cycle. The per-hour cost is higher than owned hardware, but the flexibility to scale up/down is worth it until you've validated your workload patterns. Once you're consistently running 80%+ utilization, owned hardware starts making sense.
Q: What about smaller models? Do the economics change for 7B or 13B models?
Dramatically. A fine-tuned 7B model runs on a single A10G (~$0.75/hour on cloud), making self-hosting viable at much lower request volumes. We covered this in detail in a previous analysis of fine-tuning economics for enterprise workloads. The break-even for 7B models can be as low as 10,000 requests/day.
Q: How does this compare to using open-weight models on cloud providers (e.g., Bedrock with Llama)?
Cloud-hosted open models (AWS Bedrock, GCP Vertex AI) sit between pure API and pure self-hosted. You get the operational simplicity of an API with some of the cost benefits of open models. The trade-off: you lose fine-tuning flexibility and data residency control. For regulated industries — fintech in the UK, healthcare in Germany — this may not satisfy compliance requirements. For everyone else, it's a legitimate middle ground.
Q: We're a 5-person startup. Should we even think about this?
No. Use the API. Spend every engineering hour on product. Come back to this article when your API bill crosses $5K/month. Seriously — premature optimization of AI infrastructure is one of the most common wastes of early-stage engineering time. We've written about this pattern in our build vs buy framework — the same principles apply to AI infrastructure decisions.
Q: What about inference-as-a-service providers like Anyscale, Together AI, or Fireworks?
These sit between pure API and pure self-hosted. You get open-model pricing (significantly cheaper than OpenAI) with managed infrastructure (no GPU ops). For teams between 50K-150K requests/day who don't want to hire MLOps talent, this is often the sweet spot. The trade-off: less control than self-hosted, more cost than doing it yourself at high scale, and you're still sending data to a third party. For regulated industries, this may not satisfy data residency requirements.
What We'd Do Differently
If we started today:
- Start with the API. Always. Get your product-market fit first.
- Track your API spend weekly. Set alerts at $5K, $10K, $15K/month.
- When you hit $10K/month, start the self-hosting evaluation. Not the migration — the evaluation.
- Hire MLOps talent before you need them. The 8-week GPU procurement window is nothing compared to the 12-week hiring cycle for good MLOps engineers.
- Run hybrid for at least 3 months before going fully self-hosted. You'll discover edge cases that only show up at scale.
- Budget for ongoing model maintenance. Fine-tuning isn't a one-time cost. Plan for quarterly retraining cycles and A/B testing infrastructure.
Related Reading
- Multi-Cloud Strategy Pitfalls — the same hidden cost analysis applied to cloud infrastructure decisions
- Build vs Buy Framework — the decision framework we use for all infrastructure investments, including AI
- Secret Management Best Practices — securing API keys and model credentials in production
The question was never "self-hosted or API." It was always "at what scale does the switch make financial sense for your specific workload?"
We help engineering teams model this decision. If you're hitting $10K+/month in API costs and wondering whether it's time, let's talk.
Get a free infrastructure audit →
Subscribe to our newsletter for weekly deep-dives into infrastructure decisions that save real money.
Top comments (0)