Self-hosting an LLM pays off in two situations: you're running high enough volume that GPU cost per token beats API cost per token, or you have a data residency requirement that API calls can't satisfy no matter the price. Outside those two cases, calling OpenAI, Anthropic, or Google's API is cheaper, faster to ship, and easier to maintain than running your own inference stack.
We get this question a lot from healthcare and govtech founders specifically, because "can the data leave our network" is often not a cost question at all, it's a legal one. So the framework below splits the decision into cost and compliance, because founders usually only need one of the two answers, not both.
The cost math, worked through
A single NVIDIA A100 80GB costs roughly $2 to $3 per hour on-demand from providers like Lambda or CoreWeave. Run it 24/7 for a month and you're at $1,500 to $2,200, regardless of whether it processes one request or one million. That's the core problem with self-hosting: you're paying for capacity, not usage.
Compare that to API pricing, where you pay per token and nothing when idle. For a workload doing, say, 5 million tokens a day on a mid-tier model, API costs typically land well under $1,500/month, and you have zero ops burden. The crossover point where a dedicated GPU starts winning is usually somewhere north of 20 to 50 million tokens/day of sustained, predictable traffic, and even then you need someone maintaining the serving stack (vLLM, TGI, or similar) and handling GPU failures, driver updates, and scaling.
We went through a version of this math ourselves. Our cold-outreach tool scrapes each prospect's site with a self-hosted Firecrawl instance and a local model to extract facts and draft an email in a single call. At our volume, running that step locally was cheaper and gave us more control over rate limits than routing every scrape through a hosted API, but we didn't self-host the actual sales copywriting, that still goes to a hosted model because the volume doesn't justify dedicated GPU capacity and the quality bar is higher. Mixing the two, self-hosted for cheap deterministic tasks, API for high-stakes generation, is usually the right shape for early-stage products. It's the same reasoning we lay out in LLM cost optimization: match the model tier and hosting model to the task, not the other way around.
Where the "hidden" self-hosting costs come from
- Serving infrastructure. vLLM or Text Generation Inference handle batching and KV-cache management, but someone has to operate them, patch them, and handle OOM crashes under load.
- Model updates. Open-weight models improve fast. Committing to self-hosting means you own the fine-tuning and evaluation cycle every time a better base model ships, instead of a provider swapping it in behind an API.
- Redundancy. One GPU node is a single point of failure. Real uptime needs at least two, which roughly doubles the baseline cost we quoted above.
The compliance case, which is a different question entirely
If your driver is data residency, PHI, or a client contract that prohibits sending data to a third party, the cost math above is mostly irrelevant. You're not comparing dollars, you're comparing "can we legally do this at all." That's the situation a lot of our healthcare and government-adjacent clients are in, and it's the same territory we cover in court-ready architecture for healthcare AI.
A few things worth being precise about here:
- Self-hosting solves the "data leaves our network" problem, but it does not by itself make you HIPAA compliant. You still need encryption at rest and in transit, access logging, and a documented retention policy around the model and its logs.
- Managed API providers do offer BAA-covered, HIPAA-eligible endpoints (both OpenAI and Anthropic offer these for enterprise customers), so "we need HIPAA compliance" doesn't automatically mean "we need to self-host." Check that option before assuming you need your own GPUs.
- Government and defense contracts are a different story. Many require the model and data to sit inside an accredited environment (FedRAMP, IL4/IL5, or fully air-gapped), where a public API is a non-starter regardless of any BAA. That's where self-hosting stops being optional.
Open-weight model quality, honestly
For narrow tasks, classification, extraction, structured triage, intent routing, an open-weight model fine-tuned or well-prompted on your data can match a frontier closed model, and you keep full control over latency and privacy. Meta's Llama family and Alibaba's Qwen models are both reasonable starting points for self-hosted deployments.
For open-ended reasoning, long documents, or anything where the failure mode is "subtly wrong answer that sounds confident," the frontier API models still have an edge. Don't self-host your way into a quality regression to save money on a task where the model's judgment is the product.
A practical decision path
- Estimate real token volume, not requests. Multiply average tokens per request by daily request count. If you're under roughly 10 to 20M tokens/day, start with an API.
- Check if a HIPAA/BAA-eligible endpoint from a major provider satisfies your compliance requirement. If yes, you likely don't need to self-host.
- If neither cost nor compliance forces your hand, don't self-host. The ops overhead is a distraction from building product in the first 12 to 18 months of a company's life.
- If you do self-host, budget for redundancy and a standing on-call rotation, not just the GPU line item.
Same logic that applies to picking between managed and bring-your-own-key providers in BYOK vs. managed LLM keys: the cheapest-looking option on paper is rarely the cheapest option once you count the operational load it puts on your team.
If you're trying to work out where your product actually falls on this line, let's talk.
Originally published on the Pykero blog.
Top comments (0)