The Self-Hosted LLM Breakeven Point Isn't 2M Tokens a Day. It's a Ratio.

Self-hosting an LLM is not automatically cheaper. This is the single most common mistake in these calculations: people price the GPU, compare it against last month's API bill, and walk away with a number that has almost nothing to do with what they'll actually spend.

Savings show up under two conditions - when you've calculated the breakeven point for your specific workload, and when you've split traffic between a local model and an API instead of attempting a wholesale migration.

Enterprise spending on model APIs more than doubled in six months - from $3.5 billion in late 2024 to $8.4 billion by mid-2025, according to Menlo Ventures. Inference in production, not model development, is what drove it.

TL;DR

• A production instance on a consumer GPU costs roughly $850/month, and more than half of that is an engineer, not hardware.
• Breakeven isn't a token volume. It's $850 ÷ (your blended API price × 30) - which lands anywhere from 2.3M to 94M tokens/day depending on which tier you're displacing.
• The "2M tokens/day" rule of thumb is correct for replacing frontier calls and badly wrong for anything cheaper.
• The architecture that works is hybrid: local for high-volume and regulated traffic, API for hard reasoning.

Why teams start doing this math

Three pressures, usually arriving in this order:

1. Cost scales linearly with volume. There's no ceiling. The product grows, the bill grows with it. For an API that's both a feature (you don't pay for idle capacity) and a problem (you can't fix a budget).
2. Data leaves your perimeter. For HIPAA or PCI-DSS, an enterprise agreement is not a substitute for controlling your own infrastructure. Every prompt physically travels somewhere else.
3. Network round-trips. 50–200ms per request, which you feel in any real-time UX.

What a local instance actually costs

The figure $850/month shows up throughout this post as the cost of one production instance on a consumer GPU. Here's where it comes from - because in most write-ups this number is either pulled from thin air or collapsed into "the price of a graphics card."

A few notes on each:

• Capex. The RTX 5090 carries an official MSRP of $1,999, but GDDR7 shortages kept street prices at $3,500–4,000+ through early 2026. Add a 1,200W+ PSU and serious cooling for its 575W TDP, plus ~$1,200 for CPU, RAM, and chassis. Check current pricing before you build this into a business case - the memory market moves in weeks.

• Electricity. Substitute your own rate. The spread between jurisdictions is severalfold, not a rounding error. An RTX 4090 at 450W still costs $50+/month for the card alone.

• Operations. This is the largest line item and the one everyone underestimates. An MLOps engineer who updates models, debugs CUDA errors, and keeps monitoring honest costs more than all the hardware combined.

More than half of that total isn't silicon or electricity - it's a person. If nobody on the team has that skill set, self-hosting isn't cheaper. It just moves the cost into a budget line that doesn't exist yet.

Why this number matters
The important property of $850 is that it's fixed. It doesn't move until the server hits its throughput ceiling. Everything below is a comparison between a flat line and a linear one.

The breakeven point isn't a volume - it's a ratio

The industry rule of thumb says self-hosting becomes competitive above 2 million tokens per day. The rule works, but it's incomplete: it says nothing about what you're replacing. That turns out to be the dominant variable.

Step 1 - Blended price per tier

At a 70% input / 30% output split:

That 70/30 split is doing real work here, so adjust it. Chat and summarization workloads skew further toward input and land cheaper; agentic loops that generate long tool calls skew toward output and land considerably more expensive.

Step 2 - Monthly bill by volume

The formulas:
monthly API cost = volume in M/day × price per 1M × 30
breakeven volume = infrastructure cost ÷ (price per 1M × 30)

Step 3 - Where "2M" actually comes from

Plug $850 into the second formula and the answer splits three ways:

• Against the cheap tier - ~94M tokens/day
• Against mid tier - ~5.7M tokens/day
• Against frontier - ~2.3M tokens/day

There it is. The 2M rule is the breakeven for displacing frontier calls, not calls in general. If you're running classification through the cheapest tier available, a local server will essentially never pay for itself - and that's a correct, useful conclusion, not a failure.
It also explains why published estimates disagree by an order of magnitude. They're pricing different substitutions.

Two things that shift the math

Consolidation. That fixed $850 doesn't serve one workload. A single node can run an internal helpdesk bot, a document analysis pipeline, and a coding assistant at the same time. Three applications at 700K tokens/day each look uneconomical in isolation; together they're 2.1M on one server at the same fixed cost. Fine-tuning works the same way - with a provider, every training run is a separate charge; on your own hardware it's capacity you've already paid for.

Payback period. At a steady 10M+ tokens/day against the mid tier:

That lands squarely inside the 6–12 month range usually quoted for 10M+ workloads.

Architecture: hybrid, not migration

Moving everything local is the wrong goal. GPT-5.6, Claude Opus 5, and Gemini 3.1 Pro (as of publication - this list will age before you finish reading) simply aren't available for self-hosting. The question isn't what replaces them. It's what you stop sending to them.

What goes local

• Simple, high-volume, and sensitive traffic:
• Ticket classification and routing
• Form and document data extraction
• Boilerplate response generation
• Routine autocomplete and code review in the IDE

A support team handling hundreds of near-identical tickets a day is the archetype: predictable volume, straightforward tasks, and frequently customer PII that can't go anywhere. What counts as "local-grade" has shifted a lot, too - a well-quantized 32B now handles work that needed a 70B two years ago.

What stays on the API

• Low volume, high cost of being wrong:
• Complex contract or document analysis
• Unfamiliar codebases
• Genuinely novel architectural decisions
• Anything where multi-step reasoning quality is the product

A legal team reviewing contracts once a week is the archetype. The call is expensive and entirely justified.

What determines your savings

Published cases land anywhere from −33% to −83%, and the commonly cited range for teams that reach a hybrid setup is 40–70% versus a fully API-dependent stack. Where you fall inside that spread comes down to two things:

1. How much of your traffic is even allowed to leave the perimeter.
2. Which tier it was hitting before the move.

Displacing a cheap tier yields a smaller percentage for the same engineering effort - see the arithmetic above. The spread between two projects usually says more about their compliance posture than about their engineering.

VRAM: the constraint everything else measures against

The baseline rule: 0.5GB of VRAM per billion parameters at 4-bit quantization. If the model doesn't fit entirely in VRAM, the system falls back to CPU and throughput drops 10–100x. The gap between 30 TPS and 1 TPS is the gap between a working system and a broken one.

Sizing

Treat the speed column as a conservative floor for single-stream generation through Ollama. On an optimized serving stack the same model classes go several times faster - see the inference server table below.

Bandwidth, not TFLOPS

For inference, memory bandwidth decides:

• RTX 5090 - 1.79 TB/s GDDR7, 32GB, ~213 tok/s on 8B models
• RTX 4090 - 24GB, 128 tok/s. The 5090 is 67% ahead
• RTX 3090 - still earns its place used, for the same 24GB at a fraction of the price

CUDA remains the default. AMD (ROCm) and Intel (IPEX-LLM) will run basic inference, but with noticeably more configuration friction: a large share of models, optimized kernels, and serving frameworks are NVIDIA-first or NVIDIA-only, and some models simply won't start outside CUDA.

Quantization and a starting model

Q4_K_M is the default. Going from Q8 to Q4 nearly halves memory for quality loss you're unlikely to notice. On newer hardware, AWQ and NVFP4 (Blackwell) push the quality/memory tradeoff further.
A sensible starting model is Qwen3.5-9B (~6.6GB, 262K native context) - it fits on a consumer card with room to spare. Check the actual size of whatever tag you pull before you deploy; it varies meaningfully between quantizations.

Implementation

Step 1 - Dev setup: Ollama + Open WebUI

yaml
# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      # Ollama listens on all interfaces by default - bind to loopback,
      # expose it only through a gateway or VPN
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama_models:/root/.ollama   # models survive restarts and image updates
    environment:
      - OLLAMA_MAX_LOADED_MODELS=1    # >1 on a single card almost always means OOM
      - OLLAMA_KEEP_ALIVE=30m         # unload the model after idle
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama_models:
  webui_data:

Two things worth knowing before you run this:

• Resource floor. Budget at least 8GB RAM for containers and 50+ GB of disk for models. Ollama detects hardware on its own: GPU if there is one, CPU if there isn't, just slower.
• The migration lever. Ollama exposes an OpenAI-compatible API on port 11434. Anything built on the OpenAI SDK switches to the local endpoint by changing one variable. No code changes.

Step 2 - Production: vLLM
Ollama handles up to 4 concurrent requests. That's fine for development and testing. The moment real simultaneous users show up is the moment to move to vLLM.

Those TPS ranges span multiple model sizes and GPU generations - the low end is larger models on an RTX 4090, the high end is 8B on a 5090. Use them to compare tools against each other, not as a benchmark. Measure your own numbers on your own workload.

A note on the multipliers in marketing material: published PagedAttention figures range from 19x to 24x over HuggingFace Transformers depending on benchmark, model, and dataset. The metric that actually matters is P99 under your real concurrency. A reasonable expectation from vLLM on a typical profile: under 100ms at 128 concurrent requests.

Ray Serve and TGI win for teams already living in the corresponding ML stack. Without that stack, they add complexity without a payoff.

bash
docker run --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:latest \
  --model "$MODEL_REPO" \                # Hugging Face repo id
  --served-model-name local-default \    # stable name for clients, decoupled from repo id
  --host 0.0.0.0 --port 8000 \
  --api-key "$VLLM_API_KEY" \            # basic auth
  --quantization awq \                   # only if you actually have AWQ weights
  --gpu-memory-utilization 0.90 \        # 0.95+ hits OOM on KV-cache spikes
  --max-model-len 16384 \                # the main lever on VRAM consumption
  --max-num-seqs 64 \                    # ceiling on concurrent sequences
  --tensor-parallel-size 1                # >1 only for multi-GPU

Three flags people trip over:

• --ipc=host is mandatory. Without it, shared memory is capped at the default 64MB and the process dies during batching.
• --gpu-memory-utilization counts against total card memory, not free memory. If anything else is already resident on the GPU, dial it down manually.
• --max-num-seqs is your real concurrency ceiling. vLLM removes the hard 4-request limit, but concurrency isn't unbounded - it's this flag plus whatever VRAM is left for KV-cache. Capacity planning starts there.

Step 3 - The router

The simplest router that works is a static map from task type to model. The temptation to put a classifier in front of inference is strong, but a classifier is one more model call, one more source of latency, and one more thing that can fail. A static map can be debugged with grep and never lies to you.

The more consequential decision is what happens when the local instance goes down, and one policy for all traffic doesn't work here:

• Ordinary tasks fail over to the API. A slightly slower response, and that's the end of it.
• Regulated traffic fails hard - queue it or return 503. Automatically falling back to an external API with sensitive data is precisely the incident this whole architecture was built to prevent.

python
import os
import httpx
from openai import (
    OpenAI,
    APIConnectionError,
    APITimeoutError,
    InternalServerError,
    RateLimitError,
)

LOCAL = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key=os.environ["VLLM_API_KEY"],
    timeout=httpx.Timeout(30.0, connect=2.0),
    max_retries=1,
)
# Example uses an OpenAI-compatible provider. For Claude or Gemini,
# use that vendor's SDK or a proxy layer with an OpenAI-compatible endpoint.
FRONTIER = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

LOCAL_TASKS = {"classify", "extract", "summarize_ticket", "autocomplete"}

LOCAL_MODEL = os.environ["LOCAL_MODEL"]        # must match --served-model-name
FRONTIER_MODEL = os.environ["FRONTIER_MODEL"]

# Deliberately narrow: connection, timeout, 5xx, 429.
# A 400 or 401 is a bug in our config, not a local outage - let it surface.
TRANSIENT = (APITimeoutError, APIConnectionError, InternalServerError, RateLimitError)


def route(task: str, messages: list, regulated: bool = False):
    # `regulated` is set by the calling service: PCI-DSS for payment data,
    # HIPAA for PHI. Different rules, one thing in common:
    # this traffic does not leave the perimeter.
    if not (regulated or task in LOCAL_TASKS):
        return FRONTIER.chat.completions.create(model=FRONTIER_MODEL, messages=messages)

    try:
        return LOCAL.chat.completions.create(model=LOCAL_MODEL, messages=messages)
    except TRANSIENT:
        if regulated:
            raise          # no fallback: a refusal beats a leak
        return FRONTIER.chat.completions.create(model=FRONTIER_MODEL, messages=messages)

Catching a bare Exception here would defeat the point - a config error would quietly route to the API disguised as a local outage.

Things that will bite you

The silent CPU fallback

Ollama doesn't crash when a model won't fit in VRAM. It offloads layers to CPU and starts serving 1 TPS instead of 30. Alerts stay quiet because the service is technically healthy, so the first signal you get is users complaining.
Check after every model update:

bash
ollama ps    # PROCESSOR column should read "100% GPU", not "47%/53% CPU/GPU"
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Context window isn't free

Doubling the window barely dents throughput but does cost memory: the KV-cache grows linearly with context length, so 8K→16K doubles it. As a share of total consumption, that's typically +15–20% for a 9B-class model with Q4 weights - but run the numbers for your own model, because the fraction changes with model size and window length.

Maxing out --max-model-len and getting a silent CPU fallback is a well-established way to kill production on a Friday evening. Time is better spent on RAG (pgvector, Weaviate, Elasticsearch): memory stays predictable and the retrieved context is more relevant than a bloated window ever is. A smaller model with a solid pipeline routinely beats a larger one with no context - faster, and on cheaper hardware.

Permissions in the vector store

The common mistake is treating it as one shared index for all users. A query from one user legitimately surfaces another user's confidential document, and technically no rule was broken. "Who can retrieve which document" is a question to answer before deployment, not after.

The fix is document-level permissions, in one of two shapes:

• Separate indexes per role, or
• Filtering retrieval results by the requesting user's rights before context reaches the model

MoE math doesn't work like dense math

The temptation is to size VRAM by active parameters. A 235B MoE with 22B active costs roughly what a dense 22B costs to run - but every expert weight still has to be resident in memory. The win is quality per unit of compute, not memory savings.

*Network isolation
*
The inference server shouldn't be reachable from outside without a specific reason: private subnet, access through an internal gateway or VPN. Ollama listening on all interfaces by default is convenient for local testing and unacceptable in production.

When not to self-host

Being honest here is more useful than being enthusiastic. Three scenarios where self-hosting clearly loses:

1. Low or unpredictable volume. Below 1M tokens/day the API is cheaper at every tier - the infrastructure never pays for itself. With spiky traffic you have to size the local server for peak, which means it idles most of the time. APIs don't have that problem.
2. You need frontier models. The gap between top proprietary models and the best open weights keeps narrowing, but for genuinely hard multi-step reasoning it's still there.
3. No MLOps expertise. Self-hosted inference isn't set-and-forget. Models need updating, CUDA errors arrive at the worst possible moment, and performance degrades as load patterns shift. Without someone who can handle that, maintenance overhead eats the token savings faster than you'd expect - and in the $850 breakdown above, that line item is more than half the total.

Open question

The most debatable piece of this setup is the router itself. A static map is transparent and free, but it can't tell a simple ticket from a hard one inside the same category, so some requests go local when they'd be better off escalated.

So: how do you split traffic between local and frontier models?

• Hardcoded rules?
• A dedicated router model at the front door?
• Confidence-based escalation, where the local LLM hands off on its own? And at your volumes, does the classifier overhead actually pay for itself?

Full TCO breakdown, quantization format comparison, RAG and fine-tuning architectures, and the complete breakeven tables are in the full self-hosted LLM guide, which this post is drawn from.