DEV Community

Cover image for Self-host an LLM or use an API? A taxi versus car break-even sum
Deployed Lab
Deployed Lab

Posted on Originally published at deployedlab.dev

Self-host an LLM or use an API? A taxi versus car break-even sum

"Can we just run the model ourselves?" is a question that arrives the moment an AI pilot gets a real bill. The honest answer is a short piece of arithmetic, plus a few questions about the customer.

Here is the picture I use, then the sums.

Taxi or car?

  • An API is a taxi. You pay per ride, which here means per token. Use nothing and you pay nothing.
  • Self-hosting is a car. You rent GPUs (the chips that run the model) and run an open-weight model yourself. The cost is fixed each month, and extra trips are almost free.

Running a model to get an answer is called inference. The question is which side of the crossover point you are on.

There is a middle option too: a managed service that runs inside your own cloud account. Think of a rental car with a driver. The data stays with you, and someone else keeps the GPUs running.

Ask these questions in order

  1. Data rules first. If text may not leave the customer's own cloud account, a public API is out before cost matters.
  2. Then skills. Who can run GPUs in production? If nobody, lean towards a managed service in their account.
  3. Then volume. High and steady use favours the car.
  4. Not measured yet? Start on the API and measure.

The break-even sum

All numbers below are made up for the example. Use your own.

  • API price: $2 per million tokens
  • Pilot use: 800 million tokens a month, so the bill is 800 x $2 = $1,600
  • GPUs: $4,000 a month for one set
  • People (part-time engineer and on-call time): $2,000 a month
  • One set of GPUs can serve 4,000 million tokens a month

Self-hosting costs $4,000 + $2,000 = $6,000 a month, used or not.

break-even = fixed cost / API price per million tokens
           = $6,000 / $2
           = 3,000 million tokens a month
Enter fullscreen mode Exit fullscreen mode

The pilot uses 800 million, far below 3,000. Stay on the API.

Now try ten times the use, which is 8,000 million tokens:

  • API: 8,000 x $2 = $16,000
  • One set serves only 4,000 million, and GPUs come in whole sets, so you need two: 2 x $4,000 = $8,000
  • Add $2,000 for people: $10,000

Self-hosting saves about $6,000 a month on paper. Check quality and on-call before you celebrate.

Owning the car is three layers of work

  • Runtime: the model running on a GPU.
  • Infrastructure: many copies, shared load, GPU supply.
  • Tooling: releases, watching for problems, tests, cost counting.

The weakest layer sets the limit. And someone has to be on call at 3 a.m. That person's time is a real cost.

Caveats

  • The sum is a first filter, not a quote. It ignores quality, traffic spikes and setup time.
  • Idle GPUs cost the same as busy ones. Low use means you pay for hardware that does nothing.
  • "Open-weight means free" is a mix-up. The download is free. GPUs and people are not.
  • Prices change, and tokens are priced per million. Re-run the sum with current numbers.
  • A cheaper model that fails your test questions is not cheaper.
  • You do not have to pick one for good. Many teams self-host the steady load and rent the spikes.

Try it with your own numbers

The full chapter has a decision tree, diagrams of the three layers, and a break-even calculator where you can change the price, GPU cost and volume and watch the answer move. It is free with no signup: Why run models yourself?. It is one of the free chapters of Deployed Lab, a course on AI engineering and the forward deployed engineer role.

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

The taxi-versus-car framing works because it keeps utilization in the model. One addition I’d make is operational labor: patching images, model upgrades, observability, capacity incidents, and on-call time are part of the owned-car cost even when the GPU invoice is low.\n\nThe break-even point also changes by workload shape. Spiky, latency-tolerant batch work and steady interactive traffic can reach opposite conclusions with the same monthly token total. Showing both curves makes the decision much more actionable.