your gpu bill is not a model problem

The cloud bill has a very annoying habit.

It waits until everyone is talking about the exciting abstraction, then it shows up with the boring invoice.

For AI infrastructure, the exciting abstraction is the model.

Which model is smarter? Which one has better reasoning? Which one is cheaper per million tokens? Which one can explain a failing test with the confidence of someone who absolutely did not run it?

Useful questions.

Incomplete questions.

OpenCost 1.121.0 added Kubernetes inference cost tracking with llm-d and vLLM metrics. The interesting part is not only that it can calculate model and token cost. The interesting part is the distinction it forces platform teams to make.

What did the model's work cost?

And what did keeping the model ready cost?

Those are not the same question.

That difference is where many self-hosted AI spreadsheets go to die.

the model is not the workload

When teams talk about AI cost, they often jump straight to price-per-token.

That makes sense for SaaS APIs. You send tokens. You receive tokens. Somewhere in the middle, a billing system does billing-system things and your dashboard gets another little number to make you sad.

Self-hosting changes the shape.

Now you are not only buying inference.

You are running infrastructure.

There are GPUs. There is memory. There is a gateway. There are pods. There are model weights sitting in VRAM. There is routing. There is cache behavior. There are namespaces, teams, tenants, and probably one deployment nobody wants to touch because it was "temporary" in March.

This is why the OpenCost angle matters. It connects Kubernetes cost allocation to inference metrics from vLLM and llm-d. It is trying to answer cost questions in the place where the workload actually runs, not in a slide where all GPUs are magically 100% utilized and nobody ever pays for idle capacity.

That slide is beautiful.

It is also fiction.

usage cost can lie to you politely

The useful distinction is usage-based cost versus allocation-based cost.

Usage-based cost is about active work. The model processed tokens. GPU compute was consumed. Input and output tokens had real processing cost. KV cache hits changed that cost. Great.

Allocation-based cost is about availability. The model is loaded. GPU memory is reserved. The serving stack is alive. Common infrastructure is running. The system is ready to respond even if the request traffic is having a nap.

Both numbers are true.

Only one answers the build-versus-buy question.

If a self-hosted model costs one dollar per million tokens while it is actively working, and a hosted API costs two dollars per million tokens, the spreadsheet starts smiling.

"Look, we are cheaper."

Maybe.

If the GPU is busy enough.

If the model is mostly idle, the real cost per token includes the quiet hours too. A GPU with a model loaded is not free because the request queue is empty. It is just expensive in a less photogenic way.

This is the trap. Usage-based cost can make self-hosting look cheap because it ignores the carrying cost of readiness. Allocation-based cost includes the part production teams actually pay for at the end of the month.

The invoice does not care that your benchmark was efficient for seven minutes.

warm capacity is a product decision

There is a reason teams keep models warm.

Latency matters. Cold starts are bad. Users do not enjoy waiting while your platform gently remembers how to be useful. Some workloads need a model ready now, not after a scheduling adventure and a loading ceremony.

So keeping capacity warm can be the correct choice.

But it has to be a choice.

Not an accident.

This is where AI FinOps becomes platform engineering. The useful question is not "is this model expensive?" That is too vague. Everything is expensive if you squint with procurement nearby.

The better questions are more practical.

Which teams need this model warm?

Which traffic can tolerate queueing or scale-up?

Which low-volume workloads should use an external API instead?

Which models can share capacity?

Which routes should consolidate traffic to improve utilization?

Which experiments are quietly reserving production-grade hardware for demo-grade usage?

This is less glamorous than model selection. It is also where the money is.

cost labels become architecture

One detail I like in the OpenCost work is the label shape. The new metrics include things like model name, model version, namespace, workload type, and whether the cost basis is usage or allocation.

That sounds boring.

Good.

Boring labels are how the bill becomes debuggable.

Without them, AI cost turns into fog. Everyone knows the platform is expensive. Nobody knows if the expensive part is one team's chatbot, another team's batch summarizer, a forgotten staging deployment, or a model nobody retired because the migration ticket got renamed into a roadmap initiative and then disappeared.

Cost attribution is not just finance hygiene. It changes engineering behavior.

When teams can see the cost of keeping a model warm for their namespace, they can have a real conversation about latency, volume, and tradeoffs. When they only see an aggregate GPU bill, the conversation becomes vibes with a spreadsheet attachment.

We have seen this movie with Kubernetes before.

CPU requests looked like small YAML details until over-requested workloads became real money. Storage classes looked like plumbing until retained volumes became invoices. Logs looked harmless until everyone discovered that debugging output also has a billing department.

Inference is getting the same treatment.

The YAML is now connected to the token.

Wonderful and terrible, as usual.

the platform owns the gap

The gap between usage-based and allocation-based cost is the part I would watch.

It is the cost of readiness.

Sometimes that gap is healthy. A fraud system, incident assistant, or customer-facing workflow may justify warm capacity because response time matters. Fine. Put that in the architecture decision. Own it.

Sometimes the gap is waste. A model is deployed for a pilot with twelve users. A team copied production values into staging. A bigger model is serving requests that a smaller model could handle. A routing layer spreads traffic so thin that every model looks underutilized.

That is not a model problem.

That is a platform problem.

The platform owns the routing, isolation, quotas, schedules, autoscaling policy, cache behavior, and cost visibility. The model is only one component inside that system.

AI infrastructure is becoming normal infrastructure faster than people want to admit. It needs the same dull tools we already learned to respect: allocation reports, budgets, ownership labels, deployment review, capacity planning, and cleanup jobs with teeth.

Not because engineers love governance.

We do not.

Because otherwise the bill becomes the observability system.

And the bill has terrible UX.

self-hosting needs utilization, not optimism

I like self-hosting. I like owning the runtime. I like being able to tune the serving stack, inspect metrics, and avoid pretending that an API call is a complete architecture.

But self-hosting is not a personality trait.

It needs utilization.

If the allocation-based cost per million tokens beats the external API price, great. If it does not, the answer is not to argue with the spreadsheet until morale improves. The answer is to change the architecture or stop self-hosting that workload.

Route more traffic to fewer warm models.

Use smaller models where they are enough.

Separate latency-sensitive traffic from batch traffic.

Scale down experiments.

Expose cost per namespace and model version.

Make teams see the fixed cost they are asking the platform to carry.

The useful version of AI cost tracking does not end with "this model is expensive."

It ends with a decision.

Keep it warm because latency matters.

Move it to an API because utilization is bad.

Consolidate traffic because too many models are doing too little work.

Change routing because the current setup is optimized for a demo, not production.

That is the real lesson.

Your GPU bill is not a model problem.

It is a utilization story with Kubernetes labels.

Which is a very platform-engineering way for AI to become boring.

To test my projects, I use Railway. If you want $20 USD to get started, use this link.