Like any founder these days, my AI computing bill was horrendous. I was running open-weight models for several of my own products, and I started looking for cheaper options. Every few weeks I'd find a provider with better rates, migrate over, and then discover the prices had shifted again.
So I did what any developer would do when faced with a repetitive task: I built a spreadsheet.
It Started as a Database
The spreadsheet turned into a SQLite database pretty quickly. I started polling pricing APIs from the providers I was already using (Together, DeepInfra, Fireworks, a few others) and logging the results every few minutes. Just for my own reference, so I could see trends and decide when to switch.
A few weeks in, the data showed something I hadn't expected. The price for the exact same model, same weights, same architecture, varied wildly across providers. Not by 10-20%. By 2x, sometimes more. And the cheapest provider wasn't always the same one. It changed multiple times per day. Off-peak hours (US night, EU morning) were consistently cheaper. Some providers had stable pricing; others fluctuated hourly, creating windows where they'd drop well below their usual rates.
I was looking at this data every morning over coffee, manually deciding which provider to point my services at. That got old fast.
From Tracking to Routing
If I already had a database of live prices, why not just route requests automatically?
I wrote a proxy that sat between my applications and the inference providers. When a request came in, it looked up the current cheapest provider for that model and forwarded the request there. If that provider failed, it tried the next cheapest. My apps didn't need to know or care which backend was serving them.
I kept it OpenAI-compatible from the start because all my code already used the OpenAI SDK. Switching meant changing one line:
from openai import OpenAI
client = OpenAI(
base_url="https://api.aivory.net/v1", # the only change
api_key="your-aivory-key"
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Hello"}],
stream=True
)
Streaming, function calling, JSON mode, retry logic. All of it works because the API contract is identical.
Then I Started Using It for Everything
Once the router was running, I pointed all my own services at it. I run a few products that use inference heavily, and within the first month my costs dropped by about 60% without changing any application code.
That was when I thought: other people have this problem too. Most teams pick a provider, integrate their SDK, and stay there. Switching means rewriting integration code, handling different auth flows, debugging streaming behavior differences. The friction of switching is what lets providers charge different rates for what is essentially the same commodity.
What the Pricing Data Actually Looks Like
Here are real numbers from our API right now. These change constantly, which is the whole point:
The model pricing dashboard. 133 models across vendors like Anthropic, DeepSeek, Google, Meta, and Qwen, each with live per-token pricing.
DeepSeek V3.2 (13 provider endpoints tracked):
| Input (per 1M tokens) | Output (per 1M tokens) | |
|---|---|---|
| Cheapest | $0.46 | $0.83 |
| Mid-range | $0.49 | $1.13 |
| Most expensive | $0.54 | $1.38 |
Llama 3.3 70B (4 provider endpoints):
| Input (per 1M tokens) | Output (per 1M tokens) | |
|---|---|---|
| Cheapest | $0.38 | $0.51 |
| Mid-range | $0.40 | $0.56 |
| Most expensive | $0.92 | $0.92 |
Qwen 3 235B (6 provider endpoints):
| Input (per 1M tokens) | Output (per 1M tokens) | |
|---|---|---|
| Cheapest | $0.35 | $0.40 |
| Mid-range | $0.42 | $1.68 |
| Most expensive | $0.48 | $3.15 |
That last one is wild. The output price for Qwen 3 235B varies almost 8x across providers. And these are all serving the same model weights.
The Same Problem, but for GPUs
The entire endavour got me thinking about the other costs at large. Inference points aren't that common, still, but GPUs can be booked anywhere. So I checked if I could do the same manual price comparison there and upgraded my layer for GPU spot pricing.
The GPU spot marketplace. 37 GPU types across 9 providers, sorted by price. Top picks: H200 at $0.60/hr, H100 at $1.30/hr. Biggest savings: V100 at $0.07/hr.
The spreads are just as big as with inference pricing - if not even larger. An A100 80GB ranges from $0.71/hr to $2.48/hr right now, a 3.5x spread. H100s go from $1.30/hr to $7.38/hr. V100s range from $0.07/hr to $0.95/hr, a 13x spread.
Each GPU shows the price range across providers, availability status, and number of competing offers. Deploy in one click.
The marketplace tracks various gpu types across a growing list of providers right now. Per-second billing, auto-shutdown on idle, and you can set a max price so you don't get surprised.
So where's the downside?
Well, I can only think of one: Routing adds latency. Not much, but perhaps not something for everybody. However, for most interactive sessions it is probably negligible.
Also: We're open-weight only. No GPT, Claude, or Gemini. But probably okay - realistically, it is what you probably rely on for most of your actual applications anyway.
But apart from that, there is another upside - provider outages work differently with a router. If your single provider goes down, you're down. With a router, you're only down if all providers go down at the same time, which is much rarer. That turned out to be a nice side effect.
Why This Market Looks Like Early Cloud Computing
What I find genuinely interesting about all of this is that inference pricing in 2026 looks a lot like cloud computing did in 2010. Dozens of providers, no price transparency, wildly different rates for identical hardware. The only reason providers can charge 2-8x more than their competitors is because nobody is comparing in real-time.
Spot pricing for GPUs follows the same dynamics as spot instances on AWS. Providers would rather sell unused capacity at a discount than let it sit idle. But unlike AWS, where everyone knows the spot market exists, most people renting GPUs for ML workloads are still paying on-demand rates from a single provider they picked months ago.
I keep expecting the spreads to compress as the market matures, but so far the opposite has happened. New providers enter, existing ones experiment with pricing, and the range just gets wider. Maybe it'll consolidate eventually. But as long as providers are competing for utilization on hardware that depreciates fast, there's going to be a gap between whoever's desperate to fill capacity right now and whoever isn't.
If you want to check out what I built from all of this, give it a go at aivory.net/smart-inference and reach out to me for feedback.
Top comments (1)
great article, thanks for sharing!