Most small teams shipping an AI feature assume the same thing at roughly the same point: once a real client's data is going through the model, it's time to own the infrastructure. Rent a GPU, self-host the model, put "dedicated" in the sales deck.
It feels like the responsible move. It's also, for most of us, the wrong one, for longer than you'd expect.
We found this out by trying to price it, having outside eyes stress-test the plan, and discovering our own first instinct was cost-plus thinking dressed up as an infrastructure strategy.
Why the intuition breaks down
The instinct comes from a fair place: a dedicated GPU sounds like control, and control sounds like the right thing to sell to a buyer who's nervous about data residency or vendor lock-in.
The problem is volume. A GPU has a fixed cost whether it processes one question or ten thousand. A boutique AI product, especially early on, does not generate enough steady-state traffic to keep that box busy. When we modeled our actual usage, not a hypothetical enterprise rollout but real questions per day across the clients we actually have, the GPU sat idle more than 90% of the time.
That's not a rounding error. That's the box mostly doing nothing while the invoice still arrives.
The real cost curve
The mistake was measuring the wrong cost. We were comparing infrastructure spend to infrastructure spend (GPU rental vs. managed API calls) and concluding the GPU looked cheap at scale. It does, eventually. But "eventually" is the part that matters.
Two things are true at once for an early-stage AI product:
- The GPU has a breakeven client count. Below it, you're paying for idle capacity. Above it, the per-question cost drops below what a managed frontier model charges per token. That crossover, in our case, sits somewhere around a low double-digit number of concurrent active clients, not one, not three.
- The dominant cost below that line isn't infrastructure at all, it's people. Someone has to patch the box, watch it, restart it, and answer for it at 2am. That's a real, recurring, human cost that scales roughly with the number of things you operate, independent of how much traffic any one of them sees.
We'd been measuring the cost of silicon and ignoring the cost of the person keeping the silicon alive. Once we counted both, the "cheap at scale" GPU stopped looking cheap for a team with a handful of clients.
A gateway pattern that defers the decision
The fix wasn't to abandon the idea of owning hardware someday. It was to stop needing to decide on day one.
We put a routing gateway in front of every model call. By default, every client's traffic goes to a managed, frontier-grade model, billed per token, run in a data-resident region, and absorbed into a flat fee so the client never sees a token count. Owning a GPU becomes a config change you flip later, once telemetry says you've crossed the breakeven line, not an architecture decision you have to get right upfront.
A simplified version of that routing config looks like this:
# model-routing.yaml
default_route:
provider: managed-frontier
region: data-resident
billing: per-token # absorbed into client's flat fee, never surfaced
ops_overhead: none # no boxes to patch
premium_route:
provider: dedicated-gpu
activation:
trigger: concurrent_active_clients >= BREAKEVEN_THRESHOLD
action: flip_config # not a re-architecture
ops_overhead: on-call rotation required
The gateway is what makes this reversible. Nothing about the client-facing product changes when you flip the route; only the backend that's actually answering the call does.
Two honest tiers
We ended up with two tiers we can say out loud without overselling either one:
| Tier | What it actually is | Ops burden | When it makes sense |
|---|---|---|---|
| Base ("logically isolated") | Isolated tenant data, shared managed inference in-region | Near zero | Default, scales down to one client without losing money |
| Premium ("dedicated") | A GPU that's actually 1:1 for one client | Real, ongoing (on-call, patching, capacity planning) | Once volume or a hard compliance requirement justifies the fixed cost |
The important discipline here is reserving the word "dedicated" for the tier that's actually dedicated. Calling shared managed inference "dedicated" because it sounds better in a sales call is a promise you can't keep, and it's the kind of thing that comes back to bite you the first time a client asks what "dedicated" actually means.
What changed when we ran the numbers
A few things surprised us enough to change the plan outright:
- The idle ratio was worse than we guessed. We expected the GPU to be underused early on. We didn't expect "more than 90% idle" until we actually modeled real daily question volume against server capacity.
- The breakeven point was a client count, not a revenue number. It's tempting to think "once we're making enough money, buy the GPU." The real trigger is concurrency: how many clients are actually hitting the model at once, not how much any of them are paying.
- The cheapest-sounding option isn't always the one you're allowed to pick. Managed, frontier, in-region inference is the cheapest and lowest-ops path on paper. For a client with a strict sovereignty requirement, that path can be a non-starter regardless of cost, because the provider underneath it is still a foreign company subject to foreign law. That's a decision to make per client, before quoting a price, not after.
Honest limitations
This isn't a universal argument against ever owning infrastructure. A few caveats:
- The breakeven threshold is workload-specific. Ours came from our own usage pattern. A product with heavier per-request context (long documents, large retrieval windows) will hit the crossover at a different client count than a product with short, simple queries.
- "Zero ops" is relative, not absolute. Managed inference still needs monitoring, fallback routing, and a real incident plan. It's cheaper than running a GPU, not free.
- This model assumes you can walk away from a provider. If your gateway locks you into one managed backend with no realistic second option, you've just moved the vendor risk instead of removing it. Build the gateway so switching providers is also just a config change.
Getting started today
You don't need the full gateway to start protecting yourself from this mistake. Three steps:
- Model your actual usage, not a hypothetical enterprise fleet, before pricing any infrastructure decision.
- Separate the cost of the box from the cost of the person operating it. Most of the "GPU is cheap at scale" arguments quietly drop the second number.
- Put a routing layer in front of your model calls now, even a thin one, so the GPU-vs-managed decision stays reversible instead of load-bearing.
The instinct to look serious by owning the hardware is a real cost, paid upfront, on a scale you might not have yet. Pricing on what the client is actually getting, and only owning what your current volume can justify, is the boring option. It also happens to be the one that's still standing when the impressive-looking competitor's infrastructure bill catches up with them.
If you're wrestling with the same build-vs-managed decision for an AI feature, I'd genuinely like to hear how you modeled your breakeven, drop it in the comments.
Top comments (0)