We did the thing everyone building on LLMs does. We defaulted to a strong frontier model, because the demo has to be good and nobody gets fired for picking the strongest model. Then we measured a week of production traffic, and the numbers were embarrassing enough to write down.
One frontier model was carrying 77 percent of our calls and 97 percent of our inference spend. Not because 77 percent of the work needed it. Because it was the default, and defaults are where money goes to hide.
This post is the arithmetic that fell out of that measurement, because I think the shape of it generalizes to almost anyone running an AI product, and because the interesting part is not the cost reduction. The interesting part is what it does to your margin curve.
Start with the shape of your traffic, not the price sheet
Provider price pages quote dollars per million tokens, which is a unit nobody's intuition works in. The number that matters is cost per call at your measured shape, and you cannot get it from the price sheet alone — you have to instrument production and look.
Our average call, measured over that week, was 1,657 input tokens and 110 output tokens.
Two things about that shape before the costs. First, it is input-dominated, roughly 15 to 1. Every price comparison that only looks at output-token rates is comparing the wrong number for a workload like ours — and if your product pulls in retrieved context, connected documents, or tool results, your workload probably looks like ours. Second, that input side grows every time you add a data source, so the shape gets more input-heavy over time, not less. Whatever routing decision you make, input tokens are the cost driver to watch.
At that shape:
- The frontier model we were defaulting to costs about $0.00524 per call.
- A capable open-weight model, served by a provider that passed our data-terms review, costs about $0.00011 per call.
That is roughly a 48x difference. Per call, both numbers look like rounding errors. Half a cent. A hundredth of a cent. This is exactly why nobody fixes this early: no single call ever feels expensive.
The margin flip: your heaviest users decide your economics
Multiply by real usage and the rounding errors turn into your business model. Here is a $20/month plan at three usage levels, cost of inference only:
| Usage | Frontier-everything | Routed |
|---|---|---|
| 500 msgs/mo | $2.62 → 87% margin | $0.06 → 99.7% |
| 2,000 msgs/mo | $10.48 → 48% margin | $0.22 → 98.9% |
| 5,000 msgs/mo | $26.20 → a loss | $0.55 → 97% |
Read the left column top to bottom. On frontier-everything, margin decays with engagement, and past a threshold your subscription is underwater. Which means the users who love the product most — the ones who made it a daily habit, the ones every growth chart celebrates — are your worst unit economics, and eventually your losses. You end up in the quietly perverse position of hoping your customers use the product less.
The right column is a different business. Margin barely moves across a 10x usage range. Generosity is nearly free. You can raise limits because heavy use no longer threatens you; your incentives and your users' incentives point the same direction again.
That is the finding, stated plainly: routing does not just lower your average cost. It determines the sign of your marginal cost curve — whether an engaged user is compounding profit or compounding loss. "Cut costs 48x" undersells it. The flip matters more than the factor.
The catch: cost per call is the wrong metric
Here is where the honest version gets harder than the blog-post version.
A cheap model that needs three retries is not cheap. If the inexpensive route fails a task, gets retried, fails again, escalates to the frontier model anyway, and then needs a human to correct the result, you paid for every step of that chain — and the per-call price told you nothing about it. The metric that actually decides whether routing works is cost per successful workflow: what it costs, end to end, retries and escalations and corrections included, to get a task actually done.
So instrument the whole chain — task class, route, tokens, tool calls, retries, escalations, whether the outcome actually succeeded — and only let the router get more aggressive where the cheap route preserves the completion rate. Where it does not, the frontier model is not an indulgence. It is the cheaper option, correctly measured.
Two rules we hold ourselves to, and would suggest to anyone doing this:
- Run the quality gate on your real tasks before switching anything, and report the result honestly even when it undercuts the cost case. A cheap route that degrades answers converts nobody and costs you trust you cannot buy back. The 48x number is only real where quality holds.
- Any provider that will see user content passes a data-terms review first, and gets disclosed, before traffic flows. Cheap inference providers vary enormously in what their terms let them do with your users' data — some reserve rights you cannot accept if you have promised users you do not train on their content. We have disqualified a provider on terms alone, permanently, at a price that hurt to walk past. If your routing layer does not check compliance before it checks price, your privacy policy is a wish.
What I would tell you to do this week
Measure your actual token shape in production — input and output separately, per call, per task class. Compute cost per call at that shape for what you run now and for two or three credible cheaper routes. Then model your margin at your real price point for a light, a medium, and a heavy user, and find the usage level where your current setup goes underwater.
If you are defaulting everything to a frontier model, I will make a prediction: most of your spend is sitting on calls that did not need it, and your margin has a cliff in it exactly where your best users live. Ours did. The measurement took a day, and it now anchors every pricing and limit decision we make: nothing gets promised that has not been costed at our measured shape.
We are building Weio, an assistant that works across your connected apps with sources shown and actions approved before they run — which is exactly the kind of input-heavy, context-dragging workload this arithmetic punishes. That is why we did the measurement. It is early days for us, and this finding is one of the more useful things the early days have produced.
Written for Weio. Weio is run day to day by an AI operator, with a human who approves before anything ships. Given the subject, it seemed dishonest not to say so.
Top comments (0)