A founder showed me a dashboard last month with a metric I hadn't seen before: cost per completed task, the dollar amount it takes an agent to answer one real request, start to finish. Not total spend. Not cost per user. Cost per task. Once I understood why he tracked it that specifically, I couldn't stop noticing how many AI products don't.
Gross margin stops being a useful signal once compute enters the unit economics.
For most of software's history, 80% gross margin meant healthy, and 40% usually meant a services business dressed up as software, because the cost of serving one more user was close to zero. AI-native products are running closer to 52% gross margin on average now, against the 70-80% range that used to be the baseline for traditional SaaS. That's the shape of the whole category, a direct result of every response calling out to a model that costs real money to run, every single time.
The tell is that the margin number stops telling you anything on its own. Two products can report the same 55% gross margin, one well-optimized and improving, the other one growth spurt away from watching its economics worsen with scale. You can't see the difference without going one level deeper.
A single agentic task is rarely one model call. It's usually five to ten.
A chat response is one call in, one response out. An agent that plans, picks a tool, executes it, checks its own work, and then responds chains five calls together at minimum, and agents in general make three to ten times the model calls a simple chatbot does for what looks, from the outside, like the same request.
That compounding is why costs for agentic products rarely scale in a straight line with users. A team that budgets for its bill to track headcount gets blindsided when it tracks agent loops instead, growth that looks linear on a user chart can look exponential on an inference invoice. Chatbot-era intuition, where more users meant roughly proportional cost, doesn't prepare you for how quickly those extra calls compound.
Falling token prices raised the ceiling on usage.
The price of a single token has genuinely fallen. Anthropic's current Opus pricing sits at $5 per million input tokens, a 67% cut from the $15 per million tokens Opus cost two generations back. That's the kind of price drop that should shrink a bill. Instead, every agentic task multiplies that lower price by three to ten calls instead of one, so a cheaper token still buys a bigger invoice. Somewhere in that expansion sits a real threshold: below roughly 50 million tokens a month, a managed API is usually still the cheaper option; above 100 million, self-hosting starts winning on unit economics.
Most of that cost is recoverable through caching, a straightforward engineering fix.
Semantic caching, recognizing when a new request means roughly the same thing as one already answered, took one documented team from an 18% to a 67% cache hit rate and cut their bill by 73%. Prompt caching, reusing repeated context instead of recomputing it, prices cache reads at a tenth of a fresh input token, up to a 90% saving on that portion of the bill. Neither technique requires a bigger GPU budget. Both are ordinary engineering discipline applied to a cost category that used to get ignored until the invoice arrived.
I ask for this number now because it predicts more than a margin ever did.
I still spend most of my time on product and market, same as always. But I've added one question to every technical conversation: what does it cost you to finish one task, end to end, right now? The teams that can answer without opening a spreadsheet are usually the same ones who saw the cost cliff coming before it hit them.
Top comments (0)