Insights · Enterprise AI · Governance
The runaway meter: enterprise AI cost, security and quality are the same problem
A field note on why the AI bill you approved a year ago and the AI bill you’re actually running today diverged by a factor of four — and how to close the gap without lobotomising the applications you just built.
Profecia Links Editorial
14 minute read
Enterprise AI · GCC & Global
May 2026
A large enterprise we started working with earlier this year opened its first quarterly review with a slide that had one number on it: the AI vendor bill for the quarter. It was 4.2 times the approved annual budget for AI — for the quarter. The CIO’s exact words were “the meter has been running for a year and nobody knows what’s on it.”
We have now had a variation of this conversation with enough enterprises that the specifics have started to blur. The number is different each time. The shape of the problem is not.
The public conversation about enterprise AI is still dominated by capability — which model is better at which benchmark, which vendor announced which feature. The private conversation inside enterprises has shifted a year ago and is now almost entirely about administration. How do we control what we’re spending. How do we know what data is going to which vendor. How do we bring quality and cost into the same frame instead of trading one for the other. This piece is about that private conversation.
~90%
Fall in per-token frontier-model pricing since 2024
4×
Typical enterprise AI overrun vs approved budget
30–50%
Savings typically achievable through gateway architecture alone
The three shapes of AI cost overrun
We tend to see enterprise AI cost problems in three overlapping forms, each with a different root cause and each requiring a different response. Being explicit about which shape a given firm is dealing with, before doing anything else, tends to save more money than any specific optimisation. Most firms have some of all three; the mix is what matters.
SHAPE 01 · SPRAWL
Twelve teams. Twelve vendors. No aggregated leverage.
Product engineering signed up for one vendor. Data science signed up for another. Marketing built something on a third. Customer support is using a wrapper product that resells a fourth. Each team has its own contract, its own credit-card-paid API key, and its own definition of “success.” Legal doesn’t know which vendors are processing which data. Finance sees the AWS bill and the Anthropic bill and can’t reconcile either to a business outcome. The single biggest lever here is not optimisation — it is consolidation and negotiation.
SHAPE 02 · AMPLIFICATION
A proof-of-concept quietly promoted to production traffic.
An application that started as a demo has been rolled out to real users. It was engineered to work, not to work efficiently. Every user query still triggers a 40-chunk retrieval where five would do; the entire employee handbook is still being concatenated as context; the largest available model is still being used because in the original demo it was marginally better. At ten thousand daily active users, none of these decisions look like the same decision they looked like at ten.
SHAPE 03 · WRONG-MODEL ROUTING
The Michelin kitchen serving breakfast cereal.
The most subtle and often the largest. Someone made a defensible early decision — “let’s use the frontier model for everything and switch to cheaper ones later if it becomes an issue” — and later never came. Simple classification tasks run through reasoning-optimised models with ten thousand tokens of thinking trace. Summarisation runs through the highest-tier variant. The organisation has been paying a Michelin restaurant to serve breakfast cereal, and quality would arguably improve if it stopped.
Diagnosing the mix, and pricing the fix for each shape separately, is usually the first useful piece of work. It reframes an anxious general conversation about “AI is expensive” into a tractable set of specific ones.
Token economics — the parts that actually add up
The public claim that “AI is getting cheaper” is technically true and practically misleading. Per-token pricing has fallen roughly ninety percent in eighteen months across every major frontier lab. Aggregate enterprise spend has not — it has multiplied several times over — because usage has grown far faster than unit price has dropped. To have a useful cost conversation, you need to see the four levers that actually move the number.
Input tokens are not created equal. Every frontier vendor now discounts prompt caching — keeping recently-used context warm for you — by around ninety percent against the standard input rate. A production application that sends a four-thousand-token system prompt with every request but never caches it is paying full freight for the same tokens ten thousand times a day. A production application that caches them is paying full freight once and the cache rate thereafter. The difference is usually a code change of a few dozen lines.
Output tokens are where reasoning models change the game. A traditional response is two hundred to eight hundred output tokens. A reasoning model’s response, with its thinking trace, can be five thousand to twenty thousand — priced at output rates, which are two to five times input rates. This is not a criticism of reasoning models; they solve problems the previous generation could not. It is a warning against reaching for them for problems the previous generation solved perfectly well.
Batch pricing is available and mostly unused. Every major frontier lab offers roughly fifty percent off for non-realtime workloads processed asynchronously, usually within twenty-four hours. Overnight reports, backlog processing, model-based data enrichment, evaluation runs — none of these need synchronous latency. Very few enterprises use batch pricing systematically. The saving is large and the engineering cost is low.
Context length is a cost lever, not just a capability lever. The industry has trained us to think about 200K and 1M-token context windows as features. From a cost perspective they are a fixed multiplier per query. If your application routinely fills 150K tokens of context to answer a question that could be answered from 5K, you are paying thirty times what you need to for every single request. The right context strategy — hybrid retrieval, aggressive summarisation of prior turns, structured memory — is the single largest cost lever most enterprise applications have.
The gateway is the answer to most of the questions
If we had to name the single architectural pattern that separates enterprises with AI cost control from enterprises without it, it would be the internal LLM gateway. The gateway sits between every internal application and every external model provider. Every request routes through it. Nothing calls OpenAI, Anthropic, or Google directly. It is not glamorous infrastructure. It is the piece of plumbing that makes every other piece of AI governance possible.
A working gateway does five things:
It routes. Not just “send this to Anthropic” but “send simple classification to a small model, send complex reasoning to a frontier model, send anything involving customer PII to the on-premise deployment.” Routing policy lives in one place, is versioned, and can be changed without touching application code.
It attributes. Every request is tagged with the team, the application, the use case, the data classification, and the user. When Finance asks who spent what, the answer is a query, not an investigation.
It caches. Semantic caching for similar queries, prompt caching against the frontier vendors, and result caching where it makes sense. The savings are often thirty to fifty percent of production traffic cost with no perceptible impact on quality.
It enforces. Data classification tags determine which models a request is allowed to hit. Rate limits per team, per application, per user. Kill switches for runaway loops. Model allow-lists that change without a code deploy.
It observes. Structured logs of every request — latency, tokens in, tokens out, cost, model, cache hit, error, retry — flowing into your existing observability stack. If you cannot tell us the cost of your ten most expensive user journeys, you do not yet have a gateway; you have a collection of API keys.
Building this is not exotic. Off-the-shelf options exist and are improving quickly — LiteLLM, Portkey, LangSmith, and cloud-native offerings from Anthropic’s Bedrock integration, Azure AI Foundry, and Google Vertex. Selecting and configuring the right one for a specific regulatory context and existing observability stack is where our work usually begins. In several GCC deployments the answer has not been an off-the-shelf gateway at all — it has been a purpose-built one, because regulated data cannot leave the country and the international SaaS options are structurally unable to meet residency requirements. That constraint tends to appear later in an engagement than clients expect, and it changes the design significantly.
Security is not a separate track
Every conversation we have about AI cost turns, three or four sessions in, into a conversation about AI security. The two are not adjacent problems. They are the same problem seen from different sides.
Cost governance requires knowing which application sent which request to which model, tagged with which data classification. Security governance requires the same information. The gateway that enables one enables the other. The team asking “why did our spend triple?” is the same team, using the same data, as the team asking “did any of our customer PII leak into a third-party model’s training set?”. Enterprises that treat these as two workstreams end up building two gateways, two log stores, and two governance committees. Enterprises that treat them as one build one of each.
We tend to think about enterprise AI security as three concentric rings.
The data boundary. Before any request leaves your infrastructure, what is in it? Data classification tags at the source, PII detection at the gateway, redaction where appropriate, hard blocks where not. This is where the risk of “we accidentally sent our M&A pipeline to a public model” is contained — not by asking developers to be careful.
The vendor boundary. Which vendors process which classifications of data, under which contracts, with which data residency guarantees, and with which explicit assurance that inputs and outputs are excluded from training? This is negotiated and enforced, not assumed. For regulated data in the GCC — banking, government, healthcare — the vendor boundary often ends at the country line, which is why sovereign and on-premise deployments matter more here than they do in many other markets.
The model boundary. Prompt injection defences, output filtering, jailbreak monitoring, and the governance to decide what “acceptable output” means for your context. This is the newest of the three and the one where practice is still catching up to threat. It is also the ring where AI-specific expertise matters most; the previous two are recognisable extensions of familiar security disciplines.
The unspoken part
Most enterprises deploying AI at scale have written answers to none of these three questions.
Not because the questions are unclear — they are quite clear — but because AI adoption has moved faster than governance capacity. The gap between deployed AI and governed AI is currently, in our experience, the single largest un-managed operational risk in most large enterprises’ technology estates.
Closing it is neither expensive nor slow. But it does not close itself, and the vendors have no incentive to help you close it.
Monitoring — the smallest useful set of metrics
Enterprise observability teams tend to have strong opinions about metrics, and we usually do not try to add to them. What we do insist on is that the AI stack produce, at minimum, three tiers of data.
Per-request: timestamp, application, team, user, use case, data classification, model, tokens in, tokens out, cache hit or miss, latency, cost, error code if any. This is the raw material for every subsequent question.
Per-application: requests per day, cost per day, cost per user, cost per successful outcome, and at least one quality signal — accuracy on a held-out set, refusal rate, escalation rate, or whatever proxy applies to the specific use case. Quality without cost is negligent; cost without quality is meaningless.
Per-team: aggregate spend, month-over-month trend, cost per business outcome delivered, distance to approved budget. This is what Finance actually wants, and it is what turns AI from an unbounded utility bill into a manageable line item.
With these three tiers, the conversation shifts. Finance stops asking “why is AI so expensive?” and starts asking “why is the customer support application’s cost per resolved ticket rising while ticket volume is flat?” That is a productive question. It leads to a specific answer. The general question does not.
Quality without lobotomisation
The instinct once cost visibility arrives is to strip everything back — cheaper model, shorter context, lower reasoning budget. This works until it does not, and the point at which it fails is usually the point at which the business value evaporates.
The right question is not “how do we make this cheaper” in isolation. It is “for this specific use case, what is the cost-quality frontier, and where on that frontier are we currently operating?” Sometimes the answer is that we are paying for far more model than the task needs. Sometimes it is that we are already at the right point, and the way to save money is to reduce request volume through better product design. Sometimes it is that we are underpaying for the task and the cheap model is producing outputs the business cannot trust — and the “cost saving” is a quality loss disguised as a saving. All three are common, and they cannot be diagnosed without evaluation infrastructure.
The discipline that makes this manageable is having a real evaluation framework — a set of representative examples per use case, a scoring rubric that matches how the business actually measures success, and an ability to swap models or prompts and see the delta before it hits production. Enterprises that have this can change vendors, upgrade models, and consolidate on new pricing tiers quickly and confidently. Enterprises that do not are trapped between “we’re overspending” and “we’re afraid to change anything”, and the trap tends to get tighter as the vendor landscape shifts underneath them.
The administration model that actually works
Almost every mid-to-large enterprise we have worked with ends up in some version of the same structure: a small central platform team that owns the gateway, the model contracts, the observability stack, and the cross-cutting policies; and a much larger set of product and application teams that build against a stable internal interface.
The central team is not a bottleneck. It is a platform. It removes the concerns that no product team wants to own — vendor management, cost allocation, PII policy, model selection — and replaces them with a well-documented internal API. Product teams get faster because they stop having to think about the layer beneath them. Finance gets clarity because there is now a single point of truth for spend. Legal and Risk sleep better because there is now a single point of enforcement for data policy. Nothing about this is exotic; it is the same organisational pattern that emerged for cloud infrastructure in the 2010s and for data platforms in the late 2010s. AI is the next platform to go through this maturation. The enterprises adopting the structure earlier will not necessarily build better AI applications. They will operate them at materially different unit economics, and with materially better sleep.
“The gateway is not a product decision. It is the piece of infrastructure that determines whether every subsequent AI decision your enterprise makes is governable or unmeasurable. Everything else is downstream of it.”
Where we tend to be useful
We do not lead these conversations with a product pitch, because the honest answer to most of these problems is not a product — it is an operating model. What we bring, when we come into an engagement, is fifteen years of experience integrating systems that were never designed to talk to each other, which is exactly the problem enterprise AI presents at a different scale.
The specific things we are typically brought in to do fall into four categories. Diagnostic engagements to characterise the shape of a firm’s AI cost overrun and produce a prioritised optimisation plan with sized savings, usually across two to four weeks. Gateway architecture and rollout, using off-the-shelf components where they fit and custom builds where the regulatory environment demands it — a scenario we see disproportionately often in the GCC, where residency and sovereignty requirements rule out most SaaS options. Governance model design — the organisational shape of who owns the gateway, who owns model selection, who signs off on new use cases, how cost is allocated back to teams; this is where technology programmes most often fail, and where our consulting experience across ERP, CRM, and enterprise data platforms transfers most directly. And, for clients who prefer it, sustained operation of the platform under a managed-service arrangement rather than a build-and-hand-back.
We have deployed on-premise LLM infrastructure into environments where cloud AI is structurally not an option — including our own Enterprise.AI reference architecture, built around sovereign model deployment for regulated industries. That work is deliberately unglamorous; it is the reason the gateway conversation tends to be easier when we are the ones having it, because we have already been through the constraints that matter.
A closing observation
Enterprise AI is now at approximately the moment cloud computing was at in 2013. Every large firm has proven the value. Almost none has proven the operating model. The next several years will separate the enterprises that treat AI as a governed platform from those who treat it as a series of experiments with a growing invoice, and the gap between the two — in cost, in security posture, and in speed to production — will compound in the way infrastructure gaps always do.
The good news is that most of the machinery required to be on the right side of that gap is unglamorous, well-understood, and buildable in months, not years. The gateway. The evaluation harness. The data-classification tagging. The routing policy. The observability tiers. None of these are frontier work. All of them can be in place inside a quarter.
The less good news is that the invoices will continue to arrive regardless of whether the machinery is in place. The interesting question, for any large enterprise reading this, is not whether the conversation is worth having. It is whether the conversation is worth having with someone who has already had it several times before.
Work with us
The AI bill you approved. The AI bill you’re actually running.
If the gap between the two is a conversation you have been meaning to have — on cost, on security, on how the whole estate is administered — we would be glad to have it with you.
Top comments (0)