A spend cap that kills a production workload to save money is not governance. It is an outage you scheduled and forgot about.
Somebody sets a monthly budget on the resource group. Traffic spikes on a Tuesday. The cap trips, the Azure OpenAI deployment stops answering, and now your on-call engineer is explaining to a product owner why checkout summaries returned 429s during a promo. You saved a few hundred dollars and bought yourself an incident.
That is the state of most ai cost governance azure practices I see: an alert threshold bolted onto an invoice that already spiked. It tells you the disease exists. It does nothing to treat it.
Spend caps are a failure signal, not a strategy
Overspend is the symptom. The disease is architecturally unaware token consumption: every request routed to a frontier model, system prompts nobody has read in six months, RAG context that grows without bound, and no way to say which feature spent what.
A budget alert fires after the damage. By the time Azure Cost Management emails you, the tokens are burned and the invoice is committed. You are looking at a photograph of a fire.
This is why cost belongs to architects, not to a finance dashboard owned by a separate FinOps team. The people who decide which model handles which request, how prompts are structured, and whether you buy provisioned capacity are the people who decide the bill. Finance can report the number. Only engineering can change it.
If your first cost control is an alert threshold, you have monitoring. You do not have governance.
Make tokens a first-class observability metric
You cannot govern what you cannot attribute. And here is the structural problem most teams discover too late: Azure's native metrics cannot attribute cost to the thing that matters.
Azure OpenAI emits metrics like ProcessedPromptTokens and TotalTokens, and it dimensions them by model name, version, and API name. It never dimensions by tenant, by feature, or by user flow. Per the Azure Monitor logs ingestion documentation, custom telemetry has to come from you. This is not a portal setting you missed. The resource-level metric schema architecturally cannot answer "which feature spent this money."
So you build the pipeline. Wrap the SDK call site. Capture the usage object the API already returns on every completion, attach your own dimensions (feature ID, tenant ID, flow name, model ID), and ship it to a Log Analytics custom table through the Logs Ingestion API and a data collection rule. If you front your deployments with API Management, extract the same fields in an APIM policy at the gateway instead.
The Azure OpenAI usage metrics in the portal aggregate at the resource level. Per-feature and per-tenant attribution requires your own instrumentation in the app layer. Every governance decision downstream of this - routing thresholds, prompt trimming, PTU sizing - is a guess until you build it.
Once tokens are attributed, cost-per-request, cost-per-feature, and cost-per-tenant become real numbers you can query. That is the spine of ai cost management azure done properly. We go deeper on the pipeline mechanics in instrumenting Azure Monitor for AI workloads.
Tag every token to the flow that spent it, or you are optimizing blind.
Intelligent model routing is the core lever
Stop defaulting every call to a frontier model. Most production LLM traffic is not hard. Classification, extraction, short summarization, and routing decisions do not need a GPT-4-class model, and paying frontier token prices for them is where the largest share of avoidable spend hides.
Build a routing layer. Score incoming requests for complexity, send the easy ones to a small, cheap model, and reserve the expensive model for genuinely hard tasks. Add a cascade: if the small model's answer fails a confidence or validation check, escalate to the larger one.
Here is the model economics, drawn from the public pricing page. Treat these as tiers, not fixed numbers.
| Model tier | Relative token price | Best-fit workload |
|---|---|---|
| Small / mini (e.g. GPT-4o mini class) | Lowest | Classification, extraction, routing, short deterministic tasks |
| Mid (e.g. GPT-4o class) | Moderate | General reasoning, summarization, most RAG answers |
| Frontier / reasoning (e.g. o-series class) | Highest | Multi-step reasoning, hard code, genuinely ambiguous tasks |
Confirm every figure against the Azure OpenAI Service pricing page, because per-model token pricing changes and varies by region.
Now the honest part. Routing is often quoted as cutting 60 to 80 percent of LLM cost. That is an illustrative range using industry-standard inputs; calibrate against your own traffic and actuals vary. The reason it varies is the cascade tax: an escalated request pays for the small model call and the frontier call, and their latencies add rather than max. Your net savings depend entirely on your escalation rate, and your escalation rate depends on how well you calibrated the complexity threshold against a labeled eval set. There is no Microsoft-published threshold that works for your workload.
So measure net cost, not per-token price. A cascade that escalates 70 percent of requests costs more than sending everything to the frontier model directly.
This is azure llm cost optimization at its highest leverage. Design the routing layer before you tune a single prompt.
Engineer the prompt and context budget
After routing, the next pool of waste is token bloat inside each call. It hides in three places: system prompts that grew by accretion, RAG context that retrieves twenty chunks when three would answer, and identical prefixes re-sent on every request with no caching.
Fix all three. Use Azure OpenAI prompt caching to discount the static portion of your prompt. Trim context by reranking retrieved chunks and keeping only what clears a relevance bar. Ask for structured JSON output so the model stops emitting prose you immediately parse away.
Prompt caching only helps when the static prefix is large and stable. Dynamic-first prompts get no discount. Order your prompt with the cacheable content up front - system instructions and fixed context first, the variable user turn last - or the cache does nothing for you.
The instinct is to cap the context window at some token count you guessed. Do not. Token usage optimization azure works when you trim against retrieval relevance, not against a fixed number. If three chunks answer the question, sending eight is waste regardless of whether eight fits your budget. If a query genuinely needs ten, a hard cap of five degrades the answer and generates a retry, which costs more than the tokens you saved.
Trim the context window against retrieval relevance, not against a fixed token count you guessed.
PTU vs. PAYG: govern the consumption model itself
The largest single cost decision on an Azure OpenAI workload is not a prompt. It is procurement.
You have two consumption models. Pay-as-you-go bills per token with no commitment. Provisioned Throughput Units, described in the PTU documentation, reserve dedicated capacity for a fixed price over a term. Under sustained, high, predictable load, PTU can be cheaper per token and gives you latency you control. Under spiky or early-stage traffic, PAYG wins because you pay for nothing when idle.
There is no stable numeric crossover between them. Minimum PTU increments, tokens-per-PTU ratios, and discount percentages all shift with pricing and vary by model and region. Anyone quoting you a fixed break-even token count is quoting a number that expired. So do not memorize a threshold. Learn the method.
Right-size against measured traffic percentiles, not peak guesses. Pull your real per-minute token throughput from the attribution pipeline you built earlier, look at the sustained load rather than the once-a-week spike, and size PTU to the level your traffic actually holds. Run those numbers through the capacity calculator in Azure OpenAI Studio against your own data.
PTU capacity is region and model-version specific and is not guaranteed on demand. Capacity is a constraint, not only a cost. Do not architect a commitment you cannot provision - confirm availability for your exact model version and region before you design around it.
That is azure openai cost control at the procurement layer: pick the consumption model from your traffic distribution, then optimize prompts inside it. Reserving capacity for traffic you do not have burns money faster than any oversized prompt.
Bake governance into the CI/CD path
Everything above is a design decision. A design decision that nothing enforces drifts back to defaults within two sprints. Someone adds a feature, points it at the frontier model "just to ship," skips the routing layer, and doubles a prompt. Nobody notices until the invoice.
So enforce it before deploy. Treat token cost like any other regression.
Azure Policy is the right tool for the allowlist and tagging enforcement; the Azure Policy documentation covers the deny and audit effects you need. This is policy-as-code, and your architects already know how to write it.
Cost discipline enforced before deploy is governance. Discovered on the invoice, it is regret.
The core position
Governance is not an alert you add after the bill spikes. It is an architecture you commit at design time, and it has five load-bearing parts: attribute every token in your own telemetry, route requests to the cheapest model that can answer them, budget context against relevance, pick PTU or PAYG from your measured traffic distribution, and enforce all of it in CI/CD before anything ships.
Skip the attribution layer and the other four become guesswork, because Azure's native metrics structurally cannot tell you which feature spent the money. That is the part vendors selling you "FinOps for AI" gloss over, and it is the part that decides whether everything downstream is engineering or theater. It is also why we treat this as an architecture problem in how we think about AI governance on Azure.
Here is your concrete next action. This week, pick one production endpoint. Wrap its SDK call to emit the usage object with a feature tag into Log Analytics, and write one build assertion that fails when its cost-per-request exceeds a number you set today. One endpoint, one budget, one gate. That is the difference between watching the invoice and governing it.
This article was originally published at az365.ai. I'm Alex Pechenizkiy, an Azure and Power Platform solutions architect writing honest, vendor-neutral analysis of the Microsoft AI stack. More at az365.ai.
Top comments (0)