An Overview of AI Gateways with Azure API Management
As organizations scale AI workloads, they hit practical limits: per-model throughput caps, token rate limits, subscription quotas, and a growing list of Azure OpenAI deployments spread across teams and regions. A single resource cannot absorb high-volume, multi-model, multi-team traffic. Moving beyond basic prompts to an enterprise-grade platform requires a robust architecture; for more on this transition, see
capestart.com
This article explains how to design a gateway that connects multiple Azure OpenAI accounts; exposes GPT-4o, GPT-4 Turbo, embeddings, and vision models through a unified contract; and enforces identity and quotas at the edge. It also clarifies what “unlimited-style” scaling actually means, and how routing, quota management, observability, multi-tenant governance, disaster recovery, and production readiness transform a proxy into a robust platform.
Why You Need an AI Gateway with Azure API Management
Growth issues rarely start with model quality; they stem from tokens-per-minute (TPM) limits, requests-per-minute (RPM) limits, and the operational burden of each application managing its own endpoint list. Azure OpenAI enforces TPM and RPM constraints per deployment alongside subscription-level ceilings. Furthermore, teams often require different models (such as chat, embeddings, vision, or RAG scoring) hosted across multiple regions. Without a centralized gateway, every application must maintain its own host list, access keys, and retry logic.
The pattern below is the usual breaking point. A GPT-4o deployment allows only its granted tokens per minute. Adding more deployments in the same subscription still collides with subscription-level throttling. A global product needs East US for one cohort and West Europe for another, and a finance team must not burn GPT-4o budget on summarization that GPT-4o-mini can do. APIM becomes the load balancer, policy engine, and traffic router so those constraints are solved once.
Microsoft documents the underlying caps in the Azure OpenAI quotas and limits guidance. The gateway does not remove those caps. It pools independent caps and decides, request by request, which remaining envelope to use. That distinction is the subject of Unlimited-Style Scaling.
Reference Architecture
Clients, including product services, analytics jobs, and RAG pipelines, call a unified entry point such as https://ai.company.com. Azure API Management acts as the single front door. Behind APIM sit named backends: Account A (East US with GPT-4o and GPT-4o-mini), Account B (West Europe with GPT-4 Turbo and vision models), and Account C (embeddings and fine-tuned models). Log Analytics and cost dashboards sit off the hot path, consuming the metadata stamped onto each request by the gateway.
How an AI Gateway with Azure API Management Selects a Backend
A request arrives on a single completions or embeddings URL. APIM authenticates the caller, applies tenant and product quotas, and estimates how many tokens the call will consume. The router then picks a healthy backend that hosts the requested model, has remaining TPM and RPM, and meets the latency and region policy for that tenant. The backend call uses the gateway’s managed identity. The client never learns which account answered, and the response path records actual token usage so the next decision is better than the last.
APIM is the right place for this because policies already cover choose, send-request, retry, rate-limit, and token-limit. Native load balancing is only a starting point. The sections that follow specify the signals the policy must read and where those signals live.
Unlimited-Style Scaling: More Capacity, Not Infinite Capacity
“Unlimited-style” scaling means something precise, and it does not mean unlimited capacity. Each Azure OpenAI resource keeps its own TPM, RPM, and any provisioned-throughput (PTU) allocation. Those ceilings are contractual and regional. Putting APIM in front of one resource does not raise them.
The gateway can raise the aggregate quota the platform can consume by placing several independent accounts, often across multiple regions, and sometimes mixing pay-as-you-go with PTU behind one URL. Each account has its own envelope. When traffic is distributed across those envelopes, usable capacity becomes the sum of remaining healthy backend quota, minus the headroom you keep for failover. That is a larger number than one account can offer. It is still a finite number.
Important: Spreading work increases available quota and reduces the chance that one resource’s 429s stall the estate. It does not create unlimited tokens, unlimited concurrency, or immunity from regional incidents. Capacity planning and tenant admission control remain mandatory.
A practical pool looks like three accounts whose limits add rather than replace each other. Account A in East US hosts GPT-4o and GPT-4o-mini and contributes one full TPM envelope. Account B in West Europe hosts GPT-4 Turbo and vision and contributes a second envelope. Account C hosts embeddings and RAG models and contributes a third. Adding Account D later increases the pool again. Removing or failing Account B shrinks it. The platform’s real capacity is always the live sum, not a marketing infinity.
Size the pool from observed P95 tokens per request and target RPS, then keep at least one backend’s worth of spare capacity so a regional event does not immediately exhaust the survivors. APIM unit count, Redis (if used for quota state), and DNS are separate ceilings. A team that adds OpenAI accounts without watching those layers will still 429. Intelligent Routing is how the pool is consumed; Disaster Recovery is what happens when part of the pool disappears.
Intelligent Routing Logic
Round-robin is a reasonable default when backends are identical and healthy. It is not sufficient for Azure OpenAI in production. TPM and RPM are consumed unevenly because prompts vary in size, tenants are bursty, models are not deployed in every region, and a backend that is “up” may already be at 95 percent of its token window. Blind rotation will send a large completion to a resource that is about to 429, retry the same resource, and amplify the failure.
The example below is the starting policy most teams write. It is useful as a sketch and incomplete as a production router. True rotation across requests also needs shared state; an APIM variable alone is per-gateway-node and resets.
<policies>
<inbound>
<set-variable name="route" value="@(context.Variables.GetValueOrDefault<int>("roundRobin", 0) % 3)" />
<choose>
<when condition="@((int)context.Variables["route"] == 0)">
<set-backend-service base-url="https://openai-account-a.openai.azure.com/" />
</when>
<when condition="@((int)context.Variables["route"] == 1)">
<set-backend-service base-url="https://openai-account-b.openai.azure.com/" />
</when>
<otherwise>
<set-backend-service base-url="https://openai-account-c.openai.azure.com/" />
</otherwise>
</choose>
</inbound>
</policies>
A production router evaluates a candidate set on every request. It uses APIM policies and a Redis-backed or function-backed decision, rather than placing logic inside every microservice. It first drops backends that do not host the requested model or API version, that sit outside the caller’s allowed regions, or that the health checker has marked failed. Remaining candidates are scored on current TPM and RPM utilization, in-flight concurrency, recent P95 latency, remaining quota for this window, and how many times this request has already been retried against that backend. Lowest healthy score wins. Ties can fall back to weighted round-robin so a newly recovered backend is not slammed.
Retry count is a first-class input. The first attempt prefers the lowest-latency in-region backend with spare TPM. A 429 or 5xx retries to a different account, preferably in a different region, and never to a backend already tried for this request unless no alternative exists. Circuit breakers open when 429 rate, 5xx rate, probe failures, or a latency SLO breach crosses a threshold. Concurrent request caps stop a single slow model from occupying every APIM thread. Cost can be a tie-breaker: non-critical summarization prefers GPT-4o-mini when quality policy allows it.
Adaptive routing consults live utilization before set-backend-service:
<send-request mode="new" response-variable-name="quotaResponse" timeout="5" backend-id="quota-checker" />
<choose>
<when condition="@(int.Parse(((IResponse)context.Variables["quotaResponse"]).Body.As<string>()) < 80)">
<set-backend-service backend-id="aoai-eastus-a" />
</when>
<otherwise>
<set-backend-service backend-id="aoai-westeurope-b" />
</otherwise>
</choose>
Where the decision runs: APIM remains the execution point so callers keep one URL. Native APIM load balancing covers round-robin and priority. TPM-aware selection, retry-aware exclusion, and latency scoring read routing state from Redis or a private quota-checker API before set-backend-service. Clients do not implement backend lists.
Routing is only as good as the signals it reads. Those signals are owned by Quota Management and published to Observability. Failover when an entire region scores zero is defined in Disaster Recovery.
Quota Management
Quota is not a single counter. Token consumption, request rate, and backend capacity live in different layers, and the architecture is explicit about which layer is authoritative for which decision.
What is Tracked, and Where
Azure OpenAI is the source of truth for each resource’s TPM, RPM, and PTU. The service enforces those limits whether or not the gateway is well-behaved; a 429 from Azure OpenAI means that envelope is exhausted. APIM is the source of truth for product, subscription, and tenant admission control. Rate-limit, quota, and llm-token-limit policies protect noisy neighbors and implement commercial contracts. They do not replace Azure OpenAI’s own caps.
A private shared store, Azure Cache for Redis reachable only from APIM and the quota-checker, holds the near-real-time picture the router needs. For each backend, it stores estimated tokens consumed in the current sliding window, in-flight request count, last health-check result, consecutive failure count, and circuit state. For each tenant, it stores tokens and requests consumed in the current billing or throttle window. Writers are APIM outbound policies and a small ingestion function that reads prompt_tokens, completion_tokens, and total_tokens from the Azure OpenAI usage object immediately after the call. Readers are the inbound routing policy and the dashboards.
Azure Monitor and Log Analytics are the source of truth for history, not for the hot path. Every APIM request emits tenant, model, backend, token counts, latency, HTTP status, retry count, and whether a failover occurred. That telemetry builds burn-down charts and cost reports. It is not queried on the request path, because a gateway that waits on analytics cannot meet P99 targets.
How a Request Consumes Quota
On ingress, APIM identifies the tenant, checks tenant and application quotas in Redis and in APIM policies, and rejects the call with a structured 429 if the tenant is exhausted. If the tenant is allowed, the router estimates prompt tokens plus requested max_tokens, then selects a backend whose remaining TPM can absorb that reservation. The reservation is written to Redis before the backend call so concurrent requests cannot oversubscribe the same window. When the response returns, the reservation is replaced with actual usage. If the backend 429s, the reservation is released, and the retry path selects a different backend. If every backend is exhausted, APIM returns 429 with a Retry-After derived from the soonest window reset, rather than holding the client until a timeout.
This split keeps enforcement close to the metal (Azure OpenAI), admission control close to the product (APIM plus Redis), and reporting close to the business (observability and cost dashboards). Multi-tenant governance defines who may consume which envelope.
Model-Wise API Segregation
A single generic /openai/chat/completions surface works, but most platform teams also publish dedicated paths such as /gpt-4o/chat, /gpt-4-turbo/chat, and /embeddings/generate. The client keeps a stable, obvious URL. The multi-account pool stays behind that URL. APIM maps the path to the allowed model family, then the same scoring router chooses an account that actually hosts it.
Segregation is what makes tenancy and cost controls usable. Finance can receive a product that only permits GPT-4o-mini and embeddings, while a product-research product may call GPT-4o. Developers stop stuffing model names into every body. Operations still load-balance across Account A and Account B for GPT-4o without publishing either hostname. Path-based APIs and a generic completions API can coexist; the generic API simply requires a model field that the policy validates against the caller’s permission map.
Multi-Tenant Governance
APIM is the tenancy boundary. Applications do not receive Azure OpenAI keys and do not learn backend URLs. Each tenant is an APIM product with one or more subscriptions, and optionally a JWT from Microsoft Entra ID that carries tenant_id, environment, and allowed-model claims. Subscription key plus JWT is the baseline because it maps onto APIM products and analytics dimensions. Mutual TLS can be added where the threat model requires it.
Authentication is evaluated first. A missing or invalid Ocp-Apim-Subscription-Key never reaches a model. A valid key with a JWT whose tenant_id does not match the product is rejected. Authorization is a policy map: which models and deployments the tenant may call, which regions it may use, whether it may consume PTU capacity or only pay-as-you-go, and the maximum max_tokens it may request. Onboarding is a configuration change delivered through the same CI/CD path as the rest of the platform, not a new API per tenant.
Tenant quotas and rate limits sit in front of backend quotas. A tenant might be allowed 200,000 tokens per hour and 20 RPS even if the platform pool has more headroom. Those counters live in the Redis store and in APIM rate-limit-by-key policies keyed by tenant_id. When a tenant hits its cap, other tenants are unaffected. When the platform pool is exhausted, every tenant sees 429 with Retry-After, which is honest and preferable to random timeouts.
Cost attribution uses the same tenant_id, product, subscription, model, and backend dimensions on every APIM log and metric. Finance rebuilds a bill from Log Analytics without a side spreadsheet. Showback reports come from the cost panel in the operations workbook. No application is asked to self-report tokens; the gateway already saw the Azure OpenAI usage object.
Governance does not replace network isolation or backend authentication. Secrets and identities are handled in Security and Production Readiness.
Observability and Operations Dashboards
A gateway you cannot see is not operable. Telemetry is collected at APIM, at Redis, at the quota-checker, and at each Azure OpenAI account. Application Insights or OpenTelemetry stitches a request ID from the caller through APIM to the backend so a slow completion can be blamed on the right hop.
The primary operations dashboard is an Azure Monitor workbook, with a Grafana view of the same Log Analytics data if the SRE team already lives there. Request volume is shown as RPS and RPM, broken down by tenant, product, model, and backend. Token usage is split into prompt, completion, and total tokens, with a burn-down against each backend’s TPM window and each tenant’s allotment. Error health is a pair of burn rates: HTTP 429 from APIM versus 429 from Azure OpenAI, plus 5xx by hop, because those two 429s mean different things. Latency is P50, P95, and P99 for the full client-to-model path and for APIM-to-OpenAI alone.
Retry and failover panels count retries per request, circuit-open events, and how often the chosen backend was not the first preference. Cost is tokens multiplied by the model’s unit price, rolled up by tenant and by backend. Backend capacity utilization is TPM and RPM used versus granted, PTU saturation if applicable, and in-flight concurrency versus the router cap.
Alerts fire on 429 rate, 5xx rate, P99 latency, Redis unavailability, backend circuit open, and tenant quota exhaustion. Token and cost anomalies are warning-level, so finance and tenant owners can react before a hard throttle. Production never shares a workbook with development.
These signals are the same ones routing consumes in real time, and that disaster recovery uses to declare a region unhealthy. If a metric is on the dashboard but not in the router, treat that as a gap.
Disaster Recovery and Failover
The failure domain that matters is not only a single 429. An Azure OpenAI account, an entire region, APIM itself, Redis, or DNS can fail independently. Recovery is health-checked, automated, and bounded by explicit RTO and RPO rather than best-effort rerouting.
Health Checks
Each backend is probed on two channels. A synthetic probe from APIM or a function in the same VNet calls a cheap deployment at a fixed interval and records status, latency, and whether a 429 was returned. A traffic probe inspects the sliding window of real requests: 5xx rate, 429 rate, and P95 latency. Either channel can open the circuit. APIM’s own health is watched through platform metrics and a canary subscription. Redis health is watched because a dark quota store forces the router into a degraded mode: static weights, no new reservations, and more conservative concurrency caps.
Failure Scenarios And Recovery
If a single account fails or throttles, the router excludes it and shifts load to remaining accounts that host the same model, same region first. If a region fails, or if every account in that region is unhealthy, traffic fails over to the secondary region’s accounts over the same APIM when APIM is still healthy, or to a paired APIM in the secondary region. APIM Premium multi-region, or an active-passive secondary APIM, is required for that second case; a single-region Developer SKU is not a DR pattern. DNS for the gateway hostname uses a failover or latency policy. Failback is not automatic the moment a probe succeeds. A backend returns in half-open state, receives a capped share, and is fully enabled only after it meets the latency and error SLO for a defined window. Tenant quotas continue to apply during failover.
RTO and RPO
Inference is stateless, so recovery point for model calls is not applicable in the database sense: in-flight requests may fail and must be retried by the client. The RPO that matters is quota and routing state. Redis is geo-replicated or failed over with an RPO of seconds; losing the last few seconds of reservations can cause brief oversubscription, which Azure OpenAI will still cap with 429s. Configuration APIM policies, tenant maps, and Terraform state have an RPO of the last successful Git commit.
Target RTO is 5 minutes for a single backend, with the circuit opened and traffic shifted by the router with no human action; 15 minutes for an Azure OpenAI regional event if the secondary region is already warm; and 30 to 60 minutes for a combined APIM regional event that requires DNS failover. Those numbers assume the secondary region is pre-provisioned. Cold-creating a new Azure OpenAI resource during an outage is a quota request, not a failover, and must not appear in the runbook as a path to RTO.
Game days simulate a backend 429 storm and a regional OpenAI outage, and they score against these RTOs while watching the observability failover panels. The retry policy is the last safety net, not the whole strategy.
Production Readiness
A diagram is not a production gateway until the path can be created, changed, and recovered without a portal session.
Infrastructure as Code and Deployment Automation
Azure resources are declared in Bicep or Terraform: APIM (including SKU and units), Azure OpenAI accounts and deployments, backends, Redis, Key Vault, Log Analytics, diagnostic settings, and Entra app registrations used for JWT validation. APIM APIs, products, named-value references, and XML policies live in Git and are applied by the APIM DevOps toolkit or Terraform. What is not in Git is not considered deployed.
Environment Separation
Development, staging, and production are separate subscriptions or at least separate APIM instances and OpenAI accounts. They do not share Redis, they do not share subscription keys, and they do not share tenant maps. Staging is the only environment allowed to resemble production’s SKU mix closely enough for load and failover tests. Promotion is artifact-based: the same policy package moves forward; only variables and secrets change.
Secret Management
No Azure OpenAI key and no APIM subscription key live in application config or source control. Key Vault holds secrets and certificates. Named values that must exist in APIM are Key Vault references. The preferred backend credential is APIM’s managed identity with the Cognitive Services OpenAI User role on every backend account, which removes the API key from the hot path. Caller subscription keys are issued per tenant product and rotated on a defined interval.
CI/CD
One pipeline family covers three layers. The infrastructure pipeline plans and applies Bicep or Terraform on pull request and, with approval, on the production branch. The gateway pipeline lints APIM policies, runs a contract test against a mock backend, and publishes APIs. The consumer pipeline deploys calling services and runs a smoke test through the unified hostname. Production applies are gated on the staging smoke test. Rollbacks are a Git revert plus pipeline re-run, not a portal undo. A new backend is added in IaC, registered in the routing table, given a health probe, and shown on the capacity dashboard in the same change.
Implementation Outline
Build the foundation first. Provision the Azure OpenAI accounts and the target deployments in the regions you actually need, then provision APIM at a SKU that can carry the planned RPS. Enable a system-assigned managed identity on APIM and grant Cognitive Services OpenAI User on every account. Register each account as a named backend that uses that identity. Stand up Redis and Key Vault in the same private network story you intend to keep in production.
Then publish the APIs. Create model-family APIs or a generic completions API with a validated model field. Attach one inbound policy at All operations: authenticate, enforce tenant quota, estimate tokens, score backends, and call set-backend-service. Apply rate-limit, quota, and llm-token-limit at product or subscription scope so team contracts are not buried inside the router. Wire the outbound policy to write actual usage back to Redis.
Finally, add resilience. Point send-request at the quota-checker for adaptive routing, attach retry in on-error so a 429 or 503 on the first backend tries a different account, and register synthetic probes for every backend. None of this is optional if the SLA assumes the pool, not a single resource.
Security
Clients authenticate to APIM using a subscription key and, for tenant-aware APIs, a JWT issued by Entra ID. Backends authenticate APIM via Managed Identity. Azure OpenAI API keys are never exposed in client configurations nor stored as plaintext named values. Implementing additional network isolation such as APIM in Internal mode alongside Private Endpoints for Azure OpenAI and Redis aligns with this architecture and is a key hardening step for enterprise environments.
Prompt logging is an explicit, tenant-aware policy, not a default. The gateway can inject a standard system prompt or strip client headers before the model sees them, and it can mask internal versions or classified spans before the response returns. Those transformations belong in policy so every tenant inherits them.
Benefits
The payoff is operational, not cosmetic. Callers keep one URL. Traffic moves off a throttling account without a client change. Adding an account increases aggregate quota instead of creating a new integration project. Security improves because keys and hostnames stay behind APIM. Multi-model access becomes a path and a permission, not a new SDK. Cost-aware routing can send cheap work to GPT-4o-mini while reserving GPT-4o for the tenants who paid for it. None of those benefits require pretending the platform is unlimited; they require the pool, the router, and the dashboards described above.
Conclusion
A single Azure OpenAI account is not enough for enterprise-grade, multi-team, multi-model traffic. An AI Gateway with Azure API Management is the control plane that makes the alternative operable: one URL, many accounts, explicit quotas, and a router that understands more than round-robin.
Running at platform scale demands rigor: capacity-aware routing, centralized quota state across hot-path enforcement and historic tracking, and full-spectrum observability covering volume, tokens, latencies, retries, costs, and utilization. True scaling comes from aggregating independent quotas, not relying on illusory unlimited capacity.
Top comments (0)