Putting Azure OpenAI Behind an API Gateway: What the Docs Don't Tell You

We had a problem that will sound familiar to anyone who has let a team loose on Azure OpenAI.

One Foundry resource. One gpt-4o-mini deployment. One API key, copied into a config file, then into a second config file, then pasted into a Slack thread so someone could test something quickly. No idea who was spending what. No limits. If one app got stuck in a retry loop at 2am, everybody's model calls would start failing and we'd spend the morning working out whose fault it was.

The fix is an AI gateway. Azure API Management has a purpose-built wizard for this now, and the happy path genuinely is a wizard. But we hit roughly a dozen things that the documentation either glosses over or gets subtly wrong, and several of them are the kind of mistake you can't undo.

This is what actually happened, in order, including the parts where I was wrong.

What an AI gateway buys you

Before the mechanics, the argument, because it's easy to build this and then not enforce it.

Putting API Management in front of a model deployment gets you four things:

One key per application, revocable independently. Each consuming app gets its own gateway key. Revoke one, the others keep working. No app ever holds a credential for the model itself.

Spend caps that are actually enforced. A tokens-per-minute limit per application means one runaway loop can't drain the shared quota.

Attribution. Token counts land in Application Insights, dimensioned by which app made the call. "Who spent this" becomes a query instead of an investigation.

Caching. Semantically similar questions can be answered from a cache rather than paid for twice.

Keep that list in mind, because there's a trap at the end of this post where you build all of it and enforce none of it.

Step one: pick the right client compatibility mode

The wizard's first real decision is also its most expensive to get wrong, and it's presented as three innocuous radio buttons.

You are asked to choose between Azure OpenAI, Azure AI, and Azure OpenAI v1. The difference is where the deployment name lives.

With Azure OpenAI, the deployment name stays in the URL path:

POST {gateway}/{base-path}/openai/deployments/{deployment}/chat/completions?api-version=2024-12-01-preview

With the other two, the deployment name moves into the request body. That sounds like a trivial difference. It isn't, because the official AzureOpenAI Python SDK builds the first shape. Choose either of the other options and you are rewriting every client, not changing a URL.

Pick Azure OpenAI unless you have a specific reason not to. Your app code then changes by exactly one line:

client = AzureOpenAI(
    api_version="2024-12-01-preview",
    azure_endpoint="https://your-gateway.example.com/ai-prod",  # was the Foundry URL
    api_key=os.environ["GATEWAY_KEY"],                          # was the Foundry key
)

The SDK appends /openai/deployments/... itself.

There's a bonus hidden in this choice. Because the deployment name is a path segment, it behaves as a wildcard. Deploy a new model on the same Foundry resource, and it is immediately reachable through the gateway with no new API, no second wizard run, no config change. We only discovered this by asking; it's not obvious from the UI.

The base path is forever

The wizard asks for a base path and leaves the field blank. Everything else on that screen you can change later. This one you effectively cannot, because it becomes part of every client's URL.

Two rules. Make it lowercase and hyphenated, because base paths are case-sensitive and mixed case generates 404s that people waste an afternoon on. And make it specific rather than generic. openai feels natural until you onboard a second Foundry resource and discover the name is taken on that gateway instance.

The default token limit is a trap

Tick "manage token consumption" and the TPM field pre-fills with 1000.

Our deployment quota was 250,000 TPM. With max_tokens: 4096, a single verbose response can consume a meaningful fraction of 1000 tokens, so you'll start seeing 429s within a handful of test calls and reasonably conclude the gateway is broken.

Go and read your actual deployment quota first. Then set the gateway limit at or below it. Above is worse than useless: API Management forwards the traffic, the model throws the 429 instead, and you get the outage without the protection, with the error surfacing from the backend where it's harder to attribute. We set ours to 125,000, half the quota, leaving room to onboard a second consumer without touching the deployment.

Two settings on that screen deserve more attention than they get:

Limit by: Subscription. This makes the allowance per gateway key, so each app has its own bucket. The alternative, IP address, is wrong for most deployments because your apps will call from a small pool of shared outbound addresses and collide into a single bucket.

Estimate prompt tokens: on. With it off, the policy can only count tokens after the response returns, so a burst of concurrent requests all pass the check before any of them report. Worse: for streamed responses it can't count at all unless the client sends stream_options: {"include_usage": true}. Most client code doesn't. With estimation off, streaming calls quietly bypass your limit entirely.

Metrics dimensions: fewer than you think

The wizard offers nine dimensions to slice token metrics by, and it's tempting to take them all. We did, briefly.

Application Insights bills per unique combination of dimension values. Most of those dimensions are constants in a single-instance, single-API setup, so you pay for cardinality that tells you nothing.

Two are worth having:

• Subscription ID: which application spent the tokens. This is the chargeback dimension and the whole point of the exercise.
• API ID: constant today, essential the moment you add a second Foundry resource.

Skip Client IP (high cardinality, low information, since callers share outbound addresses), User ID (empty unless you're doing per-user JWT auth at the gateway), and Gateway ID / Location (constant for a single instance).

Worth knowing before you decide: Application Insights data isn't retroactive. Metrics already emitted keep their original dimensions. So adding a dimension later means old and new data can't be compared cleanly, and removing one leaves a split in your history. Get it roughly right at the start.

There are two different Application Insights connections

This one cost us a debugging session.

The token metric policy and the request telemetry logger are separate mechanisms, configured in different places, and having one does not give you the other.

The token metric policy is set up in the wizard. It emits prompt, completion, and total token counts.

The request logger is a per-API setting under Settings, Diagnostics Logs. It captures status codes, latency, which operation was called, and failures. It has its own destination field, which starts empty.

So after the wizard we could see exactly how many tokens a request consumed, but not that it had returned a 500 or taken twelve seconds. When we hit our first error, there was nothing useful to look at.

Set both. On the request logger, set sampling to 100% for a low-volume API, tick "always log errors", and switch the correlation protocol from Legacy to W3C, which is what lets you trace a request end to end across the edge, the gateway, and the backend. Leave "payload bytes to log" at 0 unless you're actively debugging, because raising it puts prompt and response content into your logs.

One more thing that surprised us: the request logger is instance-wide by default, so telemetry from unrelated APIs on the same gateway lands in the same Application Insights resource. That's not a bug. Operations are prefixed with the API ID, so filtering is trivial:

requests
| where operation_Name startswith "your-ai-api"

The only place it genuinely bites is alerting. An alert on "average duration over 5 seconds" is meaningless when one API responds in 100ms and the other takes 8 seconds. Scope alerts by operation name, not on the blended instance average.

Semantic caching is where the wizard falls apart

This is the feature with the longest prerequisite chain, and the wizard will not tell you what's missing until you're several screens in.

Your existing Redis almost certainly won't work

Semantic caching needs a vector index, which means Redis with the RediSearch module. We had a Redis instance already, running happily as a cache for another system. Its module list read RedisJSON, RedisTimeSeries, RedisBloom. No RediSearch.

Modules are fixed at creation time. You cannot add RediSearch to an existing instance. You create a new one.

RediSearch conflicts with OSS clustering

Creating that new instance, we ticked RediSearch and immediately got:

OSSCluster Cluster Policy doesn't support the selected module(s): RediSearch

The fix is one click: change Clustering Policy from OSS to Enterprise. Enterprise clustering hides sharding from the client, which is what the search module requires.

You can't set an eviction policy, and that matters

With RediSearch enabled, the eviction policy dropdown offers exactly one option: No Eviction. Azure locks it, because the search index needs its data present to stay consistent.

The consequence is easy to miss. Your cache will not self-clean. When it fills, Redis starts rejecting writes rather than dropping old entries. So the cache duration you set in the API Management policy stops being a tuning knob and becomes the only thing preventing unbounded growth. We used 3600 seconds. Size the instance with headroom.

InsufficientCapacity is a region problem, not a config problem

Our first three deployment attempts failed with a generic Conflict / ResourceDeploymentFailure. The portal's error summary was useless. The Raw Error tab had the real answer:

{
  "status": "Failed",
  "error": {
    "code": "InsufficientCapacity",
    "message": "Request failed due to insufficient capacity. Retry using a different Azure Managed Redis size or region."
  }
}

Always open Raw Error. The summary is a wrapper.

Two levers: change the size, or change the region. Note that changing size within the same performance tier often doesn't help, because those SKUs draw from the same capacity pool. Switching performance tier (Balanced to Memory Optimized, say) is more likely to find capacity than stepping from B0 to B1.

We eventually provisioned in a different region, which leads directly to the next mistake.

The external cache has a region field, and it's not decorative

Registering Redis as API Management's external cache asks you which gateway location "uses" this cache. Ours defaulted to the region the Redis instance was in, which was not the region the gateway was in. If no gateway location matches, the cache may never be used at all, silently. Set it to your gateway's region, or Default.

Also worth understanding: external cache registration is instance-wide, but using it is per-API. Registering the cache doesn't change the behaviour of any existing API, because nothing touches Redis unless a policy says so.

Cross-region caching may cost more than it saves

We ended up with the gateway in one continent and Redis in another. Roughly 200ms of round trip added to every request, hits and misses alike.

On a cache miss, that's ~200ms added to a call already taking 1 to 2 seconds. Annoying, survivable. On a hit, you still come out well ahead. The place it genuinely hurts is streaming, where the full round trip lands on time-to-first-token, which is the latency users actually perceive.

But the real question isn't latency. It's whether your prompts repeat at all. Semantic caching pays an embedding call on 100% of requests to get a saving on some fraction of them. If your traffic is mostly unique prompts, you're paying for nothing. Measure your hit rate before you defend the architecture.

I should also correct something I believed going in. I assumed Redis Enterprise with search would cost several hundred dollars a month, and used that to argue against the whole feature. Azure Managed Redis at the entry tier came in around $25/month. That's a materially different calculation, and it changed my recommendation. Check current pricing rather than trusting a number you remember.

The embeddings backend the wizard forgets to create

This one produced our only hard failure.

The semantic cache lookup policy references an embeddings backend by ID. That backend needs to exist as a registered API Management Backend, pointing at an embeddings deployment. The wizard normally creates it for you.

Ours didn't, because the wizard aborted partway through with a benign-looking error:

Error setting up semantic caching. The role assignment already exists.

The role assignment conflict was harmless. An earlier attempt had already granted the gateway's managed identity the right role. But the wizard treated it as fatal, stopped, and never created the backend or added the policies.

We added the policies by hand and got a clean 500 Internal Server Error on every request, in about 800ms, far too fast to have reached the model. The lookup policy was throwing because embeddings-backend didn't resolve.

The fix: create the backend manually as a Custom URL, with the name matching the policy exactly.

https://<your-resource>.openai.azure.com/openai/deployments/text-embedding-3-small/embeddings

Two lessons. First, if the wizard reports any error, go and verify what it actually created; don't assume a partial failure is a partial success. Second, and more generally: add one policy at a time and test between each. We added two policies and a backend dependency in one go, then had to bisect. Getting a single successful 200 before layering anything on would have made the failure obvious immediately.

The policy order, and what it means

The finished inbound chain, in execution order:

1. Validate the gateway key against the product. An unrecognised key is rejected here, before any Azure resource is touched.
2. Semantic cache lookup. Embed the prompt, search Redis. A match above the similarity threshold returns the stored answer and steps 3 and 4 never run.
3. Token rate limit. Count the call against that key's allowance.
4. Emit token metrics.
5. Route to the backend using the gateway's managed identity.

And on the way out:

1. Semantic cache store, writing the answer and its vector back to Redis.

The sequencing is deliberate: cheap rejections first, cache before the expensive call, and no credential anywhere in the path.

One thing worth internalising: a cache miss now costs you an embedding call plus a model call. Add content safety screening and you have three service round trips before the model sees the prompt. That's the honest price of a governed gateway. It's usually worth it. Just don't be surprised by the latency and go looking for a bug.

Similarity threshold: trust the description, not the warning

Setting the threshold, the portal showed us this:

Similarity score threshold above 0.2 may lead to cache mismatch. Consider using lower value.

That is backwards, at least as worded. The field description says similarity, where 1.0 is an exact match. A higher threshold means stricter matching and fewer false hits. A lower threshold is what causes mismatches: at 0.2 almost any two prompts would look similar and you'd cheerfully serve a user the answer to a question they didn't ask.

We started at 0.9. Start strict, watch your hit rate, and lower it deliberately if hits are too rare. This is the one setting where a wrong value produces plausible but incorrect output rather than an error, which makes it the most dangerous knob on the page.

Also worth scoping the cache per key with a vary-by on the subscription. Otherwise one application's answers can be served to another.

The header rename that saves you a rewrite

API Management expects its key in an Ocp-Apim-Subscription-Key header. The AzureOpenAI SDK sends credentials in an api-key header.

You can work around this in client code with default_headers, but there's a cleaner option: under the API's Settings, rename the subscription header to api-key. Now the SDK works unchanged and your app diff is genuinely one line.

Products and subscriptions: badly named, load-bearing

The terminology here trips people up, so plainly:

An API Management subscription has nothing to do with an Azure subscription. It's an API key.

A product is a bundle of APIs plus access rules. A subscription is a key issued against a product. So: product contains your API, you create a subscription on the product, it generates a key, the app sends the key.

This isn't optional decoration. Two of the settings you already configured point at nothing without it: Limit by: Subscription has no subscriptions to count per, and the Subscription ID metrics dimension has nothing to record. And of course without a key nobody can call the API at all.

The mistake that makes all of this decorative

Here's the one that matters most, and it's the easiest to skip because everything appears to be working.

When you finish, you have a governed gateway. You also still have the original Foundry endpoint, with its original keys, publicly reachable, bypassing every control you just built. Anyone holding a copy of that key gets no rate limit, no metrics, no cache, no attribution.

Your gateway is optional. And an optional control is not a control.

There are two ways to close it, with very different effort profiles.

Disable local key authentication. The endpoint stays publicly reachable, so the gateway keeps working unchanged (it authenticates with its managed identity, not a key), but the API keys stop working entirely.

az cognitiveservices account update \
  --name <your-foundry-resource> \
  --resource-group <your-rg> \
  --custom-domain <your-foundry-resource> \
  --api-properties disableLocalAuth=true

Or disable public network access and put a private endpoint in front of it. Stronger, because it removes the public attack surface entirely rather than just the key path. But it's a real project: private endpoint, private DNS zone linked to the VNet, and gateway VNet integration, which requires a Standard v2 or Premium v2 tier. If you're on a lower tier, this becomes a SKU migration with a cost attached.

If you go the private route, audit callers before disabling local auth, and do it last in the sequence. If some forgotten script or integration is hitting the model with a key today, it breaks the moment you flip that switch, silently and at whatever time of day you happened to run the command.

What I'd do differently

Prove the base path works before adding anything. One successful 200 through the gateway, with nothing but key validation in the chain. Then add the token limit, test. Then metrics, test. Then caching, test. We stacked four unproven dependencies and spent longer bisecting than the sequential approach would have taken.

Read the raw error, always. The portal's error summaries are wrappers around the useful message.

Verify what the wizard claims to have built. A partial failure looks a lot like success from the notifications panel. Check the policy XML. Check the backends list. Check the role assignment landed on the target resource.

Decide about direct access on day one. Not as a follow-up task. Until the bypass is closed, you have built a dashboard, not a gateway.

Ask the data residency question out loud. Semantic caching stores prompt text and responses. If your organisation deliberately chose a region for compliance reasons, a cache in a different geography is a real change to where data lives, not an implementation detail. It's the item least likely to surface on its own and most likely to become a problem later.

Was it worth it?

Yes, and not marginally.

The end state: every application holds its own revocable key, no application holds a credential for the model, spend is capped per application and attributable to the team that caused it, repeated questions are served from cache instead of being paid for twice, and there's one place to look when something breaks.

The build took an afternoon. Most of that afternoon was the dozen small things above, none of which are in the quickstart. Now that you know about them, it should take you an hour.