Deploying a Model-Calling Service on Azure Container Apps

#ai #machinelearning #tutorial #llm

Container Apps sits between Functions and Kubernetes: you bring an image and a port, and the platform brings Envoy, TLS, revisions and KEDA. For a service whose job is to call a model provider, three of its defaults need changing before it behaves.

The environment

An environment is the boundary: apps in one environment share a virtual network and a Log Analytics workspace, and can address each other internally. There are two types. Workload profiles is the default, supports user-defined routes, egress through NAT Gateway and private endpoints, and has a documented minimum subnet size of /27. Consumption-only is the legacy type, supports none of those, and needs a /23. Microsoft, Networking in an Azure Container Apps environment.

az containerapp env create \
  --name env-inference \
  --resource-group rg-model \
  --location westeurope

You cannot change the network type after creation. If there is any chance this service will later need to reach Azure OpenAI over private networking, supply your own virtual network now with --infrastructure-subnet-resource-id rather than retrofitting it.

Deploying a revision

az containerapp create \
  --name inference-proxy \
  --resource-group rg-model \
  --environment env-inference \
  --image myregistry.azurecr.io/inference-proxy:1.4.0 \
  --target-port 8000 \
  --ingress external \
  --min-replicas 1 \
  --max-replicas 10 \
  --cpu 0.5 --memory 1.0Gi \
  --env-vars AZURE_OPENAI_ENDPOINT="https://my-aoai.openai.azure.com/"

--target-port is the port your process listens on inside the container. It is not the port the world connects to: Container Apps exposes 80 and 443 for inbound connections and terminates TLS at an Envoy edge proxy, which then routes to your target port. Downstream connections support HTTP/1.1 and HTTP/2, and Envoy upgrades automatically when the client asks. The upstream side — how the proxy talks to your container — is set by the transport property on the ingress object, and defaults to auto-detection.

Use --ingress internal if this service is only called by other apps in the environment. That removes the public FQDN entirely, which is a stronger control than any allow-list you would otherwise write.

Secrets and the secretref indirection

A provider key should not be an environment variable literal in your deployment command, where it lands in shell history and in the ARM representation of the app. Container Apps has app-scoped secrets and a reference syntax that connects them to environment variables:

az containerapp update \
  --name inference-proxy \
  --resource-group rg-model \
  --secrets "aoai-key=<key>" \
  --set-env-vars "AZURE_OPENAI_API_KEY=secretref:aoai-key"

The secretref: prefix is the whole mechanism — the container sees a normal environment variable, and the value never appears in the template. Note that this is not the App Service @Microsoft.KeyVault(...) syntax; Container Apps has its own secret store and its own Key Vault integration, and pasting the App Service form here leaves the literal string in your environment. The same scale-rule auth array uses these secrets by name, which is how a KEDA trigger authenticates without a second copy of the credential.

Revisions are immutable

Every change to the container image, the environment variables or the scale rules creates a new revision — an immutable snapshot. Changes to things outside that set, such as traffic weights, do not. In single revision mode the newest revision takes all traffic. In multiple revision mode old revisions stay available and you split traffic between them, which is how blue/green and canary deployments work here.

One documented catch matters as soon as you add queue-driven scaling: Microsoft notes that activeRevisionsMode should be set to single when using non-HTTP event scale rules. Two revisions both consuming the same queue is two sets of replicas competing for the same messages, which is rarely what a canary was meant to test.

Probes decide when traffic arrives

Container Apps supports three probe types per container — Startup, Liveness and Readiness — over HTTP(S) or TCP. exec probes are not supported, gRPC is not supported, ports must be integers rather than named, and you may define only one of each type per container. An HTTP probe succeeds on a status code greater than or equal to 200 and less than 400; anything else is a failure.

If you enable ingress and define none of them, defaults are added to the main app container, and it is worth knowing what they are because they are TCP checks against the ingress target port:

Startup — TCP, timeout 3 seconds, period 1 second, initial delay 1 second, success threshold one, failure threshold 240. That last number is the app’s startup budget, and four minutes is generous.
Liveness — TCP on the ingress target port.
Readiness — TCP, timeout 5 seconds, period 5 seconds, initial delay 3 seconds, success threshold one, failure threshold 48.

A TCP default is exactly as strong as “the socket is open”. For a model-calling service that is usually not enough, because the port binds before the client is constructed, before configuration is validated, and before any credential has been tested. Traffic arrives at a replica that will 500 on its first request. An HTTP readiness probe that returns 503 until the model client has been built and a credential acquired closes that window:

"probes": [
  {
    "type": "Readiness",
    "httpGet": { "path": "/readyz", "port": 8000 },
    "initialDelaySeconds": 3,
    "periodSeconds": 5,
    "timeoutSeconds": 5,
    "failureThreshold": 48,
    "successThreshold": 1
  }
]

Two documented behaviours follow from this. A revision appears unhealthy if any of its replicas fails readiness, even when every other replica is fine, and Container Apps restarts the failing replica until it is healthy or the failure threshold is exceeded. And in multiple revision mode you should wait for readiness to succeed before shifting traffic to a new revision — in single revision mode the shift happens automatically once readiness returns success. Do not put a dependency on the model provider inside the readiness probe: a provider blip then marks every replica unhealthy at once and restarts your entire revision.

The replica floor, and what it costs

The documented default limits are a minimum of 0 replicas and a maximum of 10, with both configurable up to 1,000. If you define no scale rule at all, the default rule is HTTP with a minimum of 0 and a maximum of 10.

Two consequences. First, scale to zero means a cold start on the next request: image pull, process start, model client construction. For a latency-sensitive proxy, --min-replicas 1 is the whole fix, and you are billed for that replica. Second, and less obvious — Microsoft warns that if ingress is disabled and you define neither minReplicas nor a custom scale rule, your container app scales to zero and has no way of starting back up. An internal worker with no ingress and no queue rule is a permanently stopped app that reports no error.

The cost of that floor is smaller than it looks, and the reason is a billing rule worth knowing. Consumption billing has two meters — vCPU-seconds and GiB-seconds — plus a charge on HTTP requests received from outside the environment. Microsoft documents monthly per-subscription free grants of the first 180,000 vCPU-seconds, the first 360,000 GiB-seconds and the first 2 million HTTP requests, and health probe requests are not billable at all.

More importantly, a replica held at the floor can be billed at a reduced idle rate rather than the active one. Microsoft documents the conditions precisely, and all of them must hold at once: the revision is configured with a minimum replica count greater than zero, the revision is scaled to that minimum, every container in the replica has started and is running, the replica is not processing any HTTP requests, it is using less than 0.01 vCPU cores, and it is receiving less than 1,000 bytes per second of network traffic. Microsoft, Billing in Azure Container Apps.

Read that list as a design constraint rather than a discount. A background poller, a metrics scrape, a chatty sidecar or a keep-alive to the model provider can each keep a replica above one of those thresholds permanently, in which case the floor is billed at the active rate around the clock. And once the revision scales above the minimum, all of its running replicas are charged at the active rate, including the one that was idle a second earlier.

Subnet minimums, default replica limits and CLI flag names are Microsoft’s documented values at the time of writing and have changed as environment types evolved. Check the Container Apps networking and scaling articles before you size a subnet.