KEDA Scaling on Azure Container Apps for Queue-Driven Inference

Container Apps runs KEDA underneath, so any ScaledObject-based KEDA scaler translates into a scale rule. The translation is mechanical. The part that is not mechanical is choosing the target number, because it does not mean what its name suggests.

The shape of a scale rule

Scaling is limits plus rules plus behaviour. Limits are minReplicas and maxReplicas — documented defaults 0 and 10, both configurable to a maximum of 1,000. Rules come in three categories: http, tcp and custom, where custom wraps a KEDA scaler. If you define more than one rule, the app begins to scale once the first condition of any rule is met.

Converting a KEDA scaler is a two-field exercise: the scaler’s type becomes custom.type, and its metadata block becomes custom.metadata verbatim. Values are strings in the ARM representation even when they are numbers in the KEDA YAML.

The Service Bus rule

Microsoft’s worked example is the Azure Service Bus scaler, which is the right one for queue-driven inference. In ARM or Bicep:

"scale": {
  "minReplicas": 0,
  "maxReplicas": 20,
  "rules": [
    {
      "name": "azure-servicebus-queue-rule",
      "custom": {
        "type": "azure-servicebus",
        "metadata": {
          "queueName": "inference-jobs",
          "namespace": "sb-inference",
          "messageCount": "5"
        },
        "identity": "system"
      }
    }
  ]
}

Or through the CLI, where metadata is space-separated key/value pairs:

az containerapp update \
  --name inference-worker \
  --resource-group rg-model \
  --min-replicas 0 --max-replicas 20 \
  --scale-rule-name azure-servicebus-queue-rule \
  --scale-rule-type azure-servicebus \
  --scale-rule-metadata "queueName=inference-jobs" \
                        "namespace=sb-inference" \
                        "messageCount=5"

Set activeRevisionsMode to single — Microsoft documents that explicitly for non-HTTP event scale rules, and two revisions draining one queue is a confusing way to find out why.

For a topic rather than a queue, swap queueName for topicName and add subscriptionName, which the KEDA scaler requires whenever a topic is named. The scaler also accepts useRegex (default false) to match several entities by pattern, in which case operation decides how their counts are combined — sum by default, or max or avg. A fan-out pipeline with one queue per tenant can therefore be driven by a single rule, though sum across tenants scales on total backlog and max scales on the worst-off tenant, and those are different products.

Authentication without a second secret

Scale rules for Azure Queue Storage, Service Bus and Event Hubs support managed identity, and Microsoft’s guidance is to use it where possible rather than storing a connection string. That is the identity property in the ARM snippet above, or --scale-rule-identity on the CLI, taking either the string system or a user-assigned identity resource ID.

Where you must use a secret, the KEDA TriggerAuthentication object maps onto the rule’s auth array: the scaler’s parameter becomes triggerParameter, and the secret key becomes secretRef, pointing at a secret in the app’s secrets array.

--secrets "connection-string-secret=<SERVICE_BUS_CONNECTION_STRING>" \
--scale-rule-auth "connection=connection-string-secret"

Note that the scaler and your application code authenticate separately. The scale rule reads the queue depth; your container reads the messages. Granting the identity a data role on the namespace and forgetting the scaler’s own binding produces an app that processes messages correctly and never scales, which looks like a broken threshold and is a broken credential.

What messageCount actually means

messageCount is not “scale out when the queue exceeds five”. It is the target number of messages per replica, and it is the denominator in the documented scaling algorithm:

desiredReplicas = ceil(currentMetricValue / targetMetricValue)

With messageCount of 5 and 50 messages waiting, ceil(50/5) is 10 replicas. With messageCount of 1 and the same queue you get 50, which is probably not what you want and is certainly not what your model provider’s rate limit wants.

For inference the right value comes from a question that has nothing to do with CPU: how many model calls should one replica have in flight at once? That number is bounded by your provider quota divided by your replica ceiling, and by whatever concurrency your process actually handles without queueing internally. Pick messageCount to match that, then set maxReplicas so that maxReplicas × per-replica concurrency stays under the tokens-per-minute you are allowed. Autoscaling into a rate limit converts a queue backlog into a wave of 429s and burns retry budget without doing more work.

One further property of the metric decides what you are actually scaling on. The KEDA Service Bus scaler counts active messages. Dead-lettered messages are not part of the count. That is the right default — a poison message should not hold replicas open forever — but it produces a specific confusing morning: a queue that visibly contains thousands of messages, an app sitting at zero replicas, and no error anywhere. Check the active count rather than the total before concluding the rule is broken. The default messageCount in the KEDA scaler is "5" if you omit it, which is rarely the number you want and never the number you reasoned your way to.

The separate threshold for leaving zero

There are two thresholds in this scaler and almost every article only mentions one. messageCount decides how many replicas to run once the app is running. activationMessageCount decides whether the app runs at all — it is the threshold for the transition from zero replicas to one, and it defaults to "0".

With the default, a single message wakes the app. For a queue that receives a trickle of low-value work all day, that means paying a cold start and a replica for each straggler. Setting activationMessageCount to, say, 20 means the app stays at zero until a real batch accumulates, then scales on messageCount as normal.

--scale-rule-metadata "queueName=inference-jobs" \
                      "namespace=sb-inference" \
                      "messageCount=5" \
                      "activationMessageCount=20"

The trade is latency for the first message in a batch, and it is a real trade rather than a free win: below the activation threshold, work simply waits. Set it deliberately, and never above a value the workload can reach — an activation threshold higher than your typical burst is an app that never starts.

How long the burst takes to be met

The documented scale behaviour makes this predictable rather than mysterious:

• Polling interval: 30 seconds. KEDA checks the queue on that cadence, so a burst can sit for up to 30 seconds before any decision is made. This does not apply to HTTP and TCP rules.
• Scale-up step: 1, 4, 8, 16, 32, … up to the maximum. The full documented step is min(maxReplicaCount, desiredReplicas, max(4, 2*currentReplicaCount)), so going from 1 replica to 40 takes several polling intervals, not one.
• Scale-down stabilisation window: 300 seconds. A scale-in only happens if the condition holds for five minutes.
• Cool down period: 300 seconds, and Microsoft notes it only takes effect when scaling in from the final replica to zero.

Add the image pull and process start on each new replica and you have your real time-to-capacity. If that number is unacceptable, the lever is minReplicas, not the scale rule — pre-provisioned replicas are the only way to have capacity before the poll.

What scale-in does to an in-flight call

Scaling out is the easy direction. Scaling in is where a queue-driven inference worker loses work or duplicates it, and the mechanism is worth understanding before it happens rather than after.

When KEDA decides to remove a replica, the platform stops routing to it and signals the container to shut down. Your process gets a termination signal and a grace period; if it exits immediately on that signal, the model call it was waiting on is abandoned. You have paid for the tokens generated so far and produced nothing. Worse, the Service Bus message was never completed, so when its lock expires it is redelivered and a second replica pays for the same inference again.

Three things make that survivable, and all three are your code:

• Handle the termination signal. Stop accepting new messages, let the in-flight one finish, complete it, then exit. A worker that ignores the signal gets killed at the end of the grace period regardless.
• Keep one message’s work inside the grace period. Which means bounding max_tokens and setting an explicit client timeout, exactly as on the Dapr page. An unbounded generation is an unbounded shutdown.
• Be idempotent anyway. Platform maintenance, a failed liveness probe and a node moving all produce the same outcome as a scale-in, and none of them are things you scheduled. Key results on the job identifier and check before calling the model.

The 300-second scale-down stabilisation window helps here more than it looks: it means scale-in is never sudden and never reactive to a single empty poll. What it does not do is wait for your work. Microsoft also notes that during platform upgrades or maintenance you might temporarily see more replicas than expected, because new replicas are pre-warmed before traffic shifts and the extras are removed once the operation completes. A capacity alarm that fires on replica count needs to tolerate that.

Polling interval, cool down, step function and default replica limits are Microsoft’s documented values at the time of writing. Verify against the Container Apps scaling article before you build a capacity model on them.

Related

• Deploying a Model-Calling Service on Azure Container Apps
• Dapr Pub/Sub on Azure Container Apps for a Model Pipeline