DEV Community: Josef Doornink

Standing Up a GPU Cluster on AKS for vLLM

Josef Doornink — Sun, 30 Aug 2026 00:07:52 +0000

This article is Part of a series on running vLLM on AKS and walks through creating an AKS cluster with a GPU node pool, deploying vLLM onto it, and wiring up Prometheus and Grafana for visibility.

Companion pieces:
Choosing the right GPU | Why your autoscaler flaps| Source

Setup Summary

Cloud: Azure
GPU node: Standard_NV36ads_A10_v5 (1× A10, 24 GB)
Image / model: vllm/vllm-openai:latest serving Qwen/Qwen2.5-7B-Instruct-AWQ
Observability: kube-prometheus-stack (Prometheus + Grafana), KEDA, NVIDIA DCGM exporter

All commands below are bash. The steps are ordered and each one depends on the previous.

Dependency chain

The build order follows one chain: model → VRAM requirement → GPU SKU → region availability → quota.

Step 0 — Prerequisites (one-time, survives resource group deletion)

GPU quota. Request through Portal → Quotas → Compute →
This article: Requested Standard NVADSA10v5 Family vCPUs = 108 in westus (108 = 3 nodes × 36 vCPUs, matching the autoscaler's max-count 3 set in step 3).

Quota is granted per-subscription and survives resource group deletion, so this step happens once, not on every rebuild.

A quota is Azure's per-subscription limit on how much of a resource (here, GPU vCPUs in a specific VM family) you're allowed to provision at once. New subscriptions start at 0 for GPU families since it's expensive and can be abused.
You need it because without an approval, az aks nodepool add for a GPU will fail outright. The request goes through manual Azure approval, so it has to happen before you plan to build.

*Prerequisites *

Existing Azure Subscription:

Local tooling:

Helm 3+
kubectl
Bash

Bash Variables to set for use through the setup

RG=<resource-group-name>
CLUSTER=<cluster-name>
LOCATION=<preferred-location>

Step 1 — Create Resource group

az group create -n $RG -l $LOCATION

Step 2 — Create AKS cluster, on a CPU system pool

az aks create -g $RG -n $CLUSTER \
  --node-count 1 --node-vm-size Standard_D2s_v5 \
  --generate-ssh-keys

The GPU does not go on this pool. Every AKS cluster requires a system node pool for cluster-critical pods (CoreDNS, metrics-server), and system pools cannot scale to zero — so a GPU placed here runs, and bills, 24/7 regardless of load. A D2s_v5 CPU node covers the system pods cheaply; the GPU pool created in step 3 is where scale-to-zero actually happens.

Step 3 — GPU node pool, tainted and scaled from zero

az aks nodepool add \
  -g $RG --cluster-name $CLUSTER \
  -n gpu \
  --node-vm-size Standard_NV36ads_A10_v5 \
  --node-count 0 --enable-cluster-autoscaler --min-count 0 --max-count 3 \
  --node-taints sku=gpu:NoSchedule --labels sku=gpu

What each flag does:

--node-count 0 + --enable-cluster-autoscaler — scale-from-zero. No node exists until a pod requires one, so there's no GPU spend until step 7. The cluster autoscaler evaluates a pending pod against the pool's declared taints/labels/VM size to decide whether it would fit, then provisions a node if so — which is why the taint and label must be set on the pool at creation, not discovered later from a running node.
--min-count 0 --max-count 3 — bounds the pool between 0 and 3 nodes. (Design decision)
--node-taints sku=gpu:NoSchedule — blocks ordinary CPU pods from landing on the GPU node once it exists.
--labels sku=gpu — the label the vLLM pod's nodeSelector targets in step 7.

Taint, toleration, and nodeSelector do three separate jobs: the taint repels pods by default, a toleration permits a specific pod to ignore that taint, and a nodeSelector steers a pod toward a specific node. A toleration alone doesn't guarantee placement — it only lifts the block. The vLLM pod spec in step 7 carries both the toleration and the nodeSelector because both are required.

The above scaling sets the --node-count and --min-count to 0; this may or may not be desirable for your use case. Keeping a node or 2 warm can help with latency, but there is cost associated with that. Choose whichever best fits your use case.

Step 4 — Get kubeconfig for cluster communication

az aks get-credentials -g $RG -n $CLUSTER
kubectl get nodes          # expect only the system node — the GPU pool is still at 0

Step 5 — NVIDIA device plugin

AKS does not install this by default. Without it, a GPU node never advertises nvidia.com/gpu as an allocatable resource, and any pod requesting nvidia.com/gpu: "1" stays Pending indefinitely with no error.

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml

The upstream DaemonSet only tolerates the standard nvidia.com/gpu taint, not the custom sku=gpu taint set in step 3, so it won't schedule onto the GPU node without a patch:

kubectl patch daemonset nvidia-device-plugin-daemonset -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/tolerations/-","value":{"operator":"Exists"}}]'

A Pending vLLM pod looks identical whether the device plugin is missing, mis-scheduled, or the node just hasn't scaled up yet. kubectl describe node -l sku=gpu and checking for nvidia.com/gpu under Allocatable distinguishes between the three.

Step 6 — Observability core: Prometheus + Grafana + KEDA

NOTE: Installed before the GPU node scales up, so the stack builds on the free CPU pool. Values file: observability/kps-values.yaml — 6-hour retention, an 8 Gi PV on managed-csi, Grafana on ClusterIP, Alertmanager disabled.

No StorageClass to create beforehand — AKS ships a built-in managed-csi class (provisioner disk.csi.azure.com, WaitForFirstConsumer binding), which the Prometheus PVC above uses directly.

Add the chart repos first (one-time per workstation):

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kedacore https://kedacore.github.io/charts
helm repo update

helm upgrade --install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  -f observability/kps-values.yaml --timeout 10m

helm upgrade --install keda kedacore/keda -n keda --create-namespace --timeout 5m

Verify:

kubectl get pods -n monitoring          # prometheus, grafana, kube-state-metrics Running
kubectl get pods -n keda                # keda-operator + metrics-apiserver Running
kubectl api-resources | grep scaledobject   # confirms KEDA's CRDs landed

prometheus-node-exporter is configured to tolerate sku=gpu (operator: Exists) in the values file, so it lands on the GPU node automatically once it scales up — no separate install step needed for CPU/memory/disk metrics from that node.

Grafana's admin password is admin, set in the values file. Acceptable for a cluster torn down daily; not for anything long-lived.

Step 7 — Deploy vLLM

kubectl apply -f deployment.yaml
kubectl get pods -w

Expected sequence: Pending → cluster autoscaler provisions an A10 node (~3–5 min) → ContainerCreating → image pull (~1 min, 8.8 GB) → model weights load → 1/1 Running.

The pod spec ([deployment.yaml](https://github.com/JDoornink/vLLM_on_K8s/blob/main/deployment.yaml)) is where the taint/toleration/nodeSelector from step 3 get consumed:

spec:
  tolerations:
    - key: sku
      operator: Equal
      value: gpu
      effect: NoSchedule   # matches --node-taints sku=gpu:NoSchedule on the GPU nodepool
  nodeSelector:
    sku: gpu               # matches --labels sku=gpu on the GPU nodepool
  containers:
    - name: vllm-gpu
      image: vllm/vllm-openai:latest
      args:
        - --model
        - Qwen/Qwen2.5-7B-Instruct-AWQ
        - --quantization
        - awq
        - --gpu-memory-utilization
        - "0.85"
        - --max-num-seqs
        - "32"
      resources:
        limits:
          nvidia.com/gpu: "1"    # ensures only one pod per GPU node

Two settings worth explaining:

--gpu-memory-utilization 0.85, not the vLLM default of 0.92. At 0.92, vLLM tried to claim 21.82 GiB of the 24 GB card but only ~21.37 GiB was free, and engine init failed. The model fits — this is a headroom setting, not a capacity limit. Full VRAM accounting is covered in GPU sizing.
nvidia.com/gpu: "1" in resources.limits is what makes one-pod-per-node a scheduling constraint rather than a convention: Kubernetes tracks the node's GPU as consumed once this pod is placed, so a second replica can't land on the same node and the autoscaler brings up a new one instead.

Step 8 — Service + smoke test

kubectl apply -f [service.yaml](https://github.com/JDoornink/vLLM_on_K8s/blob/main/service.yaml)
kubectl get endpoints vllm-openai-gpu        # must show pod IP:8080, confirming the selector matched

kubectl port-forward svc/vllm-openai-gpu 8080:8080 &
curl -s localhost:8080/v1/models | jq        # should list id "vllm-openai-gpu"

The Service is ClusterIP — reachable only via port-forward, no public IP. Switch to type: LoadBalancer only if the endpoint needs to be reached from outside the cluster (e.g., load-testing from a separate machine).

Step 9 — GPU metrics: DCGM exporter

vLLM's /metrics endpoint reports request/queue stats but nothing about the GPU itself — no utilization, VRAM, temperature, or power. NVIDIA's DCGM exporter is a DaemonSet that reads the GPU directly and exposes it to Prometheus. It requires the GPU node to already be up (step 7) and, like the device plugin, must tolerate the sku=gpu taint to schedule there.

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update gpu-helm-charts
helm upgrade --install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  -n monitoring -f observability/dcgm-values.yaml --timeout 5m

kubectl rollout status ds/dcgm-exporter -n monitoring --timeout=120s

Two failure modes, both fixed in observability/dcgm-values.yaml:

OOMKilled at a 256Mi memory limit (exit 137). DCGM's field collection needs more headroom than that. Fixed with a 1Gi limit.
scrapeTimeout must be ≤ interval. The chart defaults scrapeTimeout to 25s; with interval: 15s, that combination makes the generated ServiceMonitor invalid, and the Prometheus operator drops the target with no visible error — the ServiceMonitor and Service objects both exist, Prometheus is healthy, but the target never appears. Fixed by setting serviceMonitor.scrapeTimeout: 10s.

Step 10 — Wire vLLM into Prometheus

DCGM ships its own ServiceMonitor, discovered automatically because kps-values.yaml sets serviceMonitorSelectorNilUsesHelmValues: false (Prometheus picks up every monitor object in the cluster, not just ones with a specific release label). vLLM needs a PodMonitor instead, since it's scraped directly on the pod's metrics port:

kubectl apply -f observability/vllm-podmonitor.yaml

Verify both targets are up, not just present:

kubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-prometheus 9090:9090 &
curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.labels.job|test("vllm|dcgm";"i")) | "\(.health)  \(.labels.job)"'
# expect:  up  default/vllm-openai-gpu   AND   up  dcgm-exporter

curl -s 'http://localhost:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL'      # GPU utilization series
curl -s 'http://localhost:9090/api/v1/query?query=vllm:num_requests_running' # vLLM's own series

Step 11 — Access Grafana

kubectl port-forward -n monitoring svc/kps-grafana 3000:80 &
# browse http://localhost:3000  (admin / admin)

GPU-level and application-level metrics now land in the same Prometheus, on the same time axis — the data the capacity-planning math in GPU sizing and the autoscaler experiments in the flapping article are built from.

Verification checklist

# device plugin landed on the GPU node and is Running
kubectl get pods -n kube-system -o wide | grep -i nvidia

# GPU is now advertised as an allocatable resource
kubectl describe node -l sku=gpu | grep -A8 Allocatable      # expect: nvidia.com/gpu: 1

# vLLM is actually serving
kubectl logs -f -l app=vllm-openai-gpu                       # look for "Application startup complete"
kubectl get pods -o wide                                     # confirm one pod per GPU node

If any of these fail, work backward through steps 5 → 3 rather than re-running the deployment. A Pending vLLM pod is almost always caused upstream of vLLM itself.

Teardown

az group delete -n $RG --yes --no-wait

Deletes the cluster, both node pools, and the auto-created node resource group (MC_*) that holds the managed disks, including the Prometheus PV. The GPU quota grant from step 0 is untouched and persists for the next rebuild.

To keep the cluster but stop GPU spend without a full teardown:

kubectl scale deploy vllm-openai-gpu --replicas=0    # the GPU pool's autoscaler drains the node 1→0

Gotchas

The GPU pool must be a separate user pool, not the default/system pool. System pools can't scale to zero.
The NVIDIA device plugin needs an explicit toleration for a custom taint. It only tolerates the standard nvidia.com/gpu taint by default.
--gpu-memory-utilization 0.85, not the 0.92 default, on a 24 GB A10 — a headroom setting, not a capacity limit. See GPU sizing.
DCGM's scrapeTimeout must be ≤ its scrape interval, or the Prometheus operator drops the target with no visible error.
A crash-looping vLLM pod leaks GPU memory into the retry. Delete the pod so VRAM releases before the next attempt.

Summary

Follow the dependency chain — model → VRAM → GPU SKU → region → quota — and request quota ahead of time, since it starts at zero and requires manual approval.
Keep the GPU pool separate from the system pool so scale-to-zero is possible.
Taint the GPU pool, and pair it with both a toleration and a nodeSelector on the pod. A toleration alone only permits placement; it doesn't steer it.
Install the NVIDIA device plugin manually — AKS doesn't, and it needs the custom taint tolerated before it schedules.
Tear down with the resource group, not the cluster alone — it removes the managed disks and leaves quota untouched.

This gets the environment running. The remaining questions — how large a GPU the model actually needs, and what signal the autoscaler should watch — are covered in the companion articles GPU sizing and the flapping article.

Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess

Josef Doornink — Wed, 19 Aug 2026 00:24:15 +0000

Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess

OK, you're a senior SRE, you've been hearing incessantly about AI models, but aren't quite sure how to determine the correct node size to host your model. - If so ... you're in the right place.

Part of a series on running vLLM on AKS. Companion piece: How to avoid flapping. GPU infrastructure setup — coming soon.

This piece walks through estimating GPU memory requirements from both a model's parameter count or a concurrent requests requirement.

After reading this article you will have enough knowledge to pick a GPU family with confidence.

Disclaimer: this process is a rule-of-thumb filter, not a precise calculation — the last step covers how to get exact numbers once the model is actually running.

Background: What actually consumes GPU memory

AI models live in GPU memory — VRAM — and engines such as vLLM provide novel techniques for managing that memory efficiently [paper], but the model isn't the only thing consuming it.
Below is a short list of things that consume our precious VRAM:

Model weights — the parameters themselves. The big fixed cost: loaded once, never shrinks.
KV cache — working memory for in-flight requests. Every token of every active request holds its attention keys/values here. This is the one that determines throughput: more KV cache = more concurrent requests.
Everything else — activations (the temporary tensors of a forward pass) plus CUDA/framework overhead. You don't calculate these by hand; vLLM measures activations with a profiling pass at start-up and prints it for our consumption.

The sizing question is really: after weights and overhead, how much is left for the KV cache — and is that enough for your traffic?

OK, lets get started

Step 1 — Choose a model

Guidance on which model to choose is outside the bounds of this article.

What matters here: once you have a candidate, everything below can be read off its spec sheet — you can then run this method on every model on your shortlist and eliminate the ones that don't fit your requirements.

For demonstration purposes I will use Hugging Face's Qwen2.5-7B-Instruct-AWQ model card, config card

Step 2 — What to look for in the spec sheet

What	Name	Value Qwen2.5-7B-Instruct-AWQ
Parameter count	model card / Number of Paremeters	~7.6 B ("7B")
Quantization	`quantization_config`	AWQ, 4-bit
Layers	`num_hidden_layers`	28
KV heads	`num_key_value_heads`	4
Head dimension	`hidden_size / num_attention_heads`	3584 / 28 = 128

The first two size the weights. The last three size the KV cache per token.

That's the whole shopping list.

Step 3 — Calculate the VRAM for the weights

Rule of thumb: weights ≈ parameter_count × bytes_per_parameter and many models have different precision offerings

Precision	Bytes/param	7.6 B params
fp16 / bf16	2	~15.2 GB
int8	1	~7.6 GB
AWQ / 4-bit	~0.5	~3.8 GB

Note on Weights and quantization: the above table compares the same model, just with different quantization:
Quantization is storing each weight in fewer bits than it was trained in. Models train in 16-bit float, so every parameter costs 2 bytes (16 bits); quantizing re-encodes them as 8-bit or 4-bit integers.

So the above table shows same model, same parameter count. Only the memory footprint for your model changes.
Every GB you don't spend on weights is a GB left for KV cache, which allows serving more concurrent requests resulting in more happy customers.

Already this helps guide decisions:

The fp16 model variant (~15 GB) plus workspace would nearly fill a 24 GB GPU before serving a single request.
The AWQ variant (~5.6 GB) leaves the majority of VRAM free for the KV cache.

Same model, same GPU — wildly different serving capacity.^^

In truth, the AWQ actually consumes ~5.6 GB, not the advertise ~3.8 GB above. This can be seen by looking at the sum of the .safetensors file sizes on the repo's Files and versions tab For this model that's exactly two files: model-00001-of-00002.safetensors (~4.0 GB) and model-00002-of-00002.safetensors (~1.6 GB). These are the weight tensors themselves — the files vLLM downloads and loads into VRAM at startup — so their combined size is the weights footprint (~5.6 GB).

So now you know your weights footprint, - its time to look choose a GPU that fits your requirements.

Below are links to some of the large cloud providers specification sheets.

GCP, Azure, AWS

NOTE — reserve room for the engine: vLLM pre-claims a fraction of total VRAM and is set using the --gpu-memory-utilization flag.

The vLLM engine fits weights + activations + KV cache inside that claim, leaving the remainder as headroom for CUDA overhead and fragmentation.

The default is 0.92; experiments showed that was too aggressive on our GPU and we run 0.85.

So for the first pass, we are going to use Azures Standard_NV36ads_A10_v5 processor. (Note VRAM is listed under the Accelerators Tab in the documents — the "Memory (GiB)" column on the Basics tab is the VM's system RAM, not the GPU's.)
| Name | Accelerators | VRAM (GB) |
|---|---|---|

| Standard_NV36ads_A10_v5 | 1 | 24 |

Step 4 — Determine your token budget.

Now the centerpiece.

On our A10 (24 GB): take 85% of it, subtract ~5.6 GB of weights and vLLM's measured activation/overhead reservation, and roughly 13.76 GiB remains for the KV cache.

How many tokens fit in that? Each token in flight stores a key and a value vector in every layer:

bytes_per_token* = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes
                = 2 × 28 × 4 × 128 × 2 (fp16)
                = 57,344 bytes ≈ 57 KB per token

[*Pope et al., 2022](https://arxiv.org/abs/2211.05102) 
Derivation assumes standard attention (MHA/MQA/GQA); sliding-window, MLA and hybrid SSM models cache differently and need a different formula.

Divide the remaining KV cache by the bytes per token:

Token budget  = 13.76 GiB / 57,344 bytes ≈ 257,584 tokens

And convert tokens into the unit you actually care about — concurrent requests (assuming ~1,000 tokens per request):

Concurrent_requests = 257,584 / 1,000 ≈ 258

One A10 can hold roughly 258 average requests in flight. That single number is what connects GPU shopping to capacity planning — it's the same Concurrent_requests the flapping article builds its theoretical autoscaling threshold from.

Step 5 — Pick the GPU from the no. of requests you will have to support. (Which is what you will likely be doing anyway)

In reality the requirement is the fixed thing (ie 100 concurrent requests at peak) and the hardware is what you get to choose.

So lets walk through how to solve for the GPU sizing from the given requirements

KV bytes needed = concurrent_requests × avg_tokens_per_request × bytes_per_token
VRAM target     = (KV bytes + weights) / gpu_memory_utilization

Requirements/Assumptions — 100 concurrent requests at ~1,000 tokens each, same token size as above since we have already decided our model:

KV bytes needed = 100 × 1,000 × 57,344 bytes ≈ 5.7 GB
VRAM target = (5.7 + 5.6) / 0.85         ≈ 13.3 GB

13.3 GB is your shopping floor: any card below it can't hold this workload, and the fractional A10 sizes (4/8/12 GB) are eliminated on the spot.
The A10's 24 GB clears it with room to spare — which is the useful answer, because "fits with headroom" is what lets you absorb a traffic spike without a second replica.

Now you have baseline requirements and can choose the model that fits those requirements BEFORE you've spent a dollar on hardware.

Step 6 — Verify at boot: don't trust the estimate

OK, now the cluster and service is alive with the desired GPU (infrastructure setup is covered in a companion piece — coming soon).
The truth at startup is finally available becuase the truth comes from vLLM itself — at startup it profiles the hardware and prints exactly what it measured:.

INFO ... Available KV cache memory: 13.76 GiB
INFO ... GPU KV cache size: 257,584 tokens

kubectl logs <vllm-pod> | grep -i "kv cache" and compare against your Step 4 numbers.
If they're close, your mental model of the card is correct.

If they're way off, something in your assumptions is wrong (usually the quantization variant or the gpu-memory-utilization value) — better to find out now than after you've sized a node pool around it.

OK - there you go ... after a couple of tries you're officially an expert at determining the correct GPU to handle your inference model.

Summary

Model — pick a candidate; everything below reads off its spec sheet, so you can run this on a whole shortlist.
Spec sheet — five values from the model card and config.json: parameter count, quantization, num_hidden_layers, num_key_value_heads, and hidden_size / num_attention_heads.
Weights — parameter count × bytes per parameter, then confirm against the real .safetensors file sizes (the rule of thumb runs low on quantized models).
Token budget — VRAM × gpu-memory-utilization, minus weights, ÷ bytes-per-token = tokens; ÷ tokens-per-request = Concurrent_requests. Note 0.85 is a safer starting point than vLLM's 0.92 default.
Calculate GPU from Requests SLO — start from the concurrency you actually need and solve for the VRAM target instead; anything below that number is off the shortlist.
Verify — the vLLM startup log is ground truth; grep it and reconcile against your Step 4 estimate.

Next in the series: that Concurrent_requests number is the foundation of a defensible autoscaling threshold — and why even a defensible threshold isn't enough: Your vLLM Autoscaler Is Flapping Because You Picked the Wrong Signal.

What about multiple GPUs?

Everything above sized one pod on one GPU. Fair question: what happens when the cluster has more than one? There are three scenarios here, and they're worth keeping distinct — because only one of them changes the math you just learned.

Scenario 1 — many single-GPU nodes (what this series runs). Each pod owns one GPU and holds a full copy of the model; traffic is load-balanced across replicas. This is data parallelism, and it's already a multi-GPU cluster — the flapping article scales exactly this fleet from 1 to 3 GPUs. Nothing in the sizing math changes: total capacity is simply C × replicas.

Scenario 2 — multi-GPU nodes, still one GPU per pod. Some VM sizes pack multiple cards (e.g. Azure's NV72ads_A10_v5 has 2× A10). Keep nvidia.com/gpu: "1" per pod and Kubernetes schedules two vLLM pods onto one node. The sizing math is still unchanged — each pod sees its own 24 GB card. This holds as long as GPU time-slicing is disabled; with it enabled, two pods can share one physical card with no memory isolation and none of the numbers above apply.

What does change is operational :

Scale-up gets faster — a new replica can land on the already-running node in seconds, instead of waiting minutes for node provisioning.
Cost granularity gets coarser — the node is the billing unit, so one busy pod next to an idle GPU slot still costs full price.
Blast radius grows — losing one node now kills two replicas.

Scenario 3 — multiple GPUs per pod: tensor parallelism. This is the one that changes the math. When the model doesn't fit on any single card — Llama-70B at fp16 is ~140 GB of weights alone — vLLM can split every layer across GPUs with --tensor-parallel-size N, and the pod requests nvidia.com/gpu: N. Three consequences for sizing:

Weights and KV cache are sharded across the N GPUs — weights divide roughly evenly, and the kv_heads term in the bytes-per-token formula divides across GPUs too. (Note the constraint: with Qwen's 4 KV heads, tensor parallelism beyond 4 stops dividing cleanly.)
Interconnect becomes a sizing axis. Every forward pass exchanges activations between the GPUs, so NVLink vs. plain PCIe meaningfully changes throughput. VRAM stops being the only number you shop on.
Autoscaling stakes go up. One replica now costs N GPUs, so every flap cycle is N× more expensive — the signal-choice lessons from the flapping article matter more here, not less.

The clean rule of thumb: more traffic → more replicas (Scenarios 1–2); bigger model → tensor parallelism (Scenario 3). Reach for Scenario 3 only when weights plus a workable KV budget exceed the biggest single card you can buy — otherwise replicas are simpler, cheaper, and fail more gracefully.

Gotchas

The 0.92 default failed on our A10. vLLM tried to claim 21.82 GiB but only ~21.37 was actually free → engine init crashed. The model fit fine — this was a headroom knob, not a "need a bigger GPU" problem. --gpu-memory-utilization 0.85 fixed it.
The weights rule of thumb underestimates quantized models. params × 0.5 bytes says 3.8 GB; the real artifact is ~5.6 GB (fp16 embeddings + quantization scales). Check the actual file sizes on the repo.
Crash-looping pods leak VRAM. If the vLLM pod crashes and restarts, the retry can fail on less-than-expected free VRAM. Delete the pod so the VRAM fully releases before the next attempt.

how to host GPUS on Azure using KEDA

Josef Doornink — Tue, 11 Aug 2026 17:52:59 +0000

Your vLLM Autoscaler Is Flapping Because You Picked the Wrong Signal — Not the Wrong Number

Josef Doornink — Mon, 10 Aug 2026 18:11:56 +0000

Your vLLM Autoscaler Is Flapping Because You Picked the Wrong Signal — Not the Wrong Number

Part of a series on running vLLM on AKS. Companion piece: GPU sizing. Infrastructure setup — coming soon.

So... you've picked the right GPU for your use case, provisioned the cluster and node pool, deployed your model behind vLLM, installed KEDA and picked the request-queue as the autoscaling parameter as per the common recommendation: "Tune the request queue to obtain the preferred latency, and use batch size if you can't hit your preferred latency."

Then you watch it in production:

A burst hits, a new pod spins up — and by the time it's ready, the burst is gone. The queue drained, so the autoscaler scales back down.
Minutes later the next spike repeats the cycle. The replica count oscillates instead of settling.
What gives??

That's flapping and it's costing you.

What flapping is, and why you should care

Flapping refers to a resource rapidly switching between two states and in this case: a pod Up/Running or Down/Failed.

I can hear you say: "So what ... The requests still get served, right?"

True, but flapping costs you:

Money — you pay for GPU nodes that spin up and tear down without ever serving traffic.
Latency — the mechanism meant to absorb load spends its time flapping instead. Requests sit in the queue while pods churn.
Trust — pods and nodes appearing and vanishing "at random" is exactly the chaos you don't want when debugging under pressure.

This article shows how to avoid flapping by using the right type of signal and why request-queue is the wrong choice.

We demonstrate this by walking through three progressive designs for the autoscaler and its defining parameter(s), reasoning about the output, and finally coming to a conclusion about a signal that actually holds steady, and why.

The three progressive steps we will walk through are:

Attempt 1 — arbitrary threshold (flaps, and over-scales)
Attempt 2 — calculated threshold (still flaps)
Attempt 3 — stabilized design (stable)

Setup

Cloud: Azure AKS 1.35.6
GPU node: Standard_NV36ads_A10_v5 (1× A10, 24 GB)
Image / model: vllm/vllm-openai:latest / Qwen2.5-7B-Instruct-AWQ
Autoscaler: KEDA ScaledObject, Prometheus trigger on vllm:num_requests_waiting
Visibility: Prometheus + Grafana

- Constraint: `nvidia.com/gpu: "1"` → one pod per GPU node, so every replica needs a new node (this matters later)

Attempt 1 — an arbitrary threshold (flaps, and over-scales)

Pick a low, round queue number that sounds reasonable (5) and move on — sound familiar?

This over-scales badly because the underlying Kubernetes HPA controller — which KEDA delegates the actual scaling decision to — computes desiredReplicas = ceil(total_waiting / threshold) (Kubernetes HPA algorithm details). So a transient queue of 40 demands 8 replicas. Fine for a proof-of-concept, but in production each of those replicas cost you.

It's better to calculate the threshold from the hardware specs, not a guess.

So ... let's calculate a real one.

Here is the scaledobject for attempt1 if you are curious. scaledobject-attempt1-guess.yaml

Attempt 2 — a calculated threshold (still flaps)

Assumptions:

Average tokens/request ≈ 1,000
Average completion time = 10s (how long it takes vLLM to generate a response once processing starts — prefill + decode)
Queue-wait budget W_max ≤ 5 seconds (SLO)

To calculate the threshold, we first need to determine how much cache is available to our model

Read from the model's startup log (the engine profiles this at boot): this is VRAM left after weights + activation + overhead.

Result: Available KV cache memory: 13.76 GiB

Next we determine the bytes per token based on the model's specifications (all from the model's config.json):

bytes_per_token = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes
                = 2 × 28 × 4 × 128 × 2 (fp16) = 57,344

Calculate how many tokens are available given the cache size and the bytes required per token
257,584 ≈ 13.76 GiB / 57,344 bytes_per_token

Determine how many requests our GPU can handle at once given our assumption(s) (~1,000 tok/req)
Concurrent_requests = 257,584 tokens / 1,000 tokens/request ≈ 258

(Full VRAM → token-budget derivation is in the GPU sizing companion piece.)

Finally, calculate the Threshold from an SLO (Little's Law — how many requests may queue before wait breaches W_max):

# Calculate how many requests will be completed every second
Drain_rate = Concurrent_requests / avg_completion_time = 258 / 10s ≈ 25.8 req/s

# Calculate the threshold to increase the pod number by 1.
Threshold = W_max × Drain_rate = 5s × 25.8 ≈ 129

129 is a defensible number, derived from measured capacity and a stated SLO. Not a guess.

However, for demonstration purposes, the rest of this article runs on a deliberately shrunk rig:

Demo vs. production numbers: --max-num-seqs 32 caps concurrency at 32 so the queue is reachable at demo scale. Measuring with unique per-request prompts gave a real drain rate of μ ≈ 0.46 req/s; with a W_max = 15s budget, the same formula calibrates to T = 15 × 0.46 ≈ 7, versus C ≈ 258 / T ≈ 129 for the full-size pod. Don't get caught up on the smaller numbers — same method, same core point; it's just easier to demonstrate the flapping behavior and saves me money. deployment.yaml

Apply the ScaledObject (scaledobject-attempt2-calculated.yaml):

triggers:
  - type: prometheus
    metadata:
      serverAddress: http://kps-kube-prometheus-stack-prometheus.monitoring.svc:9090
      query: sum(vllm:num_requests_waiting)
      threshold: "7"

Apply the load (load_test.py):

Let's walk through what happened:

Requests start arriving, running count climbs to 32 (one pod's cap) — no queue yet, all requests are actually running.
Once running is saturated at 32, new arrivals start queuing instead. The queue grows.
Queue crosses 7 (the threshold) a little before 20:00 → KEDA scales to 2 replicas.
While that second pod is still loading (model weights + readiness, ~90s), the queue keeps growing — the new pod isn't serving yet, so it does nothing to relieve the backlog. By the time it's ready, the queue is way past 7 (peak ~50).
KEDA polls again, sees the queue still far over threshold, and scales to 3 pods.
Now with more capacity coming online, the queue starts draining. It falls below threshold, and KEDA scales back down (2, then 1) — the queue graph and replica graph fall together around 20:04–20:07.
The load never stopped: it's the same continuous arrival rate (0.6 req/s) throughout. The moment replicas dropped to 1 again, that single pod was immediately outmatched by the same steady arrivals, so the queue re-formed and crossed 7 again almost immediately (~20:08–20:10) — a second flap cycle.
Eventually the load generator's 12-minute run ends, arrivals stop, the queue drains for real, and replicas settle back to the floor of 1.

Under steady load, the oscillation isn't coming from the traffic — it's coming from the shape of the signal.

The queue reads ~0 until the pod is full, then climbs; the autoscaler is just asking "is this threshold met or not?"

The binary nature of the signal is what causes the flap.

Why you might not have noticed this in the wild: Kubernetes' HPA defaults to a 300s scale-down stabilization window (0s scale-up), so out of the box the flap is slowed to a ~5-minute cycle — infrequent enough to miss at a glance, but you pay for it on every swing. (Kubernetes HPA defaults)
For this demo, stabilizationWindowSeconds was set to 0 in Attempt 2, to expose the flap within a short window instead of stretching it to 5+ minutes:
advanced:
  horizontalPodAutoscalerConfig:
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 0

So even picking the "right" queue-depth number doesn't save you.
Is there a different metric we can use instead??

Now you're asking the right questions.

Attempt 3 — the stabilized design (stable)

Two requirements to stabilize the system:

Scale on a signal that moves before trouble, and
Damp scale-down so a dip can't tear down capacity you just paid for.

First, let's look at some metrics and rank vLLM's metrics by what they tell you and when:

Signal	Shape	Tells you
`num_requests_waiting`	0 until full, then climbs	you're already overloaded (lagging)
`num_requests_running`	proportional, flat once at `max_num_seqs`	how busy, up to the cap
`kv_cache_usage_perc`	smooth 0–1 (fraction)	*how close to full — before* the queue forms (leading)**

KV-cache utilization is the right signal for vLLM because it rises before requests queue, and it is the binding resource the capacity math is about.
Scaling up at ~80% buys you lead time — you add capacity while the current pod still has headroom, INSTEAD OF AFTER it's drowning.

triggers:
  - type: prometheus
    metadata:
      serverAddress: http://kps-kube-prometheus-stack-prometheus.monitoring.svc:9090
      query: sum(vllm:kv_cache_usage_perc)   # V1 engine name; older builds use gpu_cache_usage_perc
      threshold: "0.80"

IMPORTANT: Use sum, not avg — this is the part that actually kills the flap. sum(kv_cache_usage_perc) measures total demand across pods (in "pod-fulls"), which doesn't change when you add a replica — so ceil(sum / 0.80) has a stable fixed point. An avg falls as you add pods, drops back under the threshold, and reintroduces the exact oscillation you're trying to kill. That's the deeper rule behind this whole article: a good autoscaling signal is one whose total is invariant to replica count. num_requests_waiting fails it (collapses to 0 once you have enough capacity); sum(kv_cache_usage_perc) passes it.

OK, so how do we dampen the scale-down?

Pair the signal with scale-down damping ≥ your measured lead time L, so a momentary dip never tears down capacity you just paid minutes to bring up:

advanced:
  horizontalPodAutoscalerConfig:
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 120   # >= L (node + engine init + readiness)

Stable ScaledObject scaledobject-attempt3-stabilized.yaml:

Apply the load again (load_test.py):

We now have a stable scale-up for processing.
It works without trying to find the exact right threshold from the request queue!
We don't even need to walk through the different points — the autoscaler handles it exactly like we want it to because we chose the right signal!

Oh, big deal... "the docs already say to add a cache trigger"

They do — and that's the trap. Some (e.g. this guide) suggest keeping the queue-depth trigger and adding a second cache trigger, assuming the two combine into "scale only if both agree."

They don't, by default. KEDA evaluates each trigger independently and takes the max of their desired replica counts — that's OR logic, not AND. So the lagging, bistable queue-depth trigger can still drive scale-up and scale-down on its own, and the flap comes right back. (KEDA docs)

If you want the leading signal to govern scaling, use it alone — drop the queue-depth trigger. If you genuinely need both conditions, don't just list two triggers and hope: that silently keeps the OR/max default. AND-style logic is possible, but only if you explicitly use the opt-in scalingModifiers.formula to combine the named triggers yourself. Adding a plain second trigger feels safer and quietly isn't.

Summary

Flapping is a signal-choice bug wearing a threshold costume. In order of impact:

Scale on a leading, proportional signal (kv_cache_usage_perc), not a lagging saturation one (num_requests_waiting).
Query it as a sum, not an average — sum(vllm:kv_cache_usage_perc) measures total demand across pods, which doesn't change when you add a replica. Additional Suggestions:
Use scalingModifiers if you need to combine metrics. Example here: scaledobject-scalingmodifiers.yaml.
Size scale-down damping to your actual lead time.

The final working scaledObject used for Attempt 3 can be seen here — trigger, threshold, and damping in one place.

You will likely still want to calculate your threshold, but — it's the last decision, not the first.
Get the signal right and most of the flapping is gone before you tune anything.

Alternatives

Keep pods warm

You may not need the perfect signal. The cheapest lever against the timescale mismatch is over-provisioning — set minReplicaCount > 1 so warm replicas absorb bursts without waiting minutes for a cold node.

What it fixes: the lead-time problem directly, and it makes any residual flapping cheaper (no cold starts) and rarer (you scale from a higher floor). What it doesn't fix: the marginal 2→3 decision still rides whatever signal you chose. And it costs money — idle GPUs are the expensive kind.

For a small deployment, warm headroom + a reasonable threshold is often good enough — the leading-signal work is where it pays off at scale or when you're cost-sensitive.

More pods per node

If one-pod-per-GPU-node causes node churn, why not pack several pods per node? A couple of reasons this doesn't help:

Flapping is a control-loop stability problem — pack three pods per GPU and the replica count still oscillates, just faster.
Each pod gets a fraction of the 24 GB, so a fraction of the KV cache and a lower concurrent-request capacity (C).

What changes at real scale

Everything above is one GPU — basically a teaching rig. Once you're running a real fleet, the strategy actually flips.

Instead of reacting fast to every queue blip, you want to over-provision warm headroom and scale infrequently. If possible, use predictive scaling — forecasting on daily/weekly patterns, not waiting on any live metric — for the traffic you can actually see coming, and let reactive scaling handle only the residual. Even that reactive layer should be the leading-signal kind from Attempt 3, not lagging queue-depth.

What's next: KV-cache-aware routing

That's a topic for another article, but worth knowing it exists: at fleet scale, how you route requests across pods (e.g. KV-cache-aware routing) becomes another lever alongside scaling. Not a replacement for getting the scaling signal right — just the next layer once you have more than a few pods.

DEV Community: Josef Doornink

Standing Up a GPU Cluster on AKS for vLLM

Setup Summary

Dependency chain

Step 0 — Prerequisites (one-time, survives resource group deletion)

Step 1 — Create Resource group

Step 2 — Create AKS cluster, on a CPU system pool

Step 3 — GPU node pool, tainted and scaled from zero

Step 4 — Get kubeconfig for cluster communication

Step 5 — NVIDIA device plugin

Step 6 — Observability core: Prometheus + Grafana + KEDA

Step 7 — Deploy vLLM

Step 8 — Service + smoke test

Step 9 — GPU metrics: DCGM exporter

Step 10 — Wire vLLM into Prometheus

Step 11 — Access Grafana

Verification checklist

Teardown

Gotchas

Summary

Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess

Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess

OK, you're a senior SRE, you've been hearing incessantly about AI models, but aren't quite sure how to determine the correct node size to host your model. - If so ... you're in the right place.

Background: What actually consumes GPU memory

OK, lets get started

Step 1 — Choose a model

Step 2 — What to look for in the spec sheet

Step 3 — Calculate the VRAM for the weights

| Standard_NV36ads_A10_v5 | 1 | 24 |

Step 4 — Determine your token budget.

Step 5 — Pick the GPU from the no. of requests you will have to support. (Which is what you will likely be doing anyway)

Step 6 — Verify at boot: don't trust the estimate

OK - there you go ... after a couple of tries you're officially an expert at determining the correct GPU to handle your inference model.

Summary

What about multiple GPUs?

Gotchas

how to host GPUS on Azure using KEDA

Your vLLM Autoscaler Is Flapping Because You Picked the Wrong Signal — Not the Wrong Number

Your vLLM Autoscaler Is Flapping Because You Picked the Wrong Signal — Not the Wrong Number

What flapping is, and why you should care

Setup

- Constraint: nvidia.com/gpu: "1" → one pod per GPU node, so every replica needs a new node (this matters later)

Attempt 1 — an arbitrary threshold (flaps, and over-scales)

Attempt 2 — a calculated threshold (still flaps)

To calculate the threshold, we first need to determine how much cache is available to our model

Attempt 3 — the stabilized design (stable)

Oh, big deal... "the docs already say to add a cache trigger"

Summary

Alternatives

Keep pods warm

More pods per node

What changes at real scale

What's next: KV-cache-aware routing

- Constraint: `nvidia.com/gpu: "1"` → one pod per GPU node, so every replica needs a new node (this matters later)