DEV Community

iapilgrim
iapilgrim

Posted on

Building an AKS + ACI Virtual Node Lab (Step-by-Step)

#aks #aci #fargate

In this tutorial, we’ll build a hybrid Kubernetes architecture using:

  • Azure Kubernetes Service (AKS)
  • Azure Container Instances (ACI)
  • AKS Virtual Nodes (serverless burst capacity)

By the end, you’ll have:

  • A working AKS cluster
  • A delegated subnet for ACI
  • Virtual Node enabled
  • A pod successfully running on ACI
  • A troubleshooting checklist for common failures

🏗 Architecture Overview 

AKS Cluster
│
├── VM Node Pool (steady workloads)
│   └── baseline-app
│
└── Virtual Node (ACI-backed)
    └── aci-test
Enter fullscreen mode Exit fullscreen mode

Key behavior:

  • VM-backed pods → 10.240.x.x
  • ACI-backed pods → 10.3.x.x (delegated subnet)

That IP difference proves true network separation.

🧰 Prerequisites

  • Azure CLI logged in
  • Subscription with permissions
  • Kubernetes CLI (kubectl)
  • Region that supports ACI

🛠 Step 1 — Register Required Providers

This step is commonly missed and causes ProviderFailed.

az provider register --namespace Microsoft.ContainerInstance
az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.Network
Enter fullscreen mode Exit fullscreen mode

Verify:

az provider show --namespace Microsoft.ContainerInstance --query registrationState -o tsv
Enter fullscreen mode Exit fullscreen mode

Should return:

Registered
Enter fullscreen mode Exit fullscreen mode

🛠 Step 2 — Create VNet + Delegated Subnet

ACI requires subnet delegation.

RG=rg-aks-agic-demo

LOCATION=eastus2

az network vnet create \
  -g $RG \
  -n aks-vnet \
  --address-prefix 10.0.0.0/8 \
  --subnet-name aks-subnet \
  --subnet-prefix 10.240.0.0/16

az network vnet subnet create \
  --resource-group $RG \
  --vnet-name $VNET_NAME \
  --name aci-subnet \
  --address-prefix 10.3.0.0/16 \
  --delegations Microsoft.ContainerInstance/containerGroups
Enter fullscreen mode Exit fullscreen mode

🛠 Step 3 — Enable Virtual Node Add-On

This is the key step many forget.

az aks enable-addons \
  -g $RG \
  -n aks-aci-cluster \
  --addons virtual-node \
  --subnet-name aci-subnet
Enter fullscreen mode Exit fullscreen mode

Verify:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

You should see something like:

kubectl get nodes
NAME                                STATUS   ROLES    AGE     VERSION
aks-nodepool1-38580963-vmss000000   Ready    <none>   4h16m   v1.33.6
aks-nodepool1-38580963-vmss000001   Ready    <none>   4h16m   v1.33.6
virtual-node-aci-linux              Ready    agent    27m     v1.25.0-vk-azure-aci-1.6.2
Enter fullscreen mode Exit fullscreen mode

🛠 Step 5 — Deploy a Pod to Virtual Node

Create:

apiVersion: v1
kind: Pod
metadata:
  name: aci-test
spec:
  containers:
  - name: aci-test
    image: mcr.microsoft.com/azuredocs/aci-helloworld
  nodeSelector:
    kubernetes.io/role: agent
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f aci-test.yaml
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

Expected:

aci-test   1/1 Running   10.3.x.x   virtual-node-aci-linux
Enter fullscreen mode Exit fullscreen mode

Logs:

kubectl logs -f aci-test
Enter fullscreen mode Exit fullscreen mode

🧪 Validation Checklist

Check Expected Result
kubectl get nodes Shows virtual-node-aci-linux
Pod IP In ACI subnet range
Pod NODE virtual-node-aci-linux
Logs stream Successfully

🧯 Troubleshooting Guide

❌ Pod Stuck in ProviderFailed

Cause 1: Provider Not Registered

Fix:

az provider register --namespace Microsoft.ContainerInstance
Enter fullscreen mode Exit fullscreen mode

❌ Pod Stuck in Pending

Cause 2: Missing Toleration

Virtual node is tainted.

Add:

tolerations:
- key: virtual-kubelet.io/provider
  operator: Exists
Enter fullscreen mode Exit fullscreen mode

❌ Error: Subnet Not Delegated

Check:

az network vnet subnet show \
  -g $RG \
  --vnet-name aks-vnet \
  -n aci-subnet \
  --query delegations
Enter fullscreen mode Exit fullscreen mode

Must show:

Microsoft.ContainerInstance/containerGroups
Enter fullscreen mode Exit fullscreen mode

❌ Virtual Node Not Appearing

Check addon:

az aks show -g $RG -n aks-aci-cluster --query addonProfiles.virtualNode
Enter fullscreen mode Exit fullscreen mode

If disabled:

az aks enable-addons \
  -g $RG \
  -n aks-aci-cluster \
  --addons virtual-node \
  --subnet-name aci-subnet
Enter fullscreen mode Exit fullscreen mode

❌ IP Not in ACI Subnet

If pod IP looks like VM subnet:

  • NodeSelector incorrect
  • Toleration missing
  • Pod scheduled to VM pool

Check:

kubectl describe pod aci-test
Enter fullscreen mode Exit fullscreen mode

Look at Node: field.

💡 When to Use Virtual Nodes

✅ Burst scaling
✅ Cost optimization
✅ Multi-tenant sandbox workloads
✅ Untrusted job execution
✅ CI/CD runners

Avoid for:

❌ DaemonSets
❌ Privileged containers
❌ Stateful workloads

💰 Cost Model Insight

  • VM nodes → Always billed
  • ACI → Per second billing
  • Ideal for unpredictable workloads

🧹 Cleanup 

az group delete -n $RG --yes --no-wait
Enter fullscreen mode Exit fullscreen mode

🏁 Final Result

You now have a hybrid AKS architecture:

  • Stable workloads on VM nodes
  • Serverless burst capacity via ACI
  • Isolated networking
  • Production-ready troubleshooting knowledge

Top comments (0)