DEV Community

iapilgrim
iapilgrim

Posted on

Troubleshooting Azure Application Gateway Ingress Controller 403 Error

#azure #kubernetes #networking #tutorial

🛠

Fixing ErrorApplicationGatewayForbidden in AKS + Application Gateway

When running:

  • Azure Kubernetes Service
  • Azure Application Gateway
  • Azure Application Gateway Ingress Controller

You may encounter this error inside AGIC logs:

ErrorApplicationGatewayForbidden
StatusCode=403
Microsoft.Network/applicationGateways/read
Enter fullscreen mode Exit fullscreen mode

This tutorial explains:

  1. ✅ Why this happens
  2. 🔍 How to diagnose it
  3. 🛠 How to fix it properly
  4. 🔐 How to prevent it next time

📌 The Error

From AGIC logs:

kubectl logs -n kube-system deploy/ingress-appgw-deployment
Enter fullscreen mode Exit fullscreen mode

You’ll see something like:

Unexpected status code '403' while performing a GET on Application Gateway
AuthorizationFailed
Enter fullscreen mode Exit fullscreen mode

And:

The client '<object-id>' does not have authorization to perform action
'Microsoft.Network/applicationGateways/read'
Enter fullscreen mode Exit fullscreen mode

🧠 Root Cause

AGIC runs using a managed identity.

That identity must have Azure RBAC permissions to:

  • Read Application Gateway
  • Modify listeners
  • Modify backend pools
  • Modify routing rules

If those permissions are missing → AGIC cannot configure App Gateway → 403 error.

This usually happens when:

  • App Gateway was created manually
  • Cross-resource-group setup
  • Subscription policies restrict automatic role assignment
  • Infrastructure created in separate steps

🏗 How AGIC Actually Works (Important to Understand)

Flow:

Ingress YAML
   ↓
AGIC watches cluster
   ↓
Calls Azure API
   ↓
Modifies Application Gateway config
Enter fullscreen mode Exit fullscreen mode

AGIC is essentially a controller that:

  • Talks to Azure ARM API
  • Updates App Gateway dynamically

Without RBAC → it cannot call Azure API.

🔍 Step 1 — Identify AGIC Managed Identity

Run:

az aks show \
  --resource-group <RG> \
  --name <AKS_NAME> \
  --query addonProfiles.ingressApplicationGateway.identity.objectId \
  -o tsv
Enter fullscreen mode Exit fullscreen mode

Save the output:

AGIC_OBJECT_ID=<value>
Enter fullscreen mode Exit fullscreen mode

That is the identity failing in the logs.

🔎 Step 2 — Verify Missing Role

Check current assignments:

az role assignment list \
  --assignee $AGIC_OBJECT_ID \
  -o table
Enter fullscreen mode Exit fullscreen mode

You’ll likely see:

  • No Contributor on App Gateway
  • No Reader on Resource Group

That’s the problem.

🛠 Step 3 — Fix RBAC Properly

✅ 1. Assign Reader on Resource Group 

az role assignment create \
  --assignee $AGIC_OBJECT_ID \
  --role Reader \
  --scope /subscriptions/<SUB_ID>/resourceGroups/<RG>
Enter fullscreen mode Exit fullscreen mode

Why?

AGIC reads resource group metadata.

✅ 2. Assign Contributor on Application Gateway

Get App Gateway ID:

APPGW_ID=$(az network application-gateway show \
  --name <APPGW_NAME> \
  --resource-group <RG> \
  --query id -o tsv)
Enter fullscreen mode Exit fullscreen mode

Assign Contributor:

az role assignment create \
  --assignee $AGIC_OBJECT_ID \
  --role Contributor \
  --scope $APPGW_ID
Enter fullscreen mode Exit fullscreen mode

Why Contributor?

AGIC must:

  • Update listeners
  • Update backend pools
  • Update HTTP settings
  • Update routing rules

Reader is not enough.

🔄 Step 4 — Restart AGIC 

kubectl rollout restart deployment ingress-appgw-deployment -n kube-system
Enter fullscreen mode Exit fullscreen mode

Then check logs again:

kubectl logs -n kube-system deploy/ingress-appgw-deployment
Enter fullscreen mode Exit fullscreen mode

The error should disappear.

You should now see:

Applied App Gateway configuration
Enter fullscreen mode Exit fullscreen mode

🧪 Step 5 — Validate End-to-End

Test your endpoint:

https://<front-door-or-appgw-url>
Enter fullscreen mode Exit fullscreen mode

Traffic should now:

Front Door (optional)
   ↓
Application Gateway
   ↓
AKS
Enter fullscreen mode Exit fullscreen mode

No more retry loop.

🛡 Production Best Practice (Important)

Instead of giving full Contributor on the entire resource group:

🔐 Use Least Privilege

Scope Contributor only to:

/resourceGroups/<RG>/providers/Microsoft.Network/applicationGateways/<APPGW_NAME>
Enter fullscreen mode Exit fullscreen mode

Even better:

Create a custom RBAC role limited to:

Microsoft.Network/applicationGateways/*
Enter fullscreen mode Exit fullscreen mode

For finance / banking / regulated environments.

🚨 Common Variations of This Problem

Symptom Cause
AGIC keeps retrying Missing RBAC
403 only on update Missing Contributor
Works initially, fails later Identity changed
Cross-subscription setup Wrong scope

🧠 Prevention Checklist (Use This Next Time)

When creating AKS with AGIC:

az aks create \
  --enable-addons ingress-appgw \
  --appgw-id <ID>
Enter fullscreen mode Exit fullscreen mode

Immediately after:

  1. Get AGIC identity
  2. Assign:
  • Reader on RG
  • Contributor on App Gateway
    1. Verify with az role assignment list

Make this part of your infrastructure checklist.

🏦 Enterprise Architecture Insight

In large organizations:

  • Network team owns Application Gateway
  • Platform team owns AKS
  • RBAC must be explicitly granted

This error is extremely common in enterprise environments.

Understanding it makes you significantly stronger in Azure architecture.

🎯 Final Summary

Problem

AGIC 403 AuthorizationFailed

Cause

Managed identity missing RBAC permissions

Fix

Assign:

  • Reader → Resource Group
  • Contributor → Application Gateway

Result

AGIC can successfully reconcile Ingress → App Gateway config

Top comments (0)