DEV Community

Fully automated creation of an AAD-integrated Kubernetes cluster with Terraform

Christian Dennig on December 15, 2019

Introduction To run your Kubernetes cluster in Azure integrated with Azure Active Directory as your identity provider is a best practice...

Read full post

Toan Nguyen • Mar 5 '20 • Edited

Your article is super helpful! I have read a lot of samples, but your article is the best.

Notes: Please make sure the account to run the terraform script having the role "Owner" to run the "azurerm_role_assignment" or you will get an error "does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write"

Sebastien Gissinger • May 3 '20

You can also give it the "User Access Administrator" role. It has less permissions than "Owner".

Christian Dennig • Apr 24 '20

Thank you for your feedback!

BarnumD • Apr 7 '20

The kubernetes_cluster_role_binding - aad_integration was enough to get me logged into the dashboard, but then there was a bunch of errors like configmaps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list resource "configmaps" in API group "" at the cluster scope. I hadd to add the following for that to work

resource "kubernetes_cluster_role_binding" "service_account" {

  metadata {
    name = "${module.lbl_default.id}-service-account"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "kubernetes-dashboard"
    namespace = "kube-system"
  }
  depends_on = [
    azurerm_kubernetes_cluster.aks
  ]
}

Christian Dennig • Apr 24 '20

Be careful with running the dashboard as „cluster-admin“. It a very „popular“ attack vector!

Zenifer Cheruveettil • Jun 15 '20 • Edited

now AD integration is available as a preview feature..I am a bit unsure in this case whether terraform is the best way. A few az commands looks much more readable than a full page HCL code, granted I will lose the declarative way of spinning the cluster. Any thoughts?

c4m4 • Jul 14 '20

You can already use: role_based_access_control {
enabled = "true"
azure_active_directory {
managed = true
}
}

Christian Dennig • Jun 18 '20 • Edited

My bet: they will soon have it also in Terraform. That will make the whole thing much cleaner.

Tim Brigley • Feb 7 '20

Thanks very much for this, especially the “magic permission GUIDs”, they help solve some automation that we've been wanting, and your detailed explanation here is extremely helpful.

Also, I wanted to add a link to the guide and code for creating the original service principle using cloud shell. I thought my existing SP had proper access until encountering some unauthorized errors.

terraform.io/docs/providers/azurea...

-T

Piers Karsenbarg • Feb 2 '21

I wanted to add some extra automation, but I'm getting stuck behind the To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ASGRA765S to authenticate.

What needs to be done to get around that?

c4m4 • Jul 10 '20

I fighting with this error: ad_integration.tf line 61, in resource "azuread_application" "aks-aad-client":
61: id = azuread_application.aks-aad-srv.oauth2_permissions.0.id

This value does not have any indices.

c4m4 • Jul 13 '20

Solved using id = tolist(azuread_application.aks-aad-srv.oauth2_permissions)[0].id

Viresh Doshi • Mar 28 '20

Very useful and clear! I am in the process of building a secure AKS and it’s good to get some better insight into the terraform aspect of using AAD

dd2repo • Mar 26 '20

With kube_admin_config i always get: Failed to configure: tls: failed to find any PEM data in certificate input. kube_config is working fine. Any ideas?

Christopher Grainger • Apr 10 '20

It's most likely base64 encoded. Try wrapping in base64decode().

Dimitar Elenkov • Sep 10 '20

Greetings, the article is a really good one.
I have an issue during the deployment when Terraform is working on the phase with resource "kubernetes_cluster_role_binding" "aad_integration".
The deployment failed with the following error:

Error: Post "terraaks01aad-f165a586.hcp.westeur... acquiring a token for authorization header: acquiring a new fresh token: initialing the device code authentication: autorest/adal/devicetoken: Error occurred while handling response from the Device Endpoint: Error HTTP status != 200

on main.tf line 251, in resource "kubernetes_cluster_role_binding" "aad_integration":
251: resource "kubernetes_cluster_role_binding" "aad_integration" {

Any ideas about that issue. Thanks in advance.

Dimitar Elenkov • Sep 11 '20

I was able to fix that issue. Deployment works after I changed the Terraform kubernetes provider section to the following:
provider "kubernetes" {
load_config_file = false

host = azurerm_kubernetes_cluster.aks.kube_admin_config.0.host
username = azurerm_kubernetes_cluster.aks.kube_admin_config.0.username
password = azurerm_kubernetes_cluster.aks.kube_admin_config.0.password

client_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_certificate)
client_key = base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_key)
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.cluster_ca_certificate)
}

MaxJ • Feb 26 '21 • Edited

I love your article. By far the best instructions I found.

Turns out there is a breaking change in how terraform handles the oauth2_permissions github.com/hashicorp/terraform-pro...

What is working for me:
id = [for permission in azuread_application.appreg-aicloud-aks-server.oauth2_permissions : permission.id][0]
instead of:
id = "${azuread_application.aks-aad-srv.oauth2_permissions.0.id}"