DEV Community

Fully automated creation of an AAD-integrated Kubernetes cluster with Terraform

Christian Dennig on December 15, 2019

Introduction To run your Kubernetes cluster in Azure integrated with Azure Active Directory as your identity provider is a best practice...
Collapse
 
toanxyz profile image
Toan Nguyen • Edited

Your article is super helpful! I have read a lot of samples, but your article is the best.

Notes: Please make sure the account to run the terraform script having the role "Owner" to run the "azurerm_role_assignment" or you will get an error "does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write"

Collapse
 
cdennig profile image
Christian Dennig

Thank you for your feedback!

Collapse
 
sgissinger profile image
Sebastien Gissinger

You can also give it the "User Access Administrator" role. It has less permissions than "Owner".

Collapse
 
senseiwu profile image
Zenifer Cheruveettil • Edited

now AD integration is available as a preview feature..I am a bit unsure in this case whether terraform is the best way. A few az commands looks much more readable than a full page HCL code, granted I will lose the declarative way of spinning the cluster. Any thoughts?

Collapse
 
c4m4 profile image
c4m4

You can already use: role_based_access_control {
enabled = "true"
azure_active_directory {
managed = true
}
}

Collapse
 
cdennig profile image
Christian Dennig • Edited

My bet: they will soon have it also in Terraform. That will make the whole thing much cleaner.

Collapse
 
barnumd profile image
BarnumD

The kubernetes_cluster_role_binding - aad_integration was enough to get me logged into the dashboard, but then there was a bunch of errors like configmaps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list resource "configmaps" in API group "" at the cluster scope. I hadd to add the following for that to work

resource "kubernetes_cluster_role_binding" "service_account" {

  metadata {
    name = "${module.lbl_default.id}-service-account"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "kubernetes-dashboard"
    namespace = "kube-system"
  }
  depends_on = [
    azurerm_kubernetes_cluster.aks
  ]
}
Collapse
 
cdennig profile image
Christian Dennig

Be careful with running the dashboard as „cluster-admin“. It a very „popular“ attack vector!

Collapse
 
rktbrigley profile image
Tim Brigley

Thanks very much for this, especially the “magic permission GUIDs”, they help solve some automation that we've been wanting, and your detailed explanation here is extremely helpful.

Also, I wanted to add a link to the guide and code for creating the original service principle using cloud shell. I thought my existing SP had proper access until encountering some unauthorized errors.

terraform.io/docs/providers/azurea...

-T

Collapse
 
pierskarsenbarg profile image
Piers Karsenbarg

I wanted to add some extra automation, but I'm getting stuck behind the To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ASGRA765S to authenticate.

What needs to be done to get around that?

Collapse
 
c4m4 profile image
c4m4

I fighting with this error: ad_integration.tf line 61, in resource "azuread_application" "aks-aad-client":
61: id = azuread_application.aks-aad-srv.oauth2_permissions.0.id

This value does not have any indices.

Collapse
 
c4m4 profile image
c4m4

Solved using id = tolist(azuread_application.aks-aad-srv.oauth2_permissions)[0].id

Collapse
 
vireshdoshi profile image
Viresh Doshi

Very useful and clear! I am in the process of building a secure AKS and it’s good to get some better insight into the terraform aspect of using AAD

Collapse
 
dielenko profile image
Dimitar Elenkov

Greetings, the article is a really good one.
I have an issue during the deployment when Terraform is working on the phase with resource "kubernetes_cluster_role_binding" "aad_integration".
The deployment failed with the following error:

Error: Post "terraaks01aad-f165a586.hcp.westeur... acquiring a token for authorization header: acquiring a new fresh token: initialing the device code authentication: autorest/adal/devicetoken: Error occurred while handling response from the Device Endpoint: Error HTTP status != 200

on main.tf line 251, in resource "kubernetes_cluster_role_binding" "aad_integration":
251: resource "kubernetes_cluster_role_binding" "aad_integration" {

Any ideas about that issue. Thanks in advance.

Collapse
 
dielenko profile image
Dimitar Elenkov

I was able to fix that issue. Deployment works after I changed the Terraform kubernetes provider section to the following:
provider "kubernetes" {
load_config_file = false

host = azurerm_kubernetes_cluster.aks.kube_admin_config.0.host
username = azurerm_kubernetes_cluster.aks.kube_admin_config.0.username
password = azurerm_kubernetes_cluster.aks.kube_admin_config.0.password

client_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_certificate)
client_key = base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_key)
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.cluster_ca_certificate)
}

Collapse
 
dd2repo profile image
dd2repo

With kube_admin_config i always get: Failed to configure: tls: failed to find any PEM data in certificate input. kube_config is working fine. Any ideas?

Collapse
 
cigrainger profile image
Christopher Grainger

It's most likely base64 encoded. Try wrapping in base64decode().

Collapse
 
if54uran profile image
MaxJ • Edited

I love your article. By far the best instructions I found.

Turns out there is a breaking change in how terraform handles the oauth2_permissions github.com/hashicorp/terraform-pro...

What is working for me:
id = [for permission in azuread_application.appreg-aicloud-aks-server.oauth2_permissions : permission.id][0]
instead of:
id = "${azuread_application.aks-aad-srv.oauth2_permissions.0.id}"