Bootstrap your GitOps-enabled AKS cluster with Terraform: A code sample using the Flux v2 K8s Extension

#tutorial #kubernetes #terraform #flux

In my previous posts, we learned how to get started with GitOps on AKS using the K8s extension for AKS.

Then, we took a look at the Flux CLI and explored how it can be used to bootstrap your cluster and generate FluxCD manifests so that we can use GitOps to implement GitOps 🤯, and implemented Flux's image update automation capability.

From there, we built on the concept of image update automation, and showed you how you can use Flagger to automate canary deployments.

Much of this was done using the Flux CLI to generate Flux CRD manifests, but you might be thinking...

"All that Flux CLI stuff is cool, but I want to bootstrap my AKS cluster to use GitOps, using the AKS extension!"

In this post, I'll show you how to do just that... with my favorite IaC tool, Terraform!

What's in the box?

To get started, head over to my GitHub repo and fork/clone it. This is where I have all the Terraform code you need to deploy and bootstrap your AKS cluster using a single terraform apply command.

At a high-level, the Terraform code will do the following:

Create a resource group
Create an AKS cluster with the Istio service mesh add-on and external ingress gateway enabled
Create a Kubernetes Secret in the flux-system namespace to store GitHub credentials for private Git repository access
Install the Flux v2 AKS extension with the following components enabled:
- image-automation-controller
- image-reflector-controller
- notification-controller
Configure the Flux v2 AKS extension with the following Kustomizations:
- The dev-app Kustomization deploys the GitRepository resource which points to the flux-k8s-ext branch in my AKS Store Demo Manifests repo, and Kustomization resource to deploy the AKS Store Demo application
- The dev-image Kustomization deploys the ImageRepository, ImagePolicy and ImageUpdateAutomation resources to automate image updates for the store-front application; this Kustomization has a dependency on the dev-app Kustomization
- The dev-flagger Kustomization deploys the HelmRepository, HelmChart, and HelmRelease to install Flagger along with the OCIRepository and Kustomization resource to install the Flagger Loadtester; this Kustomization also has a dependency on the dev-app Kustomization
- The dev-canary-store-front Kustomization deploys the Canary resource that is used to automate progressive delivery of the store-front application; this Kustomization has a dependency on the dev-flagger Kustomization

As an added bonus, I've also included Azure Managed Grafana and Azure Managed Prometheus in case you wanted to explore the monitoring capabilities Flux on AKS.

To deploy, simply follow the instructions found in the README of the repo.

Kustomizations, Kustomizations, Kustomizations...

There are some differences between the way Flux CLI bootstraps your GitOps cluster and the way you can bootstrap your AKS cluster using the AKS extension.

When Flux CLI bootstraps a cluster, it creates folders called /clusters/dev and /clusters/dev/flux-system. The /clusters/dev/flux-system folder contains manifests that are used to install Flux and store GitRepository source used for bootstrapping.

Here is what the directory structure looked like when we bootstrapped using Flux CLI.

.
├── README.md
├── base
│   ├── kustomization.yaml
│   ├── makeline-service.yaml
│   ├── mongodb.yaml
│   ├── order-service.yaml
│   ├── product-service.yaml
│   ├── rabbitmq.yaml
│   ├── store-admin.yaml
│   ├── store-front.yaml
│   ├── virtual-customer.yaml
│   └── virtual-worker.yaml
├── clusters
│   └── dev
│       ├── aks-store-demo-kustomization.yaml
│       ├── aks-store-demo-source.yaml
│       ├── aks-store-demo-store-front-image-policy.yaml
│       ├── aks-store-demo-store-front-image-update.yaml
│       ├── aks-store-demo-store-front-image.yaml
│       ├── flagger-helmrelease.yaml
│       ├── flagger-loadtester-kustomization.yaml
│       ├── flagger-loadtester-source.yaml
│       ├── flagger-source.yaml
│       └── flux-system
│           ├── gotk-components.yaml
│           ├── gotk-sync.yaml
│           └── kustomization.yaml
└── overlays
    └── dev
        ├── kustomization.yaml
        └── namespace.yaml

7 directories, 25 files

The /clusters/dev/flux-system/gotk-sync.yaml file is the Kustomization resource that Flux uses to configure the cluster. It tells Flux to deploy all resources found in the /clusters/dev directory.

With the AKS extension, the gotk-* manifests are not saved or used from our repo, so we can discard them. However, in order to deploy the resources, we do need to make Flux aware that we want these resources deployed.

The files in the /clusters/dev directory needs to be split out and installed as separate Kustomizations.

Here is what the directory structure ends up looking like to bootstrap a cluster using the AKS extension.

.
├── README.md
├── base
│   ├── kustomization.yaml
│   ├── makeline-service.yaml
│   ├── mongodb.yaml
│   ├── order-service.yaml
│   ├── product-service.yaml
│   ├── rabbitmq.yaml
│   ├── store-admin.yaml
│   ├── store-front.yaml
│   ├── virtual-customer.yaml
│   └── virtual-worker.yaml
├── clusters
│   └── dev
│       ├── flagger
│       │   ├── flagger-helmrelease.yaml
│       │   ├── flagger-loadtester-kustomization.yaml
│       │   ├── flagger-loadtester-source.yaml
│       │   ├── flagger-source.yaml
│       │   └── kustomization.yaml
│       └── image-update
│           ├── aks-store-demo-store-front-image-policy.yaml
│           ├── aks-store-demo-store-front-image-update.yaml
│           ├── aks-store-demo-store-front-image.yaml
│           └── kustomization.yaml
└── overlays
    └── dev
        ├── canary
        │   ├── kustomization.yaml
        │   └── store-front-canary.yaml
        ├── ingressgateway.yaml
        ├── kustomization.yaml
        └── namespace.yaml

9 directories, 25 files

As you can see, the image-update and flagger components have been split into their own Kustomization directories. I also created a canary Kustomization to deploy the Canary resource and moved it down to the overlays/dev directory since it is more specific to my app deployment.

By taking this directory structure approach, we can link the Kustomizations together using the dependsOn property and ensure things get installed and deployed in the right order.

So in my fluxcd.tf I am first deploying the dev-app Kustomization, then the dev-image and dev-flagger Kustomizations (which depends on dev-app), and finally the dev-canary-store-front Kustomization (which depends on dev-flagger).

When you run the Terraform, it will take roughly 15-20 minutes end-to-end, but at the end of it all, you'll have a fully functional AKS Store Demo app fully configured with image update automation and canary deployments 🥳

Conclusion

Using the Flux CLI to generate Flux manifests is a great way to get started with GitOps on AKS. It gives you a way to understand exactly what CRDs need to be deployed and how the manifests are written. However, if you are looking for an Azure native (and supported way) of bootstrapping your cluster, you'll need to plan for a few more Kustomizations and nest them using Flux dependencies. Just a little more planning but not much more.

Now go and bootstrap your GitOps cluster, the Azure way! 🚀

Continue the conversation

If you have any feedback or suggestions, please feel free to reach out to me on Twitter or LinkedIn.

You can also find me in the Microsoft Open Source Discord, so feel free to DM me or drop a note in the cloud-native channel where my team hangs out!

Peace ✌️