Using Terraform to push files to Git Repo for GitOps

Note: This post was updated because I incorrectly stated that the GitHub Terraform provider didn't delete files in the remote repository when they were removed from the Terraform code. The issue arose because I was using GitHub Actions to run the plan and apply steps, but the GitHub token was not propagating correctly during the plan step. This caused Terraform to fail to delete the files in the remote repository and resulted in multiple commits with each execution. I apologize for any confusion this may have caused and have edited the article to provide accurate information.

I have been working on a personal project named Smart-cash to improve some skills and learn new ones.

In this article, I will share my thoughts about using Terraform in the GitOps process, specifically to create the manifest and push it to the Git repo.

The basics

GitOps relies on a Git repository as the single source of truth. New commits imply infrastructure and application updates.

Imagine a Git repository where you push all the manifests of the Kubernetes resources you want to create in your cluster. These are pulled by a tool or script that runs a "kubectl apply", creates the resources, and checks the Git repo for new changes to apply. This, at a high level, is GitOps.

Setting up the scenario

For this case, the K8 cluster will run in AWS EKS, and Terraform is being used as an IaC tool.

A basic cluster can be created using Terraform. You can check an example here.

FluxCD installation can be done using the official documentation or you can check this.

I will not explain some Flux concepts like sources and Kustomizations; you can check that in the links shared previously.

Creating the YAML files

Let's say that we want to create a namespace for the development environment, we can use the following YAML:

apiVersion: v1
kind: Namespace
metadata:
  name: develop
  labels:
    test: true

We can push this file to GitHub and wait for FluxCD to do the magic.

Now let's say that we want to create a service account and associate it with an AWS IAM role, the YAML can be:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-test-develop
  annotations:  
    eks.amazonaws.com/role-arn: arn:aws:iam::12345678910:role/TEST

This looks easy but what happens if we have multiple environments or if We don't yet know the ARN of the role because this is part of our IaC?

Here is where Terraform gives us a hand.

You can create something like a template for the manifest and some variables that you can specify with Terraform. The two manifests would look like:

apiVersion: v1
kind: Namespace
metadata:
  name: ${ENVIRONMENT}
  labels:
    test: true

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-test-${ENVIRONMENT}
  annotations:  
    eks.amazonaws.com/role-arn: ${ROLE_ARN}

Notice the ${ENVIRONMENT} and ${ROLE_ARN} variables added.

We can use the Terraform GitHub provider to push the file to the repository. Let's check the following code to push the service account:

resource "github_repository_file" "sa-test" {
  repository          = data.github_repository.flux-gitops.name
  branch              = main
  file                = "./manifest/sa-manifest.yaml"
  content = templatefile(
    "sa-manifest.yaml",
    {
      ENVIRONMENT = var.environment
      ROLE_ARN = aws_iam_role.arn
    }
  )
  commit_message      = "Terraform"
  commit_author       = "terraform"
  commit_email        = "example@example"
  overwrite_on_create = true
}

The arguments repository and branch allow us to specify the remote repo and the branch where we want to push the file. The file argument is the location in the remote repository where we want to put the file.

The content argument is where we pass the values to the variables created in the template, in this case ENVIRONMENT and ROLE_ARN, the values are a terraform variable and the reference to a Terraform resource that creates the role.

overwrite_on_create argument is needed because if you run Terraform again, it will show an error because the file already exists in the repo.

Pros

1. Pushing the manifests using Terraform avoids the manual tasks of committing and pushing them, allowing us to automate more steps.
2. We can integrate this process into our pipeline, so a full environment can be ready when the pipeline finishes.
3. Terraform count can be used when there are many manifests to push, avoiding repetitive code.

Comments

[Jack]

Thank you for writing this article. While it didn't quite answer the question I was hoping to have answered, it did address another question I had that I wasn't expecting to find an answer to here.

My company is currently using Atlantis to automatically apply Terraform for GitOps. I've heard that with Flux, you can automatically apply Terraform; however, we are currently using ArgoCD. If switching to Flux would indeed allow us to automatically apply Terraform, we could potentially get rid of Atlantis and have one less tool. I was hoping to learn more about this from your article, but it wasn't covered.

On the other hand, you did cover another issue we face. Our Terraform and Kubernetes manifests (ArgoCD) don't communicate with each other. For example, when Kubernetes generates resources like load balancers, we have to hardcode the ARN in our Terraform templates or look them up with a Terraform data object. Similarly, we have to hardcode resource references for resources created in Terraform in our Kubernetes manifests. Your solution would allow us to use Terraform references in our Kubernetes manifests, which would be a great improvement.

[Daniel German Rivera]

Hey, thanks for reading and commenting
i have not tried FluxCD to apply to terraform but i will, that can help to reduce some pipelines that i have.
In the other side, yes, you can avoid hardcode values in the manifest and do it more dynamic, but double-check that Terraform doesn't delete the files in the repo. you can try and let me know your thoughts :)