TL;DR: With the mass adoption of Terraform and becoming the de facto tool for developers to build, and manage their cloud infrastructure at scale, most companies today, who rely heavily on Terraform for their infrastructure management, choose to do so with an orchestration tool. In this blog, we'll review the way to govern Terraform States using Gitlab Enterprise
Terraform has become the nearly ubiquitous way to provision services in a cloud native era. However, when we start to build our infrastructure using Terraform’s as-code approach, there are a few things we need to consider in order to be able to manage these operations at scale, for a diversity of decentralized services, and for distributed teams. At Firefly, we often encounter the challenges of managing IaC at scale, as part of our effort to help organizations discover and manage their many cloud assets.
Terraform Management at Scale
With the mass adoption of Terraform and becoming the de facto tool for developers to build, and manage their cloud infrastructure at scale, most companies today, who rely heavily on Terraform for their infrastructure management, choose to do so with an orchestration tool. These tools complement the suite of tools Hashicorp provides, to help get a handle on the many modules and providers, frameworks and services being provisioned with the sheer scale of cloud operations, as well as a remote backend to maintain the state for your infrastructure.
The companies that choose a fully managed orchestration tool, oftentimes will select Hashicorp’s very own Terraform Cloud (SaaS solution), however like all tools that gain widespread popularity ––there are many non-Hashicorp alternatives to ride the wave of the tool’s success, as well. Terraform Cloud's benefits are a fully remote backend, native integration with GitHub, State versioning, and advanced features for infrastructure stakeholders, such as platform engineers, DevOps teams and cloud engineers.
However, there is another option that is gaining popularity for large-scale Terraform operations, and that is the GitOps approach, ones who decide to deploy their infrastructure using GitHub Actions or other built-in CI pipelines applications. The most popular tool for this use case is Atlantis. Atlantis is a basic solution that integrates automatically with each pull request (PR) and enforces best practices for infrastructure deployments as they are defined in the company policies such as: the code owner, code reviewers, unit tests using tools like TerraTest, among others.
When you choose the GitOps method, this will not come with the managed backend, and therefore this will still be required for those looking to maintain state for their IaC. Terraform currently supports out-of-the-box integration with: AWS S3, GCS, Hashicorp Consul, Kubernetes, and HTTP.
As we all know though, many companies today have chosen to work with Gitlab on-prem, for many reasons, and therefore all of the Github and Github Actions integrations become less relevant with this choice.
Terraform States Using Gitlab Enterprise
Those companies who choose GitLab as their primary source code management (SCM) platform, will also many times choose to deploy their infrastructure using dedicated GitLab pipelines. This leaves us with the question: but what about the Terraform state?
Introducing a new feature for remote backends inside Gitlab. We knew this was just what we needed as a Gitlab shop. However, when we came to try and enable it, we found very little documentation to help us…and so we had to go down the rabbit hole of researching how to configure and setup remote backends with specific requirements dictated in the Gitlab API, and we’d like to share with you some of the excellent intel we uncovered.
We’ll start with some of the challenges we immediately encountered. Configuring the Gitlab backend proved itself quite complex, having to understand the Gitlab configuration syntax in-depth and the various S3 configurations to actually get this set up. Once we managed to configure our S3 bucket as the dedicated data store for our Terraform states, we found that these are all encrypted using AES 256 inside the S3 by Gitlab. What this means is that once encrypted, this state is no longer accessible inside Terraform. This requires you to use Gitlab APIs to download them and be able to use them in your environment.
This is where it gets tricky. So we’ve chosen to deploy and orchestrate our code using Gitlab. Great. Next we want to leverage their new capability of managing state - but this means we can’t actually manage our Terraform State if they are encrypted and not accessible to Terraform.
This adds a particular layer of complexity for this use case, because when you work in the modern engineering format of CI/CD, Gitlab will increase your version number with each deployment, to maintain the log and change history of deployed versions. All of this is fine, and important as an engineering best practice - however this introduces a few gaps when it comes to Terraform state management.
If we take a look at the Gitlab API documentation the way to download the state is as follows:
curl --header "Private-Token: " "https://gitlab.example.com/api/v4/projects//terraform/state//versions/"
This means that in order to be able to download the state you have to have a few critical pieces of information:
- Your Access Token
- Your Project ID
- Your State Name
- Your Version Number
Not only does one rarely know the specific name of their deployment, it’s very rare to know the latest version number (Gitlab doesn’t expose this in the UI, only if you hover over the deployment or click on it will you see this number in the URL)––as this is constantly changing with continuous deployment.
In very large scale operations, there are hundreds of environments running Terraform all the time, and news ones constantly being deployed. Not to mention different kinds of environments–– development, staging, production, with all of these having multiple dev accounts. It’s a needle in a haystack.
We felt like we hit a wall. We knew there had to be a better way. We went back to researching.
Gitlab GraphQL API for Terraform State Management
After digging deeper, we found a gold mine. There IS another way.
We found a hidden GraphQL API that reveals all of your Gitlab environments built through GitLab Pipelines which enables you to extract quite simply all of the critical information you will need to be able to download and access the Terraform State.
See it in action - below is the GraphQL code snippet that enables you to query and extract the required data.
POST https://{{GITLAB-HOST}}/api/graphql { "operationName": "getStates", "variables": { "projectPath": "sefi/tf-demo", "first": 50, "after": null, "last": null, "before": null }, "query": "query getStates($projectPath: ID!, $first: Int, $last: Int, $before: String, $after: String) {\n project(fullPath: $projectPath) {\n id\n terraformStates(first: $first, last: $last, before: $before, after: $after) {\n count\n nodes {\n ...State\n __typename\n }\n pageInfo {\n ...PageInfo\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment State on TerraformState {\n id\n name\n lockedAt\n updatedAt\n deletedAt\n lockedByUser {\n ...User\n __typename\n }\n latestVersion {\n ...StateVersion\n __typename\n }\n __typename\n}\n\nfragment User on User {\n id\n avatarUrl\n name\n username\n webUrl\n __typename\n}\n\nfragment StateVersion on TerraformStateVersion {\n id\n downloadPath\n serial\n updatedAt\n createdByUser {\n ...User\n __typename\n }\n job {\n id\n detailedStatus {\n id\n detailsPath\n group\n icon\n label\n text\n __typename\n }\n pipeline {\n id\n path\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }
This API returns the latest version of all environments in the project.
{ "data": { "project": { "id": "gid://gitlab/Project/", "terraformStates": { "count": 1, "nodes": [ { "id": "gid://gitlab/Terraform::State/", "name": "", "lockedAt": null, "updatedAt": "2022-08-02T19:55:26Z", "deletedAt": null, "lockedByUser": null, "latestVersion": { "id": "gid://gitlab/Terraform::StateVersion/", "downloadPath": "/api/v4/projects//terraform/state//versions/0", "serial": 0, "updatedAt": "2022-08-02T19:55:26Z", "createdByUser": null, "job": null, "__typename": "TerraformStateVersion" }, "__typename": "TerraformState" } ], "pageInfo": { "hasNextPage": false, "hasPreviousPage": false, "startCursor": "", "endCursor": "", "__typename": "PageInfo" }, "__typename": "TerraformStateConnection" },
Using the response, we can download the latest version of the Terraform State leveraging the previously mentioned Gitlab API:
https://{{GITLAB-HOST}}/API/v4/projects//terraform/state//versions/
That’s it! It’s that easy.
For anyone using Gitlab On-Prem or Enterprise, leveraging Gitlab Pipelines, there really is no need to add more tooling to the stack for Terraform orchestration and management. You can leverage the built-in Gitlab support and Terraform’s integration with S3. With the GraphQL API access you can now access the required info to download your state from storage via the Gitlab API.
Top comments (1)
Great post @sefige_1 🔥💪🏽