This is how you want to manage your Terraform modules

What if I told you there is a way to manage all your private terraform modules, in a mono-repo, with independent versioning, without using git tags? After researching for a proper open-source tool, I found the right one for the job.

Intro

A private registry is needed.

I looked for a way to manage private terraform modules like public ones.
In my company, we write tailor-maid modules that describe our infrastructure. These modules have to be private.
Terraform recommends each module has its own git repository, yet, this has the burden of managing and syncing multiple repositories.

The second option suggested is to use a mono-repo, and reference the module's version using git tags. At first, this seemed alright,
but there was a drawback — you had to give up the terraform version syntax. You no longer can use the convenient version = ~> 1.0.0 module parameter.

It seemed to me that there had to be a better way.

The problem

When it comes down to managing my infrastructure, I prefer the mono-repo approach. Since our terraform modules are small configuration blocks,
it makes sense. It is much easier to find a module in a single repository rather than searching multiple repositories for a module.
Our module release workflow looked something like this

• Our repository was structured with sub-directories per module
• We used Git tags for module versioning in the form of moduleName-vX.Y.Z
• Updates to module versions were hand-delivered to clients (even minor patches)

Our root modules (where we execute terraform plan and apply) reference modules using git tags:

terraform {
  source = “https://github.com/my-org/my-repo?ref=s3-v1.0.0"

  .. module inputs ..
}

This has worked well for a while. But once we had released a new module version and wanted to use it, there was no convenient way to apply that.

We had to go through our root modules using this module and update their references. (We could write a simple script, but we decided not to. More on that later)

The problem we had with this approach is, you can’t use the version argument in the terraform block.
This argument lets you specify a range of acceptable versions instead of a hard-coded one.

For example, you can provide version constraints such as version = “~> 1.2.0, < 2.0” which allows incrementing the “patch” automatically every time you run terraform init.
No need to manually update a patch release, and no need to write a bash script. Terraform can manage that reliably for us,
with a better API, if we can only discover a way to use this feature.

Manually managing our tags was chaotic. It is not an easy task to manage independent module versions in a mono-repo using the Git tags approach.

A simpler approach is to release a version for the whole repository. That means every new release includes all the modules together.
This couples the modules into a single artifact, which is easier to manage, but at the cost of development velocity.
You cannot release minor patches just for your module, no matter how small the code change is.

Towards a better future

The structure and workflows we had in place worked; just as not as well or easily as we wanted to.
What we needed was to be able to use terraform versioning syntax, while maintaining our mono-repo.
To achieve that, we need to treat our terraform modules as artifacts; something we can archive, version, and release independently.

After doing some research, we found an open-source project called Terralist.

Terralist is a private Terraform registry for providers and modules following the published HashiCorp protocols.
It provides a secure way to distribute your confidential modules and providers. That looked like a project that might help us to solve our problem, so we decided to try it out.

So we maintain our mono-repo as it is. We created a job in our CI system that archives a single terraform module and uploads it to Terralist.
Since it's a private registry, clients need to authenticate to be able to download the modules. After authentication (using terraform login <terralist-url>)
we could use our private modules just as we use the public modules. Yes, it means we can use the versioning syntax I was talking about.

Here is how we use our modules nowadays

 terraform {
     source = “https://my-terralist.com/my-org/s3/aws”
     version = “~> 1.0.0”

     .. module inputs ..
 }

This client code will retrieve automatic patch updates on every execution from our CI system (if you execute terraform locally, you would need to re-run terraform init).
If we make larger code changes to one of our modules, for example, something that might break the API
we would increment the major or minor version, so it wouldn't impact our clients.

Terralist

Terralist has an API that lets you upload a module from a git repository. It means you don't need to archive the module yourself,
just point Terralist to its location.

Terralist will clone the repository and create the artifact for you. Let's walk through an example.

I have a demo module, which resides in my terraform.git repository under the modules/demo directory. In the module itself, we keep a version.tf
with the major and minor versions of the module. These values change only when we introduce a change that breaks our existing API
(e.g, adding a new mandatory parameter without defaults). In such case we update the major or minor manually.

A patch number in the semver represents a safe change, such as a bug fix or non-breaking changes to the module.
This value is calculated based on the build number of our CI. We don't really care about its value, because the version constraint
applied is in the form of version = "~> 1.0.0". This automatically updates to the latest patch on every execution.

As part of our module build process, we read the values of version.tf and append the patch. Then we upload the module to Terralist using this API call:

 curl -X POST registry.example.com/v1/api/modules/demo/aws/1.0.10/upload \
      -H "Authorization: Bearer x-api-key:$TERRALIST_API_KEY" \
      -d '{ "download_url": "https://github.com/example-org/terraform/archive/refs/heads/master.zip//modules/demo" }'

The module version is composed of major.minor coming from the version.tf file, and the .patch is the build ID, which is incremented with every run and guaranteed to be unique.

Pay close attention to the double // -- this instruct Terralist to:

1. Download an archived repository
2. Extract it locally
3. Make an archive only from the path modules/demo
4. Upload it to the registry (basically upload it to S3 and update the registry database)

It's an elegant solution: I tell Terralist where my module is, and what version it is tagged with, and it takes care for everything else.

Summary

There are various ways to manage your infrastructure code.

It depends on multiple factors, such as team size, how much you are willing to spend on 3rd party tools, and your company policy, to name a few.
As of today, I've been using Terraform for more than 3 years, relying solely on the open-source ecosystem. That means my team and I manage everything related to our infrastructure (it might change soon, as our infrastructure size has grown).

Terralist was a helpful addition to our stack; it solved a problem we had in our existing workflow, with minimal effort and no code changes to our existing modules.
We just had to upload and version them to the new registry.
Now, we can release each module independently and decide if we want our clients to automatically upgrade their module version.

Happy Terraforming.

For further reading, HashiCorp documentation contains a lot of good information. Check out the links below

• Terraform Module Blocks
• Finding and using modules
• Reuse configuration with modules (tutorial)