Use Terraform module from a private git project with Atlantis

Let's use Terraform modules hosted in private repository on Gitlab -
https://developer.hashicorp.com/terraform/language/modules/sources#generic-git-repository with Atlantis for the CI/CD.

https://www.runatlantis.io/ is a tool for collaborating on Terraform, enables developers and operators to run terraform plan and apply directly from Terraform pull requests. Atlantis then comments back on the pull request with the output of the commands.

Here the components:

Use cases

Best practices suggest always using a registry, but there are situations where you might prefer to use a Git repository as a module source:

• Publishing on a registry requires more effort, such as setting up a CI/CD pipeline
• You need to set up, configure, secure, and maintain a registry
• The development workflow is slower because you have to wait for the module to be published
• You might not want to publish a module under active development on a registry, but you still want to use it from a root module for testing purposes

Context

Atlantis is running on kubernetes, git credentials username / password are provided by 2 env variables sourced from a kubernetes secret.

We dont want to write a .git-credentials file (see https://www.runatlantis.io/docs/server-configuration#write-git-creds) because it saves the credentials in plaintext on disk!

Terraform code

This is the easiest part:

module "vpc" {
  source = "git::https://gitlab.com/company/terraform/vpc.git?depth=1&ref=v1.2.0"
}

This will not work, because there is no authentication configuration!

Atlantis code

Many solutions found on the internet suggest configuring Git globally to allow Terraform to clone the repository. While this might work, setting up a global configuration is not a good idea.

Since Atlantis is a shared service, we want to restrict access so that only our Atlantis workflow can fetch the Terraform module.

Git credentials helper

Here the magic! Many cli tools like docker, kubectl, git ... propose a credential manager and contexts:

https://git-scm.com/docs/gitcredentials#_credential_contexts

So we must configure a credentials helper (just a small bash code) for context: "https://gitlab.com/company/terraform" to give username and password sourced from env variables:

echo "username=${ATLANTIS_GITLAB_USER}";
echo "password=${ATLANTIS_GITLAB_TOKEN}";

This is a simple shell script that output a .env format. You can use another gitlab service account, or even better & secure, generate a password with vault generator https://github.com/splunk/vault-plugin-secrets-gitlab.

In our scenario, we want to configure this only for the current Atlantis workflow, fortunately, Git allows us to configure credentials using environment variables!

https://git-scm.com/docs/git-config#Documentation/git-config.txt-GITCONFIGCOUNT

If GIT_CONFIG_COUNT is set to a positive number, all environment pairs GIT_CONFIG_KEY_ and GIT_CONFIG_VALUE_ up to that number will be added to the process’s runtime configuration.

Atlantis workflow

workflows:
  my-workflow:
    plan:
      steps:
        - env:
            name: GIT_CONFIG_COUNT
            value: "1"
        - env:
            name: GIT_CONFIG_KEY_0
            value: "credential.https://gitlab.com/company/terraform.helper"
        - env:
            name: GIT_CONFIG_VALUE_0
            value: |
              !f() { echo "username=${ATLANTIS_GITLAB_USER}"; echo "password=${ATLANTIS_GITLAB_TOKEN}"; }; f
        - init
        - plan