Automating the injection of CI/CD runtime information into Terraform provider

As a DevOps engineer or software developer, you may have encountered scenarios where you must inject CI/CD runtime information into your Terraform provider code.

This information could be anything from environment-specific variables to runtime configuration values only available during the CI/CD process. However, provider usage is evaluated very early on in the Terraform run, before we have enough context to do variable interpolations, so you can't use variables there (like you can normally do with standard resources and environment variables).

I stumbled upon a real-world use case dealing with some Terraform code to create AWS resources, where I need to add information about the role ARN to be assumed, and this information cannot be statically inserted in the code, because this code needs to be executed with different roles depending on some condition and/or constraints.

Another use case is to add default tags to all providers, such as the build number, to ensure consistency across all created AWS resources (and maybe you can't be sure that a default_tags entry is present in every provider).

To automate the process of injecting CI/CD runtime information into our Terraform provider, we'll introduce the tool hcledit. With hcledit and some other manipulation, we can insert data into the Terraform code using regex/grep to find the correct place to add it.

How did I do it? Let's see the code! Here's my Bash function:

initialize_provider() {
  # allows the last command in a pipeline to be executed in the current shell environment, rather than a subshell
  shopt -s lastpipe
  # this is a simple regex to match every provider
  regex='provider[[:blank:]]\+"\([[:alnum:]_-]\+\)"[[:blank:]]*{'

  # Search for files in the current directory that contain the regular expression
  for file in $(grep -rl "$regex" .); do
    echo "Processing file: $file"
      # Create temporary file with the same name as the original file
      temp_file="${file}.tmp"
      cp "$file" "$temp_file"
      # declare an array variable
      declare -a arr

      # Modify file contents and write to temporary file
      grep "$regex" "$file" | while IFS= read -r line; do
          # Extract the provider name, which is the second word without the quotes and braces
          pn=$(echo "$line" | sed "s/$regex/ \1/g" | tr -d '"{}' | cut -d' ' -f2)
          echo "Found provider: $pn"
          #add the provider name to the array
          arr+=("$pn")
      done

      # Iterate over the array
      for provider_name in "${arr[@]}"; do
          # Add the assume_role block to the provider
          hcledit block append provider.$provider_name assume_role -f $temp_file -u --newline
          hcledit attribute append provider.$provider_name.assume_role.role_arn \"$ROLE_ARN\" -f $temp_file -u
          # Add the default_tags block to the provider
          hcledit block append provider.$provider_name default_tags -f $temp_file -u --newline
          hcledit attribute append provider.$provider_name.default_tags.tags \"$BUILD_ID\" -f $temp_file -u
      done
      cat "$temp_file"
  done

  # Move temporary files to original file names
  for temp_file in *.tmp; do
      mv "$temp_file" "${temp_file%.tmp}"
  done
}

Let's clarify what it does:

1. The first line enables the lastpipe shell option, which allows the last command in a pipeline to be executed in the current shell environment, rather than a subshell. It is most useful if you call this function from another process (i.e. your CI/CD pipeline).

2. regex is a regular expression that matches every provider in the Terraform code. I need to match multiple providers because you can have more than one, for example when you deploy in different regions. Beware that, if you use non-AWS providers, you may change this regex to exclude them or, in general, better match your needs.

3. The script searches for files in the current directory that contain the regex regular expression.

4. For each file found, the script creates a temporary file with the same name as the original file and copies the contents of the original file to the temporary file.

5. The script declares an empty array called arr.

6. The script reads through the contents of the original file and extracts the names of all the providers matching the regex regular expression. Each provider name is added to the arr array.

7. The script iterates over the arr array and appends an assume_role block to each provider in the Terraform code, using hcledit, which is much more convenient than Bash directly.

8. For each assume_role block, the script appends a role_arn attribute with the value of $ROLE_ARN, using hcledit.

9. Then the script uses hcledit again to add a default_tags block to each provider in the Terraform code.

10. For each default_tags block, the script appends a tags attribute with a list containing the value of $BUILD_ID.

11. Finally, the script moves the temporary files to their original file names by removing the .tmp extension.

In conclusion, by automating the injection of CI/CD runtime info into your Terraform code with tools like hcledit and a little bit of scripting know-how, you can easily add environment-specific variables and runtime configuration values to your Terraform code, making it more efficient and less error-prone.

Happy automating!