DEV Community

Karun Japhet
Karun Japhet

Posted on

3 2

Version controlled configuration and secrets management for Terraform

Terraform is a tool to build your infrastructure as code. We've been having a few challenges while trying to figure out how to how to manage configuration and secrets when integrating terraform with our CD pipeline.

Life before version control

Before we can do that, it's important to understand build process before we began on this journey.
Terraform managed environments

Our build model for this project was branch based. Each environment maps to a branch (main -> dev, uat -> uat and production -> production). All other (feature) branches only ran the plan stage against the dev environment.

As you can notice, the configurations, secrets and keys are all maintained on the build agent. This means, every developer wanting to run plan and test their changes needs to replicate the terraform_variables directory. Any mistakes in doing so masks actual issues that your pipeline might face leading to delayed feedback.

Next, let's look at what our codebase looked like

terraform
├── module-1
│   ├── backend.tf
│   ├── data.tf
│   ├── resources.tf
│   ├── provider.tf
│   └── variables.tf
├── module-2
│   ├── backend.tf
│   ├── data.tf
│   ├── resources.tf
│   ├── provider.tf
│   └── variables.tf
└── scripts
    └── provision
        ├── apply.sh
        ├── init.sh
        └── plan.sh
Enter fullscreen mode Exit fullscreen mode

The provisioning scripts help us consistently run different stages across modules. Each module is an independent area of our infrastructure (such as core networking, HTTP services etc.)

Each of the provisioning scripts accepted a WORKSPACE_NAME (branch for execution that maps to the environment terraform is running for) and MODULE_NAME (module being executed).

init.sh ran the terraform init stage of the pipeline downloading the necessary plugins and initializing the backend

#!/bin/bash
set -e
cd $MODULE_NAME
echo "init default.tfstate"
terraform init -backend-config="key=default.tfstate"
echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME
echo "init $MODULE_NAME/terraform.tfstate"
terraform init -backend-config="key=$MODULE_NAME/terraform.tfstate" -force-copy -reconfigure
view raw init.sh hosted with ❤ by GitHub

plan.sh ran the terraform plan stage allowing users to review their changes before applying them.

#!/bin/bash
set -e
cd $MODULE_NAME
echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME
echo "plan with var file ~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
terraform plan -var-file=~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars -out=$MODULE_NAME.tfplan -input=false
view raw plan.sh hosted with ❤ by GitHub

apply.sh applied the changes onto an environment. Developers do not run this command from local to ensure consistency on the environment

#!/bin/bash
set -e
cd $MODULE_NAME
echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME
echo "apply with var file ~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
terraform apply -var-file=~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars -auto-approve
view raw apply.sh hosted with ❤ by GitHub

Version controlling configuration

We moved the variables into the config directory by making a directory for every branch for each of the 3 environments we had.

terraform
├── config
│   ├── main
│   │   ├── module-1.tfvars
│   │   └── module-2.tfvars
│   ├── production
│   │   ├── module-1.tfvars
│   │   └── module-2.tfvars
│   ├── uat
│   │   ├── module-1.tfvars
│   │   └── module-2.tfvars
├── module-1
│   └── ...
├── module-2
|   └── ...
└── scripts
    ├── provision
    │   ├── apply.sh
    │   ├── functions.sh
    │   ├── init.sh
    │   └── plan.sh
    └── test_variable_names.sh
Enter fullscreen mode Exit fullscreen mode

According to terraform's documentation, you can export a variable that your terraform codes need with a prefix of TF_VAR.

functions.sh provides convenience functions to read the configuration and secrets.

#!/bin/bash
function fetch_variables() {
workspace_name=$1
module_name=$2
echo $(cat ../config/$workspace_name/$module_name.tfvars | sed '/^$/D' | sed 's/.*/TF_VAR_& /' | tr -d '\n')
}
view raw functions.sh hosted with ❤ by GitHub

fetch_variables read the tfvars file, removes empty lines (that were added for readability), prefixed the name with TF_VAR and joined all entries into a single line. The string this method returns can be used as a prefix to the terraform command while running plan and apply making them environment variables.

Updated plan and apply scripts are placed in the secrets management section for brevity

Testing configuration files

The only limitation is that none of these variables can have a hyphen in the name because of shell variable naming rules. As with any potential mistake, a test providing feedback helps protect you from run time failures. test_variable_names.sh does this check for us.

#!/bin/bash
function parse_and_test_properties_entries() {
prop=$1
if [[ "$prop" == "" || $prop = \#* ]]; then
continue
fi
key="$(cut -d'=' -f1 <<<"$prop")"
if [[ $key =~ "-" ]]; then
echo "$filename contains \"$key\" which contains a hyphen"
exit 1
fi
}
function parse_file() {
filename=$1
OLD_IFS=$IFS
props=$(cat $filename)
IFS=$'\n'
for prop in ${props[@]}; do
parse_and_test_properties_entries $prop
done
IFS=$OLD_IFS
}
base_dir="config"
for sub_dir in $(find $base_dir -mindepth 1 -maxdepth 1 -type d); do
workspace_name=${sub_dir#"$base_dir/"}
for input_file in config/$workspace_name/*.tfvars; do
parse_file $input_file
done
echo "All variables are named correctly in config/$workspace_name"
done

Version controlling secrets

Secrets like passwords can be version controlled in a similar way though they require encryption to keep them safe. We're using OpenSSL with a symmetric key to encrypt our secrets. Each secret is put into a tfsecrets file (internally a property file just like tfvars files for configuration). When encrypted, the file will have an extension of .tfsecrets.enc. When the plan or apply stages are executed, files are decrypted in memory (and not on disk, for security reasons) and used the same way.

functions.sh gets a new addition to support reading all secrets

function fetch_secrets() {
workspace_name=$1
module_name=$2
secret_key_for_workspace=$(eval "echo \$SECRET_KEY_$workspace_name")
echo $(openssl enc -aes-256-cbc -d -in ../config/$workspace_name/$module_name.tfsecrets.enc -pass pass:$secret_key_for_workspace | sed '/^$/D' | sed 's/.*/TF_VAR_& /' | tr -d '\n')
}
view raw functions.sh hosted with ❤ by GitHub

The astute amongst you probably noticed that we're using OpenSSL v1.0.2s because v1.1.x changes the syntax on encryption/decryption of files. Also, you might have noticed the use of environment variables like SECRET_KEY_main, SECRET_KEY_uat and SECRET_KEY_production as the encryption keys. These values are stored on our CI server (in our case GitLab) which makes these values available to our CI agent during execution.

For local development, we have scripts to encrypt and decrypt configuration files either one at a time or in bulk per environment. It's worth noting that re-encryption of the same file will show up on your git diff since the encrypted file's metadata changes. Only check in encrypted files when their contents have changed (helping you debug future issues)

encrypt.sh takes SECRET_KEY as an environment variable for making local usage easier.

#!/bin/bash
set -e
if [ -z "$SECRET_KEY" ]; then
echo "Set a SECRET_KEY for \"$WORKSPACE_NAME\" encryption"
exit 1
fi
function encrypt_file() {
input_file=$1
target_file="$input_file.enc"
echo "Encrypting $input_file to $target_file"
openssl enc -aes-256-cbc -salt -in $input_file -out $target_file -pass pass:$SECRET_KEY
rm -f $input_file
}
if [ -z $1 ]; then
echo "Usage:"
echo " ./scripts/encrypt.sh <filePathFromProjectRoot>"
echo " ./scripts/encrypt.sh all"
exit 2
elif [ "$1" == "all" ]; then
for input_file in config/$WORKSPACE_NAME/*.tfsecrets; do
encrypt_file $input_file
done
else
encrypt_file $1
fi
view raw encrypt.sh hosted with ❤ by GitHub

decrypt.sh also takes the same SECRET_KEY as an environment variable for making local usage easier.

#!/bin/bash
set -e
if [ -z "$SECRET_KEY" ]; then
echo "Set a SECRET_KEY for \"$WORKSPACE_NAME\" decryption"
exit 1
fi
function decrypt_file() {
input_file=$1
target_file=${input_file%".enc"}
echo "Decrypting $input_file to $target_file"
openssl enc -aes-256-cbc -d -in $input_file -out $target_file -pass pass:$SECRET_KEY
rm -f $input_file
}
if [ -z $1 ]; then
echo "Usage:"
echo " ./scripts/decrypt.sh <filePathFromProjectRoot>"
echo " ./scripts/decrypt.sh all"
exit 2
elif [ "$1" == "all" ]; then
for input_file in config/$WORKSPACE_NAME/*.tfsecrets.enc
do
decrypt_file $input_file
done
else
decrypt_file $1
fi
view raw decrypt.sh hosted with ❤ by GitHub

Testing secret files

If all files for an environment aren't checked with the same key, you'll face a runtime error. Since files can be encrypted individually, you must test if all files have been encrypted correctly. This test is also useful when you're rotating the SECRET_KEY for an environment.

test_encryption.sh needs SECRET_KEY_<env> values set so it can be executed locally.

#!/bin/bash
base_dir="config"
for sub_dir in $(find $base_dir -mindepth 1 -maxdepth 1 -type d); do
workspace_name=${sub_dir#"$base_dir/"}
password_var_name="\$SECRET_KEY_$workspace_name"
secret_key_for_workspace=$(eval "echo $password_var_name")
if [ -z "$secret_key_for_workspace" ]; then
echo "Variable $password_var_name has not been set. Unable to test"
exit 1
fi
for input_file in config/$workspace_name/*.tfsecrets.enc
do
openssl enc -aes-256-cbc -d -in $input_file -pass pass:$secret_key_for_workspace &> /dev/null
if [ $? != 0 ]; then
echo "Unable to decrypt $input_file with $password_var_name"
exit 1
fi
done
echo "Successfully decrypted all secrets in config/$workspace_name"
done

End result

Our final project structure contains the following files

terraform
├── config
│   ├── main
│   │   ├── module-1.tfvars
│   │   ├── module-1.tfsecrets.enc
│   │   ├── module-2.tfvars
│   │   └── module-2.tfsecrets.enc
│   ├── production
│   │   ├── module-1.tfvars
│   │   ├── module-1.tfsecrets.enc
│   │   ├── module-2.tfvars
│   │   └── module-2.tfsecrets.enc
│   ├── uat
│   │   ├── module-1.tfvars
│   │   ├── module-1.tfsecrets.enc
│   │   ├── module-2.tfvars
│   │   └── module-2.tfsecrets.enc
├── module-1
│   └── ...
├── module-2
|   └── ...
└── scripts
    ├── decrypt.sh
    ├── encrypt.sh
    ├── provision
    │   ├── apply.sh
    │   ├── functions.sh
    │   ├── init.sh
    │   └── plan.sh
    ├── test_encryption.sh
    └── test_variable_names.sh
Enter fullscreen mode Exit fullscreen mode

plan.sh uses functions.sh to load configuration and secrets

#!/bin/bash
set -e
source $(dirname "$0")/functions.sh
cd $MODULE_NAME
echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME
echo "plan with var file config/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
config=$(fetch_variables $WORKSPACE_NAME $MODULE_NAME)
secrets=$(fetch_secrets $WORKSPACE_NAME $MODULE_NAME)
eval "$secrets $config terraform plan -out=$MODULE_NAME.tfplan -input=false"
view raw plan.sh hosted with ❤ by GitHub

apply.sh uses functions.sh in a similar fashion

#!/bin/bash
set -e
source $(dirname "$0")/functions.sh
cd $MODULE_NAME
echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME
echo "apply with var file config/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
config=$(fetch_variables $WORKSPACE_NAME $MODULE_NAME)
secrets=$(fetch_secrets $WORKSPACE_NAME $MODULE_NAME)
eval "$secrets $config terraform apply -auto-approve"
view raw apply.sh hosted with ❤ by GitHub

And thus, our terraform project requires no data from the CI agent and can be executed perfectly from any box as long as it has the latest code checked out and the correct version of terraform.

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more