We know storing credentials or other sensitive values in a configuration file (e.g. Kubernetes yaml file) is bad, but how can we get values easily replaced without having to do a complicated string substitution or writing a custom Python script?
I often have personal environment variable files for projects that I use to store credentials and configurations in. Before working on a project I would the corresponding configuration file into the shell session. However, these files cannot be stored in git repositories or shared with coworkers or bots. Even worse, sometimes the repositories have files in them that need to be changed, which is dangerous, because it’s easy to accidentally commit these files.
Well as it turns out, there already is a good solution and it is called envsubst. We can use envsubst to substitute environment variable placeholders inside configuration files and we can even pipe it into other commands like Kubernetes' kubectl.
envsubst < config.txt
Some systems have gettext with envsubst preinstalled. However, if it is missing, you can install it using a package manager. For macOS you can use homebrew:
brew install gettext
Learn more about homebrew in my article about setting about your development machine in one script
Let’s say, we have an existing configuration file, that want to give to someone or use with a bot. Ideally we don’t want to include credentials in that file.
# my configuration file server: https://gitlab.com/skofgar username: foo_user password: mymonkey
We don’t want to check that information into a git repository nor do we want it laying around or send it to someone like this , but how can we improve this? Lets replace the information we don’t want in the file with environment variables:
server: $SERVER_URL username: $USER_NAME password: $USER_PASSWORD
Then define these environment variables either by defining them in the shell session with:
export SERVER_URL=https://gitlab.com/skofgar export USER_NAME=foo_user export USER_PASSWORD=mymonkey
or save them to a file (e.g. .env) and then loading them into your current shell session by using source .env
Make sure to use export, otherwise your variables are considered shell variables and might not be accessible to envsubst
To run an actual substitution, perform the following commands.
> envsubst < config.txt server: https://gitlab.com/skofgar username: foo_user password: mymonkey
It is also possible to write your substitution to a new file:
> envsubst < config.txt > confidential_config.txt
It is possible to pipe the output into other commands like less or kubectl for Kubernetes (k8s).
# pipe into less > envsubst < config.txt | less # pipe a deployment "deploy.yml" into kubectl apply > envsubst < deploy.yml | kubectl apply -f -
This is a great way to improve your Continuous Integration and Continuous Deployment (CI/CD) pipelines or just simplify your own workflow. I use this for a project where we have to replace credentials that are temporarily needed, that we didn’t want to check into the git repository and that a CI pipeline also needs to access.
Let me know in the comments what you think of this solution and if you have come across an alternative approach.
Here’s a summary of all steps combined:
# print content of configuration file > cat config.txt server: $SERVER_URL username: $USER_NAME password: $USER_PASSWORD # load environment variables > source .env # replace environment variables in file content > envsubst < config.txt server: https://gitlab.com/skofgar username: foo_user password: mymonkey # replace environment variables and write to new file > envsubst < config.txt > confidential_config.txt # pipe into less > envsubst < config.txt | less # pipe a deployment "deploy.yml" into kubectl apply > envsubst < deploy.yml | kubectl apply -f -
Some of the sources I used for this article are:
- envsubst for Kubernetes: https://serverfault.com/a/843883
- gettext documentation: https://www.gnu.org/software/gettext/
Originally published at Skofgar’s Blog.