TLDR: Configuration management is hard. Fix it before digging a deeper hole.
Below is the sample .env file that looks pretty standard and innocent. This does not necessarily map to a real-world configuration where you could expect this file to contain way more environment variables.This is pretty standard in projects and we used to roll with it until I started to deal with configurations over the years in regards to managing, provisioning, rotations, policy enforcement, and secrets management.
1. Explicit over Implicit
Never allow optional/default value for a configurable variable. How many times you have seen localhost in your staging/QA environments? How many times you are not able to connect to the database because it used a default value such as Postgres or Testuser because you missed configuring the database in the foo environment?
This gets even difficult when you have to hand off your project you are working on to the Dev ops engineer and they have to deal with your default values for the configurable variable. The next day, your deployed application redirects to localhost:5000/callback?… when the user is authenticated.
REMEDY
Crash your application and scream with giant and ugly stack trace until configuration variables are explicitly jotted or provisioned.
2. Separation of Build and Run time Configuration
I often see running applications that are deployed in the cloud have environment variables such as ECR_REPO_URL, AWS_ACCESS_KEY, etc. I immediately smell the situation where developers are polluting their run time environment variables with build-time environment variables.
ECR_REPO_URL was used to push the application's OCI image to a remote repository and AWS_ACCESS_KEY was used to authenticate against AWS to upload artifacts to s3. These variables were clearly used to package the application and not used during the application's runtime. And yet we still put those build-time environment variables in our classic .env file all the time which are supposed to be used for runtime.
REMEDY
Separate your environment variables explicitly for different stages in your development and deployment pipelines.
For instance, you could use .env.build for build and .env.run for runtime configuration. Over time, you have to manage configurations for integration tests, documentation, etc but hope you got the idea.
3. Mode is better than Prefixed Variable name
In the above .env file, you could see that it is TEST_APP_HOST for the test environment and APP_HOST for the production environment. TEST is prefixed to signify that the TEST_APP_HOST variable is used for the app host in the test environment. This is just a horrible idea in the first place.
This should remind your code that looks something like this
It becomes exponentially difficult to maintain environment variables based on the environment as a number of environments such as staging, QA, pre-production, etc are introduced. And the worst part is, your unit and integration tests in your development won't ensure the full coverage as you won't be testing the code that branches off when you configure it to production.
REMEDY
Use mode to signify the environment and keep the rest of the environment variables without any prefixes.The below code example should give you the idea where we branch off using mode and keep the rest of the statements the same across environments.
The above code example uses Golang as a programming language but should be fairly simple. We enabled tracer and disabled debugging in production and kept the same variable name APP_HOST and APP_PORT irrespective of different environments.
4. Synchronize .env and .env.example files
Hey, could you pass me your .env file?
Let's admit it, your .env.example file always looks "short and sweet" over time while your .env file includes variables that were added during development which does not exist in the .env.example file. You have to copy-paste your environment files to your team member while they are bootstrapping their project because he/she could not run the project as few environment variables were missing which was not included in the .env.example file.
REMEDY
Make .env.example part of your program and crash your app with the ugly exception (if your language has one) if there is a deviation between your .env and .env.example. This will ensure that your .env and .env.example are in sync and no one in the team has to guess what is missing. You could use tools such as sync-dotenv to enforce this.
5. Handle Secrets secretly
We tend to manage our secrets similar to how we manage other environment variables such as DB_USER, DB_HOST. DB_PASSWORD is a secret and requires way more policies around it such as rotations, leasing, revoking auditing, and much more. These secrets are usually dynamic i.e changes over time. Since these secrets require more bookkeeping than the rest of the environment variables, it's best to use tools that are best fit for managing such complexities.
REMEDY
It really depends on the deployment platform. If you are using Kubernetes, you could use a Config map object to store your configuration and a Secrets object to store your sensitive information. You could also opt for Vault by Hashicorp which manages complexities around secrets. If you don't want to manage the Vault Cluster, you could go for services such as Secret Hub.
Top comments (2)
Great article @robusgauli
Another remedy I would like to suggest is Configu. It's open-source, language-agnostic, and it simplifies configuration management, saving you all the overheads you described throughout the article.
Great article, thank you so much!