DEV Community

Mikko Koivunalho
Mikko Koivunalho

Posted on

Manage Environment Configs

#dotenv #environment #perl #docker

TL;DR

An app’s config is everything that is likely to vary between deploys (staging, production, developer environments, etc).
The Twelve-Factor App

Storing the often changing parts of configuration in environment variables is
one of the principles of The Twelve-Factor App.

From this principle follows the need to:

  1. ensure that all the required environment variables are set with
    appropriate values, and

  2. store those environment variables and their values in easily accessible ways suitable both for development and for running in production.

Both are typical DevOps problems. To help solve them, use Env::Assert and Env::Dot, two programs which, while doing two very different things, are designed to work in unison.

Env::Assert

Env::Assert was born from frustration. One too many times:

$ PLAEC='Stockholm'
$ if [[ "$PLACE" == '' ]]; then echo "Normal OK"; fi
OK
Enter fullscreen mode Exit fullscreen mode

... And the program fails with no errors!

Not quite what we want!

Another example, from a real life Docker execution script:

perl -Ilib bin/repos-gh-yaml.pl --verbose         \
    | perl -Ilib bin/repos-yaml-csv.pl --verbose  \
    | az storage blob upload --data @-            \
        --content-type 'text/csv'                 \
        --content-encoding 'UTF-8'                \
        --content-language 'en_US'                \
        --name "$blob_name"                       \
        --container "$CONTAINER_NAME"             \
        --account-name "$AZURE_STORAGE_ACCOUNT"   \
        --sas-token "$AZURE_STORAGE_SAS_TOKEN"
Enter fullscreen mode Exit fullscreen mode

If the environment variables are wrongly set, or not set at all, it won't become evident until after the run has started. It could take hours before the run reaches the point when they are used.

Describe The Environment

Env::Assert, or rather the executable envassert, that comes with it provide an easy way to find out if the environment variables are what we require them to be.

envassert is a CLI command to assert that your environment variables match your Environment Description.

Envdesc or Environment Description is a way to describe which environment variables are required by your program.

Environment Description is written in a file. Default file name is .envdesc.

.envdesc actually looks a lot like a .env file, except instead of
defining variables and their content, it defines regular expressions
which control the variables' content. These regexps are Perl's
extended regular expressions (m/<regexp>/msx).

Example .envdesc:

CONTAINER_NAME=^[a-z0-9-]{1,}$
AZURE_STORAGE_ACCOUNT=^[a-z0-9]{1,}$
AZURE_STORAGE_SAS_TOKEN=^[?].*$
GITHUB_TOKEN=^[[:word:]]{1,}$
Enter fullscreen mode Exit fullscreen mode

In normal circumstances, envassert only verifies the variables that you specifically describe. If you want more control over your environment, there is the meta command envassert (opts: exact=1)
which will make envassert also assert that the environment doesn't contain any unknown variables.

## envassert (opts: exact=1)
USER=^username$
HOME=^/home/username$
PATH=^/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin$
Enter fullscreen mode Exit fullscreen mode

Running Env::Assert

You can create an airtight environment description to verify environment variables in both test and production. Just run envassert as the first command during container execution or any script run:

envassert --env-description /home/me/.envdesc \
    || ( echo 'Break execution ...' 1>&2 && exit 1 )
Enter fullscreen mode Exit fullscreen mode

If it detects problems, envassert will report errors and exit with an error, e.g.:

$ envassert
Environment Assert: ERRORS:
    variables:
        FIRST_VAR: Variable FIRST_VAR is missing from environment
        FOURTH_VAR: Variable FOURTH_VAR has invalid content
Enter fullscreen mode Exit fullscreen mode

Running Self-Contained

A .envdesc file is really convenient for a bigger app which may have many disconnected parts and execution scripts. But if you have only a single script which nevertheless is dependent on having certain predefined environment variables, you can also include the .envdesc file in the script. An example:

#!/usr/bin/env sh
envassert --stdin <<'EOF' # Ensure the required environment.
NUMERIC_VAR=^[[:digit:]]+$
TIME_VAR=^\d{2}:\d{2}:\d{2}$
EOF
echo "${NUMERIC_VAR}: ${TIME_VAR}"
Enter fullscreen mode Exit fullscreen mode

Using Env::Assert in a Program

Env::Assert is a Perl language module. If your application is a Perl script or package, you can also call Env::Assert directly in the code.

If you know you will always have a .envdesc file in the working directory, call:

use Env::Assert 'assert';
Enter fullscreen mode Exit fullscreen mode

But it would probably be better to specify the Environment Description file. Other parameters are also available. break_at_first_error will make Env::Assert to only report the first error it detects:

use Env::Assert assert => {
    envdesc_file => 'another-envdesc',
    break_at_first_error => 1,
};
Enter fullscreen mode Exit fullscreen mode

Inlining the description file is also possible:

use Env::Assert assert => {
    exact => 1,
    envdesc => <<'EOF'
NUMERIC_VAR=^[[:digit:]]+$
TIME_VAR=^\d{2}:\d{2}:\d{2}$
EOF
};
Enter fullscreen mode Exit fullscreen mode

Env::Dot

Env::Dot is the other piece of the puzzle, the one which will provide the environment repeatably and reliably.

There is plenty of existing DotEnv solutions. Env::Dot, however, can offer a few unique features. The .env files are treated more like source files, not as ready shell (Unix standard sh or Bash) files. With meta commands user can specify if the .env file is compatible with shell or is written in the more limited format that Docker is using:

For standard shell:

# envdot (file:type=shell)
VAR="value"
Enter fullscreen mode Exit fullscreen mode

For Docker:

# envdot (file:type=plain)
VAR=My var value
Enter fullscreen mode Exit fullscreen mode

You can chain .env files. When seeing meta command read::from_parent**Env::Dot** will search for another.env` file in any parent directory. It will load the first .env file it finds from the current directory upwards to root. If you have several applications in different subdirectory which share some environment variables but also have some unique ones, you can place the common ones in the parent directory and refer to it:

# envdot (read:from_parent)
DIR_VAR="dir"
COMMON_VAR="dir"
Enter fullscreen mode Exit fullscreen mode

Env::Dot uses environment variable ENVDOT_FILEPATHS to read dotenv files located somewhere else than in the current work dir. You can specify several file paths; just separate them by ":". Env::Dot will load the files in the reverse order, starting from the last. This is the same ordering as used in PATH variable: the first overrules the following ones, that is, when reading from the last path to the first path, if same variable is present in more than one file, the later one replaces the one already read.

If you are using Windows, separate the paths by ";"!

For example, if you have the following directory structure:

project-root
| .env
+ - sub-project
  | .env
Enter fullscreen mode Exit fullscreen mode

and you specify ENVDOT_FILEPATHS=project-root/sub-project/.env:project-root/.env, then the variables in file project-root/.env will get replaced by the more specific variables in project-root/sub-project/.env.

In Windows, this would be ENVDOT_FILEPATHS=project-root\sub-project\.env;project-root\.env

Env::Dot Executable

Use executable envdot to bring the variables into your shell.
The executable is distributed together with Env::Dot package.

envdot supports the following Unix shells: sh and its derivatives, including bash and ksh, csh and its derivative tcsh', and fish`.

Normally the variables are created in a way that also exports them into any subsequent programs which are run in the same shell, i.e. they become environment variables. However, envdot can also create them as simple variables only for the current process.

Examples of usage:

eval `envdot --no-export --shell csh`
eval `envdot --dotenv subdir/.env`
ENVDOT_FILEPATHS='../.env:subdir/.env:.env' eval `envdot`
Enter fullscreen mode Exit fullscreen mode

Using Env::Dot in a Program

Env::Dot is a Perl language module. If used in code, having the .env file is not mandatory. By default, Env::Dot will do nothing if there is no .env file. You can also configure *Env::Dot * to break execution if there is no .env file.

# If your dotenv file is `.env` or there is no `.env` file:
use Env::Dot;

# If you have a dotenv file in a different filepath:
use Env::Dot read => {
    dotenv_file => '/other/path/my_environment.env',
};

# When you absolutely require a `.env` file:
use Env::Dot read => {
    required => 1,
};
Enter fullscreen mode Exit fullscreen mode

Existing environment variables always take precedence to dotenv variables. A dotenv variable (variable from a file) does not overwrite an existing environment variable. This is by design because
a dotenv file is to augment the environment, not to replace it. This means that you can override a variable in .env file by creating its counterpart in the environment.

An example of how that works in a normal shell:

#!/usr/bin/env sh
unset VAR
echo "VAR='Good value'" >> .env
perl -e 'use Env::Dot; print "VAR:$ENV{VAR}\n";'
# VAR:Good value
VAR='Better value'; export VAR
perl -e 'use Env::Dot; print "VAR:$ENV{VAR}\n";'
# VAR:Better value
Enter fullscreen mode Exit fullscreen mode

If your .env file(s) contain variables which need interpolating,
for example, to combine their value from other variables or execute a command to produce their value, you have to use the envdot program. Env::Dot does not do any interpolating. It cannot because that would involve running the variable in the shell context within the calling program.

Env::Assert And Env::Dot

If you are in the habit of using .env files, .envdesc complements it. Commit your .envdesc file into your repository and it will act as a template for user or developer to create his/her .env file which should not be committed into Git anyway.

Top comments (0)