A better way to do environment variables

Matt — Wed, 06 Mar 2024 22:05:12 +0000

Running software will always require configuration values to declaratively steer the outcome of its functionality.

We have traditionally utilised environment variables to handle all of our important values. Whilst it works and there are benefits, it has become a very inflexible way to build software in light of evolved patterns.

Using things like secrets manager services, feature flag services and config files, we can accelerate our development with more confidence and care.

Bondi, Larry Snickers

Current State & Criticisms

Take this example snippet from a small imaginary TypeScript monorepo consisting of a website using Vite-config, Node backend and utilising environment variables to manage all build-time and runtime outcomes:

.env.staging

Note: assume this will be set properly upon deployment and not using files

ENV=staging
NODE_ENV=production
VITE_TITLE="Medium"
FEATURE_1_ENABLED=true
SOME_SECRET=xxx
VITE_API_URL=api.medium.com

api.ts

const { 
  ENV, 
  NODE_ENV, 
  FEATURE_1_ENABLED,
  SOME_SECRET,
} = process.env;

app.ts

const { 
  VITE_TITLE,
  VITE_API_URL,
} = import.meta.env;

Let's take a closer look at the environment variables:

# required at build-time, boot-time & runtime
# because of this, values like these will always need to be environment 
# variables
ENV=staging
NODE_ENV=production

# static, frontend-only & non-sensitive values
VITE_TITLE="Medium"
VITE_API_URL=api.medium.staging

# this is a non-sensitive, runtime boolean value
# feature flags should ideally be dynamic and personalised
FEATURE_1_ENABLED=true

# this is a sensitive runtime value
SOME_SECRET=xxx

This looks fine, so what are the drawbacks here?

Developer Experience & Confidence

You could imagine in a large complex product, this file could move into the 100's of lines. Now multiply this by the amount of environments you have and things can more easily go wrong. These are testable, of course, but since they are coming from the environment the tests are being run in, rather than the code or a service, it's environment dependent.

Security

Yes, someone needs access to your server to see these values. What if someone is testing a production build locally and accidentally commits a secret value? Some IDE's store local history for weeks.

Static vs Dynamic

Changing environment variables on a server is the only option we have here and will require a service restart in most cases, whilst 'feature flag' type values should have the ability to be dynamically changed on the fly i.e., from an admin interface or be personalised for workspaces or users through a session variable.

Deployment

Where do these values get set? For local dev, it's right in front of us, but when we deploy our environment, we begin to set most of these in some separate platform, like a CI/CD pipeline config, which is away from our codebase and becomes a bit of a black box.

Icebergs, Belle Co

Solution

tl;dr do these steps

Categorise your values:
i. environment values
ii. secret values
iii. non-secret values
iv. dynamic & personalised values
Use a typed environment variable file
Use a secrets manager service
Use code config files that are apart of your codebase (file per environment)
Use a feature flag service

Let's go deeper!

1. Categorising values

This will help you understand where you should move your existing environment variables to.

i. Environment Values

Environment values are values that declaratively tell the service or app how it should run in the given environment.

Traits

required at build time
required at boot time
often required during runtime
not sensitive
immutable

Examples

environment determinant e.g., Node Environment is production NODE_ENV=production

ii. Secret values

Traits

sometimes required at build time
required at boot time
required during runtime
sensitive
potentially mutable, generally immutable

Examples

third-party API keys e.g., Stripe API Key STRIPE_API_KEY=abc-123

iii. Static, non-secret values

Traits

required at build time
required during runtime
not sensitive
potentially mutable, generally immutable

Examples

UI content variables e.g., meta page title PAGE_TITLE=My Website
Feature flag value e.g., toggling a new feature on in certain environment NEW_SEARCH_BAR_UI_ENABLED=false

iv. Dynamic, personalised values

Traits

required at build time
required during runtime
not sensitive
mutable

Examples

Feature previews e.g., subset of organisations in your app can preview a new feature before the rest of the users can e.g., NEW_SEARCH_BAR_UI_ENABLED=true

2. Use a typed environment variable file

Let's say this is what our environment variables look like:

Environment (however you set this):

DEBUG=false
DEPLOYMENT=staging
NODE_ENV=production

Since we will always need some values from the environment, the least we can do is wrap some strictness around the types for confidence.

Using TypeScript as an example, we can create an environment.d.ts file so we get completion and type safety when accessing process.env:

environment.d.ts:

declare namespace NodeJS {
  export interface ProcessEnv {
    DEBUG: string;
    DEPLOYMENT: string;
    NODE_ENV: string;
  }
}

Keep in mind that they are all string | undefined in Typescript, whether or not you have assigned these to a number, boolean or otherwise, e.g.,

✅ if (process.env.DEBUG === 'false') { ...

❌ if (!process.env.DEBUG) { ...

znv is pretty good at improving the types of your environment also.

3. Secrets Manager service

I would suggest starting with these as environment variables as they change infrequently.

However, there may come a time where you want these to be dynamic, in the sense that you want to change them without incurring any downtime. For this, we can use a Secrets Manager service.

Off the shelf Secret Manager services can get expensive, so you may want to roll your own, however, the responsibility of designing the encryption process, accessor patterns and probably an admin UI now rely with your team.

4. Config files

When infrastructure-as-a-service and infrastructure-as-code became a reality, there was a notion that we could have 1..n environments. Whilst this is still true, things like feature flagging helped us move code to production in a safer manner, since we could obfuscate new features from users until they were ready and still deliver new features, fixes and technical refreshes.

With this paradigm shift, lot's of teams moved toward trunk-based development. In doing so, we have come full-circle in a way, way back to a time where we had multiple physical servers for the various stages of SDLC; staging and production.

staging: production-like environment where we can sanity test new code. This generally reflects the code in main before production does.
production: the thing customers use

Additional environments can include:

development: a safe space to perform architecture or third-party changes without disrupting the flow of all code to production. This is generally for developers only.
local: for local development
test: for test-running environments

Given the above, we can have configuration files which store safe values for each environment using types:

Note: as this grows, separate files would be welcome

config/getConfigValue.ts

// we can separate this into a `staging.ts` file
const staging: ConfigKeys = {
    PAGE_TITLE: 'My website',
    NEW_SEARCH_BAR_UI_ENABLED: true
}

// we can separate this into a `production.ts` file
const production: ConfigKeys = {
    PAGE_TITLE: 'My website',
    NEW_SEARCH_BAR_UI_ENABLED: false
}

// type safe config
type ConfigKeys = {
    PAGE_TITLE: string;
    NEW_SEARCH_BAR_UI_ENABLED: boolean;
};

type Config = keyof ConfigKeys;

// we can add as many environments as we need, even a 'default' environment for fallback (although this could be production for safe use in most cases)
type DeploymentEnvironment =
    | 'staging'
    | 'production';

type ConfigKeysForEnv = Record<DeploymentEnvironment, ConfigKeys | undefined>;

const configKeysForEnv: ConfigKeysForEnv = {
    staging,
    production,
};

export function getConfigValue<T extends Config>(
    key: T,
    deploymentEnvironment: DeploymentEnvironment,
    override?: ConfigKeys[T]
): ConfigKeys[T] {
    const deployment = deploymentEnvironment || 'production';

    if (!(deployment in configKeysForEnv)) {
        throw new Error(
            `Unsupported deployment environment provided for config: ${deploymentEnvironment}`
        );
    }

    const config = configKeysForEnv[deployment];
    if (!config || !(key in config)) {
        throw new Error(
            `Config key does not exist for deployment environment: ${key}`
        );
    }

    const value = config[key];

    if (override !== undefined) {
        return override;
    }

    return value;
}

Yeah yeah, development 'code' next to production 'code', there are trade-offs.

5. Feature Flag service

Conclusion

Let me know how you go moving away from using environment variables for everything.

Please feel free to leave any feedback as I'm always happy to chat about different opinions & experiences and learn new things.

Note: I like using a common image theme for my posts so whilst Bondi makes no sense in a technical blog, nerd talk and code by itself would be boring.

Downsides to this, multi language environment will need to incorporate independent implementations per language. Whilst I can say this is the cost of doing business, properly, this is overhead
Security
Testable
testing production out locally
let's other people in your team who are not technical manage the commercial side of the product re: feature flags
production secrets being committed

I might be a minority and a bit bias, but low code / no-code solutions generally irk me and I would often suggest avoiding them. Like tennis, playing in the middle is risky.

but but but development values in production and vice versa

Run time will pick up env var change

Overrides for an env in db (singleton table) and overrides at workspace and even user level

Current impls

Dev has changed due to this evolution

Low code no code sucks

Dynamic data > low code

The best get better all the time 💫

Matt — Thu, 09 Jan 2020 08:54:20 +0000

Every team’s continued success is up to how much they keep improving their people, processes and product/s.

Continuous improvement can be difficult for even the most refined and experienced of teams and this can be due to many things; time constraints, execution, following through on actions and so on.

We’re building a platform that helps teams practice continuous improvement seamlessly within your iterations so you can deliver only the best value for our customers.

Starting with a free retro tool, the team at Better have lot’s of plans to help make your team keep on improving, such as automation and integrations, so stay tuned over the coming months.

In the meantime, we’re always happy to hear your feedback, so please let us know what you think so far and what you’d like to see next!

DEV Community: Matt

A better way to do environment variables

Current State & Criticisms

Developer Experience & Confidence

Security

Static vs Dynamic

Deployment

Solution

1. Categorising values

i. Environment Values

Traits

Examples

ii. Secret values

Traits

Examples

iii. Static, non-secret values

Traits

Examples

iv. Dynamic, personalised values

Traits

Examples

2. Use a typed environment variable file

3. Secrets Manager service

4. Config files

5. Feature Flag service

Conclusion

The best get better all the time 💫