DEV Community: Josh Cole

How to Publish an NPM Package from a GitLab CI/CD Pipeline

Josh Cole — Wed, 08 Jun 2022 22:00:54 +0000

TL;DR

Take a look at the .gitlab-ci.yml configuration file in my recent safe-env-vars project. This shows how I configured GitLab’s built-in CI/CD pipelines to automatically build and publish the package to npm. This post will explain the steps required to set this up in detail.

What’s in a pipeline?

Long before GitHub joined the party with GitHub Actions it was possible to run continuous integration and deployment (CI/CD) workloads on GitLab. For the best part of a decade now, we’ve been able to use the same tool that hosts our code repositories to run our pipelines. I was previously a big fan of Circle CI, especially when they introduced support for Docker containers, but I haven’t needed to use these guys in years.

But why do we care about CI/CD pipelines in the first place? Well, continuous integration (CI) and continuous deployment (CD) are essential for ensuring software quality and agility in distributed teams. They allow us to run automated quality checks, execute our test suites, scan for security vulnerabilities, publish and deploy our apps, and a myriad of other important tasks. They’re also great at saving time on small projects such as npm packages!

At a high level, CI/CD pipelines cover these areas:

Building — Installing dependencies, compiling your code or application, and anything else required to prepare your project for testing and deployment.
Verifying — Performing static analysis of your codebase, assets, and compiled artifacts to ensure they meet quality and security standards.
Testing — Executing your test suite to confirm everything works as expected and no bugs have crept in.
Deploying — Publishing your package or deploying your application to the cloud.
Reporting — Providing insight into build errors, development velocity, security vulnerabilities, and a lot more.

What’s in a GitLab pipeline?

Recently I wrote a post about environment variables where I discussed an npm package I had published called safe-env-vars — that package is automatically built and published using GitLab CI/CD (and it’s open source) so if you want to skip ahead why not take a browse of its code repository. The main point of interest for those wishing to skip ahead 🏃💨 is the .gitlab-ci.yml file which tells GitLab about the jobs we want to execute in the pipeline.

Before we can get into the nitty gritty there are a few terms GitLab uses that we ought to define:

Pipeline — A single CI/CD execution consisting of several stages and jobs, typically triggered by a commit or a merge into a specific branch. You can have different pipelines for different purposes, for example, to verify changes in a merge request, or conduct a cloud deployment.
Stage — A grouping of related jobs that must succeed before GitLab will proceed to the next stage in the pipeline. If there are no further stages, the pipeline will complete successfully.
Job — A task you define that typically runs in its own idempotent container (e.g. npm run build, npm publish, etc). Multiple jobs within the same stage can be run in parallel as they are usually independent of each other.

GitLab pipelines are very powerful and can be configured in a lot of different ways. Some features are restricted to the paid and enterprise pricing tiers but all the features we’ll use below are available for free, as long as you have some pipeline minutes remaining! At the time of writing GitLab provides 400 free minutes per month for pipeline execution, which is more than ample for our needs here.

Our goal

So what are we aiming to do here? When I created the safe-env-vars package I had two goals:

Automatically publish my package to npm whenever I committed or merged into the main branch so I didn’t need to do this manually.
Run some code quality checks to catch any silly mistakes. I didn’t want to spend much time setting up these checks so I opted to limit them to code linting and unit testing.

Our pipeline will consist of three stages and four jobs:

Stage Name	Job Name	Job Description
Prepare	Install	Runs npm install to install key development dependencies such as TypeScript and Jest, and to build our package using the TS compiler.
Verify	Lint	Runs ESLint to verify our code meets the agreed linting rules.
	Test	Runs Jest to verify all unit tests pass and nothing has broken.
Publish	Publish	Runs npm publish to publish our package to the npm registry.

Our final pipeline will look something like this:

Step by step guide

Step 1: Create your package

Before you configure the pipeline you’ll need to have created a package with a suitable package.json file. Besides including the usual properties such as the dependecnies, author, and so on, you need to make sure the “main” property points to the JavaScript file which is the entrypoint to your package.

For TypeScript projects you’re going to need a few extra bits, but you can ignore these if you’re working with vanilla JavaScript (but why 😭):

The “types” property should point to the entrypoint for exported type definitions.
The “scripts.build” property should call the TS compiler, something like "build": "npx tsc --outDir ./dist".
The “scripts.prepublishOnly” property should look like "prepublishOnly": "npm run build" in order to automatically build your package before publishing with npm publish.

You might end up with a package.json file looking similar to this:

{
    "name": "my-package",
    "version": "1.0.0",
    "main": "dist/src/main.js",
    "types": "index.d.ts",
    "scripts": {
        "build": "npx tsc --outDir ./dist",
        "lint": "npx eslint . --ext .ts,.tsx,.js,.jsx",
        "prepublishOnly": "npm run build",
        "test": "npx jest --watch",
        "test:cov": "npx jest --coverage --verbose --runInBand"
    },
  "dependencies": {...},
  "devDependencies": {...},
}

My personal preference is to have additional scripts defined in the package.json for linting (lint), testing (test), and testing with coverage (test:cov).

Step 2: Add an npm ignore file

In the root of your code repository you’ll want to add a file called .npmignore and within it list all the paths you wish to exclude from your published package. This is important because there are some files you don’t need to include, and some you definitely shouldn’t include.

The syntax is very similar .gitignore and you might end up with something like the below. This ensures the src directory doesn’t get included (for TypeScript projects we only want our dist directory published). It also lists the various configuration files which aren’t important for people to actually use the package in their apps and just add unnecessary weight to a node_modules directory.

src/
.gitlab-ci.yml
.eslintignore
.gitignore
.eslintrc.js
.gitmodules
.prettierrc.js
jest.config.mjs
tsconfig.json
dist/**/*.spec.*

Step 3: Add the CI/CD config file

In the root of your code repository create an empty .gitlab-ci.yml file with the following contents (the full reference for GitLab’s config file is available here: https://docs.gitlab.com/ee/ci/yaml):

image: node:16

variables:
  ARTIFACT_DOWNLOAD_ATTEMPTS: 2 # Number of retries for restoring saved artifacts.
  FORCE_COLOR: '1' # Whether to force colour output in compatible tooling.
  GET_SOURCES_ATTEMPTS: 2 # Number of retries for git commands.
  GIT_STRATEGY: fetch # How to pull submodules.
  GIT_SUBMODULE_STRATEGY: recursive # How to treat nested git submodules.
  RESTORE_CACHE_ATTEMPTS: 2 # Number of retries for restoring cached files.

stages:
  - prepare
  - verify
  - publish

The YAML above is in three parts:

The image string specifies the Docker image and tag to use (by default) for each of the jobs, in our case Node.js version 16.x. This can be overriden in each individual job later if needed. Advanced developers might want to create and publish their own Docker images with any additional dependencies their pipeline requires.
The variables dictionary lists the environment variables to make available to each job. These can also be customised for each job. I’ve started you here with some useful variables that customise the behaviour of the pipeline (see the comments beside each for what they do). For example, the FORCE_COLOR variable is used by many (but not all) tools to force them to output ANSI colour codes even when they can’t detect a TTY device.
Finally, the stages array lists the names of each of the three stages that will house our jobs. We will define each of the jobs next.

Step 4: Add the “install” job

At the bottom of your config file add the following top-level key, install, which defines the job named “install”. The install.stage property defines which stage the job belongs to, in this case the “prepare” stage.

install:
  stage: prepare
  interruptible: true
  retry: 1
  dependencies: []
  artifacts:
    name: 'deps_$CI_COMMIT_REF_SLUG'
    paths:
      - node_modules/
    when: on_success
    expire_in: 12h
  script:
    - npm config set -- '//registry.npmjs.org/:_authToken' "${NPM_AUTH_TOKEN}"
    - npm install --no-progress --no-audit --no-fund --verbose
  only:
    - merge_requests
    - develop
    - main

The full keyword reference on GitLab’s website will explain all the various properties defined above, but the important ones to note are:

“artifacts” — Tells GitLab which path(s) to hold on to between jobs, in this case, that’s the node_modules directory as we’ll be running npm install in this job only, and preserving the installed dependencies for use in subsequent jobs to save time.
“script” — The lines of the shell script to execute for the job, in this case to set the NPM auth token that allows publishing to the registry. The NPM_AUTH_TOKEN environment variable should be set in the GitLab UI for your project or group and not comitted to your code repository! You can find the correct page in GitLab by clicking CI/CD in the left-hand menu of any project or group and going to the section titled “variables”.
“only” — An array listing the events that can trigger the job to be included in the pipeline, in this case we execute the job for merge requests and for commits/merges to the develop and main branches. If none of the conditions match, the job will not be included in the pipeline.

Step 5: Add the “lint” job

Add the next job, lint, at the bottom of the config file:

# Lints the codebase.
lint:
  stage: verify
  interruptible: true
  retry: 1
  dependencies:
    - install
  script:
    - npm run lint
  only:
    - merge_requests
    - develop
    - main

Here, we have a similar set of properties defining what our job does, including stage, this time grouping our job in the verify stage, and script which simply runs the npm run lint script we defined in our package.json.

There is a new property to pay attention to, however. In order to execute our lint script ESLint and its associated dependencies need to be installed. Each CI/CD job is run in a new container and so doesn’t have any of the changes introduced in the preceeding install job, such as the node_modules directory.

In order to bring our installed node_modules into this lint job we need to specify the dependencies array and list our install job inside it. This will cause GitLab to pull in all the artifacts specified in the install job (i.e. node_modules) and put them back in the same places they were before. As far as npm will be concerned, the node_modules directory was always there!

Step 6: Add the “test” job

Add the next job, test, at the bottom of the config file:

# Runs the test suite.
test:
  stage: verify
  interruptible: true
  retry: 1
  dependencies:
    - install
  script:
    - npm run test:cov
  only:
    - merge_requests
    - develop
    - main

Once again, our job specifies the install job as a dependency so the node_modules directory is available again, and we specify a simple bash script that runs npm run test:cov - this simply executes our test runner, Jest, with coverage turned on. We also put this job under the verify stage and it will be run in parallel with the lint job to save time.

You might have noticed we specify a property called interruptible: true which simply tells GitLab it is allowed to kill our pipeline whilst this job is running if needed. This is useful when a new pipeline wants to start (i.e. something new has been committed to the same branch) that would otherwise have to wait for the current pipeline to finish, saving us both time and pipeline minutes!

Step 7: Add the “publish” job

Add the next job, test, at the bottom of the config file:

# Publishes the package to npm.
publish:
  stage: publish
  interruptible: false
  retry: 1
  dependencies:
    - install
  script:
    - npm config set -- '//registry.npmjs.org/:_authToken' "${NPM_AUTH_TOKEN}"
    - npm publish --verbose
  resource_group: 'deployment-$CI_COMMIT_REF_SLUG'
  rules:
    - if: '$CI_DEPLOY_FREEZE != null'
      when: never
    - if: '$CI_COMMIT_REF_NAME != "main" && $CI_COMMIT_REF_NAME != "develop"'
      when: never
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_REF_NAME == "develop"'
      when: never
    - when: on_success

Here we set the interruptible property to false because we don’t want GitLab to kill the pipeline halfway through a publish, leaving us in a situation where we can’t be sure what state our package is in… did that version get deployed or not? 🙀📦

We also specify retry: 1 which tells GitLab to retry the job one additional time if it fails. This can be useful in cases where you’re relying on side effects for the job to succeed, for example npm publish makes HTTP requests to the npm registry which could fail due to transient network faults or a problem on npm’s side. In those cases, we want to automatically retry in case the problem resolves itself.

The other property of import here is rules where we list an array of objects. Each object defines a rule for when our publish job should run. We’re using this property instead of only here because we need to define much more complex behaviour than only will allow.

In the above code we have specified four rules and the first rule to match will be applied:

If the CI_DEPLOY_FREEZE environment variable is set, never execute the publish job.
If the branch we’re running on isn’t main or develop, never execute the publish job.
If we’re in the scope of a merge request, never execute the publish job.
Otherwise, and if the rest of the pipeline has been successful, then execute the publish job.

The syntax for rules is very powerful and allows us to specify a complex set of conditions for when the job should run (or not). There is far more to this than I can possibly detail here, so I encourage you to refer to the GitLab documentation: https://docs.gitlab.com/ee/ci/yaml.

Step 8: Commit and push

Once your package and the GitLab CI/CD config file are ready, you can commit and push your changes up. If everything has gone to plan, you should see a pipeline appear in the GitLab UI for your repository under CI/CD in the left-hand menu. By clicking into the pipeline you can follow through the jobs as they execute, and drill down into the output logs for each individual job.

Below you can see an example of four pipelines that have been run. The first pipeline at the top of the list is the most recent and was run against the main branch. The second pipeline down was run on a merge request and does not include the publish job as the rules we configured exclude it from being included in merge request pipelines.

Clicking into the first pipeline shows the stages and jobs that were executed. As this pipeline was run against the main branch our rules caused the publish job to be included and the package was subsequently (and automatically) published to npm. Happy days! 🥳

From here if we click into the test job we can see a snippet of the output emitted by Jest during the job. There is also some other log output which isn’t shown here for brevity.

Conclusion

In conclusion, it’s actually very simple to setup a powerful CI/CD pipeline in GitLab, especially if all you’re doing is publishing a package to the npm registry. Pipelines are a great way to verify the quality and robustness of your code, package, or application, and they are a fantastic timesaver when it comes to automatic publishing and deployment. To configure a pipeline in GitLab all you need is a .gitlab-ci.yml file in the root of your repository. You can find the keyword reference for that file on GitLab’s website.

You’re Doing Environment Variables All Wrong - A Node.js Perspective

Josh Cole — Sun, 08 May 2022 23:43:06 +0000

TL;DR

Environment variables aren’t always what you expect and it’s painful to check each one. Instead, use a library such as safe-env-vars to do the hard work and be safe in the knowledge your environment variables won’t cause you any headaches.

Oh, what?

Environment variables are easy, you say, we’ve been working with environment variables for our entire careers... how could we possibly be “doing them wrong”?! Well, as American computer scientist Jim Horning said, “Nothing is as simple as we hope it will be”. And in this case, a risk is introduced every time you ‘set and forget’ a variable. Let’s explore the problem, or rather, problems.

Let’s start at the top

So what are environment variables and why do we use them? Put simply, environment variables are pieces of state (read; string values) that we store in the ‘environment’ that our application is running in. This state is usually set via one of the mechanisms provided by the operating system, shell, or container orchestrator, which is responsible for our application process.

Environment variables are a simple mechanism, and this is good thing because a lot of engineering is not so simple.

“Simplicity is prerequisite for reliability. “ — Edsger Dijkstra.

Often in engineering we need to iteratively refactor and rework our solutions until we reach a good balance between readability and functionality. Here, simplicity is our friend because it makes it easier to understand what our code is doing and why. We’re far less likely to end up with misbehaving, buggy software if it’s simple.

See, it’s mostly upside!

Well yes, there is an awful lot of upside. As we shall see storing state in the environment allows us to do several very useful things that would otherwise be risky or time consuming.

1. Change configuration at will

We can change the behaviour of our application whilst avoiding risky activities such as changing source code, and time consuming chores such as re-compiling, re-deploying, testing, and so on. If we need to rotate API keys, turn feature flags on or off, or adjust some other behaviour, we can do all this from the comfort of our chairs simply by deploying the new values and restarting our applications.

2. Keep secrets hidden

We can store secrets separately to our source code. This helps us mitigate the risk of sensitive values such as API keys, credentials, and so on that would put our users at risk if they were to be exposed. This way, if a nefarious actor gains access to our source code, they won’t get their hands on the secrets at the same time. It makes it harder for them to do us damage.

3. Stay on the right side of regulation

In regulated industries it’s often necessary to limit personnel access to sensitive systems to a limited number of specific people. By storing the secrets separately to the source code, the engineers can still do their jobs effectively without the keys to the kingdom sitting within their reach.

4. Set different values per engineer or environment

Whilst working locally we often need to use different values for API keys, feature flags, and behaviour flags that make sense whilst developing but not in deployed environments. The same can be said of automated testing where tests may need to change the application’s behaviour and inputs to test particular aspects.

Each deployed environment can be given a different set of environment variables, for instance to keep production secrets isolated and separate from staging secrets. As with local development, we can also change the values in our staging/testing environments independently of the other environments as needed. Flexibility is great!

5. Use dot env files

In the expansive JavaScript universe a common pattern is to use the dot-env package to read in environment variables from a local .env file that is not committed to the repository. This is a much quicker (and importantly more visible) alternative to setting environment variables in the actual environment. Engineers can change the values quickly and easily whilst developing as the need arises.

So what’s the problem?

There are a few. These are all risks that we need to mitigate for, vulnerabilities that can leave us open to attack, and mistakes that can cause unexpected behaviour at the worst times. Even in the best case scenario, badly behaving environment variables can waste a significant amount of time, especially in dynamically typed languages such as JavaScript.

“Seek simplicity but distrust it.” — Alfred North Whitehead.

We need to be careful not to fall in to one of the myriad traps. In each case, it’s hard if not impossible to predict how our application will behave. Sometimes issues are immediately obvious, but in many instances we won’t know about an issue until it randomly rears its head at the most inconvenient time.

1. Missing values

The most obvious risk here is that a value could be missing. This is more likely to be the case on our local machines where one developer makes a change that requires an environment variable we haven’t got set in our local environment. It’s less likely to happen in deployed code which has gone through several layers of reviews and testing, but it can still happen with complex systems. We’re only human after all!

LOG_LEVEL="TRACE"
#API_KEY="..."
DATABASE_URL="..."

Oops, we disabled the API_KEY value and forgot about it. Or perhaps our colleague added ACCESS_TOKEN_TTL in their latest commit and you haven’t noticed you need to add it to your local .env file.

2. Empty values

Similar to missing values, it’s possible for the value of an environment variable to end up as an empty string. Perhaps that was intentional (though it probably shouldn’t be), but how would we know?

LOG_LEVEL=""

What exactly does the above mean to you? Does it mean we want to turn logging off entirely? Does it mean we want to use the default log level and we don’t care what it is? Or (more likely) has something broken that we need to fix? Ask your friends, you might find they have diverging expectations to you.

3. Arbitrary values

Environment variables are often used for boolean values such as feature flags. Booleans have some big downsides which I won’t go into here, but safe to say those boolean values are arbitrary and different engineers will use different values.

For example:

FEATURE_FLAG_AAA="true"
FEATURE_FLAG_B="TRUE"
FEATURE_FLAG_c="yes"
FEATURE_FLAG_c="Y"
FEATURE_FLAG_c="1"

As humans, we instantly know that all these values all represent the same thing, that a particular feature flag has been toggled on. We rely on conventions and consistency to ensure we don’t fall into the trap of using different values in different places, but good intentions won’t always help when herding cats 🐈 (engineers).

The same can be said if you use enum values, such as with log levels (INFO, DEBUG, TRACE, etc). Obviously you could end up with an invalid value that may throw a spanner in the works unless you validate the value you read from the variable... but how many of us really do that? 🌚

4. Incorrect types

We covered the problem with boolean values above, it’s a similar story if you need to use a value as a number. Environment variables are always read in as strings regardless of what value you’ve stored in them:

FEATURE_FLAG_AAA="true"
SOME_NUMBER="3"

Maybe you need the SOME_NUMBER value to be a number so TypeScript will allow you to pass it to the nice library you want to use. Do you parse the value to an integer like this?

const value = Number.parseInt(process.env.SOME_NUMBER);
someNiceLibrary(value);

And what if that value gets changed to a float in one environment but not another?

SOME_NUMBER="3.14"

Suddenly your application is freaking out but you don’t know why. Your seeing some weird behaviour but you don’t know why, or perhaps worse, you’re seeing an error message stack trace that is a red herring and points you totally in the wrong direct for an hour whilst your customer is yelling at you.

You might argue that this issue is more likely to occur in JavaScript than other languages, but unexpected behaviour is always a risk when dealing with side effects like environment variables.

5. Optional values

Another consideration is that sometimes we really do want values to be optional, where things like the following may be totally valid given our context:

#FEATURE_FLAG_AAA="true" # 1. comment out a value we don't need at the moment.
FEATURE_FLAG_AAA="" # 2. or set it to an empty value (not so good!)

If we’re manually checking environment variables to ensure they exist we need to leave this one variable unchecked as it may be optional. This introduces the human element whereby future engineers may not add in presence checks where needed because they see they aren’t consistently applied to all variables. The variable is implicitly optional and this leaves it open to interpretation by the reader. Better to be explicit when variables are optional as the majority (i.e. the default) will be required.

6. Hidden environment variables

It’s a poor (but sadly common) practice for engineers to read in an environment variable at the point they want to use it, for instance:

function calculateCommission(amount: number): number {
  return amount * Number.parseInt(process.env.COMMISSION_RATE);
}

What’s the problem here? Well our nice calculateCommission function can exhibit odd behaviour if our COMMISSION_RATE environment variable is missing or set to some weird value. Perhaps the engineer that wrote this forgot to update the documentation to indicate that the commission rate needs to be configured in the environment and you didn’t realise you needed to do it. Whoops.

7. Behaviour and security

Environment variables are side effects. You might say they add impurities to our code. Our application can’t control the values it’s reading from the environment and must accept what it’s given. This means environment variables are akin to user input and carry the same risks. ☠️

The value of an environment variable could be unexpected, or worse, malicious. Best case, the value triggers a visible error that leads you down the garden path for an hour or two before you figure out what’s actually causing the issue. Worst case, you have exposed your application to input you can’t trust (and you have trusted it absolutely) without verifying it’s authenticity or correctness, and now you have been storing sensitive data in the attacker’s message queue for the last 2 weeks rather than your own. 😬

Right, how do we sidestep these issues?

Simplicity is fantastically splendiferous, except when it’s not.

“Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains.” — Steve Jobs.

The trick as with all ‘user’ input outside our sphere of control, is to trust but verify, or in our case, trust but validate. There are a few things you want to do for every value you read in from the environment:

Presence checks - ensure expected environment variables are defined.
Empty checks - ensure expected values are not empty strings.
Value checks - ensure only expected values can be set.
Typecasting - ensure values are cast to the expected type at the point you read them in.
Single entry point - ensure all variables are pulled in at the same place, and not smeared around your codebase for people to stumble upon later.
Dot env - read values from both a .env file and the environment.

Writing the code to do this for every project would be a pain, but the good news is, I’ve already done that for you.

Package: safe-env-var

safe-env-vars will read environment variables from the environment as well as a .env file in a safe way with full TypeScript support. By default, it will throw an error if the environment variable you're trying to read is undefined or empty.

It’s very quick to get started with basic usage if all you’re doing is reading in string values that are always required:

import EnvironmentReader from 'safe-env-vars';

const env = new EnvironmentReader();

export const MY_VALUE = env.get(`MY_VALUE`); // string

You can explicitly mark variables as optional:

export const MY_VALUE = env.optional.get(`MY_VALUE`); // string | undefined

Or you can allow the variables to be an empty value, though I would discourage this for the reasons stated in the discussion above:

export const MY_VALUE = env.get(`MY_VALUE`, { allowEmpty: true }); // string

You can even cast the type of the value as you’d expect:

// Required
export const MY_BOOLEAN = env.boolean.get(`MY_BOOLEAN`); // boolean
export const MY_NUMBER = env.number.get(`MY_NUMBER`); // number

// Optional
export const MY_BOOLEAN = env.optional.boolean.get(`MY_BOOLEAN`); // boolean | undefined
export const MY_NUMBER = env.optional.number.get(`MY_NUMBER`); // number | undefined

And finally, you might want to check whether the variable is one of the allowed values. This check always occurs after the presence/empty checks and typecasting the value.

export const MY_NUMBER = env.number.get(`MY_NUMBER`, { allowedValues: [1200, 1202, 1378] ); // number

See the docs for more usage information and examples.

Recommended pattern

I would recommend you have a single point of entry for the environment variables in your application. One place where you read in all the values needed by the different modules and functions. This ensures that there’s only one place to look and one place to change when making modifications.

I like to structure my single point of entry in JavaScript/TypeScript projects like this:

/src/
    /main.ts
    /config/
        /env.ts
        /constants.ts
        /index.ts

./config/env.ts

import EnvironmentReader from 'safe-env-vars';

const env = new EnvironmentReader();

export const COMMISSION_RATE = env.number.get(`COMMISSION_RATE`); // number

./config/constants.ts

export const SOME_CONSTANT_VALUE = 123;
export const ANOTHER_CONSTANT_VALUE = `Hello, World`;

./config/index.ts

export * as env from './env';
export * as constants from './constants';

...and the usage?

import * as config from './config';

const { COMMISSION_RATE } = config.env;
const { SOME_CONSTANT_VALUE } = config.constants;

export function calculateCommission(amount: number): number {
  return amount * COMMISSION_RATE;
}

This results in a very clean way of working with configurable environment variables as well as constant values. The benefits of this approach are that there is a single point of entry for the environment variables in your application, and every usage of these values directs the reader back to that entry point.

Conclusion

Don’t fall into the trap of believing that because you’ve been using environment variables for years that they’re safe and can’t surprise you. It’s better to trust but verify the values you’re reading using a robust and time-saving library such as safe-env-vars* which does the hard work for you.

*Alternative options may exist. 🙃

Engineering Principles for Unruly Artists

Josh Cole — Wed, 30 Dec 2020 12:32:13 +0000

Leading a team of intelligent and passionate engineers can feel too much like herding cats at times. Problem solving of any kind is a deeply creative activity, and with creativity comes mess. Lots of mess. This is particularly the case when there isn't just one artist painting on canvas, but many, with their respective paint brushes fighting to manifest their unique ideas.

We all come from different backgrounds, have differing educations, and unique life experiences. These individual perspectives are built up over a lifetime and they can and should lead to healthy (and most importantly, productive) technical discussions.

However, that's not always the case! I've seen teams where the conflicting viewpoints constantly trigger meaningless debates. Debates that divert the team's finite energy and brainpower from the key tasks at hand.

I've seen teams where specific members have such entrenched views they can't begin to see a different way of working. They often end up going off on their own and writing code only they can work with effectively.

What's the result? Paint everywhere.

This crazy artistry is why we see the most effective teams being those that are well disciplined, working in lockstep. Each member playing their own unique role but in concert with everybody else.

It's tempting to look at that discipline and determine that it must come from an autocratic command and control approach, with rules and orders flowing down from the top to the implementers below.

This is "easy" for leadership to do, it's also incredibly lazy. Nobody has to think about integrating conflicting ideas, even if that means good ideas, better ideas, are lost. Because the "best" way, the only way, has already been dictated.

The problem is, it would be impossible to write up a complete list of rules for how to engineer software. The list would be insanely long, and many of the rules would conflict with each other. Not to mention you'd need to know the context of each situation to judge whether the rules had been followed appropriately.

This kind of dictatorial rule-setting ends up being counter-productive as it requires either dumb obedience or cognitive overload. Neither are desirable when cognitive energy is at such a staggering premium. Hiring more people and more expensive skill sets simply isn't the smartest move.

So what's a better way to manage creative individuals?

Practical Principles

A good analogy for principles are the guide rails bowling alleys provide for children and the perennially inaccurate. They don't guarantee you're going to hit a strike on your first bowl but they do ensure the resulting output is within an acceptable range.

Successful entrepreneur Elon Musk and others talk about reasoning from first principles in order to solve problems more effectively. Why should software engineering be any different?

Instilling a set of principles in a team allows for much faster implementation and iteration. There's just no need to think about how to architect solutions or to confirm every decision with a superior, because an acceptable range of "hows" has already been defined.

Anything that reduces cognitive load and opens up more mental capacity for solving problems is a winner in my book. The trick is for the principles to leave sufficient scope for innovation, personal flair, and team input, without being so loose as to be useless.

The principles I personally like to follow err towards maintainability and practicality. I have a bias for getting stuff done rather than indulging in an academic sense of correctness, because for me, it's more important to have a mostly correct but useful output, than a perfectly correct pipedream.

That probably comes from a background in innovation rather than engineering. A different perspective for sure. Now of course this won't be appropriate for all situations so please modify as appropriate!

My Principles

This list is my current best effort as of December 2020 and I fully expect it to change over time as I learn new things. The principles below are in ascending order and build upon each other.

1. Simplicity

Generally said, it's better to favour simple solutions over complex ones. Sometimes the overarching solution may be inescapably complex, but the constituent parts of that solution can still be kept simple themselves.

Simplicity is good for many reasons, not least because we're only human and we will all make mistakes. In short, simple means easy to understand... easy to understand means safe to change... safe to change means a low risk for introducing bugs... and so on.

Typically I'd consider a concept to be simple if a developer can hold the entire working knowledge of it in their head at one time. Obviously this doesn't apply to entire software systems, but rather to each individual part.

2. Consistency

It's no good just choosing simplicity over complexity. The way code is written and the way problems are solved should also be broadly consistent across a codebase. Having 500 unique approaches might yield marginal innovation in some areas, but it's painful, time consuming, and costly to maintain such unlimited variety.

Consistency also applies to conforming to the generally accepted way of doing things both inside and outside your current team and organisation. That's because developers coming from elsewhere will probably expect things to work a certain way.

We have international standards for a reason, the same reason that we tell developers "don't repeat yourself" and consider "code reusability". Doing things differently and reinventing the wheel for no good reason introduces cognitive overhead. And bugs sure love to take advantage of overloaded developers.

3. Legibility

Simple and consistent code lends itself nicely to legibility - the quality of being clear to read and easy to digest. Maybe you can impress yourself with the shortest possible piece of code. Great. Pat on the back. Now give that to someone else after you're long gone, or even your future self next year, and hope they can figure it out in a sensible amount of time.

Whether it's showing off or laziness, short code is a pain to unpick. It's a pain to review. It's a pain to modify. A bit more verbosity to make your intentions clear goes a long way to explaining your thought process to other developers.

Engineering is a team sport that requires constant communication through code, and as with any form of communication you can't expect others to magically devine what's going on inside your head. Make it explicit. As in life it's better to over-communicate than to under-communicate.

4. Responsibility

In all likelihood, code that is simple, consistent, and legible, probably only does one thing. That's because code that only has one responsibility is much easier to reason about. It's easier to understand how the internals work and how it fits into the larger picture as a whole.

Singular responsibility is a sturdy design principle that applies at all levels. From the application as a whole, to each microservice, class, function, variable, all the way down to individual lines of code. Everything benefits from having a clear, singular purpose.

It also makes it infinitely easier to reuse code in multiple places when that code only does one thing, but does it well. It becomes possible to stitch together (compose) units of code in multiple different ways to produce multiple different outcomes. A core tenet of functional programming.

5. Predictability

Code that is simple, consistent, legible, and has a singular responsibility, provides an amazing foundation for predictability. Now we're really getting to the crux of the matter, because for me, predictability is the holy grail of building reliable, bug-free software.

Often our software can seem like a wild beast. We release it into the wilderness* (*also called production) with what we think is a good understanding of how it will behave, only to find that when our wild beast meets reality its behaviour is anything but rational.

Of course we use automated testing, code quality checks, partial roll-outs, automated rollbacks, code reviews, and a whole host of other techniques to assess the behaviour of our code. And these things are absolutely necessary in an imperfect world.

But we could have made our lives a whole lot easier if only we had started with code that's predictable in the first place. That means following all of the above principles as well as more concrete techniques such as minimising side effects, avoiding shared state, and so on.

If we can predict how our code is going to behave before we release it into the wild we'll be able to pre-empt and correct much more of its undesired behaviour before it impacts users.

6. Reliability

Hopefully by now you get how this works. Code that is simple, consistent, legible, with a singular responsibility, and is predictable, has the potential to be considerably more reliable.

Reliability refers to two things here; firstly that the software conforms to whatever specifications have been set out, whether that comes from internal stakeholders or from the customer(s) themselves.

This is important because we need to know what the software should be doing even if that's at a high level (we don't want to be doing waterfall here). As discussed above we can use automated testing and associated techniques to assess this.

Secondly, reliability refers to the software behaving as expected when end-users get their hands on it out in the wild (AKA production). Now I'm not saying that it should be perfectly bug free. A known bug that reliably results in the same behaviour every time it occurs may be a nuisance but it's a manageable nuisance.

Reliable software is software that consistently behaves in the same way every time (that also includes being "up" and available when a user needs to access it, in case you're asking). The key here is consistent behaviour, users will be able to work around limitations and bugs, and as a team you'll be able to fix them that much easier.

Conclusion

So there you have it. For software to be reliable it must first be predictable, which requires that each part has a single responsibility, that the code is legible, and that problems are solved in a consistent way, recognising that simple solutions are usually better than complex ones.

These are the six principles by which I guide my teams to design and build high quality software. It may be an ever evolving set of principles but it's a set that's carried me a fair distance so far, and I hope with a bit more tweaking, much further into the future.

Comments, war stories, and differing opinions are welcomed! :)

5 VS Code Extensions For Team Productivity

Josh Cole — Tue, 17 Nov 2020 20:23:51 +0000

Is your team as productive as it could be? Maybe it's close, although I'd put money on there being room for improvement in there somewhere.

Now, IDE extensions aren't going to solve major problems by any stretch of the imagination, but they will help you eek out that little bit more productivity without asking anything more of the team.

And that's what productivity is often about; lots of small tweaks and adjustments that allow humans to do what they do best (craftsmanship and creative work) by leveraging better tools.

As Confucius said;

"If a craftsman wants to do good work, he must first sharpen his tools."

The following are five VS Code extensions that I encourage every engineer on my team to utilise. Each of them has a small but valuable incremental contribution to productivity, hence the recommendation.

5 - Todo Tree

Description

Todo Tree quickly searches your workspace for comment tags like TODO, HACK, and FIXME, displaying them in a tree view in the sidebar.

Clicking a TODO within the tree will open the file and navigate to the correct line, whilst the comments themselves will be highlighted so they're visually annoying and less likely to get missed. This can be configured in settings.

Why

I generally don't like to see "todo" comments like this left in code, particularly code that's supposedly "finished" and/or going into production.

// TODO: implement new feature
// FIXME: this is a known issue I haven't fixed yet
// HACK: revisit this later when we have more time

That being said, life is messy and it's not always possible to avoid such things. Developers are bound to write these comments and they sometimes get accidentally committed to source control, it happens. 🤷‍♂️

And there are times when the comments have actual value such as when pushing code at the end of the workday to act as reminders for tomorrow, or if a developer is off sick and someone else needs to pick up where they left off.

I don't tend to encourage these comments, but given the fact they will crop up somewhere we might as well have a good tool for finding and dealing with them. This is especially useful when code is changing quickly in the early stages of a new project.

4 - GitLens

Description

GitLens provides a more visually friendly UI to work with Git repositories with shortcuts for many common uses of the tool.

It helps with visualising code authorship through surfacing useful information directly in the code editor, and via tools for git blame, history navigation, and code comparisons.

The extension also comes with lots of customisation options to adjust its behaviour to your individual tastes.

Why

I've used Git for every project for many years as I suspect you have too. Git is a complex tool to master fully and engineers in your team will be at different stages in their learning.

That's where this extension comes in. Simplification of a complex tool is a valuable endeavour. This extension also means developers don't have to switch context between their IDE, the command line and your VCS provider.

Since Microsoft's purchase of GitHub the Git integration in VS Code has come on leaps and bounds. Perhaps in the future this extension won't be necessary at all.

For now, GitLens enables some valuable insights into the evolution of your codebase without needing to resort to the command line or the online tools of your VCS.

3 - Editorconfig

Description

EditorConfig helps maintain consistent coding styles for multiple developers working on the same project across various editors and IDEs.

This extension reads the .editorconfig committed to your project's repository and automatically applies the settings within when you switch to the project. Full definition of the options available can be found here: https://editorconfig.org

Why

For me, consistency is an important principle of high quality engineering. Consistency reduces mistakes because it reduces the cognitive load on those reading or writing the code, and it quickly becomes implicitly understood how something should or could work without needing to deep dive every time a similar problem crops up.

Editor Config has been around for quite some time and it's one of those very simple tools that ensures a modicum of consistency between developers. I like to keep git diffs as clean as possible, and seeing changes to things like trailing whitespace, newlines, indentation characters, and so on quickly becomes painful.

The extension automatically sets the correct IDE settings for each project you switch into based on an .editorconfig file in the repository root.

Make a decision on the style to follow and be done with it. All developers with this extension installed will then be writing code in the same style, keeping git diffs clear of whitespace noise.

2 - Code Spell Checker

Description

Code Spell Checker is a basic spell checker that works well with camelCase code. The goal of the extension is to catch common spelling errors whilst keeping the number of false positives low.

It's not designed to be as comprehensive as a word processor spell checker but it does well enough to prevent silly mistakes.

The extension also supports many more languages than US English which is a boon for those of us who work in other variants of English or entirely different languages.

Why

Spelling mistakes are common. That might not be a dealbreaker for passive code such as comments, documentation, strings containing user notifications, and so on, but it can have a huge knock-on effect and mistakes tend to live forever.

See the case of the HTTP Referrer header that was misspelled in the original specification over 20 years ago and continues to live on forever more.

The misspelling of referrer was introduced in the original proposal by computer scientist Phillip Hallam-Baker to incorporate the "Referer" header field into the HTTP specification.[7] The misspelling was set in stone by the time (May 1996) of its incorporation into the Request for Comments standards document RFC 1945[8] (which 'reflects common usage of the protocol referred to as "HTTP/1.0"' at that time); document co-author Roy Fielding remarked in March 1995 that "neither one (referer or referrer) is understood by" the standard Unix spell checker of the period.[9] "Referer" has since become a widely used spelling in the industry when discussing HTTP referrers; usage of the misspelling is not universal, though, as the correct spelling "referrer" is used in some web specifications such as the <code>Referrer-Policy</code> HTTP header or the Document Object Model.[3]

View on Wikipedia

Incorrectly named functions, variables and so on tend to be hard to change once baked in, and may cause confusion and frustration for every developer that ever needs to read or use that code in the future.

Mistakes like this tend to take more energy to correct than you might want to expend. So save yourself the eternal embarrassment and get it right first time.

1 - Prettier

Description

Prettier is an opinionated code formatter. It enforces a consistent style by parsing your code and re-printing it with its own rules that take the maximum line length into account, wrapping code when necessary.

Prettier supports: JavaScript, TypeScript, Flow, JSX, JSON, CSS, SCSS, Less, HTML, Vue, Angular, GraphQL, Markdown, and YAML. Full definition of the options available can be found here: https://prettier.io

Why

Now I know not all engineers will be writing code in languages that Prettier supports, but for those developers who are it's a huge timesaver. Similar to the Editor Config extension above, Prettier automates work that is valuable but quickly becomes a huge time sink such as formatting and organising code to make it consistent and readable.

Prettier is useful for the languages it supports because those languages are so flexible they can be written in many styles. Not to mention that most of them don't have official style guides or formatting tools, unlike more modern languages.

I can't count how many hours I've seen wasted by developers arguing over whether semicolons are necessary in JavaScript, or whether indentation should be tabs or spaces, or reading git diffs that are mired with stylistic changes.

Whilst there are sensible answers to all these crinkles, I've seen these debates continue to rage unnecessarily in every team. Prettier puts an end to all that by enforcing an opinionated style on all developers with only a small amount of flexibility to change its behaviour.

What I love about Prettier is that code styling is applied automatically on save so I don't need to even think about how tidy I'm being, Prettier does that for me. Plus one for productivity!

🌍 Website | 📇 LinkedIn | 🐥 Twitter | 📕 Dev.to | 📗 Medium