DEV Community: Kostas Bariotis

Why you should not use process.exit()

Kostas Bariotis — Thu, 13 Jan 2022 00:00:00 +0000

I've been working with different teams and projects during the past few years and a common theme I've been seeing is the use of Node.js' process.exit() function.

Have you ever seen something like this?

// ...

process.on('SIGTERM', async () => {
  await server.close();
  process.exit(1);
});

// ...

So what's the problem with the above? Let's see.

What is `process.exit()`?

Node.js gives us the ability to forcefully exit the current process. Emphasis to forcefully because as we may all know, Node.js is an asynchronous event-driven runtime, meaning a process may have asynchronous operations running in the background or waiting in the event loop and process.exit() will ignore them and exit itself with out waiting.

How it works

Let's see an example. Consider the below snippet that will print the first 10 numbers of the fibonacci sequence.

const initial = [0, 1];

for (let i = 2; i <= 10; i++) {
  initial[i] = initial[i - 2] + initial[i - 1];
  console.log(initial[i]);
}

If you try to run the above, you will notice that the process will exit as soon as it finishes and will return control back to the terminal.

Let's see another example with an asynchronous operation this time.

let i = 0;
setInterval(() => {
    console.log('Infinite Loop Test interval n:', i++);
}, 5000);

This time if you try to run the above, it will never exit. It will just keep printing the above message to the terminal every 5 seconds. And the reason for that is that a Node.js process won't exit until the event loop is empty. But setInterval just keeps adding operations to the event loop in order to keep its recurring execution.

If we now add a process.exit() at the end of it, we won't even see the first message logged in the terminal, and thats how process.exit() forcefully exits a process.

The dangerous nature of `process.exit()`

Sometimes there are caveats that people may not be aware. Consider the below snippet from a simple web service that runs an HTTP server and a queue consumer.

The service replicates itself and lives behind a load balancer that will send a SIGTERM event whenever wants to make a new deployment for example.

The service is configured to close and exit the process when ever it receives the SIGTERM event which in a first glance it makes sense.

// ...

process.on('SIGTERM', async () => {
  await server.close();
  logger.error('Uncaught exception was thrown.');
  process.exit(1);
});

// ...

There are two caveats to look out fore here.

First, the above doesn't guarantee that the log will be properly written as the process of writing to the output is an asynchronous IO operation. The above is quite dangerous because you will end up in a situation where your process is just randomly halting without any explanation!

Second, we mentioned that that service runs an HTTP service but also a queue consumer. We've handled the closure of the HTTP server when we received the SIGTERM event but we haven't close the queue consumer. That means that asynchronous operations may be waiting to be processed or they may even be processing at that specific moment. When we call process.exit() those operations will be terminated and we will be looking afterwards to find what happened.

What should we do

It's simple. Never use process.exit() under no circumstances unless you specifically need to do it.

Node.js docs state that:

"In most situations, it is not actually necessary to call process.exit() explicitly. The Node.js process will exit on its own if there is no additional work pending in the event loop. The process.exitCode property can be set to tell the process which exit code to use when the process exits gracefully."

Make sure that you're correctly handling the initialisation of your IO operations in your app but that you also properly handling the termination of them. Once that's done, the process will have no reason to stay running and it will exit by it self.

The above will ensure the normal operation of any Node.js service.

Do you have any examples of how the process.exit() can be dangerous? Let me know in the comments, I would love to read them.

How I made a web app to create presentations

Kostas Bariotis — Mon, 17 May 2021 18:36:58 +0000

I had that idea for a while in my head. I was tired of every time I had to create a presentation and I was spending more time deciding where the header should go and if it's five pixels higher than it should be and if the colors match.

Today I'm glad to introduce you to Presentador. Now I can focus on writing the content of my presentation while Presentador takes care of the positioning of the elements and the colors. 🛠

In this post, I would like to explore some challenges I found along the way of creating a web app for presentations.

Presentation mode

A presentation app at the very least will allow you to present, meaning zooming the contents of the slides to play nicely on bigger screens, projectors, etc. When your presentation slides are basically HTML pages being rendered on a web browser, you can't rely on scaling using pixels or relative units like em. Different screens will have different resolutions and different user settings. So how do we scale to always make sure that the slides will keep their original shape? CSS scale to the rescue.

With scale we are able to resize our page in two dimensions. First, we get the viewport size of the current browser and then calculate how much to scale. Hats off to Reveal.js for giving me the idea. 💡

User text selection

I wanted to be able to edit all text-based components in place in each slide. I started experimenting with contenteditable and at first, it was serving its purpose greatly. Until I decided to add a simple WYSIWYG interface to allow me to add bold and italic text as well as links.

The biggest challenge I faced there was working text selection. The Selection is still experimental and besides that, I found it to be quite complicated. I wanted something easier to work with.

Enter Rangy. It makes working with user selection, a walk in the park. Unfortunately, it is outdated and hasn't been maintained for some time, but I still found it stable and simple. With some experimentation I was able to load it onto my app and I'm not exaggerating when I say: it saved lots of hours. Hats off to the author.

I'm using it to save and restore the user selection as well as to wrap or unwrap an HTML element around a specific selection, so to make it bold, italic or transform it to a link. 💥

Local file system

Saving presentations on the cloud you say? Well yeah but honestly, it wouldn't offer me that much at this point. These days, I'm working constantly from one single laptop so I wasn't worried about losing my work. But I had to save my work somewhere. Well, let's use that hard drive then.

File System Access API is here and while no major compatibility exists yet, this library will fallback to native implementation if the browser doesn't support it.

Now I'm able to save my work in a local file, store it on my cloud account and load it whenever I need it. 👍

Conclusion

There you go. Those were my biggest challenges creating a web app for creating presentations.

It was a great journey. Do use it and let me know if any feedback you may have. I'm looking forward to that. 🙏

📽 Presentador: Build a presentation by writing Markdown

Kostas Bariotis — Sun, 13 Dec 2020 17:00:13 +0000

More often than not, when I'm creating a presentation, I find myself being distracted on each slide by the looks of it and start editing and altering the fonts and colors and sizes.

All I want is to write a presentation and make my points. That's why I built Presentador.

📽 With Presentador, you write your presentation in Markdown files and it will produce a beautiful website. No code or configuration required.

It's easy! Let me show you how:

Write your presentation

Create a folder and start writing your presentation in Markdown files. Create one file for each slide and name them after the order you want them to take, e.g. 1.md, 2.md, etc.

Use regular Markdown. Presentador will parse each slide and assign a predefined layout to each of them based on the elements in them.

If you find a combination of elements that could be presented differently, please open an issue. 👍

Build your presentation

Run:

> npx presentador build -d FOLDER_NAME

and Presentador will produce a website ready to be hosted anywhere you want.

While developing your presentation, run npx presentador serve -d FOLDER_NAME for instant changes while you're editing it.

That's it!

Head over to https://www.presentador.dev to see it in action. 🙂

Let me know what you think, I really need your feedback to make Presentador better.

Also, Presentador is open source. Let's build it together. 🤝

Learning to code by doing essential little projects

Kostas Bariotis — Fri, 03 Jul 2020 21:21:25 +0000

Photo from Unsplash

When I was learning to code for the first time, I was always eager to find tutorials out on the wild to follow in order to better understand concepts that I was not familiar with.

I was selective though. Not just any tutorial would do. I wanted by the end of my sit to have something to show. Something that would make sense.

I remember learning PHP and my favorite tutorial would be a login page. I learned a lot by following lots of different tutorials that were about a simple form. I learned about authenticating users, correctly hashing and storing passwords, common security-related issues, and more just by doing a simple login page. I had learned about the authentication system.

It was fun!

But I can't find that any more. Tutorials nowadays are all about common libraries and technologies that even if you learn to use, you would still not have an understanding of the fundamentals. Just the fundamentals to use that specific library.

I want to fix this.

I want to make learning to code fun again. To make it about learning the fundamentals.

I want to build a mailing list. I will send you an exercise every week. A simple little project with a clear objective and of essential use. If you stuck or you have any questions, I will be right behind my inbox to assist you.

This is for beginners to start upskilling by doing useful projects that can actually show. Even for intermediates that are looking for motivation to sharpen their existing skills.

Let's make code fun again

Are you interested? Send me an email to kostas@bariotis.com. I will start sending out the first projects in a few weeks.

Thank you. Speak to you soon.

Web development exercises for beginners

Kostas Bariotis — Sat, 16 May 2020 15:57:01 +0000

What are some great web development exercises you did while you were learning how to code?

I remember when I was learning, my first language was PHP. At the time, JavaScript wasn't booming yet so we had to do lots of backend alongside frontend work. My two main tutorials were Login forms and Contact forms.

I specifically remember those because Login forms taught me lots about security, correctly hashing passwords, how to protect from SQL injection, not revealing too much information in your errors, and others.

Contact forms taught me about user experience, I learned a lot about HTML and CSS, styling buttons and inputs, etc.

What was/is your favorite exercise or tutorial?

Setting up a GraphQL server with TypeScript

Kostas Bariotis — Sat, 21 Mar 2020 00:00:00 +0000

This article will go through setting up the basic structure of a GraphQL server with TypeScript. I will use apollo's Koa server package that I found to be very straightforward and also will set up some basic smoke tests with mocha and supertest. Let's go!

GraphQL, by providing its own definition language, it's often hard to co-exist with our high-level language of choice. But things don't have to be complicated, so let's try to keep it simple and create a maintainable structure and keep separate concerns as possible.

I will assume that you have a TypeScript project set up already in your preferred way. If not, you will find a complete example by the end of this blog post. Bear with me till then.

Kick off

Let's install some required packages first:

> npm install --save apollo-server-koa koa koa-router

> npm install --save-dev chai supertest mocha

Great, now let's start with setting up koa and the apollo-server-koa package. We will start with our apps' entrypoint, say server.ts:

// src/server.ts

import Koa from "koa";
import KoaRouter from "koa-router";
import { ApolloServer, gql } from "apollo-server-koa";

We will first import our required modules.

async function main() {
  const app = createApp();
  const port = process.env.PORT || 3100;

  app.listen(port);

  console.log(`Listening on port ${port}`);
}

Then we define a main function that will create a Koa app and start listening. We will come back to it later.

// src/server.ts

// ...

export function createApp(): Koa {
  const app = new Koa();

  const router = new KoaRouter();

  const server = new ApolloServer({
    typeDefs: gql(`
      type RootQuery

      type RootMutation

      schema {
        query: RootQuery
        mutation: RootMutation
      }
    `),
    context: ({ ctx }) => ctx,
    formatError: errorHandler,
    resolvers: {
      RootQuery: {},
      RootMutation: {},
    }
  });

  router.get("/healthz", ctxt => {
    ctxt.body = "ok";
  });

  router.post("/graphql", server.getMiddleware());
  router.get("/graphql", server.getMiddleware());

  app.use(router.routes());
  app.use(router.allowedMethods());

  return app;
}

The createApp function, as mentioned before, will construct our Koa app, along with the GraphQL schema and server. At the moment, our GraphQL schema is empty and does pretty much nothing. But notice how we were able to specify a Koa route for our GraphQL endpoint and a /healtz endpoint for our load balancer. Let's move on.

// src/server.ts

// ...

const errorHandler = err => {
  console.log("Error while running resolver", {
    error: err
  });

  // Hide all internals by default
  return new Error("Internal server error");
};

Our error handler is very basic and as is, will always respond with a 500 and a mystic error message. We will change that later on.

// src/server.ts

// ...

if (require.main === module) {
  main();
}

The final piece is an instruction to only call main if the file was called from the CLI and wasn't required from another module. That's important for our tests and we will see it right now.

Let's write some tests right away.

// src/server.test.ts

import supertest from "supertest";

import { createApp } from "./server";

const request = supertest(createApp().listen());

describe("Server", () => {
  it("should respond for health check", async () => {
    await request
      .get("/healthz")
      .expect("Content-Type", /json/)
      .expect(200);
  });
});

Here we are using createApp to create and return an instance of our Koa app and pass it to supertest. That will give us the environment we need to make requests. We start off with a basic request to /healtz and expect to have a valid 200 OK response.

Now let's add our first GraphQL query. Let's go back to where we defined our Appolo server:

// server.ts

// ...

const server = new ApolloServer({
  typeDefs: gql(`
    schema {
      query: Query
    }

    type Query {
      hello(name: String): String
    }
  `),
  resolvers: {
    Query: {
      hello: function hello(
        root: {},
        args: { name: string },
        context: {}
      ): String {
        return `Hello ${args.name}`;
      }
    }
  },
  formatError: errorHandler,
});

// ...

Our GraphQL schema is pretty basic now, we added a hello query and an appropriate resolver that will do nothing other than just returning the passed argument. Let's write a test before we move on:

// src/server.test.ts

// ...

describe("GraphQL Server", () => {
  it("should query hello", async () => {
    await request
      .post("/graphql")
      .send({
        query: `
          {
            hello(name: "Kostas")
          }
        `
      })
      .expect("Content-Type", /json/)
      .expect(200)
      .then(response => {
        expect(response.body.data.hello).to.eq("Hello Kostas");
      });
  });
});

Amazing, a very basic test for our query.

Scaling up

Now let's go back to our server definition. Notice how fast it will grow as we start to add more queries and mutations. Let's break it up. Start by creating a folder src/schema. That's where we will store our GraphQL definitions. Then a folder src/queries where we will store our GraphQL resolvers. Let's move away from the hello query.

// src/schema/hello.graphql

extend type Query {
  hello(name: String): String
}

And our query:

// src/queries/hello.ts

export default {
  Query: {
    hello: function hello(
      root: {},
      args: { name: string },
      context: {}
    ): String {
      return `Hello ${args.name}`;
    }
  }
}

And finally the server:

// src/server.ts

import { readFileSync } from 'fs';
import { join as pathJoin } from 'path';

// ...

  const server = new ApolloServer({
    typeDefs: gql(
      readFileSync(pathJoin(__dirname, "schema/hello.graphql")).toString()
    ),
    resolvers: {
      Query: helloQuery.Query
    },
    formatError: errorHandler
  });

// ...

Amazing! Now run our tests again and they should pass.

Now to avoid having to add new lines every time we add a new query or mutation, let's do something slightly more sophisticated. Let's traverse our schema folder and build our server based on that.

// src/server.ts

// ...

// Fetch all schema definition files
const schemaFiles = readdirSync(pathJoin(__dirname, "schema"))
  .filter(file => file.indexOf(".graphql") > 0);

// Concatanate them to create our schema
const schema = schemaFiles
  .map(file => readFileSync(pathJoin(__dirname, `schema/${file}`)).toString())
  .join();

// Based on these files, bring their respective query resolvers
const queryResolvers = schemaFiles
  .map(file => file.replace(".graphql", ""))
  .map(file => require(pathJoin(__dirname, `queries/${file}`)).default)
  .reduce(
    (initial, current) => ({
      ...initial,
      ...current.Query
    }),
    {}
  );

const server = new ApolloServer({
  typeDefs: gql(`
    type Query

    schema {
      query: Query
    }

    ${schema}
  `),
  resolvers: {
    Query: queryResolvers
  },
  formatError: errorHandler
});

// ...

Run our tests again, they should pass again. Try to start your server as well.

Did you know that apollo's server comes with a development GraphQL client? Start your server and head to /graphql on your browser on your servers' host and port.

More types

So far, we've been doing great. Notice the hello query how it returns a String type. That's great, but with TypeScript and GraphQL, we are able to define more complex type structures. As a matter of fact, we are definitely going to need them. So let's change the query to return a more complicated object.

// src/queries/hello.ts

interface HelloResponse {
  name: String;
  greeting: String;
}

export default {
  Query: {
    hello: function hello(root: {}, args: { name: string }): HelloResponse {
      return {
        name: args.name,
        greeting: `Hello ${args.name}`
      };
    }
  }
};

// src/schema/hello.graphql

type HelloResponse {
  name: String
  greeting: String
}

extend type Query {
  hello(name: String): HelloResponse
}

Amazing, now let's change our test as well.

// src/schema/hello.ts

// ...

  it("should query hello", async () => {
    await request
      .post("/graphql")
      .send({
        query: `
          {
            hello(name: "Kostas") {
              name
              greeting
            }
          }
        `
      })
      .expect("Content-Type", /json/)
      .expect(200)
      .then(response => {
        expect(response.body.data.hello.name).to.eq("Kostas");
        expect(response.body.data.hello.greeting).to.eq("Hello Kostas");
      });
  });

// ...

And run our tests again. Hopefully, they should pass as nothing changed.

Conclusion

That was it. I hope that you have been following along and managed to keep your tests green along the way.

You can find a complete example of the above on GitHub. Give it a pass and see what differences you have with your own or simply clone it and play around with it.

Let me know in the comments what you thought of the article and remember to share it with your colleagues.

Gatsby with Docker multi-stage builds

Kostas Bariotis — Fri, 08 Nov 2019 00:00:00 +0000

I'm probably late to the game, but I have just discovered Docker's new (well..) feature, the multi-stage builds. At first, it came handy for building Go binaries, starting with a golang base image, compiling the project and then continuing with a scratch image to actually run the binary. Here's how it helped me build the containers for the Discover project. Superb!

But then I started thinking about other cases and suddenly it struct me! Frontend baby! In this article I will go through building a Dockerfile suitable for holding a Gatsby project. This Dockerfile will be able to serve a development environment with the help of docker-compose, but also creating a final image from nginx ready to go up on your kubernetes cluster (or wherever really).

So, let's get on.

The process

In a frontend project there are usually two distinct processes. The development and the build. Development will spin up a local server, probably with webpack, some file-watching daemon, etc. The build process will build everything up producing the final artifacts that will go on your server. create-react-app anyone?

The base in each of these processes is the same. Install Node, fetch npm packages, and so on.

Gatsby in particular, has two commands, gatsby develop and gatsby build.

The Dockerfile

Let's start with the base image. Here's a very common Dockerfile for building a Gatsby project.

FROM node:10 as node

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm ci

COPY . .

EXPOSE 8000

CMD ["gatsby", "build"]

Pretty basic.

Now let's add a docker-compose.yaml file to help us with local development. You may have one of these already probably serving a local API, so embedding it into your workflow will be easy peasy.

version: "3.4"

services:
  website:
    container_name: gatsby_website
    build:
      context: ./
      dockerfile: Dockerfile
    ports:
      - 8000:8000
    command: ./node_modules/.bin/gatsby develop -H 0.0.0.0
    volumes:
      - /usr/src/app/node_modules
      - .:/usr/src/app
    environment:
      - NODE_ENV=development

Notice how we are overriding the command so instead of running gatsby build inside the container, the gatsby develop process will kick in instead. Try it by running running docker-compose up. The local service should start and you will be able to make any changes and watch them go live.

The deployment

But now, we would like to actually build our website and put it inside an nginx container. That container will then be deployed in a kuberentes cluster. Let's do some modifications to our files above:

FROM node:10 as node

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm ci

COPY . .

CMD ["gatsby", "build"]

+ FROM nginx as server
+ 
+ EXPOSE 80
+
+ COPY --from=node /usr/src/app/public /usr/share/nginx/html

version: "3.4"

services:
  website:
    container_name: gatsby_website
    build:
      context: ./
      dockerfile: Dockerfile
+ target: node
    ports:
      - 8000:8000
    command: ./node_modules/.bin/gatsby develop -H 0.0.0.0
    volumes:
      - /usr/src/app/node_modules
      - .:/usr/src/app
    environment:
      - NODE_ENV=development

So now I've added a second stage to our Dockerfile that starts from nginx and also copies all the artifacts from the previous stage. docker-compose has also been accommodated to stop at the first stage so it will never reach the second one.

Let's build the image now with Docker:

> docker build -t gatsby-image .

That's it! Now our Dockerfile will produce an nginx container with our final website deployed in. docker-compose will continue to work as. Brilliant!

Conclusion

And there you go. A single Dockerfile to use for both development and production in conjunction with docker-compose. Life just became simpler.

I'm sure more use cases can come out of that. I would love to hear how are you using it! Hit me in the comments below.

Guide to package-lock.json

Kostas Bariotis — Tue, 22 Oct 2019 00:00:00 +0000

This article first appeared on LogRocket's blog.

In this article, we'll look at package-lock.json, why it's important, and how it's best used along with NPM CLI in your day-to-day life.

History

NPM version 5 introduced package-lock.json as a mechanism to capture the exact dependency tree installed at any point in time.

This helps with collaboration across different environments in which you want everyone fetching dependencies for a specific version of your project to fetch the same tree.

package.json defines the required dependencies and also their respective versions using semantic versioning. However, semantic versioning can be tricky.

Consider a dependency stated as "express": "^4.16.4".

The publisher of this module (without using package-lock.json) would have express version 4.16.4 installed since they installed the latest version.

If express has published a new version by the time I download this module and try to install dependencies on it, I can download the latest version.

The caret symbol tells us exactly that.

The problem with the above is that if version 4.17.x contains a bug, my local setup will fail, but the publisher's will continue to work fine on the previous version.

The same thing could happen in the production environment, and you'd have no idea why it was failing.

Prior to NPM version 5, you would use shrinkwrap. It differs from package-lock.json because it's allowed to be published with your module on the NPM registry, whereas package-lock.json is not.

If all members can use NPM+5, it's best to go with package-lock.json for unpublished projects.

But if you are developing a module and you intend to publish it, you might need to think about whether you want the clients to install the exact dependency tree you dictate, or if you want to be more flexible about it. Here's a more detailed version on the subject.

So, package-lock.json will describe the exact dependency tree currently installed. The format is described in NPM documentation page.

By committing it on your VCS--which you should absolutely do--you are able to go back in history and replicate the exact dependency tree from that time.

Make sure to always commit package-lock.json to your VCS to keep track of exact dependency trees at any given time.

It will ensure that all clients that download your project and attempt to install dependencies will get the exact same dependency tree. Furthermore, it'll ensure you're able to check out previous commits and replicate the dependencies state of each commit.

`package.json` vs `package-lock.json`

Make sure you don't change package-lock.json directly. That's being handled automatically by NPM. It reflects changes made to package.json to package-lock.json and keeps it up to date.

However, this only happens if you use NPMs' CLI to make changes. If you manually change package.json, don't expect package-lock.json to update. Always use the CLI commands, like install, uninstall, etc.

How to use the NPM CLI

NPM will auto generate a package-lock.json when you first use it in a fresh project.

Then, you can use NPM as normal.

npm install (with specific modules as arguments)

install can be used with the names of modules to install as arguments, which will alter both package.json and package-lock.json since the dependency tree will change.

Consider the following example:npm install express body-parser cors

npm install (without arguments)

install will attempt to install all dependencies in respect to package-lock.json.

A key point here is that install can alter package-lock.json if it registers that it's outdated.

For example, if someone manually alters package.json--say, for example, they remove a package since it's just a matter of removing a single line--the next time that someone runs npm install, it will alter package-lock.json to reflect the removal of the previous package.

That can be tricky. Imagine pulling the latest version of your project, running npm install to get up to date, only to find that you immediately have a bunch of changes in your tree that make no sense.

It's also highly likely that the changes in your tree would make no sense to the people reviewing your changes.

npm uninstall

Similar to install but with names of modules to remove as arguments. Will alter both package.json and package-lock.json.

npm update

update will read package.json to find any dependencies that can be updated. Subsequently, it will construct a new dependency tree and update the package-lock.json as well.

Remember semantic versioning? Say we have a dependency in our package.json stated as ^1.4.5.

The ^ character tells NPM to check if there's a newer version under the 1.X.X scope and if there is, to install that. Similarly, the ~ character will only go up to hot-fixes, or 1.4.X.

You could also omit the special character and keep a fixed version, which makes package-lock.json less helpful (but not useless).

npm ci

ci will install all dependencies in respect to package-lock.json similar to install. The key difference here is that it won't alter package-lock.json under any circumstances.

Its purpose is to be used by environments, e.g. build servers, where installation happens in an automated way.

Conclusion

Remember these key takeaways when using 'package-lock.json':

Don't use npm install without arguments to fetch dependencies--use npm ci for that. You can use the npm install to install specific dependencies.

Use npm ci everywhere when you only want the local dependencies tree--even on your local develpment environment.

Make npm update a repetitive task, say once a month, to update your dependencies. (or use a service like dependabot, but make sure that you have a good test coverage).

This way you ensure that you keep your dependencies up to date and you avoid bubbling up technical debt.

More references :

Differences between npm install and npm ci: https://stackoverflow.com/questions/52499617/what-is-the-difference-between-npm-install-and-npm-ci
NPM CLI source code: https://github.com/npm/cli/blob/latest/lib/install.js
Semantic versioning: https://blog.npmjs.org/post/162134793605/why-use-semver
Why does “npm install” rewrite package-lock.json? https://stackoverflow.com/questions/45022048/why-does-npm-install-rewrite-package-lock-json

Dependency injection in Node.js

Kostas Bariotis — Sat, 20 Jul 2019 00:00:00 +0000

This article is about how to achieve dependency injection in Node.js using awilix, a Dependency Injection Container I have been personally using for quite some time now. I will use unit testing as a use case where dependency injection can be helpful, but needless to say, its purpose is far more greater than just that.

I've seen different patterns over the years and this is a summary of what led me to use dependency injection. Node.js, and JavaScript, being a dynamic language, can forgive different techniques. My goal is to achieve a standard technique to use across all of my projects and teams I am working on.

The case of unit testing

We strive to write more unit tests for our functions. Though they can be tough especially in situations where a function is having side effects, that is when it's communicating with the outer world directly and not from its standard input or output. Let's have a look:

const client = require('axios');

const externalServiceRoot = 'https://api.example-external-service.com';

async function callExternalService(anArgument) {
  const { response } = await client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

  if (!response.success) {
    throw new Error('Response doesn\'t look good');
  }

  return response.data;
}

module.exports = { callExternalService };

Here's a simple module that you intent to use in your application. Probably multiple times.

It can be tested using Nock, a network interceptor, that will sit in the network card and assert certain connections going through. I am assuming that though to not be a unit test, but rather an integration test since the network request will actually reach the network card (or Node.js' http module). Also, it will make the test much slower since will go all the way through axios.

How can we unit test that? We mock the axios object with a one that actually does a fake request. This way the test will only test the code inside the function. Will be much faster and will actually do only what is intended to do, that is to only test the code inside our function.

How can we do that though in Node.js?

Monkey patch

I have seen this over the years especially in the early days of Node.js, the days of prototype inheritance. The axios module would be attached to the exported object so it could be seen from the outside world. Using the _ convention we could denote that is private. But tests would be able to patch it and pass a stub.

Here's an example:

const axios = require('axios');

function ExternalServiceConstructor() {
  this._client = axios;
}

ExternalServiceConstructor.prototype.callExternalService(anArgument) {
  const { response } = await this._client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

  if (!response.success) {
    throw new Error('Response doesn\'t look good');
  }

  return response.data;
}

module.exports = new ExternalServiceConstructor();

The module is now exporting a singleton and you will be able to access _client thus able to stub it. (Nowadays, same effect can be achieved using the Class keyword or Object.create for less boilerplate code)

I don't like this approach for two reasons.

First, since the exported module is a singleton, it means that if a test stubs the _client property, it will remain stubbed across all other places that have been required. You should be really careful to reset the stubbed version so other test files can function properly.

Second, it exposes public properties, which makes it dangerous for consumers. People will try to be smart and alter it.

Dependency injection

Dependency injection is not a new concept and quite well known in other languages where monkey patching is not really feasible. In that, you are responsible to pass any dependencies of the module in runtime. Here's our first example with the axios module injected in our functions' signature:

const externalServiceRoot = 'https://api.example-external-service.com';

async function callExternalService(client, anArgument) {
  const { response } = await client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

  if (!response.success) {
    throw new Error('Response doesn\'t look good');
  }

  return response.data;
}

module.exports = { callExternalService };

Now the consumer is responsible to pass the module whenever he's calling the callExternalService function. That can be painful though and not be providing the best developer experience. First, the functions signature is bloated with mixed arguments and second you may have to do that all the time which creates redundancy and duplication.

Here are some other patterns I've seen and used my self.

Function factory

Use currying from functional programming to create a factory that will produce our final function.

const externalServiceRoot = 'https://api.example-external-service.com';

function makeCallExternalService(client) {
  return async function callExternalService(anArgument) {
    const { response } = await client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

    if (!response.success) {
      throw new Error('Response doesn\'t look good');
    }

    return response.data;
  }
}

module.exports = { makeCallExternalService };

Call the makeCallExternalService with the client and you have your function.

Dependencies always as the first argument

Have a convention to always pass all dependencies as the first argument of the function.

const externalServiceRoot = 'https://api.example-external-service.com';

async function callExternalService(dependencies, anArgument) {
  const { response } = await dependencies.client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

  if (!response.success) {
    throw new Error('Response doesn\'t look good');
  }

  return response.data;
}

module.exports = { callExternalService };

Dependency injection container

A dependency injection container is kind of what you think. It contains all of your dependencies. It's responsibility is to construct all of your classes or modules thus abstracting that bit away from your business logic.

In addition, it handles the wiring of your modules as well. Instead, of directly requiring our externalService module, we are asking it from the container and it will make sure to return the needed function ready with its dependencies.

In Node.js, I have only used one such container and really never looked back. It's the awilix container.

awilix is really powerful and once you get on track with it, it can really abstract most of the dependency injection hassling away from you.

We will see an example shortly, but before that, let's add another minor requirement to our externalService module. Let's require that we want the externalServiceRoot variable to be injected as well, because we want to hit a different endpoint based on the environment our app runs. It's different for production/staging and the local development. Dependency injection cannot only be used for stubbing dependencies.

Here's the final function using plain dependency injection.

async function callExternalService(client, externalServiceRoot, anArgument) {
  const { response } = await client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

  if (!response.success) {
    throw new Error('Response doesn\'t look good');
  }

  return response.data;
}

module.exports = { callExternalService };

Right. Now the signature really starts to become bloated. Let's start with awilix. We will start by feeding configuration values that will come from environment variables.

const { createContainer, asValue } = require('awilix');

const container = createContainer();

container.register({
  externalServiceRoot: asValue(process.env.EXTERNAL_SERVICE_ROOT)
});

Great, now our container is aware of the configuration values. asValue simply instructs awilix to pass that value as is, whenever someone requires the externalServiceRoot variable.

What I personally like to do though is adding an extra guard here to not allow our app to continue without that value. You could use a configuration manager to handle that for you, but for now let's do some changes.

const { createContainer, asFunction } = require('awilix');

const container = createContainer();

container.register({
  externalServiceRoot: asFunction(() => {
    if (!process.env.EXTERNAL_SERVICE_ROOT) {
      throw new Error('EXTERNAL_SERVICE_ROOT is not defined.')
    }

    return process.env.EXTERNAL_SERVICE_ROOT;
  }
});

Now asFunction is a little different, as it will actually run that function whenever someone requires that variable.

Great. That's our first dependency of the callExternalService function. Now we need to pass the axios module. In an ideal world, you would actually pass a module that follows a specific interface, otherwise you are coupling your code to axios and making it harder later to switch to another HTTP client library. For now, let's suppose that axios follows our standard interface.

const { createContainer, asFunction, asValue } = require('awilix');
const axios = require('axios');

const container = createContainer();

container.register({
  client: asValue(axios)
});

container.register({
  externalServiceRoot: asFunction(() => {
    if (!process.env.EXTERNAL_SERVICE_ROOT) {
      throw new Error('EXTERNAL_SERVICE_ROOT is not defined.')
    }

    return process.env.EXTERNAL_SERVICE_ROOT;
  }
});

Now our container is aware of our HTTP client as well. It's time to put everything together. We will use the function factory from above.

const { createContainer, asFunction, asValue } = require('awilix');
const axios = require('axios');

const container = createContainer();

// The container will be passed to this function with
// everything is contained. awilix is smart enough to
// understand what exactly you are requiring.
function makeCallExternalService({ client, externalServiceRoot }) {
  return async function callExternalService(anArgument) {
    const { response } = await client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

    if (!response.success) {
      throw new Error('Response doesn\'t look good');
    }

    return response.data;
  }
}

container.register({
  callExternalService: asFunction(makeCallExternalService)
})

container.register({
  client: asValue(axios)
});

container.register({
  externalServiceRoot: asFunction(() => {
    if (!process.env.EXTERNAL_SERVICE_ROOT) {
      throw new Error('EXTERNAL_SERVICE_ROOT is not defined.')
    }

    return process.env.EXTERNAL_SERVICE_ROOT;
  }
});

module.exports = container;

That's nice. We created the factory that takes our container as an input, thus we can request any value we passed to it so far and produces the function we want injected with these values.

When we pass the factory makeCallExternalService to awilix, it knows to run the function and pass its return value to whoever is requiring it. The return value will be our final function with all of its dependencies.

Let's try and use that function now.

// require the container from above
const container = require('./container');

// Will fail if the environment doesn't contain
// the EXTERNAL_SERVICE_ROOT variable
const callExternalService = container.resolve('callExternalService');

// Call the function that was produced from the
// makeCallExternalService function factory
callExternalService('argument')

Amazing. Everything is wired and being handled by the container it self. We can use this function across our application and we can also do that for integration testing possibly with nock. But what's also amazing, is that our function is now able to be tested against stubs. Let's do that.

// -- src/callExternalService.js

// We have moved our function to another file

function makeCallExternalService({ client, externalServiceRoot }) {
  return async function callExternalService(anArgument) {
    const { response } = await client.post(`${externalServiceRoot}/an/endpoint`, anArgument);

    if (!response.success) {
      throw new Error('Response doesn\'t look good');
    }

    return response.data;
  }
}

module.exports = { makeCallExternalService };

// -- callExternalService.test.js

// This is our unit test

// This time we require our function directly,
// not through the container
const { makeCallExternalService } = require('./callExternalService')

// HTTP Client stub
const client = {
  put: Promise.resolve({ response: { data: { success: false } } });
}

// Produce the function under test
// by passing our stubs
const callExternalService = makeCallExternalService({
  externalServiceRoot: 'FAKE_ROOT',
  client,
})

describe('callExternalService', () => {
  context('with false success response', () => {
    it('should throw', () => {
      expect(() => callExternalService('argument')).to.throw('Error');
    })
  })
})

There you have it. Unit testing our function with stubs. We have our container loaded with our dependencies and modules that we can use across our app. We don't have to worry about wiring our modules together. Every time we write a new module, we know exactly where to find any dependency, how to request them, and all we focus on is the task on hand.

Conclusion

I advice you to take a look on its repository and go through its documentation and examples. Also, I have found this API boilerplate that uses awilix, and also makes uses of many best practices. It's a great read even if you don't intent to use it as is.

The above was an oversimplified example of how dependency injection with awilix can be achieved on Node.js. In practice, as the project grows, other complications arise, like the container being bloated it self. awilix is an amazing library though and has most of this issues solved already.

I would be curious to know how do you handle dependency injection in your projects. Leave a comment below.

This is a cross-post from my blog. Find the original article here.

NODE_ENV: So easy to miss your step

Kostas Bariotis — Sat, 19 Jan 2019 00:00:00 +0000

I broke our main API last week. I merged a PR of mine I had been working for a while. It got two approvals from two co-workers, green test suite and it was working perfectly fine locally and on our staging environment, the one that we test things before we send them to production. Somehow, the Node.js server failed to boot on production and we had to rollback to the last deployment, resulting into a 2-3 minutes downtime.

Well there were so much to learn from that adventure, but the one that we are going to focus on this post is, why did the Node.js server failed to boot on production while was working perfectly fine locally and on our pre-production staging environment? Keep reading.

What happened?

Wanna see the PR that brought our production down? Here it is:

I literally moved a dependency from the dependencies section to devDependencies. Reinstall my dependencies locally and rerun the server and was working perfectly. I then deployed to our staging and again, worked perfectly fine. But, failed to boot on production.

Specifically, the production server failed to boot cause it wasn't able to find the chance module.

Yes! That's obscure! Well not so much. Let's take it from the beginning.

The Node.js server was mostly running in three different environments. It was using the NODE_ENV to denote the current environment. It could take three different values based on where it was running, development for when it was running locally, staging for when it was running on our staging environment and production for when it was running on our production environment.

Now here's an interesting piece from npm docs:

With the --production flag (or when the NODE_ENV environment variable is set to production), npm will not install modules listed in devDependencies.

Yes! The NODE_ENV will actually impact your production environment and will enlarge the gap between production and other environments. Let's revisit a famous quote from the The twelve-factor app:

Keep development, staging, and production as similar as possible.

Unfortunately, by setting our NODE_ENV to a value based on the current environment it was running on, we were actually making our environments parity bigger.

The immediate action we took after that incident was simple. To decouple our application environment from NODE_ENV. We introduced the APP_ENV variable, renamed all NODE_ENV occurrences to that and then moved NODE_ENV to only have values development when running locally or while running unit tests and production for all other environments.

Another aspect of this is that some other library may be using this variable as well without even knowing about it. And it won't be looking for multiple values, it will only be looking for production vs anything else.

Now where did NODE_ENV came from and why does npm actually uses it? Node.js documentation mention nothing about such a variable. Well the NODE_ENV variable became famous from the Express.js framework where it was using it to decide whether it should enable some development features on production. After people started to use it, other projects started to adopt it as well and we reached today.

To conclude

Like I mention above, we learned quite a few things from that incident, but our immediate action was to rename NODE_ENV in an attempt to keep to our online environments as similar as possible. 🤓

Were there any actions or decisions you took recently to address this issue?

Aggregated log management

Kostas Bariotis — Thu, 27 Dec 2018 00:00:00 +0000

I have been building aggregated logging mechanisms in all my projects and I always thought of it as an industry standard practice. Lately I've noticed that this is not the case and lots of people are not taking it seriously.

This post is an attempt to gather my thoughts on the subject and hopefully help others understand it. Here it comes.

Why logging

OS processes that usually run web applications are always bound to the standard output. Can be a file, the terminal or a GUI made by the hosting provider. Whatever that is, I usually tend to throw a bunch of logging statements to my functions to be able to trace their execution trail later.

It's also useful to pack those traces with metadata. A request for password reset was initiated by a specific user or the creation of a specific TODO item failed yesterday. Hopefully, I will be able to trace back those requests and see more information.

Say for the password reset request we will end up having something like this:

82c838eb-736a-447b-b9e1-10dfe2bbe575 - Tue, 04 Dec 2018 20:27:16 GMT - POST /user/resetPassword - Initiated
82c838eb-736a-447b-b9e1-10dfe2bbe575 - Tue, 04 Dec 2018 20:27:16 GMT - User 145123 requested to reset password
82c838eb-736a-447b-b9e1-10dfe2bbe575 - Tue, 04 Dec 2018 20:27:16 GMT - User found, resetting password
82c838eb-736a-447b-b9e1-10dfe2bbe575 - Tue, 04 Dec 2018 20:27:16 GMT - Password reset
82c838eb-736a-447b-b9e1-10dfe2bbe575 - Tue, 04 Dec 2018 20:27:16 GMT - Attempt to send email
82c838eb-736a-447b-b9e1-10dfe2bbe575 - Tue, 04 Dec 2018 20:27:16 GMT - Email sent
82c838eb-736a-447b-b9e1-10dfe2bbe575 - Tue, 04 Dec 2018 20:27:16 GMT - POST /user/resetPassword - 200 OK

The above is a simplified example but you can see the detail granularity that we have already. We could easily spot an exception thrown in the middle of the process, find out its the exact time and reason about our system's behavior.

But the above will quickly result to lots of lots of logs especially in a high traffic environment. Fortunately, there are the tools that will help us find the information we need much easier.

Why aggregated

By aggregating those logs, we are implying that there are more than one processes or even completely independent systems that will generate logs. Indeed, even the smallest web application will be consisted of various parts like several instances of our app that exposes a REST API, the database and possibly a load balancer. Instead of having different places to check for your logs, it would be easier to aggregate all these in the same place and be able to traverse them all together.

Even better, we could trace the execution across our different systems, see any type of log or error that may arise during the request. Our code could be all valid but we may have forget to add an index in our DB.

The ELK model

As with every brilliant idea in this industry, ELK strives for separation of concerns. ELK stands for E lasticSearch, L ogstash and K ibana. It combines those three tools in order to put a great log aggregation system in place.

ElasticSearch uses Lucene under the hood. It will store and index our logs for us to be able to search in them. Logstash (now Filebeat) is responsible for wiring the standard outputs of our systems with ElasticSearch and finally Kibana is the GUI that will help us traverse our logs and create charts and visualizations.

Self hosted

ELK stack is open source and can be self hosted by you. As with all self hosted software, though it means that you may have to monitor the ES cluster and provide regular updates to Kibana.

AWS provides a managed service to create ES cluster of which they come with Kibana installed as a plugin. You could also self-host Kibana your self in an ECS task to have more freedom over it.

3rd-party solutions

papertrail is a great log management tool to centralize your logs in a single place. In contrast with ELK, papertrail is only a simple terminal where you can search for text occurrences. It doesn't provide the analytics and visualizations features of ELK.

Loggly is similar to ELK and provides all of it features plus a few more, like alarms. Alarms can be handy when you need to be notified e.g. when a certain occurrence comes up or when the number of a computed field exceeds a certain threshold.

Cloudwatch is AWS's native solution for managing logs in their cloud. If you're using EC2 for example you will be able to find each instance's logs in Cloudwatch. From there you can search in them, create triggers on specific text occurrences or even redirect them to other services. You could stream those logs (using Lambda functions) to an external ELK based stack such as Loggly or AWS ElasticSearch cluster.

Both papertrail and Loggly are now owned by the same company, but that was not the case when I was first introduced to them. I am not affiliated by this company in any way.

The log standard

For the indexing service to be able to better parse and analyze your logs, you must be able to provide a standard log format. There is no standard log standard that everyone uses but rather it seems like it is different for every case.

Node.js' bunyan logger that is extensively being used provides a good starting point.

{
  "name": "myserver",
  "hostname": "banana.local",
  "pid": 123,
  "req": {
    "method": "GET",
    "url": "/path?q=1#anchor",
    "headers": {
      "x-hi": "Mom",
      "connection": "close"
    },
    "remoteAddress": "120.0.0.1",
    "remotePort": 51244
  },
  "level": 3,
  "msg": "start request",
  "time": "2012-02-03T19:02:57.534Z",
  "v": 0
}

The above will be parsed and indexed by ES so you will be able to search and aggregate them.

Imagine that you wake up in the morning and you read support request from a customer who tried to use your service while you were sleeping but couldn't. What could have gone wrong? You will be able to answer it by searching those logs, see the path that the user took before crashing and eventually find the error along with the stack trace.

But this not the only valuable information we can get out of this.

Analytics

By having these information stored, other interesting answers are coming out:

What's the most used API path?
Do we get any suspicious headers?
Add the HTTP status code in the mix: How many 500 are we getting?
Add the username in the mix: Audit a particular's user activity

Be careful with what user identifying information you are storing. 2018 was the year that GDPR came in practice. By storing (or even worse, providing information to 3rd-party services such as Loggly) user identifiable information you are in violation of the GDPR law.

See how you are able to answer basic questions (or even more complex ones) for your HTTP traffic, systems execution or even business specific operations.

Conclusion

I consider log management to be a foundational piece for every software architecture. I would always design it and have it available as early as possible.

As the architecture grows, log management can be a tedious task. Here are some insights on StackOverflow's monitoring process. But till then, small to medium size architecture should have a way to simply:

Trace back in time execution of individual requests
Analyse and answer basic questions about your architecture

Engineering KPIs

Kostas Bariotis — Mon, 27 Aug 2018 19:47:49 +0000

What sort of KPIs do you use on your engineering processes and infrastructure? Sprint Velocity, No of hotfixes, API response times, client bundle file size, etc, etc..

DEV Community: Kostas Bariotis

Why you should not use process.exit()

What is process.exit()?

How it works

The dangerous nature of process.exit()

What should we do

How I made a web app to create presentations

Presentation mode

User text selection

Local file system

Conclusion

📽 Presentador: Build a presentation by writing Markdown

Write your presentation

Build your presentation

Learning to code by doing essential little projects

Web development exercises for beginners

Setting up a GraphQL server with TypeScript

Kick off

Scaling up

More types

Conclusion

Gatsby with Docker multi-stage builds

The process

The Dockerfile

The deployment

Conclusion

Guide to package-lock.json

History

package.json vs package-lock.json

How to use the NPM CLI

npm install (with specific modules as arguments)

npm install (without arguments)

npm uninstall

npm update

npm ci

Conclusion

Dependency injection in Node.js

The case of unit testing

Monkey patch

Dependency injection

Function factory

Dependencies always as the first argument

Dependency injection container

Conclusion

NODE_ENV: So easy to miss your step

What happened?

To conclude

Aggregated log management

Why logging

Why aggregated

The ELK model

Self hosted

3rd-party solutions

The log standard

Analytics

Conclusion

Engineering KPIs

What is `process.exit()`?

The dangerous nature of `process.exit()`

`package.json` vs `package-lock.json`