DEV Community: RisingStack 👉 Node.js, K8S & Microservices

Node.js Async Best Practices & Avoiding the Callback Hell

RisingStack 👉 Node.js, K8S & Microservices — Mon, 30 Mar 2020 15:27:54 +0000

In this post, we cover what tools and techniques you have at your disposal when handling Node.js asynchronous operations: async.js, promises, and async functions.

After reading this article, you’ll know how to use the latest async tools at your disposal provided by Node.js!

Node.js at Scale is a collection of articles focusing on the needs of companies with bigger Node.js installations and advanced Node developers. Chapters:

Using npm:

Node.js Internals Deep Dive:

Building with Node.js

Advanced Node.js Project Structuring
JavaScript Clean Coding Best Practices
Node.js Async Best Practices (this article)
Event sourcing with Examples
CQRS (Command Query Responsibility Segregation) Explained

Testing + Node:

Node.js in Production:

Node.js + Microservices:

Asynchronous programming in Node.js

Previously we have gathered a strong knowledge about asynchronous programming in JavaScript and understood how the Node.js event loop works.

If you have not read these articles, I highly recommend them as introductions!

The Problem with Node.js Async

Node.js itself is single-threaded, but some tasks can run in parallel thanks to its asynchronous nature.

But what does running in parallel mean in practice?

Since we program a single-threaded VM, it is essential that we do not block execution by waiting for I/O, but handle operations concurrently with the help of Node.js's event-driven APIs.

Let’s take a look at some fundamental patterns, and learn how we can write resource-efficient, non-blocking code, with the built-in solutions of Node.js.

The Classical Approach - Callbacks

Let's take a look at these simple async operations. They do nothing special, just fire a timer and call a function once the timer finished.

function fastFunction (done) {
  setTimeout(function () {
    done()
  }, 100)
}

function slowFunction (done) {
  setTimeout(function () {
    done()
  }, 300)
}

Seems easy, right?

Our higher-order functions can be executed sequentially or in parallel with the basic "pattern" by nesting callbacks - but using this method can lead to an untameable callback-hell.

function runSequentially (callback) {
  fastFunction((err, data) => {
    if (err) return callback(err)
    console.log(data)   // results of a

    slowFunction((err, data) => {
      if (err) return callback(err)
      console.log(data) // results of b

      // here you can continue running more tasks
    })
  })
}

<a href="https://twitter.com/share?text=Never%20use%20the%20nested%20callback%20approach%20for%20handling%20asynchronous%20%23nodejs%20operations!%22%20via%20%40RisingStack;url=https://blog.risingstack.com/node-js-async-best-practices-avoiding-callback-hell-node-js-at-scale/">
    Never use the nested callback approach for handling asynchronous #nodejs operations!" via @RisingStack
</a>

Avoiding Callback Hell with Control Flow Managers

To become an efficient Node.js developer, you have to avoid the constantly growing indentation level, produce clean and readable code and be able to handle complex flows.

Let me show you some of the tools we can use to organize our code in a nice and maintainable way!

#1: Using Promises

There have been native promises in javascript since 2014, receiving an important boost in performance in Node.js 8. We will make use of them in our functions to make them non-blocking - without the traditional callbacks. The following example will call the modified version of both our previous functions in such a manner:

function fastFunction () {
  return new Promise((resolve, reject) => {
    setTimeout(function () {
      console.log('Fast function done')
      resolve()
    }, 100)
  })
}

function slowFunction () {
  return new Promise((resolve, reject) => {
    setTimeout(function () {
      console.log('Slow function done')
      resolve()
    }, 300)
  })
}

function asyncRunner () {
    return Promise.all([slowFunction(), fastFunction()])
}

Please note that Promise.all will fail as soon as any of the promises inside it fails.

The previous functions have been modified slightly to return promises. Our new function, asyncRunner, will also return a promise, that will resolve when all the contained functions resolve, and this also means that wherever we call our asyncRunner, we'll be able to use the .then and .catch methods to deal with the possible outcomes:

asyncRunner()
  .then(([ slowResult, fastResult ]) => {
    console.log('All operations resolved successfully')
  })
  .catch((error) => {
    console.error('There has been an error:', error)
  })

Since node@12.9.0, there is a method called promise.allSettled, that we can use to get the result of all the passed in promises regardless of rejections. Much like Promise.all, this function expects an array of promises, and returns an array of objects that has a status of “fulfilled” or “rejected”, and either the resolved value or the error that occurred.

function failingFunction() {
  return new Promise((resolve, reject) => {
    reject(new Error('This operation will surely fail!'))
  })
}

function asyncMixedRunner () {
    return Promise.allSettled([slowFunction(), failingFunction()])
}

asyncMixedRunner()
    .then(([slowResult, failedResult]) => {
        console.log(slowResult, failedResult)
    })

In previous node versions, where .allSettled is not available, we can implement our own version in just a few lines:

function homebrewAllSettled(promises) {
  return Promise.all(promises.map((promise) => {
    return promise
      .then((value) => {
        return { status: 'fulfilled', value }
      })
      .catch((error) => {
        return { status: 'rejected', error }
      })
  }))
}

Serial task execution

To make sure your tasks run in a specific order - maybe successive functions need the return value of previous ones, or depend on the run of previous functions less directly - which is basically the same as _.flow for functions that return a Promise. As long as it's missing from everyone's favorite utility library, you can easily create a chain from an array of your async functions:

 function serial(asyncFunctions) {
     return asyncFunctions.reduce(function(functionChain, nextFunction) {
         return functionChain.then(
             (previousResult) => nextFunction(previousResult)
         );
     }, Promise.resolve());
 }

serial([parameterValidation, dbQuery, serviceCall ])
    .then((result) => console.log(`Operation result: ${result}`))
    .catch((error) => console.log(`There has been an error: ${error}`))

In case of a failure, this will skip all the remaining promises, and go straight to the error handling branch. You can tweak it some more in case you need the result of all of the promises regardless if they resolved or rejected.

 function serial(asyncFunctions) {
     return asyncFunctions.map(function(functionChain, nextFunction) {
         return functionChain
             .then((previousResult) => nextFunction(previousResult))
             .then(result => ({ status: 'fulfilled', result }))
             .catch(error => ({ status: 'rejected', error }));
     }, Promise.resolve());
 }

Converting callback functions to promises

Node also provides a handy utility function called "promisify", that you can use to convert any old function expecting a callback that you just have to use into one that returns a promise. All you need to do is import it in your project:

const promisify = require('util').promisify;
function slowCallbackFunction (done) {
  setTimeout(function () {
    done()
  }, 300)
}
const slowPromise = promisify(slowCallbackFunction);

slowPromise()
  .then(() => {
    console.log('Slow function resolved')
  })
  .catch((error) => {
    console.error('There has been an error:', error)
  })

It's actually not that hard to implement a promisify function of our own, to learn more about how it works. We can even handle additional arguments that our wrapped functions might need!

function homebrewPromisify(originalFunction, originalArgs = []) {
  return new Promise((resolve, reject) => {
    originalFunction(...originalArgs, (error, result) => {
      if (error) return reject(error)
      return resolve(result)
    })
  })
}

We just wrap the original callback-based function in a promise, and then reject or resolve based on the result of the operation.

Easy as that!

For better support of callback based code - legacy code, ~50% of the npm modules - Node also includes a callbackify function, essentially the opposite of promisify, which takes an async function that returns a promise, and returns a function that expects a callback as its single argument.

const callbackify = require('util').callbackify
const callbackSlow = callbackify(slowFunction)

callbackSlow((error, result) => {
  if (error) return console.log('Callback function received an error')
  return console.log('Callback resolved without errors')
})

#2: Meet Async - aka how to write async code in 2020

We can use another javascript feature since node@7.6 to achieve the same thing: the async and await keywords. They allow you to structure your code in a way that is almost synchronous looking, saving us the .then chaining as well as callbacks:

const promisify = require('util').promisify;

async function asyncRunner () {
    try {
      const slowResult = await promisify(slowFunction)()
      const fastResult = await promisify(fastFunction)()
      console.log('all done')
      return [
        slowResult,
        fastResult
      ]
    } catch (error) {
      console.error(error)
    }
}

This is the same async runner we've created before, but it does not require us to wrap our code in .then calls to gain access to the results. For handling errors, we have the option to use try & catch blocks, as presented above, or use the same .catch calls that we've seen previously with promises. This is possible because async-await is an abstraction on top of promises - async functions always return a promise, even if you don't explicitly declare them to do so.

The await keyword can only be used inside functions that have the async tag. This also means that we cannot currently utilize it in the global scope.

Since Node 10, we also have access to the promise.finally method, which allows us to run code regardless of whether the promise resolve or rejected. It can be used to run tasks that we had to call in both the .then and .catch paths previously, saving us some code duplication.

Using all of this in Practice

As we have just learned several tools and tricks to handle async, it is time to do some practice with fundamental control flows to make our code more efficient and clean.

Let’s take an example and write a route handler for our web app, where the request can be resolved after 3 steps: validateParams, dbQuery and serviceCall.

If you'd like to write them without any helper, you'd most probably end up with something like this. Not so nice, right?

// validateParams, dbQuery, serviceCall are higher-order functions
// DONT
function handler (done) {
  validateParams((err) => {
    if (err) return done(err)
    dbQuery((err, dbResults) => {
      if (err) return done(err)
      serviceCall((err, serviceResults) => {
        done(err, { dbResults, serviceResults })
      })
    })
  })
}

Instead of the callback-hell, we can use promises to refactor our code, as we have already learned:

// validateParams, dbQuery, serviceCall are higher-order functions
function handler () {
  return validateParams()
    .then(dbQuery)
    .then(serviceCall)
    .then((result) => {
      console.log(result)
      return result
    })
    .catch(console.log.bind(console))
}

Let's take it a step further! Rewrite it to use the async and await keywords:

// validateParams, dbQuery, serviceCall are thunks
async function handler () {
  try {
    await validateParams()
    const dbResults = await dbQuery()
    const serviceResults = await serviceCall()
    return { dbResults, serviceResults }
  } catch (error) {
    console.log(error)
  }
}

It feels like a "synchronous" code but still doing async operations one after each other.

Essentially, a new callback is injected into the functions, and this is how async knows when a function is finished.

Takeaway rules for Node.js & Async

Fortunately, Node.js eliminates the complexities of writing thread-safe code. You just have to stick to these rules to keep things smooth:

As a rule of thumb, prefer async, because using a non-blocking approach gives superior performance over the synchronous scenario, and the async - await keywords gives you more flexibility in structuring your code. Luckily, most libraries now have promise based APIs, so compatibility is rarely an issue, and can be solved with util.promisify should the need arise.

If you have any questions or suggestions for the article, please let me know in the comments!

In case you're looking for help with Node.js consulting or development, feel free to reach out to us! Our team of experienced engineers is ready to speed up your development process, or educate your team on JavaScript, Node, React, Microservices and Kubernetes.

In the next part of the Node.js at Scale series, we take a look at Event Sourcing with Examples.

This article was originally written by Tamas Hodi, and was released on 2017, January 17. The revised second edition was authored by Janos Kubisch and Tamas Hodi and it was released on 2020 February 10.

Generating a Static Site with Hugo + Netlify in 15 minutes

RisingStack 👉 Node.js, K8S & Microservices — Fri, 21 Feb 2020 14:10:55 +0000

In this article, I’m going to show how you can quickly generate a static site with Hugo and Netlify in an easy way.

What are static site generators, and why do you need one?

Simply put, a static site generator takes your content, applies it to a template, and generates an HTML based static site. It’s excellent for blogs and landing pages.

Benefits:

Quick deployment
Secure (no dynamic content)
Fast load times
Simple usage
Version control

So, what are the popular options in terms of static site generators?

Gatsby (React/JS)
Hugo (Go)
Next.js (React/JS)
Jekyll (Ruby)
Gridsome (Vue/JS)

These are the most starred projects on GitHub. I've read about Hugo previously, and it seemed fun to try out, so I’m going to stick with Hugo.

What is Hugo?

The official website states that Hugo is the world's fastest static website engine.

We can confirm that it’s really fast. Hugo is written in Golang. It also comes with a rich theming system and aims to make building websites fun again.

Let’s see what we got here.

Installing Hugo

Mac:

brew install hugo

Linux:

sudo apt-get install hugo

or

sudo pacman -Syu hugo

To verify your install:

hugo version

Using Hugo

Create a new project:

hugo new site my-project

Add a theme for a quick start. You can find themes here.

cd my-project
git init
git submodule add https://github.com/budparr/gohugo-theme-ananke.git themes/ananke

Add the theme to the config file.

echo 'theme = "ananke"' >> config.toml

Add some content.

hugo new posts/my-first-post.md

It should look something like this:

---
title: "My First Post"
date: 2020-01-05T18:37:11+01:00
draft: true
---

Hello World!

There are lots of options (tags, description, categories, author) you can write to the front matter details.

You can read more about the details here.

Take a look at what we made:

hugo server -D

Open http://localhost:1313

Understanding Hugo’s directory structure

.
├── archetypes
├── assets (not created by default)
├── config.toml
├── content
├── data
├── layouts
├── static
└── themes

archetypes: Archetypes are content template files that contain preconfigured front matter (date, title, draft). You can create new archetypes with custom preconfigured front matter fields.
assets: Assets folder stores all the files, which are processed by Hugo Pipes. (e.g. CSS/Sass files) This directory is not created by default.
config.toml: Hugo uses the config.toml, config.yaml, or config.json (if found in the site root) as the default site config file. Instead of a single config file, you can use a config directory as well to separate different environments..
content: This is where all the content files live. Top level folders count as content sections. If you have devops and nodejs sections, then you will have content/devops/first-post.md and content/nodejs/second-post.md directories.
data: This directory is used to store configuration files that can be used by Hugo when generating your website.
layouts: Stores templates in the form of .html files. See the Styling section for more information.
static: Stores all the static content: images, CSS, JavaScript, etc. When Hugo builds your site, all assets inside your static directory are copied over as-is.
themes: Hugo theme of your choice.

Styling our static site

Remember, we applied a theme before. Now, if we inspect the themes folder, we can see the styling files.

But beware!

DO NOT EDIT THESE FILES DIRECTLY.

Instead, we will mirror the theme directory structure to the root layouts folder.

Let's say I want to apply custom CSS to the theme.

The theme has a themes/theme-name/layouts/partials folder, where we can find some HTML templates (header.html, footer.html). Now we will edit the header.html template, so copy the content from this file to layouts/partials/header.html and be careful to create the same directory structure like the theme's into the root layouts folder.

layouts/partials/header.html

themes/theme-name/layouts/partials/header.html

Create a custom CSS file: static/css/custom-style.css.

Add the custom css file to config.toml:

[params]
 custom_css = ["css/custom-style.css"]

Open layouts/partials/header.html:

Add this code inside the <head> tag:

{{ range .Site.Params.custom_css -}}
   <link rel="stylesheet" href="{{ . | absURL }}">
{{- end }}

Now you can overwrite CSS classes applied by your theme.

Deploying our static site to Netlify

One of the benefits of a static site is that you can deploy it easily. Netlify or AWS S3 is a very good choice for hosting a static site. Let’s see how to deploy it to Netlify.

Requirements:

Netlify account
Github repository

What to do on Netlify

.1. Create a git repository

.2. Create a netlify.toml file into the root of your project with the content below.

[build]
publish = "public"  // default hugo build folder
command = "hugo --gc --minify" // hugo build command

[context.production.environment]
HUGO_VERSION = "0.62.1"
HUGO_ENV = "production"
HUGO_ENABLEGITINFO = "true"

.3. Now, if you push your code to Github, Netlify will deploy the site, and blogging shall start.

.4. Connect Netlify and your Git repository

.5. Build and deploy

You can also take a look at the Hugo official Netlify hosting details for further information.

AWS S3 + CI

We will be using Terraform to create an S3 bucket that will host our static site. I assume you have an AWS account.

Create an s3_bucket.tf file in your project and insert the content below to it:

resource "aws_s3_bucket" "my-static-site" {
  bucket = "my-static-site"
  acl    = "public-read"
  policy = <<POLICY
{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"PublicRead",
      "Effect":"Allow",
      "Principal": "*",
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::my-static-site/*"]
    }
  ]
}
POLICY

  website {
    index_document = "index.html"
    error_document = "index.html"
  }
}

Then, folllow these steps:

execute terraform init
insert the following code snippet

export AWS_ACCESS_KEY_ID="anaccesskey"
export AWS_SECRET_ACCESS_KEY="asecretkey"
export AWS_DEFAULT_REGION="us-east-1"
terraform plan

execute terraform apply

This will create a public readable AWS S3 bucket, which will host the static site.

Now we have to take care of building the static site on our own: we need a CI tool.

Setting up CircleCI for continuous integration

At RisingStack, we’re usually using CircleCI for these kinds of tasks. Let’s set up the build process.

Create the following files in the project root: .circleci/config.yml

Apply the following code to config.yml:

version: 2
jobs:
  build:
    docker:
      - image: cibuilds/hugo:latest
    working_directory: ~/hugo
    environment:
      HUGO_BUILD_DIR: ~/hugo/public
    steps:

      # install git
      - run: apk update && apk add git

      # checkout the repository
      - checkout

      # install git submodules for managing third-party dependencies
      - run: git submodule sync && git submodule update --init

      - run:
          name: install AWS CLI (first install pip, the Python package manager)
          command: |
            apk add --update python python-dev py-pip build-base
            pip install awscli

      # build with Hugo
      - run: HUGO_ENV=production hugo -v -d $HUGO_BUILD_DIR

      - run:
          name: test our generated HTML files
          command: |
            htmlproofer $HUGO_BUILD_DIR --allow-hash-href --check-html \
            --empty-alt-ignore --disable-external

      # `deploy` step: identical to a `run` step, but uses only one container:
      # /docs/2.0/configuration-reference/#deploy 
      - deploy:
          name: deploy to AWS
          command: |
            if [ "${CIRCLE_BRANCH}" = "master" ]; then
              aws s3 sync $HUGO_BUILD_DIR \
              s3://my-static-site --delete
            else
              echo "Not master branch, dry run only"
            fi

Note: To add AWS credentials, locate the AWS Permissions link under the Permissions section in CircleCI.

Moving forward with Hugo

Hugo offers so much more you should know about. Discover the official documentation here.

If you'd like to check out the very simple page I created, head over to https://bmz-codez.com/

In case you professional software development services, feel free to reach out to us using our website: https://risingstack.com/nodejs-development-consulting-services