DEV Community: Rishabh Rawat

Understanding Offset and Cursor-Based Pagination in Node.js

Rishabh Rawat — Wed, 29 May 2024 15:00:41 +0000

Pagination means dividing a big chunk of data into smaller pages. It unlocks performance benefits for the backend, while also improving UX by sending manageable pieces of data to a client.

In this article, we'll explore offset-based and cursor-based server-side pagination approaches, compare them, and implement cursor-based pagination in Node.js.

Let's get started!

Types of Pagination

There are two broad pagination approaches:

Client-side pagination
Server-side pagination

The significant difference between them lies in where the pagination process takes place. In client-side pagination, the complete dataset is fetched from a database and partitioned into pages of a given size on the client side. Meanwhile, in server-side pagination, the client only fetches the data required for a given page number. When a user navigates to the next page, the client makes a fresh call to the server for new data.

So, for smaller datasets, client-side pagination works well, but server-side pagination best fits large data use cases.

In the coming sections, we'll learn about two primary ways to perform server-side pagination.

Before we proceed, we need a sample API to work with.

Prerequisites

To follow this tutorial, you'll need:

A PostgreSQL server installed and running on your local machine (I prefer psql to interact with the database).
Node.js and npm installed, with Node.js version >= 16.
A basic understanding of a REST API with the Express.js framework.

This article contains some code snippets, but you can find the complete working code on GitHub.

Installing Our Sample Node.js API

Here are the steps to run the sample API locally.

Clone the repo:

git clone git@github.com:Rishabh570/pagination-nodejs.git

Install the dependencies:

npm install

Source the environment variables.

You can find the env variables in config.js at the project's root level. Rename the .env.example file to .env and fill in your desired values for the environment variables.

If Postgres is not set to trust local connections, provide a correct password for your Postgres user. To turn off password authentication, edit your pg_hba.conf file (run SHOW hba_file; as a superuser to find the file location) to trust all local connections. It should look like this:

host    all             all             0.0.0.0/0             trust

Once done, you can load environment variables with source .env on macOS. Otherwise, try replacing "source" with "." For more information, refer to the 'source command not found in sh shell' StackOverflow thread or load the environment variables directly into the terminal.

Run the server:

npm start

Create a table in your Postgres database. Connect to your Postgres database locally and run the below command to create a table named "items".

create table items (
  id bigint generated by default as identity,
  created_at timestamp with time zone not null default now(),
  name text not null,
  description text null,
  constraint item_pkey primary key (id),
  constraint item_id_key unique (id),
  constraint item_name_key unique (name)
);

Load sample data in your items table by hitting the API endpoint — POST /item/create.

You should now have an API server running locally and connected to a local Postgres database. Feel free to run the queries in this article locally to solidify your understanding.

In the following sections, let's look at offset-based and cursor-based pagination, starting with offset-based first.

Understanding Offset-Based Pagination

Offset-based pagination is intuitive and straightforward, which explains why it's been widely adopted. The key parameters behind this pagination type are offset and limit clauses.

The offset clause helps skip past a set of rows, while limit limits the query output.

The latency of executing queries on a database with offset-based pagination increases as the data grows. Here's the planning and execution time when we run it on our sample database (you can run the same commands in your local Postgres database):

The latency grows by more than 10x when the offset parameter is increased:

The good part about offset-based pagination is that it can adapt quickly to almost any pagination style. It's straightforward to implement, whether you're implementing sequential or random access.

For implementing sequential access (previous/next, infinite scroll, or load more pagination style), the offset is simply increased or decreased to get the desired data. For instance, the offset is four times the page size to load the fifth page. And for returning to the first page, the offset is zero times the page size.

Drawbacks of Offset-Based Pagination

It's important to note that our table from the last section stores very minimal data — name, description, created_at, and id. The data is much bigger in real-world scenarios, and latencies can exceed the linear proportionality scale.

If we keep performance-related concerns aside, offset-based pagination has another major drawback — duplicate or missing entries.

Let's understand with an example.

We have ten rows in a table, the page size is five, and the rows are ordered by most recent first. Here's how the two pages ideally look:

Imagine a new row is inserted into the table while a user is on the first page. When the user visits the next page, a duplicate (or already seen) entry creeps into the following page:

Similarly, if one of the entries on the first page gets deleted when the user visits the second page, they permanently miss that entry:

The user never gets to see the fifth entry because it's on the first page now, and the user is on the second page.

Cursor-based pagination is immune to such issues because it doesn't use the offset clause. We'll look into cursor-based pagination in detail in the next section. But from what we have observed so far, offset-based pagination can be summarized as follows:

Pros

Intuitive and straightforward to implement.
Works well for datasets that don't change frequently.
Implementing any pagination logic (sequential and random access) is straightforward.

Cons

Noticeable performance drops as the dataset grows.
Unreliable for frequently changing data, as it can show duplicate entries or miss them.

Let's look at how cursor-based pagination offers better potential in terms of performance and how it handles the problem of missing/duplicate entries.

Understanding Cursor-Based Pagination

Cursor-based pagination leverages the where clause to get relevant data. Compared to offset-based pagination, this is a better approach in terms of both performance and reliability.

The stark difference in the query planner's number of rows is immediately visible, and so are the planning and execution times:

Offset-based pagination is almost 9x slower compared to cursor-based pagination.

It's important to mention that the cursor field plays a crucial role in the correctness of the result. Any column can work as a cursor if the values follow a strictly increasing or decreasing order. Sometimes, that means going out of the way to add such columns to the table (a timestamp or an incrementing/decrementing ID, for instance).

To emphasize the importance of a strictly increasing/decreasing order, let's analyze a potential issue I faced while working on the API:

In our items table, created_at is a timestamp column (with timezone) that's precise to the millisecond.
But new Date().toISOString() in Javascript doesn't provide the same precision. As a result, our API stores duplicate values from two rows if they are created close enough.

This leads to unexpected behavior when fetching a new page from the database. I fixed it by putting some time intervals (big enough for JavaScript to notice) between every created item.

Looking at all the aspects of cursor-based pagination, the pros and cons are as follows:

Pros

Significant performance upside on bigger datasets.
Works for frequently changing datasets — no duplicates or missing entries problems.

Cons

Relatively complex to implement — especially with random access pagination style (where the user can jump to the fifth page from the first one).
Cursor selection plays a crucial role in the correctness of output.

There are merits to both offset- and cursor-based approaches — let's explore which one to pick for the needs of your application.

Choosing the Right Pagination Strategy for Your Node.js App

It largely depends on the scope and impact. If you're working on a smaller dataset, it's not worth the hassle of setting up cursor-based pagination. Going ahead with an intuitive and relatively simple approach will allow you to focus on bigger goals for your app (e.g., bringing in more customers).

However, latency becomes noticeable as your database hits millions of rows. The random access pattern can add to latency and throughput. Depending on the use case, embracing complexity might be the only way forward. We'll talk more about UX-imposed challenges later in this article.

Let's see how we can set up cursor-based pagination next.

How to Implement Cursor-Based Pagination

The idea is to track a column's value with a cursor and use that to navigate across different pages. We'll implement sequential access and use the most recent first sorting order. The pagination style can be any one of the following:

"Next" and "previous" buttons to navigate
An infinite scroll
A load more button

Here's an overview of the next/previous pagination style:

The only difference in an infinite scroll or a load-more variant is that the user cannot return to the previous page because there's no previous page. The data just keeps getting appended.

Regardless of the pagination style you're going with, the initial load will fetch a fixed number of entries from the database:

SELECT * FROM items order by created_at desc limit 10;

We get back ten entries sorted in the desired order:

When the user visits the second page, the where clause helps us fetch the next set of ten entries:

SELECT * FROM items WHERE created_at < '2023-11-24T10:07:32.443Z' order by created_at desc limit 10;

We've picked the smallest created_at value for the where clause. It gives us the following ten entries (second page):

So far, so good.

The query is similar if a user goes back to the first page:

SELECT * FROM items WHERE created_at > '2023-11-24T10:07:31.438Z' order by created_at asc limit 10;

We've used the largest created_at value and filtered out the entries greater than that with a max cap of ten again. But there's a catch. The sorting order has changed in the query. Let's look at the diagram below and understand why it is required.

First of all, let's highlight the known pieces of this setup:

The page size is ten.
The entries are sorted by most recent first.
There are 30 entries in the table, with each row showing the timestamp (number on the tiles).
We are currently on the last page. The current largest and smallest timestamps are highlighted on the diagram.

The first step to move back to the previous page is simple — we want to find all the rows where the timestamp exceeds the current largest timestamp (highlighted in the diagram). In our case, twenty rows match this filter.

The second step is to pick the ten most recent rows from the results obtained in the first step. However, sorting the rows in descending timestamp order will give us rows 30 to 21. We want rows 20 to 11. This is why we need to sort the rows in ascending order.

This sudden change in the sorting order can be problematic if the client expects a sorted response from the API. This implementation detail can be ironed out during the design phase.

If a client shares a sorting preference in the request, the server can sort the results before sending them to the client:

rows.sort((a, b) => {
  if (
    new Date(a.created_at).toISOString() ===
    new Date(b.created_at).toISOString()
  ) {
    if (order === "desc") return +b.id - +a.id;
    return +a.id - +b.id;
  }

  if (order === "desc") return new Date(b.created_at) - new Date(a.created_at);
  return new Date(a.created_at) - new Date(b.created_at);
});

If the sorting order changes when a user has progressed to future pages, then usually the first page resets with the new sorting order. For example, if the user requests changing the sorting order from most recent first to the oldest first while on page 2, it should reset to the first page (with entries sorted by oldest first).

If this feels like an information dump, reviewing the complete code should give you a good birds-eye view of the implementation:

async function queryRunner(postgres, dir, order = "desc", limit = 10) {
  let result;

  // Filter and fetch rows with created_at > largestTimestamp when:
  // 1. we want to fetch the results for the next page in an ascending order, or
  // 2. we want to fetch the results for the previous page in a descending order
  if (
    dir &&
    ((dir === "next" && order === "asc") ||
      (dir === "previous" && order === "desc"))
  ) {
    result = await postgres.query(
      `SELECT * FROM items WHERE created_at > '${largestTimestamp}' order by created_at asc limit ${PAGE_SIZE};`
    );
  }
  // Filter and fetch rows with created_at < largestTimestamp when:
  // 1. we want to fetch the results for the next page in a descending order, or
  // 2. we want to fetch the results for the previous page in an ascending order
  else if (
    dir &&
    ((dir === "next" && order === "desc") ||
      (dir === "previous" && order === "asc"))
  ) {
    result = await postgres.query(
      `SELECT * FROM items WHERE created_at < '${smallestTimestamp}' order by created_at desc limit ${PAGE_SIZE};`
    );
  }
  // If direction (or, dir) is not provided, fetch the first page
  else {
    result = await postgres.query(
      `SELECT * FROM items order by created_at ${order} limit ${limit};`
    );
  }

  if (result && result.rows && result.rows.length === 0) return [];
  const rows = result.rows;

  // sort rows in the order that client expects
  rows.sort((a, b) => {
    if (
      new Date(a.created_at).toISOString() ===
      new Date(b.created_at).toISOString()
    ) {
      if (order === "desc") return +b.id - +a.id;
      return +a.id - +b.id;
    }

    if (order === "desc")
      return new Date(b.created_at) - new Date(a.created_at);
    return new Date(a.created_at) - new Date(b.created_at);
  });

  // Update largest and smallest timestamps from fetched data
  if (order === "desc") {
    largestTimestamp = new Date(rows[0].created_at).toISOString();
    smallestTimestamp = new Date(
      rows[rows.length - 1].created_at
    ).toISOString();
  } else {
    largestTimestamp = new Date(rows[rows.length - 1].created_at).toISOString();
    smallestTimestamp = new Date(rows[0].created_at).toISOString();
  }

  return result.rows;
}

Now that we have implemented cursor-based pagination for our Node.js API, it's time to look at how this approach plays out with random access patterns.

UX-Imposed Challenges

There are a couple of ways to implement pagination in the UI:

Sequential access - Infinite scroll or "Load more" or "prev/next" buttons.
Random access - Numbered page.

Offset-based pagination works (ignoring performance bottlenecks) with both access patterns. However, when using cursor-based pagination, random access can be problematic. Let's quickly recap how cursor-based pagination works:

Fetching the first page involves a fetch with a limit clause (without any cursor).
For the second page, the combination of where and limit is used, e.g., where created_at < some_timestamp and limit = 10.

If we look closer, fetching the second page (and so forth) requires the smallest/largest timestamp of the first page. To allow random access, i.e., jump to the fifth page from the first one, we need to know the smallest/largest timestamp of the fourth page.

This leads to the following conclusions:

Enabling random access is only possible when you know each page's smallest/largest timestamp values.
Precomputing and storing the smallest/largest timestamps that correspond to every page only helps a little if the data keeps changing (with rows being added or deleted).

Even though computing this on every request is better than offset-based pagination, it is undoubtedly a more complex approach, so best avoided for smaller-scale applications.

Wrapping Up

In this post, we've learned why pagination is necessary, the types of pagination, and two primary ways to implement server-side pagination. We also discussed some UX-imposed challenges when working with cursor-based pagination.

Now, you should have sufficient knowledge to implement end-to-end pagination and improve the performance of your Node.js API.

Thanks for reading!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Build an Idempotent API in Node.js with Redis

Rishabh Rawat — Wed, 28 Feb 2024 14:24:31 +0000

In a world full of microservices and distributed systems, building robust and reliable APIs is of paramount concern. One key concept that plays a vital role in achieving reliability is idempotency.

In this article, we'll delve into the world of idempotency, exploring what it is, why it's important, and how it can be implemented to solve the persistent problem of duplicate processing in APIs. Along the way, we'll see how to build an idempotent API in Node.js with Redis.

What We'll Cover

Here's what we'll talk about in this post:

What is idempotency?
Why idempotency is important
Idempotent HTTP methods
A duplicate processing problem using a sample Node.js API
Adding idempotency to sequential requests in Node.js
Idempotency for parallel duplicate requests in Node.js
Real-world applications

You can see all the code used in this post on GitHub.

What Is Idempotency?

An operation is considered idempotent if applying it multiple times has the same effect as applying it once. In a REST API, that means performing multiple HTTP requests should have the same outcome (not output) as that of a single HTTP request.

Why Idempotency Is Important

Idempotency is a crucial ingredient in building reliable and scalable APIs. While it might be acceptable to have an API that's not idempotent for a personal project, it's a red flag in the context of a distributed system.

In a typical distributed system, various services interact with each other. To complete an operation, a request has to pass through multiple stages, potentially encountering network issues, disk failures, delays in scaling up, or even the occasional service going offline. In such complex scenarios, it's almost certain that requests will need to be retried at some point.

This is where having idempotent operations is absolutely crucial. For retries to work successfully, all operations must adhere to the principle of idempotency. Retrying requests should not produce incorrect or unintended results.

Idempotent HTTP Methods

Some HTTP methods are idempotent by default. For example, fetching details for a resource using the GET method is idempotent because it'll always give the same result for a given input. Similarly, the following methods are also idempotent:

HEAD
OPTIONS
TRACE
PUT
DELETE

Note that the last two methods (PUT and DELETE) are idempotent but unsafe. A safe HTTP method cannot alter the state of resources on the server side. To summarize:

All safe HTTP methods are idempotent.
PUT and DELETE are idempotent, but unsafe since they can alter the server state.

This leaves us with POST and PATCH HTTP methods, which are non-idempotent. We will learn how to make these methods idempotent in the coming sections.

A Duplicate Processing Problem Using a Sample Node.js API

A request can fail due to network partitions, disk failures, timeouts, throttling, or DNS lookup issues. There are a lot of real-world scenarios where idempotency helps. Let's zip through a few:

Websites typically show a “please wait, already processing” popup when you accidentally double-click or try to leave a page while it's processing your request.
Clicking on the elevator button multiple times doesn't change its behavior. The elevator still goes to the desired floor.
Wishlisting an item that's already been added to your wishlist on Amazon simply brings that item to the top of your wishlist, but it still holds the same items.

The applications of idempotency are not just limited to one use case. This is precisely why it's such a valuable concept to learn as a developer. It's language agnostic, so there are no barriers to entry.

To understand the problem and get to a working solution, our journey will include a sample Node.js API 👇:

async function create(req, res) {
  try {
    const { longURL } = req.body;

    // New long URL, generates a new and unique slug
    const slug = await urlService.generateNewSlug();

    // save short URL <> long URL mapping to the database
    await urlService.saveToDB(slug, longURL);

    // Return the newly generated short slug
    return res.status(HTTP_STATUS_CODES.SUCCESS).json({
      status: true,
      message: "Short slug created successfully",
      data: { slug },
    });
  } catch (err) {
    return res.status(HTTP_STATUS_CODES.INTERNAL_SERVER_ERROR).json({
      status: false,
      message: "Could not create short slug",
      error: err,
    });
  }
}

It creates a new short slug for every long URL passed to it and saves it into the database. Let's give it a run:

{
  "status": true,
  "message": "Short slug created successfully",
  "data": {
    "slug": "UfBHWg"
  }
}

This is functional but has two flaws. We'll uncover them in the next sections.

Adding Idempotency to Sequential Requests in Node.js

Our current API implementation returns a different short slug every time you request the same long URL. Not only is this redundant, but it's a waste of CPU and storage resources. We need to ensure that there's only one short slug present for a given long URL at any point in time.

To achieve idempotency, we can leverage our saved mappings in the database. For a given long URL, we can check if a slug exists and return early if that's the case.

Here's the updated code that generates a short slug:

async function create(req, res) {
  try {
    const { longURL } = req.body;

    // Check if the mapping already exists
    const mapping = await service.urlService.findByLongURL(longURL);
    if (mapping) {
      return res.status(HTTP_STATUS_CODES.SUCCESS).json({
        status: true,
        message: "Short slug created successfully",
        data: { slug: mapping.slug },
      });
    }

    // New long URL, generates a new and unique slug
    const slug = await urlService.generateNewSlug();

    // save short URL <> long URL mapping to the database
    await urlService.saveToDB(slug, longURL);

    // Return the newly generated short slug
    return res.status(HTTP_STATUS_CODES.SUCCESS).json({
      status: true,
      message: "Short slug created successfully",
      data: { slug },
    });
  } catch (err) {
    return res.status(HTTP_STATUS_CODES.INTERNAL_SERVER_ERROR).json({
      status: false,
      message: "Could not create short slug",
      error: err,
    });
  }
}

Note that we now check for the mapping in our database before generating a new slug. If the mapping exists, we return the stored slug. Otherwise, we generate a new one (and store that in the database).

This ensures we're processing a long URL only once.

Problem solved. But is it?

Idempotency doesn't mandate the same API response for every duplicate request. For instance, when using a create API, the initial request might return a 200 status code, while a subsequent duplicate request could instead return a 202 status code and not create the resource again. Even though the response differs, the API's behavior remains idempotent.

Idempotency for Parallel Duplicate Requests in Node.js

Let's fire multiple requests to our short slug generator in parallel. Here's our script:

const async = require("async");
const axios = require("axios");

const payload = {
  method: "post",
  url: `http://localhost:8201/url/create`,
  headers: { "Content-Type": "application/json" },
  data: {
    longURL: "http://example.com/100",
  },
};

function shortenURL(reqPayload) {
  return axios(reqPayload).then((resp) => {
    return resp.data;
  });
}

async.parallel(
  [
    function (callback) {
      return shortenURL(payload)
        .then((res) => {
          console.log("res1: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 1 = ", err.message));
    },
    function (callback) {
      payload.data.num += 1;
      return shortenURL(payload)
        .then((res) => {
          console.log("res2: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 2 = ", err.message));
    },
    function (callback) {
      payload.data.num += 1;
      return shortenURL(payload)
        .then((res) => {
          console.log("res3: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 3 = ", err.message));
    },
    function (callback) {
      payload.data.num += 1;
      return shortenURL(payload)
        .then((res) => {
          console.log("res4: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 4 = ", err.message));
    },
    function (callback) {
      payload.data.num += 1;
      return shortenURL(payload)
        .then((res) => {
          console.log("res5: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 5 = ", err.message));
    },
    function (callback) {
      payload.data.num += 1;
      return shortenURL(payload)
        .then((res) => {
          console.log("res6: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 6 = ", err.message));
    },
    function (callback) {
      payload.data.num += 1;
      return shortenURL(payload)
        .then((res) => {
          console.log("res7: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 7 = ", err.message));
    },
    function (callback) {
      payload.data.num += 1;
      return shortenURL(payload)
        .then((res) => {
          console.log("res8: ", res.data.slug);
          callback(null, res.data.slug);
        })
        .catch((err) => console.log("err 8 = ", err.message));
    },
  ],
  (err, results) => {
    if (err) {
      console.log("Final error: ", err.message);
    } else {
      console.log("results: ", results);
    }
  }
);

And here's the output when we run the above script:

results:  ['s9SW7p', 's9SW7p', 's9SW7p', 'el42vA', 'TswIY7', 'aFwbK7', 'AYcaBM', 'gU6fP8']

Our barebones API isn't faring well when hit with a couple of parallel duplicate requests. Even when the requests are precisely the same, the output is different. We need to ensure that parallel duplicate requests do not cause unintended side effects due to a Node.js context switch.

Welcome, locks!

Distributed locks are crucial in environments where different processes or clients have access to shared resources. In such environments, locks are used to prevent race conditions — scenarios where two or more operations access a shared resource and attempt to modify it simultaneously, leading to unpredictable outcomes. Our API faces such challenges when parallel duplicate requests are executed. By setting an exclusive lock, we ensure that only one request can be processed at a time.

We'll implement an exclusive lock mechanism using Redlock (Redis Lock) in our API. Redlock is a distributed locking solution that helps prevent the concurrent execution of code blocks by different processes.

Since race conditions happen only when parallel duplicate requests are fired, we can set an exclusive lock in the control flow to allow only one request at a time. It's important to understand that two duplicate requests can be processed, but not at the exact same time.

Let's acquire an exclusive lock using Redlock at the entry point:

async function create(req, res) {
  let lock;
  try {
    // Try to acquire the lock
    lock = await service.urlService.acquireLock(
      "URL:CREATE:ExclusiveLock",
      100
    );

    const { longURL } = req.body;

    // Check if the mapping already exists
    const mapping = await service.urlService.findByLongURL(longURL);
    if (mapping) {
      return res.status(HTTP_STATUS_CODES.SUCCESS).json({
        status: true,
        message: "Short slug created successfully",
        data: { slug: mapping.slug },
      });
    }

    // New long URL, generate a new and unique slug
    const slug = service.urlService.generateNewSlug();

    // save short URL <> long URL mapping to the database
    await service.urlService.saveToDB(slug, longURL);

    // Return the newly generated short slug
    return res.status(HTTP_STATUS_CODES.SUCCESS).json({
      status: true,
      message: "Short slug created successfully",
      data: { slug },
    });
  } catch (err) {
    return res.status(HTTP_STATUS_CODES.INTERNAL_SERVER_ERROR).json({
      status: false,
      message: "Could not create short slug",
      error: err,
    });
  } finally {
    /**
     * Manually release the lock when the operation is complete
     *
     * NOTE: Redlock's release method is not a no-op, it throws an error if you try to release an already expired lock ([more here](https://github.com/mike-marcacci/node-redlock/issues/168#issuecomment-1165700540)). Setting a small TTL usually triggers this unexpected behavior.
     * As a workaround, we're ignoring the errors from lock.release()
     */
    if (lock) await lock.release().catch(() => {});
  }
}

Now, each request initially attempts to acquire an exclusive lock. If it's unsuccessful, the request is rejected, resulting in a lock acquisition error. Here's how the output appears when running the same script, which sends parallel duplicate requests:

As expected, only one of those 8 parallel duplicate requests can generate a slug. The others fail during the lock acquisition stage. We have finally made our random slug generator truly idempotent.

Real-World Applications

What I find appealing about idempotency is its versatility. You'll notice it being applied in different situations and technologies, regardless of the tools used. As I mentioned earlier, it's actually quite commonly employed in:

Cases where achieving a one-time action is crucial, such as in transactional systems like banks and stock exchanges.
Various GitOps tools like ArgoCD, where reapplying the same configuration doesn't lead to redundant deployments.
Working with React.js, because setting the same state doesn't re-render a component.
Sound financial systems, so they don't suffer from double payment issues where the customer is accidentally charged twice.

Wrapping Up

In this article, we learned what idempotency is, how it can prevent duplicate processing problems, and how to use it in your Node.js application. You now have one more tool at your disposal to build better APIs.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Throttling a Kafka Queue in Node.js

Rishabh Rawat — Wed, 14 Feb 2024 11:34:36 +0000

Coming from an HTTP-only background, message queues look very fascinating. Unlike HTTP, you can just push events to the queue and move on. After all, they're a big part of what makes async communication possible between microservices.

In this article, we'll take a look at Apache Kafka and how you can build a durable throttling layer with it.

Let's get started!

What We'll Cover

Here's what we'll go through in this post:

Why use Kafka?
The problem with normal consumption
How to enable manual commits to throttle consumption
Granular control with async queue
Putting it all together
Real-world applications
Wrapping up

Prerequisites

To follow along with this tutorial, you will need:

Working knowledge of building REST API with Express.js (Node.js version >= 16)
Basic knowledge of Apache Kafka
(Optional) AppSignal account (for application performance monitoring)
A running MongoDB cluster on your Atlas account

This article contains a few code snippets. You can refer to the complete working code along with the getting started guide on GitHub.

Installing Kafka

You can skip this section if you have Apacha Kafka already installed on your machine. The complete quickstart is available in the official Kafka documentation. Here's what we need to do:

Download Kafka as indicated in step 1 of the documentation above.
Once installed, go to the downloaded Kafka folder. Your Kafka installation version might differ.

cd kafka_2.13-3.3.1/

Run zookeeper and server by running these commands in separate terminal tabs. The commands must be run in the given order.

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

If one stops or exits the process, try to restart both. You should only proceed when both scripts are in a running state and there isn't an expiring session.

Open a new tab in your terminal and create a test topic. We'll use this topic throughout the article.

bin/kafka-topics.sh --create --topic testTopic2 --bootstrap-server localhost:9092

To confirm if the topic has been successfully created, you can run the following command:

bin/kafka-topics.sh --describe --topic testTopic2 --bootstrap-server localhost:9092

You should have Apache Kafka running on localhost:9092 after following these steps. We can proceed to the next steps.

Setting up AppSignal for Node.js

This is an optional step. We're going to integrate AppSignal to monitor and learn more about our Node.js application. You need to set up an AppSignal account to follow this section.

We need two things to integrate AppSignal into our project:

AppSignal application name. You can find this in the config.js file under the "appName" field. I've created an app called "throttle-kafka-nodejs".
Your AppSignal API key, sourced as "APPSIGNAL_PUSH_API_KEY" from the ENV file.

You can find the AppSignal SDK in the src/appsignal.cjs file.

Let's now turn to why we should use Kafka in the first place.

Why Use Kafka for Node.js?

Apache Kafka provides a durable queue for your events with built-in reliability. It becomes pretty evident how Kafka can handle billions of events once you look at its architecture.

Durability

In contrast to other event queues where the events are lost upon consumption, this is not the case with Kafka. Unsurprisingly, that's one of its biggest selling points. Having access to the events even after consumption gives a sense of safety that is unparalleled.

The events are stored on disk and have a default retention period of 7 days. The retention period is configurable, and increasing it comes at the cost of a bigger disk.

Robustness

Kafka is designed with robustness in mind. Here's a simplistic generalized overview:

Brokers are nodes, a topic is a queue, and you can have various topics. Each topic is divided into a predetermined number of partitions (think database partitioning).

Writes go to the leader and are replicated to the followers on separate nodes (to ensure high availability or HA). Every partition is also copied to other nodes or brokers (configured via a replication factor setting) for HA.

When a broker dies, the broker leaders have to be re-elected on one of the live brokers. For instance, if broker 3 dies in our above-configured cluster, the leader for partition 3 will be elected from either broker 1 or broker 2. This happens by the cluster automatically and is not something you should worry about.

The Problem with Normal Consumption

Now that we know what makes Kafka one of the best candidates for the pub-sub mechanism, we should give it a run. We won't change any default configs just yet. We have a basic producer and a batch consumer set up in our Node.js application.

Let's push some events and see how our application performs:

router.post("/produce", async (req, res) => {
  try {
    const { events } = req.body;
    const producer = await kafkaProducer;
    const promises = events.map(async (event) => {
      const payload = {
        topic: config.kafka.topic,
        messages: [{ value: JSON.stringify(event) }],
      };

      return producer.send(payload);
    });

    await Promise.all(promises);

    return res
      .status(200)
      .json({ status: true, message: "Produced event successfully" });
  } catch (err) {
    return res
      .status(500)
      .json({ status: false, message: `Could not produce event: ${err}` });
  }
});

Since there's no way to stop consumption cycles, our application consumer will keep fetching the events in batches of the default size — 10MB. However, our application may be unable to process the events at the same rate. This causes the application to accumulate events in memory, making it prone to unexpected downtime due to resource starvation.

Moreover, autocommit is turned on by default in various Kafka Node.js client libraries, including KafkaJS. As the name suggests, all the events you fetch are committed automatically after a batch is successfully processed. But what if an application faces downtime in the middle of batch execution? The same events will be fetched again from Kafka in the next cycle, causing duplicate events.

But this isn't a bug, it's a feature. It ensures "at least once" semantics — see the documentation. Your use case determines your approach (e.g., autocommit in a financial system should be a no-go).

In this article, we're focusing on "exactly once" semantics and so want to avoid autocommit.

Monitoring the Anomalies in Our Node.js Application

The default batch limit of 10 MB might be too much (or too little) depending on the size of one event. Fetching events in excess can lead to an unbearable load on the consuming application. To observe and look out for anomalies, we're using AppSignal for Node.js.

We can analyze slow API endpoints and database queries for our application without setting up any configs or enabling integrations.

It's evident that our database is not having a good time. Let's improve this — time to reduce the batch size and switch to manual Kafka commits.

Manual Kafka Commits to the Rescue

Let's turn some knobs to ensure our application doesn't get overwhelmed.

We'll update our Kafka consumer to incorporate batching. We'll fetch a manageable batch size from Kafka and keep autocommit turned off, instead performing manual commits.

Here's our consumer function:

initConsumer: async (mongoClient) => {
  const consumer = KafkaJS.consumer({
    groupId: config.kafka.consumer,
    topic: config.kafka.topic,
    maxBytesPerPartition: config.kafka.maxBytesPerPartition,
    maxBytes: config.kafka.maxBytes,
    allowAutoTopicCreation: true,
  });

  await consumer.connect();

  consumer.on("consumer.commit_offsets", (event) => {
    console.info("✅ offset is committed");
  });

  await consumer.subscribe({ topic: config.kafka.topic, fromBeginning: false });

  await consumer.run({
    autoCommit: false,
    eachBatchAutoResolve: false,
    eachBatch: async ({
      batch,
      resolveOffset,
      heartbeat,
      commitOffsetsIfNecessary,
      uncommittedOffsets,
      isRunning,
      isStale,
      pause,
      resume,
    }) => {
      console.log("🛑 Pausing consumer group");
      pause([{ topic: config.kafka.topic }]);

      console.log(`🚧 Processing ${batch.messages.length} messages 🚧`);
      for (let message of batch.messages) {
        if (!isRunning() || isStale()) break;

        // Processing the event
        tracer.startActiveSpan(
          "insert | kafka event | SUCCEEDED",
          async (span) => {
            const Item = mongoClient.collection("items");

            // Send query data to AppSignal
            setCategory("query.mongodb");
            setBody(`Item.insertOne()`);
            setTag("operationType", "insert");

            // Save event data to DB
            await Item.insertOne(message);

            span.end();
          }
        );

        resolveOffset(message.offset);
        await heartbeat();

        commitOffsetsIfNecessary([message.offset]);
      }

      console.log("🟢 Resuming consumption for next batch");
      consumer.resume([{ topic: config.kafka.topic }]);
    },
  });
};

Let's run through this step-by-step:

We've explicitly set fetchMaxBytes and turned off autocommit.
Upon successful execution, we mark the event as processed by invoking commitOffsetsIfNecessary manually (this was automated earlier).
Once the events are committed to offset, we resume consumption.
The commits are happening after processing every message. This way, we don't wait for the whole batch to finish processing before a commit can happen.

With these changes, we've improved the reliability of the pub-sub setup through manual commits and smaller batches. Let's explore how we can improve this further in the next section.

Granular Control with Async Queue in Kafka

As you might've noticed, we specify the batch size, not the number of events. In fact, the number of events we get in a batch is completely dependent on the size of one individual event. This leaves room for unwanted surprises. We don't want our application to get burdened with more events than it can process.

Welcome, async queue!

We will build a second throttling layer with async queue. With this queue, our application will only process a set number of events at once. The number is specified by the concurrency parameter.

Let's understand with an example.

The second throttling layer lets us specify the exact number of events we want to process concurrently. Here's the updated consumer code:

initConsumer: async (mongoClient) => {
  const concurrentQueue = async.queue(async (queueData) => {
    const data = JSON.parse(queueData.value);
    console.time(`${data.hash} took`);
    console.log("processing in queue: ", data);

    // AppSignal event capturing, can comment if throttling has unexpected behaviour
    tracer.startActiveSpan("insert | kafka event | SUCCEEDED", async (span) => {
      const Item = mongoClient.collection("items");

      // Send query data to AppSignal
      setCategory("query.mongodb");
      setBody(`Item.insertOne()`);
      setTag("operationType", "insert");

      // Save event data to DB
      await Item.insertOne(data);

      console.timeEnd(`${data.hash} took`);

      span.end();
    });
  }, config.kafka.queueConcurrency);

  const consumer = KafkaJS.consumer({
    groupId: config.kafka.consumer,
    topic: config.kafka.topic,
    maxBytesPerPartition: config.kafka.maxBytesPerPartition,
    maxBytes: config.kafka.maxBytes,
    allowAutoTopicCreation: true,
  });

  // Connect to the consumer group
  await consumer.connect();

  consumer.on("consumer.commit_offsets", (event) => {
    console.info("✅ offset is committed");
  });

  // Subscribe to the topic
  await consumer.subscribe({ topic: config.kafka.topic, fromBeginning: false });

  // Run the consumer
  await consumer.run({
    autoCommit: false, // This is turned off since we're doing manual commits
    eachBatchAutoResolve: false,
    eachBatch: async ({
      batch,
      resolveOffset,
      heartbeat,
      commitOffsetsIfNecessary,
      uncommittedOffsets,
      isRunning,
      isStale,
      pause,
      resume,
    }) => {
      console.log("🛑 Pausing consumer group");
      pause([{ topic: config.kafka.topic }]);

      let offsets = [];
      console.log(`⬇️ Fetched ${batch.messages.length} messages from batch`);
      for (let message of batch.messages) {
        if (!isRunning() || isStale()) break;

        // Process the event
        concurrentQueue.push(message);

        offsets.push(message.push);

        resolveOffset(message.offset);
        await heartbeat();
      }

      // Handler for draining queue
      // Resumes the consumer to fetch next batch only when concurrentQueue is empty
      concurrentQueue.drain(function () {
        commitOffsetsIfNecessary(offsets);
        console.log("🟢 Resuming consumption for next batch");
        consumer.resume([{ topic: config.kafka.topic }]);
      });
    },
  });

  process.on("drainQueue", concurrentQueue.drain);
};

We are now fetching a batch from Kafka and pushing it to the queue. The queue makes sure only a predetermined number of events get processed concurrently.

Putting It All Together

With our updated pub-sub setup, let's analyze how the events are processed. We're producing a total of 10 events where the batch size is set to 1 KB and queue concurrency is 2.

initConsumer: async (mongoClient) => {
  const concurrentQueue = async.queue(async (queueData) => {
    const data = JSON.parse(queueData.value);
    console.time(`${data.hash} took`);
    console.log('processing in queue: ', data);

    // AppSignal event capturing, can comment if throttling has unexpected behaviour
    tracer.startActiveSpan('insert | kafka event | SUCCEEDED', async (span) => {
      const Item = mongoClient.collection('items');

      // Send query data to AppSignal
      setCategory('query.mongodb');
      setBody(`Item.insertOne()`);
      setTag('operationType', 'insert');

      // Save event data to DB
      await Item.insertOne(data);

      console.timeEnd(`${data.hash} took`);

      span.end();
    });
  }, config.kafka.queueConcurrency);

  const consumer = KafkaJS.consumer({
    groupId: config.kafka.consumer,
    topic: config.kafka.topic,
    maxBytesPerPartition: config.kafka.maxBytesPerPartition,
    maxBytes: config.kafka.maxBytes,
    allowAutoTopicCreation: true,
  });

  // Connect to the consumer group
  await consumer.connect();

  consumer.on('consumer.commit_offsets', (event) => {
    console.info('✅ offset is committed');
  });

  // Subscribe to the topic
  await consumer.subscribe({ topic: config.kafka.topic, fromBeginning: false });

  // Run the consumer
  await consumer.run({
    autoCommit: false, // This is turned off since we're doing manual commits
    eachBatchAutoResolve: false,
    eachBatch: async ({
      batch,
      resolveOffset,
      heartbeat,
      commitOffsetsIfNecessary,
      uncommittedOffsets,
      isRunning,
      isStale,
      pause,
      resume,
    }) => {
      console.log('🛑 Pausing consumer group');
      pause([{ topic: config.kafka.topic }]);

      let offsets=[];
      console.log(`⬇️ Fetched ${batch.messages.length} messages from batch`);
      for (let message of batch.messages) {
        if (!isRunning() || isStale()) break;

        // Process the event
        concurrentQueue.push(message);

        offsets.push(message.push);

        resolveOffset(message.offset)
        await heartbeat();
      }

      // Handler for draining queue
      // Resumes the consumer to fetch next batch only when concurrentQueue is empty
      concurrentQueue.drain(function() {
        commitOffsetsIfNecessary(offsets);
        console.log('🟢 Resuming consumption for next batch');
        consumer.resume([{ topic: config.kafka.topic }]);
      });
    },
  });


  process.on('drainQueue', concurrentQueue.drain);
},


getProducer: async () => {
  const producer = KafkaJS.producer({
    allowAutoTopicCreation: false
  });

  producer.on('producer.connect', (event) => {
    console.info('Kafkajs producer connection', { extra: event });
  });
  producer.on('producer.disconnect', () => {
    console.error(new Error('Kafkajs producer disconnected'));
  });
  await producer.connect();
  return producer;
}

Here's the output of the above code:

Note that we produced 10 events, but we're only focusing on one of the fetch cycles that pulled 7 events. We can confirm a few things by studying the above output:

Batched consumption is working as expected. Consumption is paused after a batch is fetched and is only resumed once the batch is completely processed.
Our second throttling layer (i.e., queue) only processes two events at once. As soon as one of the two is successfully processed, another event is picked from the fetched batch for processing.

Real-World Kafka for Node Applications

Kafka pub-sub is widely used for a variety of use cases across the industry, such as:

In an e-commerce store, to update inventory and send stock refresh notifications to users.
To send user events to a post-processor for lazy insertions into an analytics database.
To build an audit trail pipeline where any modifications to an entity will be pushed to a topic and consumed by an audit service.

This is one of the reasons that companies like Uber, LinkedIn, Twitter, Netflix, PayPal, and many more use Apache Kafka.

Wrapping Up

In this post, we explored how to batch Kafka events with manual pause-resume functionality to build a scalable Kafka pub-sub pipeline. We talked about autocommit and how keeping it on isn't good for "exactly once" semantics. We also added a queue for granular control over a batch of events and consumed it at our own pace.

This mechanism of throttling Kafka enables your Node.js application to handle incoming load more gracefully and prevents choking of resources, thus promoting scalability. There's no one-size-fits-all solution; you can always customize the pipeline according to your needs and goals.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

How to Optimize MongoDB Performance for Node.js

Rishabh Rawat — Wed, 22 Nov 2023 12:49:55 +0000

To update a document in MongoDB, I used to fetch it, update the values, and save back the entry. I would question the need for an update method. Looking back, it's evident that performance optimizations were hardly a concern when working on a personal project.

Working with a larger dataset is a whole different story, though. This is where no-code tools can't help. In this article, I'll share some of my learnings when it comes to working in MongoDB with millions of documents. We'll see how best to optimize performance.

Let's get started!

What We'll Cover

Here's what we'll go through in this post:

Data modeling
Starting with explain
Provisioning your hardware resources
Monitoring MongoDB queries
Managing indexes
Limiting query output
Using projection
Connection pooling with Mongoose
Batch operations
Soft deletes
Caching layer for faster access
Aggregation best practices

Data Modelling in MongoDB

When designing the data layer, it's absolutely crucial to know the end goals of a product. These are some of the crucial questions you must ask yourself:

Do you want to show all the items on a single page?
Does owner info need to be present with the item description? Would the owner name suffice?
Can owners update their profile details? How often? Once a day/week/month?

Why? Because answering questions like these helps you chart relationships between your entities (in this case, owner and item). That's why drawing an Entity Relationship Diagram helps.

For example, if owners can edit their profile details, embedding them in every item they list might not be the right choice (as you'll need to update their profile details for every item they have previously listed). Whereas if items reference the owner document, you only need to update the owner details once.

This is where embedding vs. referencing comes in:

Embedding is when you take the complete owner object and put it inside an item object. This means the owner object is added to every item someone creates. This is data duplicity (or, more formally, denormalization).
Referencing means just adding a unique identifier in the item's object for the owner, solving the data duplicity issue.

To summarize, the pros and cons of the embedding approach are as follows:

Pros:

Faster read
Atomic behavior (during writes)

Cons:

Data duplication
Limit on max document size (16MB)

Now let's turn to analyzing queries using explain in MongoDB.

Starting with `explain`

MongoDB provides a very neat way to analyze your queries with explain. You can learn a lot about how your query performs by looking at the query plan and execution stats. We have a sample query on MongoDB playground:

db.collection.aggregate([
  {
    "$project": {
      "_id": 0
    }
  }
]).explain("executionStats")

Let's understand the explain output:

{
  "stages": [
    {
      "$cursor": {
        "executionStats": {
          "executionStages": {
            "docsExamined": 5,
            "executionTimeMillisEstimate": 0,
            "nReturned": 5,
            "stage": "COLLSCAN",
          },
          "executionSuccess": true,
          "executionTimeMillis": 0,
          "nReturned": 5,
          "totalDocsExamined": 5,
          "totalKeysExamined": 0
        },
        "queryPlanner": {
          "rejectedPlans": [],
          "winningPlan": {
            "direction": "forward",
            "stage": "COLLSCAN"
          }
        }
      },
      "executionTimeMillisEstimate": NumberLong(0),
      "nReturned": NumberLong(5)
    },
    {
      "$project": {
        "_id": false
      },
      "executionTimeMillisEstimate": NumberLong(0),
      "nReturned": NumberLong(5)
    }
  ]
}

I've removed most of the output for brevity, as we're only interested in these things:

queryPlanner.winningPlan.stage: This indicates COLLSCAN, i.e., a collection scan.
executionStats.nReturned: Shows the number of documents returned in the output.
executionStats.totalKeysExamined: Since the query doesn't use an index, this value is 0.
executionStats.totalDocumentsExamined: This shows the total number of documents that the query has to go through.
executionStats.executionTimeMillis: This is the time (in ms) the query took.
executionStats.executionStages: In the case of a complex query, there might be multiple stages involved. This shows all of them.

To summarize, running explain on your queries tells you whether an index is being used, all the execution stages, and the amount of time your query takes.

Provisioning your Hardware Resources

As the common (unspoken) rule in the distributed systems world goes — always provision your hardware resources with a sufficient buffer. Is usage around 1.7 GB? Don't go for a t3.small EC2 instance with 2 GB of RAM. Going for neck-to-neck calculations often results in chaos.

Insufficient RAM, CPU, or I/O can lead to higher latencies, sluggish queries and operations, and even aggregation pipeline execution times can become unbearable.

To know the sweet spot for hardware, a good first step is:

Regularly monitor CPU, memory, disk I/O, and network during peak load times. Check if your database is capable of handling the load. If you use a tool like AppSignal, it automatically monitors host and memory metrics for you.
Run the mongostat command to view metrics such as connections, read/write per second, virtual memory, resident memory, network in/out, and much more.
Run MongoDB's built-in database profiler (it comes with an overhead) to collect stats about your queries and analyze the slow ones.
You can even go for an external monitoring solution — MongoDB OpsManager or Atlas, which adds nice visualizations to better understand the patterns.

But it's hard to improve something you can't observe. Time to monitor our queries.

Monitoring MongoDB Queries for Node.js

The goal of monitoring your MongoDB queries is to ensure you are the first to know when anomalies occur. A slow query not only degrades user experience but can also affect other clients (e.g., there might be no idle connections in a pool due to longer holds).

Logging does not help in observing query performance data. It's simply not the right tool for the job. In times like this, where you need a holistic view of your application performance, Application Performance Monitoring (APM) helps.

I've set up AppSignal for my Node.js API. And the best part is there are no extra steps needed to enable the MongoDB/Mongoose integration.

Let's use an example.

Here's a GET /item/:hash route that fetches an item by hash and returns it to the client:

const tracer = trace.getTracer("my-nodejs-app");

router.get("/:hash", async (req, res) => {
  try {
    const { hash } = req.params;

    setRootName("Fetch Item");

    tracer.startActiveSpan("find | Items | SUCCEEDED", async (span) => {
      setCategory("query.mongodb");
      setBody(`Item.findOne({ hash: ${hash}})`);
      setTag("operationType", "fetch");

      const Item = mongoClient.collection("items");
      const item = await Item.findOne({ hash });

      res.json({
        item,
        status: 200,
        message: "Item read successfully!",
      });

      span.end();
    });
  } catch (err) {
    res.json({
      item: null,
      status: err.code || err.statusCode || 500,
      message:
        err.message || "Something went wrong while reading item from DB!",
    });
  }
});

Please note that setting the category to query.mongodb is important because AppSignal detects MongoDB events by looking for the ".mongodb" suffix in the event name. You can read more about structuring event names in AppSignal's documentation.

Now we're all set. Slow queries will start appearing on our AppSignal dashboard:

With the help of the above stats, it's much easier to pinpoint the root cause and fix it.

Managing Indexes

Not indexing generally doesn't cause a bottleneck — over-indexing does. While you should ensure all necessary fields are indexed, you should also review indexes periodically.

Scattered data can result in index fragmentation, which, in turn, negatively impacts your query performance. I'd suggest re-indexing your collections during scheduled maintenance to ensure they are always up-to-date. Here's how:

db.itemsCollection.reIndex();

You can analyze your queries using explain to understand your query in-depth and add/update the corresponding index appropriately:

db.itemsCollection.find({ price: { $gt: 10 }}).explain("executionStats")

Don't hesitate to remove indexes that are stale and not frequently used. The cost of building an index should not scare you into running suboptimal queries.

In the next section, we'll dive into how best to limit query output.

Limiting Query Output

While ignoring query output for a database with a small dataset might not hurt, limiting query output becomes essential for a bigger dataset. Showing a million results on a single list is not only unproductive, it may even crash the UI. Performing searches or operating filters becomes a nightmare.

It is completely up to you and your team to decide when to implement pagination. While everyone starts with a small dataset, the question is: do you want to control the beast now or live in the present?

Putting a limit on your MongoDB queries is simple, yet has a big impact:

db.items.find({}).limit(10);

Now, let's see how we can specify the fields we want to see in our output.

Using Projection in MongoDB

Common data fetching methods in MongoDB like find() and findOne() allow you to specify the fields you want in the output. This is called projection, and you can specify the fields you want like this:

db.items.find({}, { name: 1 }).limit(10);

This way, only the names of all ten items will be returned by the query. Alternatively, you can do negative projection, i.e., specify the fields you do not want in the output. For instance, you can remove _id from the output like this:

db.items.find({}, { _id: 0}).limit(10);

Projection is also available in the MongoDB aggregation pipeline. You can add the project stage in your pipeline like this:

db.collection.aggregate([
  {
    "$project": {
      rating: {
        "$round": "$rating"
      },
      "_id": 0
    }
  }
])

Try it out yourself in the Mongo playground.

Now we'll turn to using Mongoose for pooling connections.

Connection Pooling with Mongoose

Pooling database connections helps skip the cold start problem of setting up a standalone connection every time. You specify the number of pooled connections and pick an already established connection from the pool of idle connections whenever you need it.

Using a connection pool is an easy performance improvement, not only for databases but when making network calls as well. For example, Axios provides connection pool support out of the box.

With Mongoose, creating a connection pool is simple:

mongoose.createConnection(uri, { maxPoolSize: 10 });

Just specify the maxPoolSize in the options object, and you're done. But, you might ask, what's the upper limit? There isn't a formal one, not that I could find. But increasing your maxPoolSize to 10k might not be a good idea.

With more connections, you're spreading your resources thin, and connections are practically useless since allocated resources can't perform any operations (e.g., insufficient RAM causes sort to fail or perform slowly).

Moving on: let's see how batch operations can improve performance in MongoDB.

Batch Operations

To perform write operations on every request, a dedicated network round trip is involved. Your database takes a hit when you have thousands of such requests (bursty traffic patterns).

Batching write operations helps you avoid those redundant network round trips and, therefore, increases throughput and performance. Let's take a look at an example using bulkWrite() from the MongoDB documentation:

const { MongoClient } = require("mongodb");

// Replace the uri string with your MongoDB deployment's connection string.
const uri = "<connection string uri>";

const client = new MongoClient(uri);

async function run() {
  try {
    const database = client.db("sample_mflix");
    const theaters = database.collection("theaters");

    const result = await theaters.bulkWrite([
      {
        insertOne: {
          document: {
            location: {
              address: {
                street1: "3 Main St.",
                city: "Anchorage",
                state: "AK",
                zipcode: "99501",
              },
            },
          },
        },
      },
      {
        insertOne: {
          document: {
            location: {
              address: {
                street1: "75 Penn Plaza",
                city: "New York",
                state: "NY",
                zipcode: "10001",
              },
            },
          },
        },
      },
      {
        updateMany: {
          filter: { "location.address.zipcode": "44011" },
          update: { $set: { is_in_ohio: true } },
          upsert: true,
        },
      },
      {
        deleteOne: { filter: { "location.address.street1": "221b Baker St" } },
      },
    ]);

    console.log(result);
  } finally {
    await client.close();
  }
}
run().catch(console.dir);

Just like the aggregation pipeline, you can pass various operations to the bulkWrite method. MongoDB performs each of the operations sequentially and doesn't perform the remaining operations in case any operation fails (by default). However, you can override this behavior by passing the ordered: false option to the bulkWrite method.

Now onto soft deletes.

Soft Deletes

There are a couple of benefits to soft deletes. Permanent deletion makes it hard to recover from incidents if a rollback is required. This might not be an issue if you have backups or an audit trail that allows for point-in-time recovery (PITR). But recovery is not always possible due to cascading dependencies. However, restoring data in the case of a soft delete is as simple as flipping the is_deleted flag in the affected documents.

But as always, there's a trade-off. To implement soft delete, your collections need to have a is_deleted boolean field to indicate the state of the document. And it's easy to forget to add this condition (is_deleted: false) when you write a query.

To ensure your database size is not taken up by deleted documents, you can run a scheduled cleanup script to remove the documents (where is_deleted is true) from all of your collections.

Hard deletes leave their legacy in unpleasant ways, and recovery is not always possible. Here's the stardust we can learn from.

Next up, we'll take a quick look at caching.

Caching Layer for Faster Access

I'm sure we've all used caching at some point in our development endeavors, and rightfully so. We do not want to hit the database on every request. The biggest challenge of caching is invalidation. When you invalidate a cache can make all the difference. It's not about being 100% consistent: a little bit of inconsistency is also fine sometimes.

Whenever you design a caching layer, always keep the use case at the forefront. It'll often help you decide how to deal with invalidation. And if you're using a multi-threading environment, be mindful of false sharing.

You can use Redis to perform application-level caching — it even allows for a distributed cache. I'd recommend reading how the Redlock algorithm works.

Finally, we'll wrap up by exploring some aggregation best practices for MongoDB.

Aggregation Best Practices in MongoDB

If you've used aggregations in MongoDB, you know their importance. Aggregation pipelines help you get custom insights out of your data without manipulating data manually (which gets ugly and error-prone).

Building a performant aggregation pipeline is important, otherwise it's very likely to bring the whole database down. Just imagine doing a find operation without any filters.

There are a few things to keep in mind when building a MongoDB aggregation pipeline:

Try to reduce the sample space as early as possible in the aggregation pipeline (ideally in the first stage). This is because the output of a stage is passed onto the next stage. Reducing the sample space ensures less data is loaded onto the memory and it's easier to work with.
If you can, avoid allowDiskUse while using the group stage. Try to make sure your data is safely below memory limits. Pulling data onto a disk levies a significant performance penalty.
Be mindful of streaming vs. blocking stages in an aggregation pipeline. For example, group and sort are blocking stages: i.e., they require complete data to be available before they can start doing the work. In other words, they cannot leverage streaming, as a grouping and sorting operation requires all data to be present at once.

And that's it!

Wrapping Up

In this post, we've explored the most critical strategies available to optimize MongoDB performance. Irrespective of the use case, these strategies will ensure your database is never a bottleneck.

But there's one thing to remember: no optimization can cure bad schema design. Doing things well at the beginning saves you a lot of time down the road.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Tracking Errors in a Node.js Application

Rishabh Rawat — Wed, 06 Sep 2023 14:11:59 +0000

Production bugs slow down velocity and often affect the complete trajectory of your release roadmap. It helps if you have a robust error tracking setup to rely on.

In this article, we'll look at how to make tracking errors in your Node.js application more convenient, automated, and safe.

Let's begin!

The Two Error Types

Every application we build has an end goal. The code reflects that goal and allows the user to perform intended actions. Since the real world is complex, situations may arise that aren't handled well by an application. This leads to unexpected and unwanted outputs, defeating the end goal of that application. Diverging from the happy path is referred to as an error.

Errors can be put into two buckets:

Operational errors
Programming errors

Operational errors arise during the active use of your application. Also known as runtime errors, they help us find edge cases missed during development. For example, a user adds a negative age, gives negative tips on a food delivery app, or enters a malicious script in an input field, etc. These should be properly handled by developers.

Programming errors, on the other hand, arise when an error slips through due to a missing error handler in the code. Examples include:

Assigning a string to a number.
Not catching a rejected promise.
Accessing an object's non-existent properties.

What Is Error Tracking?

Error tracking is a process where the abnormal behavior of your application is observed and monitored with the goal of fixing it before it affects end users. Error tracking entails logging, monitoring, creating alerts, and anomaly detection based on historical usage patterns.

Proactively tracking errors and resolving them provides a smooth user experience and helps in building trust. Relying on your users to report errors is not only unreliable, but is bad for your business and brand.

How to Track Errors in Node.js

Tracking errors involves anticipating an error and writing code to handle unexpected/unwanted scenarios — in other words, situations when your application does not function how it is designed. It's the only (right) way to steer the correct course and ensure your application is on the right track.

There are various ways to identify errors in a Node.js application:

Callbacks
Promises
Try-Catch

We'll use try-catch throughout this article. The basic idea is to run a piece of code and wrap it in the try block, and the error handler goes in the catch block. If your application behaves normally, the catch block will never run.

Here's an example snippet, using try-catch to handle such situations:

try {
  operationThatMightFail();
} catch (err) {
  log.error(err);
  throw err;
}

Now that we've run through vanilla error handling, let's move on to the exciting part. The above error handler is a good place to start, but we can do better. As you might've noticed, it doesn't provide much value to us. Looking at that error message wouldn't necessarily help us during an incident. There's a better way. Let's explore that.

Creating a Custom Error Handler

We want every error to have sufficient context for it to be actionable. For example, if the stack trace of an error is overridden, debugging can take unreasonably long. Balancing rich error reports with concise syntax is key. Error handlers shouldn't distract us from the main application logic.

To tackle this, we can set educated defaults to avoid passing certain details in every error handler. Various common configs can be abstracted away, such as:

Passing error context to an APM (Application Performance Monitoring) tool
Logging the error
Setting custom flags that help during analysis and monitoring
Returning an HTTP status code to the client

For this, we can create a custom error class that extends Error and adds some nice defaults. Let's have a look at one:

const { sendError } = require("@appsignal/nodejs");
const { HTTP_STATUS_CODES } = require("../../constants");

class AppError extends Error {
  constructor(message, statusCode, isOperational = true) {
    super(message);

    this.name = "AppError";
    this.statusCode = statusCode || HTTP_STATUS_CODES.INTERNAL_SERVER_ERROR;
    this.isOperational = isOperational;
    Error.captureStackTrace(this);

    // send to APM
    sendError(this);
  }
}

Now we can technically go ahead and plug this custom error handler into our code everywhere. It'll work. But it's not required. We will only use it in our controller layer. Any errors that happen in services/utils/helpers/etc., will be reported to the error handler at the controller layer.

This leverages event bubbling and avoids redundant error handlers. Moreover, it helps to avoid overriding error stack traces by accident.

Better Error Messages in Your Node.js App

While a "something went wrong" error message might be sufficient for your end users (hint: it's not), it doesn't help developers working on your application.

As a developer, you want an error message to at least give you a semblance of the root cause(s), if not to paint the complete picture. Again, the ultimate goal is to make your errors actionable.

Here are some best practices for writing error messages:

Make them concise
Avoid ambiguity
Avoid acronyms/slang/technical jargon
Make them human-readable
Make sure they are actionable and help the developer in finding the root cause
Avoid duplicate error messages

Some examples of a good error message can be:

"Instance does not have SES:sendEmail permission".
"Failed to establish database connection".
"Failed to upload the file, size is greater than 10 MB".

Handling Programming Errors

We can now track all sorts of operational errors with the help of our newly created custom error class. It's time to handle programming errors.

Two such errors are uncaughtException and unhandledRejection.

process.on("uncaughtException", (err) => {
  logger.fatal(`uncaughtException error has occurred: ${err}`);
  process.exit(1);
});

process.on("unhandledRejection", (err) => {
  logger.fatal(`uncaughtRejection error has occurred: ${err}`);
  process.exit(1);
});

These handlers ensure that even the errors with no handling get tracked and get your much-needed attention. These errors must be acted upon with urgency (P0), because if they're overlooked, your service may go into an unexpected and less predictable state.

How to Choose Log Levels for Your Errors in Node.js

Our application looks much more resilient now. This section is a refresher and a general guide to help you choose log levels for your (error) logs. Every log level indicates a different purpose.

Trace

"I'm here" kind of logs. With trace logs, you generally don't want to track the state of the variables but want to know the control flow for a particular request.

Debug

Debug logs go one step further than trace logs: you are keen on tracking the state of the variables. For example, authenticated user details, params being passed to services, flags set in database queries, etc.

Info

These logs are for general info which doesn't require immediate action or attention. They do not represent an unwanted code state and so can be ignored under normal circumstances. Some examples of info logs can be loaders initializing successfully, successful events ingestion and drain logs, etc.

Warn

These are not errors but unwanted states. The warn log level can be used if it's okay for a request to fail due to an inappropriate payload. This is an unwanted state, but an expected unwanted state. For example, a request failure log due to an idempotency lock is an expected error.

Error

Contrary to warnings, error logs indicate an unexpected, unwanted code state. But a service can continue to function in case of such errors. For example, if a request fails due to the service host temporarily not being reachable, it's an error log. Error logs are actionable.

Fatal

These take the P0 priority. Fatal errors must be acknowledged and acted upon immediately as they (usually) bring a service down.

But Wait, What Errors Should I Track Exactly?

There's one important rule when logging errors: only include essential information. As much as it's important to provide sufficient context, it is also a developer's responsibility to avoid noise in logs. Why?

Every log needs to be stored on disk, and it has a cost. Processing logs requires disk IOPS, which can easily become a bottleneck if your service gets under bursty or high sustained load. The same IOPS quota is used to serve requests as well.

Here are some of the important things you should track:

Request metadata
Request payload (if too big, a key that can uniquely identify the payload)
Success/failures at checkpoints (with necessary context)
Final response

Restarting Gracefully When Needed

There are situations when a critical error hits, and there's no other option but to shut down and restart the application (e.g., your application's memory crosses its threshold).

A direct (and potentially dangerous way) to do this is to directly process.exit(1) and get moving with a fresh restart. But that can lead to issues if your storage layer or any other stateful component was left in an awkward situation.

The better way is to restart gracefully by closing all the open loops. Restart the application only when every component is ready.

We'll listen to SIGTERM and SIGINT on the process and call our utility to stop the server gracefully:

process.on("SIGTERM", (signal) => loaders.close(server));
process.on("SIGINT", (signal) => loaders.close(server));

And here's the loader module:

module.exports = {
  run: () => {
    return new Promise((resolve, reject) => {
      const mongoClient = mongoose.getConnection();
      // other dependencies init goes here...

      resolve({ mongoClient });
    });
  },
  // runs before app crash/shutdown
  close: async (server) => {
    if (!server) {
      logger.warn("Cannot stop server, it's not running");
      process.exit(0);
    }

    // Fetching DB conn from loader as it is idempotent
    const { mongoClient } = await module.exports.run();

    server.close(async (err) => {
      // closing connection(s)
      await mongoClient.close();

      if (err) process.exit(1);
      process.exit(0);
    });
  },
};

Based on your specific use case, it may be okay not to exit the process. Also, process managers like PM2 allow automatic restarts if your application crashes.

Learning from Your Node.js Errors with AppSignal

Looking at your errors holistically gives a better perspective and helps you observe repetitive behaviors in your code that otherwise may not catch your attention.

AppSignal makes error tracking proactive and convenient, and you can sign up for a 30-day free trial (no credit card details required).

For example, I can see a cluster of errors on this graph. I can simply zoom in and check if those errors correlate to the release that went out that day / a vendor outage / anything else that might help me get to the root cause and decide on further actions.

You can even set up alerts to notify you if any anomalies are detected:

AppSignal for Node.js also offers logging, host and uptime monitoring, and performance metrics.

Best Practices to Reduce Errors in Node.js

While tracking and monitoring errors is good, it's best to prevent them as much as possible.

Here are some best practices that can eliminate trivial errors:

Use static type checking to catch trivial errors during the compilation stage (e.g., TypeError: Cannot assign number to string).
Add pre-commit hooks and tests in your CI level to ensure breaking builds don't go out.
Load test your services to observe how your application behaves under heavy load. There might be no errors on a normal load, but you may observe app memory spikes under bursty traffic patterns.
Regularly review and remove logs that don't serve any specific purpose and add to the noise. This also saves disk space and helps in case of a limited/small IOPS.

Wrapping Up

In this article, we first defined error tracking and looked at the different types of errors. We then explored how to track your Node.js errors in the right way, including creating a custom error handler. We also saw how to avoid tracking trivial errors to reduce noise in your alerts.

Hopefully, you learned something that you can apply to your projects right now. You may also find our Node.js Error Handling: Tips and Tricks post useful.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Complete Guide to Multi-Provider OAuth 2 Authorization in Node.js

Rishabh Rawat — Sun, 15 May 2022 17:39:27 +0000

OAuth 2 authorization makes the user authentication journey very seamless. It enhances the user experience, minimizes the attack surface, and encourages a definite & limited authorization model.

In this guide, we will take a look at how you can build a complete OAuth 2 authorization workflow in a nodejs application using Passportjs. We will be focusing on the back-end in this guide i.e., Nodejs, MongoDB, and Passportjs.

There are three major goals, divided into checkpoints:

With that said, you are expected to have a working node.js application using MongoDB as a database. It won't cover starting a project from scratch.

In case you need a basic application running, you can clone the repository from here.

This guide contains the code snippets, a link to a Github branch, and a demo for each checkpoint. So you can expect to follow along and implement alongside.

How it is structured

This is a step-by-step guide to implementing an OAuth 2 authorization in your nodejs application using multiple providers (Google, Github, Amazon).

It provides the ability to cross-sync multiple social accounts so that you can log in using any one of them.

As a third and final checkpoint, you will learn how to support multiple Google logged-in accounts. This is very similar to what Gmail offers and allows you to switch accounts without having to authenticate every time you switch.

Here’s how it is structured:

Implementing OAuth 2.0 authorization.
Adding the ability to cross-sync multiple providers.
Extending the code to allow adding multiple Google accounts.

This is going to be a comprehensive guide covering the steps as well as troubleshooting the roadblock(s) that come along the way. Feel free to go through different sections to scope things out.

OAuth 2 Overview

If you're starting today, don't use OAuth 1. It had a lot of issues (limit on providers, hard to scale, etc.) and is deprecated now.

OAuth 2 is designed to provide authorization with delegated authentication. OAuth 2 does not provide a user authentication mechanism, by design.

Here's a quick recap on Authentication vs. Authorization:

Authentication makes sure a user is who they're claiming to be.

Whereas Authorization governs what the user has access to.

An OAuth 2 application delegates the authentication to services that host a user account and asks for (limited) authorization from those services, after the user has given consent.

To understand with the help of an example, it is like informing Google (through user consent) that it is okay for Todoist to access your Google profile information and update your Google Calendar on your behalf.

Here is the step-by-step breakdown of how the OAuth 2 authorization flow works:

User wants to use Todoist by signing in to Google.

Todoist acknowledges the user's request and shows an authorization request (or a consent screen).
User gives the consent and the consumer (Todoist) receives an authorization code from Google. It is a way to identify which consumer was authorized.
Consumer then goes to the authorization server (or Google) with the authorization code.
Authorization server recognizes the valid authorization code and gives an access token to the consumer application.
Consumer requests access to user resources using the access token.
The consumer application successfully receives the authorization to access user resources (in this case, Google calendar's read + write access).

The benefit? Todoist never gets to know your Google password. Thus, you're safe in case Todoist suffers a security breach.

We used the authorization code implementation of the OAuth 2. But there are other ways to implement it also.

And yes, there are trade-offs here too. For instance, you would need a separate integration (in the case of Passportjs, a different strategy) for each social provider you plan to support in your application.

I hope this gave you a general overview of how the OAuth 2 authorization works.

The theory's over. Let's move on to the next step 👇🏻.

Create API keys for all providers

Before we start working on our backend API, let’s create the credentials for the providers that we want to support. This will avoid context switches when we get to the implementation.

Google

Visit the credentials page.

Use the already selected project or create a new one.
Visit the Consent screen page and fill in the required details. For our use case, here’s what we will do:
1. Select user type to be external (if asked).
2. App name can be the same as our project’s name, i.e., nodejs-social-auth-starter.
3. Enter your email in support email and developer contact email inputs.
4. Click “save & continue”.
5. Next, it asks for scopes. Enter profile and email. Again, save and continue.
6. Review everything and proceed.
Create a new OAuth Client ID.
1. Select the application type to be “Web Application”.
2. Most importantly, we will fill the “Authorized redirect URIs” to be http://localhost:3001/api/auth/google/callback.
Save and proceed. You will find the newly created OAuth client ID on the crendentials page.

Github

For Github, head over to your Settings > Developer Settings > OAuth apps and create a new app.

NOTE: You will need to also create the client secret manually after following the above instructions.

Amazon

Visit Amazon developers console.
Create a new security profile.
Note down the OAuth2 credentials in your .env file.
Go to your newly created profile's web settings:

Fill the Allowed Origins and Allowed Return URLs fields.

Setting up starter application

Throughout the article, we will be working with a sample project which can you find here.

We are using Expressjs for the backend server, MongoDB as a storage layer, and Passportjs for implementing the OAuth 2 authentication in our application.

To follow along, make sure to do the following:

Clone the repo:
Install the dependencies using npm install

That's it! You should be able to run the server by running the command npm start.

There are several branches in the repository:

base: Starter project setup; choose this to start from scratch
basic-oauth: Contains basic passport OAuth implementation
main: Basic OAuth2 + allows cross-sync between providers
multiple-google-accounts: Basic OAuth2 + contains the multiple logged-in Google accounts feature

You can choose to start from scratch (basic express server setup). Feel free to check out different branches to see different states of the code.

To make it easier to follow along, the base branch contains the commented out changes of basic-oauth branch. So you can go through the first section of this guide and progressively uncomment code snippets to see them in action.

User model

Before jumping into the implementation, let's understand the fields in our User schema and why we need them.

NOTE: Only a single MongoDB document can exist for a given email i.e., one email cannot be a part of more than one account.

Here's our User schema:

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

// Schema to store the information about other logged in accounts
const accountSchema = new Schema({
  name: String,
  userId: String,
  email: String
});

// create User Schema
var UserSchema = new Schema({
  name: String,
  connectedSocialAccounts: {
    type: Number,
    default: 1
  },
  otherAccounts: [accountSchema],
  google: {
    accessToken: String,
    email: String,
    profileId: String,
  },
  github: {
    accessToken: String,
    email: String,
    profileId: String,
  },
  amazon: {
    accessToken: String,
    email: String,
    profileId: String,
  }
});

const User = mongoose.model('users', UserSchema);
module.exports = User;

We have dedicated fields for all the social providers to store their access token, profile Id, and email. Additionally, we have two special fields:

otherAccounts: It stores all the other accounts user has logged in from.
connectedSocialAccounts: It is a count of providers synced to the logged-in account.

We don't need to worry about these fields for now. We will cover them in great detail in the later section.

Okay, enough theory. Let's start coding 🚀.

Configure Passportjs

Passportjs is authentication middleware for Node.js and it is very modular (has ~500 authentication strategies) and flexible (complete control over how the authentication flow works). Another great thing I liked about Passportjs is that once logged in, it populates the request.user with the user details (provides serialize & deserialize functions for flexibility).

We will work with Google, Amazon, and GitHub APIs in this article. You can go ahead and add more strategies to your application if you like.

To configure Passportjs, we need to set up a session store, initialize Passportjs & its sessions, and use express-session to store the cookie in our session store.

Let's go through them one by one:

Setting up session store

We'll be using connect-mongo as our session storage layer.

npm install connect-mongo

Finished installing? Awesome! Let's set up our mongo session store.

const MongoStore = require('connect-mongo');
const { databaseURL, databaseName } = require('@config');

module.exports = {
  run: () => MongoStore.create({
    mongoUrl: databaseURL,
    dbName: databaseName,
    stringify: false,
    autoRemove: 'interval',
    autoRemoveInterval: 1 // In minutes
  })
};

Finally, make sure to run this loader. In our case, we include this in our main loader file which runs on application startup:

const mongooseLoader = require('./mongoose');
const expressLoader = require('./express');
const passportLoader = require('./passport');
const sessionStore = require('./sessionStore');

module.exports = {
  run: async ({ expressApp }) => {
    const db = await mongooseLoader.run();
    console.log('✌️ DB loaded and connected!');

    const mongoSessionStore = sessionStore.run();

    await expressLoader.run({ app: expressApp, db, mongoSessionStore });
    console.log('✌️ Express loaded');

    passportLoader.run();
  }
}

Install and configure the express-session package

Passportjs is just a middleware for Expressjs applications. Hence it does not have any storage layer to store the user sessions. For that reason, we need to use a separate storage solution for our user sessions.

There are two options:

Cookie session package - cookie contains all the user session details
Express session package - cookie only contains the session ID, session data is stored in the backend.

We will go with the second approach as that is more secure.

express-session provides a lot of options for session stores. While the default is a memory store, we’ll be using a mongo store for better security, scalability, and reliability of data.

Why MongoDB for the session store? Because we are already using it for our application data.

Let's install the express-session package first:

npm install express-session

Once installed, we need to configure this in our express server:

app.use(expressSession({
  name: cookieName,
  secret: 'keyboard cat',
  resave: false,
  saveUninitialized: false,
  unset: 'destroy',
  cookie: {
    httpOnly: false,
    maxAge: 300000, // 5 min
  },
  store: mongoSessionStore
}));

Now that we have the sessions middleware in place, we don’t need to care about storing sessions.

The next step is to set up Passportjs and enable sessions 🔑.

Initialise passport and enable passport sessions

Let's quickly install the package first:

npm install passport

Two steps to complete the initial setup:

Initializing passport and sessions
Inject serialize and deserialize middleware in our express loader

The first step is a plug & play mechanism to enable Passportjs for our application. And the second step allows us to tell Passportjs what we want to put in the user session and consequently in request.user.

Initializing is quick, just put these lines after the express-session middleware in the express loader:

// Enable passport authentication, session and plug strategies
app.use(passport.initialize());
app.use(passport.session());

That was fast! Here’s the basic serialize & deserialize middleware we’ll put in our express server:

    passport.serializeUser(function(user, done) {
      process.nextTick(function () {
        done(null, user._id);
      });
    });
    passport.deserializeUser(function(id, done) {
      process.nextTick(function () {
        User.findById(id, function(err, user){
            if(!err) done(null, user);
            else done(err, null);
          });
      });
    });

Serialize function tells Passportjs what to store inside the user sessions. Deserialize function attaches the result to the request.user.

Since we want the complete user object to be present in request.user, we find the user document using the userId stored in the session. Alternatively, we can choose to store the complete user object in the session too. That way, we won't have to perform a database query in our deserialize function.

We are going ahead with the above approach because it makes switching accounts easier. This will become more clear when we perform hot reloading of our user sessions in the third section of this guide.

If you're still unclear on serialize and deserialize functions, you can check out this visualization for a better understanding. Worth checking out.

That's it! We're done with the basic Passportjs setup 🎊.

Adding Google OAuth login

Now that we have all project setup and dependencies installed, we are now ready to look at the authentication using Google OAuth.

To set up Google's OAuth2 authentication using Passportjs, we need to follow these steps:

Create a Passportjs strategy for the provider (eg. Google)
Add the authentication routes for the provider
Add a middleware to check for authentication
Adding the logout functionality

Let's implement Google OAuth2.

Create a passport strategy for Google

We need a passport strategy for every provider we add to our application. A strategy includes our OAuth2 API credentials for the provider, some custom options, and a verify function.

Credentials are given to the applications that are registered at Google's developer console. Verify function is where developers can provide the logic of how they want to identify users, preprocess the data, perform validations and create database entries.

Passportjs also provides documentation for nearly every strategy. We will follow the documentation for Google OAuth2 strategy in this section.

Let’s look at our basic passport strategy for Google:

const passport = require('passport');
const GoogleStrategy = require('passport-google-oauth20').Strategy;

const User = require('@models/user');
const config = require('@config');
const { default: mongoose } = require('mongoose');
const mongoSessionStore = require('../../loaders/sessionStore').run();

passport.use(new GoogleStrategy({
  clientID: config.googleClientId,
  clientSecret: config.googleClientSecret,
  callbackURL: config.googleCallbackUrl,
  scope: ['profile', 'email'],
  passReqToCallback: true,
  },
  async function(req, accessToken, refreshToken, profile, done) {
    try {
      const email = profile['_json']['email'];
      if(!email) return done(new Error('Failed to receive email from Google. Please try again :('));

      const user = await User.findOne({ 'email': email });

      if (user) {
        return done(null, user);
      }
      const newUser = await User.create({
        name: profile.displayName,
        profileId: profile.id,
        email: email,
        accessToken,
      });
      return done(null, newUser);
    } catch (verifyErr) {
      done(verifyErr);
    }
  }
));

module.exports = passport;

We pass two parameters to our Google strategy:

The options object - it contains credentials, scope, and passReqToCallback setting which makes the request object available in the verify callback function.
Verify callback function as the second parameter. This is where you can customize the logic based on your needs and build custom logging journeys.

This Google strategy will definitely evolve when we extend the functionality later in the article. But for now, this strategy helps us create new users in the database if they don’t exist. And we return the user object in the callback. Short and sweet.

Where does this callback send the data we pass? To Passport's serialize and then deserialize function. Serialize function attaches the user Id to request.session.passport.user. The deserialize function fetches & stores the user object in request.user.

🚧 [Roadblock] Patching node-oauth to workaround Google APIs

While working on the project, you might experience a roadblock with the Google OAuth2 strategy.

Google API sometimes closes the connection early causing the node-oauth callback to immediately get invoked, which is fine. But when the Google servers perform the connection reset, it goes into the error callback and node-oauth calls the callback again which leads to InternalOAuthError.

This is a known issue and there is a comment in the code highlighting this.

The impact? OAuth flow might not work for Google. But there's a workaround 💡.

You need to make a slight change in the error callback in your node-modules/node-oauth/ package to skip invoking the callback if it is already invoked once.

request.on('error', function(e) {
+    if (callbackCalled) { return }
     callbackCalled = true;
     callback(e);
});

To make sure this patch gets on to the remote repository, you can use the patch-package to modify node-oauth's code.

This was a solid ~4 hours journey for me, I hope this workaround helped you avoid it.

Add authentication routes for Google

Looking at the documentation, we need two routes:

First starts the authentication flow by redirecting the user to the consent screen.
Google provides an auth code once the consent has been given by the user. We need the second route to handle that redirection and complete the auth flow.

This is a quick one, we will add these routes to our auth routes module (/api/auth/...):

router
  .route('/google/callback')
  .get(passportGoogle.authenticate('google', { failureRedirect: '/login', successReturnToOrRedirect: '/' }));

router
  .route('/google')
  .get(passportGoogle.authenticate('google'));

NOTE: I prefer the above way of defining routes as it gives the flexibility of attaching multiple controllers for different HTTP verbs for the same exact route.

And we're done with the routes. Time for our authentication check middleware 👇🏻.

Add authentication middleware for protected routes

Passportjs attaches the .isAuthenticated() method to the request object which allows us to conveniently check if the user is logged in.

Here's our middleware:

function ensureAuthenticated(req, res, next) {
  if (req.isAuthenticated()) {
    return next(); // user is logged in
  }
  res.redirect('/login');
}

We can also use req.user to check for a logged-in user.

Adding the logout functionality

The project's front-end has a logout button but we haven't handled it on the backend yet. To log out a user, we need to expire the user session and the session cookie on the client-side.

Once it is done, we will redirect the user to the login page (/login; handled by our front-end app).

router
  .route('/logout')
  .get(function(req, res, next) {
    req.session.destroy(function(err) {
      if(err) return res.redirect('/');
      res.clearCookie('sid');
      res.redirect('/login');
    });
  });

express-session gives us a method to destroy the session which is an extended version of (req.logout()). While req.logout() only removes the user information from the session, the destroy method deletes the whole session document altogether.

Once the session is deleted, we remove the cookie from the client-side and redirect the user back to the login page.

Users can't access the protected routes (routes behind the authentication check middleware) even if they directly enter the URL in the address bar and hit ENTER.

You can follow the exact same steps to implement OAuth authentication for other providers. Feel free to check out this branch for the implementation for other providers.

Authentication milestone achieved 🥁🥁🥁

Woah! If you're following along, you surely deserve this:

We have supported:

Login using Google OAuth 2 flow using Passportjs,
Authentication check middleware to deny accessing protected routes anonymously, and
Logout functionality

📺 Here's the walkthrough of what we have built till now:

Let's keep the flow going and move on to our next section, which is, adding the ability to cross-sync providers.

Implementing cross-sync for social providers

Welcome to the second section of this guide where you will learn how to implement cross-sync functionality for different social OAuth providers (Google, Github, and Amazon).

Why implement such a feature? TL;DR: Better UX ✨.

There can be several reasons a user might want to have multiple social accounts linked to your website. They might've lost control over one of their social accounts, forgotten their password, or simply don't want to share a specific email address to prevent bloat & spam on that address.

Whatever the reason might be, users always love to have the ability to login into your website using any one of their social accounts (Google, Facebook, Twitter, Instagram, and Github are some examples).

Who uses it? There are a lot of real-world products that use this feature, albeit calling it something else.

Todoist uses it, for instance. If you are a Todoist user, you can find it in your account settings:

We want to achieve the same thing with our application i,e., to allow users to log in using any one of their connected accounts. If you have connected your Google and Github accounts to the application, you should be able to log in to your account using anyone of them.

There are four things to keep in mind to implement this:

How the user will connect/disconnect the providers?
How to connect different providers to a single user account?
How to make sure that the user doesn't disconnect all of the connected providers from their account?
Show the status of connected and disconnected (or yet-to-connect) providers on the UI.

Let's understand and find an answer to these questions 💡.

Routes for connecting and disconnecting providers

We can use the same route for connecting a new provider that we use for Google OAuth login. This is possible because the verify function in Google's passport strategy is flexible (remember from the first section?).

We can tweak the logic inside the verify function based on the requirements. This is such a powerful feature and it also saves one additional route for connecting (or linking) a new provider.

To disconnect or unlink a provider from the user account, we would need a dedicated route. This route will delete all the provider data from the user document in MongoDB.

Let's take a look.

router.get('/google/disconnect', async (req, res) => {
  if(req.user.connectedSocialAccounts > 1) {
    await disconnectGoogle(req.user);
  }
  res.redirect('/');
});

Making the request to /api/auth/google/disconnect invokes our disconnectGoogle handler (in src/services/user/index.js) which removes all the Google-specific data from the user document.

async function disconnectGoogle (user) {
  if (!user || !user.google) return;
  await User.findOneAndUpdate({ _id: user._id }, { $unset: { google: 1 }, $inc: { connectedSocialAccounts: -1} });
}

Linking different providers to a single user account

The first obvious data point is that there must be a logged-in user when a request to link a new provider comes. Otherwise, the request is treated as a login request, not a provider sync request.

We will leverage this piece of information to fine-tune Google's passport strategy and add the support for connecting a new provider.

Let's visualize it with a flowchart:

Profile User or (P.U.) simply means the email Id with which the user is trying to log in. The Logged-in user (or L.U.) refers to the currently logged-in user's account.

We have defined a top-level separation in how we handle a logged-in user vs. an anonymous user.

We link the Google account of a user to their logged-in account in only two conditions:

When the account (specifically the account's email, let's call it ProfileEmail) with which the user is trying to log in does not exist in the database, for any user.
When the ProfileEmail is already linked to the logged-in user, but for a different provider (since a user can have multiple social accounts with the same email).

In all other scenarios, we either create a brand new user (if not already exist) and treat it as a completely different account (not linked with the Logged-in user or L.U.), or we do nothing.

Our updated Google strategy:

const passport = require('passport');
const GoogleStrategy = require('passport-google-oauth20').Strategy;

const User = require('@models/user');
const config = require('@config');

passport.use(new GoogleStrategy({
  clientID: config.googleClientId,
  clientSecret: config.googleClientSecret,
  callbackURL: config.googleCallbackUrl,
  scope: ['profile', 'email'],
  passReqToCallback: true,
  },
  async function(req, accessToken, refreshToken, profile, done) {
    try {
      const email = profile['_json']['email'];
      if(!email) return done(new Error('Failed to receive email from Google. Please try again :('));

      const user = await User.findOne({
        $or: [
          { 'google.email': email },
          { 'amazon.email': email },
          { 'github.email': email },
        ]
      });

      if (req.user) {
        if (!req.user.google || (!req.user.google.email && !req.user.google.accessToken && !req.user.google.profileId)) {
          /**
           * proceed with provider sync, iff:
           * 1. req.user exists and no google account is currently linked
           * 2. there's no existing account with google login's email
           * 3. google login's email is present in req.user's object for any provider (indicates true ownership)
           */
          if(!user || (user && user._id.toString() === req.user._id.toString())) {
            await User.findOneAndUpdate({ '_id': req.user._id }, { $set: { google: { email: email, profileId: profile.id, accessToken }, connectedSocialAccounts: (req.user.connectedSocialAccounts + 1) }});
            return done(null, req.user);
          }
          // cannot sync google account, other account with google login's email already exists
        }
        return done(null, req.user);
      } else {
        if (user) {
          return done(null, user);
        }
        const newUser = await User.create({
          name: profile.displayName,
          connectedSocialAccount: 1,
          google: {
            accessToken,
            profileId: profile.id,
            email: email
          }
        });
        return done(null, newUser);
      }
    } catch (verifyErr) {
      done(verifyErr);
    }
  }
));

module.exports = passport;

Keeping track of connected providers

We need to keep track of the number of connected providers to every user account to make sure we don't allow disconnecting (or unlinking) a provider if it is the last one.

To achieve this, we had already defined a field in our user schema earlier. It is called connectedSocialAccounts. It is always initialized to a value of 1, as there will be at least one social provider connected at any point in time.

You would have noticed we increment the count of connectedSocialAccounts whenever we connect a new provider. Similarly, we lower it by one for every disconnection.

Showing the status for all providers

We need to show the status of all providers on the UI. But how does the client know about the status of all providers? We request the details from our server.

This is somewhat related to how the client-side code is written but I'll explain how it works. You can refer to the nodejs code here.

Whenever the user successfully logs in, we fetch the user details from our backend server.
For connected (or linked) providers, our front-end checks if the user object contains google, github, and amazon. It shows the option to disconnect for only those providers who are present given that the number of connected providers is more than one.
For disconnected (or yet-to-be-linked) providers, it simply shows the buttons to connect them.

Cross-Sync Achieved 🎉🎉🎉

Way to go!

Noice! You have successfully reached the second checkpoint 🎊.

Take a breath. Admire what you have achieved 📺 🥳.

Code up till this point is available in the main branch of the repo. Feel free to take a peek if you’d like.

Now we are heading towards the final stop, i.e., adding the support for multiple logged-in accounts 🚀🎆.

This is not a common feature to have on websites and hence I couldn't any resource covering it.

In the coming section, I'll walk you through my thought process and how I came up with the approach to implement this. And how you can too 🤩.

Here we go 👨🏻‍💻.

Adding support for multiple logged-in accounts

This feature is very niche and suitable for only specific use cases. You won’t find this in a lot of products. But I wanted to explore how it can be implemented.

Just for context, here’s how it looks for Gmail:

You are most likely familiar with how Gmail works, let me highlight the features we are interested in:

Clicking any profile loads the data (inbox, labels, filters, settings, etc.) for that account.
You can sign out of all accounts at once.
You can log in to multiple Google accounts.

Looking at these requirements, there are a couple of things we can be certain about:

Gmail indeed loads different user data when you switch between different Google accounts.
It doesn't ask for your password when you switch accounts. It indicates all the accounts are authenticated. So either Google is storing different sessions for all the user accounts (and loading based on request query param authuser?) or they are hot reloading a single user session in the backend based on again, request query param.
It allows signing out of all user accounts at once. This would be very straightforward if you have a single session for multiple user accounts.
It shows a list of currently logged-in Google accounts on the profile popup. This clearly indicates they are storing this information somewhere.

These observations have helped us progress somewhat closer to our goal.

We now have a better understanding of how we can approach this. But there is one decision you need to make before you progress further.

💭 One session per user document or one session per unique user?

Let's understand this with help of an example.

You are an end-user of this application. You have signed in using one of your Google accounts (say G.A1). After signing in, you went ahead and added (not to be confused with connected/linked) another Google account (say G.A2).

Having one session per user will lead you to have two sessions in the session store (because you technically have two user accounts or two separate MongoDB user documents).
Having one session per unique user will assign only one session for both of your accounts as both represent the same end-user.

This is a key decision you need to make when implementing this feature as everything else depends on it.

We will be going ahead with the second option i.e., one session per unique user.

Why? Simply because one session is easier to manage. We can hot reload the session when the user wants to switch accounts, and deleting a single session will log all the user accounts out.

This also means that you get logged out from all of your accounts as soon as the session expiry hits.

Tracking all logged-in accounts

When a user is logged in, we need to know what other logged-in accounts that user has, if any. We can store the user Ids of other logged-in accounts in every user document.

Whenever the user adds a new account, we update both user documents (the existing one and the new one that just got added) with the user Id, name, and email of the other one.

We can then extend this for more than two accounts and make sure to update the otherAccounts field in each user document whenever a new Google account gets added.

Now that we have finalized our approach, let's proceed to the next step where we update our Google strategy to support multiple logged-in accounts.

Let's first visualize all possibilities (no, not 14000605 😉):

If the user is not logged in, the user goes through a simple OAuth flow
However, if the user is logged in, we create a new user document and populate the otherAccounts flag. Finally, we inject the newly created user's id into the session object (more on this later).

Based on the above considerations, here's our updated passport strategy for Google:

const passport = require('passport');
const GoogleStrategy = require('passport-google-oauth20').Strategy;

const User = require('@models/user');
const config = require('@config');
const { default: mongoose } = require('mongoose');
const mongoSessionStore = require('../../loaders/sessionStore').run();

passport.use(new GoogleStrategy({
  clientID: config.googleClientId,
  clientSecret: config.googleClientSecret,
  callbackURL: config.googleCallbackUrl,
  scope: ['profile', 'email'],
  passReqToCallback: true,
  },
  async function(req, accessToken, refreshToken, profile, done) {
    try {
      const email = profile['_json']['email'];
      if(!email) return done(new Error('Failed to receive email from Google. Please try again :('));

      const user = await User.findOne({ 'email': email });

      if (req.user) {
        if (req.user.email !== email) {
          if (user && req.user.otherAccounts.find((accountObj) => user._id === accountObj.userId)) {
            return done(null, user); 
          }
          else {
            // fresh request to add "other logged in account"
            // step 1
            const newUser = await User.create({
              name: profile.displayName,
              email,
              profileId: profile.id,
              accessToken,
              otherAccounts: [ ...req.user.otherAccounts, { userId: req.user._id, name: req.user.name, email: req.user.email } ],
            });


            // step 2: update otherAccounts for already logged in users
            req.user.otherAccounts.forEach(async (otherAccount) => {
              await User.findOneAndUpdate({ '_id': otherAccount.userId }, { $push: { otherAccounts: { userId: newUser._id, email: newUser.email, name: newUser.name } } });
            });

            // step 3: : update otherAccounts for logged in user
            const existingUser = await User.findOne({ '_id': req.user._id });
            existingUser.otherAccounts.push({ userId: newUser._id, email: newUser.email, name: newUser.name });
            await existingUser.save();

            // update session in mongo
            mongoSessionStore.get(req.sessionID, (err, currentSession) => {
              currentSession.passport.user = new mongoose.Types.ObjectId(newUser._id);
              mongoSessionStore.set(req.sessionID, currentSession, (updateErr, finalRes) => {
                // return the new user
                return done(null, newUser);
              });
            });
          }
        } else {
          return done(null, req.user);
        }
      } else {
        if (user) {
          return done(null, user);
        }
        const newUser = await User.create({
          name: profile.displayName,
          email,
          accessToken,
          profileId: profile.id,
          otherAccounts: [],
        });
        return done(null, newUser);
      }
    } catch (verifyErr) {
      done(verifyErr);
    }
  }
));

module.exports = passport;

We have successfully updated our Google strategy and made sure each user document contains the references to the other logged-in accounts 👌🏻.

Switching between different logged-in accounts

This looks very similar to how Gmail provides the option to switch accounts. We have a profile popup that shows all the logged-in accounts and clicking on anyone loads that user account into session.

But how do we hot reload the session?

We are using MongoDB as our session store with the help of connect-mongo npm package. This allows saving the session in the same database we are storing the application data.

Let's check out what a session collection holds:

[
  {
    _id: 'PcFbwsKJQsFHNtH5TksWbCMmuDC7odjH',
    expires: ISODate("2022-05-12T12:31:36.554Z"),
    session: {
      cookie: {
        originalMaxAge: 120000,
        expires: ISODate("2022-05-12T12:31:35.530Z"),
        secure: null,
        httpOnly: false,
        domain: null,
        path: '/',
        sameSite: null
      },
      passport: { user: ObjectId("627b5024419f6964528642b3") }
    }
  }
]

Let's look closely at the passport object in the session. It only contains the user Id (since we only pass the user Id to the callback during passport.serialize).

This gives us conclusive proof that Passportjs takes this user Id and runs the passport.deserialize to load the user into the session.

This also means that we only need to somehow replace this user Id if we want to hot reload a user into the session (without going through the whole authentication flow again).

Fortunately, connect-mongo has a concept of events. We can leverage the setter method it provides to update the session whenever we need.

But doesn't this mean that we can (mistakenly) inject a user Id into the session for a completely different user? Doesn't this pose a security risk?

Yes, it has the potential. That's why we have introduced the concept of otherAccounts in the user schema.

⭐️ Users can switch to another logged-in account only if the user Id of the second account is present in the otherAccounts array of the first one.

We enforce this in the account switch route:

router.get('/google/switch/:userId', ensureAuthenticated, async (req, res) => {
  const { userId } = req.params;
  const currentSessionId = req.sessionID;
  const newUserId = new mongoose.Types.ObjectId(userId);

  if (req.user.otherAccounts && !req.user.otherAccounts.find((otherAcc => otherAcc.userId === userId))) {
    // not authorized to switch
    return res.redirect('/');
  }

  mongoSessionStore.get(currentSessionId, (err, sessionObj) => {
    if (err) {
      res.redirect('/');
    }
    else {
      sessionObj.passport.user = newUserId;
      mongoSessionStore.set(currentSessionId, sessionObj, (updateErr, finalRes) => {
        if(updateErr) {
          console.log('error occurred while updating session');
        }
        res.redirect('/');
      });
    }
  });
});

This is a protected route so an anonymous user can't even access this.
We are checking if the otherAccounts array contains the user Id that the logged-in user is trying to switch to.

Combining these practices, we have made it much more secure for the users 🔐.

🎉 We have completed the final step 🎉

With the third and final checkpoint, you have completely built the fully functional OAuth 2 authentication & authorization mechanism with the ability to add multiple logged-in accounts.

You can find the complete code for this checkpoint here ✨.

📺 Final walkthrough of the application:

You are a rockstar programmer and definitely believe in patience! This is no easy feat.

I tried my best to make sure this guide is light to read, skimmable, and to the point.

You can choose to walk away from your screen for a while, have a glass of water, and take a break.

You have earned it 🚀.

Conclusion

And that's it! We have covered a lot of ground in this article. We talked about how we can implement OAuth authentication using Passportjs in an Expressjs application with multiple social providers and the ability to sync multiple social accounts to a single user account. Additionally, we also looked that how we can have multiple user accounts logged in at the same time.

The main reason I jotted this down is that I couldn't find any resource explaining the things covered in this article. And, building this project will definitely come in handy next time I (and certainly you) need an OAuth2 boilerplate. What's better than having a headstart on your next awesome project 😄.

I hope it helped you implement OAuth 2 authentication without any major issues. If you feel there is something missing or can be better explained, please feel free to drop a comment below. This will help everyone who lands on this article.

I would also love to know your experience with OAuth 2. For me, it was an if-it-works-don’t-touch-it thing, but now I definitely have a better understanding of what goes on behind the scenes.

Happy authenticating 🔐.

What next?

There are a lot of things that you can explore. If social authentication using OAuth 2 is the first authentication & authorization mechanism you are learning, you can check out other types of strategies out there.

Two-Factor Authentication (2FA) and Single Sign-On (SSO) are the two things I would love to learn next in the authentication realm.

Security through obscurity is also fascinating, you can take a peek and decide if you want to explore it further.

And just a final reminder, there is never a perfect plan to learn things. It's okay to learn (& break) things that you find intriguing and connect the dots along the way. I found this article really helpful on the topic.

Resources

In addition to all the resources mentioned in this guide, you can check out the following resources to further deepen your understanding and expand your horizons:

5 Tiny Developer Workflow Tips to Improve Your Productivity

Rishabh Rawat — Tue, 10 May 2022 15:08:16 +0000

As a software developer, we're constantly on the move trying to learn and build new things. It is all about acquiring knowledge and implementing it. This requires us to constantly reassess what's the best way to get the job done (you must've heard don't reinvent the wheel a hundred times by now, right?).

In this article, I wanted to share some of my revelations as quick tips that will improve your productivity at work. It won't even take you 2 minutes, but it might save you from a lot of headaches.

Here are 5 tiny, seemingly insignificant but useful tips:

💾 Format on save in VSCode
🌉 Squash commits before rebase (whenever you can)
🗻 Find and replace in VSCode
🖨️ Use VSCode extension for console logs
🔌 Shut down your mac (at least occasionally)

The rest of the article will expand on each of the above points. If you were only looking for tips, there you have it. You can choose to skip what follows. But read if you want to learn more.

Format on save in VSCode

How many times have you manually corrected formatting issues in your code? We've all been there during our web development journey.

No more manual labor of formatting the code. Install Prettier on VSCode.

Once you have the prettier extension installed,

Head over to your VSCode settings (cmd + , on mac) and check format on save in editor option.
To confirm, simply open any of your files with intentional formatting mistakes and just hit save.

If you want fine-grained control over formatting rules or you have different formatting preferences across different projects, simply create a .prettierconfig file inside each of your projects with your desired formatting rules.

To learn more, check out the Prettier's official documentation for VSCode.

Squash commits before you rebase (whenever you can)

If you're working on a shared repo, chances are that the base branch has moved while you were working on your feature branch. In such scenarios, merge conflicts are inevitable.

This is something I've witnessed firsthand, not realizing that I can completely avoid the pain of hunching over, carefully resolving conflicts, and spending way too much time unnecessarily.

Problem? Whenever the base branch you're working on moves, you usually rebase your feature branch to make sure it has latest changes. If you have multiple commits in your feature branch, resolving merge conflicts becomes ‌harder during rebase.

While you cannot avoid the merge conflicts, what you can avoid is the time spent on resolving them.

Here's the pro tip: squash all the commits in your feature branch, and then perform the rebase on the base branch.

How it helps? It works because:

Your single squashed commit contains your latest changes, so it's easy to identify your changes vs changes on the base branch during conflict resolution. And,
You just need to resolve the conflict once as opposed to multiple stages of conflict resolution for rebasing each commit.

Find & replace in VSCode

I'll be the first to admit, I've been using the cmd + D command for ages to select all the occurrences of a word in VSCode.

Here's what I used to do for find and replace in VSCode:

Highlight any word in VSCode (either by double-clicking on the word or by pressing cmd + D while the cursor is on that word).
Keep hitting cmd + D until all the occurrences are highlighted.
Type out whatever I want to change those words to.

But it is case-insensitive. Most often, I don't want that because it may end up changing things by accident‌. Now, I've made it case-sensitive by default.

Just hit cmd + F and make sure to select the Aa icon.

Here's the modified find & replace:

Highlight any word in VSCode (either by double-clicking on the word or by pressing cmd + D while the cursor is on that word).
Enter the new word you want to replace the highlighted word with.
Hit Replace all to replace all occurrences at once.

With this, you can now find & replace any word in your code without worrying about case 😌.

Use VSCode extension for console logs

We all have been there. Consoling the heck out of the program to figure out what’s the issue. Given the number of times I console log, ‌I needed some hotkey for doing that.

You can install javascript console utils VSCode extension. It lets you:

Write an empty console log by hitting cmd + shift + L.
Write console for any variable by highlighting it and hitting the same set of keys.
If you want to remove all console logs from a file, press cmd + shift + D.

Neat ✨

Shut down your mac (at least occasionally)

I used to never shut down my mac because it is convenient. Just open the lid and bam, it's ready to go. It never made sense to shut it down. Until recently when it started stuttering between trivial operations.

While I'm aware there's a lot of debate around this topic and both sides (shut down vs. sleep) have reasons to prefer one thing over the other.

But based on my personal experience, not shutting down my mac for months caused performance & thermal bottlenecks.

Now I use a combination of both. I shut down my mac on the weekends. You can read more on the Apple's official forum.

Done tweaking? Awesome!

That's it! These were some of quick tips that you can incorporate in your workflow right now to elevate your productivity at work.

Let me know how many of these are already in your productivity arsenal and if there's any other micro fix you follow for better developer experience or quality of life improvement at work.

Unit Testing Essentials for Express API: A Step-by-Step Guide

Rishabh Rawat — Sun, 27 Mar 2022 07:46:14 +0000

Unit testing is a very important aspect of software development. It involves testing the smallest units of code (eg. functions or methods) and if written well, they act as a guard rail whenever your code violates its expected behavior.

In this article, we'll cover all the steps required to write unit tests for your node express API.

By the end of this article, you'd have an understanding of how unit tests work, how to write them, and how to be less lost finding the bugs.

Here's the overview of how the article is structured:

🌠 Our express API
🧪 Let's begin testing
🔎 Install mocha and chai
🧩 Create your first test
✨ Running your first test
🔮 Try async
🪝 Before and After hooks
📈 One unit at a time
🎭 Stubbing private resources with Rewire
🏭 Testing our database
🐫 Testing our routes
👓 Check your coverage
🏎️ Test driven development
💠 Conclusion
🏄🏼‍♂️ Resources

Our express API

We'll be using a simple express API throughout this article to demonstrate unit testing. You can find the code on Github.

The API only provides five endpoints:

GET /health/sync - returns 'OK' synchronously
GET /health/async - returns 'OK' asynchronously
GET /item/:hash - fetches an item from MongoDB with matching hash value
POST /item - creates new item in MongoDB
PUT /item - updates item's hash value in MongoDB

Let’s begin testing

We are now ready to write some unit tests. We'll be using mocha and chai for our API. Mocha is open-source, can be combined with various assertion libraries, and has great community support. Moreover, it is used by Coursera, Asana, Intuit, and the like.

There are several components (or units) that we need to test in our API:

Controllers (or services or business logic handlers) - it is responsible for performing the business logic and returning the result.
MongoDB models (database schemas) - Constraints are defined in the schema and are enforced at runtime.
Express routes - It is responsible for mapping the HTTP requests to the corresponding controller and eventually returning the response to the client.

Install mocha and chai

First off, we need to install mocha and chai:

npm install -D mocha chai

Done? Great! Time to create our first test ⚡.

Create your first test

Let's start by creating a test file. Here's how the current project structure looks like:

- src
-- controllers
---- item.controller.js
---- health.controller.js
-- models
---- item.model.js
-- routes
---- index.js
---- item.route.js
---- health.route.js
-- tests
---- health.spec.js
-- app.js

We'll be adding our tests inside the tests folder. We have created a new file health.spec.js inside the folder.

Let's start writing some basic tests for our /health API:

describe('Test /health', () => {
  describe('Health check on /sync', () => {
    it('health should be okay', () => {
      const actualResult = healthCheckSync();
      expect(actualResult).to.equal('OK');
    });
  });
});

describe block

We use this outer-most describe block to group related test suites similar to how we've structured our application code.

You can also create nested describe blocks to contain your test suites. For example, here's how we will structure the tests for /health endpoint:

- describe('Test /health')
-- describe('Test /health/sync')
-- describe('Test /health/async')

We will be adding a lot unit tests for our express API and it is usually a good idea to split your unit tests across different files corresponding to different modules (or business logic).

NOTE: It is helpful to write messages in test suites to showcase what your code does from product's perspective instead of application/software perspective.

it block

This is the place where we actually write our test suites and check for assertions, return values, etc.

Running your first test

Now that we have our first test suite ready, we are all set. To run the test we've written, let's add the following line to the package.json file inside the scripts section:

"test": "mocha ./src/tests/*.spec.js"

This script will look for all the files inside the tests folder and run them using mocha. So we just need to run the below command whenever we want to run our test suites:

npm test

And here we have our first test suite passing 🎉!

If you want to run mocha in watch mode to automatically trigger on file changes, you can have another test script in your package.json file like this:

"test:watch": "mocha --watch ./src/tests/*.spec.js"

Try async

The tests we write are probably going to test async operations that happen across our express API. Let’s write a test for our /health/async endpoint as well which is async in nature:

describe('Test /health', () => {
  describe('Health check on /sync', () => {
    it('health should be okay', () => {
      const actualResult = healthCheckSync();
      expect(actualResult).to.equal('OK');
    });
  });

  describe('Health check on /async', () => {
    it('health should be okay', async () => {
      const actualResult = await healthCheckAsync();
      expect(actualResult).to.equal('OK');
    });
  });
});

We get a lot of flexibility here because mocha supports multiple ways we can handle async operations in our test suites:

We can use async/await like show above,
We can have the thenables attached which perform assertions on the result, or
We can use done parameter with callbacks to handle the result and perform assertions.

Before and After hooks

We sometimes need to run some setup/teardown code before/after each test suite. For example, we might need to connect to a database before each test suite and disconnect it after each test suite.

In a describe block, you get access to the following hooks:

before - runs before all the tests inside the describe block runs
beforeEach - runs before each test inside the describe block runs
after - runs after all the tests inside the describe block have run
afterEach - runs after each test inside the describe block has run

Let's take a look at the following example for clarity:

describe('Test /health', () => {
  before('before', () => {
    console.log('Ran before all the test suites');
  });

  after('after', () => {
    console.log('Ran after all the test suites');
  });

  beforeEach('beforeEach', () => {
    console.log('Ran before EACH test suite');
  });

  afterEach('afterEach', () => {
    console.log('Ran after EACH test suite');
  });

  describe('Health check on /sync', () => {
    it('health should be okay', () => {
      const actualResult = healthCheckSync();
      expect(actualResult).to.equal('OK');
    });
  });

  describe('Health check on /async', () => {
    it('health should be okay', async () => {
      const actualResult = await healthCheckAsync();
      expect(actualResult).to.equal('OK');
    });
  });
});

Running the above code gives the following output:

We can observe that:

Before and after hooks ran at the start and end of the outer-most describe block.
BeforeEach and afterEach ran before & after each test suite (i.e. each test or it block).

One unit at a time

When testing a function, the idea of unit testing is to only test that function and not the other stuff present in that function. So if a function involves a database call, we don’t actually want to make that database call when testing. Here's why:

We're performing "unit" test on the function, not the database.
Any problem in the database would cause the function to fail for no reason.

We'll test our readItem function to understand this better. But first, let's install the necessary dependencies by running the following command:

npm install -D rewire sinon sinon-chai

NOTE: Sinon enables us to create stubs, spies, mocks with any unit testing framework. Rewire provides us setter and getter functions to replace the original functions with our own.

Now that we have our dependencies ready, let's look at the test suites for readItem:

describe('Testing /item endpoint', () => {
  let sampleItemVal;
  let findOneStub;

  beforeEach(() => {
    sampleItemVal = {
      name: 'sample item',
      price: 10,
      rating: '5',
      hash: '123456891'
    };

    findOneStub = sandbox.stub(mongoose.Model, 'findOne').resolves(sampleItemVal);
  });

  afterEach(() => {
    itemController = rewire('../controllers/item.controller');
    sandbox.restore();
  });

  describe('GET /', () => {
    it('should return error when called without hash', async () => {
      itemController
        .readItem()
        .then(() => {
          throw new Error('⚠️ Unexpected success!');
        })
        .catch((err) => {
          expect(result).to.be.instanceOf(Error);
          expect(err.message).to.equal('Invalid item id');
        });
    });

    it('should succeed when called with hash', async () => {
      itemController
        .readItem('someRandomHash')
        .then((item) => {
          expect(item).to.equal(sampleItemVal);
        })
        .catch((err) => {
          throw new Error('⚠️ Unexpected failure!');
        });
    });
  });
});

Let's go through it step-by-step:

We stub the findOne method of mongoose in the before hook for /item endpoint.
We reset the itemController to the original one after each test suite (i,e., "it" block) runs.
We restore the sandbox afterEach test suite to reset the stubs (it is generally a good practice to use sandbox).
In the first test suite, we call readItem without hash.
We expect the result to be an error and make an assertion inside the catch block.
In the second one, we pass the hash. It results with a sampleItemVal because we stubbed the findOne method.
We make the assertion on the result.

As expected, tests pass seamlessly:

We have now successfully tested our readItem function by stubbing out the findOne method of mongoose. This is one of the extremely important concept to understand when writing unit tests.

NOTE: Test one unit at a time. Pay extreme care that your tests don't reach out for the production resources (for example, production database and API calls over the network).

Stubbing private resources with Rewire

There is a lot that we have covered starting from scratch. If you're following along, you would've started noticing how the same process can be replicated to test most of the things.

Let's try to replicate the same process to test our updateItem function:

exports.updateItemHash = async function (hash) {
  try {
    if (!hash) {
      throw new Error('Incomplete arguments');
    }

    let item = await Item.findOne({
      hash
    });
    item.hash = getUniqueHash(item);

    return await item.save();
  } catch (err) {
    return Promise.reject(err);
  }
};

As you can see, there is a helper function we're using here called getUniqueHash. And unfortunately, we can't access this function outside the module since it is not exported.

function getUniqueHash(item) {
  if (!item) return null;
  const currentHash = item.hash;
  let newHash = nanoid(10);

  while (newHash === currentHash) {
    newHash = nanoid(10);
  }
  return newHash;
}

If you look at the documentation of sinon stub, you'll see that we cannot use the stub in this case.

For it to work, we would need to use the rewire package. It is just like require but comes with a setter and getter function to modify the behavior of private functions and variables in modules.

Let's see the test suite for updateItem and understand how it works:

describe('Testing /item endpoint', () => {
  let sampleItemVal;
  let findOneStub;
  const sampleUniqueHash = '1234567891';

  beforeEach(() => {
    sampleItemVal = {
      name: 'sample item',
      price: 10,
      rating: '5',
      hash: sampleUniqueHash
    };

    findOneStub = sandbox.stub(mongoose.Model, 'findOne').resolves(sampleItemVal);
  });

  afterEach(() => {
    itemController = rewire('../controllers/item.controller');
    sandbox.restore();
  });

  describe('PUT /', () => {
    let getUniqueHashStub, saveStub, result, sampleUpdatedItemVal;
    const sampleUpdatedHash = '9876543219';

    beforeEach(async () => {
      // forcefully restore sandbox to allow re-write of findOneStub
      sandbox.restore();

      // Stub to mock getUniqueHash's Functionality
      getUniqueHashStub = sandbox.stub().returns(sampleUpdatedHash);

      sampleUpdatedItemVal = {
        ...sampleItemVal,
        hash: sampleUpdatedHash
      };
      // save stub to return updated item
      saveStub = sandbox.stub().returns(sampleUpdatedItemVal);

      // make findOneStub return save() method in addition to sampleItemVal
      findOneStub = sandbox.stub(mongoose.Model, 'findOne').resolves({
        ...sampleItemVal,
        save: saveStub
      });

      // Use rewire to modify itemController's private method getUniqueHash
      itemController.__set__('getUniqueHash', getUniqueHashStub);
    });

    it('should throw invalid argument error', () => {
      itemController
        .updateItemHash()
        .then(() => {
          throw new Error('⚠️ Unexpected success!');
        })
        .catch((err) => {
          expect(result).to.be.instanceOf(Error);
          expect(err.message).to.equal('Incomplete arguments');
        });
    });

    it('should update item hash successfully', async () => {
      result = await itemController.updateItemHash(sampleUniqueHash);
      expect(findOneStub).to.have.been.calledWith({
        hash: sampleUniqueHash
      });
      expect(findOneStub).to.have.been.calledOnce;
      expect(saveStub).to.have.been.calledOnce;
      expect(result).to.equal(sampleUpdatedItemVal);
    });
  });
});

Let's go through this step-by-step again:

We have stored the initial unique hash in sampleUniqueHash variable.
Inside test suites for PUT endpoint, we have stored the updated unique hash inside sampleUpdatedHash variable.
We need a slightly different stub for findOne so we have completely restored/reset the sinon sandbox. This will allow us to write a new stub for findOne.
We have created a stub for getUniqueHash function which will be invoked instead of the original private function inside itemController.
On line 41, we have created a new stub for findOne which contains the save method in addition to the sample item value.
We are using rewire to modify the private function and replace it with our stub.
In first test suite, we have called updateItemHash with an empty hash. It should throw an error.
In second test suite however, we have called updateItemHash with a valid hash. It should update the hash and return the updated item.

This lands us shiny green check marks ✅ on the terminal:

Keeping up the momentum, let's test our mongoose models in the next section ✴️.

Testing our database

We usually put some constraints on our models when we create schemas. This ensures that our data follows certain characteristics and is consistent.

Here's our item schema:

const mongoose = require('mongoose');
const Schema = mongoose.Schema;

const itemSchema = new Schema({
  name: {
    type: String,
    required: true
  },
  rating: {
    type: String,
    required: true
  },
  price: {
    type: Number,
    required: true
  },
  hash: {
    type: String,
    required: true,
    unique: true,
    minlength: 10,
    maxlength: 10
  }
});

module.exports = mongoose.model('Item', itemSchema);

Let's create a new file named model.spec.js inside our tests folder. We will add some basic checks for our model to make sure that our data respects those constraints:

describe('Testing Item model', () => {
  let sampleItemVal;

  beforeEach(() => {
    sampleItemVal = {
      name: 'sample item',
      price: 10,
      rating: '5',
      hash: 'hashGreaterThan10Chars'
    };
  });

  it('it should throw an error due to missing fields', (done) => {
    let item = new Item();

    item.validate((err) => {
      expect(err.errors.name).to.exist;
      expect(err.errors.rating).to.exist;
      expect(err.errors.price).to.exist;
      expect(err.errors.hash).to.exist;

      done();
    });
  });

  it('it should throw an error due to incorrect hash length', (done) => {
    let item = new Item(sampleItemVal);

    item.validate((err) => {
      if (err) {
        expect(err).to.be.instanceOf(ValidationError);
        // this is expected, do not pass err to done()
        done();
      } else {
        const unexpectedSuccessError = new Error('⚠️ Unexpected success!');
        done(unexpectedSuccessError);
      }
    });
  });

  it('it should create the item successfully with correct parameters', (done) => {
    let item = new Item({
      ...sampleItemVal,
      hash: '1234567891'
    });

    item.validate((err) => {
      if (err) {
        const unexpectedFailureError = new Error('⚠️ Unexpected failure!');
        done(unexpectedFailureError);
      } else {
        expect(item.hash).to.equal('1234567891');
        done();
      }
    });
  });
});

We have create three test suites to check for three things:

Item validation should fail when we don't pass required fields.
Item validation should fail when we don't pass the correct hash length.
Item should be created successfully when we pass the right parameters.

Tests pass successfully 🎉:

Testing our routes

Finally, let's test our express app routes. We will create a new file named routes.spec.js inside our tests folder.

Just for your reference, here's how the final project structure looks like:

- src
-- controllers
---- item.controller.js
---- health.controller.js
-- models
---- item.model.js
-- routes
---- index.js
---- item.route.js
---- health.route.js
-- tests
---- health.spec.js
---- item.spec.js
---- model.spec.js
---- routes.spec.js
-- app.js

The next step would be to install supertest package from npm. It makes testing HTTP requests really easy and clean.

npm install -D supertest

Finished installing? Awesome!

Let's add some tests for our routes now:

describe('Testing express app routes', () => {
  afterEach(() => {
    app = rewire('../app');
    sandbox.restore();
  });

  describe('Testing /item route', () => {
    let sampleItemVal, hash;

    beforeEach(() => {
      hash = '1234567891';
      sampleItemVal = {
        name: 'sample item',
        price: 10,
        rating: '5',
        hash
      };
      sandbox.stub(itemController, 'readItem').resolves(sampleItemVal);
      sandbox.stub(itemController, 'createItem').resolves(sampleItemVal);
      sandbox.stub(itemController, 'updateItemHash').resolves(sampleItemVal);
    });

    it('GET /:hash should successfully return item', (done) => {
      request(app)
        .get(`/item/${hash}`)
        .expect(200)
        .end((err, response) => {
          expect(response.body).to.have.property('message').to.equal('Item read successfully!');
          expect(response.body)
            .to.have.property('item')
            .to.have.property('name')
            .to.equal('sample item');
          expect(response.body).to.have.property('item').to.have.property('price').to.equal(10);
          expect(response.body).to.have.property('item').to.have.property('rating').to.equal('5');
          expect(response.body).to.have.property('item').to.have.property('hash').to.equal(hash);
          done(err); // err is null in success scenario
        });
    });

    it('POST / should successfully create a new item', (done) => {
      request(app)
        .post('/item/')
        .send(sampleItemVal)
        .expect(200)
        .end((err, response) => {
          expect(response.body).to.have.property('message').to.equal('Item created successfully!');
          expect(response.body)
            .to.have.property('item')
            .to.have.property('name')
            .to.equal('sample item');
          expect(response.body).to.have.property('item').to.have.property('price').to.equal(10);
          expect(response.body).to.have.property('item').to.have.property('rating').to.equal('5');
          expect(response.body).to.have.property('item').to.have.property('hash').to.equal(hash);
          done(err);
        });
    });

    it('PUT / should successfully update hash for a given item', (done) => {
      request(app)
        .put('/item')
        .send(hash)
        .expect(200)
        .end((err, response) => {
          expect(response.body).to.have.property('message').to.equal('Item updated successfully!');
          expect(response.body)
            .to.have.property('item')
            .to.have.property('name')
            .to.equal('sample item');
          expect(response.body).to.have.property('item').to.have.property('price').to.equal(10);
          expect(response.body).to.have.property('item').to.have.property('rating').to.equal('5');
          expect(response.body).to.have.property('item').to.have.property('hash').to.equal(hash);
          done(err);
        });
    });
  });
});

It follows a similar structure to what we've been doing so far. We are essentially:

Stubbing the controllers because we want to test routes, not controllers (we've tested them already).
Making the request using supertest and asserting the response.

Before you run the tests, make sure to update your test script to include the --exit flag:

"test": "mocha ./src/tests/*.spec.js --exit",

This makes sure your tests exit once finished.

There we go on our successful tests streak 🚀:

NOTE: To test protected routes, simply pass the authentication headers or auth credentials when making the request. You can read more on supertest documentation.

Check your coverage

Code coverage is the indication of the code percentage covered under tests. Now that we have finished writing them, it would be nice to see the code coverage of our unit tests.

Code coverage often affects developer confidence. But there is a catch. A 100% code coverage does not necessarily mean that your code is perfect.

TL;DR: code coverage is just the percentage of code covered by the tests. It does not tell whether the tests cover all the scenarios.

Let's take one example.

We have a function named getUniqueHash in our API:

function getUniqueHash(item) {
  const currentHash = item.hash;
  let newHash = nanoid(10);

  while (newHash === currentHash) {
    newHash = nanoid(10);
  }
  return newHash;
}

And here's one unit test:

describe('Test getUniqueHash'), () => {
  it('should return a new hash', () => {
    const item = {
      hash: '1234567890',
    };
    const newHash = getUniqueHash(item);
    expect(newHash).to.not.equal(item.hash);
  });
});

This test technically covers the function getUniqueHash but it doesn’t cover all the scenarios.

What if the length of the hash generated in the function changes to 100? What if it is null? How about a malicious script as a string? Test coverage won't be affected but the functionality will be, drastically.

NOTE: The situation does not change even if we stub the nanoId result. The library on production can choose to return a different output in the future. While we cannot control it, we can prepare for it with a unit test covering that scenario.

Now that we have that out of the way, let's add coverage to our app.

First step, let's install nyc:

npm install -D nyc

Next, let's add the following script to the package.json file:

"coverage": "nyc --reporter=text npm test"

And we're done! You can now run the coverage script with npm run coverage and see the coverage report in the terminal.

Here's how it looks for our express API:

Optional: Brief on Test Driven Development (TDD)

Test-driven development is a practice where unit tests for a feature are written before that feature is developed. Development happens progressively to fix each test case until all of them pass.

Here is a brief overview of how TDD works:

Write a unit test for the feature to be implemented
Run the tests; they all fail.
Implement the feature to fix the first unit test. It passes, rest of them fails.
Repeat the above steps until all of the tests pass.

This is an agile way which makes the development strictly combined with tests and refactoring.

The obvious benefit of going with this approach is reliability and developer confidence. As the tests are written before implementing the feature, it makes sure that developed features cater to every test case.

NOTE: Developing an application first usually distorts the developers' view of the bigger picture when writing unit tests. And, it is more likely to result in a missed test case.

One common argument against this approach is the speed of development which is highly affected because the development is now combined with testing.

You can read more about test driven development here if you're curious.

Now you go, captain!

And that wraps up our unit testing endeavor! If you reached here, congrats! You are now equipped with the required knowledge to embark on your unit tests journey.

We covered a lot of ground in this article. We talked about the benefits of unit testing, how can we integrate it in our express JS API, and use stubs to mock external libraries and private functions. We also touched on the test coverage and test-driven development (TDD) programming paradigm.

Having said that, there are bound to be things that were not covered in this article. So I encourage you to use this as a starting point and explore further according to your requirements.

I hope you found this useful and actionable to implement in your express.js API. For any queries and feedback please feel free to reach out in the comments or hit me up on Twitter.

Resources

Unit test library documentation: Mocha
Assertion library we used: Chai (comes with a lot of plugins worth exploring)
Standalone test spies, stubs and mocks for JavaScript: Sinon
HTTP assertions library: Supertest
Monkey patching for private functions and variables: Rewire
Code coverage: Nyc
Express.js API used in this article: Github

Want more such articles?

You can join my newsletter here. I'll share the articles with you as they come out.

Caolan Asyncjs vs Async/Await: Which One to Use for Async Operations in NodeJS

Rishabh Rawat — Tue, 01 Mar 2022 04:57:17 +0000

Working with JavaScript we all have come across asynchronous operations at some point in our web development journey. There are various ways you can handle an asynchronous operation in JavaScript/nodeJS, can be either using callbacks, promises or async/await. This gives developers so much flexibility in code and that's the reason you can still find different approaches in the real world projects today.

If not handled well, asynchronous operations can prove to be harmful in subtlest of ways. We all know callback hell right?

In this article we'll take a look at Caolan's asyncjs library, how it provides easy-to-read way of working with asynchronous operations in JavaScript/nodeJS and if it's still needed for the usual control flows.

Here's the overview of what we'll cover:

✨ Async operations in javascript
👓 Handling async flows with asyncjs
🧪 Using async/await
🔎 You might still need asyncjs
🧩 Conclusion
🏄🏼‍♂️ What next?

Let's jump right in 🏊

Async operations in javascript

Asynchronous operations in nodeJS/JS are the operations that cannot return the result immediately. It can be a network call or a database operation, for example.

As it does not make sense for the execution to get halted there waiting for the async operation to finish, callbacks & promises came to solve the problem.

With callback/promise, we tell the event loop what to do when the result of the async operation arrives.

The callback/promise gets pushed to the event loop and gets revisited in the next iteration. This process repeats if the async operation doesn't resolve by the next iteration of the event loop.

Here's a sample callback based approach of working with async operations:

someAsyncOperation(function (err, data) {
  if (err) {
    console.log(`Some error occurred. Look at it => ${err}`);
  } else {
    data.forEach((item, index) {
      asyncProcessingOfItem(item, function (itemErr, isProcessed) {
        if (itemErr) {
          console.log(`Some error occurred while processing item. Here's that beast => ${err}`);
        } else if (isProcessed) {
          console.log(`${item} processed succesfully!!!`);
        } else {
          console.log(`${item} could not be processed :(`); 
        }
      })
    })
  }
})

Yes, the code doesn't look clean and credit goes to callbacks. If you want to understand more about callbacks and callback hell, there's a whole website dedicated to this. Check it out here.

This situation was vastly improved with the asyncjs library. Let’s see how asyncjs library contributed to better readability 👇

Handling async flows with asyncjs

The library provides an easy way to deal with asynchronous functions in NodeJS. In addition to a good collection of functions for arrays and objects, there are various control flows that the library provides for making developers life easy.

Asyncjs library also provides support for promises and async/await but I'll be showing examples using callbacks.

async.series

This flow allows you to put as many handlers as you want and they'll run in series one after the other. The output of one does not depend on the previous handler (unlike async.waterfall).

async.series([
    function(callback) {
        setTimeout(function() {
            // do some async task
            callback(null, 'one');
        }, 200);
    },
    function(callback) {
        setTimeout(function() {
            // then do another async task
            callback(null, 'two');
        }, 100);
    }
], function(err, results) {
    console.log(results);
    // results is equal to ['one','two']
});

In the above example, two async functions run in series and the final callback contains an array with the returned values from those functions.

If there's any error in any function, no further handler will be executed and the control will directly jump to the final callback with the thrown error.

async.parallel

This control flow comes handy when the handlers are not dependent on each other at all. You can trigger all of them at once. By parallel, we only mean kicking off I/O tasks if any, if your functions do not perform any I/O or use any timers, the functions will be run in series in synchronous fashion. Javascript is still single-threaded.

async.parallel([
    function(callback) {
        setTimeout(function() {
            callback(null, 'one');
        }, 200);
    },
    function(callback) {
        setTimeout(function() {
            callback(null, 'two');
        }, 100);
    }
], function(err, results) {
    console.log(results);
    // results is equal to ['one','two'] even though
    // the second function had a shorter timeout.
});

Again, error in any of the handlers will cause the execution of all the remaining handlers to be skipped.

async.race

This is exactly similar to Promise.race, result from the final callback will come from whichever function calls the callback first.

async.race([
    function(callback) {
        setTimeout(function() {
            callback(null, 'one');
        }, 200);
    },
    function(callback) {
        setTimeout(function() {
            callback(null, 'two');
        }, 100);
    }
],
// main callback
function(err, result) {
    // the result will be equal to 'two' as it finishes earlier
});

Using async/await

The control flows that we've seen in the previous section can be replicated using async/await without the need of asyncjs library. Let's recreate those examples using async/await:

async.series

try {
  const resultFromFn1 = await asyncFnThatReturnsOne();
  const resultFromFn2 = await asyncFnThatReturnsTwo();
  return [resultFromFn1, resultFromFn2];
} catch (err) {
  console.log(err);
}

Assuming the above code block is inside an async function, we have easily replicated the async.series functionality here.

We're making sure that asyncFnThatReturnsOne resolves and returns the result first before asyncFnThatReturnsTwo can run.
Final result array is exactly same as before i.e., ['One', 'Two']. It does not matter whether asyncFnThatReturnsOne takes longer than asyncFnThatReturnsTwo.
We're catching error using try-catch block.

async.parallel

try {
  const result = await Promise.all([    // result = ['One', 'Two']
    asyncFnThatReturnsOne(),
    asyncFnThatReturnsTwo()
  ]);
} catch (err) {
  console.log(err);
}

We're firing both async functions in parallel and have wrapped them in Promise.all. We're awaiting that and voila, we have the same result!

async.race

Similarly, we can use promises to recreate a race scenario without needing asyncjs library:

const promise1 = new Promise((resolve, reject) => {
  setTimeout(resolve, 500, 'one');
});

const promise2 = new Promise((resolve, reject) => {
  setTimeout(resolve, 100, 'two');
});

// Both resolve, but promise2 is faster
const result = await Promise.race([promise1, promise2]);
console.log(result);  // output = 'two'

However, asyncjs library provides some benefits that makes it worth it. One thing to keep in mind, it is possible to make your own custom solution and recreate everything from scratch. But it is generally not good idea to reinvent the wheel when there's already a library that does exactly what you want.

You might still need asyncjs

We have seen a few scenarios where it doesn't make much sense to install asyncjs library. But there are other use-cases where asyncjs can prove worthy and save you from writing your own custom solutions.

async.queue

This queue utility helps you write a worker function and provide a set of tasks to be processed by the worker function. Tasks are run in parallel upto a max limit known as concurrency limit. Tasks are picked up as soon as the concurrent workers running becomes less than the concurrency limit.

const async = require('async');

// specify how many worker execute task concurrently in the queue
const concurrent_workers = 1;

const queue = async.queue((object, callback) => {
  let date = new Date();
  let time = date.toISOString();

  // Log processing start time
  console.log(`Start processing movie ${object.movie} at ${time}`);

  // simulated async operation, can be network/DB interaction
  setTimeout(() => {
    date = new Date();
    time = date.toISOString();

    // Log processing end time
    console.log(`End processing movie ${object.movie} at ${time} \n`);
    callback(null, object.movie);
  }, 1000);
}, concurrent_workers);

queue.drain(function () {
  console.log('all items have been processed');
});

// add total of 8 tasks to be processed by the worker function
for (let i = 0; i < 8; i++) {
  queue.push({ movie: `Spiderman ${i}`, excitement: `${100 * i}` });
  console.log(`queue length: ${queue.length()}`);
}

Feel free to play around by tweaking the concurrent_workers number and see how it affects the async operations being processed. Playground link available here.

This is very useful in making sure that you don't attempt to run more tasks in parallel than your CPU/disk can take. Remember, the parallel aspect is only for the I/O and timers. If all of your tasks have I/O and you're running unlimited number of them in parallel, you're server will crash because of high Disk I/O usage and resource starvation.

async.queue provides a good use-case of throttling applications because of the ability to set a max cap on the number of parallel execution.

Check out async.priorityQueue which is similar to async.queue but offers the priority mechanism to make sure higher priority tasks don't starve out because of low priority tasks.

async.retry

It is sometimes possible that a request fails with no fault of our application (eg. network connection issue). You can use async.retry to make the same request X number of times until a success response is received. For example, trying and failing the same request 3 times gives us certainty in our judgments of service behavior.

async.retry(
  {times: 5, interval: 100},
  someAPIMethod,
  function(err, result) {
    // process the result
});

In above example, we're firing someAPIMethod 5 times with a 100ms interval. Callback is immediately called with the successful result if any method succeeds. In case no method success, callback is called with an error.

There are other control flows in asyncjs which can come in really handy, you can check them out here.

Conclusion

This was a short overview of asyncjs library, some of the control flows it provides and how we can replicate the same flows using async/await. We also looked at a few cases where using asyncjs can prove really helpful and saves you from reinventing the wheel.

I hope it gave you some perspective on the benefits of the library and how we should understand our specific use-case before jumping onto 3rd party solutions (one commit is enough sometimes 🙃)

What next?

The documentation of asyncjs is quite straightforward and easy to read. As we've only seen a couple of use cases in this article, I'd recommend to go the asyncjs documentation and check out other possibilities with the library. You can also try to replicate the same using async/await to solidify your understanding of where the library might still make sense.

How To Use PostgreSQL Triggers for Automated Processing of Data

Rishabh Rawat — Sat, 05 Feb 2022 09:30:52 +0000

Database triggers are a way to run a piece of code when a predefined operation occurs on the database. You can relate to triggers as event handlers in javascript. In javascript, you can set up an event handler for a button click. Database triggers are just like that.

In this article, we'll go through the basics of setting up triggers in PostgreSQL and a few examples to see how they can be used. Triggers can be created for data changes or database events. We'll be looking at data changes i.e., triggers that run BEFORE/AFTER INSERT, UPDATE, DELETE operations on a table.

The article is divided into the following sections:

What is a trigger
Types of triggers
Trigger function
2 steps to write a trigger in PostgreSQL
Data visibility in triggers
Examples
Drop trigger
Conclusion

What is a trigger

Taking from the docs, a trigger is a specification that the database should automatically execute a particular function whenever a certain type of operation is performed.
We can use triggers on tables (partitioned or not), views, and foreign tables.

Triggers can be defined to execute either before or after any INSERT, UPDATE, or DELETE operation, either once per modified row, or once per SQL statement.

A trigger must have a trigger function that runs whenever a supported operation occurs. Creating a trigger includes providing the trigger function. It takes no arguments and returns the type trigger.

NOTE: The trigger function receives its input arguments through a particular variable inbuilt into PostgreSQL. We don't need to pass any function arguments in the definition.

Types of triggers

There are a lot variations in how a trigger gets invoked. Let's look at them:

Row level triggers
Statement level triggers

Row-level triggers run for each row in the table whereas statement-level triggers run once per transaction.
For example, if a single transaction updates a thousand rows then the statement-level trigger will only run once but the row-level trigger will run a thousand times.

Triggers are also classified according to whether they fire before, after, or instead of the operation. These are referred to as BEFORE triggers, AFTER triggers, and INSTEAD OF triggers respectively.

For per-row triggers:

The return value is ignored for row-level triggers fired AFTER an operation so they can return NULL.
BEFORE level trigger can return NULL to skip the operation for the current row. This instructs the executor to not perform the row-level operation that invoked the trigger (the insertion, modification, or deletion of a particular table row).
For row-level BEFORE INSERT/UPDATE triggers, the returned row becomes the row that will get inserted or updated. It allows the trigger function to modify the row being inserted or updated.
Trigger functions invoked by per-statement triggers should always return NULL.

NOTE: Triggers in a transaction execute in the same transaction (BEFORE or AFTER). Doing a rollback for those X transactions means all the trigger changes will be undone in the same X transactions.

Trigger function

A trigger function is created with the CREATE FUNCTION command, declaring it as a function with no arguments and a return type of trigger for data change triggers.
Special local variables named TG_something are automatically defined to describe the condition that triggered the call.

Trigger functions can be written in most of the procedural languages including PL/pgsql, Perl, Python.
With PL/pgSQL function as a trigger, there are some variables automatically created at the top-level block.

Note that the function must be declared with no arguments even if it expects to receive some arguments specified in CREATE TRIGGER.
Such arguments are passed via TG_ARGV.

We'll take a look at some of the common ones:

NEW

Holds the new database row for INSERT/UPDATE operations in row-level triggers. This variable is null in statement-level triggers and for DELETE operations.

OLD

Holds the old database row for UPDATE/DELETE operations in row-level triggers. This variable is null in statement-level triggers and for INSERT operations.

TG_WHEN

A string of BEFORE, AFTER, or INSTEAD OF, depending on the trigger's definition.

TG_LEVEL

Indicates whether trigger is ROW or STATEMENT level.

TG_OP

Shows the operation that triggered the trigger. Can be INSERT, UPDATE, DELETE, or TRUNCATE.

TG_ARGV[]

The arguments from the CREATE TRIGGER statement. The index counts from 0. Invalid indexes (less than 0 or greater than or equal to tg_nargs) result in a null value.

2 steps to write a trigger in PostgreSQL

A fully functioning trigger in PostgreSQL can be created in just two steps:

Define a trigger function
Create the trigger using that trigger function

We'll create a trigger function first which can be used by a trigger:

CREATE OR REPLACE FUNCTION function_name()
  RETURNS trigger
  LANGUAGE plpgsql
AS $$
DECLARE
  -- declare variables if needed
BEGIN
  -- function body goes here
END;
$$;

Now that we have the trigger function, we can create the trigger:

CREATE OR REPLACE TRIGGER name
  { BEFORE | AFTER } { event }
  ON table_name
  [ FOR [ EACH ] { ROW | STATEMENT } ]
  [ WHEN ( condition ) ]
  EXECUTE { FUNCTION | PROCEDURE } function_name ( arguments )

CREATE OR REPLACE TRIGGER will either create a new trigger or replace an existing trigger. If you don't want that, use CREATE TRIGGER instead. It won't create the trigger if it already exists.

The trigger will be associated with the specified table and will execute the specified trigger function when certain operations are performed on that table BEFORE/AFTER the operation has happened.

NOTE: The same trigger function can be used to create multiple triggers if written properly.

You can also put conditions on when the trigger should fire using the WHEN clause.
In row-level triggers, the WHEN condition can examine the old and/or new values of columns of the row.
It is not useful in the case of statement-level triggers because they cannot refer to any values in the table.

NOTE: If multiple triggers of the same kind are defined for the same event, they will be fired in alphabetical order by name.

Some use-cases for different triggers

Row-level BEFORE triggers are used for checking or modifying the data that will get inserted or updated.
For example, a BEFORE trigger might get used to insert the current time into a timestamp column or to check that two elements of the row are consistent. It can also act as a validation stage where you reject the unwanted data (eg. null checks).

Row-level AFTER triggers are often used to propagate the updates to other tables or make consistency checks against other tables.
Reason? AFTER trigger always sees the final value of the row while a BEFORE might not. There might be other BEFORE triggers firing after it.

If possible, using the BEFORE trigger is better from a performance standpoint. We shouldn't save the information about the operation until the end of a statement if not needed.

NOTE: If a trigger function executes SQL commands then these commands might fire triggers again. This is known as cascading triggers. There is no direct limitation on the number of cascade levels.
The Cascades can cause a recursive invocation of the same trigger.

For example, an INSERT trigger might execute a command that inserts an additional row into the same table, causing the INSERT trigger to be fired again.
It is the trigger programmer's responsibility to avoid infinite recursion in such scenarios.

Data visibility in triggers

There are many use-cases for triggers and accessing the new or old values of a row in a table is one of them.
The ability to look into the data allows us to perform various actions that wouldn't be possible otherwise.
Audit logs, check constraints, and version history are some of the common and straightforward use-cases.

Keeping this in mind, it is important to know when we can access the old and new data of a row in the table.

In a statement-level trigger, none of the changes are visible to BEFORE triggers, whereas all changes are visible to AFTER triggers.
Any modifications to the data that happen during INSERT, UPDATE, DELETE is not visible to BEFORE level triggers because the change hasn't happened yet.
BEFORE trigger can see the effects of previous runs of BEFORE triggers.
All the changes are visible to AFTER triggers because the change has already happened.

Examples

Let's take a look at some examples showing different use cases of triggers. These are very straightforward examples, but they are not the only use cases.

Checking constraints

We can set a check constraint on a table to ensure that certain column values are not null or have certain values.
This can be achieved with the help of a BEFORE INSERT trigger.

CREATE OR REPLACE FUNCTION name_null_checker() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
    IF NEW.name IS NULL THEN
        RAISE EXCEPTION 'Name cannot be null';
    END IF;
    RETURN NEW;
END;
$$;

We have written a basic trigger function that checks if the name of the book is null and throws an exception if it is.
With this step, we have created a re-usable trigger function that can be used by multiple triggers to check the name constraint.
Properly written, this trigger function would be independent of the specific table it is triggering on.

Now, let's create a trigger that will fire BEFORE an INSERT operation on the books table.

CREATE TRIGGER name_null_checker_trigger BEFORE INSERT ON books
FOR EACH ROW EXECUTE FUNCTION name_null_checker();

And now we have a basic trigger in place to perform constraint check on books table.

Audit logging

Audit logs are a way to keep track of changes to a table. We will set up a trigger to log all INSERT, UPDATE and DELETE on the books table.

We need to create a books audit table to store the audit logs. Let's first create it:

CREATE TABLE IF NOT EXISTS books_audit_store (
  name VARCHAR(128) UNIQUE NOT NULL,
  price float(2) NOT NULL,
  rating INT NOT NULL,
  operation VARCHAR(64) NOT NULL,
  modified_at TIMESTAMP NOT NULL
);

Now that we have our books_audit_store table we can create the trigger function which will perform the INSERT SQL command:

CREATE OR REPLACE FUNCTION audit_logger()
  RETURNS trigger
  LANGUAGE plpgsql
AS $$
BEGIN
  INSERT INTO books_audit_store (name, price, rating, operation, modified_at) 
    VALUES (NEW.name, NEW.price, NEW.rating, TG_OP, now());
  RETURN NEW;
END;
$$;

Finally, creating the trigger on books_audit_store table:

CREATE TRIGGER audit_logger_trigger
AFTER UPDATE ON books
FOR EACH ROW
WHEN (OLD.* IS DISTINCT FROM NEW.*)
EXECUTE FUNCTION audit_logger();

Here's the books table before update:

We've updated the "Scrum" book title to be "Scrum master":

As a result of the update on books table with a different value, we got our audit log entry in books_audit_store table:

Drop trigger

A trigger can be dropped by issuing a DROP TRIGGER statement.

DROP TRIGGER triggerName ON tableName;

To drop the trigger, the user must be the owner of the table on which the trigger is defined.

We can add IF EXISTS to the statement to drop the trigger only if it exists:

DROP TRIGGER IF EXISTS triggerName ON tableName;

NOTE: In the SQL standard, trigger names are not local to tables, so you don't need to specify the table name in the DROP TRIGGER statement.

Conclusion

This is just the introduction to triggers and how you can use them with functions to create post processors, validators, and audit loggers.
The possibilities are endless, going through the official docs of CREATE TRIGGER would be a good next step. It has some important notes and examples which you can explore.

I'd be writing about more extensive applications of triggers once this information fits better with my current understanding of databases. Just connecting the dots would be a good start.

Learn how to use group in mongoDB aggregation pipeline (with Exercise)

Rishabh Rawat — Wed, 19 Jan 2022 15:58:16 +0000

The group stage in the MongoDB aggregation pipeline helps us group data using any field in the MongoDB document. It is one of the most important and commonly used stages in the MongoDB aggregation pipeline.

In this article, we'll look at the $group stage and use various features that it provides. We'll be working with a sample collection of movies. There's going to be a playground link for each query so you can practice and learn by doing.

There's also an exercise at the end of this article for you to try out. It will help you solidify your understanding once you've finished this article.

Here are the things we will cover in this article:

Find distinct using group by ♟
Group by using multiple fields 🎳
Using accumulator functions 🧩
Using with $project ⚙️
Sorting the results 📈
$group vs $project stage 🪞
Thing to note 📝
Conclusion 🎬
Exercise 🎐

Establishing the data

Before we jump into the aggregation pipeline and the group stage, we need some data to work with. I'm taking an example Movies collection for understanding the concept here. Again, there'll be links to the playground for each query throughout the article.

Here's the Movies collection with only 5 documents containing random data:

{
  "name": "Spidey One way home",
  "release_year": "2021",
  "rating": 9,
  "starring": [
    "Tom Hanks",
    "Tom Holland",
    "Mark Zucks",
    "Samy"
  ],
  "runtime": 120,
  "totalReviews": 2000,
  "director": "Jon What"
},
{
  "name": "The Arrival of a Train",
  "release_year": "1896",
  "rating": 6,
  "starring": [
    "Shawn Ching",
    "Looker Blindspot",
    "Tom Hanks"
  ],
  "runtime": 115,
  "totalReviews": 720,
  "director": "Ricky"
},
{
  "name": "Lost persuit of adventure",
  "release_year": "2005",
  "rating": 7.1,
  "starring": [
    "Jimmy simmon",
    "Catarina"
  ],
  "runtime": 150,
  "totalReviews": 823,
  "director": "Ricky"
},
{
  "name": "Jungle Warrior",
  "release_year": "2016",
  "rating": 5.9,
  "starring": [
    "Stormer",
    "Carmony",
    "Tom Hanks"
  ],
  "runtime": 150,
  "totalReviews": 1368,
  "director": "Whim Wailer"
},
{
  "name": "The Last of the us all",
  "release_year": "2005",
  "rating": 8.5,
  "starring": [
    "Samy",
    "George wise",
    "Pennywise"
  ],
  "runtime": 120,
  "totalReviews": 1800,
  "director": "Jon What"
}

Now that we have our sample collection, it's time to explore the $group stage ⚡

Find distinct using group by

To find the distinct items in a collection we can use the group stage on any field that we want to group by. This field will be unique in the output. Let's group the movies by their release year:

{
  $group: {
    _id: "$release_year"
  }
}

(playground link)

Here's the output of the above query. Note that we only got unique release year values in the output.

[
  { "_id": "1896" },
  { "_id": "2016" },
  { "_id": "2021" },
  { "_id": "2005" }
]

Group by using multiple fields

Similar to grouping by a single field, we might want to group the data with more than one field as per our use case. MongoDB aggregation pipeline allows us to group by as many fields as we want.

Whatever we put inside the _id field is used to group the documents i.e., it returns all the fields present inside the _id field and groups by all of them.

Let's group the movies by their release year and their runtime:

{
  $group: {
    _id: {
      "release_year": "$release_year",
      "runtime": "$runtime"
    }
  }
}

(playground link)

Grouping by release year and their runtime gives us this output:

[
  {
    "_id": {
      "release_year": "2005",
      "runtime": 150
    }
  },
  {
    "_id": {
      "release_year": "2021",
      "runtime": 120
    }
  },
  {
    "_id": {
      "release_year": "2016",
      "runtime": 150
    }
  },
  {
    "_id": {
      "release_year": "2005",
      "runtime": 120
    }
  },
  {
    "_id": {
      "release_year": "1896",
      "runtime": 115
    }
  }
]

Instead of using a single field to group by, we are using multiple fields in above scenario. The combination of release year and runtime acts as the unique identifier for each document.

Using accumulator functions

There are a lot of accumulator functions available in the group stage which can be used to aggregate the data. They help us carry out some of most common operations on the grouped data. Let's take a look at some of them:

$count accumulator

$count accumulator is used to count the number of documents in the group. This can be combined with our group by query to get the total number of documents in the group.

Let's apply this to our movies collection:

{
  $group: {
    _id: "$release_year",
    totalMovies: { $count: {} }
  }
}

(playground link)

We'll get the total movies released in each year:

[
  {
    "_id": "2016",
    "totalMovies": 1
  },
  {
    "_id": "1896",
    "totalMovies": 1
  },
  {
    "_id": "2021",
    "totalMovies": 1
  },
  {
    "_id": "2005",
    "totalMovies": 2
  }
]

$sum accumulator

We can use the $sum accumulator to add up all the values in a field. Let's group the movies by their rating and sum up the reviews to understand if there's a correlation between movie rating and the number of reviews.

{
  $group: {
    _id: "$rating",
    totalMovies: {
      $sum: "$totalReviews"
    }
  }
}

(playground link)

And here we can see that there is a slight correlation between the number of reviews and the movie rating:

[
  {
    "_id": 9,
    "totalMovies": 2000
  },
  {
    "_id": 6,
    "totalMovies": 720
  },
  {
    "_id": 5.9,
    "totalMovies": 1368
  },
  {
    "_id": 8.5,
    "totalMovies": 1800
  },
  {
    "_id": 7.1,
    "totalMovies": 823
  }
]

$avg accumulator

We might want to examine which year has the highest average movies rating for analytical purposes. Let's see how we can get those stats from our data:

{
  $group: {
    _id: {
      year: "$release_year",
    },
    avgRating: {
      $avg: "$rating"
    }
  }
}

(playground link)

We are first grouping the movies by the release year and then calculating the average rating for each release year. Here's the output of the above query:

[
  {
    "_id": { "year": "2016" },
    "avgRating": 5.9
  },
  {
    "_id": { "year": "1896" },
    "avgRating": 6
  },
  {
    "_id": { "year": "2021" },
    "avgRating": 9
  },
  {
    "_id": { "year": "2005" },
    "avgRating": 7.8
  }
]

$push accumulator

We want to look at all the ratings movies received for every release year. Let's use the $push accumulator to get all the movie names for each year:

{
  $group: {
    _id: {
      year: "$release_year",
    },
    ratings: {
      $push: "$rating"
    }
  }
}

(playground link)

All the movie ratings for each release year are pushed into an array:

[
  {
    "_id": { "year": "1896" },
    "ratings": [ 6 ]
  },
  {
    "_id": { "year": "2016" },
    "ratings": [ 5.9 ]
  },
  {
    "_id": { "year": "2021" },
    "ratings": [ 9 ]
  },
  {
    "_id": { "year": "2005" },
    "ratings": [ 7.1, 8.5 ]
  }
]

$addToSet accumulator

You can consider this to be like the $push accumulator. $addToSet only adds the value to the array if it doesn't exist already. This is the only difference between $addToSet and $push.

Let's group by rating and see which (unique) release years produced each:

{
  $group: {
    _id: {
      rating: "$rating"
    },
    releasedIn: {
      "$addToSet": "$release_year"
    }
  }
}

(playground link)

We get ratings along with their unique release years:

[
  {
    "_id": { "rating": 8.5 },
    "releasedIn": [ "2005" ]
  },
  {
    "_id": { "rating": 7.1 },
    "releasedIn": [ "2005" ]
  },
  {
    "_id": { "rating": 9 },
    "releasedIn": [ "2021" ]
  },
  {
    "_id": { "rating": 6 },
    "releasedIn": [ "1896" ]
  },
  {
    "_id": { "rating": 5.9 },
    "releasedIn": [ "2016" ]
  }
]

$min accumulator

Let's say we want to find out successful release years for the movies. A year is considered successful if the all the movies released during that year have rating greater than 7. Let's use the $min accumulator to get the successful years:

{
  $group: {
    _id: {
      year: "$release_year"
    },
    minRating: { $min: "$rating" }
  }
},
{
  "$match": {
    minRating: { $gt: 7 }
  }
}

(playground link)

We have grouped the movies collection using the release_year field.
In addition to that, we have added minRating field which maintains the minimum rating for each release year.
We have also applied a $match stage to filter out the years which don't have a minimum rating greater than 7.

[
  {
    "_id": {
      "year": "2021"
    },
    "minRating": 9
  },
  {
    "_id": {
      "year": "2005"
    },
    "minRating": 7.1
  }
]

$first accumulator

This accumulator is different from the $first array operator which gives first element in an array. For each grouped documents, $first accumulator gives us the first one.

Let's fetch the highest rated movie for every release year. Since we want to get the highest rated document from the each group, we need to sort the documents before passing them to the group stage.

{
  "$sort": {
    "release_year": 1,
    "rating": -1
  }
},
{
  $group: {
    _id: "$release_year",
    highestRating: {
      $first: "$rating"
    }
  }
}

(playground link)

We are sorting using two fields here, release_year and rating. Let's understand the output of sort stage first:

[
  {
    "rating": 6,
    "release_year": "1896"
  },
  {
    "rating": 8.5,
    "release_year": "2005"
  },
  {
    "rating": 7.1,
    "release_year": "2005"
  },
  {
    "rating": 5.9,
    "release_year": "2016"
  },
  {
    "rating": 9,
    "release_year": "2021"
  }
]

(playground link)

The output is first sorted on the basis of ascending release year and then for each year, the movies are sorted in descending order of rating.

This sorted output is then passed to the group stage which groups the documents by their release year. For example, group stage is working with two documents for release year 2005:

{
  "rating": 8.5,
  "release_year": "2005"
},
{
  "rating": 7.1,
  "release_year": "2005"
}

Let's call these "shortlisted documents" for release year 2005. This happens for all (unique) release years. Group stage picks the first element from these shortlisted documents (which has the highest rating because ratings are sorted in descending order).

Combining the sort and group stages, here's the final output of the query:

[
  {
    "_id": "2016",
    "highestRating": 5.9
  },
  {
    "_id": "1896",
    "highestRating": 6
  },
  {
    "_id": "2021",
    "highestRating": 9
  },
  {
    "_id": "2005",
    "highestRating": 8.5
  }
]

NOTE: Passing sorted documents to $group stage does not guarantee that the order will be preserved.

Using with $project

The movie rating is a floating point number. We'll round that off to the nearest integer to get the movie rating as a whole number. Let's also group movies by their modified ratings:

{
  "$project": {
    rating: {
      "$round": "$rating"
    }
  }
},
{
  $group: {
    _id: "$rating",
    movies: {
      $sum: 1
    }
  }
}

(playground link)

We used $project stage to round off the rating to the nearest integer.
We used $group stage to group the movies by their modified rating.

Here's the output of the above query:

[
  {
    "_id": 7,
    "movies": 1
  },
  {
    "_id": 8,
    "movies": 1
  },
  {
    "_id": 9,
    "movies": 1
  },
  {
    "_id": 6,
    "movies": 2
  }
]

The possibilities are endless. You can combine many other stages, perform some filters, put conditions or even $$REMOVE the documents.

Sorting the results

The year with the highest movie minutes might give us some insights on movies production and its correlation with audience attention spans over the years. So let's understand how to achieve that:

{
  $group: {
    _id: "$release_year",
    totalRuntime: {
      "$sum": "$runtime"
    }
  }
},
{
  "$sort": {
    "totalRuntime": -1
  }
}

(playground link)

We are fetching the total runtime of all the movies released in a particular year and then sorting them in descending order with the help of $sort stage:

[
  {
    "_id": "2005",
    "totalRuntime": 270
  },
  {
    "_id": "2016",
    "totalRuntime": 150
  },
  {
    "_id": "2021",
    "totalRuntime": 120
  },
  {
    "_id": "1896",
    "totalRuntime": 115
  }
]

It is evident from this query that the attention spans of the target audience have been decreasing in non-uniform way over the years.

$group vs $project stage

We have an n:1 relationship between input and output documents in the group stage. But, we have a 1:1 relationship in the $project stage.

In group stage we usually get a count, sum, average of documents based on the grouping key (or _id), or even build an array. All of these operations take n number of documents and the output of group is a single document with the aggregated values.

On the other hand, we include/exclude fields, perform field transformations within a single document in case of project stage in aggregation pipeline,

Thing to note

$group stage has a limit of 100 megabytes of RAM

If you're working with a massive dataset and you receive an error during group stage execution, you might be hitting the memory limit. If you want to increase it, use allowDiskUse option to enable the $group stage to write to temporary files on disk.

The reason for this issue is very well stated in the mongoDB docs:

NOTE: Pipeline stages operate on streams of documents with each pipeline stage taking in documents, processing them, and then outputting the resulting documents. Some stages can't output any documents until they have processed all incoming documents. These pipeline stages must keep their stage output in RAM until all incoming documents are processed. As a result, these pipeline stages may require more space than the 100 MB limit.

Conclusion

And that was how the group stage works in mongoDB aggregation pipeline. We looked at how we can group data using single fields (distinct count), multiple fields, sort them, how we can carry out complex computations by adding conditions to group stage and the subtle difference between group and project stage. I hope you find this useful and interesting.

Let me know your thoughts and feedback on Twitter.

Exercise

To make sure you understand the concepts, I have curated a couple of questions related to what we've learned in this article. You can download exercise PDF below. It also contains working mongoDB playground links containing the answers for all the questions. Be honest, don't cheat 🙂.

5 quick questions on group stage with answers

How to use project in MongoDB aggregation pipeline

Rishabh Rawat — Mon, 20 Dec 2021 14:46:50 +0000

If you have ever worked with MongoDB, chances are you've heard of or even come across the aggregation pipeline that comes with it. Aggregation pipeline in MongoDB is a very powerful feature to get insights out of your data that is not possible with simple find operation.

In this article, we're going to take at the project stage. We do have the projection support in find queries as well but it cannot achieve what project stage in the aggregation pipeline can. Today we're going to explore some of the common use-cases for the project stage:

Projecting all fields
Working with arrays
Naming field in project stage
Filtering fields
Literal fields
Conditionally removing fields

Establishing the data

Before we jump into the aggregation pipeline and the project stage, we would need some data to work with. I'm taking an example Restaurants collection for understanding the concept here. I'll be linking to a MongoDB playground for each query so that you can play around and see how it affects the output.

Here's the Restaurants collection with only 5 documents containing random data:

{
    name: 'Thai Restaurant',
    location: 'London',
    notOpenOnDays: ['Monday', 'Thursday'],
    entryFee: '$5',
    rating: '3',
},
{
    name: 'Indian.com',
    location: 'Broughton',
    notOpenOnDays: ['Wednesday'],
    entryFee: '$0',
    rating: '4',
},
{
    name: '23rd Street Pizza',
    location: 'Fallowfield',
    notOpenOnDays: ['Sunday'],
    entryFee: '$2',
    rating: '4',
},
{
    name: '7 STAR Pizza',
    location: 'Newcastle',
    notOpenOnDays: ['Saturday', 'Sunday'],
    entryFee: '$10',
    rating: '2',
},
{
    name: 'A Cake A Shake',
    location: 'London',
    notOpenOnDays: ['Monday'],
    entryFee: '$1',
    rating: '4',
},

Projecting all fields

In the MongoDB aggregation pipeline, a pipeline stage has access to only those fields which are included in the previous stage.

So if you want to include all fields, you can include everything in the project stage so that it gets passed to the next one. Any field missed is not going to make it to the next stage of the pipeline.

MongoDB gives us a way to state which fields we want to include and exclude. So if you want to include only the name, the project stage would be:

{
    $project: { name: 1 }
}

You can also put true or any number greater than 1 to show inclusion of a field but let's just stick to 1 for simplicity & conciseness.

Similarly, if you want to exclude the field you'd do the opposite:

{
    $project: { name: 0 }
}

And, if you want to include everything in the Restaurants collection, here's what the project stage would look like:

{
    $project: {
        name: 1,
        location: 1,
        notOpenOnDays: 1,
        entryFee: 1,
        rating: 1,
    }
}

NOTE: Including a field in the project stage implicitly means excluding all the other fields and vice-versa.

Working with arrays

Creating arrays

Creating a new array field is just as creating any non-array field, the only difference is that the field value is wrapped in [] to make it an array.

We can do it like this:

{
    $project: {
        review: [ "$rating" ]
    }
}

Now review field will be an array with rating values in it.

Extracting element(s) at specific index

To get an array element at a specific index, we use use $arrayElemAt like shown below,

{
    $project: {
        fieldName: { $arrayElemAt: ["$arrayFieldName", index] }
    }
}

With the latest MongoDB version 4.4, we now also have dedicated operators to get first and last elements in an array. The operators are $first and $last.

To get the first element,

{
    $project: {
        review: [ "$rating" ]
    }
},
{
    $project: {
        newReview: { $first: "$review" },
        _id: 0
    }
}

We are putting the restaurant rating in an array named review and in the next project stage we are extracting the first element of the array created in the stage before. This gives us:

{ "newReview" : 3 }
{ "newReview" : 4 }
{ "newReview" : 4 }
{ "newReview" : 2 }
{ "newReview" : 4 }

NOTE: We can chain as many project stages as we want in an aggregation pipeline.

To get the last element,

{
    $project: {
        review: [ "$rating" ]
    }
},
{
    $project: {
        newReview: { $last: "$review" },
        _id: 0
    }
}

This will give the same output as there is only one element in the array currently, let's add some values to the reviews array.

{
    $project: {
        review: [ 1, 2, 3, "$rating" ]
    }
},
{
    $project: {
        newReview: { $last: "$review" },
        _id: 0
    }
}

We would get the ratings as the output of this as we have it in the last position:

{ "newReview" : 3 }
{ "newReview" : 4 }
{ "newReview" : 4 }
{ "newReview" : 2 }
{ "newReview" : 4 }

Filtering values

Filter lets you write custom filter which is applied to every element in the array and only the elements which pass the filter are passed as output to next stage.

  {
    $project: {
      _id: 0,
      openOnWeekend: {
        $filter: {
          input: "$notOpenOnDays",
          as: "notOpen",
          cond: {
            $and: [
              { $ne: ["$$notOpen", "Saturday"] },
              { $ne: ["$$notOpen","Sunday"] },
            ]
          }
        }
      }
    }
  }

Output of the above aggregation query would only contain the restaurants which are open on weekends:

[
  {
    "openOnWeekend": [ "Monday", "Thursday" ]
  },
  {
    "openOnWeekend": [ "Wednesday" ]
  },
  {
    "openOnWeekend": []
  },
  {
    "openOnWeekend": []
  },
  {
    "openOnWeekend": [ "Monday" ]
  }
]

NOTE: We use $ for values present in our document and $$ for fields that are introduced within a pipeline stage.

Transforming data

$map operator is just like the array.map() in javascript, it takes an input array, applies an expression on each item and return the transformed array.

Here you can directly transform data as compared to filter where you need to provide a condition to handle the data appropriately.

Let's transform the notOpenOnDays array and shorten the weekday names:

  {
    $project: {
      _id: 0,
      newPrice: {
        $map: {
          input: "$notOpenOnDays",
          as: "noOpenOn",
          in: {
            $substrCP: [ "$$noOpenOn", 0, 3 ]
          }
        }
      }
    }
  }

We have the substring operator to get only first 3 characters and remove the rest:

[
  {
    "newPrice": [ "Mon", "Thu" ]
  },
  {
    "newPrice": [ "Wed" ]
  },
  {
    "newPrice": [ "Sun" ]
  },
  {
    "newPrice": [ "Sat", "Sun" ]
  },
  {
    "newPrice": [ "Mon" ]
  }
]

There are a bunch of other array expression operators, you can refer them here.

Naming field in project stage

The aggregation pipeline provides great flexibility when it comes to naming fields. Any name you specify in the project stage acts as the "new" name for that field from that point forwards.

You give an existing field a new name which to effectively rename it:

{
    $project: {
        closedOn: "$notOpenOnDays"
    }
}

Now the output will contain closedOn field with notOpenOnDays array as value.

Literal fields

By default, whatever you put in the project stage is treated as an expression and it is evaluated to a value. You can use $literal operator if you want MongoDB to treat any field's value like a constant or literal.

Let's say we want to find out the restaurants where the entry fee is exactly equal to "$1", we can leverage literal operator to achieve this:

  {
    "$project": {
      affordable: {
        $eq: [
          "$entryFee",
          {
            $literal: "$1"
          }
        ]
      }
    }
  },
  {
    "$project": {
      _id: 0,
      status: {
        "$cond": {
          "if": "$affordable",
          "then": "restaurant is affordable 🍕🍟",
          "else": "Not affordable 🙈"
        }
      }
    }
  }

You can chain as many project stages as you need in the aggregation pipeline. Output of the above pipeline evaluates to 👇🏻

[
  {
    "status": "Not affordable 🙈"
  },
  {
    "status": "Not affordable 🙈"
  },
  {
    "status": "Not affordable 🙈"
  },
  {
    "status": "Not affordable 🙈"
  },
  {
    "status": "restaurant is affordable 🍕🍟"
  }
]

Conditionally excluding fields

In the above example, we saw a bunch of redundant information in the output. We are only interested in knowing the affordable restaurants so it makes sense to not output the other ones.

With MongoDB 3.6, you can remove fields from the output conditionally. Let's modify our above project stage to only output affordable restaurants:

  {
    "$project": {
      name: 1,
      affordable: {
        $eq: [
          "$entryFee",
          {
            $literal: "$1"
          }
        ]
      }
    }
  },
  {
    "$project": {
      _id: 0,
      status: {
        "$cond": {
          "if": "$affordable",
          "then": "$name",
          "else": "$$REMOVE"
        }
      }
    }
  }

We're now including the name in the first project stage to make sure it is available in the next stage.
$cond operator is updated to show the name of only the affordable restaurants.

Output:

[
  {},
  {},
  {},
  {},
  {
    "status": "A Cake A Shake"
  }
]

Conclusion

So to summarize we have seen that the project stage in the MongoDB aggregation pipeline can do much more than just include & exclude fields. We can create arrays, modify its elements, perform filters, slice strings, conditionally remove fields and get data as per our needs.

If you think something is missing or have any questions, please post it down in the comments. I'd be happy to connect!

DEV Community: Rishabh Rawat

Understanding Offset and Cursor-Based Pagination in Node.js

Types of Pagination

Prerequisites

Installing Our Sample Node.js API

Understanding Offset-Based Pagination

Drawbacks of Offset-Based Pagination

Pros

Cons

Understanding Cursor-Based Pagination

Pros

Cons

Choosing the Right Pagination Strategy for Your Node.js App

How to Implement Cursor-Based Pagination

UX-Imposed Challenges

Wrapping Up

Build an Idempotent API in Node.js with Redis

What We'll Cover

What Is Idempotency?

Why Idempotency Is Important

Idempotent HTTP Methods

A Duplicate Processing Problem Using a Sample Node.js API

Adding Idempotency to Sequential Requests in Node.js

Idempotency for Parallel Duplicate Requests in Node.js

Real-World Applications

Wrapping Up

Throttling a Kafka Queue in Node.js

What We'll Cover

Prerequisites

Installing Kafka

Setting up AppSignal for Node.js

Why Use Kafka for Node.js?

Durability

Robustness

The Problem with Normal Consumption

Monitoring the Anomalies in Our Node.js Application

Manual Kafka Commits to the Rescue

Granular Control with Async Queue in Kafka

Putting It All Together

Real-World Kafka for Node Applications

Wrapping Up

How to Optimize MongoDB Performance for Node.js

What We'll Cover

Data Modelling in MongoDB

Starting with explain

Provisioning your Hardware Resources

Monitoring MongoDB Queries for Node.js

Managing Indexes

Limiting Query Output

Using Projection in MongoDB

Connection Pooling with Mongoose

Batch Operations

Soft Deletes

Caching Layer for Faster Access

Aggregation Best Practices in MongoDB

Wrapping Up

Tracking Errors in a Node.js Application

The Two Error Types

What Is Error Tracking?

How to Track Errors in Node.js

Creating a Custom Error Handler

Better Error Messages in Your Node.js App

Handling Programming Errors

How to Choose Log Levels for Your Errors in Node.js

Trace

Debug

Info

Warn

Error

Fatal

But Wait, What Errors Should I Track Exactly?

Restarting Gracefully When Needed

Learning from Your Node.js Errors with AppSignal

Best Practices to Reduce Errors in Node.js

Wrapping Up

Complete Guide to Multi-Provider OAuth 2 Authorization in Node.js

How it is structured

OAuth 2 Overview

Create API keys for all providers

Google

Starting with `explain`