DEV Community: Deepal Jayasekara

How Not To Measure Time in Programming

Deepal Jayasekara — Sat, 12 Mar 2022 00:02:54 +0000

a.k.a. how the likes of Date.now() will ruin your metrics.

If you fly between two airports that are in two different time zones, would you calculate the time it took for you to fly, by checking the difference between the ‘wall clock’ times of either airport?

If your answer is ‘no’, then let me explain why you shouldn’t do the same in your code, using a little but a very interesting experiment.

Continue reading on Deepal’s Blog »

Atomic Microservices Transactions with MongoDB Transactional Outbox

Deepal Jayasekara — Thu, 17 Feb 2022 00:23:54 +0000

Atomic Transactions with Polled Outbox

One of the drawbacks of the microservices architecture is the difficulty to implement atomic transactions. While this limitation can be overcome by using patterns such as Saga, there are certain situations where we can use an alternative approach to solve this problem without having to implement the rather complicated Saga pattern.

Let’s consider a scenario where we need to dispatch an event when a user verifies their email after signing up. The event will then be consumed by other services for reporting, updating the customer profile on a CRM system and sending a customer a welcome email.

Continue reading on Deepal’s Blog »

Use Streams to Build High-Performing Node.js Applications

Deepal Jayasekara — Wed, 09 Feb 2022 13:33:20 +0000

The moment you type something on a keyboard, read a file from a disk or download a file over the internet, a stream of information (bits) flows through different devices and applications.

If you learn to work with these streams of bits, you'll be able to build performant and valuable applications. For example, think of when you watch a video on YouTube. You don't have to wait until the full video downloads. Once a small amount buffers, it starts to play, and the rest keeps on downloading as you watch.

Node.js includes a built-in module called stream which lets us work with streaming data. In this article, we will explain how you can use the stream module with some simple examples. We'll also describe how you can build pipelines gluing different streams together to build performant applications for complex use cases.

Before we dive into building applications, it's important to understand the features provided by the Node.js stream module.

Let's get going!

Types of Node.js Streams

Node.js streams provides four types of streams:

Readable Streams
Writable Streams
Duplex Streams
Transform Streams

See the official Node.js docs for more detail on the types of streams.

Let's look at each stream type at a high level.

Readable Streams

A readable stream can read data from a particular data source, most commonly, from a file system. Other common uses of readable streams in Node.js applications are:

process.stdin - To read user input via stdin in a terminal application.
http.IncomingMessage - To read an incoming request's content in an HTTP server or to read the server HTTP response in an HTTP client.

Writable Streams

You use writable streams to write data from an application to a specific destination, for example, a file.

process.stdout can be used to write data to standard output and is used internally by console.log.

Next up are duplex and transform streams, which you can define as 'hybrid' stream types built on readable and writable streams.

Duplex Streams

A duplex stream is a combination of both readable and writable streams. It provides the capability to write data to a particular destination and read data from a source. The most common example of a duplex stream is net.Socket, used to read and write data to and from a socket.

It's important to know that readable and writable sides operate independently from one another in a duplex stream. The data does not flow from one side to the other.

Transform Streams

A transform stream is slightly similar to a duplex stream, but the readable side is connected to the writable side in a transform stream.

A good example would be the crypto.Cipher class which implements an encryption stream. Using a crypto.Cipher stream, an application can write plain text data into the writable side of a stream and read encrypted ciphertext out of the readable side of the stream. The transformative nature of this type of stream is why they are called 'transform streams'.

Side-note: Another transform stream is stream.PassThrough, which passes data from the writable side to the readable side without any transformation. Though this might sound trivial, Passthrough streams are very useful for building custom stream implementations and pipelines (e.g., creating multiple copies of one stream's data).

Read Data From Readable Node.js Streams

Once a readable stream is 'connected' to a source that generates data (e.g., a file), there are a few ways to read data through the stream.

First, let's create a sample text file named myfile, with 85 bytes of 'lorem ipsum' text:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur nec mauris turpis.

Now, let's look at two different methods of reading data from a readable stream.

1. Listen to 'data' Events

The most common way to read data from a readable stream is by listening to 'data' events emitted by the stream. The following program demonstrates this approach:

const fs = require("fs");
const readable = fs.createReadStream("./myfile", { highWaterMark: 20 });

readable.on("data", (chunk) => {
  console.log(`Read ${chunk.length} bytes\n"${chunk.toString()}"\n`);
});

The highWaterMark property, passed as an option to fs.createReadStream, determines how much data buffers inside the stream. The data is then flushed to the reading mechanism (in this case, our data handler). By default, readable fs streams have their highWaterMark set to 64kB. We deliberately override this to 20 bytes to trigger multiple data events.

If you run the above program, it will read 85 bytes from myfile in five iterations. You will see the following output in the console:

Read 20 bytes
"Lorem ipsum dolor si"

Read 20 bytes
"t amet, consectetur "

Read 20 bytes
"adipiscing elit. Cur"

Read 20 bytes
"abitur nec mauris tu"

Read 5 bytes
"rpis."

2. Use Async Iterators

An alternative way of reading data from a readable stream is by using async iterators:

const fs = require("fs");
const readable = fs.createReadStream("./myfile", { highWaterMark: 20 });

(async () => {
  for await (const chunk of readable) {
    console.log(`Read ${chunk.length} bytes\n"${chunk.toString()}"\n`);
  }
})();

If you run this program, you will get the same output as the previous example.

State of a Readable Node.js Stream

When a listener is attached to a readable stream's 'data' events, the stream switches to a 'flowing' state (unless it is explicitly paused). You can inspect the stream's flowing state using the stream object's readableFlowing property.

We can demonstrate this using a slightly modified version of our previous example with the 'data' handler:

const fs = require("fs");
const readable = fs.createReadStream("./myfile", { highWaterMark: 20 });

let bytesRead = 0;

console.log(
  `before attaching 'data' handler. is flowing: ${readable.readableFlowing}`
);
readable.on("data", (chunk) => {
  console.log(`Read ${chunk.length} bytes`);
  bytesRead += chunk.length;

  // Pause the readable stream after reading 60 bytes from it.
  if (bytesRead === 60) {
    readable.pause();
    console.log(`after pause() call. is flowing: ${readable.readableFlowing}`);

    // resume the stream after waiting for 1s.
    setTimeout(() => {
      readable.resume();
      console.log(
        `after resume() call. is flowing: ${readable.readableFlowing}`
      );
    }, 1000);
  }
});
console.log(
  `after attaching 'data' handler. is flowing: ${readable.readableFlowing}`
);

In this example, we read from myfile via a readable stream, but we temporarily 'pause' the data flow for 1s after reading 60 bytes from the file. We also log the value of the readableFlowing property at different times to understand how it changes.

If you run the above program, you will get the following output:

before attaching 'data' handler. is flowing: null
after attaching 'data' handler. is flowing: true
Read 20 bytes
Read 20 bytes
Read 20 bytes
after pause() call. is flowing: false
after resume() call. is flowing: true
Read 20 bytes
Read 5 bytes

We can explain the output as follows:

When our program starts, readableFlowing has the value null because we don't provide any mechanism of consuming from the stream.
After the 'data' handler is attached, the readable stream changes to 'flowing' mode, and readableFlowing changes to true.
Once 60 bytes are read, the stream is 'paused' by calling pause(), which, in turn, changes readableFlowing to false.
After waiting for 1s, the stream switches to 'flowing' mode again by calling resume(), changing readableFlowing to true. The rest of the file content then flows through the stream.

Processing Large Amounts of Data with Node.js Streams

Thanks to streams, applications do not have to keep large blobs of information in memory: small chunks of data can be processed as they are received.

In this section, let's combine different streams to build a real-life application that can handle large amounts of data. We'll use a small utility program that generates an SHA-256 of a given file.

But first, let's create a large 4GB dummy file for testing. You can do this using a small shell command, as follows:

On macOS: mkfile -n 4g 4gb_file
On Linux: xfs_mkfile 4096m 4gb_file

After creating our dummy 4gb_file, let's generate the SHA-256 hash of the file without using the stream module:

const fs = require("fs");
const crypto = require("crypto");

fs.readFile("./4gb_file", (readErr, data) => {
  if (readErr) return console.log(readErr);
  const hash = crypto.createHash("sha256").update(data).digest("base64");
  fs.writeFile("./checksum.txt", hash, (writeErr) => {
    writeErr && console.error(err);
  });
});

If you run the above code, you may get the following error:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (4294967296) is greater than 2 GB
at FSReqCallback.readFileAfterStat [as oncomplete] (fs.js:294:11) {
code: 'ERR_FS_FILE_TOO_LARGE'
}

The above error occurs because the JavaScript runtime cannot handle arbitrarily large buffers. The max size of a buffer that the runtime can handle depends on your operating system architecture. You can check this by using the buffer.constants.MAX_LENGTH variable in the built-in buffer module.

Even if we didn't see the above error, keeping large files in memory is problematic. The physical memory we have available will restrict the amount of memory our application can use. High memory usage can also cause poor application performance in terms of CPU usage, as garbage collection becomes expensive.

Reduce Your App's Memory Footprint Using `pipeline()`

Now, let's look at how we can modify our application to use streams and avoid encountering this error:

const fs = require("fs");
const crypto = require("crypto");
const { pipeline } = require("stream");

const hashStream = crypto.createHash("sha256");
hashStream.setEncoding("base64");

const inputStream = fs.createReadStream("./4gb_file");
const outputStream = fs.createWriteStream("./checksum.txt");

pipeline(inputStream, hashStream, outputStream, (err) => {
  err && console.error(err);
});

In this example, we use the streaming approach provided by the crypto.createHash function. It returns a "transform stream" object hashStream, generating hashes for arbitrarily large files.

To feed the file content into this transform stream, we have created a readable stream — inputStream — to 4gb_file using fs.createReadStream. We pipe the output from the hashStream transform stream to the writable outputStream and the checksum.txt, created using fs.createWriteStream.

If you run the above application, you will see that the checksum.txt file populates with the SHA-256 hash of our 4GB file.

Using `pipeline()` vs `pipe()` for Streams

In our previous example, we used the pipeline function to connect multiple streams. An alternative common approach is to use the .pipe() function, as shown below:

inputStream.pipe(hashStream).pipe(outputStream);

However, using .pipe() in production applications is not recommended for several reasons. If one of the piped streams is closed or throws an error, pipe() will not automatically destroy the connected streams. This can cause memory leaks in applications. Also, pipe() does not automatically forward errors across streams to be handled in one place.

pipeline() was introduced to cater for these problems, so it's recommended you use pipeline() instead of pipe() to connect multiple streams. We can rewrite the above pipe() example to use the pipeline() function, as follows:

pipeline(inputStream, hashStream, outputStream, (err) => {
  err && console.error(err);
});

pipeline() accepts a callback function as the last parameter. Any forwarded errors from any of the piped streams will call the callback, so it's easier to handle errors for all streams in one place.

Wrap Up: Reduce Memory and Improve Performance Using Node.js Streams

Using streams in Node.js helps us build performant applications that can handle large amounts of data.

In this article, we covered:

The four types of Node.js streams (readable, writable, duplex, and transform streams).
How you can read data from readable Node.js streams by either listening to 'data' events or using async iterators.
Reducing the memory footprint of your applications by using pipeline to connect multiple streams.

A quick, small word of warning: You likely won't encounter many situations where streams are a necessity, and a stream-based approach can increase the complexity of your application. Make sure you confirm that the benefits of using streams outweigh the complexity they'll bring.

I'd encourage you to read the official Node.js stream documentation to learn more and to explore more advanced use cases of streams out there.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Experimenting with HTTP/3 — How the Next Gen Web is being built on UDP

Deepal Jayasekara — Sat, 23 Oct 2021 23:32:57 +0000

HTTP3 is fundamentally different from its predecessors we knew. It ditches TCP and is built upon a novel UDP based protocol, QUIC.

Continue reading on Deepal’s Blog »

Abusing Network Protocols for Secret Communication

Deepal Jayasekara — Wed, 18 Aug 2021 18:47:28 +0000

How to hide your communication in plain sight with Network Covert Channels

Continue reading on Deepal’s Blog »

NodeJS Streams in Practice

Deepal Jayasekara — Sun, 25 Jul 2021 20:19:53 +0000

Turbo boost your Node.js application with Streams

Continue reading on Deepal’s Blog »

Starting Your Own Tech Blog

Deepal Jayasekara — Mon, 28 Jun 2021 21:52:23 +0000

Tips and tricks for building your technical writing portfolio

Continue reading on Deepal’s Blog »

A Visual Guide to NodeJS Streams

Deepal Jayasekara — Thu, 10 Jun 2021 01:18:28 +0000

In NodeJS, stream module provides the capability to work with streams. Even if you haven’t used stream module explicitly there are a lots…

Continue reading on Deepal’s Blog »

Understanding Async Resources with Async Hooks

Deepal Jayasekara — Sun, 16 May 2021 19:37:56 +0000

Advanced NodeJS Internals — Part 3

Continue reading on Deepal’s Blog »

JavaScript Event Loop vs Node JS Event Loop

Deepal Jayasekara — Sun, 16 Aug 2020 00:01:57 +0000

A guide to understanding the distinction between the event loop in browser and Node

Continue reading on Deepal’s Blog »

Five Misconceptions on How NodeJS Works

Deepal Jayasekara — Thu, 19 Mar 2020 01:31:42 +0000

Background Image Courtesy: https://wallpapercave.com/mythology-wallpapers

This article is based on a Brown Bag session I did at comparethemarket.com on “Five Misconceptions on How NodeJS Works”.

NodeJS was born in 2009 and it has gained massive popularity throughout the years because of one reason. It’s just JavaScript! Well, it’s a JavaScript runtime designed to write server-side applications, but the statement that “It’s just JavaScript” is not 100% true.

JavaScript is single-threaded, and it was not designed to run on the server-side where scalability was a critical requirement. With Google Chrome’s high-performance V8 JavaScript Engine, the super cool asynchronous I/O implementation of libuv, and with a few other spicy additions, Node JS was capable of bringing client-side JavaScript to the server-side enabling writing super-fast web servers in JavaScript that are capable of handling thousands of socket connections at a time.

Building Blocks of Node JS

NodeJS is a massive platform built with a bunch of interesting building blocks as the above diagram describes. However, due to the lack of understanding of how these internals pieces of Node JS work, many Node JS developers make false assumptions on the behaviour of Node JS and develop applications which lead to serious performance issues as well as hard-to-trace bugs. In this article, I’m going to describe five such false assumptions which are quite common among many Node JS developers.

Misconception 1 — EventEmitter and the Event Loop are related

NodeJS EventEmitter is intensively used when writing NodeJS applications, but there’s a misconception that the EventEmitter has something to do with the NodeJS Event Loop, which is incorrect.

NodeJS Event Loop is the heart of NodeJS which provides the asynchronous, non-blocking I/O mechanism to NodeJS. It processes completion events from different types of asynchronous events in a particular order.

(Please check out my article series on the NodeJS Event Loop, if you are not familiar with how it works!)

NodeJS Event Loop

In contrast, NodeJS Event Emitter is a core NodeJS API which allows you to attach listeners functions to a particular event which will be invoked once the event is fired. This behaviour looks like asynchronous because the event handlers are usually invoked at a later time than it was originally registered as an event handler.

An EventEmitter instance keeps track of all events and listeners associated with an event within the EventEmitter instance itself. It does not schedule any events in the event loop queues. The data structure where this information is stored is merely a plain old JavaScript Object where the object properties are the event names (or “types” as someone may call) and the value of a property is one listener function or an array of listener functions.

Oversimplified Diagram of how event handlers are attached to an event in EventEmitter

When the emit function is called on the EventEmitter instance, the emitter will SYNCHRONOUSLY invoke the listener functions registered to the event in a sequential manner.

If you consider the following snippet:

The output of the above snippet would be:

handler1: myevent was fired!
handler2: myevent was fired!
handler3: myevent was fired!
I am the last log line

Since the event emitter synchronously executes all the event handlers, the line I am the last log line won’t be printed until all the listener functions are invoked.

Misconception 2—All callback-accepting functions are asynchronous

Whether a function is synchronous or asynchronous depends on whether the function creates any asynchronous resources during the execution of the function. With this definition, if you are given a function, you can determine that the given function is asynchronous if it:

Calls a native JavaScript/NodeJS asynchronous function (e.g, setTimeout, setInterval, setImmediate, process.nextTick, etc.)
Performs a native NodeJS async function(e.g, async functions in child_process, fs, net, etc.)
Uses Promise API (includes the usage of async-await)
Calls a function from a C++ addon which is written to be asynchronous (e.g, bcrypt)

Accepting a callback function as an argument does not make a function asynchronous. However, usually asynchronous functions do accept a callback as the last argument (unless it’s wrapped to return a Promise). This pattern of accepting a callback and passing the results to the callback is called the Continuation Passing Style. You can still write a 100% synchronous function using the Continuation Passing Style.

Synchronous functions and Asynchronous functions have a significant difference in terms of how they use the stack during the execution.

Synchronous functions occupy the stack during the entire duration of its execution, by disallowing anyone else to occupy the stack until it returns.

In contrast, asynchronous functions schedule some async task and return immediately hence removing itself from the stack. Once the scheduled async task is completed, any callback provided will be called and the callback function will be the one who occupies the stack again. At this point, the function which initiated the async task will no longer be available on the stack since it has already returned.

With the above definition in your mind, try to determine whether the following function is asynchronous or synchronous.

In fact, the above function can be synchronous and asynchronous depending on the value passed to the data.

If data is a falsy value, the callback will be called immediately with an error. In this execution path, the function is 100% synchronous as it does not perform any asynchronous task.

If data is a truthy value, it’ll write data into myfile.txt and will call the callback after the file I/O operation is completed. This execution path is 100% asynchronous due to the async file I/O operation.

Writing function in such an inconsistent way (where the function behaves both synchronously and asynchronously) is highly discouraged because it will make an application’s behaviour unpredictable. Fortunately, these inconsistencies can easily be fixed as follows:

process.nextTick can be used to defer the invocation of callback function thereby making the execution path asynchronous.

Alternatively, you can use setImmediate instead of process.nextTick in this case, which will more or less give the same result. However, process.nextTick callbacks have a higher priority comparatively thereby making it faster than setImmediate.

If you need to learn more about the difference between process.nextTick and setImmediate, have a look at the following article from my Event Loop series.

Timers, Immediates and Process.nextTick— NodeJS Event Loop Part 2 | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Oct 9, 2020 ・
blog.insiderattack.net

Misconception 3— All CPU-intensive functions are blocking the event loop

It is a widely known fact that CPU-intensive operations block the Node.js Event Loop. While this statement is true up to a certain extent, it is not 100% true as there are some CPU-intensive functions which do not block the event loop.

In general, cryptographic operations and compression operations are highly CPU-bound. Due to this reason, there are async versions of certain crypto functions and zlib functions which are written in a way to perform computations on the libuv thread pool so that they do not block the event loop. Some of these functions are:

crypto.pbkdf2()
crypto.randomFill()
crypto.randomBytes()
All zlib async functions

However, as of this writing, there’s no way to run CPU-intensive operation on the libuv thread pool using pure JavaScript. Yet, you can write your own C++ addon which will give you the ability to schedule work on the libuv thread pool. There are certain 3rd party libraries (e.g. bcrypt) which perform CPU-intensive operations and uses C++ addons to implement asynchronous APIs for CPU bound operations.

Misconception 4— All asynchronous operations are performed on the thread pool

Modern operating systems have built-in kernel support to facilitate native asynchrony for Network I/O operations in an efficient way using event notifications (e.g, epoll in linux, kqueue in macOS, IOCP in windows etc.). Therefore, Network I/O is not performed on the libuv thread pool.

However, when it comes to File I/O, there are a lot of inconsistencies across operating systems as well as in some cases within the same operating system. This makes it extremely hard to implement a generalised platform-independent API for File I/O. Therefore, File system operations are performed on the libuv thread pool to expose a consistent asynchronous API.

dns.lookup() function in dns module is another API which utilises the libuv thread pool. The reason for that is, resolving a domain name to an IP address using dns.lookup() function is a platform-dependent operation, and this operation is not a 100% network I/O.

You can read more about how NodeJS handles different I/O operations here:

Handling IO — NodeJS Event Loop Part 4 | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Aug 16, 2020 ・
blog.insiderattack.net

Misconception 5— NodeJS should not be used to write CPU-intensive applications

This is not really a misconception, but rather was a well-known fact about NodeJS which is now obsolete with the introduction of Worker Threads in Node v10.5.0. Although it was introduced as an experimental feature, worker_threads module is now stable since Node v12 LTS, therefore suitable for using it in production applications with CPU-intensive operations.

Each Node.js worker thread will have a copy of its own v8 runtime, an event loop and a libuv thread pool. Therefore, one worker thread performing a blocking CPU-intensive operation does not affect the other worker threads’ event loops thereby making them available for any incoming work.

If you are interested in learning how Worker Threads work in detail, I encourage you to read the following article:

Deep Dive into Worker Threads in Node.js | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Jul 6, 2021 ・
blog.insiderattack.net

However, at the time of this writing, the IDE support for worker threads is not the greatest. Some IDE’s does not support attaching the debugger to the code run inside a worker thread other than the main worker. However, the development support will mature over time as a lot of developers have already started adopting worker threads for CPU-bound operations such as video encoding etc.

I hope you learned something new after reading this article, and please feel free to provide any feedback you have by responding to this.

100% Unit Test Coverage — Is that a Myth?

Deepal Jayasekara — Fri, 07 Feb 2020 19:46:11 +0000

Background Image Courtesy: https://www.goodfon.com/wallpaper/world-pc-technology-satellite.html

We all need to embrace the bitter fact that we all, as developers, hated writing unit tests at some point in our career. While some of us still keep hating writing tests (which I don’t blame for), I kind of developed a weird interest in writing unit tests after working with a number of JavaScript (mainly Node.js) projects over the time. And many times, I have seen people arguing about the acceptable unit test coverage both in meetings as well as online developer forums.

After observing all those dramas, and after having terrible experiences myself throughout the years, I thought I should very briefly write down my two cents on writing unit tests with proper coverage. While I jot these down based on my experience in writing unit tests for Node.js applications, I strongly believe that these facts are universal for any type of application written in any programming language. And, I’m pretty sure you might have more experience than me regarding this topic, so feel free to let me know your opinions about this topic which certainly would help me, as well as the other readers.

This article was originally posted in:

100% Unit Test Coverage — Is that a Myth? | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Feb 13, 2020 ・
blog.insiderattack.net

Why do you need unit tests? Isn’t integration tests sufficient?

One problem with the unit tests is that if your unit tests are passing, it still doesn’t mean that your application will operate correctly. The reason is, as we all know, unit tests only stubs/mocks the dependencies and test the individual building blocks of your application. In contrast, “Integration tests” assert whether your application behaves properly once all those building blocks were put together. Then why do we write unit tests at all?? Why can’t we satisfy from integration tests?

We need to understand the purpose of unit tests to answer this question.

Unit tests are there to increase the developer confidence that whatever the change made in the future won’t break the existing functionality.

Can’t we get the same confidence level by integration tests? Not really.

Running integration tests is usually an expensive operation as it involves communicating with real or at least near-real dependencies. This is not something you can do every time you make a code change as it affects productivity.

Another reason is that it is extremely hard to reproduce and test all the execution paths including the edge-cases in integration tests whereas, in unit tests, it’s relatively easy to manipulate the execution path by fine-grained dependency stubbing to test those scenarios.

80% Coverage or 100% coverage

I have seen many times in many projects that people agreeing to 80% as a good test coverage number. I’m strongly against this decision because I still do not have the answers to the following two questions:

How do you quantify the acceptable test coverage? Who and how can come someone up with an exact number?
If 80% coverage is acceptable, which 80% of the code would you cover?

If you are a project manager, and you expect 80% of the unit tests coverage from your developers, you should know that they are going to leave the most critical 20% of the code untested and cover the most comfortable 80% just to make you happy!

In my opinion, the tests should cover as much of your code as possible and preferably 100%. Any piece of code you left untested can be changed by another developer at any time, leading to a potential break of the functionality going unnoticed.

However, as we all know, the test coverage is measured in multiple ways such as line coverage, branch coverage, function coverage etc. Obtaining 100% line coverage is not that hard. But, does 100% line coverage mean that the entire code is properly unit tested? This leads us to our next topic.

Line Coverage vs Branch Coverage

A line is considered to be covered if any of the statements in that line was touched during the tests. But if the code execution splits into multiple branches in a particular line, line coverage will not correctly cover all the possible execution paths. The execution paths , also known as branches are the different paths your application logic might take during the execution. For example, the following line shows a statement with two branches.

const result = isEveryoneHappy ? happyFunc() : sadFunc();

The above line is considered to be covered in test coverage tools if the code execution hit this line regardless of the value of isEveryoneHappy. But depending on the value of isEveryoneHappy, the code execution might take either happyFunc() or sadFunc() path which could probably result in two completely different outcomes.

Therefore, branch coverage is much more powerful and a more accurate representation of the test coverage.

Achieving 100% branch coverage is not that hard at all, given that you write your code in a testable way and use the correct tools at your disposal to stub the dependencies and make your code follow the different branches.

Last but not the least, always make sure that you have covered the most important assertions which are related to your application’s functionality when you write tests. 100% unit test coverage will not help you if you haven’t identified the most important functionalities that need to be tested. Once the source code is 100% covered and all the tests are properly written to assert all the required functionality, it’ll be a huge investment which will ease future development.

I hope I left you something important to think about when writing unit tests. Anyway, this topic is open to suggestions, and feel free to let me know your thoughts on this in comments.

DEV Community: Deepal Jayasekara

How Not To Measure Time in Programming

Atomic Microservices Transactions with MongoDB Transactional Outbox

Use Streams to Build High-Performing Node.js Applications

Types of Node.js Streams

Readable Streams

Writable Streams

Duplex Streams

Transform Streams

Read Data From Readable Node.js Streams

1. Listen to 'data' Events

2. Use Async Iterators

State of a Readable Node.js Stream

Processing Large Amounts of Data with Node.js Streams

Reduce Your App's Memory Footprint Using pipeline()

Using pipeline() vs pipe() for Streams

Wrap Up: Reduce Memory and Improve Performance Using Node.js Streams

Experimenting with HTTP/3 — How the Next Gen Web is being built on UDP

Abusing Network Protocols for Secret Communication

NodeJS Streams in Practice

Starting Your Own Tech Blog

A Visual Guide to NodeJS Streams

Understanding Async Resources with Async Hooks

JavaScript Event Loop vs Node JS Event Loop

Five Misconceptions on How NodeJS Works

Misconception 1 — EventEmitter and the Event Loop are related

Misconception 2—All callback-accepting functions are asynchronous

Timers, Immediates and Process.nextTick— NodeJS Event Loop Part 2 | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Oct 9, 2020 ・ blog.insiderattack.net

Misconception 3— All CPU-intensive functions are blocking the event loop

Misconception 4— All asynchronous operations are performed on the thread pool

Handling IO — NodeJS Event Loop Part 4 | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Aug 16, 2020 ・ blog.insiderattack.net

Misconception 5— NodeJS should not be used to write CPU-intensive applications

Deep Dive into Worker Threads in Node.js | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Jul 6, 2021 ・ blog.insiderattack.net

Further Readings:

100% Unit Test Coverage — Is that a Myth?

100% Unit Test Coverage — Is that a Myth? | by Deepal Jayasekara | Deepal’s Blog

Deepal Jayasekara ・ Feb 13, 2020 ・ blog.insiderattack.net

Why do you need unit tests? Isn’t integration tests sufficient?

80% Coverage or 100% coverage

Line Coverage vs Branch Coverage

Reduce Your App's Memory Footprint Using `pipeline()`

Using `pipeline()` vs `pipe()` for Streams

Deepal Jayasekara ・ Oct 9, 2020 ・
blog.insiderattack.net

Deepal Jayasekara ・ Aug 16, 2020 ・
blog.insiderattack.net

Deepal Jayasekara ・ Jul 6, 2021 ・
blog.insiderattack.net

Deepal Jayasekara ・ Feb 13, 2020 ・
blog.insiderattack.net