elanizi43998

Posted on Nov 11

Node.js Streams Explained

#node #javascript #webdev #backend

What Moving the Pyramids Can Teach You About Node.js Streams 🏛️

Imagine you have a 3GB file that you need to upload/download using your Node.js server. How do you solve this task?

Your first instinct as a Node.js developer might be to use an npm package. Sounds great to me!

But what if I told you that while you're treating the file, you also need to add some business logic? Let's say, for example, the file is a CSV, and you need to keep certain lines and ignore others—while saving the ignored lines in a separate file. Can your npm package handle that?

Probably. Is it easier, simpler, scalable? Probably yes. But now you need another package and more complex code in your codebase, which might affect performance.

Now let's take a step back. Before downloading the package, did you ask yourself what technique the creators of these packages use under the hood? I'll answer that for you—it's most likely streams, and that's what we're going to talk about in this post.

So buckle up, and let's see what this thing can do for your app! 🚀

To have a better understanding of streams, let's start by visualizing what problem they solve.

Imagine we have a company, and a client comes to us saying they need to move the Great Pyramid of Giza from Egypt to Britain. We have all the resources to do it in a short period of time. That means using 100% of our capacity to complete the task. But if we go all in, we won't be able to serve other clients because all our manpower is tied up. In the end, we'd be forced to give up the job, our team would be exhausted, and everyone would be upset.

While trying to figure out how to move the pyramid without shutting down the rest of our business, one team member points out something: our main goal is to move the pyramid, so why move it all at once? Why not move it piece by piece, using only part of our resources, so we can keep working on other requests too?

Bingo! That's exactly it—we did what was asked and still kept everyone happy.

This is exactly what streams do in Node.js. The company is our Node.js app. The pyramid is the large file. The manpower represents our computing resources: RAM, CPU, and storage. The team that breaks down and moves the pyramid piece by piece, that's the stream.

What Are Streams?

Now let's speak technically. According to the official documentation:

A stream is an abstract interface for working with streaming data in Node.js. The node:stream module provides an API for implementing the stream interface.

Pretty simple and self-explanatory, but the pyramid already helped us get the idea.

Types of Streams

There are 4 (or 3) types of streams in Node.js: Readable, Writable, Duplex & Transform.

Each one allows us to:

Readable: read data from a source in chunks. We can read from a file, an HTTP request, etc.
Writable: write data to a destination in chunks. We can write to a file, an HTTP request, etc.
Duplex: both read and write streams, like sockets.
Transform: same as Duplex, while also being able to transform the data. In our pyramid example, it's like taking a brick and turning it into a ball (though we might get a strongly worded letter from the Egyptian government 😆).

There's many types of Readable, Writable and Transform streams provided by Node.js but today we will focus on the most basic form of them starting by:

Readable Stream:

const fs = require('fs');
const readable = fs.createReadStream('largeFile.csv');
readable.on('data', chunk => {
  console.log('Received:', chunk.toString());
});

The code snippet above allows us to create a readable stream to read from our file each time we receive a chunk on the 'data' event (We will talk about this later), then it displays our data after parsing it into a string, since out chunks come as a buffer

Writable Stream:

const fs = require('fs');
const writable = fs.createWriteStream('error.txt',);
writable.write('Error');
writable.end();

Here we create a writable stream that we will save on a file named error.txt, then we can write into it using the .write() function as much as we like, when we are done we can finish by calling the .end() that can also accept data to be written into the file if you want to both write and close the stream.

Now that we can read and write, let's make things more interesting. What if we want to do both at the same time and also modify the data as it's being transferred?

Duplex & Transform Stream:

A Duplex stream is essentially a stream that's both readable and writable. Think of it like a twoway pipe for TCP connections, WebSockets, or anything that needs back and forth communication.

A Transform stream is a special kind of Duplex. It doesn't just pass data along it transforms it on the fly.

Let's take a real example: compressing a big file while reading it.

const {
  createReadStream,
  createWriteStream
} = require('fs')
const { createGzip } = require('zlib')

const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const gzipStream = createGzip()
const destStream = createWriteStream(dest)

srcStream.on('data', data => {
  const canContinue = gzipStream.write(data)
  if (!canContinue) {
    srcStream.pause()
    gzipStream.once('drain', () => {
      srcStream.resume()
    })
  }
})

srcStream.on('end', () => {
  const remainingData = gzipStream.read()
  if (remainingData !== null) {
    destStream.write(remainingData)
  }
  gzipStream.end()
})

gzipStream.on('data', data => {
  const canContinue = destStream.write(data)
  if (!canContinue) {
    gzipStream.pause()
    destStream.once('drain', () => {
      gzipStream.resume()
    })
  }
})

gzipStream.on('end', () => {
  destStream.end()
})

Conclusion

Now that you know about Node.js Streams, you can optimize your code, minimize dependency on external packages, and even drop a few buzzwords in your work (like I did 😅) to show who's really in charge. Who knows one day, you might even move the pyramids from Egypt to the British Museum!

Useful Resources

Here are some useful links:

P.S. This post was lightly polished by ChatGPT for grammar and punctuation, but all ideas and examples are 100% mine.

DEV Community