Have you been using Node.js for a while and only until now have you heard about the word pipe? You tried checking the documentation but still can’t figure out what it means or does? In this article, I will clarify those doubts by explaining what .pipe or piping is and how to use it in Node.js. To ensure you understand the article, previous knowledge about streams is strongly recommended.
What Does .pipe() Method Do?
The method .pipe was added in v0.9.4 of Node.js and its purpose is to attach a writeable stream to a readable stream allowing to pass the readable stream data to the writeable stream. One good way to understand this concept is by thinking about PVC pipes and connecting two pipes.
For the sake of explanation, let’s assume the first PVC pipe is a readable stream and the second pipe is a writeable stream. The method .pipe will be the orange pipe fitting which will connect both pipes allow for the water, or the data, to flow from one pipe to another.
How to Use The .pipe() Method?
In this pipe implementation, we are going to create a simple HTTP server that will read data from a file and send the response to the client.
1. Let’s start by creating the HTTP server using the http package that returns some data.
const http = require('http');
http.createServer(function(req, res) {
res.write('hello!');
res.end();
}).listen(8080);
Let’s make sure it works, by making a request to our server using curl.
curl localhost:8080
Or another option is to open a new tab http://localhost:8080/. Once you make the request, you should receive “hello!”.
We are going to pause for a second. Let’s recall the anatomy of an HTTP transaction. An HTTP transaction is made of a server, created by the method createServer which in itself is an EventEmitter. When an HTTP request hits the server, node calls the request handler using the req and res objects, which are request and response respectively, for dealing with the transaction.
The req or request object is an instance of the IncomingMessage object. The IncomingMessage object is a child object of a ReadableStream.
The res or response object is an instance of the ServerResponse object. The ServerResponse object is a child object of a WriteableStream.
Therefore, we know we have a writeable and a readable stream.
2. We are going to create a data.txt file in the same directory folder, and save some information. For the sake of making things clear, I will save the following text: “This is data from the data.txt file”.
3. Remove the existing logic from the event handler.
4. We are going to read the content of the data.txt file using the fs package using fs.createReadStream. The fs.createReadStream will return a ReadableStream. We are going to use that ReadableStream to pipe or pass the data from data.txt file to the response object, which is a WriteableStream.
const http = require('http');
const fs = require('fs');
http.createServer(function(req, res) {
// generete readable stream to read content of data.txt
const readStream = fs.createReadStream(__dirname + '/data.txt');
// pass readable stream data, which are the content of data.txt, to the
// response object, which is a writeable stream
readStream.pipe(res);
}).listen(8080);
Once updated the event handler’s logic, make a request to http://localhost:8080/ and you should see data.txt data.
Only Works With Readable Streams
Remember, the pipe method can only be used in readable streams. Don’t let yourself fool by your IDE in case it suggests the pipe method in a writeable stream.
In case you attempt using the .pipe method using a writeable stream, like in the example below:
At the moment of executing this code, it will throw the following error.
Error [ERR_STREAM_CANNOT_PIPE]: Cannot pipe, not readable
Pushing Data No Matter Readable Stream’s Flowing Mode
If you are familiar with readable streams, you will know there are two modes in which data flows, flowing and paused mode. You can use the pause() or resume() method to update the flowing mode.
const http = require('http');
const fs = require('fs');
http.createServer(function(req, res) {
const readStream = fs.createReadStream(__dirname + '/data.txt');
readStream.on('data', function(chunk) {
console.log('this is the data from file', chunk);
});
readStream.pause();
console.log('on pause: readable flowing', readStream.readableFlowing);
readStream.resume();
console.log('on resume: readable flowing', readStream.readableFlowing);
res.write('Hello!')
res.end();
}).listen(8080);
If you run the example above, you will only read data from the data.txt file whenever the readable stream flowing mode is set to true which is enabled by using the resume() method. If the flowing mode is set to false, it will never read the content of the data.txt file.
However, when using the pipe method the flowing mode will automatically be set to true ensuring the data is passed from one stream to another. We can confirm this if we try to pause the flowing mode prior to piping both streams.
const http = require('http');
const fs = require('fs');
http.createServer(function(req, res) {
// generete readable stream to read content of data.txt
const readStream = fs.createReadStream(__dirname + '/data.txt');
readStream.on('data', function(chunk) {
console.log('this is the data from file', chunk);
});
readStream.pause();
console.log('on pause: readable flowing', readStream.readableFlowing);
readStream.pipe(res);
}).listen(8080);
After making a request to the server, we will still receive the content from the data.txt file.
Don’t Confuse The pipe Method With The Event pipe
If you have never heard of the word “pipe” or “piping” when working with streams, there is a slight chance you could find the wrong information if you go to Node.js documentation and start finding for the word “pipe”. When you do a quick search, you will find two options.
If you find the first option, you will notice it is an event listener that writeable streams can set when a readable stream uses the pipe method to pass the data from one stream to another. The event pipe is only available on writeable streams. We are going to use our simple server API to demonstrate the event pipes.
const http = require('http');
const fs = require('fs');
http.createServer(function(req, res) {
const readStream = fs.createReadStream(__dirname + '/data.txt');
// setting pipe event listener before triggering the pipe method in the readable stream
// otherwise, the pipe event listener won't be triggered if set after triggering the pipe method
res.on('pipe', function(src) {
console.log('Triggered the pipe event listener whenever a source readable stream pipes the writeable stream');
});
readStream.pipe(res);
}).listen(8080);
In other words, calling the pipe method on the readable stream causes the pipe event listener to be triggered on the writeable stream.
It is important to mention to define the pipe event listener prior to calling the pipe method from the readable stream. Attempting to call the pipe method prior to setting the event listener in the writeable stream won’t work.
Why You Should Use .pipe Method?
Streams are one of the most powerful and fundamental concepts of Node.js applications. They allow us to handle data in a more efficient way as pieces of data can be transported in smaller chunks preventing you from running out of memory and maintaining good performance in your applications.
Therefore, using the pipe method is an effective and easy solution to push data between streams. In that way, we avoid storing too much data that needs to be manipulated or modified all at the same time. Also, the code will be shorter, elegant, and easy to follow.
Top comments (2)
realy good and musr read
Great!