DEV Community

Cover image for Node.js 101 - Streams
Eric The Coder
Eric The Coder

Posted on

Node.js 101 - Streams

Whats are Streams?

Streams are used to process (read and write) data piece by piece (chunks) without completing the whole read and write operation and at the therefore without keeping all the data in memory.

Youtube or Netflix are good example of streams. You don't have to wait for video to fully load. The process is done piece by piece (chucks). So you can then start watching even if the entire file is not yet download

In Node.js there are Readable Streams and Writable Streams. Readable Streams can for example be a file read or a http request data.

Writable Streams is the opposite of Readable Streams so for example a http responses or file to send

Here an example of a large data file read

const fs = require('fs')
const server = require('http').createServer()

server.on('request', () => {
    // No need to load the entire file to memory
    // fs.readFile('data.txt', (err, data) => {
    //    if (err) console.log(err)
    //    res.end(data);
    // })

    // Create a Readable Streams
    const readable = fs.createReadStream('data.txt')

    // Pipe the Stream chunk to a writable Stream
Enter fullscreen mode Exit fullscreen mode

The readable.pipe() method attaches a Writable stream to the readable, causing it to switch automatically into flowing mode and push all of its data to the attached Writable. The flow of data will be automatically managed so that the destination Writable stream is not overwhelmed by a faster Readable stream.


That's it for today. Tomorrow the journey continue. Stay tune!

Follow me on Twitter: Follow @justericchapman

Top comments (1)

micalevisk profile image
Micael Levi L. C.

Streams are great and all but error handling is complex with it.
I've learned with this great talk by Matteo Collina that fs.createReadStream('data.txt').pipe(res) isn't the right way to stream a file over HTTP because this will create a memory leak. Here Matteo explains why.