If you've ever used fs.readFile() to load a large file, you might have noticed high memory usage or even application crashes.
The reason? fs.readFile() loads the entire file into memory before your application can process it.
Node.js solves this problem with Streams and Buffers.
Let's understand why they are one of the most important concepts for every backend developer.
📌 What is a Stream?
A Stream is a way to process data chunk by chunk instead of loading everything into memory at once.
Think of watching a movie on Netflix. The entire movie isn't downloaded before playback starts—you receive small chunks continuously.
That's exactly how Streams work.
Why use Streams?
- ✅ Memory Efficient
- ✅ Better Performance
- ✅ Handles Large Files
- ✅ Ideal for Network Communication
- ✅ Used in File Uploads & Downloads
📦 What is a Buffer?
A Buffer is a temporary memory area that stores raw binary data before it's processed or transmitted.
Whenever Node.js reads data from a file or receives data from a network, it first stores it in a Buffer.
Buffer = Temporary Binary Storage
🌊 Types of Streams
Node.js provides four types of Streams:
1️⃣ Readable Stream
Reads data from a source.
Examples:
fs.createReadStream()- HTTP Request
2️⃣ Writable Stream
Writes data to a destination.
Examples:
fs.createWriteStream()- HTTP Response
3️⃣ Duplex Stream
Supports both reading and writing.
Example:
- TCP Socket
4️⃣ Transform Stream
Reads, modifies, and writes data.
Examples:
- Compression
- Encryption
- Decompression
🔄 pipe() — Connecting Streams
One of the most useful Stream methods is pipe().
Instead of manually reading chunks and writing them yourself, you can directly connect a Readable Stream to a Writable Stream.
This keeps the code simple and automatically handles data flow efficiently.
🚦 Backpressure
Imagine a fast producer sending data to a slow consumer.
Without flow control, memory usage keeps increasing.
Node.js solves this using Backpressure.
When the Writable Stream becomes slower, Node pauses the Readable Stream until the destination catches up.
Backpressure = Flow Control
💧 What is highWaterMark?
highWaterMark defines the maximum amount of data that can be buffered before backpressure is applied.
Think of it as a water level limit.
Once the internal buffer reaches this limit:
- Reading pauses
- Writable Stream processes buffered data
- Reading resumes automatically
📊 Stream Flow
Large File
│
Readable Stream
│
Chunks
│
Internal Buffer
│
HighWaterMark Reached
│
Backpressure
│
Writable Stream
│
Output
💡 Key Interview Takeaways
- Streams process data chunk by chunk.
- Buffers temporarily store binary data.
-
pipe()connects Readable and Writable Streams. - Backpressure prevents memory overflow.
-
highWaterMarkcontrols when backpressure starts. - Streams are ideal for large files, uploads, downloads, and video streaming.
🎯 Final Thoughts
Streams and Buffers are at the heart of Node.js performance.
If you're building APIs that handle file uploads, large downloads, logs, media, or network communication, understanding these concepts will help you write scalable and memory-efficient applications.
What real-world use case have you used Streams for? Share it in the comments! 👇
Top comments (0)