Mastering Node.js Streams and Pipelines

Streams are Node.js's superpower for handling large datasets efficiently. Let's dive into streams and pipelines.

Why Streams?

1. Memory efficiency
2. Time efficiency
3. Composability

Types of Streams

1. Readable
2. Writable
3. Duplex
4. Transform

Basic Stream Usage

const fs = require('fs');

const readStream = fs.createReadStream('big.file');
const writeStream = fs.createWriteStream('output.file');

readStream.on('data', (chunk) => {
  writeStream.write(chunk);
});

readStream.on('end', () => {
  writeStream.end();
});

Enter Pipelines

Pipelines simplify stream composition and error handling.

const { pipeline } = require('stream/promises');
const fs = require('fs');
const zlib = require('zlib');

async function compressFile(input, output) {
  await pipeline(
    fs.createReadStream(input),
    zlib.createGzip(),
    fs.createWriteStream(output)
  );
  console.log('Compression complete');
}

compressFile('big.file', 'big.file.gz').catch(console.error);

Custom Transform Streams

const { Transform } = require('stream');

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

pipeline(
  process.stdin,
  upperCaseTransform,
  process.stdout
).catch(console.error);

Performance Tips

1. Use highWaterMark to control buffering
2. Implement stream.Readable.from() for async iterables
3. Leverage stream.finished() for cleanup

Common Pitfalls

1. Ignoring backpressure
2. Mishandling errors
3. Neglecting to end writable streams

Streams shine with large datasets or real-time data processing. Master them for scalable Node.js applications.

cheers🥂