DEV Community

Cover image for Why a Node.js Transform error listener is not enough
Chris Rodriguez
Chris Rodriguez

Posted on Originally published at chriseugenerodriguez.com

Why a Node.js Transform error listener is not enough

A failed Node.js stream chain has two jobs to complete: report the failure and close the connected participants. Those jobs are easy to confuse when a Readable feeds a Transform and then a Writable.

The example below uses object mode to make lifecycle behavior visible. The source emits a small set of values, the Transform uppercases each value until it reaches a chosen failure marker, and the destination records the output it receives. Keeping the fixture in memory avoids file, socket, and buffering details that would obscure the central question.

The important implementation decision is the boundary around the chain. Repeated .pipe() calls connect the stages for data flow. If the Transform fails, an error listener can observe that failure and prevent it from being unhandled. But observing an error is not the same as cleanup: the failed Transform may be destroyed while the source remains readable and the destination remains writable or unfinished. Earlier chunks may already have reached the destination, so partial output is expected.

For a chain that should succeed or fail as one unit, pass every connected stage to stream.pipeline() and await the resulting promise. A Transform error rejects that operation and Node coordinates teardown of the connected streams. On the successful path, the transformed values reach the Writable and it finishes normally.

Inspect more than the error message when validating this behavior. Record the output collected before failure, whether the source, Transform, and destination were destroyed, whether the source ended, and whether the destination finished. Those observations distinguish an error that was merely surfaced from a chain that was actually closed.

There is an important limit: this pattern demonstrates stream-object lifecycle behavior, not throughput, memory use, or universal file-descriptor behavior for every custom source and sink. .pipe() is not inherently wrong; it remains a reasonable choice when application code deliberately owns teardown and covers every failure path. The practical rule is narrower: choose pipeline() when the whole connected chain needs one completion boundary and one cleanup boundary.

Working example

import { Readable, Transform, Writable } from 'node:stream';
import { pipeline } from 'node:stream/promises';

function makeChain(chunks, failOn) {
  let index = 0;
  const received = [];

  // A one-item high-water mark prevents this source from eagerly ending before
  // the transform gets a chance to reject a later item.
  const source = new Readable({
    objectMode: true,
    highWaterMark: 1,
    read() {
      if (index === chunks.length) {
        this.push(null);
        return;
      }
      this.push(chunks[index++]);
    },
  });

  const transformer = new Transform({
    objectMode: true,
    highWaterMark: 1,
    transform(chunk, _encoding, callback) {
      if (chunk === failOn) {
        callback(new Error(`transform rejected: ${String(failOn)}`));
        return;
      }
      callback(null, String(chunk).toUpperCase());
    },
  });

  const destination = new Writable({
    objectMode: true,
    write(chunk, _encoding, callback) {
      received.push(chunk);
      callback();
    },
  });

  return { source, transformer, destination, received };
}

function state(chain) {
  return {
    output: chain.received,
    sourceDestroyed: chain.source.destroyed,
    sourceEnded: chain.source.readableEnded,
    transformDestroyed: chain.transformer.destroyed,
    destinationDestroyed: chain.destination.destroyed,
    destinationFinished: chain.destination.writableFinished,
  };
}

// This deliberately uses ordinary .pipe(). Its error listener prevents an
// unhandled-error crash, but it does not implement cleanup for the other ends.
export async function processWithPipe(chunks, failOn) {
  const chain = makeChain(chunks, failOn);
  const error = await new Promise((resolve) => {
    chain.transformer.once('error', resolve);
    chain.source.pipe(chain.transformer).pipe(chain.destination);
  });

  return { error, ...state(chain) };
}

// This is the application-facing alternative for a connected stream chain.
export async function processWithPipeline(chunks, failOn) {
  const chain = makeChain(chunks, failOn);
  let error = null;

  try {
    await pipeline(chain.source, chain.transformer, chain.destination);
  } catch (caught) {
    error = caught;
  }

  return { error, ...state(chain) };
}
Enter fullscreen mode Exit fullscreen mode

Read the full article: https://chriseugenerodriguez.com/blog/why-does-pipe-leave-a-failed-node-stream-chain-open

Top comments (0)