DEV Community: r9v

await doesn't mean 'meanwhile': the bug every JavaScript engineer writes in Python

r9v — Mon, 06 Jul 2026 20:49:01 +0000

There's a bug you'll write in your first week of Python async, and you'll write it precisely because you're good at JavaScript.

In JavaScript, this is a concurrency idiom:

const userPromise = fetchUser(id); // starts now
const ordersPromise = fetchOrders(id); // starts now, runs alongside
const user = await userPromise; // both already in flight
const orders = await ordersPromise;

Call early, await late. The two fetches overlap, total time is roughly the slower of the two, and every JS engineer has this pattern in muscle memory because it's the cheap way to get parallel I/O without reaching for Promise.all. It works because calling an async function in JavaScript starts it immediately. The body runs synchronously up to the first await, the network request is dispatched before the call even returns, and what you get back is a promise that's already hot. By the time you await it, you're not starting anything, you're subscribing to something that's been running the whole time.

Now you learn Python, you meet async def, and you translate the idiom one line at a time:

user_coro = fetch_user(id)
orders_coro = fetch_orders(id)
user = await user_coro
orders = await orders_coro

It runs and returns the right data, raises no error and logs no warning, and it's exactly as slow as running the two fetches one after the other, because that's what it does. Nothing started at the call. fetch_user(id) executed none of the function body, it built a coroutine object and handed it to you inert, and the first line of fetch_user ran only when the await forced it to. Your second fetch sat frozen until the first one finished end to end. The idiom that meant "meanwhile" in JavaScript means "later, in order" in Python, and the syntax is close enough that nothing about the code tells you the meaning changed.

Eager promises, lazy coroutines

The split is in what a call does. A JavaScript async function is eager: invoke it and the work begins, unconditionally, whether or not anyone ever awaits the result. A Python async def function is lazy: invoke it and you get a coroutine object, which is best understood as a frame that hasn't started, the function's code and arguments packaged up with an instruction pointer parked before the first line. Python will happily let you create one with no event loop running at all, pass it around, store it in a list, and none of that executes anything. The only warning the runtime ever gives you is if you throw one away without awaiting it ("coroutine 'fetch_orders' was never awaited"), and notice that our buggy translation never triggers it, since we do await both. Awaited-but-sequentialized is invisible.

await differs to match. In JavaScript, await promise means "suspend me until that already-moving thing completes." In Python, await coroutine means "run this now, as part of me." Mechanically it's delegation, the direct descendant of the generator yield from, and the awaiting coroutine and the awaited one fuse into a single call stack that suspends and resumes as a unit. The event loop never sees your fetch_orders as an independent piece of work it could interleave with something else. From the loop's point of view there was only ever one runnable thing on your behalf, so there's nothing to overlap. That's the whole bug: in JS the concurrency happened at the call, and awaiting late just collected it, while in Python the call created potential and the await is where all the actual execution lives, one at a time, in the order you wrote them.

Tasks are the eager thing

Python does have JavaScript's hot promise, it just makes you ask for it. Wrapping a coroutine in a Task schedules it on the event loop as an independent unit, and from that moment it runs whenever the loop gets control, whether you're awaiting it or not:

user_task = asyncio.create_task(fetch_user(id))
orders_task = asyncio.create_task(fetch_orders(id))
user = await user_task # both in flight since creation
orders = await orders_task

This is the honest translation of the JavaScript original, overlap and all. The rough mapping, if you want it as a table:

JavaScript	Python
calling `fetchUser(id)`	`asyncio.create_task(fetch_user(id))`
a pending `Promise`	a `Task`
`await promise`	`await task`
`Promise.all([...])`	`asyncio.gather(...)` or `TaskGroup`
no equivalent	a bare coroutine object

The bottom row is the one to internalize. JavaScript has no value that means "an async computation, defined but not started," the closest you get is a thunk like () => fetchUser(id), which is why libraries that need laziness (retry helpers, React Query's queryFn) all take functions rather than promises. Python hands you that lazy value as the default, and the eager one costs a wrapper.

For the fan-out case, gather accepts bare coroutines and wraps each in a Task internally, which is why it delivers real concurrency while your await-in-a-loop doesn't:

results = [await process(item) for item in items] # sequential, N round-trips end to end
results = await asyncio.gather(*(process(item) for item in items)) # concurrent

And since 3.11 the structured version is TaskGroup, which scopes the tasks to a block and refuses to let them outlive it:

async with asyncio.TaskGroup() as tg:
    user_task = tg.create_task(fetch_user(id))
    orders_task = tg.create_task(fetch_orders(id))
user = user_task.result()
orders = orders_task.result()

Why Python went lazy

It's tempting to read the laziness as a wart, but it's a position in a real design argument, and Python's side of it has aged well.

Part of it is lineage. Python coroutines grew out of generators, which are lazy by nature, and async/await (3.5, 2015) was largely new syntax over that machinery, so a coroutine inherited the generator's character: a computation as a value, created anywhere, driven by whoever holds it. A JavaScript promise needs the runtime's machinery the moment it's created; a Python coroutine object doesn't even need a running event loop to exist, which is part of why the same async def can be driven by asyncio, trio, or anything else that speaks the protocol.

The rest is a stance about implicit background work. An eager promise is a fire-and-forget by default, execution that continues whether or not anyone is watching, and JavaScript spent years growing machinery to cope with the consequences, unhandled rejection tracking being the obvious one. Even the call-early-await-late idiom itself has a sharp edge in JS: if the first await throws, the second promise is now rejecting with nobody listening. Python's choice means no work starts implicitly, and anything that should run in the background must be named as a Task, which gives it a lifecycle you're forced to think about. That thinking matters, because a Task you create and drop can be garbage-collected mid-flight, a genuinely nasty gotcha that deserves its own post, and TaskGroup exists precisely to make task lifetimes structural rather than something you track by discipline.

The production shape of the bug

What makes this one dangerous is that it produces no failure, only a latency signature. The endpoint returns correct data, the tests that check payloads pass, and the only symptom is that a request which fans out to four downstream calls takes the sum of their latencies instead of the max. If those calls are 50ms each, you shipped a 200ms endpoint that should be a 50ms one, and nothing in your logs or your test suite will ever flag it. It just sits there as baseline slowness that everyone attributes to "Python being slow" or the network, and it survives because the code reads correctly to anyone whose async instincts were trained in JavaScript, which in a lot of teams is everyone.

The audit is mechanical enough to do on review. Any time two awaits appear in sequence, ask whether the second call actually needs the first one's result. If it does, the sequence is honest and there's nothing to fix. If it doesn't, the awaits are serializing work that could overlap, and the fix is gather, a TaskGroup, or explicit create_task, whichever fits the structure. The deeper habit is to stop reading await as "meanwhile, collect this" and start reading it as "run this here," because in Python the await isn't a checkpoint where concurrency you already launched comes home, it's the place in the program where the work happens. Once that reading settles in, the lazy model stops feeling like a trap and starts feeling like what it is, a language handing you computations as values and letting you decide, explicitly, which of them get to run at the same time.

The two-Reacts bug: when packages aren't singletons

r9v — Thu, 02 Jul 2026 20:05:30 +0000

"Invalid hook call. Hooks can only be called inside of the body of a function component."

You read the error, you open the component, and the hook is exactly where it's supposed to be, top level of a function component, no condition, no loop, no class in sight. The docs page for this warning lists three possible causes. The first is mismatched react and react-dom versions, the second is breaking the Rules of Hooks, and the third is "more than one copy of React in the same app," and nearly everyone reads the first two and skips the third, because who has two copies of React?

You, probably. It's the most common cause of that error in any setup involving a component library, a monorepo, or npm link, and it has a nastier sibling that doesn't throw at all: a context Provider that sets a value while every consumer under it calmly reads the default. No error, no warning, just wrong data, and you can lose an afternoon to it before you even suspect the module graph.

Hooks are a module-level trick

To see why a second copy breaks things, you have to look at what useState actually is, because it can't be what it pretends to be. When your component calls useState, the function has no argument telling it which component instance is asking, no fiber, no id, nothing. It works anyway because react-dom sets a shared mutable slot right before calling your component, a "current dispatcher" that lives at module scope inside the react package, and useState just reads that slot and delegates. The entire hooks API is two packages coordinating through a module-level singleton. React even tells you how load-bearing and private this is by the export name it travels under, which in React 18 was literally __SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED.

Now duplicate the package. The renderer is react-dom copy A, and during render it sets the dispatcher on react copy A's shared slot. Your component, bundled elsewhere, imported useState from react copy B. Copy B's dispatcher slot was never set, it's null, and a null dispatcher is precisely the condition that throws "invalid hook call." The error text blames your component, but nothing about the component is wrong. Two module instances each hold half of a conversation that was designed to happen inside one.

Context fails the same way, just silently. createContext returns an object, and that object's identity is the whole mechanism: the Provider writes the current value into the context object it was created from, and useContext reads from the context object you pass it. If the Provider came from copy A and your useContext received copy B's context object, the write and the read touch two different objects, the value never lands, and the consumer falls back to the default value like no Provider exists. Nothing throws, because nothing is illegal. You just have two parallel context systems that never talk.

How you end up with two

The honest answer is "more ways than you'd think," and the fix differs depending on which one you're in.

The classic is a failed dedup. Two things in your tree depend on React with version ranges that no single version satisfies, so the package manager does the correct thing and nests a second copy inside the stricter dependent. Nobody chose this, it fell out of range arithmetic in somebody's package.json, usually a UI library that pinned React instead of declaring it as a peer.

The one that bites library authors is npm link. You're developing a component library against a real app, so you link it. The library has React in its own node_modules, a devDependency it needs for tests and Storybook, and when the app's bundler follows the symlink it resolves the library's import React by walking up from the file's real location on disk, straight into the library's own copy. The app renders with its React, the library's components call hooks on theirs. Everything type-checks, everything builds, the first render throws. Symlinked development is basically a machine for manufacturing this bug, which is why it's the first thing to suspect the moment "invalid hook call" appears in a linked setup.

Monorepos have their own variants. Workspace hoisting can leave two apps on two React versions with shared packages resolving to one or the other depending on install order. pnpm, doing the strictly correct thing, instantiates a package once per peer-dependency combination, so a shared component library resolved against two React versions in the same workspace genuinely exists twice, on purpose. And module federation setups double React by default, since every remote bundles its own unless you explicitly declare it shared and singleton.

There's also a version of this where the version number is identical and you still get two copies. A package that ships both CJS and ESM builds can be loaded through both in one process, some of your graph requiring it while other parts import it, and since module caches key on the resolved file rather than the package name, the two builds are two module instances with two sets of module-level state. The dual-package hazard, in Node's terminology. It doesn't tend to hit React itself, but it hits the same class of library, anything stateful at module scope.

Confirming it

Before touching config, get proof. The package manager will tell you the truth about the tree:

npm ls react        # every copy, and who pulled it in
pnpm why react
yarn why react

One line mentioning deduped is health. Two distinct entries with different paths is your bug.

For symlink setups where the tree looks clean but you're still suspicious, check identity at runtime, which is the test that can't lie:

// in the app
window.React1 = require("react");
// in the library entry
window.React2 = require("react");
// in the console
window.React1 === window.React2; // false = two copies

If you have a bundle analyzer wired up, it gives the same answer visually, react appearing twice at two paths.

The fixes

On the app side, the durable workaround is forcing resolution to a single path. Vite has it as a first-class option:

// vite.config.js
export default {
  resolve: {
    dedupe: ["react", "react-dom"],
  },
};

and in webpack the same move is an alias, resolve.alias: { react: path.resolve("./node_modules/react") }. Every import of react anywhere in the graph, symlinked or nested, now lands on one file, so there's one module instance and one dispatcher. For linked-library development this is less a workaround than standard equipment, and tools like yalc, which copy instead of symlinking, dodge the problem from the other side.

On the library side there's a real fix, and it's the one that protects your users instead of asking them to defend themselves: React belongs in peerDependencies, never in dependencies, with a copy in devDependencies for your own tests.

{
  "peerDependencies": { "react": ">=18" },
  "devDependencies": { "react": "^19.0.0" }
}

A peer dependency is exactly this contract made explicit, "I use React but I must share your instance." And the same rule applies at bundle time, react marked external in your rollup or tsup config, because a copy of React compiled into your dist files is a second copy nothing can dedupe away. Module federation has its own spelling of the contract, shared: { react: { singleton: true } }.

Packages aren't singletons, resolutions are

The React story is just the loudest instance of something more general, and the general version explains a family of bugs that look unrelated.

At runtime there's no such thing as a package. Node's CJS cache keys on resolved filename, the ESM cache keys on URL, and bundlers key on resolved module path, so "one package" is really "however many distinct resolutions your graph produces." Module-level state is a singleton only while that count is one. Which means any library holding state at module scope, a dispatcher, a context registry, a class definition used in instanceof checks, has silently signed the same contract React signed, and breaks in its own way when the count hits two.

You've probably met the other family members. graphql-js checks for this explicitly and throws "Cannot use GraphQLSchema from another module or realm," one of the most helpful errors in the ecosystem, because they decided a loud failure beats a confusing one. styled-components warns about several instances being initialized and themes that vanish. An err instanceof MyCustomError returning false for an error that is, by any human reading, exactly that class, because the class was defined twice and the handler compared against the other one. Zod schemas failing instanceof ZodType inside a linked package. None of these are separate bugs to memorize, they're one bug wearing different libraries.

The mental model that survives all of it: npm installs packages, but the runtime only ever sees modules, one instance per resolved path, and identity is the only thing module-level state can coordinate on. If you consume libraries, npm ls is how you count instances when identity-shaped weirdness appears. If you write one and you hold anything at module scope, you're a singleton by contract, so declare it with a peer dependency, and consider checking identity at runtime and failing loudly, the graphql-js way, because your users will hit this, probably via npm link, probably on a Friday.

The Node.js bug that's invisible to your monitoring

r9v — Wed, 24 Jun 2026 14:21:24 +0000

Your health check is a single line. res.send('ok'). It used to take a millisecond. Then traffic ramped up one afternoon and p99 went to 400ms, and you spent the next three hours staring at dashboards that all said the same thing, which was nothing.

CPU is moderate, event loop lag is flat, memory looks healthy, and your APM is reporting that the request took 400ms while telling you nothing about why. No slow database spans, no slow downstream calls, no errors or GC pauses. The time was spent in a place your APM can't see.

The place is the libuv thread pool. Standard Node observability is built around the event loop, and the pool is a different queue with different occupants, sitting just out of reach of every dashboard you have.

What lives on the pool

Node's event loop runs your JavaScript. Anything that would block the loop, because it's CPU-heavy or because it's blocking I/O on a syscall that has no real async kernel variant, gets pushed to a separate pool of OS threads inside libuv. That pool defaults to four threads. Four, total, for the whole process, regardless of how many cores the machine has.

What runs there is more than people expect. The obvious ones are fs.readFile, fs.writeFile, and the rest of fs, because most filesystems on Linux don't have a real async API at the syscall level and libuv emulates it with blocking calls on worker threads. Crypto goes there too, including pbkdf2, scrypt, randomBytes in its async form, and anything heavy enough that a sync version would block the loop. zlib runs gzip and deflate on the pool. And dns.lookup, which most apps use without thinking, hits the pool through getaddrinfo, which is a blocking call.

That last one trips people. There are two DNS APIs in Node. dns.lookup looks async and behaves async to your code, but the underlying getaddrinfo system call is blocking, so libuv runs it on the pool. dns.resolve uses c-ares directly, which is a real async resolver and doesn't touch the pool at all. Almost all code uses dns.lookup because it's the default everywhere: http.get, https.request, the standard agent, the pg and mysql drivers, all of it goes through getaddrinfo. So your outbound traffic and your database calls are quietly taking pool slots for name resolution, and you didn't write any of that code, and your APM has no concept of it.

The starvation pattern

Four threads is enough for almost everything until it isn't. The classic trigger is bcrypt at login.

A bcrypt.hash(password, 12) call takes roughly 250ms on commodity hardware and runs entirely on the pool. If four people log in at the same time, all four slots are busy for 250ms. The fifth request hits a queue and waits until one of the four finishes before its hash even starts, so it pays 250ms of queue time before it pays its own 250ms of work, and you've turned a 250ms login into a 500ms one without doing anything extra. Scale that to a hundred concurrent logins and the last person waits something like six seconds, none of which shows up in CPU charts, none of which shows up in event loop lag, all of which is correctly attributable to a libuv pool queue that nobody is graphing.

The second classic is large gzip. A 50MB response body compressed with zlib.gzip takes a noticeable fraction of a second even on fast hardware, and during that time it's holding a pool slot. If you have a route that does this and it gets hit four times concurrently, every other pool consumer in the process is queued behind those compressions. Unrelated fs.readFile calls wait, bcrypts in other handlers queue behind them, DNS lookups join the back of the line. The whole pool is jammed by something that has nothing to do with the requests that are slow.

And the third, the one nobody suspects, is the DNS issue from earlier. Under heavy outbound traffic, every fresh HTTP connection your service makes is a getaddrinfo call on the pool. Most of the time those are fast and the OS caches them, but a DNS hiccup in your network stack, or a misconfigured TTL on an internal service, can put four resolutions in flight at once and stall the rest of the pool for hundreds of milliseconds at a stretch. Then your bcrypts and gzips wait for the network nobody thinks of as competing with crypto for compute.

Why the dashboards lie

A request stuck on the pool queue isn't doing anything visible. It's not on CPU, it isn't running JavaScript, and it isn't in the event loop. It's parked, waiting for libuv to dequeue it, and libuv doesn't ship a counter for that queue depth.

This is the gap. Every tool you have is measuring CPU, event loop, or specific spans you instrumented. The event loop lag metric, the one everyone watches, is genuinely useful but measures the wrong thing here, because the event loop is fine. It's processing other requests at full speed, running timers, accepting new connections. The work that's slow isn't on the loop at all. The metric was designed to catch a different class of bug, where your own JS code is heavy enough to block the loop directly, and against that class it works. Against pool starvation it returns "everything is great" while a queue piles up two function calls over.

APMs make this worse because they create the illusion of completeness. You see request spans, DB spans, slow query traces, error rates, all of it broken down, and the natural conclusion is that if nothing in the breakdown is slow, the request must be slow somewhere outside the application, maybe the network or the load balancer. So you go look there, and there's nothing there either, and you come back and the request is still 400ms. The instinct then is to assume the monitoring is right and the slowness must be a phantom, when actually the monitoring is silent about the slice of the runtime where the time is going.

The signature

Latency without lag is the signature.

If your p99 is climbing under load and the event loop lag metric stays flat, the suspect list is short. Either the request is waiting on something external, or it's waiting on the pool. External you can usually rule out with span traces and network logs, and once you have, the pool is what's left.

The cleanest way to confirm is to bump UV_THREADPOOL_SIZE and see what happens. Set it to 64 in your environment, restart the process, run the same load profile, and watch the latency. If p99 drops noticeably, your pool was queueing and now isn't. If p99 doesn't move, your pool wasn't the bottleneck and you can look elsewhere. It's a ten-minute A/B that rules out half the possible causes, and most teams never run it because the pool isn't in their mental model of where time goes.

The other diagnostic is to grep your code and your dependencies for the usual suspects. Any bcrypt, argon2, crypto.pbkdf2, crypto.scrypt, sync-shaped DNS, large zlib operations, or heavy fs work on a hot path is a candidate. If you find one and it's on every request, you have your answer before you even run the experiment.

The fixes

Bumping the pool is the cheapest move and almost always the right first move. The default of four is a 2010-era number from when machines had two cores and SSDs were exotic, and it's almost always too small for a modern Node service. Sixteen to sixty-four is reasonable for most workloads. The pool threads are cheap when idle, so the cost of overprovisioning is small, and the cost of underprovisioning is the thing this whole post is about.

The hard ceiling is high (1024 in current Node), but you almost never want to go that far. The pool shares CPU with your event loop, and if you have hundreds of threads all wanting to run, the kernel scheduler will mix them with your JS work in ways that hurt latency in different places. Pick a number that gives headroom for the burst sizes you actually see, not the worst case you can imagine.

The deeper fix is to get the worst offenders off the pool entirely. bcrypt should run in a worker thread pool you control, sized to your CPU count, not borrowed from libuv. worker_threads with a small fixed pool of two or four workers handles auth load cleanly and never touches the libuv pool, so bcrypt stops competing with fs and DNS for slots. Large zlib work should use the streaming variants where you can, because createGzip doesn't sit on a single slot for the whole compression, it processes chunks and releases the slot between them.

For DNS, the move is to switch to dns.resolve for anything you control, or to set up a DNS cache layer like cacheable-lookup in your HTTP agent. Both bypass the getaddrinfo path and stop your outbound traffic from competing with crypto for slots. Most HTTP clients have a way to inject a lookup function, and most projects never use it.

fs is harder to dodge because there's no real alternative on Linux outside of io_uring, which Node's support for is still experimental. The practical answer is the same as for everything else: don't do filesystem work on a hot request path, batch where you can, prefer streams to avoid holding a slot for the duration of a large file, and keep per-request file work small.

When to suspect this

The shape to watch for is latency that rises with concurrency while CPU stays moderate, event loop lag stays flat, no slow span appears in the trace, and the slowness clusters on routes that touch auth, compression, files, or DNS. If three of those four boxes are checked, it's the pool until proven otherwise.

The reason this post exists is that nothing in standard Node monitoring will lead you here. The metric you'd want, libuv pool queue depth, isn't exposed by default. The metric everyone watches, event loop lag, will positively tell you it isn't the loop while saying nothing about the pool. You have to know the pool exists and that your code is sitting on it behind a queue, and once you do, the diagnostic and the fix are both fast. Until then, you're staring at green dashboards while users wait.

Node streams aren't hard anymore

r9v — Wed, 27 May 2026 14:34:24 +0000

Node streams have a reputation. James Halliday wrote a "stream-handbook" repo over a decade ago that became canonical, and the existence of a handbook told you everything. For years, asking a Node developer about backpressure was a way to find out which job they were about to quit.

The reputation was earned, but it's mostly obsolete now. The API got fixed in stages between 2018 and 2021, and nobody made a clean announcement of it, so the cultural memory stayed at "streams are scary" while the actual code became, for most uses, boring.

Why they were hard

The short version: three eras of API coexisting in one type, errors that didn't propagate through .pipe(), backpressure as an advisory boolean that ninety percent of code ignored, and four different events all roughly meaning "done" ('end', 'finish', 'close', plus a null from read()). The longer version is the stream-handbook itself.

Pre-2018, writing correct stream code meant remembering, for every pipeline, that you had to attach 'error' to every stage because .pipe() didn't propagate errors and a downstream failure would orphan the upstream stages, hanging the process or leaking file descriptors. You had to check the return value of .write() and listen for 'drain', because otherwise your writable buffer grew until the process OOMed. You had to know which mode (paused or flowing) your readable was in, because attaching a 'data' listener flipped it, and once flowing you could lose chunks if you attached late. And you had to distinguish 'end' (no more reads) from 'finish' (all writes flushed) from 'close' (resource released), because they fired at different times and meant different things.

What changed

The fix came in pieces.

In 2018, Node 10 shipped stream.pipeline(). It propagates errors, destroys all stages on failure, and cleans up resources, the single biggest improvement to streams in Node's history, and it arrived without fanfare. The same release made Readables async iterables, so for await (const chunk of stream) started working, hiding the entire mode/event apparatus behind one line.

In 2020, Node 15 added stream/promises, giving you await pipeline(...) and await finished(stream). Promise-native, no event handlers.

In 2021, Node 16.5 added Web Streams as experimental (they became stable in Node 21, late 2023), a different model entirely, promise-based, never inheriting from EventEmitter, for code that wants out of the legacy.

Most stream tutorials today still walk you through these chronologically: events first, then .pipe(), then pipeline(), then for await, then Web Streams. You finish with five overlapping mental models and the feeling that streams have too many APIs.

There's a better framing.

The map

Every Node stream API fits into one 2×2:

	Readable	Writable
Consume	how you take data out	how you put data in
Implement	how you produce data inside	how you receive data inside

Four cells. Every API you've ever seen on a stream lives in exactly one of them. The complexity of Node streams comes from each cell having multiple layers of API stacked on top, accreted over different versions, and the clarity comes from seeing the cells as separate problems.

Let's fill them in.

Consume Readable, "how do I take data out of this thing"

Three layers, low to high:

// Layer 1: paused mode, manual
const chunk = readable.read();
readable.on("readable", () => {
  /* try .read() again */
});

// Layer 2: flowing mode, push
readable.on("data", (chunk) => handle(chunk));
readable.on("end", () => done());

// Layer 3: async iteration (Node 10+)
for await (const chunk of readable) handle(chunk);

Layer 3 hides everything below it. Backpressure is automatic because await blocks the loop, errors throw out of the for await, and stream mode is irrelevant because you never see it. For consumption, this is the only API you need now.

Consume Writable, "how do I put data into this thing"

// Layer 1: manual, with backpressure
const ok = writable.write(chunk);
if (!ok) await once(writable, "drain");
writable.end();
writable.on("finish", () => done());

// Layer 2: pipeline (Node 10+, promise variant Node 15+)
await pipeline(source, writable);

There's no for await equivalent for writables, you can't async-iterate a sink. The modern answer is to never call .write() yourself, let pipeline() do it. The boolean-return-value-plus-drain dance is still the underlying protocol, but your application code shouldn't be running it.

Implement Readable, "I want to produce data"

// Layer 1: raw
class MyReadable extends Readable {
  _read(size) {
    this.push(chunk);     // returns false when buffer is full
    this.push(null);      // signal end
  }
}

// Layer 2: from an async iterable (Node 12+)
const r = Readable.from(async function* () {
  for await (const row of db.query(...)) yield row;
}());

Readable.from covers most cases. You write a generator, sync or async, and Node wraps it into a Readable that handles _read, push, backpressure, and end-of-stream for you. For most custom Readables, this is what you reach for.

The raw _read/push API is still current, it's what you fall back to when you need fine control, things like multiple data sources or integration with a non-iterable producer.

Implement Writable, "I want to receive data"

const w = new Writable({
  write(chunk, encoding, callback) {
    sink.send(chunk).then(callback, callback);
  },
});

This is the cell that didn't get sugar. There's no Writable.from(asyncFn) that wraps a consumer function into a properly backpressured Writable. You either implement the write(chunk, enc, cb) callback form or you extend the Writable class and override _write. Calling callback() participates in backpressure, Node won't call write again until you signal completion.

Most application code doesn't need to implement Writables, the built-in ones (fs.createWriteStream, the HTTP response, network sockets) cover the common sinks. When you do need one, the raw API is verbose but fine.

What the map shows

The two consume cells got modern wrappers, for await and pipeline(). Most application code lives in those two cells, which is why most stream code is now easy.

The two implement cells got partial sugar (Readable.from) or none (Writable). This is where the historical complexity still lives, and it's narrower than the reputation suggests, you have to be writing your own stream type to hit it.

Web Streams are the same 2×2 with different method names. Consumption goes through getReader().read(), production through a controller's enqueue() from inside the source. The Writable side has its own surface, a WritableStream constructed with a write() method on its underlying sink. The shape is identical, what's missing is the EventEmitter row beneath each cell. Web Streams never accumulated that layer, so there's one API per cell instead of two or three.

What this means in practice

If you're consuming streams in application code, you're in the top row of the map, and you should be using:

// Reading
for await (const chunk of readable) {
  /* ... */
}

// Writing / piping
await pipeline(source, transform, destination);

Everything below that, the raw event API and .pipe() without pipeline, is the legacy stack. It still works, and you'll encounter it in older code, in Express middleware, in libraries that haven't updated. You should be able to read it, but you shouldn't be writing it.

If you're implementing custom streams, you're in the bottom row, where the API is still raw. Readable.from covers most Readable cases via generators. For Writables, you write new Writable({ write(c, e, cb) {} }) and it's fine, the verbosity is the cost.

If you're starting a new project that might run outside Node, on Workers or in browsers, skip the Node stream API entirely and use Web Streams. Same model, smaller surface, portable.

The reputation, revisited

Node streams were hard because the API accumulated rather than redesigned. Each fix added a new layer without removing the old ones, and for a long time, that meant you had to learn all the layers to write correct code.

That's no longer true. The top of the stack, for await and pipeline(), handles the common cases cleanly. The bottom is still there for performance-critical paths and custom stream implementations, but it's no longer on the daily path.

The handbook isn't wrong, it's just no longer the prerequisite it was.