DEV Community: Roman Dvornov

Web Streams in Depth, Part 1: From Callbacks to Streams

Roman Dvornov — Mon, 04 Aug 2025 16:23:30 +0000

Introduction: The Challenge of Modern Data

JavaScript has come a long way from its humble beginnings as a scripting language for small web page interactions. Today, it powers everything from massive web applications to high-performance backend servers. But as the scale and complexity of our applications have grown, so too have the demands on how we handle data.

Consider these scenarios:

Loading a 50MB, or even 1GB JSON file in a browser for visualization.
Processing a multi-gigabyte log file on a Node.js server.
Uploading or downloading large files, possibly compressed, over the network.
Transforming data on the fly — decompressing, decoding, parsing, or even analyzing it — without ever loading it all into memory.

If you’ve ever tried to handle large data in JavaScript, you’ve probably run into the limitations of the classic APIs: callbacks, Promises, and even the venerable JSON.parse() and JSON.stringify(). They’re great for small data, but quickly fall apart when you hit the big leagues. Out-of-memory errors, UI freezes, and slow, unresponsive apps are all too common — even with files in the tens or hundreds of megabytes.

So how did we get here? And what’s the solution? The answer, as you might suspect, is streams — but not just any streams. In this article, we’ll trace the evolution from callbacks to modern Web Streams, explore how they fit into the broader JavaScript ecosystem, and set the stage for building robust, scalable, and universal data pipelines.

1. Callbacks, Promises, and the Need for Streams

Let’s start with the basics. For years, JavaScript’s answer to asynchronous data was the callback. You’d read a file, fetch data, or process an event, and pass in a function to be called when the data was ready.

fs.readFile('large-file.json', (err, data) => {
    if (err) throw err;
    processData(data);
});

This works fine for small files, but what if the file is huge? You’re forced to load the entire thing into memory before you can do anything with it. Not only is this inefficient, it’s dangerous — one big file and your process could crash.

Promises made things a bit nicer, but didn’t solve the core problem:

fs.promises.readFile('large-file.json')
    .then(data => processData(data));

Still, you’re loading everything at once. For truly large data, you need to process it incrementally — as it arrives, chunk by chunk, without ever holding the whole thing in memory.

2. Node.js Streams: The First Revolution

Node.js was among the first to tackle this problem head-on with its Stream API. Streams let you read or write data piece by piece, as it becomes available. The classic example is reading a file:

const fs = require('fs');

const stream = fs.createReadStream('large-file.json');
stream.on('data', chunk => {
    // Process each chunk as it arrives
});
stream.on('end', () => {
    // All done!
});

This is a huge improvement. Now, you can process arbitrarily large files without running out of memory. Streams became the backbone of Node.js’s I/O model — files, network sockets, HTTP requests and responses, even process stdin/stdout.

Node.js streams come in several flavors:

Readable: Data flows out (e.g., reading a file)
Writable: Data flows in (e.g., writing to a file)
Duplex: Both readable and writable (e.g., a TCP socket)
Transform: A duplex stream that transforms data as it passes through (e.g., compression)

Streams are event emitters, with events like data, end, error, and close. You can also pipe streams together:

fs.createReadStream('input.txt')
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream('output.txt.gz'));

However, Node.js streams aren’t perfect. They have a reputation for being tricky — subtle bugs, confusing modes (flowing vs paused), and a somewhat idiosyncratic API. They’re also Node.js-specific — not available in browsers, and not always easy to polyfill.

3. The Rise of Protocols: Iterables and Async Iterables

Meanwhile, JavaScript itself was evolving. ES6 introduced the iterable protocol — objects that can be looped over with for...of. ES2018 brought async iterables, which let you loop over data that arrives asynchronously, using for await...of.

// Synchronous iterable
for (const item of [1, 2, 3]) {
    console.log(item);
}

// Asynchronous iterable
async function* asyncNumbers() {
    yield 1;
    await new Promise(r => setTimeout(r, 100));
    yield 2;
    yield 3;
}

for await (const item of asyncNumbers()) {
    console.log(item);
}

This was a game-changer. Now, any object that implements the right protocol (Symbol.iterator or Symbol.asyncIterator) can be consumed in a standard way, regardless of where the data comes from.

Node.js quickly adopted this pattern. Since Node.js 10 (April 2018), Readable streams implement Symbol.asyncIterator, so you can do:

const fs = require('fs');

async function processFile(filename) {
    for await (const chunk of fs.createReadStream(filename)) {
        // Process each chunk
    }
}

This is not only more idiomatic, it’s less error-prone — no more juggling event listeners, just a simple loop.

for await...of prefers Symbol.asyncIterator, but if not present, falls back to Symbol.iterator. This means you can use for await...of with both async and sync iterables. This subtlety is often overlooked, but it’s what makes protocol-based APIs so flexible.

4. Enter Web Streams: The New Standard

But what about the browser? For years, there was no standard streaming API in the browser. You had to use callbacks, Promises, or hacky workarounds like FileReader for files, or XHR for network data.

That changed with the introduction of the Web Streams API — a set of standard, promise-based streaming primitives now available in all major browsers (with some features still rolling out). The Web Streams API brings three main classes:

ReadableStream: For reading data
WritableStream: For writing data
TransformStream: For transforming data as it passes through

These are designed to be universal — the same API in browsers, Node.js (since v16.5, July 2021), Deno, and Bun.

Here’s a minimal example of a ReadableStream:

const stream = new ReadableStream({
    start(controller) {
        controller.enqueue('hello');
        controller.enqueue('world');
        controller.close();
    }
});

const reader = stream.getReader();
reader.read().then(console.log); // { value: 'hello', done: false }
reader.read().then(console.log); // { value: 'world', done: false }
reader.read().then(console.log); // { value: undefined, done: true }

But you rarely use the low-level API directly. Instead, you consume streams with async iteration — if your environment supports it.

Unlike Node.js streams, Web Streams did not support Symbol.asyncIterator from the start. This feature was proposed in 2017, but only landed in browsers much later:

Firefox: February 2023 (v110)
Chromium: April 2024 (v124)
Safari: Not yet supported as of mid-2025 (see bug 194379)

Node.js, Deno, and Bun support it, but in browsers, you must check compatibility. If not available, you can always fall back to using .getReader().

// Fallback for environments without Symbol.asyncIterator
const reader = stream.getReader();
while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    // Process value
}

Notice the pattern: everything is built on protocols. If an object implements Symbol.asyncIterator, you can for await...of it. This is the glue that lets streams, iterators, and generators all work together.

5. Timeline: How Streaming Features Arrived

To appreciate how far we’ve come, here’s a timeline of key streaming features in JavaScript environments:

May, 2015: First browser implementation of Web Streams (Chromium 43)
January, 2019: Firefox completes initial Web Streams support
April, 2018: Node.js 10 adds Readable[Symbol.asyncIterator]
May, 2020: Deno 1.0 launches with Web Streams support
July, 2021: Node.js 16.5 ships experimental Web Streams, including ReadableStream[Symbol.asyncIterator]
April, 2022: Node.js 18.0 stabilizes Web Streams and fetch()
July, 2022: Bun 0.1.1 launches with Web Streams support
February, 2023: Firefox 110 adds ReadableStream[Symbol.asyncIterator]
April, 2024: Chromium/Edge 124 adds ReadableStream[Symbol.asyncIterator]
2023–2025: ReadableStream.from() lands in Firefox, Node.js 20.6, Deno, Bun (not yet in Chromium or Safari)

This drawn-out process means that, even today, you must be careful about which features are available in your target environment.

6. Node.js Streams vs Web Streams: A Comparison

At this point, you might be wondering: how do Node.js streams and Web Streams relate? Are they compatible? Which should you use?

Let’s break it down.

Feature	Node.js Streams (node:stream)	Web Streams API
Class Names	Readable, Writable, Duplex, Transform	ReadableStream, WritableStream, TransformStream
Duplex	Duplex stream (readable & writable)	No DuplexStream, but can compose ReadableStream and WritableStream
API Style	EventEmitter-based, protocol-driven (since Node 10)	Promise-based, protocol-driven
Iteration	`for await...of` (since Node 10)	`for await...of` (modern browsers, Node.js 16.5+), `getReader()` always available
Piping	`.pipe()`	`.pipeThrough()`, `.pipeTo()`
Conversion	`Readable.from(iterable)`, `fromWeb()`, `toWeb()` (Node.js 17+)	`ReadableStream.from()` (Firefox, Node.js 20.6+, Deno, Bun), manual polyfill elsewhere
Buffering	`highWaterMark`, `objectMode`	`QueuingStrategy`, BYOB (bring your own buffer)
Error Handling	Events (`error`)	Promise rejection, `controller.error()`

Symmetry and differences:

If the class name ends with "Stream", it’s a Web Stream; if not, it’s a Node.js Stream.
Node.js has a built-in Duplex stream; Web Streams do not, but you can compose a ReadableStream and WritableStream to achieve similar functionality (we’ll cover this in a future article).
Node.js streams are event-based, but since Node 10, they’re also protocol-driven via Symbol.asyncIterator.
Web Streams are promise-based and protocol-driven in most environments today (except Safari).

Interoperability

Suppose you have a Web Stream and want to use it as a Node.js stream:

const { Readable } = require('stream');

const webStream = getWebStreamSomehow();
const nodeStream = Readable.from(webStream); // or Readable.fromWeb(webStream)

Or, going the other way (Node.js stream to Web Stream):

const { Readable } = require('stream');

const nodeStream = fs.createReadStream('file.txt');
const webStream = Readable.toWeb(nodeStream); // Node.js 17+

And in environments with ReadableStream.from() (Firefox, Node.js 20.6+, Deno, Bun):

const webStream = ReadableStream.from(nodeStream);

The difference between Readable.from() and Readable.fromWeb() lies in the options passed as the second parameter. Readable.from() and ReadableStream.from() accept any iterable value, not just streams. If ReadableStream.from() isn’t available, you can always write a polyfill (we’ll show how in the next article).

7. Why Protocols Matter: The Universal Abstraction

The beauty of this evolution is that protocols — not classes — are the real foundation. If your object implements the right protocol, it can be consumed as a stream, an iterator, or a generator, regardless of where it came from.

This means you can write code that works everywhere:

In Node.js, Deno, Bun, or the browser
With files, network responses, blobs, or custom data sources
Asynchronously, and synchronously in some cases

For example, a function that processes data from any source:

async function processChunks(iterable) {
    for await (const chunk of iterable) {
        // Process each chunk
    }
}

This will work with:

A Node.js Readable stream
A Web ReadableStream (in environments that support async iteration)
An async generator
Any object implementing Symbol.asyncIterator or even just Symbol.iterator (since for await...of falls back to sync iterables)

While this protocol-driven approach is powerful, keep in mind that browser support for ReadableStream[Symbol.asyncIterator] is still catching up. As of mid-2025, only Firefox and Chromium-based browsers support it; Safari does not. For maximum compatibility, you may need to use .getReader().

What’s Next

Web Streams, built on protocol-driven abstractions, are now the backbone for scalable, universal data handling in JavaScript — across browsers and servers alike. In the next articles, we’ll move from fundamentals to hands-on practice: consuming data from every source, building transformation pipelines, transferring streams between contexts, and mastering streaming uploads and downloads.

Next up:
Part 2: Consuming Data with Web Streams — Files, Blobs, Responses, and Beyond (coming soon)

A surprisingly simple trick: Converting FormData to multipart/form-data string and back using Response

Roman Dvornov — Thu, 05 Dec 2024 16:01:07 +0000

While working on another article about converting various values into ReadableStream, I stumbled upon a neat trick to convert FormData into a multipart/form-data string and back using the Response. Surprisingly, FormData doesn't provide this functionality out of the box. However, by leveraging the methods of Response, we can achieve this quite easily.

It seems this approach is unknown or little-known. I don't think the ability for such conversion was intentionally designed by the authors of the Response API; it's more of a side effect, which makes the solution even more interesting. So, I decided to write this article to share my findings.

Converting `FormData` to a multipart/form-data string

Let's suppose we have an instance of FormData:

const formData = new FormData();
formData.set('value', 123);
formData.set(
  'myfile',
  new File(['{"hello":"world"}'], 'demo.json', { type: 'application/json' })
);

Our goal is to obtain a multipart/form-data string from this FormData instance. Common suggestions involve manually serializing FormData into a string or performing complex manipulations via Request, or using a library for it. However, we can simply use the Response constructor, which accepts FormData as the body, and then call its text() method to get the desired string representation:

// FormData -> multipart/form-data string
function convertFormDataToMultipartString(formData) {
  return new Response(formData).text();
}

const multipartString = await convertFormDataToMultipartString(formData);

console.log(multipartString);
// ------WebKitFormBoundaryQi7NBNu0nAmyAhpU
// Content-Disposition: form-data; name="value"
//
// 123
// ------WebKitFormBoundaryQi7NBNu0nAmyAhpU
// Content-Disposition: form-data; name="myfile"; filename="demo.json"
// Content-Type: application/json
//
// {"hello":"world"}
// ------WebKitFormBoundaryQi7NBNu0nAmyAhpU--

If needed, we can also extract additional values like the Content-Type header and the boundary:

// FormData -> multipart/form-data
async function convertFormDataToMultipart(formData) {
  const response = new Response(formData);
  const contentType = response.headers.get('Content-Type');
  const boundary = contentType.match(/boundary=(\S+)/)[1];

  return {
    contentType,
    boundary,
    body: await response.text(),
  };
}

const multipart = await convertFormDataToMultipart(formData);

console.log(multipart);
// {
//   contentType: 'multipart/form-data; boundary=----WebKitFormBoundarybfJIH5LgEGPqNcqt',
//   boundary: '----WebKitFormBoundarybfJIH5LgEGPqNcqt',
//   body: '------WebKitFormBoundarybfJIH5LgEGPqNcqt\r\n...'
// }

Instead of using text(), we can also use other methods like arrayBuffer() to obtain the encoded content of FormData as an ArrayBuffer. The advantage of this type of value is that it's a transferable object and can be efficiently transferred between workers using postMessage() (meaning without copying, unlike a string). The FormData object itself is not transferable.

Converting multipart/form-data string back to `FormData`

To reverse the process, we again utilize Response, but this time we use its formData() method. A key aspect of this method is that it requires the correct Content-Type header, which must include the multipart/form-data type and the boundary used to separate the parts. If we have the boundary value, we can insert it into the header. However, we might only have the multipart string and nothing else. In that case, we can extract the boundary from the string using a regular expression.

// multipart/form-data string -> FormData
async function convertMultipartStringToFormData(multipartString) {
  const boundary = multipartString.match(/^\s*--(\S+)/)[1];

  return new Response(multipartString, {
    headers: {
      // Without the correct header, the string won't be parsed,
      // and an exception will be thrown on formData() call
      'Content-Type': 'multipart/form-data; boundary=' + boundary,
    },
  }).formData();
}

const restoredFormData = await convertMultipartStringToFormData(multipartString);

console.log([...restoredFormData]);
// [
//   ['value', '123'],
//   ['myfile', File]
// ]

Just like with serializing FormData, we can convert not only strings but also ArrayBuffer, TypedArray, and ReadableStream into FormData, since Response accepts such values as the body value.

Conclusion

One potential downside is that both types of conversions are asynchronous operations. Other than that, the method seems quite reliable and can be used for debugging or when conversion is needed outside of fetch().

This method has wide browser support (it should work in any browser released after 2017). The method is also applicable in Node.js, Deno, and Bun if they support the Response object (i.e., versions where fetch() is available).

I hope you find this approach as intriguing as I did. It's a simple yet effective way to convert FormData to a multipart/form-data string (and not only a string) and back, leveraging the capabilities of the Response API. If you have any thoughts or know of other methods to achieve the same result, feel free to share in the comments!

Discovery.js tutorials: quick start

Roman Dvornov — Mon, 16 Sep 2019 12:35:47 +0000

This and the following tutorials will guide you through the process of building a solution based on discoveryjs projects. As a goal we will get a NPM dependencies inspector, i.e. an interface for exploring structure of node_modules.

Note: Discoveryjs is at an early stage, so some things might to be changed to be more useful and simpler. If you have an idea how to make something better, let us know on discoveryjs issues.

TL;DR

Below you will find an overview of discoveryjs key concepts. You can explore all the sources from the tutorial combined in a repo on GitHub or try how it works online.

Prerequisites

Before we start, we need a project to analyze. It may be a new project or an existing one, the only requirement is that it has a node_modules inside (a subject of our analysis).

As a first step, we need to install discoveryjs view and CLI tools:

npm install @discoveryjs/discovery @discoveryjs/cli

As our next step, we need to launch a discovery's server:

> npx discovery
No config is used
  Models are not defined (model free mode is enabled)
Init common routes ... OK
Server listen on http://localhost:8123

And open http://localhost:8123 in a browser to see something:

That's a model-free mode, where nothing is pre-configured. You can choose any JSON file via "Load data" button or drop it right on the page, and start exploring it.

However, we need something specific, in particular, we need to get a node_modules structure. Let's add some configuration.

Add a configuration

As you might have noticed, there was a message No config is used when we first launched the server. So let's create a config file named .discoveryrc.js with following content:

module.exports = {
    name: 'Node modules structure',
    data() {
        return { hello: 'world' };
    }
};

Note: If you're creating a config file in current working directory (i.e. in a root of project), then no additional action is needed. Otherwise, you need to pass a path to the config file with --config option, or specify it in package.json this way:

{
   ...
   "discovery": "path/to/discovery/config.js",
   ...
}

Ok, let's restart the server to apply a config:

> npx discovery
Load config from .discoveryrc.js
Init single model
  default
    Define default routes ... OK
    Cache: DISABLED
Init common routes ... OK
Server listen on http://localhost:8123

As you can see, a config file that we created is being used now. And there is a default model, which we defined (discovery can run in multi-model mode, we'll cover this approach in later tutorials). Let's see what we get in the browser:

What's we see here:

name is used as a header of the page;
result of data method invocation is displayed as main content of the page

Note: data method must either return data or a promise, that will be resolved to data.

Our basic setup is ready, now can move on to the next step.

Context

Before moving forward, let's look on report page (click Make report to open it):

At first glance, that's the same as index page... But we can change everything! For example, we can recreate an index page, that's easy:

Notice how a header is defined: "h1:#.name". That's a header level 1 with #.name as a content, which is a Jora query. # is referencing to a context of query. To see what it contains just enter # in a query editor and use default view:

So now you know where you can get a current page ID, its params and other stuff.

Collecting data

Back to our project, currently we used a mock data, but we need to collect real data. So we should create a module and change data value in the config (btw, you should not to restart a server after such changes):

module.exports = {
    name: 'Node modules structure',
    data: require('./collect-node-modules-data')
};

The source of collect-node-modules-data.js:

const path = require('path');
const scanFs = require('@discoveryjs/scan-fs');

module.exports = function() {
    const packages = [];

    return scanFs({
        include: ['node_modules'],
        rules: [{
            test: /\/package.json$/,
            extract: (file, content) => {
                const pkg = JSON.parse(content);

                if (pkg.name && pkg.version) {
                    packages.push({
                        name: pkg.name,
                        version: pkg.version,
                        path: path.dirname(file.filename),
                        dependencies: pkg.dependencies
                    });
                }
            }
        }]
    }).then(() => packages);
};

I used @discoveryjs/scan-fs package, which simplify file system scanning by defining rules. In package's readme an usage example can be found, so I took it as a basis and reworked as needed. And now we have some info about node_modules content:

Much better! Despite it's just a JSON, we can dig into it and gain some insights. For example using signature popup we can found out a count of packages and how many of them has more than one physical instance (due to different versions or problems with package deduplication).

Although we have got some data, we need more details. For example, it's good to know to which physical instance resolves every of dependencies. Nevertheless improvements on data fetching is out of scope of this tutorial. So we just take @discoveryjs/node-modules (which also built on @discoveryjs/scan-fs) and get most details about packages with ease. collect-node-modules-data.js simplifies dramatically:

const fetchNodeModules = require('@discoveryjs/node-modules');

module.exports = function() {
    return fetchNodeModules();
};

And data about node_modules is now looks like this:

Preparation script

As you may have noticed, some packages contains deps – a list of dependencies. Each dependency has resolved field, where value is a reference to a package physical instance. The reference is a path value of one of the packages, since each path value is unique. To resolve a reference to a package, we need to use an additional code (e.g. #.data.pick(<path=resolved>)). But for sure, it would be much suitable to have such references already resolved.

Unfortunately, we can't resolve references on data collecting stage, as this will lead to circular references and data duplication, as well as making data transfer problematic. Nevertheless, there is a solution for this – a special script called prepare. That script is defined in config, and is invoked for any new data for discovery instance. Let's start with config:

module.exports = {
    ...
    prepare: __dirname + '/prepare.js', // Note: value is a path to a module
    ...
};

And then define a prepare.js:

discovery.setPrepare(function(data) {
    // do something with data or/and with discovery instance
});

In this module we specified prepare function for a discovery instance. This function invoked every time before data is applied to discovery instance. That's a good place to resolve references:

discovery.setPrepare(function(data) {
    const packageIndex = data.reduce((map, pkg) => map.set(pkg.path, pkg), new Map());

    data.forEach(pkg =>
        pkg.deps.forEach(dep =>
            dep.resolved = packageIndex.get(dep.resolved)
        )
    );
});

Here we create a package index, where key is package's path value (which is unique). After that we go through all packages and each dependency, and replace resolved value for a reference to a package. That's a result:

It's much easier to make queries to dependency graph now. Here's how to get a dependency cluster (dependencies, dependency dependencies, etc.) for a specific package:

Unexpected success story: exploring tutorial's data I found a problem in@discoveryjs/cli (with the query .[deps.[not resolved]]), it had a typo in a peer dependency reference. This has been fixed immediately. This is a case in point.

I suppose, it's a good time to show up some numbers and packages with duplicates on index page.

Setup default page

First of all we need to create a page module, e.g. pages/default.js. default is used since index page has that slug and we can override it (most things in discoveryjs can be overridden). We might start with something simple, like this:

discovery.page.define('default', [
    'h1:#.name',
    'text:"Hello world!"'
]);

Now we need to link the module in the config:

module.exports = {
    name: 'Node modules structure',
    data: require('./collect-node-modules-data'),
    view: {
        assets: [
            'pages/default.js'  // a reference to page's module
        ]
    }
};

Checking up in a browser:

It works!

Let's show some counters, by changing pages/default.js this way:

discovery.page.define('default', [
    'h1:#.name',
    {
        view: 'inline-list',
        item: 'indicator',
        data: `[
            { label: 'Package entries', value: size() },
            { label: 'Unique packages', value: name.size() },
            { label: 'Dup packages', value: group(<name>).[value.size() > 1].size() }
        ]`
    }
]);

Here we define an inline list of indicators. A data value is a Jora query, that produces an array of entries. A package list is used as a data source (a data root), so we get a list length (size()), a number of unique names (name.size()) and number of groups by a name that has more than a single member (group(<name>).[value.size() > 1].size()).

Not bad. However, it would be better to have a link to selected entries besides numbers:

discovery.page.define('default', [
    'h1:#.name',
    {
        view: 'inline-list',
        data: [
            { label: 'Package entries', value: '' },
            { label: 'Unique packages', value: 'name' },
            { label: 'Dup packages', value: 'group(<name>).[value.size() > 1]' }
        ],
        item: `indicator:{
            label,
            value: value.query(#.data, #).size(),
            href: pageLink('report', { query: value, title: label })
        }`
    }
]);

First of all data value was changed, now it's a regular array with a few objects. In addition size() method was removed for each value queries.

Also subquery was added to indicator view. Such queries produce a new object, where value and href property values are computing. For value it performs a query using query() method and pass data to it from a context, then apply size() method to a query result. For href it uses pageLink() method to generate a link to report page with specific query and title. After those changes indicators became clickable (notice that their values became blue) and much functional.

To make index page a bit useful, let's add a table with duplicated packages.

discovery.page.define('default', [
    // ... the same as before

    'h2:"Packages with more than one physical instance"',
    {
        view: 'table',
        data: `
            group(<name>)
            .[value.size() > 1]
            .sort(<value.size()>)
            .reverse()
        `,
        cols: [
            { header: 'Name', content: 'text:key' },
            { header: 'Version & Location', content: {
                view: 'list',
                data: 'value.sort(<version>)',
                item: [
                    'badge:version',
                    'text:path'
                ]
            } }
        ]
    }
]);

The same data as for Dup packages indicator is used for the table. Additionally, a package list was sorted by a group size in reverse order. The rest setup is for columns (btw, often you don't need to setup them). For Version & Location column we defined a nested list (sorted by a version), where each item is a pair of version badge and path to instance.

A package page

Currently we have only overall view of the packages. It might be useful to have a specific package page. To archive this we need to create a new module pages/package.js and define a new page:

discovery.page.define('package', {
    view: 'context',
    data: `{
        name: #.id,
        instances: .[name = #.id]
    }`,
    content: [
        'h1:name',
        'table:instances'
    ]
});

In this module we define a page with slug package. As a root view context view is used, that's a non-visual view which helps define common data for nested views. Notice that we use #.id to get a package name, that comes from an URL, i.e. http://localhost:8123/#package:{id}.

Don't forget to include new module to the config:

module.exports = {
    ...
    view: {
        assets: [
            'pages/default.js',
            'pages/package.js'  // here you go
        ]
    }
};

And here is a result in a browser:

It's not so impressive, but OK for now. More complex views will be created in next tutorials.

Sidebar

Since now we have a package page, it's nice to have a list of all the packages. We might to define a special view sidebar for this, which renders when defined (is not defined by default). Let's create a new module views/sidebar.js:

discovery.view.define('sidebar', {
    view: 'list',
    data: 'name.sort()',
    item: 'link:{ text: $, href: pageLink("package") }'
});

Now we have a sidebar with all the packages:

Looks good. But with a filter it might be much user friendly. Extending sidebar definition:

discovery.view.define('sidebar', {
    view: 'content-filter',
    content: {
        view: 'list',
        data: 'name.[no #.filter or $~=#.filter].sort()',
        item: {
            view: 'link',
            data: '{ text: $, href: pageLink("package"), match: #.filter }',
            content: 'text-match'
        }
    }
});

Here we're wrapped the list into content-filter view, which will provide an input value converted to RegExp (or null when empty) as filter value in the context (a name may be changed via name option). Also we used #.filter to filter data for the list. Finally, a link view definition was extended to highlight matching parts using text-match view. And there is a result:

In case you dislike default style of something, you may tweak styles as you want. Suppose, you want to change a sidebar width. Then you need to create a style file (views/sidebar.css would be a good choice):

.discovery-sidebar {
    width: 300px;
}

And to include a reference to this file in the config, just like with JavaScript modules:

module.exports = {
    ...
    view: {
        assets: [
            ...
            'views/sidebar.css',  // you may specify *.css files in assets too
            'views/sidebar.js'
        ]
    }
};

Auto linking

Last chapter of this tutorial is about links. As you can see above, we made a link to a package page via pageLink() method. Beside that we need to specify link text as well. But how about make it a bit simpler?

To simplify linking, we need to define a link resolver. A good place for this is prepare script:

discovery.setPrepare(function(data) {
    ...

    const packageIndex = data.reduce(
        (map, item) => map
            .set(item, item)        // key is item itself
            .set(item.name, item),  // and `name` value
        new Map()
    );
    discovery.addEntityResolver(value => {
        value = packageIndex.get(value) || packageIndex.get(value.name);

        if (value) {
            return {
                type: 'package',
                id: value.name,
                name: value.name
            };
        }
    });
});

We added a new map (an index) for packages here, and used it for an entity resolver. The entity resolver makes an attempt to translate passed value into a package descriptor when possible. A package descriptor contains:

type – slug of instance type
id - unique reference to an instance, used as page ID in links
name – used as captions for links

Last step, we need to attach this type to certain page (a link should lead somewhere, isn't it?).

discovery.page.define('package', {
    ...
}, {
    resolveLink: 'package'  // link `package` entities to this page
});

The first effect of those changes, is that some values in struct view now marked with a badge link to the package page:

And now you may apply auto-link view for an package object or name:

As an example, sidebar can be slightly refactored:

    // before
        item: {
            view: 'link',
            data: '{ text: $, href: pageLink("package"), match: #.filter }',
            content: 'text-match'
        },

    // with `auto-link`
        item: {
            view: 'auto-link',
            content: 'text-match:{ text, match: #.filter }'
        }

Conclusion

Now you have a basic knowledge of discoveryjs key concepts. Next tutorials will continue to guide you through these topics in more depth.

You can explore all the sources from the tutorial combined in a repo on GitHub or try how it works online.

Follow @js_discovery on Twitter and stay tuned!

DEV Community: Roman Dvornov

Web Streams in Depth, Part 1: From Callbacks to Streams

Introduction: The Challenge of Modern Data

1. Callbacks, Promises, and the Need for Streams

2. Node.js Streams: The First Revolution

3. The Rise of Protocols: Iterables and Async Iterables

4. Enter Web Streams: The New Standard

5. Timeline: How Streaming Features Arrived

6. Node.js Streams vs Web Streams: A Comparison

Interoperability

7. Why Protocols Matter: The Universal Abstraction

What’s Next

A surprisingly simple trick: Converting FormData to multipart/form-data string and back using Response

Converting FormData to a multipart/form-data string

Converting multipart/form-data string back to FormData

Conclusion

Discovery.js tutorials: quick start

TL;DR

Prerequisites

Add a configuration

Context

Collecting data

Preparation script

Setup default page

A package page

Sidebar

Auto linking

Conclusion

Converting `FormData` to a multipart/form-data string

Converting multipart/form-data string back to `FormData`