Nested Dependencies in Frontend

So you got this awesome idea and now you want to actually do it. I'm pretty sure you do not want to start from scratch, so let's use existing open source packages.

If you want to play along, all the code is on github.

For our example case, we wanna use lit-element and lit-html.

mkdir nested-dependecies-in-frontend
cd nested-dependecies-in-frontend
npm install lit-element lit-html@1.0.0 --save-exact

Note: we are using pinned versions on purpose here.

Then we just load both packages in our main.js.

import { LitElement } from "lit-element";
import { html } from "lit-html";

console.log(LitElement);
console.log(html);

In order to find out how big our app will be, we would like to create a rollup bundle. First, install Rollup:

npm install -D rollup

Then create a rollup.config.js

export default {
  input: "main.js",
  output: {
    file: "bundle.js",
    format: "iife"
  },
};

Next, add "build": "rollup -c rollup.config.js && du -h bundle.js" to our package.json's scripts block, so we can easily build the file and output it's file size.
Lets run it via npm run build :)

(!) Unresolved dependencies
https://rollupjs.org/guide/en#warning-treating-module-as-external-dependency
lit-element (imported by main.js)
lit-html (imported by main.js)

Oh! It doesn't work! 😭

OK, I've heard this one before... We need to add some plugins so that Rollup will understand the way node resolution (i.e. bare module specifiers like import { html } from 'lit-html') works.

npm i -D rollup-plugin-node-resolve

+ import resolve from "rollup-plugin-node-resolve";
+
   export default {
    input: "main.js",
    output: {
      file: "bundle.js",
      format: "iife"
    },
+  plugins: [resolve()]
  };

$ npm run build
# ...
created bundle.js in 414ms
96K     bundle.js

So that seems to work fine. 💪

What Happens if Someone Prefers yarn?

Doing a yarn install and then a build should result in the same output, right?

$ yarn install
$ yarn build
# ...
created bundle.js in 583ms
124K    bundle.js

Wow! That is unexpected - 124K for the yarn build vs. 96K for npm?
It seems the yarn build contains some extra files... maybe a package was duplicated?

$ yarn list --pattern lit-*
├─ lit-element@2.2.0
│  └─ lit-html@1.1.0
└─ lit-html@1.0.0

Yup, both lit-html versions 1.0.0 and 1.1.0 are installed.
The reason is most likely that we pinned lit-html to version 1.0.0 in our root dependency when we installed it with the npm install --save-exact lit-html@1.0.0 command, above.

While npm seems to dedupe it fine, I don't feel safe using npm because if the dependency tree becomes bigger npm also likes to install nested dependencies.

$ npm ls lit-element lit-html
├─┬ lit-element@2.2.0
│ └── lit-html@1.0.0  deduped
└── lit-html@1.0.0

Also specially when you use some beta (e.g. 0.x.x) dependencies it becomes very tricky. As in this case SemVer says every 0.x.0 release means a breaking change. This means 0.8.0 is treated as incompatible with 0.9.0. Therefore even if the APIs you are using would work just fine with both versions you will always get nested dependencies which may break your application silently. e.g. there will be no warning or information on the terminal 😱

How Node Resolution Works

In nodejs, when you import a file using a bare specifier, e.g. import { LitElement } from "lit-element"; Node's module resolver function gets the string lit-element, and begins searching all of the directories listed in module.paths for the importing module, which you can inspect like any other value in the node REPL:

$ node
module.paths
[
  '/some/path/nested-dependencies-in-frontend/node_modules',
  '/some/path/node_modules',
  '/some/node_modules',
  '/node_modules',
]
# unimportant folders are hidden here

Basically, node looks into every node_modules folder, starting in the module's parent directory and moving up the file tree, until it finds a directory name which matches the module specifier (in our case, lit-element). The resolution algorithm always starts at the current module's parent directory, so it's always relative to where you are importing the file from. If we would inspect module.paths from within lit-element's directory, we'd see a different list.

$ cd node_modules/lit-element
$ node
module.paths
[
  '/some/path/nested-dependencies-in-frontend/node_modules/lit-element/node_modules',
  '/some/path/nested-dependencies-in-frontend/node_modules',
  '/some/path/node_modules',
  '/some/node_modules',
  '/node_modules',
]

Now we can understand what node's nested dependencies are. Every module can have it's own node_modules directory, ad nauseum, and imports referenced in that module's files will always look in their closest node_modules directory first...

Pros of Nested Dependencies on Node	Cons of Nested Dependencies for Frontend
Every package can have their own versions of every dependency	Shipping the same code twice means longer download and processing times
Packages are not influenced by dependencies of other packages in the application	Stuff might break if the same code is imported twice from two different locations (e.g. performance optimizations via WeakMaps or singletons)
There is no "high fee" to pay for accessing many extra files.	Checking if a file exists is an extra request
On the server, you usually do not care too much about how much extra code (in files size) there is	Overall, in short, your site will get slower

The Problems

In short, automatic module resolution that prefers nesting may be dangerous for frontend.

• We care about loading and parsing performance
• We care about file size
• Some packages must be singletons (i.e. unique in the module graph) to work properly in our application
  • Examples include lit-html and graphql
• We should be in full control of what ends up on the client's browser

Node-style module resolution, which was designed for a server-side environment, can turn these concerns into serious issues when adopted in the browser.
IMHO, even if node resolution makes it technically possible, loading the code for a complex data-grid more than once should never be our goal as frontend developers.

Solutions

Thankfully, there are solutions to these problems that we can use today, and proposals on the horizon which will altogether eliminate the need for such workarounds in the future.

Making it Work Today

Here are some tips to work with bare module specifiers in your front end code today:

• Make sure that the modules in your dependency tree all use similar version ranges of their common dependencies
• Avoid pinning specific package versions (like we did above with npm i -S lit-html@1.0.0) wherever possible
• If you're using npm:
  • Run npm dedupe after installing packages to remove nested duplicates.
  • You can try deleting your package-lock.json and do a fresh install. Sometimes it magically helps 🧙‍♂️
• If you're using yarn:
  • Consider using yarn resolutions to specify your preferred version of any duplicated packages

A Look Into the Future

If we could tell the JavaScript environment (i.e. the browser) exactly at which path to find the file specified by some string, we would have no need for node-style resolution or programming-time deduplication routines.
We'd write something like this and pass it to the browser to specify which paths mapped to which packages:

{
  "lit-html": "./node_modules/lit-html.js",
  "lit-element": "./node_modules/lit-element.js"
}

Using this import map to resolve package paths means there would always only be one version of lit-html and lit-element, because the global environment already knows exactly where to find them.

Luckily ✨, this is already a proposed spec called import maps. And since it's meant for the browser there's no need to do any transformation at all! You just provide the map and you don't need any build step while developing?

Sounds crazy 😜? Let's try it out! 🤗

Note: Mind you this is an experimental API proposal, it hasn't been finalized or accepted by implementers.

It currently only works in Chrome 75+, behind a flag.
So enter chrome://flags/ in the URL bar and then search for Built-in module infra and import maps and enable it.
Here is a direct link to it: chrome://flags/#enable-built-in-module-infra.

Using Import Maps in the Browser

In order to use an import map, let's create an index.html file.

<html lang="en-GB">
<head>
  <script type="importmap">
    {
      "imports": {
        "lit-html": "./node_modules/lit-html/lit-html.js",
        "lit-html/": "./node_modules/lit-html/",
        "lit-element": "./node_modules/lit-element/lit-element.js",
        "lit-element/": "./node_modules/lit-element/"
      }
    }
  </script>
  <title>My app</title>
</head>

<body>
  <crowd-chant>
    <span slot="what">Bare Imports!</span>
    <span slot="when">Now!</span>
  </crowd-chant>

  <script type="module" src="./main.js"></script>
</body>

</html>

and adjust the main.js.

import { html, LitElement } from "lit-element";

class CrowdChant extends LitElement {
  render() {
    return html`
      <h2>What do we want?</h2>
      <slot name="what"></slot>
      <h2>When do we want them?</h2>
      <time><slot name="when">Now!</slot></time>
    `;
  }
}

customElements.define("crowd-chant", CrowdChant);

Save the file then serve it locally by running npx http-server -o in the same directory.
This will open http://localhost:8080/ where you will see your custom element rendered on screen. 🎉

What kind of black magic is this 🔮? Without any bundlers, tools, or build step, we wrote a componentized app with the kind of bare specifiers we've come to know and love.

Lets break it down:

import { html } from 'lit-html';
// will actually import "./node_modules/lit-html/lit-html.js"
// because of
// "lit-html": "./node_modules/lit-html/lit-html.js",

import { repeat } from 'lit-html/directives/repeat.js'
// will actually import "./node_modules/lit-html/directives/repeat.js"
// beacause of
// "lit-html/": "./node_modules/lit-html/",

So this means

1. You can import packages directly since the package name is mapped to a specific file
2. You can import subdirectories and files, since packageName + '/' is mapped to its directory
3. You must not omit the .js when importing a file from a subdirectory

What Does this All Mean for my Production Build?

It's important to once again note that this is still experimental technology. In any event, you may still want to do an optimized build for production sites using tools like Rollup. We are exploring together what these new APIs will do for our websites and apps. The underlying import-maps proposal is still unstable, but that shouldn't stop us from experimenting and extracting utility from it. After all, most of us are comfortable using babel to enable experimental syntax like decorators, even though that proposal has at time of this writing at least four flavours.

If you want to try import maps today even in unsupported browsers, you'll need either a build step or a runtime solution like systemjs. For the build-step option, you'll replace the rollup-plugin-node-resolve with something that respects your import map instead of using node resolution.

And wouldn't it be really nice if you could just point rollup to your index.html and have it figure out what your entry points are and if there is an import map?

That's why at open-wc we're releasing experimental support for import maps with our rollup-plugin-index-html.

And you can read all about it here on dev.to. Watch this space for the announcement 😉.

Follow us on Twitter, or follow me on my personal Twitter.
Make sure to check out our other tools and recommendations at open-wc.org.

Thanks to Benny and Lars for feedback and helping turn my scribbles to a followable story.

Comments

[Luiz Américo]

Hi, nice article. I'm also exploring the possibility of use native browser import, to avoid bundling.

Some considerations:

• In the Backbone ecosystem, the issue of bundling duplicates of dependencies also hit it, long ago. The solution (or workaround) was to define the basic dependencies (Backbone, underscore) as peerDependencies letting for the app define as dependency.

• To native import work, all dependencies must be distributed as ES module. Sometimes is not so simple

• There's the possibility to use imports map with a polyfill. Here's an example

• Using native imports is nice for small apps but for apps with many dependencies the distribution may be difficult (there's the need to resolve all dependencies in node_modules and elsewhere and upload together with app main source, ensuring relative path is adjusted)

• In the other side, for testing demoing libraries / small apps, it rocks (compare the needed build setup to run the react version of the example app cited above)

[Thomas Allmer]

thxxx :)

1. yes "moving" the decision to the app itself can work in certain situations but it also "moves" more complexity to the app which is not ideal in many cases as well 🙈
2. that is soooo true 😭 we experience it ourselves quite a lot as well... you can publish a "fork" like npmjs.com/package/@bundled-es-modu... but yeah that always comes with maintenance 🙈 I am afraid there is no simple way forward - all we can do is encouraging projects to adopt es modules. I think when node will support es modules a lot will change... it's going to happen - slowly but steadily...
3. yeah actually we love that polyfill and we are building something that will use it - stay tuned 🤗
4. yes, for now, we recommend import maps purely for development - production environment should still use a build for performance. We recommend our rollup setup which respects import maps while building your performance optimized files.
5. yeah having no need for any build step while developing feels so freeing :)