Native ESM in Node.js w/ require() fallbacks and support for all front end compilers!

#javascript #node #webpack #testing

Native ESM support was unflagged in Node.js CURRENT and LTS a few months ago. Once I started diving in it turned out to be a bit more difficult than I anticipated.

One thing I worried about was navigating the differences between the way front-end compilers might interpret ESM and the way Node.js does. If I want to split up the entry points for browser, ESM and require they need to all understand the same package.json properties.

Mikeal Rogers

@mikeal

How widely supported is the new conditional `exports` map in package.json across compilers? /cc @webpack @pikapkg @browserify nodejs.org/dist/latest-v1…

18:24 PM - 08 Jun 2020

2 31

That answer was “no!” Compilers do not yet understand Node.js’ export map.

If you want consumers of your library to be able to import it with require() you’ll need to use an export map and this mapping will be used by Node.js but invisible to compilers.

This means a few things:

You’ll probably want to set { “type”: “module” } in your package.json in order to use ESM everywhere by default. This will make Node.js interpret the .js files in your project as ESM and compilers can already detect ESM in the source file. There’s really no benefit to using .mjs unless you want to maintain separate source files of identical implementations, and you probably don’t.
You won’t be able to use an export map the way they were intended, which is to allow something like import main from ‘packageName/defaults’ because that’s not a valid file path and this mapping won’t be visible to the compilers.

Mikeal Rogers

@mikeal

Am I missing something, or is there just *no* way to require() a module/file written in ESM w/ { type: module } in Node.js?

All I can find are examples of separate entry points, which means you are implementing in cjs and then just exposing an ESM entry point.

14:48 PM - 19 Jun 2020

0 14

You can use import to load Node.js modules written to the old module standard, but you cannot require() an ESM module, so the compatibility only flows in one direction.

You quite literally have to have a separate source file, written in the old module format, and overlayed against your ESM files in an export map if you want to support require().

Here’s an example from js-multiformats which has many exports.

 "exports": {
    ".": {
      "import": "./index.js",
      "require": "./dist/index.cjs"
    },
    "./basics.js": {
      "import": "./basics.js",
      "require": "./dist/basics.cjs"
    },
    "./bytes.js": {
      "import": "./bytes.js",
      "require": "./dist/bytes.cjs"
    },
    "./cid.js": {
      "import": "./cid.js",
      "require": "./dist/cid.cjs"
    },
    ...
}

Compiling these with rollup was pretty simple once @mylesborins pointed me in the right direction, but I needed a bit more.

Here’s another example from js-multiformats.

import globby from 'globby'
import path from 'path'

let configs = []

const _filter = p => !p.includes('/_') && !p.includes('rollup.config')

const relativeToMain = name => ({
  name: 'relative-to-main',
  renderChunk: source => {
    while (source.includes("require('../index.js')")) {
      source = source.replace("require('../index.js')", "require('multiformats')")
    }
    while (source.includes("require('../")) {
      source = source.replace('require(\'../', 'require(\'multiformats/')
    }
    return source
  }
})

const plugins = [relativeToMain('multiformats')]
const add = (pattern) => {
  configs = configs.concat(globby.sync(pattern).filter(_filter).map(inputFile => ({
    input: inputFile,
    output: {
      plugins: pattern.startsWith('test') ? plugins : null,
      file: path.join('dist', inputFile).replace('.js', '.cjs'),
      format: 'cjs'
    }
  })))
}
add('*.js')
add('bases/*.js')
add('hashes/*.js')
add('codecs/*.js')
add('test/*.js')
add('test/fixtures/*.js')
console.log(configs)

export default configs

You want to compile all the .js files and all the tests. There’s a lot that can go wrong in this translation so compiling a version of each test that uses require() is quite useful. It also ensure the exported interface remains the same for each entry point.

You also need to make sure you compile out any relative imports in the tests and instead use the local package name. Node.js will resolve the local package name correctly, but if if you use a relative import you’ll actually skip the export map entirely and that will fail.

It would be tempting to just migrate the tests away from relative imports but compilers often don’t support lookups against the local package name the way Node.js does so you can’t.