Loading WASM as ESM in NodeJS

What is a module?

The module in JavaScript is a logical namespace within which we will define the functions or/and values. Then we can export these functions or/and values and import them into some other namespaces.

In the NodeJS world, we have CommonJS modules.

What is CommonJS?

CommonJS modules were created for server and Desktop. With CommonJS the import and export syntax looks something like this:

// importing from a node_modules
const lodash = require('lodash');
// importing from another module
const localFunction = require('./some-module').someFunction;

// Exporting 
module.exports = {
    exportValue: someValue,
    exportFunction: someFunction
}

In servers, (mostly) all the necessary JavaScript files are located in the filesystem. This means they can be loaded synchronously. So the CommonJS module system is

• compact
• allows synchronous loading
• built for the servers

But CommonJS does not have live bindings. CommonJS modules have dynamic structure, this makes it extremely difficult to static checking, optimizing, eliminating dead code with the bundlers.

The bundlers or loaders ecosystem does some intelligent hacks to make this happen.

Also in CommonJS, it is very hard to identify and fix cyclic module dependencies. Sometimes it might lead to infinite loops.

ES Modules

On the other side, the web has a highly undefined network. That introduces latency which makes it hard to work. But still, the web is the most awesome thing happened.

There were multiple attempts made to make the module system for the web. But finally, ES2015 gave us ESModules (or ECMAScript Modules).

// Importing a node modules
import lodash from 'lodash';
// Importing a function from another ESModule
import {someFunction} from './some-module';

// Exporting values or functions
export const exportValue = someValue;
export function exportFunction = someFunction;

The ESModules are built for the web. That is they have the support for asynchronous loading.

It is also important to have clear, concise and compact statements that are easy to understand both for people writing it and for the loaders or bundlers.

ESModules are

• more compact
• loads asynchronously
• built for the web
• cyclic module dependencies are managed efficiently
• static structure makes it easy to check, optimize, eliminate dead code

The ECMAScript modules are slowly stabilizing itself in the NodeJS ecosystem. It definitely took a while but it is all for good. We planned and deliver ESModules in NodeJS. Check out more details here.

Currently, JavaScript that we write for Node and browser environment is different. This makes it difficult for library authors, developers and others. Making JavaScript isomorphic between Node and Browser will be awesome. It will reduce a lot of boilerplate code.

Bringing ESModules to NodeJS makes us bridge the gap between Node and browser.

Narrower the bridge is better the ecosystem will be.

ESModules comes to Node.js

In browsers, we differentiate the ESModules in the script tag using type="module". Likewise in the NodeJS world, we will differentiate the ESModules using .mjs extension.

We can import the .mjs files using the import syntax.

// add.mjs
export let methodCalled = 0;

export function add(x, y) {
    methodCalled++;
    return x+y;
}

// main.mjs
import assert from 'assert';
import {methodCalled, add} from './add.mjs';

assert(methodCalled, 0); 
assert(add(13, 31), 44);
assert(methodCalled, 1);

We can compile and run the above code using node --experimental-modules main.mjs.

The experimental-modules flag specifies the Node to load the main.mjs file as an ESModule.

No default resolution

Currently, the modules implementation does not resolve to index file or add extensions .mjs. That is

import {add} from './add'; // is not a valid ESM import
                          // nor it looks for index file in add folder

No mixing syntaxes in .mjs files

With the current implementation, you cannot mix and match the syntax. That is the .mjs files should only use import statements to import.

const assert = require('assert');
             ^
ReferenceError: require is not defined

Loading js files inside mjs files

The ESModules JavaScript file(.mjs) can import the CommonJS file(.js).

To import the .js we have to use the createRequire function.

// add.js

let methodCalled = 0;

function add(x, y) {
    methodCalled++;
    return x+y;
}

module.exports = {
     methodCalled,
     add
}

Inside the .mjs file let us import the add.js file.

//main.mjs

import { createRequire } from 'module';      // ---1
const require = createRequire(import.meta.url); // ---2

const { add } = require('./add.js'); // ---3

console.log(add(13, 10)); // 23

1. We import the createRequire from the module. The module is available inside the NodeJS code.

2. Then we define the require. The require uses import.meta.url. Check out more about import.meta here

3. Load the library using the require function.

Then we can use the add function, just like any other imported function.

Loading mjs files inside js files

It is not possible to do this.

How does ESModules work?

There is an absolutely awesome blog from Lin Clark here.

There are three phases in the ESModules loading:
1. Fetch and Parse
2. Link
3. Evaluate

Fetch & Parse

As the name indicates in this phase the JavaScript file mentioned is fetched from any URL given. The URL can be a remote location (usually in browsers) or an absolute File URL. Once fetched, the file is parsed.

During parsing the dependencies(or modules) are identified progressively. Then it fetches all the modules and parses them. The parsing ensures the JavaScript is having a valid syntax.

The phase ends with creating a Module record. Consider Module record as an instance that holds all the things that are defined inside the module. Things like import, export and others.

Linking phase

During this phase, the links to export and import are mapped using the module record. The linking will just link the values to a location rather than to a value. This enables live bindings for the imports.

So the values that are imported will always reflect the live value.

Evaluate

During this phase,

• the module's lexical scope is initialized
• functions are hoisted
• function declarations are initialized the JavaScript code is evaluated and the values are filled in the memory location.

Enters the WebAssembly Modules

WebAssembly is the cool new kid in the block. It brings maintainable performance and native code to the browser.

ESM in WASM

Currently, the ESModules integration for WebAssembly Modules is in Stage1.

Let us see the main difference between loading a WebAssembly Module as an ES Module over the JavaScript.

There are three phases in the ESModules loading (similar to JavaScript):

• Fetch and Parse
  • The binary format is parsed and validated.
• Link
  • No Function initialization happens here
• Evaluate
  • Initialize the modules
  • Run the start function
  • Function declarations are initialized

Loading WASM as ESM in NodeJS

Let us first create a WebAssembly Module. The easiest and hackiest way to generate a WebAssembly Module is using the WebAssembly Text Format.

Create a file called add.wat with the following contents

(module
  (func $add (param $p1 i32) (param $p2 i32) (result i32)
    local.get $p1
    local.get $p2
    i32.add)
  (export "add" (func $add))
)

Convert the above WebAssembly Text Format into WebAssembly Module using wabt

/path/to/wabt/build/wat2wasm add.wat -o add.wasm

It creates add.wasm

00 61 73 6d 01 00 00 00 01 07 01 60 02 7f 7f 01
7f 03 02 01 00 07 07 01 03 61 64 64 00 00 0a 09
01 07 00 20 00 20 01 6a 0b

Now we can import the WebAssembly Module like an ESModule. Let us create a file called index.mjs with the following contents.

import * as M from './add.wasm';

console.log(M.add(10, 13)); // 23

We can run the above code using two flags one for enabling the
experimental-modules and experimental-wasm-modules.

node --experimental-modules --experimental-wasm-modules index.mjs

Check out the commit where it landed in the Node core.

Reference links to explore further about --experimental-modules

NodeJS announcement

I hope this gives you a head start into ESModules. If you have any questions/suggestions/ feel that I missed something feel free to add a comment.

If you like this article, please leave a like or a comment.

You can follow me on Twitter and LinkedIn.

Thanks @MylesBorins for the awesome review. :)

Comments

[Saleh Abdel Motaal]

ESM, WASM, this article certainly talks about all the exciting things that makes 2019 a year for the books — thanks for posting!

[Sendil Kumar]

Thanks Saleh. Glad you enjoyed it. :)

[Zach Smith]

Thank you - great post that really helps clarify commonjs vs esm, and also really gave me an introduction to wasm that I understand for the 1st time.

[Sendil Kumar]

So glad that it helped. Its my pleasure 🙂