DEV Community: Jacob Smith

The quest for auth in the modern web app

Jacob Smith — Tue, 25 Oct 2022 08:18:55 +0000

From ye old days of HTML table layouts, web apps have grown tremendously sophisticated, to paralleling native app capabilities and performance in, dare I say, most use-cases (push notifications on iOS aside, for now ^[1]).

There are a number of authentication methods available nowadays, but let's first review the main challenges common to them all.

Prologue

A scene unfolds before you: a torrent of requests and responses flying back and forth to one place and another. Some venturing to the wide world, and some back to HQ. The ones back to HQ need the secret knock.

Challenge 1: Session

You don't want just anybody wondering into HQ, so to prevent that you have a secret knock; but, you also need to prevent some rando figuring out your secret knock, so you decide to change it regularly. This is the essentials of basically any authentication setup.

To mitigate the chances of a bad actor hijaking a user's active session, apps typically have a short-lived "access" token that expires after a few (~15) minutes. When that happens, most requests subsequently made to the app's server-side component(s) are rejected. This is an easily recoverable scenario. But what do you do when this happens?

Apps typically have a second, "refresh", token. The only thing this token can be used to do is request a new access token. This refresh token is typically longer-lived, at a minimum, longer than the access token (otherwise, it's useless). When the refresh token expires, that is, by design, unrecoverable (without user intervention).

Maybe you're thinking to yourself: "I got this". But suddenly, another token appears, and then another. "Where the heck are all these tokens coming from??"

Challenge 2: Plot twist! (enter the multiverse)

"Oh. my. god." You've realised users have your app running in multiple browser tabs (as many users do—the official term for these app instances is "clients").

The door to the multiverse is thrown wide open: These tokens are coming from different browser tabs.

This is actually classic concurrency. The issues here to face are most notably desynchronisation of shared data. In the case of authentication, that means the access and refresh tokens (the secret knock). If a client (an SPA running in a particular browser tab) keeps a copy of the tokens in its memory (eg const tokens = { access: '…', refresh: '…' }), that can become stale when another tab is updated. This is a mistake that I see most of the time, and it will result in confusing bugs.

The mitigation I see most often is keeping the tokens in a shared place and retrieve them on every request. This is, however, very inefficient, and depending on what shared place the tokens are put, it may not actually address the issue (ex putting them in IndexedDb re-creates the same problem, but now much more complicated).

Deus ex machina

You could address the first challenge fairly easily in a number of ways, but the multiverse makes things much trickier. However, there is a solution that addresses all of these at the same time with basically no extra work: A ServiceWorker^[2] (which you may or may not already be using, eg for a PWA). The reason for this is because this is what it was intended to do: it creates a point of commonality for multi-client scenarios as a single ServiceWorker shared by all clients. I recently implemented this with @iainjreid for Orbiit (who donated this example): JakobJingleheimer/webapp-auth-via-serviceworker (verified on 24 Oct in the latest Chrome ^[3], Firefox, and iOS Safari).

The important pieces to note are:

WebStorage (specifically, LocalStorage) because WebStorage is synchronous and locking (multiple clients cannot write at the same time—reads and writes are queued). This is critical for precluding race conditions and desynchronisation.
A fetch event listener in the ServiceWorker to decorate internal requests with authentication and handle the various session scenarios cited above.
The ServiceWorker maintains the in-memory cache of tokens; this is safe because all roads lead to Rome (all requests go through the same ServiceWorker and its cache).
The SPA is blocked from initialising until the ServiceWorker is ready. This is necessary to avoid any requests the SPA might make before the ServiceWorker can intercept them. The provided ServiceWorker is quite small (and it's important to keep it that way!), so the latency incurred waiting for it to activate is trivial (something like 3 ticks).

Fetch quest

The fetch handler in the provided example does a few things:

Ignores external requests (you don't want to tell the secret knock to some rando).
Detects transient authentication failures in internal requests, automatically triggering a refresh (if possible), and then re-tries the original request.
Detects and surfaces to the client unrecoverable authentication failures. This allows the client to consider any 401 response a cue to tell the user to log in again.

There is one minor issue with the provided implementation, which is fortunately limited to DX: Handling a request doomed to fail. I think the most correct way to handle this scenario is to cancel it via AbortController::abort() because it causes the request in DevTools' Network panel to be marked cancelled, which seems to me the most accurate representation of what has happened (the ServiceWorker has detected the request is doomed to fail and has cancelled it without hitting the network); this method also supports providing a reason. However, due to identical bugs in all major browsers ^[4][5][6], this does not work.

This leaves 2 options that achieve the same end-result (but both create red herrings for the developer):

Use AbortSignal and trigger a "network failure". This can look like an environment problem instead of expected behaviour.
Return a fake failing response (eg 401). This looks like the server responded with the failure (when in fact the request never left and never touched the server); also, this option makes it more difficult for the client (SPA) to determine what to do (now it has no way to differentiate a real problem from one already being remediated).

I opted for the former (use AbortSignal despite the misleading "network failure") because it has fewer downsides, is a bit less misleading than the other option, and should just start working when the browser bugs are fixed.

Epilogue

This quest started ~4 years ago when I originally implemented it for another client. With the latest refinement (blocking the SPA loading until the ServiceWorker is ready), I believe I've addressed all the core concerns, but if you have any unaddressed, I'd love to hear about it!

Footnotes

Apple claim they "can't" support Push Notifications in PWAs for "performance reasons". Android and desktops somehow managed to address software performance issues years ago, so one might ask "Are Apple's engineers significantly less capable than everyone else?". We can be pretty confident Apple's engineers are not inept, so one might think the next plausible explanation is Apple are lying: It's a problem of financial performance. Push Notifications are effectively a "must" for many apps, so this forces app developers to publish on the AppStore, which is hugely financially advantageous for Apple: Apple take a 30% cut of payments (which they have enforced as a requirement for inclusion in the AppStore). Fortunately, many governments around the world (ex EU, South Korea, US) have ruled Apple are violating anti-monopoly laws, and Apple must allow app developers to use the payment service of the developer's choice. So without financial or software performance issues, we might soon see this "performance reasons" issue disappear (unless Apple decide to vindictively keep up the ruse, and the world leaves them behind).
When working with a ServiceWorker, ensure you
1. Always use an Incognito window (stick to Chromium for dev—Firefox does not allow ServiceWorkers in Private mode aka an Incognito window)
2. Enable DevTools → Application → Service Workers → Update on reload.
If you don't do both of these, you will have a very bad day.
A bug in Brave causes any ServiceWorker fetch() of a CORS request to fail with blocked:other (regardless of proper CORS setup). That means this solution does not currently work at all in Brave (they're pretty quick to fix bugs).
Chromium ServiceWorker fetch AbortSignal bug
Firefox ServiceWorker fetch AbortSignal bug
Safari just ignore bug reports, so I didn't bother.

Custom ESM loaders: Who, what, when, where, why, how

Jacob Smith — Tue, 12 Jul 2022 22:31:50 +0000

Most people probably won't write their own custom ESM loaders, but using them could drastically simply your workflow.

Custom loaders are a powerful mechanism for controlling an application, providing extensive control over loading modules—be that data, files, what-have-you. This article lays out real-world use-cases. End users will likely consume these via packages, but it could still be useful to know, and doing a small and simple one-off is very easy and could save you a lot of hassle with very little effort (most of the loaders I've seen/written are about 20 lines of code, many fewer).

For prime-time usage, multiple loaders work in tandem in a process called "chaining"; it works like a promise chain (because it literally is a promise chain). Loaders are added via command-line in reverse order, following the pattern of its forebearer, --require:

$> node --loader third.mjs --loader second.mjs --loader first.mjs app.mjs

node internally processes those loaders and then starts to load the app (app.mjs). Whilst loading the app, node invokes the loaders: first.mjs, then second.mjs, then third.mjs. Those loaders can completely change basically everything within that process, from redirect to an entirely different file (even on a different device across a network) or quietly provide modified or entirely different contents of those file(s).

In a contrived example:

$> node --loader redirect.mjs app.mjs

// redirect.mjs

export function resolve(specifier, context, nextResolve) {
  let redirect = 'app.prod.mjs';

  switch(process.env.NODE_ENV) {
    case 'development':
      redirect = 'app.dev.mjs';
      break;
    case 'test':
      redirect = 'app.test.mjs';
      break;
  }

  return nextResolve(redirect);
}

This will cause node to dynamically load app.dev.mjs, app.test.mjs, or app.prod.mjs based on the environment (instead of app.mjs).

However, the following provides a more robust and practical use-case:

$> node \
   --loader typescript-loader \
   --loader css-loader \
   --loader network-loader \
   app.tsx

// app.tsx

import ReactDOM from 'react-dom/client';
import {
  BrowserRouter,
  useRoutes,
} from 'react-router-dom';

import AppHeader from './AppHeader.tsx';
import AppFooter from './AppFooter.tsx';

import routes from 'https://example.com/routes.json' assert { type: 'json' };

import './global.css' assert { type: 'css' };

const root = ReactDOM.createRoot(document.getElementById('root'));

root.render(
  <BrowserRouter>
    <AppHeader />
    <main>{useRoutes(routes)}</main>
    <AppFooter />
  </BrowserRouter>
);

The above presents quite a few items to address. Before loaders, one might reach for Webpack, which sits on top of Node.js. However, now, one can tap into node directly to handle all of these on the fly.

The TypeScript

First up is app.tsx, a TypeScript file: node doesn't understand TypeScript. TypeScript brings a number of challenges, the first being the most simple and common: transpiling to javascript. The second is an obnoxious problem: TypeScript demands that import specifiers lie, pointing to files that don't exist. node of course cannot load non-existent files, so you'd need to tell node how to detect the lies and find the truth.

You have a couple options:

Don't lie. Use the .ts etc extensions and use something like esbuild in a loader you write yourself, or an off-the-shelf loader like ts-node/esm to transpile the output. On top of being correct, this is also significantly more performant. This is Node.js’s recommended approach.

Note: tsc appears soon to support .ts file extensions during type-checking: TypeScript#37582, so you'll hopefully be able to have your cake and eat it too.

Use the wrong file extensions and guess (this will lead to decreased performance and possibly bugs).

Due to design decisions in TypeScript, there are unfortunately drawbacks in both options.

If you want to write your own TypeScript loader, the Node.js Loaders team have put together a simple example: nodejs/loaders-test/typescript-loader. ts-node/esm would probably suit you better though.

The CSS

node also does not understand CSS, so it needs a loader (css-loader above) to parse it into some JSON-like structure. I use this most commonly when running tests, where styles themselves often don't matter (just the CSS classnames). So the loader I use for that merely exposes the classnames as simple, matching key-value pairs. I've found this to be sufficient as long as the UI is not actually drawn:

.Container {
  border: 1px solid black;
}

.SomeInnerPiece {
  background-color: blue;
}

import styles from './MyComponent.module.css' assert { type: 'css' };
// { Container: 'Container', SomeInnerPiece: 'SomeInnerPiece' }

const MyComponent () => (<div className={styles.Container} />);

A quick-n-dirty example of css-loader is available here: JakobJingleheimer/demo-css-loader.

A Jest-like snapshot or similar consuming the classnames works perfectly fine and reflects real-world output. If you're manipulating the styles within your JavaScript, you'll need a more robust solution (which is still very feasible); however, this is maybe not the best choice. Depending on what you're doing, CSS Variables are likely better (and do not involve manipulating the styles at all).

The remote data (file)

node does not yet fully support loading modules over a network (there is experimental support that is intentionally very restricted). It’s possible to instead facilitate this with a loader (network-loader above). The Node.js Loaders team have put together a rudimentary example of this: nodejs/loaders-test/https-loader.

All together now

If you have a "one-off" task to complete, like compiling your app to run tests against, this is all you need:

$> NODE_ENV=test \
   NODE_OPTIONS='--loader typescript-loader --loader css-loader --loader network-loader' \
   mocha \
   --extension '.spec.js' \
   './src'

As of this week, the team at Orbiit.ai are using this as part of their development process, to a near 800% speed improvement to test runs. Their new setup isn't quite finished enough to share before & after metrics and some fancy screenshots, but I'll update this article as soon as they are.

// package.json

{
  "scripts": {
    "test": "concurrently --kill-others-on-fail npm:test:*",
    "test:types": "tsc --noEmit",
    "test:unit": "NODE_ENV=test NODE_OPTIONS='…' mocha --extension '…' './src'",
    "test:…": "…"
  }
}

You can see a similar working example in an open-source project here: JakobJingleheimer/react-form5.

For something long-lived (ex a dev server for local development), something like esbuild's serve may better suit the need. If you're keen do it with custom loaders, you'll need a couple more pieces:

A simple http server (JavaScript modules require it) using a dynamic import on the requested module.
A cache-busting custom loader (for when the source code changes), such as quibble (who published an explanatory article on it here).

All in all, custom loaders are pretty neat. Try them out with today's v18.6.0 release of Node.js!

Configuring CommonJS & ES Modules for Node.js

Jacob Smith — Mon, 03 Jan 2022 20:06:22 +0000

Configuration is always a chore, but an unfortunately necessary evil. And configuring a package for CommonJS (CJS) and ES Modules (ESM) can be a waking nightmare—not least because it has changed a dozen times in half as many years.

As one of the implementers for Node.js Loaders, touching much of Node’s internal ESM code, I get asked quite frequently “how do I make this work!?” (often with angry tears); but yet more frequently I come across packages that are just misconfigured.

My name is Jacob, and I’m here to help.

I have confirmed all the provided package.json configurations (not specifically marked “does not work”) work in Node.js 12.22.x (v12 latest, the oldest supported line) and 17.2.0 (current latest at the time)¹, and for grins, with webpack 5.53.0 and 5.63.0 respectively. I’ve prepared a repository with them so you can check them out yourself: JakobJingleheimer/nodejs-module-config-examples (the repo’s root README explains how to use it).

For curious cats, Preamble: How did we get here and Down the rabbit-hole provide background and deeper explanations. If you're just looking for a solution, jump to Pick your poison for the TLDR.

Preamble: How did we get here

CommonJS (CJS) was created long before ECMAScript Modules (ESM), back when JavaScript was still adolescent—CJS and jQuery were created just 3 years apart. CJS is not an official (TC39) standard and is supported by a limited few platforms (most notably, Node.js). ESM as a standard has been incoming for several years; it is currently supported by all major platforms (browsers, Deno, Node.js, etc), meaning it will run pretty much everywhere. As it became clear ESM would effectively succeed CJS (which is still very popular and widespread), many attempted to adopt early on, often before a particular aspect of the ESM specification was finalised. Because of this, those changed over time as better information became available (often informed by learnings/experiences of those eager beavers), going from best-guess to the aligning with the specification.

An additional complication is bundlers, which historically managed much of this territory. However, much of what we previously needed bundle(r)s to manage is now native functionality; yet bundlers are still (and likely always will be) necessary for some things. Unfortunately, functionality bundlers no-longer need to provide is deeply ingrained in older bundlers’ implementations, so they can at times be too helpful, and in some cases, anti-pattern (bundling a library is often not recommended by bundler authors themselves). The hows and whys of that are an article unto itself.

Pick your poison

This article covers configuration of all possible combinations in modern Node.js (v12+). If you are trying to decide which options are ideal, it is better to avoid dual packages, so either:

ESM source and distribution
CJS source and distribution with good/specific module.exports

You as a package author write	Consumers of your package write their code in	Your options
CJS source code using `require()`	CJS: consumers `require()` your package	CJS source and distribution
CJS source code using `require()`	ESM: consumers `import` your package	CJS source and only ESM distribution
CJS source code using `require()`	CJS & ESM: consumers either `require()` or `import` your package	CJS source and both CJS & ESM distribution
ESM source code using `import`	CJS: consumers `require()` your package	ESM source with only CJS distribution
ESM source code using `import`	ESM: consumers `import` your package	ESM source and distribution
ESM: source code uses `import`	CJS & ESM: consumers either `require()` or `import` your package	ESM source and both CJS & ESM distribution

CJS source and distribution

This the "Rum & Coke" of packages: pretty difficult to mess up. Essentially just declare the package’s exports via the "exports" field/field-set.

Working example: cjs-with-cjs-distro

{
  "type": "commonjs",                        // current default, but may change
  "engines": { "node": ">=12.22.7" },        // optional, but kind
  "exports": {
    ".": "PATH/TO/DIST/CODE/ENTRYPOINT.js",  // ex "./dist/index.js"
    "./package.json": "./package.json"       // ensure this file is importable
  }
}

Note that packageJson.exports["."] = filepath is shorthand for packageJson.exports["."].default = filepath

CJS source and only ESM distribution

The "Gin & Tonic" of packages: This takes a small bit of finesse but is also pretty straight-forward.

Working example: cjs-with-esm-distro

{
  "type": "commonjs",                         // current default, but may change
  "engines": { "node": ">=12.22.7" },         // optional, but kind
  "exports": {
    ".": "PATH/TO/DIST/CODE/ENTRYPOINT.mjs",  // ex "./dist/index.mjs"
    "./package.json": "./package.json"        // ensure this file is importable
  }
}

The .mjs file extension is a trump-card: it will override any other configuration and the file will be treated as ESM. Using this file extension is necessary because packageJson.exports.import does NOT signify that the file is ESM (contrary to common, if not universal, misperception), only that it is the file to be used when the package is imported (ESM can import CJS. See Gotchas below).

The "engines" field provides both a human-friendly and a machine-friendly indication of with which version(s) of Node.js the package is compatible. Depending on the package manager used, an exception may be thrown causing the installation to fail when the consumer is using an incompatible version of Node.js (which can be very helpful to consumers). Including this field here will save a lot of headache for consumers with an older version of Node.js who cannot use the package.

CJS source and both CJS & ESM distribution

You have a few options:

Attach named exports directly onto `exports`

The "French 75" of packages: Classic but takes some sophistication and finesse.

Pros:

Smaller package weight
Easy and simple (probably least effort if you don't mind keeping to a minor syntax stipulation)
Precludes the Dual-Package Hazard

Cons:

Hacky-ish: Leverages non-explicitly documented behaviour in Node.js's algorithm (it can but is very unlikely to change).
Requires very specific syntax (either in source code and/or bundler gymnastics).

Working example: cjs-with-dual-distro (properties)

{
  "type": "commonjs",                           // current default, but may change
  "engines": { "node": ">=12.22.7" },           // optional, but kind
  "exports": {
    ".": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js", // ex "./dist/cjs/index.js"
    "./package.json": "./package.json"          // ensure this file is importable
  }
}

Typically, you would see module.exports assigned to something (be it an object or a function) like this:

const someObject = {
    foo() {},
    bar() {},
    qux() {},
};

module.exports = someObject;

Instead, do this:

module.exports.foo = function foo() {}
module.exports.foo = function bar() {}
module.exports.foo = function qux() {}

Use a simple ESM wrapper

The "Piña Colada" of packages: Complicated setup and difficult to get the balance right.

Pros:

Smaller package weight

Cons:

Likely requires complicated bundler gymnastics (I could not find any existing option to automate this in Webpack).

Working example: cjs-with-dual-distro (wrapper)

{
  "type": "commonjs",                                   // current default, but may change
  "engines": { "node": ">=12.22.7" },                   // optional, but kind
  "exports": {
    ".": {
      "import": "PATH/TO/DIST/ESM-CODE/ENTRYPOINT.mjs", // ex "./dist/es/wrapper.mjs"
      "require": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js", // ex "./dist/cjs/index.js"
      "default": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js"  // ex "./dist/cjs/index.js"
    },
    "./package.json": "./package.json"                  // ensure this file is importable
  }
}

In order to support named exports from the CJS bundle for an ESM consumer, this will need a bit of gymnastics from a bundler but is conceptually very simple.

In certain conditions, CJS exports an object (which gets aliased to ESM's default); that object, like any object, is destructure-able. You can leverage that to pluck all the members of the object out, and then re-export them so the ESM consumer is none the wiser.

// ./dist/es/wrapper.mjs

import cjs from '../cjs/index.js';

const { a, b, c, /* … */ } = cjs;

export { a, b, c, /* … */ };

Two full distributions

The "Long Island Ice Tea" of packages: Chuck in a bunch of stuff and hope for the best. This is probably the most common and easiest of the CJS to CJS & ESM options, but you pay for it.

Pros:

Simple bundler configuration

Cons:

Larger package weight (basically double)

Working example: cjs-with-dual-distro (double)

{
  "type": "commonjs",                                   // current default, but may change
  "engines": { "node": ">=12.22.7" },                   // optional, but kind
  "exports": {
    ".": {
      "import": "PATH/TO/DIST/ESM-CODE/ENTRYPOINT.mjs", // ex "./dist/es/index.mjs"
      "require": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js", // ex "./dist/cjs/index.js"
      "default": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js"  // ex "./dist/cjs/index.js"
    },
    "./package.json": "./package.json"                  // ensure this file is importable
  }
}

ESM source and distribution

The wine of packages: Simple, tried, and true.

This is almost exactly the same as the CJS-CJS configuration above with 1 small difference: the "type" field.

Working example: esm-with-esm-distro

{
  "type": "module",
  "engines": { "node": ">=12.22.7" },       // optional, but kind
  "exports": {
    ".": "PATH/TO/DIST/CODE/ENTRYPOINT.js", // ex "./dist/index.js"
    "./package.json": "./package.json"      // ensure this file is importable
  }
}

Note that ESM is not “backwards” compatible with CJS: a CJS module cannot require() an ES Module; it is possible to use a dynamic import (await import()), but this is likely not what consumers expect (and, unlike ESM, CJS does not support Top-Level Await).

ESM source with only CJS distribution

We're not in Kansas anymore, Toto.

The configurations (there are 2 options) are nearly the same as ESM source and both CJS & ESM distribution, just exclude packageJson.exports.import.

💡 Using "type": "module"² paired with the .cjs file extension (for commonjs files) yields best results. For more information on why, see Down the rabbit-hole and Gotchas below.

Working example: esm-with-cjs-distro

ESM source and both CJS & ESM distribution

These are "mixologist" territory.

When source code is written in non-JavaScript (ex TypeScript), options can be limited due to needing to use file extension(s) specific to that language (ex .ts) and there is often no .mjs equivalent³.

Similar to CJS source and both CJS & ESM distribution, you have the same options.

There is also a 4th option of publishing only an ESM distribution and forcing consumers to use a dynamic import (await import()), but that is not quite the same and will likely lead to angry consumers, so it is not covered here.

Publish only a CJS distribution with property exports

The "Mojito" of packages: Tricky to make and needs good ingredients.

This option is almost identical to the CJS source with CJS & ESM distribution's property exports above. The only difference is in package.json: "type": "module".

Only some build tools support generating this output. Rollup produces compatible output out of the box when targetting commonjs. Webpack as of v5.66.0+ does with the new commonjs-static output type, (prior to this no commonjs options produces compatible output). It is not currently possible with esbuild (which produces a non-static exports).

The working example below was created prior to Webpack's recent release, so it uses Rollup (I'll get around to adding a Webpack option too).

Working example: esm-with-cjs-distro

{
  "type": "module",
  "engines": { "node": ">=12.22.7" },            // optional, but kind
  "exports": {
    ".": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.cjs", // ex "./dist/index.cjs"
    "./package.json": "./package.json"           // ensure this file is importable
  }
}

💡 Using "type": "module"² paired with the .cjs file extension (for commonjs files) yields best results. For more information on why, see Down the rabbit-hole and Gotchas below.

Publish a CJS distribution with an ESM wrapper

The "Pornstar Martini" of packages: There's a lot going on here.

This is also almost identical to the CJS source and dual distribution using an ESM wrapper, but with subtle differences "type": "module" and some .cjs file extenions in package.json.

Working example: esm-with-dual-distro (wrapper)

{
  "type": "module",
  "engines": { "node": ">=12.22.7" },                    // optional, but kind
  "exports": {
    ".": {
      "import": "PATH/TO/DIST/ESM-CODE/ENTRYPOINT.js",   // ex "./dist/es/wrapper.js"
      "require": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.cjs", // ex "./dist/cjs/index.cjs"
      "default": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.cjs"  // ex "./dist/cjs/index.cjs"
    },
    "./package.json": "./package.json"                   // ensure this file is importable
  }
}

💡 Using "type": "module"² paired with the .cjs file extension (for commonjs files) yields best results. For more information on why, see Down the rabbit-hole and Gotchas below.

Publish both full CJS & ESM distributions

The "Tokyo Tea" of packages: Chuck in a bunch of stuff (with a surprise) and hope for the best. This is probably the most common and easiest of the ESM to CJS & ESM options, but you pay for it.

In terms of package configuration, there are a few options that differ mostly in personal preference.

Mark the whole package as ESM and specifically mark the CJS exports as CJS via the `.cjs` file extension

This option has the least burden on development/developer experience.

This also means that whatever build tooling must produce the distribution file with a .cjs file extension. This might necessitate chaining multiple build tools or adding a subsequent step to move/rename the file to have the .cjs file extension (ex mv ./dist/index.js ./dist/index.cjs)³. This can be worked around by adding a subsequent step to move/rename those outputted files (ex Rollup or a simple shell script).

Support for the .cjs file extension was added in 12.0.0, and using it will cause ESM to properly recognised a file as commonjs (import { foo } from './foo.cjs works). However, require() does not auto-resolve .cjs like it does for .js, so file extension cannot be omitted as is commonplace in commonjs: require('./foo') will fail, but require('./foo.cjs') works. Using it in your package's exports has no drawbacks: packageJson.exports (and packageJson.main) requires a file extension regardless, and consumers reference your package by the "name" field of your package.json (so they're blissfully unaware).

Working example: esm-with-dual-distro

{
  "type": "module",
  "engines": { "node": ">=12.22.7" },                   // optional, but kind
  "exports": {
    ".": {
      "import": "PATH/TO/DIST/ESM-CODE/ENTRYPOINT.js",  // ex "./dist/es/index.js"
      "require": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.cjs" // ex "./dist/index.cjs"
    },
    "./package.json": "./package.json"                  // ensure this file is importable
  }
}

💡 Using "type": "module"² paired with the .cjs file extension (for commonjs files) yields best results. For more information on why, see Down the rabbit-hole and Gotchas below.

Use the `.mjs` (or equivalent) file extension for all source code files

The configuration for this is the same as CJS source and both CJS & ESM distribution.

Non-JavaScript source code: The non-JavaScript language’s own configuration needs to recognise/specify that the input files are ESM.

Node.js before 12.22.x

🛑 You should not do this: Versions of Node.js prior to 12.x are End of Life and are now vulnerable to serious security exploits.

If you're a security researcher needing to investigate Node.js prior to v12.22.x, feel free to contact me for help configuring.

Down the rabbit-hole

Specifically in relation to Node.js, there are 4 problems to solve:

Determining format of source code files (author running her/his own code)
Determining format of distribution files (code consumers will receive)
Publicising distribution code for when it is require()’d (consumer expects CJS)
Publicising distribution code for when it is import’d (consumer probably wants ESM)

⚠️ The first 2 are independent of the last 2.

The method of loading does NOT determine the format the file is interpreted as:

package.json’s exports.require ≠ CJS. require() does NOT and cannot blindly interpret the file as CJS; for instance, require('foo.json') correctly interprets the file as JSON, not CJS. The module containing the require() call of course must be CJS, but what it is loading is not necessarily also CJS.
package.json’s exports.import ≠ ESM. import similarly does NOT and cannot blindly interpret the file as ESM; import can load CJS, JSON, and WASM, as well as ESM. The module containing the import statement of course must be ESM, but what it is loading is not necessarily also ESM.

So when you see configuration options citing or named with require or import, resist the urge to assume they are for determining CJS vs ES Modules.

⚠️ Adding an "exports" field/field-set to a package’s configuration effectively blocks deep pathing into the package for anything not explicitly listed in the exports’ subpathing. This means it can be a breaking change.

⚠️ Consider carefully whether to distribute both CJS and ESM: It creates the potential for the Dual Package Hazard (especially if misconfigured and the consumer tries to get clever). This can lead to an extremely confusing bug in consuming projects, especially when your package is not perfectly configured. Consumers can even be blind-sided by an intermediary package that uses the "other" format of your package (eg consumer uses the ESM distribution, and some other package the consumer is also using itself uses the CJS distribution). If your package is in any way stateful, consuming both the CJS and ESM distributions will result in parallel states (which is almost surely unintentional).

Gotchas

The package.json's "type" field changes the .js file extension to mean either commonjs or ES module respectively. It is very common in dual/mixed packages (that contain both CJS and ESM) to use this field incorrectly.

// ⚠️ THIS DOES NOT WORK
{
  "type": "module",
  "main": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js",
  "exports": {
    ".": {
      "import": "PATH/TO/DIST/ESM-CODE/ENTRYPOINT.js",
      "require": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js",
      "default": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js"
    },
    "./package.json": "./package.json"
  }
}

This does not work because "type": "module" causes packageJson.main, packageJson.exports["."].require, and packageJson.exports["."].default to get interpreted as ESM (but they’re actually CJS).

Excluding "type": "module" produces the opposite problem:

// ⚠️ THIS DOES NOT WORK
{
  "main": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js",
  "exports": {
    ".": {
      "import": "PATH/TO/DIST/ESM-CODE/ENTRYPOINT.js",
      "require": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js",
      "default": "PATH/TO/DIST/CJS-CODE/ENTRYPOINT.js"
    },
    "./package.json": "./package.json"
  }
}

This does not work because packageJson.exports["."].import will get interpreted as CJS (but it’s actually ESM).

Footnotes

There was a bug in Node.js v13.0–13.6 where packageJson.exports["."] had to be an array with verbose config options as the first item (as an object) and the “default” as the second item (as a string). See nodejs/modules#446.
The "type" field in package.json changes what the .js file extension means, similar to to an HTML script element’s type attribute.
TypeScript has experimental support for the package.json "type" field and .cts and .mts file extensions.

Thank you to @geoffreybooth, @guybedford, @ljharb, @jwfwessels, and @sokra.

DEV Community: Jacob Smith

The quest for auth in the modern web app

Prologue

Challenge 1: Session

Challenge 2: Plot twist! (enter the multiverse)

Deus ex machina

Fetch quest

Epilogue

Footnotes

Custom ESM loaders: Who, what, when, where, why, how

The TypeScript

The CSS

The remote data (file)

All together now

Configuring CommonJS & ES Modules for Node.js

Preamble: How did we get here

Pick your poison

CJS source and distribution

CJS source and only ESM distribution

CJS source and both CJS & ESM distribution

Attach named exports directly onto exports

Use a simple ESM wrapper

Two full distributions

ESM source and distribution

ESM source with only CJS distribution

ESM source and both CJS & ESM distribution

Publish only a CJS distribution with property exports

Publish a CJS distribution with an ESM wrapper

Publish both full CJS & ESM distributions

Mark the whole package as ESM and specifically mark the CJS exports as CJS via the .cjs file extension

Use the .mjs (or equivalent) file extension for all source code files

Node.js before 12.22.x

Down the rabbit-hole

Gotchas

Footnotes

Attach named exports directly onto `exports`

Mark the whole package as ESM and specifically mark the CJS exports as CJS via the `.cjs` file extension

Use the `.mjs` (or equivalent) file extension for all source code files