Use URL imports before Node/TypeScript supports it

A-yon Lee — Thu, 05 Oct 2023 07:31:58 +0000

I’m not going to talk about the command-line flag --exprimental-network-imports, which isn’t working well at the time and lacks compatibility support (doesn’t support TypeScript as well).

Neither am I going to talk about DNT (Deno to NPM build tool), which isn’t working well with Node.js ecosystem. However, I will be talking about Deno, because this Node.js library I’m working on is a Deno/browser-compatible library. To be clear, this is NOT a library that allows us to use URL imports in our TypeScript, rather, it’s a library that uses URL imports in its codebase.

This library is called @hyurl/utils, which contains some useful functions I collected during the last several years in Node.js programming. Recently, I’ve been refactoring it to better support Deno and browsers. I’ve been doing a lot of Deno-compatible libraries recently, and just two days ago, I figured out the best way to refactor this library and not leave the Node.js ecosystem entirely.

I did not use DNT, If so, I won’t be able to test the code in Node.js, TS-Node more specifically, and that is not ideal because Node.js is still the primary platform of this library and needs to be tested. It’s way easier for Deno to test Node.js code than the other way around since Deno has added many Node.js compatibility layers to its runtime, and it’s safer in this way.

Below is a snapshot of some of the code in the library that uses URLs to import dependencies (these dependencies are also Node/Deno compatible, so they have equivalent versions in node_modules).

This code works well natively in Deno. But for TypeScript/VSCode to allow it, we have to add some magic to the tsconfig.json, and that is the import map (a.k.a compilerOptions.paths). To do so, I added these settings to the compilerOptions:



{
    // ...
    "noEmit": true,
    "baseUrl": "./node_modules/",
    "allowImportingTsExtensions": true,
    "moduleResolution": "Bundler",
    "types": [
        "node",
        "mocha" // I use mocha for testing, it works in Deno as well,
                // with a little magic, so I don't have to write tests twice.
                // This option is mandatory for Mocha to work in Deno.
    ],
    "paths": {
        "https://lib.deno.dev/x/is_like@latest/index.js": [
            "is-like"
        ],
        "https://lib.deno.dev/x/ayonli_jsext@latest/number/index.ts": [
            "@ayonli/jsext/number"
        ],
        // ...
    }
}

Now TypeScript/VSCode will work as well. But in order for TS-Node to work (for testing purposes), there are some other options that need to be set:



{
    // ...
    "ts-node": {
        "require": [
            "tsconfig-paths/register"
        ],
        "compilerOptions": {
            "module": "NodeNext"
        },
        "ignore": [] // required, otherwise TS-Node will ignore .ts files in node_modules
    }
}

Now TS-Node will work as well.

However I’m developing a Node.js module that will be published to NPM, so I have to replace the URLs with node modules in the compiled code. Since TSC will not work with this kind of code, I use Rollup to transpile the code instead.

Apart from the regular configurations and plugins that needed to work with TypeScript, I also added a plugin that called @rollup/plugin-replace to replace the URLs to node modules during build-time (I’ve also tried @rollup/plugin-alias, but it doesn’t work with URLs).

Since I’m building a library that transpiled to multiple targets (Node.js CJS and ESM, Browser ESM and Bundle), the rollup.config.mjs seems a little complicated, so I’ll just post the code here and explain some of the core concepts via comments.



import path from "node:path";
import { readFileSync } from "node:fs";
import { builtinModules } from "node:module";
import { fileURLToPath } from "node:url";
import { glob } from "glob";
import * as FRON from "fron";
import typescript from "@rollup/plugin-typescript";
import resolve from "@rollup/plugin-node-resolve";
import commonjs from "@rollup/plugin-commonjs";
import replace from "@rollup/plugin-replace";
import terser from "@rollup/plugin-terser";

/** @type {import("ts-node").TsConfigOptions} */
const tsCfg = FRON.parse(readFileSync("./tsconfig.json", "utf8"));
/** @type {Record<string, string>} */
export const importMap = Object.keys((tsCfg.compilerOptions.paths ?? {})).reduce((record, id) => {
    // Replace the URLs to their node modules according to the **compilerOptions.paths**.
    record[id] = (tsCfg.compilerOptions.paths ?? {})[id][0];
    return record;
}, {});
/** @type {Record<string, string>} */
const importMapWeb = Object.keys((tsCfg.compilerOptions.paths ?? {})).reduce((record, id) => {
    if (id.endsWith(".ts") && id.startsWith("https://lib.deno.dev/x/ayonli_jsext@latest/")) {
        // Package **ayonli_jsext** comes with native ESM files for the browser,
        // so replace the URLs of .ts files to the equivalent ESM .js files.
        record[id] = id.replaceAll(
            "https://lib.deno.dev/x/ayonli_jsext@latest/",
            "https://lib.deno.dev/x/ayonli_jsext@latest/esm/"
        ).slice(0, -3) + ".js";
    } else {
        record[id] = id;
    }
    return record;
}, {});
const entries = Object.fromEntries(
    glob.sync("**/*.ts", {
        ignore: ["node_modules/**", "**/*.test.ts", "**/*.d.ts", "**/*-deno.ts"],
    }).map(file => [
        file.slice(0, file.length - path.extname(file).length),
        fileURLToPath(new URL(file, import.meta.url))
    ])
);

/** @type {import("rollup").RollupOptions[]} */
export default [
    { // CommonJS
        input: entries,
        output: {
            dir: "cjs",
            format: "cjs",
            exports: "named",
            interop: "auto",
            sourcemap: true,
            preserveModules: true,
            preserveModulesRoot: ".",
        },
        external(id) {
            return String(id).includes("node_modules");
        },
        plugins: [
            replace({ ...importMap, preventAssignment: true }),
            typescript({
                moduleResolution: "bundler",
                baseUrl: "", // must reset this
            }),
            resolve({ preferBuiltins: true }),
            commonjs({ ignoreDynamicRequires: true, ignore: builtinModules }),
        ],
    },
    { // ES Module for Node.js
        input: entries,
        output: {
            dir: "dist",
            format: "esm",
            exports: "named",
            interop: "auto",
            sourcemap: true,
            preserveModules: true,
            preserveModulesRoot: ".",
        },
        external(id) {
            return String(id).includes("node_modules");
        },
        plugins: [
            replace({ ...importMap, preventAssignment: true }),
            typescript({
                moduleResolution: "bundler",
                compilerOptions: {
                    baseUrl: "", // must reset this
                    declaration: true,
                    declarationDir: "dist",
                },
                exclude: [
                    "*.test.ts",
                    "*-deno.ts"
                ]
            }),
            resolve({ preferBuiltins: true }),
            commonjs({ ignoreDynamicRequires: true, ignore: builtinModules }),
        ],
    },
    { // ES Module for Web
        input: entries,
        output: {
            dir: "esm",
            format: "esm",
            interop: "auto",
            sourcemap: true,
            preserveModules: true,
            preserveModulesRoot: ".",
        },
        external(id) {
            return String(id).startsWith("https://");
        },
        plugins: [
            replace({ ...importMapWeb, preventAssignment: true }),
            typescript({ moduleResolution: "bundler" }),
            resolve({ preferBuiltins: true }),
            commonjs({ ignoreDynamicRequires: true, ignore: builtinModules }),
        ],
    },
    { // Bundle
        input: "index.ts",
        output: {
            file: "bundle/index.js",
            format: "umd",
            name: "@hyurl/utils",
            sourcemap: true,
        },
        plugins: [
            replace({ ...importMap, preventAssignment: true }),
            typescript({ moduleResolution: "bundler" }),
            resolve({ preferBuiltins: true }),
            commonjs({ ignoreDynamicRequires: true, ignore: builtinModules }),
            terser()
        ],
    },
];

This config works well with the transpiled JS code and the URLs are replaced with node modules (and ESM URLs for the browser), however, it does not work with the output .d.ts files (don’t know why). So I ended up adding a postbuild.mjs file that looks like the following, to replace the URLs in the generated .d.ts files.



import { readFileSync, writeFileSync } from "node:fs";
import { glob } from "glob";
import { importMap } from "./rollup.config.mjs";

// Replace URL imports to node modules
for (const file of glob.sync("./dist/*.d.ts")) {
    let contents = readFileSync(file, "utf8");

    for (const [origin, target] of Object.entries(importMap)) {
        contents = contents.replaceAll(origin, target);
    }

    writeFileSync(file, contents, "utf8");
}

// Emit package.json for Node ESM
writeFileSync("./dist/package.json", `{"type":"module"}`);

Now when I run the command npx rollup -c rollup.config.mjs && node postbuild.mjs, it generates the correct code I want, the code that is suitable for Node and the web, while Deno can run with the source code.

But this isn’t the end, there are still some other things that need to be mentioned. In order to write the code once and run both in Node.js and Deno, three rules have to be honored:

Always use node: prefix for importing Node.js builtin modules (Yes, Deno has Node.js builtin module built-in). For example, import * as assert from "node:assert";
Always use .ts extension in the import path for relative imports. For example, import { ensureType } from "./index.ts";
For the test code (*.test.ts) files, DON’T import { describe, it } from "mocha"; because it will not work in Deno, use the global describe and it instead.

But how do I run the test, well, this is how:

For Node.js, use the command npx mocha -r ts-node/register *.test.ts
For Deno, obviously, this won’t work since the mocha command will always use node, however, mocha can be run in the browser, and Deno is a lot like the browser, so I wrote a script test-deno.ts to use the browser’s version of Mocha to run the tests in Deno instead. This is how:



import "https://unpkg.com/mocha/mocha.js";
import { dirname, fromFileUrl } from "https://deno.land/std/path/mod.ts";
import { globber } from "https://lib.deno.dev/x/globber@latest/mod.ts";

(window as any).location = new URL("http://localhost:0");
mocha.setup({ ui: "bdd", reporter: "spec" });
mocha.checkLeaks();

const files = globber({
    cwd: dirname(fromFileUrl(import.meta.url)),
    include: ["*.test.ts"],
    exclude: ["getGlobal.test.ts"],
});

for await (const file of files) {
    await import(file.absolute);
}

mocha.run((failures: number) => {
    if (failures > 0) {
        Deno.exit(1);
    } else {
        Deno.exit(0);
    }
}).globals(["onerror"]);

Now I can use deno run --allow-read --allow-net test-deno.ts to run the tests.

In this library, its dependencies are also Node/Deno compatible (have both versions), but what if we want to import some package that is only available on NPM (in node_modules), well, It’s quite simple, just use the npm: prefix as Deno suggested, and map the specifier to having no prefix. (In my opinion, this is just hilarious, and using the NPM version has some downsides, always prefer the raw version instead of the NPM version whenever we can.)

The blowing link is the repo of this library, check it out if you’re interested.

https://github.com/hyurl/utils

DEV Community: A-yon Lee

Use URL imports before Node/TypeScript supports it