Write once, run anywhere with Deno and Dnt

dajiaji — Wed, 06 Jul 2022 12:34:33 +0000

Since multiple non-browser JavaScript runtimes has been emerged as well as web browsers, I've tried Deno-based module development which can support multiple web browsers, Node.js and Cloudflare Workers as well. I will share my efforts as a case study.

Introduction

I recently implemented a TypeScript module named hpke-js:

dajiaji / hpke-js

A Hybrid Public Key Encryption (HPKE) module built on top of Web Cryptography API.

hpke-js

A TypeScript Hybrid Public Key Encryption (HPKE) implementation build on top of Web Cryptography API This module works on web browsers, Node.js, Deno and various other JavaScript runtimes

Documentation: jsr.io | pages (only for the latest ver.)

For Node.js, you can install hpke-js via npm/yarn:

npm install @hpke/core
# if necessary...
npm install @hpke/dhkem-x25519
npm install @hpke/dhkem-x448
npm install @hpke/chacha20poly1305
# ...or you can use the v0.x-compatible all-in-one package below.
npm install hpke-js

Then, you can use it as follows:

import {
  Aes128Gcm,
  CipherSuite,
  DhkemP256HkdfSha256,
  HkdfSha256,
} from "@hpke/core";
async function doHpke() {
  const suite = new CipherSuite({
    kem: new DhkemP256HkdfSha256(),
    kdf: new HkdfSha256(),
    aead: new Aes128Gcm(),
  });

  // A recipient generates a key pair.
  const rkp = await suite.kem.generateKeyPair(

…

View on GitHub

HPKE (Hybrid Public Key Encryption) is, roughly speaking, a standard for exchanging public keys to create a shared key for secure end-to-end encryption. One of my goals was to implement this on the Web Cryptography API and guarantee that it works with multiple JS runtimes that support this API (major web browsers, Node.js, Deno, Cloudflare Workers), in other words, to properly incorporate testing in all runtimes into CI/CD.

I started out implementing it as an npm package, but in the process of making it Deno-compatible, I made a major rewrite to make it a Deno-based structure. This allowed me to build a much cleaner development and CI/CD flow for TypeScript/JavaScript modules that work on Chrome, Firefox, Safari, Node.js, Cloudflare Workers, and Deno.

Specifically, make my codebase be for Deno, use Deno built-in formatter, linter and testing, and use dnt (Deno to Node Transform) to generate an npm package containing ESM code and to test generated code. For testing in the browser, deploy the test content linked to the generated ESM code to Github Pages and use playwright/test. For Cloudflare Workers, use wrangler to set up a worker locally for testing. A schematic diagram is shown below.

I made the building/testing flow in this diagram be done by using Github Actions at pull request time and merge time to the master branch, and also made the flow including deployment be done at the release time by using Github Actions as well.

In this article, I will introduce the definition and overview of Deno-based "JS runtime-independent module" development, the various tools used and their settings, and CI/CD on Github, using hpke-js as an example to build the above flow.

This article is intended for modules that use APIs provided by JS runtimes, such as the Web Cryptography API, but still want to ensure portability. If it is obvious that your module is runtime-independent, there is no need to build a CI like the one introduced here.

Definition
JS Runtime-Independent Module Development
- Register your module to major Registries
- Directory Structure
Tools and the Settings
- deno
- dnt
- playwright/test
- wrangler
- eggs
CI/CD on Github
- CI for Deno
- CI for Browsers
- CI for Node.js
- CI for Cloudflare Workers
- Delivery
Remaining Problems
Conclusion

Definition

In this article, "JS runtime-independent modules" refers to modules that, after release, will be available in each JS runtime as follows:

Browsers: It is available in ESM format in browsers from major CDN services (esm.sh, Skypack, etc.). It has been tested and guaranteed to work in Chrome (Blink), Firefox (Gecko), and Safari (WebKit) before release.

<script type="module">
  import * as hpke from "https://esm.sh/hpke-js@0.13.0";
  // import * as hpke from "https://cdn.skypack.dev/hpke-js@0.13.0";
</script>

Node.js: It can be installed with npm or yarn and is available in both ESM and CommonJS formats. It has been tested and guaranteed to work with all Node.js versions that claim to support it.

// CommonJS
const hpke = require("hpke-js");
// or ESM
// import * as hpke from "hpke-js";

Deno: It can be installed via major registries such as deno.land and nest.land. It has been tested and guaranteed to work with all Deno major versions (currently only 1.x) that claim to support it.

import * as hpke from "https://deno.land/x/hpke@0.13.0/mod.ts";
// import * as hpke from "https://x.nest.land/hpke@0.13.0/mod.ts";

Cloudflare Workers: The single-filed module that is downloaded from various CDNs or emitted by deno bundle, can be included in a Cloudflare Worker package and can be used.

# download from a CDN (esm.sh)
curl -o $YOUR_PATH/hpke.js https://esm.sh/v86/hpke-js@0.13.0/es2022/hpke-js.js
# or downlaod a minified version from a CDN
curl -o $YOUR_PATH/hpke.js https://esm.sh/v86/hpke-js@0.13.0/es2022/hpke.min.js
# or use `deno bundle`
deno bundle https://deno.land/x/hpke@0.13.0/mod.ts > $YOUR_PATH/hpke.js

// then import and use it
import * as hpke from "./hpke.js";

JS Runtime-Independent Module Development

As mentioned in Introduction, the point is to develop it as a Deno module and use dnt (Deno to Node Transform) to convert it into code that works with other JS runtimes.

All you need to do is read the official documentation (README and doc.deno) and develop with portability in mind, but here are the main points to keep in mind, in my opinion:

Basically, do not use Deno-dependent funcitons. However, if you have to use a Deno namespace feature or any other feature that affects portability, check to see if it has a shim that is injected when converting to an npm package with dnt (see node_deno_shims. For example, the implementation status of the shims is listed up here). Using shim will ensure that it works on Node.js.
If your module has dependent packages, use esm.sh or Skypack as much as possible. If there are corresponding npm packages, dnt will map them to the dependencies in the output package.json. In other words, they are treated as external modules.
The entry point of the module should be mod.ts compliant with customary in Deno.
Since git tags are used for versioning in deno.land, make the tag name SemVer compliant (e.g., 1.2.3). v1.2.3 is also fine, but this will cause inconsistencies in the way of specifying versions in various CDNs (sometimes with v and sometimes without). I recommend that you use 1.2.3 without v.
If you want to output CommonJS/UMD format modules, do not use Top-level await.

NOTE: It should go without saying, but please keep in mind that even though a shim is provided as a mitigation/workaround, the basic premise is that portability cannot be basically ensured if non-standardized proprietary functions of a runtime are used.

Register your module to major Registries

To develop a JS runtime-independent module, you should register your module to the following two registries in advance:

Registration with npmjs is mandatory, and deploying here will also deploy to various CDNs(esm.sh、Skypack、unpkg.com, etc.).

As a Deno module, we would still like to be able to distribute it in deno.land. You can register it by clicking Publish a module from the link above and following the instructions; note that a Github repository is required. Note that in this article, we will register the Deno module not only in deno.land but also in nest.land. It seems that nest.land is a blockchain-based immutable registry.

nest.land

Another point to keep in mind is once you have decided on a module name, you should make sure that it is not registered in any of the above registries, and then pre-register it (I failed to do this...).

Directory Structure

We will get down to business here. The next section will introduce the various tools and their settings, but before that, let's take a look at the directory structure of hpke-js and its important files.

In the past, we have to prepare package.json, package-lock.json, esbuild scripts and configuration files for eslint, jest, typescript, typedoc, etc. It tended to get messy. But after changing to Deno-based developement, it is a little cleaner. There are four configuration files in the top directory, but egg.json is not important, so there are only three files.

deno.json: settings for deno.
dnt.ts: configuration and execution script for dnt.
import-map.json: for aggregating version descriptions of dependent libraries.
egg.json: for deploying to nest.land, not necessary if only deno.land is needed.

.
├── deno.json
├── dnt.ts
├── egg.json
├── import-map.json
├── mod.ts
├── README.md
├── src
│   └── *.ts
└── test
    ├── *.test.ts  # Unit tests for Deno, which can be transformed and executed for other runtimes.
    ├── pages      # E2E Test contents for browsers.
    │   ├── index.html
    │   └── src
    ├── playwright # E2E tests for Deno.
    │   ├── hpke.spec.ts
    │   ├── package.json
    │   └── playwright.config.ts
    └── wrangler   # E2E tests for Cloudflare Workers.
        ├── hpke.spec.ts
        ├── package.json
        ├── src
        │   └── index.js
        └── wrangler.toml

Tools and the Settings

I'll introduce the following tools but do not explain how to install or how to use them basically. Please refer to the official documentation for each. Basically, I will only put my setup and introduce some key points.

deno
dnt
playwright/test
wrangler
eggs

deno

I like that deno has a built-in formatter (fmt), linter (lint), test (test), and documentation (doc). It is very Cargo like.

The deno configuration file (deno.json) is optional and does not need to be present, but for development efficiency, it is better to register a series of commands used in development and CI in tasks and so on.

First of all, I'll put hpke-js/deno.json.

{
  "fmt": {
    "files": {
      "include": [
        "README.md",
        "CHANGES.md",
        "deno.json",
        "dnt.ts",
        "egg.json",
        "import-map.json",
        "samples/",
        "src/",
        "test/"
      ],
      "exclude": [
        "samples/node/node_modules",
        "samples/ts-node/node_modules",
        "src/bundles",
        "test/playwright/node_modules",
        "test/wrangler"
      ]
    }
  },
  "lint": {
    "files": {
      "include": ["samples/", "src/", "test/"],
      "exclude": [
        "samples/node/node_modules",
        "samples/ts-node/node_modules",
        "src/bundles",
        "test/playwright/node_modules",
        "test/wrangler"
      ]
    }
  },
  "importMap": "./import-map.json",
  "tasks": {
    "test": "deno fmt && deno lint && deno test test -A --fail-fast --doc --coverage=coverage --jobs --allow-read",
    "dnt": "deno run -A dnt.ts $(git describe --tags $(git rev-list --tags --max-count=1))",
    "cov": "deno coverage ./coverage --lcov --exclude='test' --exclude='bundles'",
    "minify": "deno bundle ./mod.ts | esbuild --minify"
  }
}

The points are as follows:

fmt supports markdown and json, so README.md and so on should be included in the target.
Since hpke-js uses npm for e2e testing and so on, exclude node_module from fmt and lint.
If you use imprt-map, you should use "importMap": ". /import-map.json" is required.
In tasks.test, both deno fmt and deno lint are executed at once.
In tasks.dnt, specify the version to put in package.json with $(git describe...).

dnt

dnt (Deno to Node Transform) is a build tool that creates npm packages from code for Deno. It is best to look at the official documentation (README and doc.deno).

But I'll expose hpke-js/dnt.ts as an example.

import { build, emptyDir } from "dnt";

await emptyDir("./npm");

await build({
  entryPoints: ["./mod.ts"],
  outDir: "./npm",
  typeCheck: true,
  test: true,
  declaration: true,
  scriptModule: "umd",
  importMap: "./import-map.json",
  compilerOptions: {
    lib: ["es2021", "dom"],
  },
  shims: {
    deno: "dev",
  },
  package: {
    name: "hpke-js",
    version: Deno.args[0],
    description:
      "A Hybrid Public Key Encryption (HPKE) module for web browsers, Node.js and Deno",
    repository: {
      type: "git",
      url: "git+https://github.com/dajiaji/hpke-js.git",
    },
    homepage: "https://github.com/dajiaji/hpke-js#readme",
    license: "MIT",
    main: "./script/mod.js",
    types: "./types/mod.d.ts",
    exports: {
      ".": {
        "import": "./esm/mod.js",
        "require": "./script/mod.js",
      },
      "./package.json": "./package.json",
    },
    keywords: [
      "hpke",
      // ...省略
    ],
    engines: {
      "node": ">=16.0.0",
    },
    author: "Ajitomi Daisuke",
    bugs: {
      url: "https://github.com/dajiaji/hpke-js/issues",
    },
  },
});

// post build steps
Deno.copyFileSync("LICENSE", "npm/LICENSE");
Deno.copyFileSync("README.md", "npm/README.md");

The points are as follows:

If you want to emit UMD code, you should use scriptModule: "umd".
If you use imprt-map, you should use "importMap": ". /import-map.json" is required.

playwright/test

This was my first time to use playwright/test and found it great. I was surprised at how easy it is to do E2E testing using a browser nowadays.

My hpke-js/test/playwright/playwright.config.ts is as follows:

import { devices, PlaywrightTestConfig } from "@playwright/test";

const config: PlaywrightTestConfig = {
  projects: [
    {
      name: "chromium",
      use: { ...devices["Desktop Chrome"] },
    },
    {
      name: "firefox",
      use: { ...devices["Desktop Firefox"] },
    },
    {
      name: "webkit",
      use: { ...devices["Desktop Safari"] },
    },
  ],
};
export default config;

For now, I activate chromium, firefox and webkit and I think it pretty much cover various browser environments.

Test code (hpke-js/test/playwright/hpke.spec.ts) is as follows. Just 9 lines.

import { expect, test } from "@playwright/test";

test("basic test", async ({ page }) => {
  await page.goto("https://dajiaji.github.io/hpke-js/");
  await page.click("text=run");
  await page.waitForTimeout(5000);
  await expect(page.locator("id=pass")).toHaveText("45");
  await expect(page.locator("id=fail")).toHaveText("0");
});

Basically, since the functionality of the module has been confirmed to some extent exhaustively by unit tests, in E2E using the actual environment, we have prepared test contents that use the Web Cryptography API with all HPKE cipher suite combinations (KEM: 5 types * KDF: 3 types * AEAD: 3 types = 45) and just hit the test button and see the results.

wrangler

wrangler is a CLI tool for Cloudflare Workers.

We could have done the same test for browsers, but for Cloudflare Workers, we implemented a test API with the following interface:

/test?kem={KEM_ID}&kdf={KDF_ID}&aead={AEAD_ID}

I ran this as a local server with wrangler dev --local=true and used deno test to perform E2E testing against this server. As with playwright/test above, I just ran a basic test scenario to check the Web Cryptography API calls with all combinations of the HPKE ciphersuites.

test API implementation: hpke-js/test/wrangler/src/index.js
E2E test executed via deno test: hpke-js/test/wrangler/hpke.spec.ts

eggs

eggs is a CLI tool to deploy a package to nest.land. My setting file is (hpke-js/egg.json) as follows. It's like a package.json.

{
  "$schema": "https://x.nest.land/eggs@0.3.4/src/schema.json",
  "name": "hpke",
  "entry": "./mod.ts",
  "description": "A Hybrid Public Key Encryption (HPKE) module for web browsers, Node.js and Deno.",
  "homepage": "https://github.com/dajiaji/hpke-js",
  "files": [
    "./src/**/*.ts",
    "./src/**/*.js",
    "README.md",
    "LICENSE"
  ],
  "checkFormat": false,
  "checkTests": false,
  "checkInstallation": false,
  "check": true,
  "ignore": [],
  "unlisted": false
}

The points are as follows:

You can define version information in eggs.json, but as with dnt, pass the latest tag information with the command argument (see eggs publish in Delivery).

CI/CD on Github

Using the various tools described in the previous section, the flows in the diagram in Introduction are straightforwardly dropped into Github Actions. In this section, I show each yml file for the following GitHub Actions.

CI for Deno
CI for Browsers
CI for Node.js
CI for Cloudflare Workers
Delivery

CI for Deno

hpke-js/.github/workflows/ci.yml

Basically, I just run "deno task test" and "deno task cov" defined in deno.json described before.
In addition, I'm using CodeCov for visualizing the coverage of the test.

name: Deno CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - name: Run deno test
        run: |
          deno fmt --check
          deno task test
          deno task cov > coverage.lcov
      - uses: codecov/codecov-action@v2
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: ./coverage.lcov
          flags: unittests

CI for Browsers

hpke-js/.github/workflows/ci_browser.yml

I deploy test content in the pages job and run E2E test in the playwright-test job.

name: Browser CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read

jobs:
  pages:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - run: |
          deno task dnt
          cp npm/esm/*.js test/pages/src/
          cp -rf npm/esm/src test/pages/src/
      - uses: peaceiris/actions-gh-pages@v3
        with:
          deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
          publish_dir: ./test/pages

  playwright-test:
    needs: pages
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
      - uses: microsoft/playwright-github-action@v1
      - working-directory: ./test/playwright
        run: npm install && npx playwright install && npx playwright test

CI for Node.js

hpke-js/.github/workflows/ci_node.yml

I run deno task dnt and deno task minify on multiple versions of Node.js (16.x, 17.x, 18.x).

name: Node.js CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [16.x, 17.x, 18.x]

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - name: Run dnt & minify
        run: |
          npm install -g esbuild
          deno task dnt
          deno task minify > ./npm/hpke.min.js

In addition, considering the size limitation of Cloudflare Workers, we tried to minify the JS file by esbuild to make it as compact as possible, but it did not make much sense as a result, because, for example, esm.sh, one of the deploying destinations, creates a minified JS file. hpke-js example has a normal size of 12KB, a minified version by esbuild of 6KB, and an esm.sh version of 6.5KB.

CI for Cloudflare Workers

hpke-js/.github/workflows/ci_cfw.yml

I run wrangler dev --local=true via npm start as a background task and then run deno test.

name: Cloudflare Workers CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - uses: actions/setup-node@v3
        with:
          node-version: v16.x
      - run: deno bundle mod.ts test/wrangler/src/hpke.js
      - name: Run test
        working-directory: ./test/wrangler
        run: |
          npm install
          nohup npm start &
          deno test hpke.spec.ts --allow-net

Delivery

hpke-js/.github/workflows/publish.yml

Deployments to npmjs.com and nest.land are performed with this Github Actions.
Deployment to deno.land is done at the time of tag creation via the API of deno.land registered in WebHook (set at the time of module registration).

name: Publish

on:
  release:
    types: [created]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: v16.x
          registry-url: https://registry.npmjs.org/
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - name: Run eggs
        run: |
          deno install -A --unstable https://x.nest.land/eggs@0.3.4/eggs.ts
          eggs link ${{ secrets.NEST_API_KEY }}
          eggs publish --yes --version $(git describe --tags $(git rev-list --tags --max-count=1))
      - name: Run dnt & minify
        run: |
          npm install -g esbuild
          deno task dnt
          deno task minify > ./npm/hpke.min.js
      - working-directory: ./npm
        run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{secrets.NPM_TOKEN}}

Remaining Problems

I have set up the CI/CD flows above, but I would like to add what I feel are some of the issues.

dependabot integration is currently not possible.
- I think this is the biggest disadvantage of using Deno (in my opinion), and I would like to let dependabot update the dependency packages in import-map.json.
Tests at the time of transformation by dnt cannot be executed in parallel.
- Unit tests in hpke-js take a long time to execute because of the huge number of test vectors in the standard, so.
To begin with, the current situation where there are many major JavaScript runtimes.

Conclusion

The current situation where there are many JS runtime is still hard. As mentioned in this article, the use of dnt and Github Actions can alleviate some of the difficulty, but I would still like to see more portability ensured within the framework of standardization.

We have high expectations for the W3C Winter CG.

Thank you for reading my poor English. See you!

DEV Community: dajiaji

Write once, run anywhere with Deno and Dnt

Introduction

dajiaji / hpke-js

A Hybrid Public Key Encryption (HPKE) module built on top of Web Cryptography API.

hpke-js

Table of Contents

Definition

JS Runtime-Independent Module Development

Register your module to major Registries

Directory Structure

Tools and the Settings

deno

dnt

playwright/test

wrangler

eggs

CI/CD on Github

CI for Deno

CI for Browsers

CI for Node.js

CI for Cloudflare Workers

Delivery

Remaining Problems

Conclusion