DEV Community: asteinarson

Typescript, NodeJS and ES6/ESM Modules

asteinarson — Wed, 28 Apr 2021 12:06:01 +0000

I write this as a sequel to a previous article of getting ES6 module imports to work with other NPM modules, from TypeScript.

The CommonJS import framework (require) is how node.js was developed and getting everything smooth and working with modern static import is sometimes not so easy, and often not well documented.

I favor static imports, because the syntax and lack of obvious structure around require is just so full of magic and just works provided you have spent a number of years working inside that universe.

Basic settings

As a starter, whether a Node project defaults to CommonJS modules or ESM is decided by two settings - which must correspond.

tsconfig.json:

{
  "compilerOptions": {
    // ... 
    "module": "esnext", /* Or 'commonjs'. */

package.json:

{
   // ... 
   "type": "module",  /* Or 'commonjs' */

The problem here - relating to TypeScript

If I have a local TS file (utils.ts) like this:

// Invert keys and values of 1-level deep object
export function invert(o: Dict<BaseTypes>) {
    // ... implementation here ...
}

and then want to use it from another file:

import {invert} from './utils';

then that import will only work (everything else being default settings in TS and Node), as long as I stay with CommonJS modules.

However, when I change over to ESM modules (see below), the compiled Javascript will no longer work. Because Node.js will be trying to import exactly:

import {invert} from './utils';

And honestly, there is no such file - that is ./utils - without the file extension added to it.

The explanation is that as TypeScript transpiles this, then for CommonJS, it ends up calling require(), and some logic on top of that. And... require() accepts local javascript file names... without file extension.

So if I want my code to work with Node with ESM, I need to change it to:

import {invert} from './utils.js';

That means, I need to have different code bases, if I target CommonJS or ESM. Not very good.

Sort of going forwards...

For a while I accepted the compromise of adding .js to every local import... But then I wanted to add Jest testing on top of this. Which (together with ts-jest) does its own building of the test files (behind the scenes, I think using Babel). And ts-jest (whatever my tsconfig/package.json said) would not accept imports from explicit Javascript files:

import {invert} from './utils.js';  // ts-jest errors here

So I needed to fix it differently. (Understanding how/why ts-jest configures TypeScript/Node differently I did not want to enter).

A couple of solutions

One approach is to use the NPM module esm - however I never went in that direction. And I would like to avoid pulling in dependencies to solve this.

But it turns out there is a Node flag to do solve exactly this problem: --experimental-specifier-resolution=node. (You find it towards the bottom here).

Wow, all settled ?

Well, we also need to launch Node in some different ways (from terminal, from a Node bin script, from the VsCode debugger, and the last two presents small challenges).

Passing Node flags into the VsCode debugger

It turns out there is a launch.json entry for this:

    "configurations": [
        {
            "type": "pwa-node",
            // ...
            "runtimeArgs": ["--experimental-specifier-resolution=node"],
            "program": "${workspaceFolder}/src/cmd.ts",

(Scroll down to the bottom of this page for docs).

And it actually does the job (I found various other suggestions where to put the Node arg, however I think they targeted an old version of the Javascript debugger).

Passing Node flags into an Npm bin command

To run a JS script directly from the terminal (without prefixing it with node) we can use the bin section of package.json:

... 
"bin": {
    "my_cmd": "./path/to/my/script",
  },

However, if the target script is JS/TS (yes, it would be), we need to insert a shebang there, to make it executable:

#!/usr/bin/env node
// Rest of JS/TS here

But the obstacle here is that we cannot pass options to Node in the shebang. Sorry. We stumble into a bash fact of life here, and there's no simple way around it, for Node/TS/JS.

There is a magic hack to this over here. However that fulfills my definition of being so cryptic and non-intuitive (for anyone who did not patiently learn Bash internals) that I cannot recommend it.

The solution instead is to let the bin command point to a shell script, and let that one invoke Node with required options:

For me:

  "bin": {
    "knemm": "./shell/knemm",
  },

and knemm then being:

#!/bin/bash 
# Get directory of calling script
DIR="$( cd "$( dirname "$0" )" &> /dev/null && pwd )"
if [ "$(echo $DIR | grep '.nvm')" ]; then
    DIR="$(dirname "$(readlink -f "$0")")"
fi 
/usr/bin/env node --experimental-specifier-resolution=node $DIR/../lib/cmd-db.js $@

The explanation of the ''DIR'' part is that the current directory (inside my Node project) is lost when the command is invoke as a symlink (see below). I need to point to the JS file in a relative way, so I then need the directory. Here's background how to find the script dir.

To install this, as a global command, I run:

$ npm link

Then a global symlink appears:

$ knemm 
... command outputs, all ESM imports are resolved!

(At some point I needed to manually remove those symlinks generated by npm link as it wouldn't change those, when I edited package.json.)

Discussion

This has taken some hours (over some weeks) to work out, and I write this to summarize the effort and the learning. Partly so I remember better and then maybe it helps someone.

I hope all rough parts of using modern JS under Node will be gradually smoothened / paved out.

Of course, the last part of my solution is Linux/Bash centric. But nowadays, with WSL/WSL2, also anyone on Windows can access a good Linux environment. So I do not see a downside with that (that's how all of this was developed).

TypeScript / Node.js: Importing Knex into an ES6 module

asteinarson — Mon, 05 Apr 2021 17:02:13 +0000

A while back I wrote a file based import in TypeScript as a Node.js cli app. I used Knex for it, in a rather simple way, starting out from this code:



import Knex from 'knex'
import { Dict } from './utils.js'

let _knex: Knex;
export function connect(connection: Dict<string>) {
    let conn = {
        client: 'pg',
        connection
    }
    _knex = Knex(conn as Knex.Config);
}

It just worked, and I didn't think much into why it did, at the time.

I should add here that for Node.js projects I've tried to move over to using ES6 modules in my server side code (away from CommonJS). That can cause challenges, at times.

Yesterday, I wanted to do something similar, so I started a new project. A new package.json, and new tsconfig.json and I copied and pasted the code below. It no longer worked!

After a while, I found out that knex.js was resolved to version 0.21.18 in my original project and to 0.95.4 in my new project (by means of package.json).

Reconfiguring my new project back to CommonJS, I got it working, via this line in tsconfig.json:



    "module": "CommonJS", // It was 'ESNext' before

...and the corresponding in package.json (I removed 'type': 'module').

But I did not want to run my code on the server in CommonJS mode!

I felt the frustration of simply copying code and settings that has worked well before and sitting there with errors in my terminal... What had changed?

Different versions of Knex.js

So there was this significant jump, from v0.21.18 to v0.95.4. The issue must be there, somewhere.

I opened my two projects next to each other and popped up IDE type hints for the same imported Knex object. This is how it looked in the old project:

While this is how it looked (very similar code) in the new project:

If you look closely, you see that the first image contains a type alias for the Knex interface - this is missing in the second picture. In both cases, the code (behind the type hints) is:



import Knex from 'knex';

In the first case, the symbol Knex is apparently both the interface type of the Knex package and the function one can invoke, to connect with the database (the default export in CommonJS).

In the second case, the type information is not there anymore in the default import - it is just a function (with a signature). (If you look at my initial code segment you see that the exact identifier Knex is used in two quite different ways).

That was the difference.

How TypeScript gets away with using the same identifier as

A type (the Knex interface)
A function to be called

... I don't understand. But that was what dropped away between the earlier and later version of Knex.

Solution 1

So my change then was to name another import (to get both the function and the interface):



import { knex, Knex } from 'knex';

Then my new code actually builds and runs... but only in CommonJS mode. Built as an ES6 module, I get this on launching it:



$ node lib/cmd.js
file:///home/arst/src/mifl/lib/cmd.js:4
import { knex } from 'knex';
         ^^^^
SyntaxError: Named export 'knex' not found. The requested module 'knex' is a CommonJS module...

At this point... it felt like I had exhausted my ways forward. However, I remembered that the code originally was just one single default import. What about keeping that one, and on top of that, doing a named import of the TS interface?

Solution 2

This then was my new attempt:



import knex, { Knex } from 'knex';
let knex_conn: Knex;
async function connect(connection: Record<string, string>) {
    let conn = {
        client: 'pg',
        connection
    }
    knex_conn = knex(conn);
    return knex_conn;
}

And that turns out to work just fine, both when the code is built and run as CommonJS and as an ES module.

The key point is that the interface type and the function are two different things. And... to get to the actually exported object (from Knex.js), we have to use a default import from an ES module.

Summary

It took me a few hours to experiment my way here, and I did not really find very good TypeScript examples using this combination of both default and named imports - particularly when used from an ES module, neither in Knex documentation nor anywhere else.

What I did find (in issues for Knex on GitHub) was that people had issues running imports in TypeScript, and that some solved it by converting their code into CommonJS.

Knowing that things had worked just fine from an ES module for me, a year before, made me want to dig deeper. Thus this post.

I would guess the same pattern applies to many other primarily CommonJS based packages that one want to import with TypeScript bindings.

Exploring Directus 9

asteinarson — Sat, 03 Apr 2021 13:00:37 +0000

Intro

Maybe you have heard of the generic DB platform Directus, maybe not. In summary it is an intuitive database management tool that leans a bit towards being an application framework. I.e. the feeling when working in it is more that of designing an application specific solution (with exact use of a relational DB), as opposed to specifying exact details within tables and columns. It is this combination that makes it interesting to me.

For this article, both are fine. I will give you my description of it, point at the features I think are most innovative and describe various ways I can see myself using it.

Of course, the strengths and interesting use cases I can see depend largely on my particular background. Our backgrounds are likely a bit different. I will share a bit where I come from in this exploration. Maybe there is enough common ground, and maybe enough difference, for something interesting to be communicated. If so, stay put!

To clarify: I am not part of the Directus core team. I am however using it for small scale administrative tasks in my own work, and I see it as quite likely that I will use it product and in client work. And I do submit patches and suggestions to the project.

What is Directus ?

The basic idea is that we want to expose the structure and relationship aspects of SQL databases in an intuitive and application friendly way. The direction of it is to be able to build something specific and useful with Directus and an underlying SQL database. Basically with just working inside of the Directus app, in the browser.

In more entity parlor, Directus allows us to define entity types, their relations, to set up forms where we edit/interact with these entities and to display, filter and search tables (or collections) of such entities. All without coding! (at least to begin with...)

The Directus platform has been around and evolved for some 15+ years now. For more background look here - with history towards the bottom of the page. It is fully open source and available on GitHub.

I first got to know of Directus when it was implemented in PHP, and even if I liked aspects of it, that was a bit of a turn off for me (PHP as practical, as battle tested as it is, is maybe not where next generation technologies first appear). I've had my years in PHP (which were fine), but I was ready there.

A year ago, I discover that the team behind Directus had started migrating the server part of it over to Node.js and TypeScript. I was excited, as this was also where I was headed.

So version 9 of Directus is to be all Node.js and Typescript on the server side, and Vue.js and TypeScript on frontend. To my knowledge, it is one of the first times a more structural application framework is implemented within the Node.js platform.

At this date (early April 2021) Directus 9.0 is not released yet - but it is nearing.

My background and angle

I'm a generic developer who enjoy working on platforms that makes sense. The last decade or so I did backend / full stack work on some different ERP like platforms. The last being implementing e-commerce (and CMS) solutions on the Magento platform (yes... Php).

I learnt a few things on that road. One is that data types, schemas, entity types (sort of different words for the same thing) is ideally something flexible (or at least extensible, adaptable) on such platforms. In the end, because client needs are like that.

So it is good to be able to describe these schemas, these entity types, with a combination of more generic code together with some declarative / descriptive configuration.

I also learnt that (in the world of ERP and their relatives) this generality can become big task to accomplish - if it is to be useful.

And yes, there are some open source ERP solutions available that do this - like Odoo and ErpNext. As feature packed and full as these are, they also carry with them aspects such as large footprints, lots of legacy technology, high learning curves, little available (unpaid) support being available. Usually these systems are their own technology stacks - which are sometimes not so main stream. And the effort to implement a small system is usually significant (after all they are geared towards enterprise solutions and budgets).

So this is where something like Directus comes in. The possibility (with limited means) to build something particular, something structural, relying on the strengths on a relational database, with mostly configuration.

My "litmus test"...

I have a use case that can exercise these qualities. I want to see if it is possible to implement the accounting of my own company, largely relying on the generic capabilities of Directus.

Accounting (or bookkeeping), as boring as it can appear, relies heavily on structural relations. A verification can refer to an invoice, it contains several rows, where each refer to an account, and the sum of credit and debit on these rows is supposed to sum to exactly 0.

And that is just the beginning...

To make you worry less about my bookkeeping...: Yes, I do implement it in another system. But I'm partly motivated by the lack of flexibility (in that system) to pursue this avenue.