DEV Community: Carlos Saito

How to maintain a huge repo with sporadic access

Carlos Saito — Wed, 08 Apr 2020 06:15:54 +0000

In our team we have the healthy habit of automating our process in simple ways.

One simple way of doing it is to create scripts, snippets of code that are just run from a developer computer. It could include things like calling an API and writing a CSV file from there. We called them scripts but it can be called "CLI apps" if you prefer it. Some scripts we use them only once, others we use them every now and then (like once in a month)...

We have one repo with several of those scripts. Each script is in a directory. All are in Node.js so every directory has a package.json file. To run it, you simply do cd «directory»; npm install; node .

How do you maintain and keep updated a repo like that? We don't want to check every dependency of every directory... Maybe a monorepo?

All suggestions are welcome :)

Practical guide: Node.js 12 supports ES modules. Are your apps ready?

Carlos Saito — Tue, 23 Jul 2019 14:52:09 +0000

Updated on May 2020. By October 2020 Node.js 14 will be released. Some things have changed since the time I've written this article (July 2019)

If you are a Node.js developer either by writing Node.js apps or libraries, you probably know that Node.js 12 supports ECMAScript standard modules!

To enable ESM in Node.js 12, you still need the --experimental-modules flag. Read more in the official docs
Node.js 14 is probably going to be released with support for ESM without the flag. Read more in the official docs

The question is… Are your apps and libraries ready?

This post explores the way of writing modular apps and libraries for Node.js 12 without using any build process. The code written is exactly the code executed.

⚠️ Caution! This post is written with the "consensus" in May 2020 and the current behavior of Node.js (Node.js 12 with --experimental-modules flag and Node.js 14 without it). It is likely but not guaranteed that 14 will behave the same once it reaches LTS in October. Whenever possible, references to official 📝 Node.js documents will be included so you can read them and take your own decisions.

🧪 Things that are currently experimental but likely to remain as they are when once Node.js reaches LTS are marked with this symbol. Read them to be an early adopter of ES modules almost from the first day but don't use it in production before the LTS release.

💥 Things marked with this symbol cause breaking changes. This is specially important for library authors. They should not be used in production before the LTS release.

🛑 Things marked with this symbol are more dangerous to do, even for early adopters as they might change in the future.

The transition phase. Plan it!

Ideally Node.js projects will gradually adopt ES modules and stop using CommonJS. This is possible from Node.js 12 and it depends on developers adoption grade (and the ones of your external dependencies).

In Node.js 12 and 14, CommonJS is still the default option but ES modules would be the preferred one in future versions.

Take your time to plan the transition. Otherwise you might have two or three versions of the same project to maintain at the same time.

What can you do?

Summary

This is a 4-level transition roadmap example:

	No adoption	Soft adoption	Hard adoption	Full adoption
		🧪	💥	🛑
CommonJS	default	default	deprecated	dropped
ES modules	disabled	enabled	default	default

No adoption and full adoption levels are the equivalent of using either CommonJS only or ES modules only and needs no more explanation.

Let's dig into soft and hard adoption levels

Soft adoption. Enable ES modules, keep CommonJS as default

Goals:

Write your own code with ESM syntax.
Offer a ESM API.
Start deprecating the CJS API.
Do not do any breaking changes.
Be ready to drop CJS.

Since this is all about dependencies, in case that some some files are in ESM and some in CJS, you will face:

index.js (migrated to ESM) depends on module.js (in CJS)
index.js (in CJS) depends on module.js (migrated to ESM)

`index.js` (migrated to ESM) depends on `module.js` (in CJS)

In this case, we are migrating first index.js keeping the dependencies in CommonJS.

Rename index.js to index.mjs to enable ESM on that file and disable CJS. Now, require doesn't work anymore in this file.

If some of your dependencies are still in CommonJS, make sure to understand the interoperability between those styles.

`index.js` (in CJS) depends on `module.js` (migrated to ESM)

❓🧪 Use this depending on your trust on esm npm package or your confidence on the import() operator

Make sure that your dependencies have .esm extension. Use both named and default exports accordingly.

You need to understand the interoperability between those styles. In particular, you need to understand the npm package esm or the import() operator

As library author

🧪 These changes are not breaking. However, since ES modules are still in experimental phase, it is recommended that you perform a preminor or prepatch version or any other "beta" upgrade: run npm version preminor, npm version prepatch or similar. Publishing a stable release of a library forcing experimental features to be enabled to the users is not a good idea.

Goals as library author:

Offer two entry points: CommonJS module and ES module.
Keep the CommonJS entry point as the main one.
Recommend the mjs alternative. 🧪

If your entire library is migrated, your ESM entry file will be something like index.mjs:

// my-lib/index.mjs
export default function sayHello (name) {
  return `Hello from my-lib/esm: ${name}`
}

Create a index.cjs. Once you change your library to support esm by default, this will be the entry point for the legacy code

// my-lib/index.cjs
require = require('esm')(module)
module.exports = require('./index.mjs')

Deliver both index.mjs and index.cjs as part of your npm package (edit your package.json). Serve index.cjs as the "main" entry

{
  "main": "index.cjs",
  "files": [
    ...,
    "index.mjs",
    "index.cjs"
  ],
}

📝 Node.js may adopt a field name in package.json specifically for ES modules. In that case, you could include both "name" and that new field as well. Follow the discussion

In this stage, library users should explicitly access index.mjs if they want to use the ESM version via an explicit path.

import sayLibrary from 'my-lib/index.mjs'
console.log(sayLibrary('World'))

📝 Read more in the Node.js help

Alternative. Keep the `index.js` file (main entry) as a deprecated copy of `index.cjs`

Offer three entry points keeping index.js as the main one.

{
  "main": "index.js",
  "files": [
    "index.js",
    "index.mjs",
    "index.cjs"
  ],
}

Deprecate index.js in order to encourage CommonJS users to require the index.cjs file explicitly.

// my-lib/index.js
require = require('esm')(module)
process.emitWarning('This library will be a ES module in the next major version. If you still need to use the CommonJS version, require("my-library/index.cjs") instead')
module.exports = require('../index.mjs')

The library can also recommend to use the ES module version.

🧪 "Recommending" an experimental solution from a stable release is not a good idea. Remember to publish your library as an unstable release (like pre-release, alpha, etc.)

The library is required/imported like this:

	Before	Now	Next version
`require(my-lib)`	OK	deprecated	dropped
`require(my-lib/index.cjs)`	N/A	legacy	deprecated
`import my-lib/index.mjs`	N/A	🧪 OK	OK
`import my-lib`	N/A	N/A	recommended

Extra. Libraries with more than one file

If you offer multiple endpoints from your library like this:

const f1 = require('my-library/function1')
const f2 = require('my-library/function2')

📝 You might want to read the Package Exports Proposal (still in "pre-PR" status)

Additional notes of this phase

New code should be written with ES modules directly.
If you are writing a library, be sure that you are using the correct standard
If you are writing a library, make sure to offer a CommonJS alternative and make it the default one, even if you want to deprecate it. Make sure to test it against Node.js < 12

Transition 2. Enable ESM as default. Support CommonJS

🛑 Do not do this in production until ESM is supported without experimental flags.

Goals:

Work with ES modules by default
Still support legacy CommonJS but deprecate it
Drop deprecated code

Make the breaking change!

In the package.json file of your project add:

{
  "type": "module"
}

Now all the .js files are treated as .mjs. You cannot use require anymore in any .js or .mjs file.

Both .js and .mjs extensions works seamlessly.

You could use both to differ between scripts vs modules… But that's only an opinion. For example, you can keep an index.js file but use .mjs as extension for its dependencies.

Rename all the .js files that still use CommonJS to .cjs. Fix the imports/exports paths

Extra. Deprecate the remaining .cjs files (you should drop them soon!).

Library author. Make the breaking change!

💥 Every step in this section causes breaking changes. If you are a library author, you might want to do a single major release with all the changes

🧪 Also every step in this section in experimental status.

Recommendation: do a major + non-stable release: npm version premajor.

⚠️⚠️⚠️ In the package.json file of your project add "type" (💥 changing "type" to "module" is also a breaking change) and point to index.mjs file as the entry of the library:

{
  "type": "module",
  "main": "index.mjs" ⚠️⚠️⚠️
}

⚠️⚠️⚠️ It is not guaranteed that the field will be called "main".

📝 Actually the ofificial announcement says:

«While we are aware that the community has embraced the "module" field, it is unlikely that Node.js will adopt that field […] Please do not publish any ES module packages intended for use by Node.js until this is resolved.»

📝 However, the roadmap documentation says:

«Status quo is current --experimental-modules behavior: "main" points to exactly one file, and all file extensions are mandatory (by default), so there is no possibility of an import specifier pointing to different files in ESM versus CommonJS.»

«Status quo has consensus and will ship absent any other proposals achieving consensus. “New field” proposal lacks consensus.»

After you have set "type" to "module", using bare-paths with CommonJS (require(my-lib)) no longer works (💥 Changing the way to reach your API is a breaking change)

Once both index.mjs and index.cjs are reachable, you can delete the index.js file. Additionally, you can add a deprecation warning in index.cjs if you are planning to drop CommonJS support.

require = require('esm')(module)
process.emitWarning('CommonJS support will end in the next major version of this library')
module.exports = require('../index.mjs')

You can also mark esm (the library that we use only for legacy support) as "optional dependency" of your library. Users that uses ES modules do not need to install the library. 💥 Converting a dependency to optional is always a breaking change

	Now	Next
`require(my-lib)`	dropped💥	dropped
`require(my-lib/index.cjs)`	deprecated	dropped
`import my-lib/index.mjs`	OK 🧪	OK
`import my-lib`	OK 🧪	OK

That's it!

In future posts I will probably mention something about authoring other types of libraries: libraries for/written in TypeScript, libraries for frontend javascript, isomorphic libraries… Who knows!

Also I want to discuss about the consequences of adopting ES modules: code completion from text editors, follow the standards, etc.

Node.js 12 supports ES modules. Do you know the difference between CommonJS and ES2015+ Modules?

Carlos Saito — Mon, 15 Jul 2019 11:44:06 +0000

If you are a Node.js developer either by writing Node.js apps or libraries, you probably know that Node.js 12 supports ECMAScript standard modules! ~~(the feature will probably be stable without any experimental flag starting from LTS release this october)~~. EDIT: Node.js 12 has not dropped the need of the --experimental-modules flag. Read more in the official documentation

Do you know what are the differences between CommonJS and ES modules?

Pre Node.js 12. CommonJS (a.k.a. CJS)

Export and import

We have two ways of exporting, named and default exports

// commonjs/named.js
module.exports.sayHello = function sayHello (name) { return `Hello ${name}` }

// commonjs/default.js
module.exports = function sayHello (name) { return `Hello ${name}` }

And two ways of importing:

// index.js
// Named import without changing the name
const { sayHello } = require('./commonjs/named')

// Named import changing the name
const { sayHello: say2 } = require('./commonjs/named')

// Default import
const sayDefault = require('./commonjs/default')

console.log(sayHello('World'))
console.log(say2('World'))
console.log(sayDefault('World'))

There are some alternatives in both exporting and importing like those but they are equivalent:

// Named import
const say2 = require('./commonjs/named').sayHello

// Named export
module.exports = {
  sayHello: function sayHello (name) {
    return `Hello ${name}`
  }
}

Bare-paths. Module resolution in Node.js

require in Node.js accepts a bare path so we can declare/export libraries from a node_modules directory:

// node_modules/my-lib/package.json
{ "main": "index.js" }

// node_modules/my-lib/index.js
module.exports.sayHello = function sayHello (name) { return `Hello ${name}` }

And import them (Node.js resolves my-lib to ./node_modules/my-lib/index.js):

// index.js
const say3 = require('my-lib')
console.log(say3('World'))

The future. ES Modules (a.k.a. ESM)

Export and import

Like in CommonJS, there are two ways of exporting: named and default.

// esm/named.js
export function sayHello (name) { return `Hello ${name}` }

// esm/default.js
export default function sayHello (name) { return `Hello ${name}` }

And two ways of importing:

// index2.js
// Named import without changing the name
import { sayHello } from './esm/named.js'

// Named import changing the name
import { sayHello as say2 } from './esm/named.js'

// Default import
import sayDefault from './esm/default.js'

console.log(sayHello('World'))
console.log(say2('World'))
console.log(sayDefault('World'))

Note that the following "alternatives" exist but are not equivalent to a named export. Do not use them as equivalent to named exports

// This is NOT a named export!!
export default {
  sayHello: function (name) {
    return `Hello ${name}`
  }
}

// This will not work with the above!
import { sayHello } from './esm/variation.js'

// This works but is NOT a named import
import say from './esm/variation.js'
const { sayHello } = say

Bare paths. Module name resolution

Node.js 12 resolves bare paths properly:

// node_modules/my-esm-lib/package.json
{ "main": "index.js" }

// node_modules/my-esm-lib/index.js
export default function sayHello (name) { return `Hello ${name}` }

And import them (Node.js resolves my-esm-lib to ./node_modules/my-esm-lib/index.js):

// index2.js
import say3 from 'my-esm-lib'
console.log(say3('World'))

Interoperability

Import a CJS module into a ESM project

The dependencies are still in CommonJS:

// commonjs/named.js
module.exports.sayHello = function sayHello (name) { return `Hello ${name}` }

// commonjs/default.js
module.exports = function sayHello (name) { return `Hello ${name}` }

So you need to know what happens when you ~~require~~ import them to a ESM file.

All the module.exports object in CJS will be converted to a single ESM default export. You cannot use ESM named exports when importing CommonJS modules.

📝 From the Node.js roadmap plans: «Status quo is current --experimental-modules behavior: import only the CommonJS default export, so import _ from 'cjs-pkg' but not import { shuffle } from 'cjs-pkg')»

// index.mjs
// "Fake named import" without changing the name
import named from './commonjs/named.js'
const { sayHello } = named

// "Fake named import" changing the name
import named2 from './commonjs/named.js'
const { sayHello: say2 } = named2

// Default import
import sayDefault from './commonjs/default.js'

console.log(sayHello('World'))
console.log(say2('World'))
console.log(sayDefault('World'))

Alternative: make an intermediate module.

Enable real ESM named imports by creating an intermediate module:

// bridge/named.mjs
import named from '../commonjs/named.js'
export const sayHello = named.sayHello

Import it as named import

// index.mjs (with bridged modules)
// Named import without changing the name
import { sayHello } from './bridge/named.mjs'

// Named import changing the name
import { sayHello as say2 } from './bridge/named.mjs'

Import a ESM module into a CJS project

Your dependencies are now in ESM:

// esm/named.mjs
export function sayHello (name) { return `Hello ${name}` }

// esm/default.mjs
export default function sayHello (name) { return `Hello ${name}` }

To require them from a CommonJS file, you can use the npm package esm. This "special" require returns everything as an object of named imports. The ESM default export becomes a named import called .default on the returned object

const esmRequire = require('esm')(module)

// Named import without changing the name
const named = esmRequire('./esm/named.mjs')
const { sayHello } = named

// Named import changing the name
const { sayHello: say2 } = named

// "ESM default export" becomes a named import called "default"
const sayDefault = esmRequire('./esm/default.mjs').default

console.log(sayHello('World'))
console.log(say2('World'))
console.log(sayDefault('World'))

If you don't want to use an external package, use the import() operator. Notes:

import() returns a Promise. So you need .then() or await
import() returns everything as an object of named imports. To access the default-exported thing, you need to access the property .default on the returned object

ℹ️ Read more about import() here

// index.js
;(async function () {
  // Named import without changing the name
  const named = await import('./esm/named.mjs')
  const { sayHello } = named

  // Named import changing the name
  const { sayHello: say2 } = named

  // Default import
  const sayDefault = (await import('./esm/default.mjs')).default

  console.log(sayHello('World'))
  console.log(say2('World'))
  console.log(sayDefault('World'))
})()

Alternative: make intermediate modules using the `esm` package

Enable CJS default export:

// bridge2/default.js
require = require('esm')(module)
module.exports = require('../esm/default.mjs').default

Make other libraries ready for CJS import

// bridge2/named.js
require = require('esm')(module)
module.exports = require('../esm/named.mjs')

And require them:

// Named import without changing the name
const named = require('./bridge2/named.mjs')
const { sayHello } = named

// Named import changing the name
const { sayHello: say2 } = named

// Default import
const sayDefault = require('./bridge2/default.mjs')

That's it!

The next post will be about how to prepare your Node.js apps and libraries to support ES modules as soon as possible!

ES2018. Real-life simple usage of async iteration: get paginated data from REST APIs in 20 lines of code

Carlos Saito — Mon, 24 Sep 2018 11:53:49 +0000

The next JavaScript standard, ES2018 is here and it comes with a new big feature: asynchronous iteration. It is a enormously useful feature and I want to share with you one super simple example on how can we use it in real life.

In this post I am NOT going to explain what are async iterators or iterators. You can get those explanations in here or here

The problem. We want to fetch data from an API that is returned paginated and do stuff with every page. For example, we want to fetch all the commits of a Github repo and do some stuff with those data.

We want to separate the logic of "fetching commits" and "do stuff", so we are going to use two functions. In a Real Life™ scenario, fetchCommits would be probably in a different module and the "do stuff" part will call fetchCommits somehow:

// Imagine that this function is in a different module...
function fetchCommits(repo) {}


function doStuff() {
  const commits = fetchCommits('facebook/react')
  // do something with `commits`
}

Now, Github API will return commits paginated (like most of the REST APIs) so we will fetch the commits "in batches". We want to implement this "pagination" logic somehow in fetchCommits.

However we don't want to return all the commits together in fetchCommits, we want to do some logic for each page when they come and implement such logic in the "do stuff" part.

Solution without async iteration

To do it, we were somehow forced to use callbacks:

// Here we "do stuff"
fetchCommits('facebook/react', commits => {
  // do something with `commits`
}

Can we use Promises?. Well, not in this way because we will get only one page or the whole thing:

function doStuff() {
  fetchCommits('facebook/react').then(commits => {
    // do something
  })
}

Can we use sync generators? Well... we could return a Promise in the generator and resolve that promise outside it.

// fetchCommits is a generator
for (let commitsPromise of fetchCommits('facebook/react')) {
  const commits = await commitsPromise
  // do something
}

This is actually a clean solution, but how is the implementation of the fetchCommits generator?

function* fetchCommits(repo) {
  const lastPage = 30 // Must be a known value
  const url = `https://api.github.com/${repo}/commits?per_page=10`

  let currentPage = 1
  while (currentPage <= lastPage) {
    // `fetch` returns a Promise. The generator is just yielding that one.
    yield fetch(url + '&page=' + currentPage)
    currentPage++
  }
}

Not bad solution but we have one big issue here: the lastPage value must be known in advance. This is often not possible since that value comes in the headers when we do a first request.

If we still want to use generators, then we can use an async function to get that value and return a sync generator...

async function fetchCommits (repo) {
  const url = `https://api.github.com/${repo}/commits?per_page=10`
  const response = await fetch(url)

  // Here we are calculating the last page...
  const last = parseLinkHeader(response.headers.link).last.url
  const lastPage = parseInt(
    last.split('?')[1].split('&').filter(q => q.indexOf('page') === 0)[0].split('=')[1]
  )

  // And this is the actual generator
  return function* () {
    let currentPage = 1
    while (currentPage <= lastPage) {
      // And this looks non dangerous but we are hard coding URLs!!
      yield fetch(url + '&page=' + currentPage)
      currentPage++
    }
  }
}

This is not a good solution since we are literally hard-coding the "next" URL.

Also the usage of this could be a bit confusing...

async function doStuff() {
  // Calling a function to get...
  const getIterator = await fetchCommits('facebook/react')

  // ... a function that returns an iterator???
  for (const commitsPromise of getIterator()) {
    const value = await commitsPromise
    // Do stuff...
  }
}

Optimally, we want to obtain the "next" URL after every request and that involves to put asynchronous logic in the generator but outside of the yielded value

Async generators (`async function*`) and `for await` loops

Now, async generators and asynchronous iteration allow us to iterate through structures where all the logic outside of the yielded value is also calculated asynchronously. It means that, for every API call we can guess the "next URL" based on the headers and also check if we reach the end.

In fact, this could be a real implementation:

(The example works in node >= 10)

const rp = require('request-promise')
const parseLinkHeader = require('parse-link-header')

async function* fetchCommits (repo) {
  let url = `https://api.github.com/${repo}/commits?per_page=10`

  while (url) {
    const response = await request(url, {
      headers: {'User-Agent': 'example.com'},
      json: true,
      resolveWithFullResponse: true
    })

    // We obtain the "next" url looking at the "link" header
    // And we need an async generator because the header is part of the response.
    const linkHeader = parseLinkHeader(response.headers.link)

    // if the "link header" is not present or doesn't have the "next" value,
    // "url" will be undefined and the loop will finish
    url = linkHeader && linkHeader.next && linkHeader.next.url
    yield response.body
  }
}

And the logic of the caller function gets also really simple:

async function start () {
  let total = 0
  const iterator = fetchCommits('facebook/react')

  // Here is the "for-await-of"
  for await (const commits of iterator) {
    // Do stuff with "commits" like printing the "total"
    total += commits.length
    console.log(total)

    // Or maybe throwing errors
    if (total > 100) {
      throw new Error('Manual Stop!')
    }
  }
  console.log('End')
}
start()

Do you have any other examples on how to use async generators?

DEV Community: Carlos Saito

How to maintain a huge repo with sporadic access

Practical guide: Node.js 12 supports ES modules. Are your apps ready?

The transition phase. Plan it!

Summary

Soft adoption. Enable ES modules, keep CommonJS as default

`index.js` (migrated to ESM) depends on `module.js` (in CJS)

`index.js` (in CJS) depends on `module.js` (migrated to ESM)

As library author

Alternative. Keep the `index.js` file (main entry) as a deprecated copy of `index.cjs`

Extra. Libraries with more than one file

Additional notes of this phase

Transition 2. Enable ESM as default. Support CommonJS

Make the breaking change!

Library author. Make the breaking change!

That's it!

Further reading:

Node.js 12 supports ES modules. Do you know the difference between CommonJS and ES2015+ Modules?

Pre Node.js 12. CommonJS (a.k.a. CJS)

Export and import

Bare-paths. Module resolution in Node.js

The future. ES Modules (a.k.a. ESM)

Export and import

Bare paths. Module name resolution

Interoperability

Import a CJS module into a ESM project

Alternative: make an intermediate module.

Import a ESM module into a CJS project

Alternative: make intermediate modules using the `esm` package

That's it!

Further reading

ES2018. Real-life simple usage of async iteration: get paginated data from REST APIs in 20 lines of code

Solution without async iteration

Async generators (`async function*`) and `for await` loops

DEV Community: Carlos Saito

How to maintain a huge repo with sporadic access

Practical guide: Node.js 12 supports ES modules. Are your apps ready?

The transition phase. Plan it!

Summary

Soft adoption. Enable ES modules, keep CommonJS as default

index.js (migrated to ESM) depends on module.js (in CJS)

index.js (in CJS) depends on module.js (migrated to ESM)

As library author

Alternative. Keep the index.js file (main entry) as a deprecated copy of index.cjs

Extra. Libraries with more than one file

Additional notes of this phase

Transition 2. Enable ESM as default. Support CommonJS

Make the breaking change!

Library author. Make the breaking change!

That's it!

Further reading:

Node.js 12 supports ES modules. Do you know the difference between CommonJS and ES2015+ Modules?

Pre Node.js 12. CommonJS (a.k.a. CJS)

Export and import

Bare-paths. Module resolution in Node.js

The future. ES Modules (a.k.a. ESM)

Export and import

Bare paths. Module name resolution

Interoperability

Import a CJS module into a ESM project

Alternative: make an intermediate module.

Import a ESM module into a CJS project

Alternative: make intermediate modules using the esm package

That's it!

Further reading

ES2018. Real-life simple usage of async iteration: get paginated data from REST APIs in 20 lines of code

Solution without async iteration

Async generators (async function*) and for await loops

`index.js` (migrated to ESM) depends on `module.js` (in CJS)

`index.js` (in CJS) depends on `module.js` (migrated to ESM)

Alternative. Keep the `index.js` file (main entry) as a deprecated copy of `index.cjs`

Alternative: make intermediate modules using the `esm` package

Async generators (`async function*`) and `for await` loops