DEV Community: Kenneth Lim

ES Modules and how we got where we are

Kenneth Lim — Tue, 25 Feb 2025 21:35:55 +0000

This is a public draft of this article, there will likely be corrections, edits, rewrites for any parts below. It is a long read and I plan to eventually break it out into smaller parts when I republish it somewhere else in a more consolidated way.

We know from the history of JavaScript, at the very start, we didn't anticipate JavaScript to become so prevalent, it definitely wasn't certain JavaScript will be anything more than just an experiment. Other experiments to run code in the browser such as Flash, Java Applet, and Silverlight have come and went with JavaScript now being the de facto standard way to run code in the browser (WebAssembly technically is another way to run code in the browser but it still needs JavaScript as an interface).

That is all an excuse to say that it wasn't really anticipated at the beginning that JavaScript will be used to build the highly complex web apps and even server apps that we are using JavaScript for nowadays. Typical JavaScript files may had been at most a few kilobytes 20 years ago gradually grew and nowadays if we don't apply the different techniques that were developed over the years to keep things compact, we will be looking at several megabytes of JavaScript files being transfer for each page load, easily thousands of times more data that needed to be transferred, massively slowing page load speed down.

The more we want our web app to be capable of doing, the more complex our code becomes. The more complex our code becomes, the harder it is to manage effectively. The more features we wanted, the more code we need. The more code we need, the larger the JavaScript size we need to send to the browser. What we are trying to balance here is a tug of war from three sides:

Feature and capabilities
Transfer size/speed
Code maintainability

Stage 1 - The default pattern

Let's start at the beginning and work our way to today (as of time of writing). The basic way of including JavaScript in a web page is the <script> tag.

<script src="./script.js"></script>

I'll leave out inline script in both script tag and HTML elements as they are functionally the same. As our code get more and more complex, we would like to have more than one file to help increase maintainability: we can split up utility functions, class definitions, etc in separate files:

<script src="./utils.js"></script>
<script src="./User.js"></script>
<script src="./script.js"></script>

This is similar to how you can include libraries by including their script tags. This works because for any script that are included in the web page with a <script> tag, they all use the same global variable scope: a global variable defined in one script file is accessible in another script file directly. Now we have Feature and capabilities and some Code maintainability but in the process, we have sacrificed Transfer size/speed. How so you may ask?

The reason is a little complicated and I won't go into detail here, Rolldown has an excellent writeup for why that is the case. The short version of it is that, with each of the script tag we have, they are individual HTTP requests. Each HTTP requests takes time that is in addition to the actual data transfer time, which means that we want to minimize the number of HTTP requests: instead of making three HTTP requests in the above snippet, it would be faster if we just make one HTTP request for the same contents. That means we need to somehow combine the three files into one, ie. some sort of bundling.

Stage 2 - Line them up

To understand this next stage, we will need some example content in our scripts.

// utils.js
function capitalize(str){
  return str.charAt(0).toUpperCase() + str.slice(1);
}

// User.js
class User {
  constructor(name){
    this.name = capitalize(name);
  }
}

// script.js
let user;
const button = document.createElement("button");
button.innerHTML = "Create user";
button.addEventListener("click", () => {
  user = new User("sam");
});

console.log(user.name) // Logs "Sam"

Now what we want to achieve is a single JavaScript file that our browser can include with a single <script> tag that will do the same thing as if we included all three files separately. Well that's easy, because if you remember, all <script> tag scripts on a web page has the same global variable scope, which means that they are functionally the same if they were all in the same file:

// script.js
function capitalize(str){
  return str.charAt(0).toUpperCase() + str.slice(1);
}

class User {
  constructor(name){
    this.name = capitalize(name);
  }
}

let user;
const button = document.createElement("button");
button.innerHTML = "Create user";
button.addEventListener("click", () => {
  user = new User("sam");
});

console.log(user.name) // Logs "Sam"

<script src="./srcipt.js></script>

This will only make one HTTP request to get script.js which fulfills our requirements. But that's where we started from! We split things up into separate files intentionally to make things more maintainable, if we combine them again we are just going around in circles, trading Code maintainability for Transfer size/speed. To get the best of both worlds, we want to write our code in separate files which we then combine into a single file before we let our server serve it to the browser. This is where the first bundling technique in JavaScript comes from, it's the very advanced technique we call "concatenation". Yes it is joining string (in this case our code) together, yes I'm being sarcastic, but only because it is such a straightforward and simple idea. What we essentially do is to take the three files we have: utils.js, User.js, and script.js then create a script that we can run to join the text in these three files together.

# An example Bash script that may do so
cat utils.js >> main.js
cat User.js >> main.js
cat script.js >> main.js

<script src="./main.js"></script>

Whenever we make a change to any of our source code files, we run the concatenation script, which will update the main.js combined script which we include in our HTML to have the benefit of only making one HTTP request. Great, now we are not sacrificing Code maintainability for Transfer size/speed or vice versa!

Sidenote: This is one of the reason cited in some (but not all) style guides for JavaScript where semicolon ; is recommended at the end of every statement in JavaScript even though it is optional. The precise reason for why, I will leave it as an exercise for you to figure out.

Stage 3 - Nobody calls it iffy

Unfortunately, our app keeps getting more complicated. Currently we are exposing several variables onto the global scope: capitalize, User, user, and button, as we continue developing, we will end up exposing more and more to the global scope. Eventually, either as a result of including a library script, or as our own code get more complicated, we will come across unintended bugs. Let's say we add another file to our example:

// login.js
let user;
const loginButton = document.querySelector("#loginButton");
loginButton.addEventListener("click", async () => {
  const username = document.querySelector("#username").value;
  const password = document.querySelector("#password").value;

  const response = await fetch("http://example.com/login", {
    body: JSON.stringify({
      username: username,
      password: password
    }),
    method: "POST"
  });
  const result = await response.json();

  user = new User(result.username);
});

Next we include this new login.js file into our concatenation pipeline:

# An example Bash script that may do so
cat utils.js >> main.js
cat User.js >> main.js
cat script.js >> main.js
cat login.js >> main.js

Great, it is not that much work to add a new file to our pipeline and we added another file while needing only one HTTP requests from the browser, win-win! Except, we are expecting a bug here, what is it? Let's look at the concatenated file:

// main.js
function capitalize(str){
  return str.charAt(0).toUpperCase() + str.slice(1);
}

class User {
  constructor(name){
    this.name = capitalize(name);
  }
}

let user;
const button = document.createElement("button");
button.innerHTML = "Create user";
button.addEventListener("click", () => {
  user = new User("sam");
});

console.log(user.name) // Logs "Sam"

let user;
const button = document.querySelector("#loginButton");
button.addEventListener("click", async () => {
  const username = document.querySelector("#username").value;
  const password = document.querySelector("#password").value;

  const response = await fetch("http://example.com/login", {
    body: JSON.stringify({
      username: username,
      password: password
    }),
    method: "POST"
  });
  const result = await response.json();

  user = new User(result.username);
});

Noticed any issue here?

We have redeclarations of both user and button variables here which are not allowed in the same scope, running this script will immediately throw an error in the browser complaining about redeclaration. In the past when we only have var it is worse because one value to overwrite the other, but we should avoid using var in practice now that we have better options. We can work around this issue by renaming the second instance of user and button to something else, however that decreases Code maintainability because now we need to remember across however big our codebase is, all the possible variable names in case we get a collision. With "naming things" being one of two hardest problems in computer science, we should avoid it as much as possible. Is there some way we can still use the variable names we want for both instances?

Of course there is, that's why we are here in the first place. What we can do is to use something called an Immediately-Invoked Function Expression (IIFE), the name is a bit of a mouthful and the idea behind it touches on some esoteric feature of JavaScript.

Variable scoping

Variable scoping in JavaScript is infamous for its complexity, special cases, and easy to get wrong, it deserves its own article so I won't go into all the details here. Roughly speaking, there are two types of variable scoping in JavaScript (post-ES6): block scoping and function scoping. Block scoping means each block, represented by any combination of { /* ... */ }, creates its own scope and as such is permitted to redefine a higher level variable name. This includes things like if statements, for loops, or even manually created block by just surrounding statements with { /* ... */ }.

let a = 10;
if(someValue === true){
  let a = 20; // This is allowed
}

Function scoping on the other hand means scopes are created by the current closest function the variable is in, meaning if statements and for loops don't create their own scope.

function sum(a, b){
  var c = a + b;
}

sum(1, 2);
console.log(c); // Will throw ReferenceError

let and const are both block scope variable declarator while var is function scope, this is another reason why let and const are preferred over var these days.

This means that if we want to reuse the same variable names, we need to put them into their own scopes. While we are not using var anymore to declare variables, a function declaration in the form of function someAction(){} may still have block scope behavior, plus there are some other feature of a function that is useful for our purpose later. As such the first building block to IIFE is a function. As an example, let's use the login.js file:

// login.js
function(){
  let user;
  const loginButton = document.querySelector("#loginButton");
  loginButton.addEventListener("click", async () => {
    const username = document.querySelector("#username").value;
    const password = document.querySelector("#password").value;

    const response = await fetch("http://example.com/login", {
      body: JSON.stringify({
        username: username,
        password: password
      }),
      method: "POST"
    });
    const result = await response.json();

    user = new User(result.username);
  });
}

Here we'll put our code in an anonymous function for now, obviously this code isn't very useful at all since the function is just a declaration, the code within it does not run. The next step is to actually run the function, since we didn't assign a name to this anonymous function we can't call it by its name. However, it is entirely possible to call this function. In JavaScript, if you add () brackets to the end of anything, you are attempting to call it as a function and this includes anonymous functions. That being said, if you try to do it with just function(){}(), you will get a syntax error. That is because in this form, it is ambiguous what your intent with this code is. Our intent here is to call this anonymous function expression with a function call, which means we need to make this intent more explicit. We can do this by forcing the function declaration to be an expression by surrounding it in brackets, resulting in (function(){})(). This is the full IIFE pattern.

// login.js
(function(){
  let user;
  const loginButton = document.querySelector("#loginButton");
  loginButton.addEventListener("click", async () => {
    const username = document.querySelector("#username").value;
    const password = document.querySelector("#password").value;

    const response = await fetch("http://example.com/login", {
      body: JSON.stringify({
        username: username,
        password: password
      }),
      method: "POST"
    });
    const result = await response.json();

    user = new User(result.username);
  });
})();

We can do the same with all of our code and then concatenate them to get the following:

// main.js
(function(){
  function capitalize(str){
    return str.charAt(0).toUpperCase() + str.slice(1);
  }
})();

(function(){
  class User {
    constructor(name){
      this.name = capitalize(name);
    }
  }
})();

(function(){
  let user;
  const button = document.createElement("button");
  button.innerHTML = "Create user";
  button.addEventListener("click", () => {
    user = new User("sam");
  });

  console.log(user.name) // Logs "Sam"
})();

(function(){
  let user;
  const button = document.querySelector("#loginButton");
  button.addEventListener("click", async () => {
    const username = document.querySelector("#username").value;
    const password = document.querySelector("#password").value;

    const response = await fetch("http://example.com/login", {
      body: JSON.stringify({
        username: username,
        password: password
      }),
      method: "POST"
    });
    const result = await response.json();

    user = new User(result.username);
  });
})();

Great, now everything has its own scope and we no longer have variable name collision/redeclaration issue, but this is non-working code. If we have a closer look, yes we have put everything into their own scope, but we have put everything into their own scope: code from one file can no longer access variables from another file, including those that was intended to be accessible such as the capitalize function. To get around this, we can utilize the return value of a function.

const capitalize = (function(){
  function capitalize(str){
    return str.charAt(0).toUpperCase() + str.slice(1);
  }

  return capitalize;
})();

Since an IIFE is a function execution, we can capture its return value the same way we capture any other function return values. If all the IIFE are in the same top level scope as we have here, we won't need to do anything extra, however for some more obscure situation, there is another feature of IIFE that we can utilize which is using function parameters.

const capitalize = (function(){
  function capitalize(str){
    return str.charAt(0).toUpperCase() + str.slice(1);
  }

  return capitalize;
})();

// By using this function parameter, 
// we effectively renamed `capitalize` to `cap` and scoped it
(function(cap){
  class User {
    constructor(name){
      this.name = cap(name);
    }
  }
})(capitalize); // We pass in the in scope `capitalize` here

Putting what we have together:

// utils.js
const capitalize = (function(){
  function capitalize(str){
    return str.charAt(0).toUpperCase() + str.slice(1);
  }

  return capitalize;
})();

// User.js
const User = (function(){
  class User {
    constructor(name){
      this.name = capitalize(name);
    }
  }

  return User;
})();

(function(){
  let user;
  const button = document.createElement("button");
  button.innerHTML = "Create user";
  button.addEventListener("click", () => {
    user = new User("sam");
  });

  console.log(user.name) // Logs "Sam"
})();

(function(){
  let user;
  const button = document.querySelector("#loginButton");
  button.addEventListener("click", async () => {
    const username = document.querySelector("#username").value;
    const password = document.querySelector("#password").value;

    const response = await fetch("http://example.com/login", {
      body: JSON.stringify({
        username: username,
        password: password
      }),
      method: "POST"
    });
    const result = await response.json();

    user = new User(result.username);
  });
})();

Now only the variables we want to expose to global, capitalize and User, are exposed. Everything else will have their own scope, ie. user and button.

Stage 4 - Abstraction not distraction

One downside to IIFE is we have to use IIFE in all of our source code files, it adds noise and boilerplate to our code where we don't want it. We want to just write our code and then define what should be exposed globally and what shouldn't, in a simpler and cleaner way. We can come up with a convention for defining what values will be exported and even imported, then using automated tools, parse these convention and create the IIFE for us. What we are describing here is essentially a module standard and a bundler respectively.

Module standard

I'll only mention two of the module standards that have any traction in JavaScript and only describe one, these two standards are CommonJS and Acynchronous Module Definiton (AMD). Both of them attempts to achieve similar things with notable design and technical differences, while CommonJS is adopted by Node.js as its (and later on one of its) module system, the most notable implementation of AMD is RequireJS. AMD in general however has fallen out of use in the modern module landscape, as such I'm going to skip over AMD and just talk about CommonJS here.

CommonJS was originally designed with server side JavaScript in mind, before Node.js was even a thing, however I'm going to focus more on the browser side usage partly because server side modules are incredibly complex with numerous special cases. What we need to know is that CommonJS just happens to solve our problem of needing a way to define import and exports to our files so that we can make them into sensible IIFE later. Let's write out how we may define utils.js with CommonJS syntax:

// utils.js
function capitalize(str){
  return str.charAt(0).toUpperCase() + str.slice(1);
}

module.exports = capitalize;

and how we can use it in User.js:

// User.js
const capitalize = require("./utils.js");

class User {
  constructor(name){
    this.name = capitalize(name);
  }
}

module.exports = User;

The two key elements of CommonJS are require() and module.exports (sometimes this is just exports), which define imports and exports respectively. module.exports is essentially a variable you can assign any JavaScript value to, here in utils.js we are assigning it to the capitalize function but if needed to export more than one values, we can set module.exports to be an object instead: module.exports = { capitalize: capitalize, anotherValue: "Hello World!" }. require acts as a function that we can call which will import and evaluate to the module.exports value of the JavaScript file that the path argument is pointing to. In other words, we export whatever values we want by assigning them to module.exports then import that same values by importing them with require(). With this we can automate the process to have a bundler read these import and export statements and build a dependency graph and the relevant IIFE structure as needed. I'll expand a bit more on bundlers in the next section.

There are a couple caveat/special behavior of CommonJS that are worth pointing out. First, it is possible for multiple files to require() the same file and the evaluated value are shared across these files:

// first.js
const obj = {
  child1: {
    name: "Alex"
  },
  child2: {
    name: "Sam"
  }
};

module.exports = obj;

// second.js
const obj = require("./first.js");

// Here we change the value of `child1`
obj.child1.name = "Nick";

// third.js
const obj = require("./first.js");
require("./second.js");

console.log(obj.child1.name); // This now logs `"Nick"` instead of `"Alex"`.

Next, as you may have noticed in the example above, in third.js I used require("./second.js") while second.js does not have a module.exports nor am I assigning the require() call to a variable. This is a side effect import, essentially I'm importing the module only to have its code executed and not using its export value, if it even had any. All require() calls executes the entirety of the imported file synchronously before it moves on, which will include side effects, regardless of whether I'm doing a side effect import or regular import. However if we combine this with the first point above, remember the same require() path resolves to the same object? This only works if that code file is evaluated and executed once, which means the side effect execution only run once regardless of how many require() uses the same file path.

This is particularly important if your module has side effect. In general side effects can be tricky to deal with especially as your application gets more complex, I would strongly advise against having any side effects in CommonJS modules (obviously other than the root file) unless you really know what you are doing.

Bundler

We'll leave CommonJS as is for now and start looking at bundlers: how will we be able to transform our CommonJS module code into nicely scoped IIFEs in a single file? While concatenation is in a way a bundler too, we have already seen that we need to scope things with IIFE ourselves which we didn't want to do, at the same time I have implied that CommonJS was designed for server side JavaScript, which by the time when we started to seriously look at browser side JavaScript bundling, Node.js is already well established and gaining traction. What people soon realise is that both Node.js and browser side JavaScript are JavaScript (ok, that is obvious to everyone in the first place), which means that if we have some code that does not use specific Node.js or browser API and just process some logic, they should be usable in both environments, in other word it should be possible to share code between Node.js and the browser.

The problem is that Node.js uses CommonJS and your Node.js code is likely to be written in CommonJS but the browser doesn't run CommonJS code, it does not know what require() or module.exports are. In comes Browserify, which does exactly what it says on the tin, it browser-ify your Node.js code so that it can run in the browser. At the same time Browserify also achieve the goal of bundling for us so that our collection of CommonJS code can be bundled into just one file. Browserify essentially creates a form of IIFE encapsualtion for us while preserving the general behavior of CommonJS so that code used in Node.js and the browser don't deviate too much in behavior. I won't go into details about how Browserify worksbut you can read more here.

Soon after Browserify gained traction (or shortly before depending on how you measure), people started to explore JavaScript bundling and the general JavaScript DevOps seriously. We started to see an explosion of tools and ideas spanning not just JavaScript but also HTML and CSS too. We were getting things like linter, test runner, code transformer, task runner, languages that compile to HTML/CSS/JavaScript, frontend frameworks, etc. A notable entrant of this period is Webpack which in a way seeks to consolidate many of the functionalities above into a single tool, creating a comprehensive bundling solution that can work with all elements of the web: HTML, CSS, JavaScript, or even image assets. Many of the ideas we expect of a modern build tool/bundler were either pioneered by Webpack or popularized by it. Even today, Webpack, despite what some may say, is still a good and valid tool for JavaScript bundling.

The whole ecosystem around JavaScript tooling was and still is very rich and diverse and maybe also saturated, which means I won't be able to detail all these came before or all that exist now. Each of them bring interesting ideas and solves either different problems or the same problem in different ways. We have ESLint, TypeScript, Parcel, Babel, SWC, and so many more. Another tool I want to highlight is Rollup. However to properly understand Rollup, we need to first understand ES Modules.

Stage 5 - Rolling and Shaking

ES Modules or ESM for short, is a bit of a colloquial name for the native module specification that was introduced into JavaScript around 2015. While there are already module definition system such as CommonJS that people are already using, ESM is a module system defined as part of the JavaScript standard itself rather than by an external group or informally through some code implementation. I won't go into the JavaScript standardization process in details here but just to highlight, the JavaScript standardization process involves all the major browser vendors, server side JavaScript groups, and many other people that through set processes came to the standard JavaScript language that we use today: long gone are the days where JavaScript was put together by one person in one week, that's literally ancient history in the overall timeline of JavaScript.

Before we look deeper into ESM, it is reasonably to ask, why ESM? Why another module system and not use CommonJS?

We have seen multiple times before, each solution/innovation does not exist in a vacuum, nor was it decided arbitrarily, there is always a problem to solve and a solution to it. In the case of ESM, a bit counter intuitively, the problem to solve is not bundling but modules itself. CommonJS is limited in that it needs to work with JavaScript as it exist while ESM has the benefit of creating new semantics in JavaScript by being a new standard, one of the main difference that this leads to is that CommonJS by requirement needs to load, parse, and execute its modules synchronously while ESM can do these asynchronously. For an understanding of acynchronicity in JavaScript do have a read on it here.

CommonJS being a synchronous module loading system means that if a particular file loads three different modules a.js, b.js, and c.js in that order in the code:

const a = require("./a.js");
const b = require("./b.js");
const c = require("./c.js");

// Do something with `a`, `b`, and `c`

CommonJS will first need to load a.js from disk or from network for however long it takes, then parse it, then execute the code within a.js (remember side effects?) which may include loading additional modules. To load all of the modules recursively, CommonJS needs to execute all of the code synchronously, this is because without executing the code itself, we don't know what module is actually required to be loaded, require is just another JavaScript function call, you can call it anywhere such as in a conditional if block, we don't know if a module should be loaded or not without evaluating the if block. Only once a.js and all its child dependencies are loaded and executed then we will start loading b.js and so on.

From what we understand about loading from disk or network, we know that the JavaScript runtime does not need to wait for these asynchronous actions to finish before it starts doing something else. We also know that we can load the code for all the modules we need in parallel as long as we know in advance what that list of modules are. ESM is designed to help with that by enabling asynchronous loading and resolution of modules. Let's look at some basic import syntax of ESM before we talk about how it works:

import a from "./a.js";
import b from "./b.js";
import c from "./c.js";

// Do something with `a`, `b`, and `c`

You may be able to see some echoes of CommonJS in the ESM syntax above which would make sense since they are trying to solve very similar problems, the syntax structure is different but the main idea is the same: we import/require a from the file "./a.js". One of the main difference here from CommonJS is that import statements can only be used in the top level, ie. it cannot be used within an if statement or any other blocks, you should be able to guess why if you think about what problems we are trying to solve with ESM here as compared to CommonJS.

In the case of ESM, rather than loading, parsing, then executing a.js before doing the same with b.js then c.js, we split the operation into three steps:

Construction
Instantiation
Evaluation

The construction steps starts from the entry file, parse it and extracts all the import statements so that it knows which files to fetch next. It does this recursively until all modules are loaded and parsed. No execution or memory allocation is done at this stage and all of these can be parallelized.

Once construction is completed, we will have a full graph of all the modules that make up our programme and no other asynchronous loading is required anymore. Next the instantiation stage allocates memory that link imports with exports (we'll look at the syntax later) of each modules together.

Finally the evaluation stage starts executing the code and since all the modules are already loaded and linked in memory, there is no waiting around for any dependency to load and execute.

By using this system, we allow the asynchronous parts of the module system to be executed asynchronously and when it comes time to actually run the code, everything required is already there and ready to go. Hopefully you can see where it speeds up the overall module loading as compared to CommonJS.

Before diving even deeper, let's have a brief look at the ESM syntax a bit more, I won't go into full details here because this is not a ESM tutorial, do checkout relevant tutorials if you need more detailed explanation of the ESM syntax:

// main.js
import a from "./a.js";
import { myValue } from "./b.js";

// a.js
const myValue = 20;
export default myValue;

// b.js
export const myValue = 42;

In ESM, the export statement like the import statement can only be used in the top level, because remember we need to be able to build the module tree fully before we execute the code, which means there cannot be any condition guarding whether a variable is exported or not. Another thing to note about export that can be seen above it that there are two types of export: a default export (export default) and a named export. Rather than assigning a value to an export variable like module.exports, the export keyword is best understood as representing an operation rather than variable assignment, it exports either a default (unnamed) variable or a named variable. You can use both a default export and named exports at the same time. When it comes time to import, you will also choose either to import the default variable or named variables: import a from "./a.js" imports the default export from a.js into the variable a while import { b } from "./b.js" import the named export b from "./b.js". You can rename the import as follow if you need a different name for the imported named export `import { b as myVar } from "./b.js".

Alright, now we see that in terms of module system, ESM better leverages the asynchronous nature of JavaScript better than CommonJS can, but you may be thinking: wait a minute, all these talk about loading a module etc doesn't really matter to bundling does it? All of the code are already bundled into a single file, there is no asynchronous disk or network request to get those code, it's all already there!

That is entirely correct, as briefly pointed to before, ESM is designed to solve the module problem not the bundling problem. In a way, by using bundlers, we are bypassing the runtime benefit of ESM since bundlers that consume ESM often outputs IIFE which does not use ESM at all. Bundlers will just use ESM as a standard syntax to define the imports and exports we are actually interested in to build the IIFE. As to whether ESM should be used directly without a bundler, I would say yes for server side JavaScript (since it is rare and not very beneficial to bundle server side code) and probably not yet for browser side code (which I will defer again to this piece from Rolldown).

Finally, since we are likely to still bundle even code written with ESM, we can get back to Rollup. Rollup is by design a bundler for JavaScript code written with ESM, there is no built in support for CommonJS modules but that functionality can be added with a plugin, meaning you can use both ESM and CommonJS modules to bundle into a single bundle. The is usually done to bridge the gap between code previously written in CommonJS and ESM, you should avoid using both module system in a single project at the same time. Other than supporting the standard syntax for modules, another major reason Rollup default to supporting ESM is because of "tree-shaking". Consider the following code:

js // main.js import { myVar } from "./a.js";

js // a.js export const myVar = 20; export const anotherVar = 10;

We can see in the entry file main.js it imports from a.js however it only imports the named variable myVar and not the variable anotherVar. This means that in the final bundled code, a.js can only contain export const myVar = 20; and we can leave out the definition and export of anotherVar because we know this variable and export is never used. This is what is called dead-code elimination, "dead-code" being code that is never used. To be "dead-code", any lines of code needs to fulfill two conditions: one, it must not define any variables that is used or referenced anywhere else in the code; two, it must not have any side effects. If and only if we can determine for certainty the two conditions above can we mark a piece of code as dead-code and proceed to eliminate it, since it is not being used. Tree-shaking as a term is somewhat similar to dead-code elimination in that the final endpoint is the same, to have code that only includes the actual useful part and not any parts with no effects, usually you will see "tree-shaking" as a term more commonly used by JavaScript bundlers instead of "dead-code elimination".

js // main.js const { myVar } = require("./a.js");

js // a.js module.exports.myVar = 20; module.exports.anotherVar = 10;

With the equivalent CommonJS example above, we can see that it should be possible (with a little more extra effort) to determine statically (ie. without actually running the code) that anotherVar is never used and as such can be eliminated, which means that it is technically possible to perform tree-shaking with CommonJS. However, since require() can be called anywhere in the code, including conditionally, there are situations where it cannot be statically confirmed that any piece of code fulfills both of the requirements for dead-code elimination. This, along with other features of CommonJS, makes tree-shaking CommonJS modules more complicated and less effective than tree-shaking ESM which by default provides static guarantees on the relationship between import and exports.

Stage 6 - Lost and confused

At this point, we have caught up to where things are at the time of writing, while there are many detailed aspects of the different things touched on above such as ESM dynamic imports or Rollup's relationship with Rolldown, I'll use the excuse of leaving all these as exercise for you to find out yourselves, that is part of the fun anyway. The goal and takeaway of this article is to present a picture of the history and context behind the most prominent tooling in JavaScript. To make an informed decision about the what and why of any tooling to use, I believe understanding this context is very important. If you would like to just have a suggestion or prescription from me on bundling tooling to use, you are likely in the wrong place, but I hope all these context above equip you with the understanding and critical approach to take when deciding what tools best fits your requirements.

The community around JavaScript is pretty unique as compared to the community in some other programming languages, while from the outside it looks like JavaScript is continually chasing trends or fads when it comes to developer tools, that is just the consequence of having a large community of people actively identifying & solving solutions and sharing openly with everyone the solutions they found. At times (perhaps even now), you may feel lost and confused with all the possibilities out there, to possibly help I can give two pieces of advice: 1. just because what you are using is not new and shiny does not mean you shouldn't use it, if it works, it works; 2. just because you have something that already works, don't let that prevent you from exploring new things with a critical mindset.

Even at this point, there is no doubt things will continue to develop, just as in linguistic, a language that doesn't change is a dead language. Part of the quality that makes a good programmer is the ability to continuously learn, adapting to change, or even create those changes yourself. That last point is what everyone who developed everything in this article above did, whether it is IIFE, Browserify, ESM, Rollup, etc, if you see a need for change or new ideas, try it out and share it with the world.

Reference

Designing an addon library system for p5.js 2.0

Kenneth Lim — Fri, 21 Feb 2025 11:27:31 +0000

TLDR: If you have an existing addon library, test if it already works with p5.js 2.0, you may not need to make any changes. If there is something that isn't working, go through the flow below to troubleshoot and find a solution:

p5.registerMethod not found error message

Instead of using p5.registerMethod, use the new addon library syntax for lifecycle hooks.

// Before
p5.prototype.setDefaultBackground = function(){
  this.background("#ed225d");
};
p5.prototype.registerMethod("pre", p5.prototype.setDefaultBackground);

///////////////////////////////

// After
const myAddon = function(p5, fn, lifecycles){
  lifecycles.presetup = function(){
    this.background("#ed225d");
  };
};
p5.registerAddon(myAddon);

p5.registerPreloadMethod, _incrementPreload, _decrementPreload not found error message
- Consider switching to returning promise from your asynchronous function, read here for possible transition methods.
Anything else not working as expected
- File a bug report with p5.js

p5.js 1.x will still receive bug fixes for at least a year while p5.js 2.0 is being rolled out. Please read the final section of this article for some transition strategy your library might want to take and do contact the p5.js team if you need help or clarifications on your library's transitions.

Part 1

Addons, plugins, extensions, or any other name they may be called, these are just different names to a very common concept in the Javascript ecosystem. From the years where all our project starts with jQuery, to now where modern frameworks, libraries, or tools such as Vue, Three.js, and ESLint have some level of support for addons. Even projects that don't explicitly have an interface for addons will often see addons written for it to expand on its functionalities. That last point, expanding functionalities, is exactly the idea we are going to explore here.

Unlike a compiled language such as C, C++, Rust, or Java, Javascript is not served to the end user as binary, it is by design served as text. Also unlike other non-compiled language, Javascript is often served frequently over the internet, instead of being packaged and downloaded by the user once to be run multiple time, such as what you might do with any other applications on your computer. This puts Javascript in a position where it needs to do as much as it needed with as little code as necessary: the more code it has, the slower Javascript gets downloaded, the slower the experience it is for the end user.

One method to mitigate this need for small download size is minification and compression. We won't explore that here. The other method that we alluded to is to be as minimal as necessary, to do as much as is necessary and not more. However, it is often a challenge to know what is "necessary". Each library will need to determine that for themselves and we will explore this further in the future, but for now let's take this as a tautology and look at what it means in practice. Once a library decided the necessary functionalities it needs, it gets implemented and people can use it, great! However, there could very well be use cases where people needed a bit more functionalities that are not included, and the library doesn't want to include for various reasons. We don't want our libraries to be able to do everything possible when we are only using a small subset of its capabilities, otherwise our libraries will be excessively bloated. In these kind of cases, it would make sense to create an addon/plugin/extension library that expands on the functionalities of the library.

Depending on the design and purpose of the library, there are several different techniques when it comes to expanding functionalities, none are better than others necessarily, and some libraries are only meant to be expanded one way while others were never meant to be expanded at all. This is to say you should see the following as different options or approaches rather than any kind of value judgement on the techniques themselves.

Note: I will be using the terms "addon", "plugin", and "extension" interchangeably here depending on context and which term individual library uses. They all mean the same thing in our context.

Direct extension

Javascript for better or for worse (depending on who you ask) is a pretty flexible language, especially when it comes to its type system. Javascript is a relatively rare object oriented language that allows the modification of a class definition at runtime, ie. it is able to modify the template in which each object is created from in the middle of execution, this is not something that stricter languages will ever allow. Technically, in Javascript we are not modifying the class definition, for the class syntax that you may have come across in Javascript is mostly syntactic sugar (syntax abstraction) of the real underlying object oriented model that Javascript uses which is prototype based object oriented programming.

We will leave detailed explanation of prototype based object oriented programming for another time, for our purposes here, the key to remember is we can modify classes at any point in time. This means that even if an interface provided by the library has a certain definition, it can still be modified by code that comes later. In practice, we will do this with prototypes and it looks something like the following:

// This class is defined by the library we want to extend
class Rectangle {
  constructor(width, height){
    this.width = width;
    this.height = height;
  }

  area(){
    return this.width * this.height;
  }
}

// In the user code
const r = new Rectangle(10, 20);
console.log(r.area()); // Will print `200`
console.log(r.circumference()); // Will error since there is no `circumference` method

// The following is our "addon" code to extend the `Rectangle` class.
// We will add a `circumference` function to our `Rectangle` class.
Rectangle.prototype.circumference = function(){
  return this.width * 2 + this.height * 2;
};

// In the user code
const r = new Rectangle(10, 20);
console.log(r.area()); // Will print `200`
console.log(r.circumference()); // Will print `60` and no error

As you can see in the second block, by attaching a new function property to Rectangle.prototype, we have modified the original Rectangle class to now have a circumference method that we can call. We can attach whatever we need onto the prototype and it will all behave as if it is defined within the class itself, which our objects will then also inherit from.

This is a relatively common way of extending a class and, with libraries that provide classes for users to create object out of, is a common way of extending functionalities. However, attaching functionalities to a prototype directly using the .prototype syntax is relatively uncommon especially in libraries meant to be extended. There are a few reasons for this, one is because library authors often prefer one of the other methods mentioned below for friendlier syntax (not everyone likes prototypes or understand them) or more granular control over extension capabilities. Another reason you don't see this as much is because in libraries that does utilize prototype based extension, they may choose to obscure that fact. A notable example is jQuery.

jQuery's plugin syntax looks like the following (taken from jQuery's learning page):

$.fn.greenify = function() {
    this.css( "color", "green" );
};

$( "a" ).greenify(); // Makes all the links green.

We can see that a function is assigned to $.fn called greenify but what is this $.fn? If you are familiar with jQuery, you will know that $ is the conventional short-hand variable for the global jQuery object; fn is a property within that jQuery object, you might have guessed by now that fn is actually prototype! If you open up the web console on the jQuery page linked above and type in the following:

$.fn === $.prototype

You will see that it returns true. Plugins for jQuery are essentially attaching directly to the jQuery prototype so that all future jQuery object inherits that new functionality that the plugin has attached to the underlying class.

Extension through dedicated API

In a way, we can think of the $.fn syntax that jQuery provides as providing an extension through dedicated API, albeit a very shallow one. However, there are use cases where additional utility and functionality with the plugin system is required. This can be because the library does not just provides a class for object extension, the library provides multiple ways of creating an object, and/or the library has more functionalities that can be extended other than simply creating objects such as if it has a runtime (common for frontend frameworks). In these cases, a more prominent API will often be presented to extend the functionalities of the library, usually in the form of a function that accept some argument that defines the behavior of the plugin.

Let's have a look at what this could mean through an example. Day.js is a library designed to make handling time related data a lot easier in Javascript. It is able to parse timestamp, manipulate time data (eg. calculate the date 10 weeks from now), and format timestamp so that you can display date and time in whatever way you want with as much detail as you need. Day.js provides a plugin system that can extend the possible things you can do with a timestamp such as getting UTC time, do timezone conversions, and many more. To extend Day.js with a plugin, you will do something like the following:

// `dayjs` is the global Day.js object that we use to create timestamp object
// We first use ES6 modules to import our UTC plugin
import utc from "dayjs/plugin/utc";

// Next we extend `dayjs`
dayjs.extend(utc);

Now the global dayjs object has additional functionalities provided by the utc plugin. What is actually happening though? What is the utc plugin and what happens with dayjs.extend() function? If we look at the Day.js plugin documentation page, we will have a good idea on what is potentially happening.

export default (option, dayjsClass, dayjsFactory) => {
  // extend dayjs()
  // e.g. add dayjs().isSameOrBefore()
  dayjsClass.prototype.isSameOrBefore = function(arguments) {}

  // extend dayjs
  // e.g. add dayjs.utc()
  dayjsFactory.utc = arguments => {}

  // overriding existing API
  // e.g. extend dayjs().format()
  const oldFormat = dayjsClass.prototype.format
  dayjsClass.prototype.format = function(arguments) {
    // original format result
    const result = oldFormat.bind(this)(arguments)
    // return modified result
  }
}

A Day.js plugin is always defined as a function, this means that dayjs.extend() is actually expecting an argument with a function as a callback into the .extend() method that is then passed its own relevant arguments. These arguments are option (which is the user defined option that is also passed to .extend() if needed), dayjsClass, and dayjsFactory. In the function body, you will notice that dayjsClass is extended through the prototype syntax we have seen above so that is not new. The final argument dayjsFactory is something that is more Day.js specific, without going into all the details, attaching additional functions to dayjsFactory extends the number of ways Day.js objects can be created.

As you can see in this example, Day.js using a dedicated API for extension does not mean it won't be extending prototypes, but rather it has additional capabilities that is not strictly attached to the class itself. For more complex projects, such as Vue.js or more commonly Node.js based frameworks, using a dedicated API is often the best way to extend the functionality of a library.

Independent interface, compatible data structure

This final method of extension is not really an extension at all. The philosophy here is that instead of having users modify the provided interface of a class or other library internals, the library provides and accepts simple or well-defined data/data structures in its public interface, eliminating much of the need to extend it directly.

What this means is that instead of implementing additional functionality as a dependency onto the existing library, the additional functionality is implemented independently and will either accept outputs from the library in a well-defined format or be able to pass inputs to the library in a well-defined format. In a way, this is the most common method in Javascript because every library that does not provide explicit interface or guidance on extension is assumed to work this way, the only thing the library maintainer need to be aware of is to have said well-defined format for input/output.

<script>
// We can use Chart.js with Vue without needing to extend
// Vue with Chart.js
import { useTemplateRef, onMounted } from "vue";

const chartEl = useTemplateRef("myChart");

onMounted(() => {
  const ctx = chartEl.getContext("2d");

  new Chart(ctx, {
    type: 'bar',
    data: {
      labels: ['Red', 'Blue', 'Yellow', 'Green', 'Purple', 'Orange'],
      datasets: [{
        label: '# of Votes',
        data: [12, 19, 3, 5, 2, 3],
        borderWidth: 1
      }]
    },
    options: {
      scales: {
        y: {
          beginAtZero: true
        }
      }
    }
  });
})
</script>

<template>
  <div id="app">
    <canvas ref="myChart" />
  </div>
</template>

The main benefit of this is that plugin authors will no longer be plugin authors, they will be writing libraries of their own instead. Their end users will also no longer need the upstream dependency if they don't need it, fulfilling even more of the original goal of limiting the transfer over internet of unused code and functionalities. The resulting library have more flexibility and compatibility as well. Gone are the days where every other library were written as a jQuery plugin, these days we can just use whichever library solves our problem without needing to include jQuery if we don't need it anymore.

A final word on extension methods

These above are not necessarily extensive when it comes to all possible ways to extend a library and definitely not extensive when it comes to possible syntax (since that is possibly infinite). I believe most if not all extension variations can fit in at least one of the above category but if you see a variation that somehow don't fit into any of the above category, do let me know, I would love to have a closer look at it.

Now that we have an overview of these different techniques, let's have a look at them in the context of p5.js addon libraries.

Part 2

p5.js as a library has many aspects that a user may want to extend. These can be providing additional drawing functions, communicating with other services, or provide glue code with other libraries, amongst infinite possibilities. The two major direct expansions are providing new functions that can be called in global mode & instance mode semantically, and running hooks (ie. custom code) at certain point in the runtime such as before/after setup() runs and before/after every draw() call. Since we have looked at possible extension strategies above, let's see how they apply in the context of p5.js, starting with p5.js 1.x then we'll look at how p5.js 2.0 changes some aspects of this and why.

p5.js 1.x uses a combination of all three strategies above when it comes to how addons can be authored. Starting from the least complex to explain, the last strategy (Independent interface, compatible data structure) as we know, relies on passing data directly out of or into p5.js. As long as your library can generate parameters accepted by p5.js functions, it will work out of the box. Some examples include passing CSS color string as argument into any function that accepts colors, and using random noise generated by the noise() function in your own addon for smooth randomness.

Next, if you wish for your addon functions to be available globally in global mode, you will need to attach it to the p5.prototype object as per the first method we looked at above.

p5.prototype.polygon = function(points){
  // Draw polygon
};

Most addon libraries that want to extend p5.js will likely work this way. However, in some circumstances, addon libraries may need to also have a bit more control over p5.js, specifically the p5.js runtime. For example, instead of passively providing functions that the user can call, an addon library might want to instead reset the background color at the start of every frame automatically or set up some initial values before setup is called. In these kind of cases, p5.js provides lifecycle hooks that an addon library can hook a function onto. What lifecycle hooks means is that they are specially registered functions that will be run at specific points in the runtime (ie. from initializing the p5 object to running setup() to removing the sketch altogether, if remove() was ever called). To register these special functions onto the relevant hook, we need to use the second extension method, Extension through dedicated API, here.

// This does not need to be a function attached to
// the p5 prototype if it does not require `this`
// and it does not need to be available globally.
p5.prototype.setDefaultBackground = function(){
  // Set background to be p5 pink by default
  this.background("#ed225d");
};
p5.prototype.registerMethod("pre", p5.prototype.setDefaultBackground);

There are a number of hooks available but we will not elaborate on them here as we are interested in how it works in p5.js 2.0 instead. In p5.js 2.0, the requirements and principles stay the same: we still need to be able to attach functions that can be used in global mode and instance mode and we still need a way for library authors to hook into specific lifecycle events. To simplify the addon library authoring workflow, p5.js 2.0 now has a unified dedicated extension API that support both use cases (we are leaving "Independent interface, compatible data structure" out here since the whole point is to not have direct extension in that case). p5.js 2.0 now provides a new static function p5.registerAddon() that takes in one argument which will be the definition of your addon library. This argument is in the form of a function:

const myAddon = function(p5, fn, lifecycles){
};

p5.registerAddon(myAddon);

As you can see above the function that comprise of your addon library has three parameters p5, fn, and lifecycles, let's go through each one at a time.

`p5`

This is the global p5 object, ie. the p5 constructor. You can use it to attach or access static properties/methods if you need to. For example if you want to manually create a p5.Vector in your library:

const myAddon = function(p5, fn, lifecycles){
  const privateVec = new p5.Vector(0, 1, 0);
};

`fn`

If you remember fn being used by jQuery, you may have guessed what fn is. If you guessed fn is just an alias for p5.prototype, you will be right! We certainly can access p5.prototype through the p5 parameter in the first position, however fn provides a slight layer of abstraction to the underlying prototype system of Javascript for the library author, it is still functionally the same though.

const myAddon = function(p5, fn, lifecycles){
  fn.polygon = function(points){
    // Draw polygon
  };
};

`lifecycles`

This is likely the most significant difference and a departure from p5.js 1.x. Instead of having an API to individually register lifecycle hooks, the lifecycles parameter here by default points to an empty object {}. We register lifecycle hooks by assigning relevant keys into the lifecycle hook.

const myAddon = function(p5, fn, lifecycles){
  lifecycles.postsetup = function(){
    // Do some actions after setup finishes.
  };  

  lifecycles.predraw = function(){
    // Set background to be p5 pink by default
    // at the start of each draw call.
    this.background("#ed225d");
  }
};

The available hooks are:

presetup - Runs once before setup() runs
postsetup - Runs once after setup() runs
predraw - Runs once before every draw() runs
postdraw - Runs once after every draw() runs
remove - Runs once when remove() is called

All functions attached to the lifecycles object have access to the current p5 sketch instance through this (provided they are not defined with arrow functions), and they are all called with the await keyword so they can be async function as well.

Put altogether, we have something like the following:

// This addon sets two default background color for the sketch
// that the user can toggle between by calling the 
// `toggleBackgroundColor` function in global or instance mode.
const myAddon = function(p5, fn, lifecycles){
  const color1 = new p5.Color("floralwhite");
  const color2 = new p5.Color("midnightblue");
  let bgColor = color1;

  fn.toggleBackgroundColor = function(){
    if(bgColor === color1){
      bgColor = color2;
    }else{
      bgColor = color1;
    }
  };

  lifecycles.predraw = function(){
    this.background(bgColor);
  };
};

This syntax should have enough flexibility to cover all uses cases for extension and even if not, there will be room for easy expansion without causing problems with future compatibilities.

Part 3

This part is for those who already has an existing addon library written for p5.js 1.x and want to make it compatible with p5.js 2.0. If you are just here to learn about addon libraries and don't already have a p5.js 1.x addon library, you can skip this section.

The good news is, if your library don't extend p5.js directly it will most likely just work as we have seen above. If your library extends p5.js through attaching to p5.prototype, it will likely just work, because as we've seen, the new addon syntax also provides fn as an alias to p5.prototype and we attach methods there still.

The bad news is, if your library uses the 1.x lifecycle hook register method p5.prototype.registerMethod, you will need to do some conversion. If your addon uses p5.registerPreloadMethod, _incrementPreload, or _decrementPreload, you will also need to update your addon, for this case, please refer to the article on asynchronous p5.js 2.0.

To convert lifecycle hooks, do the following:

Take note of the relevant lifecycle hooks you are using and how they map onto the new names:
- init -> You no longer need a lifecycle hook for this, just put the relevant code in the addon function body or the presetup hook.
- beforePreload -> presetup
- afterPreload -> presetup/postsetup
- beforeSetup -> presetup
- afterSetup -> postsetup
- pre -> predraw
- post -> postdraw
- remove -> remove

Using the new addon syntax, attach the relevant function as members of the lifecycles parameter.

const myAddon = function(p5, fn, lifecycles){
  lifecycles.presetup = function(){
    // Your old `beforeSetup` code
  };
};

Note that since the lifecycle hook functions will have this resolved to the p5 sketch instance already, you no longer need to attach functions to the prototype to access this like we were doing before in the setDefaultBackground example above.

If you are using the new addon syntax, the final thing you will need to do is to call p5.registerAddon() with your defined addon function.

p5.registerAddon(myAddon);

Transition

You may have noted that if your addon library converts to use the new addon library syntax, it will no longer work with p5.js 1.x, since p5.js 1.x does not provide the p5.registerAddon() static function, amongst other reasons. There are a few possible options you can take:

Have two different versions of your library, one with p5.js 1.x compatibility and another with p5.js 2.0 compatibility.
Release a new version of your library with p5.js 2.0 compatibility. If your user needs p5.js 1.x compatibility, they can use the last released version of your library that was still compatible.

Support both syntax at the same time. This will be more complicated to implement and may not be possible in all cases. The general idea will be something like the following:

// We will attach directly to the `p5` prototype
p5.prototype.setDefaultBackground = function(){
  // Do some action
};

// For lifecycles, we check if `p5.registerAddon` is present
// or not to determine which version to use.
if(p5.registerAddon){
  // We are working with p5.js 2.0
  const addon = function(p5, fn, lifecycles){
    lifecycles.presetup = function() { /* ... */ }
  };

  p5.registerAddon(addon);
}else{
  // We are working with p5.js 1.x
  p5.prototype.registerMethod("beforeSetup", function() { /* ... */ });
}

If you are unsure how to do the conversion, especially in the specific context of your addon library, please reach out to the p5.js team by opening an issue on the p5.js library repo, directly via email, or other official channel. The p5.js team is currently, at the time of writing, going through the full list of community contributed libraries to individually assess the compatibility of the libraries and identify any action needed. You may be contacted on your library's repo about any action needed. However, feel free to attempt some tests yourself with p5.js 2.0, it may already work without needing any changes to your code.

Asynchronous p5.js 2.0

Kenneth Lim — Mon, 10 Feb 2025 17:02:04 +0000

TLDR: To convert your code from 1.x with the preload function to 2.0 with async setup function, do the following:

Move all code that is currently in the preload function into the setup function at the top.

let img, data;
function preload(){
  // img = loadImage("./my-image.png");
  // data = loadJSON("./my-data.json");
}

function setup(){
  img = loadImage("./my-image.png");
  data = loadJSON("./my-data.json");
  createCanvas(400, 400);
}

Add the async keyword before function setup.

let img, data;

async function setup(){
  img = loadImage("./my-image.png");
  data = loadJSON("./my-data.json");
  createCanvas(400, 400);
}

Add the await keyword before the load functions copied from the preload function.

let img, data;

async function setup(){
  img = await loadImage("./my-image.png");
  data = await loadJSON("./my-data.json");
  createCanvas(400, 400);
}

When first learning p5.js, you will likely have learnt that the setup function is run once at the start to setup initial variable values, create the canvas and other one time setup things, after that the draw function will run over and over again in a loop.
However, if you have ever tried loading something, say an image with p5.js before, you might have done something like this:

let img;
function preload(){
  img = loadImage("./my-image.png");
}

function setup(){
  createCanvas(400, 400);
  image(img, 0, 0, width, height);
}

What is actually going on here and why do we need the preload function, can’t we just load in setup instead? Isn’t setup meant to be where we initialize variables? What we are doing here in the preload function looks to be just setting up the variable value so why can’t we do it in setup? If you already know the answer (ie. Javascript asynchronicity) you can probably skip the next section, for those who don’t let’s look at Javascript asynchronicity.

Part 1

Javascript as a language is by design single threaded, this means that Javascript can only ever run one task at a time, this can be string concatenation, math operation, assigning variables, updating the DOM, etc. This also means that operations such as loading a file over the internet must also run on this single threaded environment. However, you will have noticed that when browsing the web, you are not constantly waiting for files to be loaded one at a time over and over again, this will make the web unusable! In Javascript as well, if we wait for every instance of a file load to complete one after another before executing the following lines of code, there will be long periods of non-activity where we, and Javascript itself, is waiting around doing basically nothing.

To get around this problem, Javascript utilizes asynchronicity.

Asynchronous
Not simultaneous; not concurrent in time; -- opposed to synchronous.

The dictionary definition may not seem super helpful in our context, however applying to file loading over the internet in Javascript, we say that we load files over the internet using an asynchronous operation. An asynchronous file loading operation - a non-simultaneous/non-concurrent file loading operation, ie. it is non-simultaneous/non-concurrent with the single thread that Javascript uses. This means a file loading operation happens outside of the single threaded context and can finish at any point of the single thread.

What this means in practice is that when you ask for a file to be loaded in Javascript through an asynchronous operation, Javascript will hand that file loading operation off to the browser or OS to do the loading while continuing to run the rest of the code, regardless of whether that file loading operation has finished or not, if you come from other programming languages that does not utilize asynchronicity, this may different from what you may expect. So what happens when the file finishes loading? In that case, whenever that is and wherever Javascript has executed the code to, the browser or OS responsible for loading the file will notify Javascript that it has finished loading the file and Javascript will then decide what to do with the loaded file.

Ok so this is all a bit handwave-y and somewhat simplified. Expressed in pseudo-code we will see the following behavior:


// The next line of code runs `loadImage()` asynchronously
// meaning it will not wait until it finishes before running
// the next line below
let img = loadImage("./my-image.png");

// The next line will run while the image is still loading.
console.log("We are loading an image");
console.log(img); 
// It is undefined what `img` will be as it is not loaded yet.
// At some point `img` may have the value loaded by `loadImage`
// but we don't know if it is 1 line of code later or 100 
// lines of code later, or even never.

Well this doesn’t seem very useful when we have no way of knowing when the file finishes loading, where has our code executed to since it can take 1 second, 5 second, or even a whole minute for the file to load. To get around this, Javascript initially uses callback functions. In the context of asynchronous file loading, a callback function is a function that we define for Javascript to run when it is notified that the file loading has completed.

loadImage("./my-image.png", function(img){
  // This is the callback function that runs when the file is
  // loaded. `img` is now populated with the loaded data
  console.log("Finished loading image");
});

// The next line will run while the image is still loading.
console.log("We are loading an image");

// The console will print "We are loading an image" then
// "Finished loading image", as the synchronous `console.log()`
// on the outside runs first without waiting for `loadImage()`
// then the `console.log()` inside runs when `loadImage()`
// completes.

This works and we have a point within our code where we know for sure that our file loading operation has completed. However, it also means that our code is now split into two different sections. We may be able to put most of our code within the callback function but that somewhat breaks the linear flow of the code and when we try to do multiple asynchronous operations one after another, we can fall into what’s called “callback hell”.

// Here we are trying to load three images one after another
loadImage("./my-image1.png", function(img1){
  console.log("Finished loading image 1");

  loadImage("./my-image2.png", function(img2){
    console.log("Finished loading image 2");

    loadImage("./my-image3.png", function(img3){
      console.log("Finished loading image 3");

      // The rest of our code ...
    });
  });
});

For a time that is what people did and I hope you can see why these same people wanted to come up with a better solution. Enter “promises”. Promises is a new way of handling an asynchronous operation finishing that preserves more of the linearity of the code while avoiding callback hell. Using promises, and loading one file after another, we can have the following code:

loadImage("./my-image1.png")
  .then(function(img1) {
    return loadImage("./my-image2.png");
  })
  .then(function(img2) {
    return loadImage("./my-image3.png");
  })
  .then(function(img3) {
    // The rest of our code ...
  });

Now with each asynchronous operation, we can wait for it to finish then continue with what we want to do with the loaded data. No matter how many files we load in sequence, we won’t get callback hell. Promises also provide additional quality of life improvements through concurrent promise resolution with Promise.all() but we won’t explain that in detail for now. This is definitely an improvement over plain old callbacks, however it still isn’t quite the same as writing linear code, we still need to encapsulate code within the callback function of the .then() method. It would be nice to be able to have code that looks as linear as possible while preserving the single threaded requirement of Javascript and asynchronicity.

This is where the async/await keywords come in. A common misconception is that async/await is a completely new method of dealing with asynchronicity in Javascript, just like how promises and callbacks are two different methods of dealing with asynchronicity, that’s not quite true. async/await is promises, just in a different packaging. The keywords are best understood in the context of a function:

async function loadImage(){
  // Some asynchronous operation that populate `img` with data
  return img;
}

In the above, the function loadImage has the async keyword before the function keyword, marking it as an asynchronous function. An asynchronous function in this context simply means a function that returns a promise. The return value of the asynchronous function will evaluate into the resolved value of the returned promise. This means that you can do the following by treating the asynchronous function as a promise:

async function loadImage(){
  // Some asynchronous operation that populate `img` with data
  return img;
}

loadImage()
  .then(function(img){
    // The rest of our code ...
  });

However, this doesn’t really solve the initial problem we want to solve with promises if we are just using the same syntax again. This is where await comes in. await can only be used in an async function (and in modern Javascript environments in the global scope but we’ll not go into this) and it is used in front of calling a promise:

async function loadImage(){
  // Some asynchronous operation that populate `img` with data
  return img;
}

async function main(){
  console.log("The image will now start to load");
  let img = await loadImage("./my-image1.png");
  // In the next line `img` will have the expected value
  console.log("We have finished loading the image", img);
}

main();

When await is used this way, it tells Javascript to wait at this point until the promise has resolved before continuing to run the code that comes after await. There can be multiple awaits in the same async function. This does not mean Javascript is going to sit around and do nothing however, remember this is the very thing we don’t want Javascript as a single threaded language to do right at the start. Instead, Javascript will continue running code that is not being awaited outside of the scope of the async function, remember an async function is just a promise like before.

async function loadImage(){
  // Some asynchronous operation that populate `img` with data
  return img;
}

async function main(){
  console.log("A");
  let img = await loadImage("./my-image1.png");
  // In the next line `img` will have the expected value
  console.log("C");
}

main();

console.log("B");
// The order in which we will see messages are:
// "A"
// "B"
// "C"
// `await` pause the function execution and "jump" out of
// the async function to continue on.

With the use of async/await, within an asynchronous function (ie. the main function in the above example), we have managed to preserve the linearity of our code and not even needing any callbacks, all the while preserving the asynchronous single thread nature of Javascript, wins all around! As such, you will see that all modern Javascript code nowadays all default to using async/await for asynchronous operations and all functions that perform asynchronous operations returns promises in anticipation that it will be used with async/await.

Part 2

Alright, so now we understand the history and the true nature of async/await being just promises underneath to enable linear code, let’s look at this in the context of p5.js. Back in 2014 when p5.js was first created, promises were not a thing in Javascript. Promises was introduced in 2015 and async/await in 2016. Javascript development was very different back then because even when promises and async/await were released in 2015/2016, browser support for these new features were very patchy, meaning that as a developer you will have to provide a patched implementation for promises if you want to use it and not being able to use async/await at all. This leaves callbacks as the most reliable solution for dealing with asynchronicity.

As we have seen, that’s not ideal, especially not in the context of p5.js. We can still use callbacks for file loading in p5.js, however that’s not a very common use case and we can see why:

let myImg;
function setup(){
  createCanvas(400, 400);
  loadImage("./my-image.png", function(img){
    image(img, 0, 0, width, height);

    // OR

    // The first few loop of draw will have an
    // `undefined` `myImg` because of asynchronicity
    myImg = img
  );
}

Absent promises and async/await, is there some other way to avoid callbacks and ensure the loaded data is already loaded? The preload function that existed up until p5.js 2.0 is such a solution that is essentially a hack made possible by how p5 itself runs. p5.js code typically has a setup function that runs at the start of the sketch, this setup function since it cannot be an async function, means that it must run its code synchronously and not wait for any asynchronous operations to complete before going onto the next line of code:

let img;
function setup(){
  createCanvas(400, 400);
  img = loadImage("./my-image.png");

  // Since `setup` is synchronous, the next line will run 
  // and error because `loadImage()` has not finished.
  image(img, 0, 0, width, height);
}

That is not ideal. What “preload” provides is a separate function that p5 can run before setup, that p5 artificially waits until registered asynchronous operations within it have completed, before starting setup. This is roughly how it works under the hood:

let preloadCounter = 0;

function loadImage(url){
  let img = {};

  preloadCounter++;
  loadURLAsync(url, function(data){
    img.data = data;
    preloadCounter--;
  });

  // This works because Javascript returns object by reference
  // and not by copy, meaning the callback function of `loadURLAsync`
  // can modify its value after the fact.
  return img;
}

// Let's say `loadImage()` is called at this point.

let interval = setInterval(function(){
  if(preloadCounter === 0){
    clearInterval(interval);
    // Loading completed, now we can run `setup()` and
    // start the sketch.
  }
  // Loading has not completed yet so we keep waiting.
}. 200);

This is not exactly how preload is implemented but the general idea is the same. As you can see in this case, while we don’t have promises and async/await, we can somewhat simulate similar behavior because we control when the sketch starts. The downsides here are that only functions that were specifically written to utilize this behavior can be used (ie. they must increment and decrement the preload counter), and that the syntax does not fit with the semantics of Javascript (this is a bit more of an advanced and philosophical argument so we won’t expand on here).

As of time of writing, we are about 10 years into promises being released in Javascript, and it is near impossible to find a Javascript environment that does not support async/await nor a library that does not use promises for asynchronous operations. Well, of course p5.js is that exception. With the Javascript ecosystem having converged on async/await and modern Javascript learning resources assumes and defaults to async/await, it makes sense for p5.js to move inline with the standard, for no other reason than to fulfill one key use case for p5.js which is to be a platform for someone new to coding to learn coding in Javascript. From a personal point of view, I have always see p5.js as the starting point for a user’s journey and not the endpoint, if the user move on from p5.js to other more advanced libraries or environments because their needs have evolved, it is not a failure of the project as long as we have equipped them to take on these new challenges. I see async/await as one of the necessary steps to bridge a user between the friendliness of p5.js and the potential complexity of a different environment. Obscuring the complexity of asynchronous operations with preload was necessary because of callbacks and its syntax but with async/await it has resolve the same problem preload was trying to solve but in a better way, because now any asynchronous functions, whether they were specifically written to be used with p5.js or not, will work with p5.js.

The motivation for this change came not entirely from p5.js itself but rather partly from a community project ml5.js. When ml5.js was being reimplemented for a new major version, discussions around preload were raised: https://github.com/ml5js/ml5-next-gen/issues/29. In particular, for ml5.js as a library to support both usage within p5.js and as a standalone library, they would need to simultaneously support a loading function both returning a promise and using preload with returning the loaded value, even if their underlying implementation and library uses promises by default.

p5.js 2.0 is a good opportunity to consider larger changes that we were not able to otherwise and introducing async/await is one of these changes. Removing preload as a function is also a change that we can only do with a major release such as 2.0 (this is due to semver restrictions) and while it is possible to also leave preload around, it would mean the next opportunity to remove it will be in p5.js 3.0. This question of whether to keep preload around for 2.0 was put to the p5.js 2.0 advisory committee, the consensus is to prioritise async/await because of its prevalence in the Javascript ecosystem and to make a clean break by not including the preload function. The point around reducing friction in students learning p5.js and learning about the wider Javascript ecosystem was also brought up as a reason to favor async/await over preload, which cannot be used with any other libraries that have asynchronous operations, out of the box.

Part 3

Now we have the context and reasoning behind the change out of the way, let’s see what you will need to do to update your code to work with p5.js 2.0. One point before we get into it though, p5.js 1.x versions already released are not going anywhere, if your sketch is already working you likely won’t need to update it. There will also be a p5.js 2.0 compatibility addon that replicates preload (and other changes) so that you can still use p5.js 2.0 with preload while you transition your syntax if you wish.

However, if you are a teacher updating your teaching material to use p5.js 2.0, or a library author updating your library to work with p5.js 2.0 without preload, or other use cases that warrant a conversion, read on!

We will first look at updating user code then we’ll look at updating library code. For user code, the trick is to first copy all the code you had in the preload function into the setup function before any of the variables are being used, you can put it above createCanvas if you wish to. Next add the async keyword before function setup so that it becomes async function setup. Finally add the await keyword before each of the loading functions that you just copied from the preload function, so img = loadImage(“./my-image.png”) becomes img = await loadImage(“./my-image.png”). And that’s it! See below for a comparison:

Before:

let img, data;
function preload(){
  img = loadImage("./my-image.png");
  data = loadJSON("./my-data.json");
}

function setup(){
  createCanvas(400, 400);
}

After:

let img, data;
async function setup(){
  img = await loadImage("./my-image.png");
  data = await loadJSON("./my-data.json");
  createCanvas(400, 400);
}

Now if you plan to use a different library to do some other asynchronous operation before your sketch starts, as long as they are using promises, you can await them in the same way in the async setup function.

For library authors, the change will likely be a simplification of your code. You will no longer need to call registerPreloadMethod, _incrementPreload, and _decrementPreload functions, they can be removed. Next the asynchronous functions previously calling the above functions now need to return a promise, the easiest way to do this is to prefix async before the function keyword like what we are doing with the setup function above. Once these are done, your async functions will work with async setup or any other async functions out there.

Before:

p5.prototype.loadCSV = function (filename){
  let result = [];

  fetch(filename)
    .then((res) => res.text())
    .then((data) => {
      data.split('\n').forEach((line) => {
        result.push(line.split(','));
      });

      this._decrementPreload();
    });

  return result;
};

p5.prototype.registerPreloadMethod('loadCSV', p5.prototype);

After:

p5.prototype.loadCSV = async function (filename){
  // We can return from `fetch` directly here without
  // the `async` keyword since `fetch` uses promises
  // but it doesn't change anything to include anyway.
  return fetch(filename)
    .then((res) => res.text())
    .then((data) => {
      let result = data.split('\n').map((line) => {
        return line.split(',');
      });

      return result;
    });
};

You may wish to preserve compatibility with 1.x for your library and there are a few options. First, you can have two versions of the library one with preload support and one with promises support. Second, you may choose to only have one version of your library now with promises support, users wishing to use p5.js 1.x can use your latest released version that still supports preload. Finally it is possible with some extra implementation to support both by still having your function return a promise and call the preload methods after checking for their existence.

p5.prototype.loadCSV = function (filename){
  if(this._incrementPreload && this._decrementPreload){
    // Here we use the old implementation
    this._incrementPreload();
    let result = [];

    fetch(filename)
      .then((res) => res.text())
      .then((data) => {
        data.split('\n').forEach((line) => {
          result.push(line.split(','));
        });

        this._decrementPreload();
      });

    return result;

  }else{
    // Here we return a promise, note that we must return
    // a promise and not use the `async` keyword
    return fetch(filename)
      .then((res) => res.text())
      .then((data) => {
        let result = data.split('\n').map((line) => {
          return line.split(',');
        });

        return result;
      });
  }
});

There are still additional ideas that are being explored in terms of asynchronous code in p5.js such as the ability to easily load files in parallel. With changes in p5.js 2.0, we have the headroom to explore these in subsequent updates and we would like to invite your participation especially as users and learners of p5.js in these discussions.