DEV Community: Lars den Bakker

Developing Without a Build (2): es-dev-server

Lars den Bakker — Sun, 04 Aug 2019 19:28:59 +0000

Developing Without a Build: es-dev-server

This article is part of a series on developing without a build:

In the introduction article, we looked into different workflows and browser APIs. In this article, we will look into how we can set up es-dev-server, and how it can help us developing without a build.

es-dev-server

es-dev-server is a composable web server that focuses on developer productivity when developing without a build step. Through options, you can opt into extra features such as caching, reloading on file changes, SPA routing, resolving bare module imports and compatibility modes to support older browsers.

Setup

To start off let's create an empty npm project and install es-dev-server:

npm init
npm i -D es-dev-server

Create an index.html in the root of your project:

<!DOCTYPE html>
<html>
<head></head>

<body>
  <div id="app"></div>
  <script type="module" src="./src/app.js"></script>
</body>

</html>

And create a dummy src/app.js file:

console.log('hello world');

We can now run our application with es-dev-server:

npx es-dev-server --open

Without any extra flags, es-dev-server acts like a regular static file server. Any extra functionality needs to be enabled explicitly.

Bare imports

One of the first things you will run into when developing without any build tools is how to deal with bare module imports like this:

import foo from 'foo';

Out of the box, the browser will throw an error, as it doesn't know how to handle these kinds of imports. In our previous article we explored how to use these imports by using import maps, an upcoming browser API.

Until import maps are properly supported in browsers, we can use the --node-resolve flag of the dev server. This will rewrite imports in your modules using NodeJS module resolution before serving them to the browser.

To see how this works let's add lit-html, a HTML templating library, to our project:

npm i -D lit-html

Change src/app.js to import it:

import { render, html } from 'lit-html';

const template = html`<p>Hello world!</p>`;
render(template, document.getElementById('app'));

Now let's restart our server, adding the node resolve flag:

npx es-dev-server --node-resolve --open

If you inspect the network tab, you will see the modules are served correctly as expected. src/app.js is rewritten to:

import { render, html } from './node_modules/lit-html/lit-html.js';

const template = html`<p>Hello world!</p>`;
render(template, document.getElementById('app'));

Watch mode

A great productivity booster is reloading the browser while you are editing files.

To enable this option, restart the server with the watch flag:

npx es-dev-server --watch --node-resolve --open

Now change anything inside one of the served files, for example the rendered html in app.js. You will see the browser reload automatically with the updated code.

Reloads are done using the EventSource API, which is not supported on Edge and IE11. The dev server injects a small script, which connects to a message channel endpoint:

Caching

es-dev-server uses the file system's last modified timestamp to return a 304 if the file hasn't changed. This significantly speeds up reloads. You can test this in your browser by turning off Disable cache and refreshing:

Folder structure

Not every project has a single index.html in the root of the project. Because es-dev-server works just like a regular web server, it can serve files from any folder.

For example, let's create a new folder called demo, and move our index.html inside it.

We will need to adjust the script src path to reflect this change:

<script type="module" src="../src/app.js"></script>

And we need to tell the server to open inside the demo folder:

npx es-dev-server --node-resolve --open /demo/

The application should be displayed without any changes.

Changing the root dir

We might be tempted to change the root directory of the web server in order to get rid of the /demo/ part in the URL:

npx es-dev-server --root-dir /demo/ --open

However, this won't work because the web server can only serve files that are within its root directory. By default, this is the current working directory. In our case, the web server needs to be able to serve the contents of the src folder, as well as the node_modules folder.

This is a common problem when working in a monorepo when you want to serve files from a package subdirectory. Many of the modules you need to serve are in the root of the project, so you need to move the root directory up two levels:

npx es-dev-server --root-dir ../../ --open packages/my-package/index.html

SPA Routing

If you are building a Single Page Application you are likely doing some form of front-end routing. In order to enable deeplinking or refreshing, the web server should return your index.html on deeper paths. This is sometimes called history API fallback.

Setting up a router is beyond the scope of this article, but the option is easy to enable using the --app-index flag:

npx es-dev-server --node-resolve --app-index index.html --open

When using the --app-index flag, the server will automatically open the server on your app's index if you don't pass an explicit path to --open.

Compatibility with older browsers

Although we can use the latest versions of the major browsers for development, we still need to support older browsers in production. We also might be using some new browsers features that are not yet supported in the latest version of all the major browsers.

It would be a shame if we have to run a production build every time we want to run our app on one of these browsers. es-dev-server supports multiple compatibility modes that help with this.

When compatibility mode is enabled, the server handles the necessary polyfills and code transformations for older browsers. This takes us into build tooling territory, so we're no longer purely "developing without build tools". I think that's fine, as we're using it only for browser compatibility. You have to opt-in to this behavior explicitly.

Let's see how it works in action. Add a dynamic import to app.js to lazy load a module when a button is clicked:

import { html, render } from 'lit-html';

async function lazyLoad() {
  await import('lit-html/directives/until.js');
}

const template = html`
  <button @click=${lazyLoad}>Click me!</button>
`;
render(template, document.getElementById('app'));

The dynamic import does not really do anything functional. If we run this on Chrome, Safari, and Firefox it works just fine. Because Edge does not yet support dynamic imports we can't run this code there.

We can turn on the lightest compatibility mode, esm, to handle this case:

npx es-dev-server --node-resolve --compatibility esm --open

With esm enabled, es-dev-server injects es-module-shims and adds a loader script to your index. You don't need to change any of your code for this. You can view the injected script in the index file:

There is some extra boilerplate, as the other compatibility modes might add more polyfills. The polyfills are hashed so that they can be cached aggressively in the browser.

Besides esm there are the modern and all compatibility modes. These modes inject polyfills for common browser APIs and use @babel/preset-env for transforming the latest javascript syntax to a compatible format.

In modern compatibility mode your code is made compatible with the latest two versions of Chrome, Safari, Firefox, and Edge.

In all compatibility mode support is extended to older browsers, all the way to IE11. On browsers which don't support es modules, they are transformed to systemjs modules.

The transformations slow down the server a bit, so I don't recommend using modern or all mode during regular development. You can create separate scripts in your package.json, and run in compatibility mode only when you view your app on older browsers.

esm mode has a negligible effect on performance, so that should be fine to use.

Import maps

In the previous article, we briefly discussed import maps as an upcoming browser feature which handles bare module imports. es-module-shims and systemjs both support import maps. If compatibility mode is enabled, the dev server takes care of adapting your import maps to work with these libraries, making this a viable workflow during development.

The import maps API is not fully stabilized yet, so if you're going down this path it's good to keep an eye how this standard evolves. Check out this article for a workflow using import maps.

More options

Check out the official documentation for more options, such as integrating the dev server as a library with other tools and installing custom middlewares.

Getting started

To get started with a project that sets up the dev server for you, use the open-wc project scaffolding:

npm init @open-wc

It sets up the project with lit-element, a web component library. You can swap this for any library of your choosing, the setup is not specific to web components.

Developing Without a Build (1): Introduction

Lars den Bakker — Sun, 04 Aug 2019 19:16:12 +0000

Developing Without a Build: Introduction

This article is part of a series on developing without a build:

In this article, we explore why and if we should do development without a build step, and we give an overview of the current and future browser APIs that make this possible. In the followup articles, we look into how es-dev-server can help us with that and how to handle testing.

Modern web development

In the early days of web development, all we needed was a simple file editor and a web server. It was easy for newcomers to understand the process and get started making their own web pages. Web development has changed a lot since then: the complexity of the tools we use for development has grown just as much as the complexity of the things that we're building on the web.

Imagine what it's like if you're coming in completely new to web development:

You first need to learn a lot of different tools and understand how each of them is changing your code before it can actually run in the browser.
Your IDE and linter likely do not understand the syntax of this framework that was recommended to you by a friend, so you need to find the right mix of plugins that makes it work.
Source maps need to be configured properly for all of the tools in the chain if you want to have any chance of debugging your code in the browser. Getting them to work with your tests is a whole other story.
You decided to keep things simple and not use typescript. You're following the tutorials and but can't get this decorators thing to work and the error messages are not helping. Turns out you didn't configure your babel plugins in the correct order...

It may sound exaggerated, and I know that there are very good starter projects and tutorials out there, but this experience is common to a lot of developers. You may have jumped through similar hoops yourself.

I think that's really a shame. One of the key selling points of the web is that it's an easy and open format. It should be easy to just get started right away without a lot of configuration and ceremony.

I'm not criticizing the build tools themselves, they all have a role and a purpose. And for a long time, using a build was the only real way to actually create complex applications on the web. Web standards and browser implementations were just not there to support modern web development. Build tools have really helped push web development forward.

But browsers have improved a lot in the past years, and there are many exciting things to come in the near future. I think that now is a good time to consider if we can do away with a big part of the tooling complexity, at least during development. Maybe not yet for all types of projects, but let's see how far we can go.

Loading modules in the browser

This is not a step by step tutorial, but you can follow along with any of the examples by using any web server. For example http-server from npm. Run it with -c-1 to disable time-based caching.

npx http-server -o -c-1

Loading modules

Modules can be loaded in the browser using regular script tags with a type="module" attribute. We can just write our module code directly inline:

<!DOCTYPE html>
<html>
<head></head>

<body>
  <script type="module">
    console.log('hello world!');
  </script>
</body>

</html>

From here we can use static imports to load other modules:

<script type="module">
  import './app.js';

  console.log('hello world!');
</script>

Note that we need to use an explicit file extension, as the browser doesn't know which file to request otherwise.

The same thing works if we use the src attribute:

<script type="module" src="./app.js"></script>

Loading dependencies

We don't write our code in just one file. After importing the initial module, we can import other modules. For example, let's create two new files:

src/app.js:

import { message } from './message.js';

console.log(`The message is: ${message}`);

src/message.js:

export const message = 'hello world';

Place both files in a src directory and import app.js from your index.html:

<!DOCTYPE html>
<html>
<head></head>

<body>
  <script type="module" src="./src/app.js"></script>
</body>

</html>

If you run this and check the network panel, you will see both modules being loaded. Because imports are resolved relatively, app.js can refer to message.js using a relative path:

This seems trivial, but it is extremely useful and something we did not have before with classic scripts. We no longer need to coordinate dependencies somewhere central or maintain a base URL. Modules can declare their own dependencies, and we can import any module without knowing what their dependencies are. The browser takes care of requesting the correct files.

Dynamic imports

When building any serious web application we are usually going to need to do some form of lazy loading for best performance. Static imports like we saw before cannot be used conditionally, they always need to exist at the top level.

For example, we cannot write:

if (someCondition) {
  import './bar.js';
}

This is what dynamic imports are for. Dynamic imports can import a module at any time. It returns a Promise that resolves with the imported module.

For example let's update the app.js example we created above:

window.addEventListener('click', async () => {
  const module = await import('./message.js');

  console.log(`The message is: ${module.message}`);
});

Now we are not importing the message module right away, but delaying it until the user has clicked anywhere on the page. We can await the promise returned from the import and interact with the module that was returned. Any exported members are available on the module object.

Lazy evaluation

This is where developing without a bundler has a significant benefit. If you bundle your application before serving it to the browser, the bundler needs to evaluate all your dynamic imports to do code splitting and output separate chunks. For large applications with a lot of dynamic imports, this can add significant overhead as the entire application is built and bundled before you can see anything in the browser.

When serving unbundled modules, the entire process is lazy. The browser only does the necessary work to load the modules that were actually requested.

Dynamic imports are supported by the latest versions of Chrome, Safari, and Firefox. It's not supported in the current version of Edge but will be supported by the new Chromium-based Edge.

Non-relative requests

Not all browser APIs resolve requests relative to the module's location. For example when using fetch or when rendering images on the page.

To handle these cases we can use import.meta.url to get information about the current module's location.

import.meta is a special object which contains metadata about the currently executing module. url is the first property that's exposed here, and works a lot like __dirname in NodeJS.

import.meta.url points to the url the module was imported with:

console.log(import.meta.url); // logs http://localhost:8080/path/to/my/file.js

We can use the URL API for easy URL building. For example to request a JSON file:

const lang = 'en-US';

// becomes http://localhost:8080/path/to/my/translations/en-US.json
const translationsPath = new URL(`./translations/${lang}.json`, import.meta.url);

const response = await fetch(translationsPath);

Loading other packages

When building an application you will quickly run into having to include other packages from npm. This also works just fine in the browser. For example, let's install and use lodash:

npm i -P lodash-es

import kebabCase from '../node_modules/lodash-es/kebabCase.js';

console.log(kebabCase('camelCase'));

Lodash is a very modular library and the kebabCase function depends on a lot of other modules. These dependencies are taken care of automatically, the browser resolves and imports them for you:

Writing explicit paths to your node modules folder is a bit unusual. While it is valid and it can work, most people are used to writing what's called a bare import specifier:

import { kebabCase } from 'lodash-es';
import kebabCase from 'lodash-es/kebabCase.js';

This way you don't specifically say where a package is located, only what it's called. This is used a lot by NodeJS, whose resolver will walk the file system looking for node_modules folders and packages by that name. It reads the package.json to know which file to use.

The browser cannot afford to send a bunch of requests until it stops getting 404s, that would be way too expensive. Out of the box, the browser will just throw an error when it sees a bare import. There is a new browser API called import maps which lets you instruct the browser how to resolve these imports:

<script type="importmap">
  {
    "imports": {
      "lodash-es": "./node_modules/lodash-es/lodash.js",
      "lodash-es/": "./node_modules/lodash-es/"
    }
  }
</script>

It's currently implemented in chrome behind a flag, and it's easy to shim on other browsers with es-module-shims. Until we get broad browser support, that can be an interesting option during development.

It's still quite early for import maps, and for most people, they may still be a bit too bleeding edge. If you are interested in this workflow I recommend reading this article

Until import maps are properly supported, the recommended approach is to use a web server which rewrites the bare imports into explicit paths on the fly before serving modules to the browser. There are some servers available that do this. I recommend es-dev-server which we will explore in the next article.

Caching

Because we aren't bundling all of our code into just a few files, we don't have to set up any elaborate caching strategies. Your web server can use the file system's last modified timestamp to return a 304 if the file hasn't changed.

You can test this in your browser by turning off Disable cache and refreshing:

Non-js modules

So far we've only looked into javascript modules, and the story is looking pretty complete. It looks like we have most of the things we need to write javascript at scale. But on the web we're not just writing javascript, we need to deal with other languages as well.

The good news is that there are concrete proposals for HTML, CSS and JSON modules, and all major browser vendors seem to be supportive of them:

The bad news is that they're not available just yet, and it's not clear when they will be. We have to look for some solutions in the meantime.

JSON

In Node JS it's possible to import JSON files from javascript. These become available as javascript objects. In web projects, this is used frequently as well. There are many build tool plugins to make this possible.

Until browsers support JSON modules, we can either just use a javascript module which exports an object or we can use fetch to retrieve the JSON files. See the import.meta.url section for an example which uses fetch.

HTML

Over time web frameworks have solved HTML templating in different ways, for example by placing HTML inside javascript strings. JSX is a very popular format for embedding dynamic HTML inside javascript, but it is not going to run natively in the browser without some kind of transformation.

If you really want to author HTML in HTML files, until we get HTML modules, you could use fetch to download your HTML templates before using it with whatever rendering system you are using. I don't recommend this because it's hard to optimize for production. You want something that can be statically analyzed and optimized by a bundler so that you don't spawn a lot of requests in production.

Luckily there is a great option available. With es2015/es6 we can use tagged template string literals to embed HTML inside JS, and use it for doing efficient DOM updates. Because HTML templating often comes with a lot of dynamism, it is actually a great benefit that we can use javascript to express this instead of learning a whole new meta syntax. It runs natively in the browser, has a great developer experience and integrates with your module graph so it can be optimized for production.

There are some really good production ready and feature complete libraries that can be used for this:

For syntax highlighting you might need to configure your IDE or install a plugin.

CSS

For HTML and JSON there are sufficient alternatives. Unfortunately, with CSS it is more complicated. By itself, CSS is not modular as it affects the entire page. A common complaint is that this is what makes CSS so difficult to scale.

There a lot of different ways to write CSS, it's beyond the scope of this article to look into all of them. Regular stylesheets will work just fine if you load them in your index.html. If you are using some kind of CSS preprocessor you can run it before running your web server and just load the CSS output.

Many CSS in JS solutions should also work if the library publishes an es module format that you can import.

Shadow dom

For truly modular CSS I recommend looking into Shadow dom, it fixes many of the scoping and encapsulation problems of CSS. I've used it with success in many different types of projects but it's good to mention that it's not yet a complete story. There are still missing features that are being worked out in the standard so it may not yet be the right solution in all scenarios.

Good to mention here is the lit-element library, which offers a great developer experience when authoring modular CSS without a build step. lit-element does most of the heavy lifting for you. You author CSS using tagged template literals, which is just syntax sugar for creating a Constructable Stylesheet. This way you can write and share CSS between your components.

This system will also integrate well with CSS modules when they are shipped. We could emulate CSS modules by using fetch, but as we saw with HTML it's hard to optimize this for production use. I'm not a fan of CSS in JS, but lit-element's solution is different and very elegant. You're writing CSS in a JS file, but it's still valid CSS syntax. If you like to keep things separated, you can just create a my-styles.css.js file and use a default export of just a stylesheet.

Library support

Luckily the amount of libraries shipping es module format is growing steadily. But there are still popular libraries which only ship UMD or CommonJS. These don't work without some kind of code transformation. The best thing we can do is open issues on these projects to give them an indication of how many people are interested in supporting the native module syntax.

I think this is a problem that will disappear relatively quickly, especially after Node JS finishes their es modules implementation. Many projects already use es modules as their authoring format, and I don't think anyone really likes having to ship multiple imperfect module formats.

Final thoughts

The goal of this article is to explore workflows where we don't need to do any building for development, and I think we've shown that there are real possibilities. For a lot of use cases, I think we can do away with most of the tooling for development. In other cases, I think they can still be useful. But I think our starting point should be reversed. Instead of trying to make our production builds work during development, we should be writing standard code that runs in the browser as is and only perform light transformations if we think it's necessary.

It's important to reiterate that I don't think that build tools are evil, and I am not saying that this is the right approach for every project. That is a choice that each team should make for themselves based on their own requirements.

es-dev-server

You can do almost everything described in this article with any regular web server. That being said, there are still web server features which can really help with the development experience. Especially if we want to run our applications on older browsers, we might need some help.

At open-wc we created es-dev-server, a composable web server that focuses on developer productivity when developing without a build step.

Check out our next article to see how we can set it up!

Getting started

To get started with developing without any build tools, you can use the open-wc project scaffolding to set up the basics:

npm init @open-wc

It sets up the project with lit-element, a web component library. You can swap this for any library of your choosing, the setup is not specific to web components.