Rebuild only when necessary in Node

If your project takes some time to prepare- maybe a compile or rewrite step- you might be waiting longer than you need to be on every load. This hits us on Google's Santa Tracker 🎅, where we use the Closure Compiler to build most games. Closure is a great optimizer, but it's not known for speed.

So the theme of this post is: slow builds are no fun, and we're going to learn how to only run them when we need to! 🎉

The Watch Primitive

Instead of building a game, scene or codebase every time you load a compiled resource or manually re-run a script, we can use NodeJS' fs.watch function to inform us whether we actually need to.

At face value, this is a straightforward method which tells you when a file has changed on-disk. Use it like this:

const fs = require('fs');
fs.watch('yourfile.txt', (eventType, filename) => {
  // something happened to 'yourfile.txt': ¯\_(ツ)_/¯
});

This is a super efficient method because it asks your operating system to let you know when something changed (not the other way around, where your program has to constantly check).

Build Usage

Let's say you're compiling some Less CSS files. You do this by compiling a file, entrypoint.less, that has dependencies:

const less = require('less');

less.render(`@import 'entrypoint.less';`).then((output) => {
  console.info(output.css);

  // contains all files that were imported, e.g:
  //   entrypoint.less => main.less => body.less
  // will result in ['entrypoint.less', 'main.less', 'body.less']
  const files = output.imports;
});

Less will provide us with a simple list of files it used in the build. Some other tools might provide you with a source map, which also contains the names of the original files.

If any of these files change, the final output is invalid, and we should rebuild it. In practice, this just means calling fs.watch on every file: 👀

  const files = output.imports;
  files.forEach((file) => {
    fs.watch(file, () => rebuildIsNeededCallback());
  });

This technically works, but it doesn't really fit into a whole build system yet. Read on! 😄👍

Caveats

While fs.watch is a powerful function, it has a few caveats. These can be summed up in a few points:

• You're not always guaranteed to be told which file has changed
• On Linux, macOS and others, fs.watch follows the inode of the watched file
  • ... if a file is moved, you'll be notified about changes in its new location
  • ... if a file is replaced, you'll be notified once, but the new file won't be automatically watched
• You need to call .close() on the result when you don't need it anymore—if you forget, your program will hold open listeners

In practice, these caveats means that you should use each call to fs.watch as a once-off hint that something has changed. 💭 Think of it this way: you can't be sure exactly what changed, but it's worth checking!

Another argument in thinking of fs.watch as a once-off: if your dependencies change by adding or removing files, it might be easier just to reset all your watchers rather than trying to keep up-to-date. 🤓

Watch Helper

Let's put the learnings above together into a small helper that'll help you invalidate code when it changes. This is what we do in Santa Tracker; we retain build output until it's no longer valid (because the underlying source has changed).

🚨 You might say "why invalidate, not just do a total rebuild?" Well unless you need the output as fast as possible, you're running an expensive compile step on every save.

So, the watch method below will accept a list of paths, watch them, and call a callback when any of them change (or a timeout passes):

function watch(paths, done, timeout=0) {
  let watchers;
  let timeoutId;
  const finish = () => {
    // To finish, we close watchers (because it's not clear
    // what state they are in), cancel the timeout callback,
    // and let the user know something changed.
    watchers.forEach((w) => w.close());
    clearTimeout(timeoutId);
    done();
  };

  if (timeout > 0) {
    // If a timeout is given, 'succeed' after ~timeout. This is
    // useful to *always* rebuild or invalidate after a time.
    timeoutId = setTimeout(finish, timeout);
  }
  watchers = paths.map((p) => fs.watch(p, finish));
}

Be sure to take a look at the code 👆, as I've left a few comments explaining what it does. Let's put this together with our Less example from above.

Less Is More

So just how can we invalidate output when the dependencies change?

We can do this with two methods and a cache variable:

• getCSS which which ensures a Promise containing the result is available; and
• buildCSS to actually rebuild when required (since this is async, it returns a Promise).

let compileCache;

async function buildCSS() {
  console.debug('rebuilding CSS...');
  const output = await less.render(`@import 'entrypoint.less';`);

  watch(output.imports, () => {
    compileCache = null;  // force a rebuild next time
  }, 60 * 1000);

  return output.css;
}

// call getCSS whenever you need CSS, and it'll always be up-to-date
function getCSS() {
  if (!compileCache) {
    compileCache = buildCSS();
  }
  return compileCache;
}

Of course, this is a very simplified example that only caches one result: if you wanted to extend it, you'd use a dictionary of outputs, each able to be invalidated if their dependencies change.

Finally

To finally hook up getCSS to the world, I'd now add a handler to your favourite NodeJS webserver so that when I load up say /compiled.css, it returns the result of getCSS, ensuring the compiled version is always up-to-date. In Polka, it might look like:

polka()
  .get('/compiled.css', (req, res) => {
    res.end(getCSS());
  });

If you're curious on more ways you might rig up a development server to do this, let me know below! 💬

Thanks

If you're using a modern packaging system (or build tool) directly, then that tool will likely be using fs.watch under the hood already. Yet, I still hope you've learned something about how you can use fs.watch to improve your build systems!

As an aside: I've personally stopped using build tools like gulp and grunt directly in favor of custom build tools or web servers that perform compilation on-demand (powered by fs.watch, as we do in Santa Tracker).

1 👋