Caching JavaScript data file results when using Eleventy

Eleventy by Zach Leatherman has become my default static site generator. It is simple, uses JavaScript, and is easy to extend. It allows me to include custom code to access additional data sources,
such as RDF datasets.

Querying data can take up some time, for example, when using an external Web API. During deployment of a website this is not a big deal, as this probably doesn't happen every minute. But when you are developing then it might become an issue: you don't want to wait for query results every time you make a change that doesn't affect the results, such as updating a CSS property, which only affects how the results are visualized. Ideally, you want to reuse these results without querying the data over and over again. I explain in this blog post how that can be done by introducing a cache.

The cache has the following features:

• The cache is only used when the website is locally served (eleventy --serve).
• The cached data is written to and read from the filesystem.

This is done by using the following two files:

• serve.sh: a Bash script that runs Eleventy.
• cache.js: a JavaScript file that defines the cache method.

An example Eleventy website using these two files is available on Github.

Serve.sh

#!/usr/bin/env bash

# trap ctrl-c and call ctrl_c()
trap ctrl_c INT

function ctrl_c() {
  rm -rf _data/_cache
  exit 0
}

# Remove old folders
rm -rf _data/_cache # Should already be removed, but just in case
rm -rf _site

# Create needed folders
mkdir _data/_cache

ELEVENTY_SERVE=true npx eleventy --serve --port 8080

This Bash script creates the folder for the cached data and serves the website locally. First, we remove the cache folder and the files generated by Eleventy, which might still be there from before. Strictly speaking removing the latter is not necessary, but I have noticed that removed files are not removed from _site, which might result in unexpected behaviour. Second, we create the cache folder again, which of course is now empty. Finally, we set the environment variable ELEVENTY_SERVE to true and start Eleventy: we serve the website locally on port 8080. The environment variable is used by cache.js to check if the website is being served, because currently this information can't be extracted from Eleventy directly. Note that I have only tested this on macOS 10.12.6 and 10.14.6, and Ubuntu 16.04.6. Changes might be required for other OSs.

Cache.js

const path = require('path');
const fs = require('fs-extra');

/**
 * This method returns a cached version if available, else it will get the data via the provided function.
 * @param getData The function that needs to be called when no cached version is available.
 * @param cacheFilename The filename of the file that contains the cached version.
 * @returns the data either from the cache or from the geData function.
 */
module.exports = async function(getData, cacheFilename) {
  // Check if the environment variable is set.
  const isServing = process.env.ELEVENTY_SERVE === 'true';
  const cacheFilePath = path.resolve(__dirname, '_data/_cache/' + cacheFilename);
  let dataInCache = null;

  // Check if the website is being served and that a cached version is available.
  if (isServing && await fs.pathExists(cacheFilePath)) {
    // Read file from cache.
    dataInCache = await fs.readJSON(cacheFilePath);
    console.log('Using from cache: ' + cacheFilename);
  }

  // If no cached version is available, we execute the function.
  if (!dataInCache) {
    const result = await getData();

    // If the website is being served, then we write the data to the cache.
    if (isServing) {
      // Write data to cache.
      fs.writeJSON(cacheFilePath, result, err => {
        if (err) {console.error(err)}
      });
    }

    dataInCache = result;
  }

  return dataInCache;
};

The method defined by the JavaScript file above takes two parameters: getData and cacheFilename. The former is the expensive function that you don't want to repeat over and over again. The latter is the filename of the file with the cached version. The file will be put in the folder _data/_cache relative to the location of cache.js. The environment variable used in serve.sh is checked here to see if the website is being served. Note that the script requires the package fs-extra, which adds extra methods to fs and is not available by default.

Putting it all together

To get it all running, we put both files in our Eleventy project root folder. Do not forget to make the script executable and run serve.sh.

When executing the aforementioned example, we see that the first time to build the website it takes 10.14 seconds (see screencast below). No cached version of the query results is available at this point and thus the Web API has to be queried. But the second time, when we update the template, it only takes 0.03 seconds. This is because the cached version of the query results is used instead of querying the Web API again.

Screencast: When the Web API is queried it takes 10.14 seconds. When the cached version of the query results is used it takes 0.03 seconds.