Pascal Schilp

Posted on Apr 17, 2020

Lessons Learned Building a COVID-19 PWA

#pwa #preact #webcomponents #htm

Introduction

A few weeks ago, a colleague of mine asked if I'd be interested to help out with some design and development for an open source COVID-19 project. The 'intelligent lockdown' in my country had just started, and with a bunch of newfound time on my hands I figured that this would be a good opportunity to do my part. And while the project is not done yet, I'm very excited to share some techniques with you on how we approached certain things.

Project Lockdown

The project is titled Project Lockdown. The basic idea is that it's a worldmap that displays the various states of countries in lockdown around the world. While lockdowns generally help countries to flatten the curve, and prevent COVID-19 from spreading, countries with a dictatorial nature may potentially see this as an opportunity to increase their grasp over a country. Governments may unnecesarily prolong lockdowns or not stick to their announced end dates for political gain.

⚠️ Any data shown in screenshots or gifs is mock data

Sourcing the data

An important question in any COVID-19 related project is: How do you source the data and is the data credible? Our data gets entered by a group of 'editors' from all around the world, and can also be submitted by users, which then gets moderated by the editors. Only data that is sourced from official government pronouncements is accepted.

The data gets entered in Google Sheets, and we have a Node script that then scrapes the Google Sheets and outputs a worldmap.json to render the map of the world and the colours of various countries in lockdown, as well as individual country JSON files, like NL.json, which then can be consumed on the frontend. The Node script runs every 5 minutes as a cron job in a GitHub Action.

We also make use of the coronatracker API to fetch some stats like the amount of infected, deaths and recoveries per country. Coronatracker is recognised by WHO.

Frontend technologies used

Usually, my colleague Lars and I are very involved with web components, and work on a project named open-wc where we develop tools and libraries for modern web development. The web components scene is very much alive and kicking, but the fact of the matter is that relatively speaking the community can still be considered smaller than communities like React, or Preact. We felt we were more likely to get more contributions and be able to onboard new developers faster if we'd use a library that a larger pool of developers were familiar with. This project is not about pushing web components after all, but for the greater good. Fortunately, we were still able to use many of our tools, like es-dev-server which is excellent for developing with ES modules, as well as our rollup-plugin-html.

Since bundlesize is a big consideration for this project, we decided to go with Preact, mixed with a some additional helpers as web components.

However, we are still big fans of buildless development, and staying close to the browser, so we decided to use htm along with it. Htm is a very small library that lets you write jsx-like syntax directly in the browser, using tagged template literals.

Being used to web components, we've gotten spoiled by the nice encapsulation that shadow DOM provides, so going back to writing global CSS was a bit of a headache. Turns out, there are many, many solutions out there for scoping CSS, many of which require a buildstep and added complexity. So again, being fans of buildless development and staying close to the browser, we decided to go with csz by Luke Jackson, which lets you write css using... tagged template literals!

We had a lot of discussion around which library to use for the map, we found Mapbox to have an incredible native-like feeling, and many nice features, but unfortunately it also comes with a very large bundlesize. Since we wanted to keep our bundle size low, we went with Leaflet instead.

Building

For building the app for production, we use Rollup. Rollup is an incredibly user friendly buildtool, and relatively easy to configure.

There are a few steps we have to take in our build:

Copy assets
Generate the service worker and apply the service worker registration to the index.html
Optimise and Compress JS
Resolve bare module specifiers like import { Component } from 'preact'
Transform non-widely supported syntax like optional chaining and import.meta

Here's roughly what our config looks like:

export default [
  /* sw build */
  {
    /* Normally, HTML is not a valid entrypoint for Rollup, but we use @open-wc/rollup-plugin-html to take care of this for us */
    input: 'index.html',
    output: {
      entryFileNames: '[hash].js',
      chunkFileNames: '[hash].js',
      format: 'es',
      dir: 'build'
    },
    plugins: [
      {
        name: 'version',
        load(id) {
          /* replace the version module with a live version from the package.json */
          if (id === versionModulePath) {
            return `export default '${packageJson.version}'`;
          }
        }
      },
      /* Handle bare module specifiers */
      resolve(),
      html(),
      /** 
        - Transform non-widely supported features like optional chaining and nullish coalescing
        - support import.meta
        - Optimise HTM by using babel-plugin-htm
      */
      babel({
        babelHelpers: 'bundled',
        presets: [require.resolve('@babel/preset-modules')],
        plugins: [
          [require.resolve('babel-plugin-htm'), { import: 'preact' }],
          [require.resolve('babel-plugin-bundled-import-meta'), { importStyle: 'baseURI' }],
          require.resolve('@babel/plugin-proposal-optional-chaining'),
          require.resolve('@babel/plugin-proposal-nullish-coalescing-operator')
        ]
      }),
      /* Compress our JavaScript */
      terser({ output: { comments: false } }),
      /* Copy some assets, we do this in the buildStart hook of rollup to ensure all assets are there when we run rollup-plugin-workbox */
      copy({
        hook: 'buildStart',
        targets: [{ src: 'data/**/*', dest: 'build/data' }],
        flatten: false
      }),
      copy({
        hook: 'buildStart',
        targets: [
          { src: 'manifest.json', dest: 'build/' },
        ],
        flatten: false
      }),
      /* Create our service worker */
      injectManifest({
        swSrc: 'build/sw.js',
        swDest: 'build/sw.js',
        globDirectory: 'build/',
        mode: 'production'
      }),
      /* Apply the service worker registration to our index.html */
      applySwRegistration({
        htmlFileName: 'index.html'
      }),
    ]
  }
];

State management

State management is a widely discussed topic in the frontend world, and there are many, many solutions out there. We decided to go with a very minimal singleton pattern that leverages EventTarget, here's what it looks like:

class TotalsService extends EventTarget {
  async getTotals(forceRefresh) {
    if (forceRefresh || !this.__totals) {
      this.__totals = fetch(new URL('../../data/totals.json', import.meta.url)).then((r) => r.json());
      await this.__totals;
      this.dispatchEvent(new Event('change'));
    }
    return this.__totals;
  }
}

export const totalsService = new TotalsService();

We can then react to changes like so:

totalsService.addEventListener('change', () => {/* update state when data has changed */})

If you're interested in learning more about this pattern, Lars is working on a more dedicated blog post on this subject, so keep an eye out at the open-wc space, or the open-wc twitter 😉

PWA

I've been excited about PWAs ever since I discovered them, and there are many exciting efforts and developments going on in the PWA scene, like Project Fugu, and PWAbuilder.

We wanted Project Lockdown to be a PWA, for several reasons:

Performance
Offline support
Native-like app experience
Development speed/maintenance

Another benefit of going with a PWA is that we were not limited or restricted to an app/play store. At some point during the project, we heard reports that Apple and Google started removing search results for COVID related apps. While trying to protect people against fake news, and apps trying to make a profit off of major tragic events is a noble cause, many legitimate apps that rely on WHO data directly also got caught up in these bans.

I'm very pleased to say that the process of PWA-ifying Project Lockdown very much felt like a plug-n-play experience. Here are some of the tools we used:

Workbox v5 via rollup-plugin-workbox - for building our service worker
pwa-asset-generator - for generating all the specific icons/assets for multiple devices
pwa-helpers - for utilities
pwa-helper-components - web components for updating your PWA, installing your PWA, dark mode, and listening for service worker updates

These tools took a lot of the heavy lifting out of our hands, and made for a very smooth process of PWA-ifying the project.

Workbox

Building our service worker was a breeze with Workbox v5 and rollup-plugin-workbox.

injectManifest({
  swSrc: 'build/sw.js',
  swDest: 'build/sw.js',
  globDirectory: 'build/',
  mode: 'production',
}),

Since we wanted some fine grained control over our service worker, we went with injectManifest rather than generateSW. Putting together the service worker felt a lot like: Pick an asset or request, decide a strategy. Is this asset or request crucial? Do we need fresh data for this? Or can we load this from the cache-first? And based on those decisions, we could very easily include routes for them in our service worker. Setting up things like strategies for google fonts, and SPA based routing was very easy due to Workbox's common and advanced strategies. Plug and play!

One of the concerns we ran into is that there are a lot of countries in the world. Clicking on a country, and opening the 'country details dialog' will fetch some data to display in the dialog. If a user is curious about a lot of countries, and clicks many of them, their storage may fill up very, very quickly. Thats why we make use of the Expiration plugin and settings like maxEntries and purgeOnQuotaError. It's important to be mindful of the users device storage!

With this setup, it could of course also happen that a user has no network connection, clicks a country, but there is no data for that country in the cache. Instead of crashing or showing errors, we show the user a friendly message instead:

Implementing the service worker with Workbox felt like a great experience overall, but I do have some criticisms:

Instead of depending on the cdn import for workbox, we felt more comfortable including bundling the service worker in our existing build process. This allowed us to write ES module imports, and have everything integrated in our process. The downside of going this way was that sometimes it was very hard to figure out which workbox module exports what, and the documentation at times felt a bit scattered.

Another thing we bumped into is that workbox makes use of process.env.NODE_ENV variables. Being a buildless kind of guy, I was a little disappointed to see these kind of variables being used for frontend-based libraries (Looking at you, redux!). Fortunately this problem was easily solved using @rollup/plugin-replace.

However, in all fairness and in defense of Workbox, this is the price I pay for deciding to bundle the service worker myself, and Workbox is a great tool.

Building the service worker

Building our service worker with rollup was relatively simple to set up as well, here's what our config for the service worker looks like:

export default [
  {
    input: 'sw.js',
    output: {
      format: 'es',
      dir: 'build'
    },
    plugins: [
      replace({ 'process.env.NODE_ENV': '"production"' }),
      resolve(),
      terser({ output: { comments: false } }),
    ]
  },
  // ...
]

On top of that, we wrote a simple inline rollup plugin to only append the service worker registration code to the index.html during build time, so we don't have to worry about it during regular development:

  {
    name: 'rollup-plugin-apply-service-worker-registration',
    generateBundle(_, bundle) {
      let htmlSource = bundle['index.html'].source;
      htmlSource = applyServiceWorkerRegistration(htmlSource);
    },
  }

The applyServiceWorkerRegistration function takes a html file as string, parses it to an AST using parse5, queries the AST for the document body, creates a new <script> tag that contains the service worker registration code, appends it to the end of the document body, and returns the new html file as a string again. Here's the code:

const { parse, serialize } = require('parse5');
const Terser = require('terser');
const { createScript } = require('@open-wc/building-utils');
const { append, predicates, query } = require('@open-wc/building-utils/dom5-fork');

function applyServiceWorkerRegistration(htmlString) {
  const documentAst = parse(htmlString);
  const body = query(documentAst, predicates.hasTagName('body'));
  const swRegistration = createScript(
    {},
    Terser.minify(`
    if ('serviceWorker' in navigator) {
      window.addEventListener('load', function() {
        navigator.serviceWorker
          .register('./sw.js')
          .then(function() {
            console.log('ServiceWorker registered.');
          })
          .catch(function(err) {
            console.log('ServiceWorker registration failed: ', err);
          });
      });
    }
  `).code,
  );

  append(body, swRegistration);
  return serialize(documentAst);
};

For my (and hopefully your) convenience, I've published this rollup plugin on NPM as well. The published version has some more configuration options, like a custom html file name, custom service worker scope, and custom service worker file name. We also use it in our building configuration at open-wc.

Additionally, we maintain a version number that keeps track of the current version of our PWA, mainly for debugging, but in the future we may want to use the version number to fetch a CHANGELOG, and when a new version of the PWA is available, we can display a "What's new?" that lists which updates the new version would contain. Updating this version variable is done automatically with a very minimal inline rollup plugin as well:

In our rollup config, we import the package.json to be able to read the latest version number, as well as the path to a JS file that exports the version number:

./rollup.config.js:

import packageJson from './package.json';
const versionModulePath = require.resolve('./src/version.js');

./src/version.js:

export default 'dev';

Then, in our plugins array in the rollup config we do the following:

{
  name: 'version',
  load(id) {
    // replace the version module with a live version from the package.json
    if (id === versionModulePath) {
      return `export default '${packageJson.version}'`;
    }
  }
},

We can then use it in our App.js like so:

import version from './version';
// During development will output 'dev', in production will always show the latest version of the package.json:
console.log(`🌐 Project Lockdown, version: ${version}`);

Pwa asset generator

We use pwa-asset-generator for, well, generating our pwa assets. Pwa asset generator is a great tool made by Önder Ceylan that allows you to very flexibly generate your assets, and automatically append them to your index.html and manifest.json. Very nicely, it even handles assets for darkmode as well, as well as apple specific icons/splash screens.

Here's what our script looks like:

"prepare-pwa-assets": "./node_modules/.bin/pwa-asset-generator Assets/logo.png src/assets/pwa --manifest manifest.json --index index.html && ./node_modules/.bin/pwa-asset-generator Assets/logo-dark.png src/assets/pwa --dark-mode --background '#303136' --splash-only --index index.html && npm run format:index",

To regenerate our assets, we can just run: npm run prepare-pwa-assets.

The only problem we had with pwa-asset-generator is that it adds a bunch of weird new lines to your index.html. And while this is easily fixed by running prettier after running pwa-asset-generator, it is a bit of nuisance to need this extra step. Alltogether though, I would very much recommend using pwa-asset-generator, its very simple to set up, and it has excellent documentation.

Lazyloading and routing

In order to improve performance and first pageload, we lazyload Leaflet, as well as the Dialog component. This way loading Leaflet doesn't block our first render, and the user will see some UI on their screen faster. We also could have opted to lazyload some of the items in the menu, but most of the content in there is only a few lines of HTML.

Additionally, we check if a new service worker is available on SPA route navigations. That way, if a user has our page open indefinitely and never refreshes, they're still able to receive updates when they make use of the app. We'll go more into that in the updating section.

Installing

For the install experience, we used a web component that I created for an earlier project in pwa-helper-components. All we had to do was install the component:

npm i -S pwa-helper-components

Import the component:

import 'pwa-helper-components/pwa-install-button.js';

and drop it in my markdown like so:

<pwa-install-button>
  <button class="my-button-styles">Install app</button>
</pwa-install-button>

The button will only show up if the PWA is installable, and made adding an install-flow plug and play! (Note: This does require having a valid service worker registered, and a manifest.json to be present)

PWABuilder is a great project that you should check out, which also ships a pwa-install web component. The reason we didn't go with this is because it's built with LitElement, and while LitElement is a great library in and of itself (and only 7kb min+gzip!) and I very much recommend you try it, we weren't comfortable with adding LitElement to our bundlesize only for one pwa-install component. The <pwa-install-button> component we did use is written with native web components, and roughly 32 lines of code (unminified).

Updating

If there is a new version of your PWA, you'll want a way that lets the user update to the latest version of your app. In order to achieve this, we used another web component; the <pwa-update-available> component, also from pwa-helper-components. Again, all I had to do was install the library, and drop the component in my markdown like so:

<pwa-update-available>
  <button class="my-button-styles">Update app</button>
</pwa-update-available>

And the component will just become visible whenever an update is available.

But, this is where we ran into a problem. The <pwa-update-available> component is conditionally rendered in the 'settings' section of the menu. Which means the user may not know an update is available until they decide to click on the 'settings' menu button. It would be nice to display some sort of user friendly indicator to let the user know an update is available. We specifically didn't want to go with a 'toast'-like pattern, because many users (myself included) experience this to be annoying. We solved this instead by using the following pattern, inspired by Jad Joubran and his talk Secrets of Native-like PWA's (timestamp: 25:49).

In order to achieve this, we extracted some logic from the <pwa-update-available> component, and turned it into a simple, small helper function that executes a callback whenever a new service worker is detected:

function addPwaUpdateListener(callback) {
  let newWorker;
  if ('serviceWorker' in navigator) {
    navigator.serviceWorker.getRegistration().then(reg => {
      if (reg) {
        reg.addEventListener('updatefound', () => {
          newWorker = reg.installing;
          newWorker.addEventListener('statechange', () => {
            if (newWorker.state === 'installed' && navigator.serviceWorker.controller) {
              callback(true);
            }
          });
        });
        if (reg.waiting && navigator.serviceWorker.controller) {
          callback(true);
          newWorker = reg.waiting;
        }
      }
    });
  }
}

We can then use it in our app-level component like so:

addPwaUpdateListener(updateAvailable => {
  this.setState({
    updateAvailable
  });
});

And toggle a subtle(!) indicator that a new update is available:

Another consideration we had is that if a user has our app open in a tab indefinitely, they may never be made aware of any updates to our app. Normally, the browser checks for updates automatically after navigations and functional events (like sync and push), but since our app is essentially a Single Page App (SPA), we simply solved this by hooking up a check to our client side SPA routing:

if ('serviceWorker' in navigator) {
  navigator.serviceWorker.getRegistration().then(registration => {
    if (registration) {
      registration.update();
    }
  });
}

Now, if a user has our app open indefinitely, they will still be able to receive new updates to the app simply by making use of the app.

Dark mode

One feature that almost all apps/PWAs have nowadays is dark mode. Honestly, it was somewhat of a nice-to-have type of feature that I thought would be implemented easily, and without a lot of work/hassle, but I have to admit I spent longer on this feature than I care to admit 😅

What I wanted to achieve was:

On initial pageload, and if there is no manual preference set yet, respect the users system preference
Then, if a user decides to toggle to dark(/light) mode, store that preference, and use that preference from now on, even on subsequent visits, because the user manually opted in.

We achieved this by registering a media query matcher on pageload to set dark mode or not. We were already using the installMediaQueryMatcher from pwa-helpers, but it's a three liner to implement this watcher yourself:

export const installMediaQueryMatcher = (mediaQuery, callback) => {
  const mediaMatcher = window.matchMedia(mediaQuery);
  mediaMatcher.addListener((e) => callback(e.matches));
  callback(mediaMatcher.matches);
};

The following logic is what took me longer to write than I thought it would, but here we are:

installMediaQueryWatcher(`(prefers-color-scheme: dark)`, preference => {
  const localStorageDarkmode = localStorage.getItem('darkmode');
  const darkmodePreferenceExists = localStorageDarkmode !== null;
  const darkMode = localStorageDarkmode === 'true';
  const html = document.getElementsByTagName('html')[0].classList;

  /* on initial pageload and no manual user preference, decide darkmode on users system preference */
  if (!darkmodePreferenceExists) {
    if (preference) {
      localStorage.setItem('darkmode', 'true');
      html.add('dark');
    } else {
      localStorage.setItem('darkmode', 'false');
      html.remove('dark');
    }
  } else {
    /* if the user has manually chosen a preference, prioritise that instead */
    if (darkMode) {
      html.add('dark');
    }
  }
});

And then in the settings menu, we have a button where the user can manually toggle between dark/light mode:

function toggleDarkmode() {
  const html = document.getElementsByTagName('html')[0].classList;

  if (html.contains('dark')) {
    html.remove('dark');
    localStorage.setItem('darkmode', 'false');
  } else {
    html.add('dark');
    localStorage.setItem('darkmode', 'true');
  }
}

I've extracted this logic to a simple <pwa-dark-mode> web component to toggle the dark class and persist it in localStorage, and a helper function to register the media query listener and it's logic and published it in pwa-helper-components, too.

That means all the user has to do is write some css, use the web component to wrap a native <button> element, and register the helper to enable dark mode:

<html>
  <head>
    <style>
      :root {
        --my-text-col: black;
        --my-bg-col: white;
      }

      .dark {
        --my-text-col: white;
        --my-bg-col: black;
      }

      body {
        background-color: var(--my-bg-col);
        color: var(--my-text-col);
      }
    </style>
  </head>
  <body>
    <p>Hello world!</p>
    <pwa-dark-mode></pwa-dark-mode>
    <script type="module">
      import { installDarkModeHandler } from 'pwa-helper-components';
      import 'pwa-helper-components/pwa-dark-mode.js';

      installDarkModeHandler();
    </script>
  </body>
</html>

As an extra nice-to-have, we also update the favicon according to the users preference:

Geolocation and Permissions

Our final progressive feature is another small, nice-to-have type of feature, but does contribute to the native-like app feel. We wanted the user to be able to allow geolocation to let the map 'snap' to their location. While developing this feature, the browser's permission pop-up would show up on pageload:

And even while developing this feature, this became very annoying, very quickly. Instead, we opted to only request permissions when a user has shown to be interested in a feature, for example by clicking the 'allow geolocation' button. If they click the button, only then do we show the browsers permission popup.

Deploying

For deploying we use Netlify. I'm a big fan of Netlify, integrating your GitHub project with Netlify is an incredibly seamless experience, and you get a lot; Automatic deployments, preview deployments on pull requests, and its very easy to add your own custom domain name to your project.

Additionally, Netlify graciously offered support for COVID-19 related projects, which we applied for and received as well. This gave us unlimited build minutes, and free plugins like analytics.

Retrospective

Contributing to a COVID-19 project overall was a great experience, but often somewhat of a rollercoaster as well. We had many different people spread out over the world, and over various communication platforms, which sometimes made it hard to reach consensus about things, like design decisions, features, etc. But I'm very happy with the contributions I was able to make to the project, and I look forward to seeing the final project released soon.

In hindsight I do somewhat regret not going with LitElement, because I think it would have been a great showcase on how to make an app with Web Components, and we also didn't end up getting that many contributions as we had hoped initially. But having said that, it was a nice breath of fresh air to work with a different library. Working with Preact and HTM was an absolute pleasure, and I would very much recommend you try them out and build something (lightweight!) cool. I will definitely be using them for more projects myself.

I hope this blog reaches you dear reader, and that maybe it gave you some ideas, or that you learned something new.

Stay safe, and stay healthy 🙏