DEV Community: Sam Vloeberghs

Lessons learned on offline capabilities with service workers using Workbox - The sequel

Sam Vloeberghs — Tue, 28 Jan 2020 14:02:31 +0000

Originally published at https://samvloeberghs.be on May 31, 2019

Target audience

This is a follow-up on my previous article about lessons learned on offline capabilities using Workbox. This article highlights some more findings and explores better/other solutions for the problems described earlier.

What will we discuss in this post?

Better detection for a first-time install of a service worker
Updating the application while it's in use

All the example code used in this post, just as this complete website, is available on Github here as an example project for you to test and try-out.

Better detection for a first-time install of a service worker

In my previous post about service workers and Workbox I used a somewhat shady technique with a refresh to activate the service worker functionality on the first load of the application. This is completely unnecessary and the same outcome can be achieved using other techniques.

If we consider the code to register the service worker from the previous post, the only thing we have to change is send an event to the service worker to claim the clients, instead of reloading the page. This is demonstrated on line 19 and 25 in the following code. The logic to detect a first service worker remained unchanged.

// service-worker-registration.ts
// Check that service workers are available
if ('serviceWorker' in navigator) {
  // Use the window load event to keep the page load performant
  window.addEventListener('load', () => {
    navigator.serviceWorker.register('/service-worker.js').then(registration => {
      if (navigator.serviceWorker.controller) {
        // let the application know our service worker is ready
        window['serviceWorkerReady'] = true;
        window.dispatchEvent(new CustomEvent('service-worker-ready'));
      }

      // A wild service worker has appeared in reg.installing and maybe in waiting!
      const newWorker = registration.installing;
      const waitingWoker = registration.waiting;

      if (newWorker) {
        if (newWorker.state === 'activated' && !waitingWoker) {
          // - no more window.location.reload();
          newWorker.postMessage({ type: 'CLIENTS_CLAIM' });
        }
        newWorker.addEventListener('statechange', () => {
          // newWorker.state has changed
          if (newWorker.state === 'activated' && !waitingWoker) {
            // - no more window.location.reload();
            newWorker.postMessage({ type: 'CLIENTS_CLAIM' });
          }
        });
      }
    })
    .catch(err => {
      console.log('service worker could not be registered', err);
    });
  });
}

Important: Please don't use the above code in your application, as it's a bad implementation! A better implementation using Workbox is discussed below.

For this to work we need to make our service worker react to messages and claim the clients for this installation of the service worker. We do this by registering an event listener on the message event:

// service-worker.js
// ... other sw functionality

self.addEventListener('message', event => {
  if (event.data && event.data.type === 'SKIP_WAITING') {
    self.skipWaiting();
  }
  if (event.data && event.data.type === 'CLIENTS_CLAIM') {
    self.clients.claim();
  }
});

If a service worker is installed, but not controlling the page yet, an event that matches the type CLIENTS_CLAIM will claim the clients and activate the service worker for those clients.

Using Workbox Window to solve the same problem

With Workbox Window we can listen for the activated event on the service worker registration. This events holds a property isUpdate that indicates if a previous version was controlling the page. If not, we can safely assume this is a first time install.

// service-worker-registration-workbox-window.ts
import { Workbox } from 'workbox-window';

const wb = new Workbox('/service-worker.js', {});

wb.addEventListener('activated', async event => {
  // `event.isUpdate` will be true if another version of the service
  // worker was controlling the page when this version was registered.
  if (!event.isUpdate) {
    // If your service worker is configured to precache assets, those
    // assets should all be available now.
    // So send a message telling the service worker to claim the clients
    // This is the first install, so the functionality of the app
    // should meet the functionality of the service worker!
    wb.messageSW({ type: 'CLIENTS_CLAIM' });
  }
});

wb.register();

Similar to our previous example, without Workbox Window, our service worker needs to be listening to messages and act accordingly on the message of type CLIENTS_CLAIM. wb.messageSW is a Workbox wrapper function that provides a message and reply mechanism using a MessageChannel but basically just sends a message to the service worker using postMessage.

Visual representation

The animated GIF shown above highlights the steps our code takes in registering a service worker for the first time. Let's go over all the steps shown in the animation:

Our application creates a new Workbox instance that registers the service worker.
The service worker installs and proceeds trough the waiting process to the activated process. At this point the service worker is not yet controlling the application (or client).
Through the workbox event system we receive the activated event. We check based on this event if the activation is an update or not. In this case it's not.
The application logic sends an event trough the messasging system of Workbox, usingpostMessage, an event CLIENTS_CLAIM instructing the service worker to take control over the active clients.
The service worker receives this event and acts accordingly, executing clients.claim().

Updating the application while it's in use

With PWA's installed on the homescreen and opened as native-like applications it's possible that those applications are running for a long time. Updating a new version of the application and installing a new service worker could potentially be extended or postponed for a long time, until the user closes and reopens the application, or refreshes the application.

It's even possible to disable the refresh by pulling down the application completely. This could be done if the functionality requires it. In case of for example a full screen map, pinching, zooming and other gestures could lead to an unexpected refresh.

In those cases the only way of updating seems to be closing and reopening the application. Fortunately it's not. There are several ways of doing an in-application update. By means of user-intent, when the application becomes visible again or by setting up an interval. Let's explore these 3 options.

Update application by user-intent

The simplest way of asking for an update to the application is by letting the user click a button. So we are gonna take this example to show how to update the application while running it. The other options and examples are just defining a different approach or intent to update.

To be able to update our application while running it we need to make sure that we keep track of the original service worker registration. Checking for a new version and performing an update is then as simple as just calling swRegistration.update(). Please consider the following ServiceWorkerService:

// app-update-user-intent.ts
import { Injectable } from '@angular/core';
import { Workbox } from 'workbox-window';

@Injectable({
  providedIn: 'root',
})
export class ServiceWorkerService {

  private swRegistration: ServiceWorkerRegistration;

  constructor() {
    this.registerServiceWorker();
  }

  public async checkForUpdate() {
    try {
      console.log('updating sw');
      return await this.swRegistration.update();
    } catch (err) {
      console.log('sw.js could not be updated', err);
    }
  }

  private async registerServiceWorker() {
    const wb = new Workbox('/sw.js', {});

    try {
      this.swRegistration = await wb.register();
    } catch (e) {
      console.log('error registering service worker', e);
    }
  }

}

This Angular example service, which can be easily ported to any framework, registers the service worker as soon as the service gets instantiated. The register method call on the Workbox instance gives us a native ServiceWorkerRegistration that we save in the serviceWorkerRegistration property.

Now any button in the application can be coupled to check for an update of the application by calling the checkForUpdate method. This method uses the previously saved swRegistration and calls update on it.

The update method tries to update the service worker by fetching the registration's URL. In case there is a byte-by-byte difference to the current service worker, the new service worker will ge installed. After its potential new resources have been cached and the logic has been loaded it will eventually go in a waiting state.

Handling the new but waiting service worker

Whenever our application gets a hold of a new but waiting service worker, we can safely assume that our application can be updated. So we show a message to the user that an action, like refreshing the page, can be invoked to load the new functionality. By using this approach we keep the responsibility with the user to load the new version.

Important*: Immediately activating the new service worker, by using skipWaiting() and/or clients.claim() for all the clients when it becomes waiting can potentially break your application. The functionality from the new service worker could not be compatible with the currently loaded old version of your application. Read more about this consideration in my previous blogpost.

Before we dive further into the code and details for this example, we will go trough the most simple use-case that describes the desired functionality:

The user already visited our application before, so there is an active service worker for our application in the browser.
During usage of the application, we deploy a new version. The user clicks a button to check and finds the new version of our application. The new service worker installs and goes into waiting state.
Our application is notified of this new version and shows the user an update button to express his further intent. Meanwhile the new service worker stays in waiting state.
When the user clicks the update button, our new service worker is informed and skips the waiting phase. As soon as the new service worker finishes his caching work and install process it becomes activated.
The application gets notified and refreshes to load the complete new version of the application. If we wouldn't reload we might have the situation that the functionality of our new service worker is not compatible with our currently loaded application.

Listening for a new service worker

Listening for a new waiting service worker using Workbox when we try an update from within the application is a little bit unexpected and different from handling a new waiting service worker after a refresh. For these situations Workbox dispatches lifecycle events like externalinstalling when updates are arriving from "external" service workers. Any version which is not the current Workbox instance or service worker registration can be considered as external.

// app-update-external.ts
import { Injectable } from '@angular/core';
import { BehaviorSubject, Observable } from 'rxjs';
import { filter, first } from 'rxjs/operators';
import { Workbox } from 'workbox-window';

declare const window: any;

@Injectable({
  providedIn: 'root',
})
export class EnvironmentService {

  public newVersionAvailable$: Observable;

  private readonly newVersionAvailable = new BehaviorSubject(false);
  private readonly applicationUpdateRequested = new BehaviorSubject(false);
  // check every 4h if a new version is available
  private readonly swUpdateInterval = 4 * 60 * 60 * 1000;

  private swRegistration: ServiceWorkerRegistration;

  constructor() {
    this.newVersionAvailable$ = this.newVersionAvailable.asObservable();
    this.registerServiceWorker();
  }

  public update() {
    this.applicationUpdateRequested.next(true);
  }

  public async checkForUpdate() {
    // ...
  }

  private async registerServiceWorker() {
    const wb = new Workbox('/sw.js', {});

    // ...

    // we use this waiting listener to handle the update we do
    // based on an interval, user intent or visibility change
    // in this case another service worker became waiting
    wb.addEventListener('externalwaiting', event => {
      // inform any functionality that is interested in this update
      this.newVersionAvailable.next(true);

      // listen to application update requests
      this.applicationUpdateRequested.pipe(
        filter((applicationUpdateRequested) => applicationUpdateRequested),
        first(),
      ).subscribe(_ => {
        // Send a message telling the service worker to skip waiting and
        // become active. We use event.sw.postMessage, and not wb.messageSw,
        // because we want to message the waiting SW and not the currently
        // active service worker
        event.sw.postMessage({ type: 'SKIP_WAITING' });
      });
    });

    // the other service worker became actived!
    wb.addEventListener('externalactivated', () => {
      // If your service worker is configured to precache assets, those
      // assets should all be available now.
      // This activation was on request of the user, so let's finally reload the page
      window.location.reload();
    });

    try {
      this.swRegistration = await wb.register();
    } catch (e) {
      console.log('error registering service worker', e);
    }
  }

}

Let's map our use case written out above onto the code:

We assume an active service worker is available while running a checkForUpdate (line 31) and a new version of the application is found and installing.
Our event listener for externalwaiting (line 43) receives a new event and updates the application that a new version is available. This value is exposed as an Observable in the application, so we can show a notification.
The event listener subscribes and starts listening to the user intent to update the application to the new version (line 48-51). When the user gives his go, an event SKIP_WAITING is sent to the service worker, instructing it to become activated (line 56).
The event listener externalactivated receives a new event, indicating that the new service worker became activated (line 61). Now we can reload to activate the complete new application (line 65)!

Important: The full code for handling all these use cases combined, including the following strategies, is a big pile of code which you can find here.

Visual representation

The animated GIF shown above highlights the steps our code takes when the application is responsible for checking for a new version of the application on the server. As mentioned this can be by user-intent, when the application becomes visible again or by setting up an interval. Let's go trough the animation step-by-step:

Our application creates a new Workbox instance that registers the service worker. The application logic checks if there is already a service worker controlling the page. In this situation, this is the case so we let our application logic know.
By means of user-intent, visibilitychange or the interval our application performs a check, checking the server for a potential new version of the service worker and application.
The new service worker installs and goes into the waiting state.
Our Workbox instance picks up this change via the externalwaiting event listener and updates the application, letting it know there is a new version available. This allows the application to show an update button to the user, giving him the option to load the new version of the application.
The user express his intent and asks for an update of the application. The application logic sends a message SKIP_WAITING to the new service worker, instructing it to advance to the activated state.
The new service worker becomes activated and updates the application trough the externalactivated event, fired by the Workbox instance. 7.This last step finally instructs our application to do a programmatic reload, activating the newly cached version of our application.

Update application when it becomes visible again

Another strategy to try and update your application can be based on the visibility of your application. As soon as your functionality is loaded, we can start listening to the visibilitychange event on the document as shown in the highlighted code below.

// app-update-visible.ts
import { Injectable } from '@angular/core';
import { Workbox } from 'workbox-window';
import { fromEvent } from 'rxjs';

@Injectable({
  providedIn: 'root',
})
export class ServiceWorkerService {

  // ...

  constructor() {
    this.registerServiceWorker();
    this.registerVisibileChangeListener();
  }

  // ...

  private registerVisibileChangeListener() {
    fromEvent(document, 'visibilitychange').subscribe(() => {
      this.checkForUpdate();
    });
  }

}

Now every time your application gets hidden, for example when opening a new tab in your browser or opening a new application on your smartphone, and you choose to make the tab visible again, or you reopen the application, the app will try to update the service worker.

Update application by setting up an interval

One more other strategy is requesting an update on an interval, for example every 4 hours. This way the user will be presented with the update possibility automatically if a new version has been deployed recently. The extra logic / code for this is a simple 3-liner as highlighted below:

// app-update-interval.ts
import { Injectable } from '@angular/core';
import { Workbox } from 'workbox-window';

@Injectable({
  providedIn: 'root',
})
export class ServiceWorkerService {

  // check every 4h if a new version is available
  private readonly swUpdateInterval = 4 * 60 * 60 * 1000

  // ...

  private async registerServiceWorker() {
    const wb = new Workbox('/sw.js', {});

    try {
      this.swRegistration = await wb.register();

      setInterval(async () => {
        this.checkForUpdate();
      }, this.swUpdateInterval);
    } catch (e) {
      console.log('error registering service worker', e);
    }
  }

}

Conclusion

Today we learned a better way of controlling the application with our service worker on the very first load. Instead of refreshing we use the clients.claim() method, but in a way that we won't potentially break our application.

Furthermore, we have explored a few ideas on how to update our application from within the application. This is particularly useful if our applications are long-living and we want to make sure our users get the latest functionalities.

Workbox and service workers in general still have many unknown capabilities that I'm learning about. But while we keep developing we explore new ideas and ways of optimizing our code. I think I will be writing more about service workers in the future :)

Special thanks to

for reviewing this post and providing valuable and much-appreciated feedback! I'm open for any other feedback so please let me know!

Angular v9 & Universal: SSR and prerendering out of the box!

Sam Vloeberghs — Sun, 26 Jan 2020 16:40:31 +0000

Originally published at https://samvloeberghs.be on January 5, 2020

Target audience

This article and guide is intented to help anybody using Angular v9 getting started with server-side-rendering (SSR) and prerendering their application. Please be advised that these instructions are only working for new or updated apps using Angular v9. Getting the same result for Angular v2-v8 requires more and custom setup.

Example project

For the example project we are using a bare minimum angular-cli generated project with routing enabled. A home (/) and about (/about) route and their components were generated as well. By using a lazy-loaded news module, an extra dynamic overview (/news) and newsdetail (/news/:id) route were configured as well.

The data for this overview and detail page is a simple JSON data object we load from the static assets folder (/assets/news.json) and looks like this:

// /assets/news.json
[
  {
    "id": 1,
    "title": "Newsitem #1",
    "short": "Lorem ipsum dolor sit amet, ...",
  },
  {
    "id": 2,
    "title": "Newsitem #2",
    "short": "Lorem ipsum dolor sit amet, ..."
  }
]

The following animation shows how our application behaves using the routes and loading the dynamic data. The full source code for this basic example application can be found here.

Adding Server-side-rendering (SSR) to your application

Normally your Angular app is only rendered as soon as your browser loads it. In this case, the only responsibility of the webserver is serving the static files of your Angular app (everything that is in the dist folder after a successful build).

With SSR, the specific route of your application, for example /about, is completely rendered on the server, just as it would render in your browser. This allows search engines like Google and social-media platforms like Facebook to index and show previews of the pages of your application, because the full HTML is available from the initial load from the server, without JavaScript required.

Getting started

To get started all you need to do is add the @nguniversal/express-engine package using the angular-cli:

ng add @nguniversal/express-engine@next
# as soon as v9 is released you can drop the "@next"

The ng add command updates your application by adding the necessary files and updating the angular.json configuration file. There are 3 new configurations added to the architect section: server, serve-ssr and prerender. All those 3 configurations have their own purpose and together they allow us to achieve the required results. Let's walk trough the changes.

// updated angular.json
{
    "$schema": "./node_modules/@angular/cli/lib/config/schema.json",
    "version": 1,
    "newProjectRoot": "projects",
    "projects": {
    "ng-v9-universal": {
      "..": "..",
      "architect": {
        "..": "..",
        "server": {
          "builder": "@angular-devkit/build-angular:server",
          "options": {
            "outputPath": "dist/ng-v9-universal/server",
            "main": "server.ts",
            "tsConfig": "tsconfig.server.json"
          },
          "configurations": {
            "production": {
              "..": ".."
            }
          }
        },
        "serve-ssr": {
          "builder": "@nguniversal/builders:ssr-dev-server",
          "options": {
            "browserTarget": "ng-v9-universal:build",
            "serverTarget": "ng-v9-universal:server"
          },
          "configurations": {
            "production": {
              "browserTarget": "ng-v9-universal:build:production",
              "serverTarget": "ng-v9-universal:server:production"
            }
          }
        },
        "prerender": {
          "builder": "@nguniversal/builders:prerender",
          "options": {
            "browserTarget": "ng-v9-universal:build:production",
            "serverTarget": "ng-v9-universal:server:production",
            "routes": []
          },
          "configurations": {
            "production": {}
          }
        }
      }
    }
  }
  "..": ".."
}

The `server` run option

A server is required to run the Angular build on your system, or the back-end, therefore the server.ts file is added to the root of your project. This TypeScript file contains all the necessary setup to render your routes and serve the static assets of your application. As you can see on line 13-15, the output path for this server is defined, as well as a tsConfig that defines how to build the server TypeScript code. Just as with the browser build we can also define the production specific environment variables.

Building this server is as easy as building your Angular application. Just run the following command in your terminal, and the server will be build into dist/ng-v9-universal/server.

npm run build:ssr

The `serve-ssr` run option

While developing your application you'll probably just run ng serve. This command will setup a dev-server for you with hot-reloading and everything else that will provide you with a nice developer experience.

Testing the server-side-rendering part can be done while developing as well. To do this, just run the application as ng run ng-v9-universal:serve-ssr and the server source files will be watched next to the application source files. Changing the code of the application or the server will automatically restart the application because of live-reloading.

Now if you reload any page, for example the news overview (/news), the initial payload will contain the fully rendered HTML for the news overview, including the dynamic data.

The `prerender` run option

A final addition is the possibility to prerender the routes of your application. For this to work the prerender builder guesses static routes using guess-parser, build by @mgechev. The routing modules of your application are analyzed and the routes found are prerendered.

To prerender the routes of your application, just run:

npm run prerender

Rendering more (dynamic) routes

As shown in the angular.json config file at line 41, it is possible to list other known routes of the application. In our case for example we could list the /news, /news/1 and /news/2 routes. This will instruct the builder to also prerender those extra routes. Especially for lazy-loaded modules this might be important, as these are not so easily guessed by guess-parser.

To render more dynamic routes, like the news detail pages in our sample application, we need some more logic. Basically you need to find a way to let the prerender know what routes to prerender. This can be done by, for example, defining a list-routes.js script that will list all the routes in a text file. This file can then by passed to the prerender script as follows:

npm run prerender --routesFile routes.txt

An example of a script to list up all the required routes is shown below. All the routes listed in this text file are added to the routes already defined in the routes array of the angular.json config file before prerendering.

// scripts/list-routes.js
const fs = require('fs');
const axios = require('axios');
const endOfLine = require('os').EOL;
const newsDataPath = 'http://localhost:4200/assets/news.json';
const routesFile = './routes.txt';

axios.get(newsDataPath).then(res => {
  const routes = [];
  res.data.forEach(newsitem => {
    routes.push('news/' + newsitem.id);
  });
  fs.writeFileSync(routesFile, routes.join(endOfLine), 'utf8');
}).catch(e => console.log(e));

The scripts in our package.json now look like this:

// updated package.json
{
  "name": "ng-v9-universal",
  "version": "0.0.0",
  "scripts": {
    "..", "..",
    "list-routes": "node ./scripts/list-routes.js",
    "prerender": "npm run list-routes && ng run ng-v9-universal:prerender --routesFile routes.txt"
  },
  "..": ".."
}

Important: While running the npm run prerender command, be sure to have the application running as well. The static file /assets/news.json needs to be available for the application to prerender all routes! You can do this by just running ng serve in another terminal.

Conclusion

I have started this blog with Angular v2 and Universal and back in 2016 it was not easy getting it set up. Universal with Angular v9 has improved developer experience a lot and implementing it is now just a matter of following clearly defined steps.

Testing your application and the server-side rendered version of it is now available as one command. Just run npm run dev:ssr to get going!

Prerendering your static routes is easy, except if you are using lazy loaded routes, these are not easy to guess by guess-parser. You still need a way to render all your routes. This can be done by listing the routes in a file and feeding it to the builder by using the --routesFile option. How you get to know the list of routes is up to you.

Custom plugins for Scully - Angular Static Site Generator

Sam Vloeberghs — Sun, 26 Jan 2020 11:18:40 +0000

Originally published at https://samvloeberghs.be on January 20, 2020

Target audience

Scully, the Static Site Generator for Angular, comes with a plugin system that allows us to control which pages to prerender of our Angular application and how to prerender them. In this article we will learn how to setup custom plugins for your specific use cases. If you care about the performance of your Angular application or website, please continue reading.

What is Scully?

Scully is a static site generator, or SSG, for Angular. It takes your full-blown Angular application and transforms the stable state of all the publicly available pages into their static HTML snapshot. You can deploy these HTML pages to a static webserver, with or without the original Angular application, and your application will load instantly. Even without JS enabled, because all the content is available in the original HTML returned from the server.

This has some important consequences and benefits. The most important one is that the content of your page is available instantly. After your HTML page has loaded, the application can still bootstrap as the Angular app, giving you the full SPA functionality. But meanwhile the user will not be blocked to read or use the page until the Angular app has started up.

Getting started with Scully

Getting started with Scully is easy. The documentation on the Github project will get you going with setting up your first project. At the bottom of the article you can find more links and resources about setting up Scully.

Support for plugins

Scully can generate your pages because of a discovery process that finds the available routes of your application by scanning all the routing modules. Some routes are easy to discover, like for example the /about route. But other routes, for example dynamic routes like /blog/:slug, need to be listed based on known data, coming from for example a DB or API interface.

The lists of available pages for these dynamic routes are built using plugins. There are two types of plugins available, router plugins and renderer plugins. As an example, Scully has a built-in router plugin of type RouteTypes.json that will load a JSON list and use the items in the list as a source for the available pages.

// scully.config.js
const {
  RouteTypes,
} = require('@scullyio/scully');

exports.config = {
  projectRoot: './src/app',
  routes: {
    '/user/:userId': {
      type: RouteTypes.json,
      userId: {
        url: 'https://jsonplaceholder.typicode.com/users',
        property: 'id',
      }
    }
  }
}

If we consider that our JSON source https://jsonplaceholder.typicode.com/users has around 10 users, this plugin will add 10 routes to the list of pages to prerender using our Angular application and Scully. This is done by defining a configuration (line 10-13) that will instruct Scully which property to use, from the objects in the list of the JSON data source, for replacing the dynamic :userId in the route definition (line 8).

And finally the complete list of pages to prerender is generated! Scully keeps track of these list in the /src/assets/scully-routes.json file.

// /src/assets/scully-routes.json
[
  {"route": "/"}, {"route": "/about"},
  ...
  {"route": "/users/1"},
  {"route": "/users/2"},
  ...
  {"route": "/user/10"}
]

The next step is building our own plugins, but let's first shine some light on the example project we will use to explain the details and decide which plugins we need to build.

Example project

The example project is a simple Angular application with a few static routes, like /about, and dynamic routes like /news that lists the available newsitems and /news/:id/:slug that shows the detail of a newsitem. The list of newsitems is retrieved via HttpClient from a static resource /assets/news.json and the newsdetail is available at /assets/news/{:id}.json.

Based on the data of /assets/news.json the list of all routes for our example application looks like the list below. Remember, Scully is generating the complete list.

// /src/assets/scully-routes.json
[
  {"route":"/"},
  {"route":"/about"},
  {"route":"/news"},
  {"route":"/news/archive"},
  {"route":"/news/1/newsitem-1"},
  {"route":"/news/2/newsitem-2"},
  {"route":"/news/5/newsitem-5"},
  {"route":"/news/99/newsitem-99"}
]

Custom router plugin

To be able to generate the list of pages for our news module, that has detail pages in the form of news/:id/:slug with 2 parameters :id and :slug, we need to program a custom router plugin for Scully. A router plugin is easy, you just need to write some code that will generate a list of routes, based on a given configuration. Let's break down the code from our newsPlugin:

// /plugins/newsPlugin.js
const {registerPlugin, routeSplit} = require('@scullyio/scully');
const {httpGetJson} = require('@scullyio/scully/utils/httpGetJson');

const NewsPlugin = 'news';

const newsPlugin = async(route, config) => {
  const list = await httpGetJson(config.url);
  const {createPath} = routeSplit(route);
  const handledRoutes = [];
  for (let item of list) {
    handledRoutes.push({
      route: createPath(item.id, item.slug)
    });
  }
  return handledRoutes;
};

const newsPluginValidator =  async conf => [];

registerPlugin('router', 'news', newsPlugin, newsPluginValidator);
exports.NewsPlugin = NewsPlugin;

The plugin function, defined at line 6, is an async function that expects the route we are handling, in this case the news/:id/:slug route. A config object as the second parameter, that is tight to the route, contains all the information we need to generate the list urls. In this case we only need a url.
At line 8 we generate a createPath function based on the given route. This function will be used to build the final page url, based on the JSON data we retrieved from the url defined in the config object and the original route.
The code that follows in the async function is the actual implementation generating all the urls available for this route.
Finally, we register the plugin to make it available for Scully. The first parameter defines the type of plugin, the second parameter marks the name of the plugin. We also export the NewsPlugin as a type. The last parameter is a validator function, but we will not cover this topic in this blogpost.

Configuring the new plugin

Configuring the new router plugin is easy! In our Scully configuration object, we define our route at line 8-11 and we give it the type of our newsPlugin. The only configuration we needed for our custom plugin was the url to fetch the JSON data from, and we are done!

// scully.config.js
const {RouteTypes} = require('@scullyio/scully');

const {News} = require('./plugins/newsPlugin');

exports.config = {
  projectRoot: './src/app',
  routes: {
    '/news/:id/:slug': {
      type: News,
      url: 'http://localhost:4200/assets/news.json',
    },
  }
};

When running Scully with npm run scully the new router plugin is used and the generated output will be a set of available pages with news details:

// /src/assets/scully-routes.json
[
    ...
    {"route":"/news"},
    {"route":"/news/archive"},
    {"route":"/news/1/newsitem-1"},
    {"route":"/news/2/newsitem-2"},
    {"route":"/news/5/newsitem-5"},
    {"route":"/news/99/newsitem-99"}
]

Custom renderer plugin

Custom renderer plugins can be used to perform some sort of modification on the generated HTML as postRenderers. These postRenderers can be defined as a default, for all rendered pages, or defined on the level of each route. In the latter case, these postRenderers will only be executed on the pages generated for that specific route.

A typical thing to do after generating the static HTML of our pages is to minify the HTML. By removing unnecessary code we can make our pages smaller and faster, which will result in more performant loading times. We can easily minify the HTML by defining a minifyHtml renderer plugin.

// /plugins/minifyHtmlPlugin.js
import { registerPlugin } from '@scullyio/scully';
import { minify, Options } from 'html-minifier';

const defaultMinifyOptions: Options = {
    caseSensitive: true,
    removeComments: true,
    collapseWhitespace: true,
    collapseBooleanAttributes: true,
    removeRedundantAttributes: true,
    useShortDoctype: true,
    removeEmptyAttributes: true,
    minifyCSS: true,
    minifyJS: true,
    removeScriptTypeAttributes: true,
    removeStyleLinkTypeAttributes: true,
    // don't remove attribute quotes, not all social media platforms can parse this over-optimization
    removeAttributeQuotes: false,
    // don't remove optional tags, like the head, not all social media platforms can parse this over-optimization
    removeOptionalTags: false,
};

export const minifyHtmlPlugin = async (html, route) => {
    const minifiedHtml = minify(html, defaultMinifyOptions);
    return Promise.resolve(minifiedHtml);
};

const minifyHtmlPluginValidator = async () => [];
export const MinifyHtml = 'minifyHtml';
registerPlugin('render', MinifyHtml, minifyHtmlPlugin, minifyHtmlPluginValidator);

The first object you see, line 4-20, is the configuration object we use for the HTML minifier. This is the same minifier I used explaining how to implement Better sharing on social media platforms with Angular Universal.
Just as with the custom router plugin for we define an async function. This function takes the HTML and associated route as input. In our case we don't need the route, we just transform the HTML by minifying it and return it as resolved Promise.
Finally, we register the plugin to make it available for Scully. The first parameter defines the type of plugin, the second parameter marks the name of the plugin. We also export the MinifyHtml as a type. We can assign a validator function as well, but this is not in the scope of this blogpost!

Configuring the new plugin

Configuring the new renderer plugin is, again, easy! In our Scully configuration object, we can define a list of postRenderers as a default for all routes using the defaultPostRenderers option. By configuring Scully like this every page that gets statically generated will be minified.

Just keep in mind that if you define a route configuration, the same postRenderers need to be attached to the route configuration using postRenderers. The defaultPostRenderers will not be executed in these cases.

// scully.config.js
const {RouteTypes} = require('@scullyio/scully');

const {MinifyHtml} = require('./plugins/minifyHtmlPlugin');
const {News} = require('./plugins/newsPlugin');

const postRenderers = [MinifyHtml];

exports.config = {
  projectRoot: './src/app',
  defaultPostRenderers: postRenderers,
  routes: {
    '/news/:id/:slug': {
      type: News,
      url: 'http://localhost:4200/assets/news.json',
      postRenderers: postRenderers,
    }
  }
};

Running Scully with npm run scully will now not only generate the correct newsdetail pages, it will also minify the HTML of the generated pages:

Using the scully-minify-html plugin

Because it is a plugin and we can share code via npm it would be silly to rewrite the minifyHtml plugin, except for learning purposes ofcourse. Installing it is as easy as running npm i -s scully-minify-html in your Angular project. This will install the plugin for you. The only thing left is to configure it in your scully.config.js file as follows:

// scully.config.js
const {RouteTypes} = require('@scullyio/scully');
const {MinifyHtml} = require('scully-minify-html');

// custom plugins in ./plugins/*.js
const {News} = require('./plugins/newsPlugin');

const postRenderers = [MinifyHtml];

exports.config = {
  projectRoot: './src/app',
  defaultPostRenderers: postRenderers,
  routes: {
    '/news/:id/:slug': {
      type: News,
      url: 'http://localhost:4200/assets/news.json',
      postRenderers: postRenderers,
    }
  }
};

Let's breakdown the code into little steps explaining what is going on:

First of all we import the required type MinifyHtml from our freshly installed Scully plugin.
At line 7 we define the postRenders list and apply it to the defaultPostRenders (line 11), to run for all routes, and/or configure it per route via postRenderers (line 16). Basically everything we did before implementing the custom renderer plugin.

Conclusion

In this article we have learned how to program and configure custom plugins for Scully. With those custom plugins we can control which pages to generate with router plugins and transform the genenerated HTML with renderer plugins.

We can configure Scully per route or we can use global configuration options to get to our solution. Keep in mind that Scully is still in full development so expect things to change.

Special thanks to

Jeffrey Bosch for reviewing this post and providing valuable and much-appreciated feedback!

Creating a simple memory cache for your Angular Universal website or application

Sam Vloeberghs — Sat, 15 Jun 2019 11:46:36 +0000

Originally published at https://samvloeberghs.be on June 14, 2019

Target audience

This article and guide on setting up a simple memory cache for an Angular Universal website or application is targeted to developers that are optimizing their server for better performance. The goal is to avoid rerendering the page of the application on the server if there is no need to. Instead we will cache the previously generated content and serve it from the cache on subsequent requests.

This concept is of course not limited to Angular applications or any other web-application framework or technology for that matter. It is a widely adopted common practice for server optimization to minimize the load on the processor. The concepts explained can easily be adopted to your framework or technology of choice.

Our example application

To explain the basics and prove the concept we have to consider an example application, like for example my simple blog. This Angular Universal implementation has some static pages containing the content in their components, and dynamic pages, loading blogposts from a JSON datasource into their components.

We have for example the static /about route and a dynamic blogpost route /posts/scroll-to-top-on-angular-router-navigation. Angular Universal is capable of knowing when the application and blogpost is completely generated. As soon as the application is stable, the static HTML is retrieved and send back as the response of the request.

Simple diagram for basic rendering

The diagram above shows the most basic example of the Angular Universal rendering flow. Let's go trough the steps shown:

The browser requests a specific page from your server, for example the dynamic page /posts/scroll-to-top-on-angular-router-navigation.
The server receives the request and renders the Angular application by fetching the content from the JSON datasource.
The renderer outputs the static HTML and the server returns this in the response to the browser.

Code for basic rendering

The code for this basic rendering case is available trough the Angular CLI. The official documentation on the Angular website will get you going really fast. For the sake of this tutorial, the following code is the most important ones for our usecase:

// basic-angular-universal-server-setup.ts

// These are important and needed before anything else
import 'zone.js/dist/zone-node';
import 'reflect-metadata';

import { renderModuleFactory } from '@angular/platform-server';
import { enableProdMode } from '@angular/core';
import * as express from 'express';
import { join } from 'path';
import { readFileSync } from 'fs';

// * NOTE :: leave this as require() since this file is built Dynamically from webpack
const { AppServerModuleNgFactory, LAZY_MODULE_MAP } = require('../../../dist/samvloeberghs/server/main');
const { provideModuleMap } = require('@nguniversal/module-map-ngfactory-loader');

// Faster server renders w/ prod mode
enableProdMode();

// Express server
const app = express();
const PORT = 80;
const DIST_FOLDER = join(process.cwd(), 'server', 'samvloeberghs');

app.set('view engine', 'html');
app.set('views', join(DIST_FOLDER, 'browser'));

// serve static files from /browser
app.get('*.*', express.static(join(DIST_FOLDER, 'browser')));

// Our index.html we'll use as our template
const template = readFileSync(join(DIST_FOLDER, 'browser', 'index.html')).toString();

app.engine('html', (_, options, callback) => {
  renderModuleFactory(AppServerModuleNgFactory, {
    document: template,
    url: options.req.url,
    extraProviders: [
      provideModuleMap(LAZY_MODULE_MAP),
    ],
  }).then(html => {
    callback(null, html);
  });
});

const ngApp = (req, res) => {
  const config = {
    req,
    res,
    preboot: true,
    baseUrl: '/',
    requestUrl: req.originalUrl,
    originUrl: 'https://samvloeberghs.be',
  };

  res.render('index', config);
};

app.get('*', ngApp);

// Start up the Node server
app.listen(PORT, () => {
    console.log(`Node server listening on port ${PORT}`);
});

res.render will return a 200 response automatically, with the static HTML of the server-side rendered page of your Angular application. Later we will learn how we can hook into this.

The rest is all Express framework and Universal configuration. If you want to learn more about the code and configuration shown above, please follow any tutorial to setup your first Universal application in NodeJS. You can also find some relevant links in the "Further reading" section at the end of the blogpost.

Why cache?

The content of our example routes or pages is very unlikely to change often, so there is absolutely no need to render the Angular application each time the page gets requested. We can simply cache the generated HTML on the first time visit, store it in a memory cache and retrieve it when the next visitor hits your website.

As soon as the content changes, we need to clear the cache and let the cache get regenerated. This can happen the first time your updated page gets a visit, but other strategies are possible as well.

A simple memory-cache implementation

Let's extend our previous diagram for basic rendering with Universal with a simple cache mechanism. The diagram above visually shows us what we need to do to, but let's go through it step-by-step:

The browser requests a specific page from your server, for example the dynamic page /posts/scroll-to-top-on-angular-router-navigation.
The server checks if there is a cached version of the page available. The first time we visit the page, there is no cached version.
The server receives the request and renders the Angular application, by fetching the content from the JSON datasource.
The server outputs the static HTML and stores it in the cache.
The server returns the same static HTML as the response to the browser.
The next visitor visits the exact same page.
You could already have seen it coming, the server checks again if there is a cached version available and reads it from the caching registry.
The server returns the cached static HTML as the response to the browser.

As you can see, this is no rocket science. It's just a little bit of extra fun work, implementing our simple cache and adapting the basic code for server side rendering. In the next sections we will learn that the extra code to reach this goal is minimal and easy to set up.

Our simple in-memory caching mechanism

First of all, let's see how our cache implementation would look like. We need a simple CRUD interface to be able to update, read and delete our cache entries. By using the package memory-cache we receive the required functionality out of the box! The API is as clear as it can be:

// memory-cache-interface.ts

const cache = require('memory-cache');

// set or overwrite a value in the cache
cache.put('foo', 'bar');

// retrieve value from the cache
const cachedValue = cache.get('foo');

// delete key from cache
cache.del('foo');

// clear cache completely
cache.clear();

Code for cached rendering

As previously mentioned, res.render is where the rendering magic happens. By default res.render('index', config) returns the static HTML on the response. res.render also accepts an optional callback we can implement with our custom behavior. We can extend the highlighted ngApp function from our basic example without caching:

// simple-cache-setup.ts

const ngApp = (req, res) => {
  const config = {
    req,
    res,
    preboot: true,
    baseUrl: '/',
    requestUrl: req.originalUrl,
    originUrl: 'http://localhost:80',
  };

  const allowedPages = ['/', '/about', '/contact'];

  if (allowedPages.includes(req.originalUrl)) {
    const entry = cache.get(req.originalUrl); // check if we have a cache entry
    if (entry) {
      res.send(entry);                        // send the cache entry
    } else {
      res.render('index', config, (err, html) => {
        cache.put(req.originalUrl, html);     // save the HTML in the cache
        res.send(html);
      });
    }
  } else {
    res.render('index', config);              // just render with no cache
  }
};

Let's go trough the code and explain what is going on:

Line 13 is a first check to see if we should leverage the cache for this specific request by checking if the requested URL is allowed for caching.
Next we check if there is a cache entry for this request. If so we return the cached HTML.
If we don't have a cache entry for this request, we continue with normal Universal SSR and we save the result HTML in the cache before we return it as the response.

Cache only the existing pages

Be careful with what you cache! You should only cache the pages that actually exist on your application or website, or make sense to cache. So avoid the caching of non-existing pages. This could potentially lead to a memory-issue on your server..

The only thing a malicious visitor has to do is visit an unlimited amount of unexisting variants (/about1, /about2, ... ) of an existing page (/about) and your (memory) cache will get flooded with 404 not found pages. Consequently your server will become very slow and might eventually even crash.

A simple way to mitigate this risk is by using (a variant of) your sitemap.xml to whitelist the allowed pages. This is implemented on line 11-13 of the code shown above. In this case we are only allowing the /, /about and /contact pages to be cached.

Possible extensions and improvements

The concepts explained above only touch the basics of implementing a caching strategy and infrastructure. There are several improvements possible that I will explain briefly.

Generate your static versions at build time
Create an interface for your cache
Use a separate caching layer
Server-side render only specific pages
Use a file based cache instead of memory cache

Generate your static versions at build time

Instead of letting the first visitor of your website be responsible for generating the first cached version of specific pages, we can generate the static versions of all the pages of our application at build time.

We can do this for example after generating our sitemap.xml. At that point we know all the available urls of our application/website and we can generate the static versions of those pages and put them in the cache.

Create an interface for your cache

Maintaining the cache of a small website like this one is easy. Everytime a new version is available, I clear the cache completely. The first visitor of a specific page is now responsible for creating the cached version.

In bigger applications an interface to dynamicaly update and clean the cache is required. For example, when using a CMS to add a page or blogpost you can trigger a cache-request right after saving the content. If you delete a page, you can whipe out the cache, to make sure your visitors are not seeing a page that is already deleted.

Be sure to implement some security on this interface. You don't want somebody malicious messing with your cache.

Use a separate caching layer

In this example we have implemented caching at the level of the serving application. This couples our server software directly to the caching infrastructure. A more advanced approach would be to install a caching layer right before the application server.

This can either be a completely different (virtual) machine, or run as an extra service on the same machine as the one running our Node.js Express server. This way we can decouple our caching infrastructure completely from our serving application.

Server-side render only specific pages

In some cases it might not be required to SSR and cache all pages of your application. You might have a public facing part of your application and a private part that requires a login. The content of the private part could be very volatile, so caching is perhaps not the best option there.

// selective-ssr-and-cache.ts

// Our index.html we'll use as our template
const template = readFileSync(join(DIST_FOLDER, 'browser', 'index.html')).toString();

app.engine('html', (_, options, callback) => {
  renderModuleFactory(AppServerModuleNgFactory, {
    document: template,
    // ...
  }).then(html => {
    callback(null, html);
  });
});

const cache = require('memory-cache');

const ngApp = (req, res) => {
  const config = {
    // ...
  };

  const allowedPages = ['/', '/about', '/contact'];

  if (allowedPages.includes(req.originalUrl)) {
    // ...
  } else {
    res.send(template); // just return the template
  }
};

We did not change much compared to the caching example. The only thing we now do is return the original template directly, imported on line 2, in case of a request that should not be SSRed and cached. This is your original index.html file, so returning this as the response to the browser will just make the browser bootstrap your Angular application.

Use a file based cache instead of memory cache

Using a cache that lives in the memory is just one option. If for some reason you don't have much RAM available, but you have plenty of disk space, you could opt for a cache living on the file system. Entries would then just be like static HTML files saved on the disk. The only drawback I see is the potential extra I/O overhead reading and writing a file introduces.

Conclusion

Setting up a simple memory cache is no rocket-science and requires minimal changes to our Universal setup. The basic idea of using a memory cache is easily extended to for example a file cache. Adapting it to your needs is easy because we can hook in to the rendering process.

We have to remember this is a very basic setup that requires extension and improvements if we are going to use this on bigger applications and projects.

Special thanks to

Frederik Prijck

for reviewing this post and providing valuable and much-appreciated feedback!

Originally published at https://samvloeberghs.be on June 14, 2019

Applications or websites build with Angular Universal

Sam Vloeberghs — Mon, 10 Jun 2019 06:52:49 +0000

I've been using Angular Universal for my blog and personal website since I first learned about the technology in October 2016!

What websites have you built using Angular and more specifically Universal? What challenges did you face? How do you like it so far?

For those who haven't heard about it yet, Angular Universal is a specific technology allowing Angular applications to be rendered on the server. This gives you some major benefits, like

better performance and load times
better SEO and support for crawlers
and many more..

Read more about it here and here. Fire away if you have specific questions!

Lessons learned on offline capabilities with service workers using Workbox

Sam Vloeberghs — Fri, 07 Jun 2019 19:34:36 +0000

Originally published at https://samvloeberghs.be on May 31, 2019

Target audience

This post is targeted to webdevelopers that are creating offline-capable applications and more specifically offline-capable applications that can be categorized as SPAs or Single Page Applications. I will share some of my lessons learned, not just to share them, but also to maybe get feedback from interested readers that might have found other ways or used other methods to solve the problems described.

What will we discuss in this post?

Serve the index.html for all deeplinks (while offline)
Don't just blindly use skipWaiting() or clients.claim()
Be careful with what you cache in the browser
Avoid using global state in the service worker

All the example code used in this post, just as this complete website, is available on Github here as an example project for you to test and try-out.

Serve the `index.html` for all deeplinks (while offline)

While developing SPA applications we often, if not always, leverage the power of routing. When we go offline we need to make sure that all routes keep working. An example:

The entry point of a visitor of your application was the /about page. The server knows that for every hit on your website, it has to return the index.html file.
After a while, this user returns to your website, but he enters directly via the /contact page with a refresh or via some link on any other website.
For some reason your application is now offline, or the user has lost connectivity during this navigation. Your service worker kicks in, tries to lookup the /about page in the cache, but has no hit, because the only static file cached is the index.html file.
Your service worker somehow needs to know which file to service for these type of requests

Using Workbox this can easily be done by checking if the request was of mode "navigate" and if so, serve the single index.html file. This index.html file should already be cached with workbox as a static asset so we request it from the precache.

// sw.js

// default page handler for offline usage,
// where the browser does not how to handle deep links
// it's a SPA, so each path that is a navigation should default to index.html
workbox.routing.registerRoute(
  ({ event }) => event.request.mode === 'navigate',
  async () => {
    const defaultBase = '/index.html';
    return caches
      .match(workbox.precaching.getCacheKeyForURL(defaultBase))
      .then(response => {
          return response || fetch(defaultBase);
      })
      .catch(err => {
        return fetch(defaultBase);
      });
  }
);

This logic gives you quicker offline and online usages, because even when your user is online the service worker will use the cached index.html, when for example opening a new tab with a different page of your application.

Shortcut available

Workbox gives you a shortcut and extra options for this behaviour, like whitelisting and blacklisting by exposing the registerNavigationRoute handler on the routing module.

// sw.js

workbox.routing.registerNavigationRoute(
  // Assuming '/single-page-app.html' has been precached,
  // look up its corresponding cache key.
  workbox.precaching.getCacheKeyForURL('/single-page-app.html'), {
    whitelist: [
      new RegExp('/blog/')
    ],
    blacklist: [
      new RegExp('/blog/restricted/'),
    ]
  }
);

Please also keep in mind that this reflects the behavior of the server in case of a SPA setup. For all requests not matching a static resource like an image, or a dynamic resource like an API, the server should return the index.html file!

Don't just blindly use `skipWaiting()` or `clients.claim()`

Using skipWaiting() or clients.claim() should not be done blindly and without thought. Unfortunately there are many articles on the internet that just instruct you to do it without much more information. The fact that you have to be careful about this is also clearly mentioned in "Service Worker Lifecycle", an article by Jake Archibald.

Note: I see a lot of people including clients.claim() as boilerplate, but I rarely do so myself. It only really matters on the very first load, and due to progressive enhancement the page is usually working happily without service worker anyway.

What could possibly go wrong?

There are several scenarios where just calling skipWaiting() or clients.claim() blindly in your service worker could be the cause of unexpected and harmful side effects.

One possible scenario is that the code functionality in your application could not match anymore with the logic in your service worker. So it might impose a breaking change. Let's consider a short step-by-step example:

The browser has your application code, including the code that registers your service worker and the service worker, cached
When loading the application, this cached content is served and loads your new service worker with updated functionality, updates the precached content and finally skips the waiting process and claims the clients available
Your user now has the old application loaded in his browser, but using the new service worker functionality, which is potentially problematic!

So it's important to always make sure that the loaded version of the logic of your application is working nicely together with the loaded version of the logic of your service worker.

Service worker active during first-time usage of your application

Not calling skipWaiting() or clients.claim() could be problematic for your usecase, especially if you want your service worker to be active from the very first time you load your application. Let's take a basic example to explain this situation:

Your application imports a 3rd party library that sends requests to a server. The URLs these requests have to be sent to are configurable, but the headers used for these requests aren't. You may want to add an Authorization header to each of these requests made.
One of the possible strategies to use could be to setup a service worker that acts as a proxy and adds this header to all relevant requests.
This functionality has to work from the very first time, but you might think this is not possible without skipWaiting() and/or clients.claim().

The only way I found so far is by checking if your page already has a service worker that is activated, or is in the process of changing his state to activated, but no new version of the service worker is waiting to be activated. In that case we can assume there was no previous service worker available. Now we can automatically reload the page as soon as the service worker activated.

// service-worker-registration.ts

// Check that service workers are available
if ('serviceWorker' in navigator) {

    // Use the window load event to keep the page load performant
    window.addEventListener('load', () => {

    navigator.serviceWorker
      .register('/sw.js')
      .then(registration => {

        if (navigator.serviceWorker.controller) {
          // let the application know our service worker is ready
          window['serviceWorkerReady'] = true;
          window.dispatchEvent(new CustomEvent('service-worker-ready'));
        }

        // A wild service worker has appeared in reg.installing and maybe in waiting!
        const newWorker = registration.installing;
        const waitingWoker = registration.waiting;

        if (newWorker) {
          if (newWorker.state === 'activated' && !waitingWoker) {
            // reload to avoid skipWaiting and clients.claim()
            window.location.reload();
          }
          newWorker.addEventListener('statechange', (e) => {
            // newWorker.state has changed
            if (newWorker.state === 'activated' && !waitingWoker) {
              // reload to avoid skipWaiting and clients.claim()
              window.location.reload();
            }
          });
        }

      })
      .catch(err => {
        console.log('service worker could not be registered', err);
      });

  });

}

The drawback is that the first time-use might show a flicker. This side-effect can be minimized by delaying the bootstrapping of your application until you know that a service worker is active and show a loading / splash screen instead. This is anyway already a requirement, as explained before.

Please let me know if you have other suggestions! I'm curious for other methods because of the refresh I'm not very happy about..*

Be careful with what you cache in the browser

When building your application and thinking about its offline strategies you should definitely consider what you are gonna cache offline. Why? Because not all browser and devices have the same capabilities in terms of storage. The following table shows what the available quota is per browser and per origin.

Browser	Limit
Chrome	<6% of free space
Firefox	<10% of free space
Safari	<50MB
IE10	<250MB
Edge	Dependent on volume size

All credits for this overview go to the authors of the article Offline Storage for Progressive Web Apps

This will most probably always be enough space for applications running in a browser on a desktop or laptop. But if you're building a content-heavy application on a tablet or phone with limited storage, this might quickly become a problem..

Imagine a tablet with 32GB of space, while there is only 3GB left as available space, because of for example pictures, music and captured videos. This means that for your application, on this tablet there is only 6% available of 3GB, which is ~184 MB.

While 184 MB might still seem like a great enough amount of space, you have to consider the possibility it might not be enough, especially if you are caching large files or assets from other domains (CORS) or opaque responses. Dealing with opaque requests can be tricky as highlighted in the article When 7 KB Equals 7 MB by Gerardo Rodriguez

In general, we want to avoid hitting a DOMException: Quota exceeded as this might potentially break our complete service-worker functionality or offline capabilities.

Workbox allows us to cleanup our caches automatically using various methods. We can invalidate caches for a specific route handler after an amount of time and we can gracefully purge our cache on quota errors.

// sw.js

// Cache the underlying font files with a cache-first strategy for 1 year.
workbox.routing.registerRoute(
  /^https:\/\/fonts\.gstatic\.com/,
  new strategies.CacheFirst({
    cacheName: 'google-fonts-webfonts',
    plugins: [
      new workbox.cacheableResponse.Plugin({
        statuses: [0, 200]
      }),
      new workbox.expiration.Plugin({
        maxAgeSeconds: 60 * 60 * 24 * 365,
        maxEntries: 30,
        purgeOnQuotaError: true // Automatically cleanup if quota is exceeded.
      })
    ]
  })
);

Avoid using global state in the service worker

I've learned this one the hard way, by debugging and debugging and debugging till I got crazy and started searching the internet. Let me first explain my usecase, which is in fact the origin of the Authorization header example above.

The application that we are developing includes 3rd party libraries like openlayers to give us very nice dynamic map functionalities.
The URLs this library uses to load the map-tiles is configurable, but the headers aren't. We needed to be able to add an Authorization header on the requests made to these URLs and using a service-worker for this seemed reasonable.
As soon as we log in, we pass the token to the service worker, using the BroadcastChannel API and we store it in a global variable let oAuthToken.
Every time our service worker sees a request for the map-tiles, we proxy the requests and add the token as a header to the request so our server/backend can successfully authorize the request.

Now what was going wrong?

It seems that service workers are potentially stopped and restarted multiple times in their lifetime because of several optimizations like battery saving algorithms. Stopping and restarting destroys and reinitializes the global scope of our service worker! So our global variable holding the token was gone, resulting in requests that did not authorize properly.

So how to correctly share the `OAuthToken` with the service worker?

The only way to successfully share the OAuthToken from our application logic with our service worker logic is by using the IndexedDB API. The library I use to simplify IndexedDB access in my service worker is idb-keyval. If you need more advanced logic you can checkout idb.

In our example below, we assume that somehow our application has saved a token in the IndexedDB. Using workbox we register a RegExp for any route we want to authenticate using our token. Let's go trough the example, step by step:

We register a route, in this case to trap all the requests going to map.png
In the capture function of the route registration, the first parameter, a check is done to see if we have a token available in the IndexedDB.
If a token is available, we modify the request in the handler, the second parameter, and add the token as an Authorization header.
If no token is available, we return a replacement resource, the not_authorized.png image, which has been cached already in our precaching setup.

// OAuth header interceptor
workbox.routing.registerRoute(
  ({ url }) => {
    return /map\.png/.test(url);
  },
  async ({ event, url }) => {

    // get the eventual token
    const customStore = new Store('swl-db', 'swl-db-store');
    const oAuthToken = await get('token', customStore);

    // if token available, set it as the Authorization header
    if (!!oAuthToken) {
      const modifiedHeaders = new Headers(event.request.headers);
      modifiedHeaders.set('Authorization', oAuthToken);
      const overwrite = {
        headers: modifiedHeaders
      };
      const modifiedRequest = new Request(url.toString(), overwrite);
      return fetch(modifiedRequest);
    }

    const defaultNotAuthedBase = '/assets/not_authorized.png';
    return caches
      .match(workbox.precaching.getCacheKeyForURL(defaultNotAuthedBase))
      .then(response => {
        return response || fetch(defaultNotAuthedBase);
      })
      .catch(err => {
        return fetch(defaultNotAuthedBase);
      });

  }
);

A possible extension for this functionality would be to check if the status code of the response coming from the server is as expected, for example a 200 OK. If not a valid or expected response code, return the cached not_authorized.png image.

Conclusion

When developing SPAs with offline capabilities, we need to make sure that our service worker is acting the same way as our server. For routing requests or navigations, the service worker should always return the index.html to keep deeplinking working when offline.

Don't just blindly use skipWaiting() or clients.claim(), but think about the possible side effects it could introduce. Instead, let the user decide when to upgrade the functionality or let a new version be applied automatically on a sub-sequent load of your application.

Be careful with what you cache offline and which devices your application will be used upon. The available space for your application depends on the free space of the device and the browser where the application is loaded.

Don't depend on global variables or state in your service worker as service workers are potentially destroyed and bootstrapped several times during their lifetime. Use IndexedDB to share state from your application with your service worker.

Special thanks to

Jurgen Van De Moere

for reviewing this post and providing valuable and much-appreciated feedback! I'm open for any other feedback so please let me know!

Originally published at https://samvloeberghs.be on May 31, 2019

Outputting JSON-LD with Angular Universal

Sam Vloeberghs — Fri, 07 Jun 2019 18:39:16 +0000

Originally published at https://samvloeberghs.be on March 12, 2019

Target audience

This article and guide on generating JSON-LD with Angular Universal is targeted to developers that want to generate the JSON that represents the Linked Data object and inject it correctly in the HTML that gets generated on the server. We will re-use and explain some techniques that the TransferState key-value cache service uses to transfer server-state to the application on the client side.

First of all; What is JSON-LD?

JSON-LD is a lightweight Linked Data format. It is easy for humans and machines to read and write. It is based on the already successful JSON format and provides a way to help JSON data interoperate at Web-scale.

Linked Data empowers people that publish and use information on the Web. It is a way to create a network of standards-based, machine-readable data across Web sites. It allows an application to start at one piece of Linked Data, and follow embedded links to other pieces of Linked Data that are hosted on different sites across the Web.

These 2 definitions of JSON-LD and Linked Data were simply taken from the website of JSON-LD. This article does not go deep into the full specification of JSON-LD and it does not teach you how to properly generate your Linked Data structures. The tools of the JSON-ld website, like for example the playground, give you all you need to validate your Linked Data structures.

A basic example

What you can see below is a typical example of a JSON-LD data object, injected into the HTML. It describes a LocalBusiness by following its definition and providing the properties that identifiy the entity. For example, the address, it's geolocation, opening hours etc are given. As you can see the address has it's own type PostalAddress. Most of the types can specificy an external link and this is how data gets interweaved on the same website or resource and even on other websites.

<!-- json-ld-example.html -->

<script type="application/ld+json">
{
  "@context": "http://schema.org",
  "@type": "LocalBusiness",
  "address": {
    "@type": "PostalAddress",
    "addressLocality": "Manhattan",
    "addressRegion": "NY",
    "postalCode":"10036",
    "streetAddress": "400 Broadway"
  },
  "description": "This is your business description.",
  "name": "Craig's Car Repair",
  "telephone": "555-111-2345",
  "openingHours": "Mo,Tu,We,Th,Fr 09:00-17:00",
  "geo": {
    "@type": "GeoCoordinates",
    "latitude": "20.75",
    "longitude": "13.98"
  },
  "sameAs" : [ "http://www.facebook.com/example",
    "http://www.twitter.com/example",
    "http://plus.google.com/example"]
}
</script>

Why should you care about JSON-LD?

You should definitely care about JSON-LD for several reasons:

It's simpler to generate, more structured and easier to read than Microdata
It links or allows your data to be linked with other entities / data subjects on the world-wide web

In general; it ads a lot more semantic value to your HTML markup by providing context to what you show on the page and makes this context machine-readable, allowing it to be parsed more easily by search engines and other crawlers on the web.

Generating your JSON Linked Data structure

In the simple case of a blog or not too complex website, generating your JSON Linked Data can be done using the same base data you use to set your meta tags for social sharing and SEO. Your most important concern is building the correct structure. How your structure looks is completely dependent on your usecase. As a basic example we will use this website, and more specifically the about page.

Using the router we first define the correct SEO data associated with our route. Just like we did before, setting the correct social share meta tags.

// about-routing.module.ts

RouterModule.forChild([{
    path: '',
    component: AboutComponent,
    data: {
      seo: {
        title: `About Sam - ${environment.seo.title}`,
        description: `I'm a 30 year old software engineer living in Belgium.`,
        shareImg: '/assets/share/about.png',
      }
    }
}])

Using a service we can subscribe to route changes to extract this data and pass the data to a service to parse the JSON-LD object.

// route-helper.service.ts

@Injectable({
    providedIn: 'root',
})
export class RouteHelper {

  constructor(
    private readonly router: Router,
    private readonly activatedRoute: ActivatedRoute,
    private readonly jsonLdService: JsonLdService
  ) {
    this.setupRouting();
  }

  private setupRouting() {
    this.router.events.pipe(
      filter(event => event instanceof NavigationEnd),
      map(() => this.activatedRoute),
      map(route => {
        while (route.firstChild) {
          route = route.firstChild;
        }
        return route;
      }),
      filter(route => route.outlet === 'primary')
    ).subscribe((route: ActivatedRoute) => {
      const seo = route.snapshot.data['seo'];
      // generate your JSON-LD object here
      const jsonLd = {
        name: seo.title,
        url: environment.url + this.router.routerState.snapshot.url,
      };
      this.jsonLdService.setData('Website', jsonLd);
    });
  }

}

The JsonLdService injected above is a simple service that caches and updates the data structure we need to output. A basic implementation of this service can be as follows, where you as a developer are still in charge of correctly structuring your object.

// ngx-seo/json-ld.service.ts

@Injectable({
  providedIn: 'root',
})
export class JsonLdService {

  private jsonLd: any = {};

  setData(type: string, rawData: any) {
    this.jsonLd = this.getObject(type, rawData);
  }

  getObject(type: string, rawData?: any) {
    let object = {
      '@context': 'http://schema.org',
      '@type': type,
    };
    if (rawData) {
      object = Object.assign({}, object, rawData);
    }
    return object;
  }

  toJson() {
    return JSON.stringify(this.jsonLd);
  }
}

Using the approach explained above our current JSON-LD data-object in memory will look like this:

{
    "@context": "http://schema.org",
    "@type": "Website",
    "name": "About Sam - Sam Vloeberghs - Freelance Webdeveloper & Software Engineer",
    "url": "https://samvloeberghs.be/about"
}

This basic example is what we wanted to achieve. The next step is getting this object outputted in our static HTML.

Injecting the JSON in the static HTML

Getting inspiration in the Angular source code

Injecting the JSON-LD data object in the DOM is only really required when we generate the HTML on the server. To get to this result I looked into the BrowserTransferStateModule(https://github.com/angular/angular/blob/master/packages/platform-browser/src/browser/transfer_state.ts) exposed via the @angular/platform-browser module.

Essentialy this module, and more specifically, the TransferState(https://github.com/angular/angular/blob/master/packages/platform-browser/src/browser/transfer_state.ts) service it provides, does the same thing we want to achieve. It caches data, the result from for example HTTP calls, in an object and holds it in memory during the application lifecycle. Apart from the HTTP calls, that's exactly what our JsonLdService does.

The server counter part, ServerTransferStateModule(https://github.com/angular/angular/blob/master/packages/platform-server/src/transfer_state.ts) exposed via the @angular/platform-server, serializes the data and injects it in the DOM before generating the HTML on the server and sending it back over the wire to the browser. Again, that is exactly what we want to achieve. This is achieved by providing an extra factory function to the BEFORE_APP_SERIALIZED token. Right after the application becomes stable all factories attached to the BEFORE_APP_SERIALIZED are executed.

// server: transfer_state.ts

export function serializeTransferStateFactory(
  doc: Document, appId: string, transferStore: TransferState
) {
  return () => {
    const script = doc.createElement('script');
    script.id = appId + '-state';
    script.setAttribute('type', 'application/json');
    script.textContent = escapeHtml(transferStore.toJson());
    doc.body.appendChild(script);
  };
}
@NgModule({
  providers: [
    TransferState,
    {
      provide: BEFORE_APP_SERIALIZED,
      useFactory: serializeTransferStateFactory,
      deps: [DOCUMENT, APP_ID, TransferState],
      multi: true,
    }
  ]
})
export class ServerTransferStateModule {
}

When the Angular application bootstraps in the browser, the TransferState service picks up the serialized state in the static HTML and unserializes it, making it available as a direct cache. This way we avoid that the application makes a similar request for the same data to the server, for which the server-side-rendered version already did the call. We don't need this part, but it's good to know.

// browser: transfer_state.ts

export function initTransferState(
  doc: Document, appId: string
) {
  const script = doc.getElementById(appId + '-state');
  let initialState = {};
  if (script && script.textContent) {
    try {
      initialState = JSON.parse(unescapeHtml(script.textContent));
    } catch (e) {
      console.warn('Exception while restoring TransferState for app ' + appId, e);
    }
  }
  return TransferState.init(initialState);
}
@NgModule({
  providers: [{
    provide: TransferState,
    useFactory: initTransferState,
    deps: [DOCUMENT, APP_ID]
  }s],
})
export class BrowserTransferStateModule {
}

Our simple BrowserJsonLdModule and ServerJsonLdModule

Following the concepts and ideas we learned from the ServerTransferStateModule and the BrowserTransferStateModule we create our own ServerJsonLdModule and BrowserJsonLdModule. The only small differences are limited to changing the type of the <script> element and injecting it in the <head> instead of right before the </body> tag. We also don't need to pickup any state from the static HTML, as is done in the BrowserTransferStateModule.

// ngx-seo/json-ld.module.ts

import { NgModule } from '@angular/core';
import { JsonLdService } from 'ngx-seo/json-ld.service';

@NgModule({
  providers: [
    JsonLdService,
  ]
})
export class BrowserJsonLdModule {
}
ngx-seo/json-ld.server.module.ts
import { NgModule } from '@angular/core';
import { DOCUMENT } from '@angular/common';
import { BEFORE_APP_SERIALIZED } from '@angular/platform-server';
import { JsonLdService } from 'ngx-seo/json-ld.service';

export function serializeJsonLdFactory(doc: Document, jsonLdService: JsonLdService) {
  const serializeAndInject = function () {
    const script = doc.createElement('script');
    script.setAttribute('type', 'application/ld+json');
    script.textContent = jsonLdService.toJson();
    doc.head.appendChild(script);
  };
  return serializeAndInject;
}

@NgModule({
  providers: [
    JsonLdService, {
    provide: BEFORE_APP_SERIALIZED,
      useFactory: serializeJsonLdFactory,
      deps: [DOCUMENT, JsonLdService],
      multi: true,
    },
  ],
})
export class ServerJsonLdModule {
}

Differentiate execution on the server

To differentiate the execution of our application on the server versus on the browser, in Angular Universal we typically create a new server module app.server.module.ts next to app.module.ts that will directly import the AppModule exported by app.module.ts. This version of the application is used in the HTML renderer configured at the server side, in most cases as a rendering engine.

If we go back to the example of the TransferState service, we see that the app.server.module.ts is importing the ServerTransferStateModule. This will overwrite the provider of the JsonLdService imported in the app.module.ts and provide the extra serialize functionality, next to also providing the TransferState service.

// app.module.ts
`
@NgModule({
  declarations: [
    AppComponent,
  ],
  imports: [
    BrowserModule.withServerTransition({appId: 'samvloeberghs'}),
    BrowserTransferStateModule,
    BrowserJsonLdModule,
    CommonModule,
    AppRoutingModule
  ],
  exports: [
    AppComponent,
  ],
  bootstrap: [AppComponent],
})
export class AppModule {
}

// app.server.module.ts

@NgModule({
  imports: [
    AppModule,
    ServerModule,
    ModuleMapLoaderModule,
    ServerTransferStateModule,
    ServerJsonLdModule,
  ],
  bootstrap: [AppComponent],
})
export class AppServerModule {
}

We do the same thing with our BrowserJsonLdModule and ServerJsonLdModule. The browser module has to be imported in app.module.ts while the server module has to be imported in app.server.module.ts.

Conclusion

Generating JSON-LD using Angular Universal is pretty straight-forward. By borrowing concepts from the Angular source code we can create our own JsonLd modules and services that enable us to output JSON-LD in the statically generated HTML on the server. The result of this code can be found live on this website, just look at the source and do a hard refresh. Why a hard refresh you might ask? Because of the service worker caching the default index.html as the app shell.

Special thanks to

Ksenia Karelskaya

Originally published at https://samvloeberghs.be on March 12, 2019

Better sharing on social media platforms with Angular Universal

Sam Vloeberghs — Thu, 06 Jun 2019 09:42:34 +0000

Originally published at https://samvloeberghs.be on February 24, 2019

Target audience

This article and guide on better sharing on social media platforms with Angular Universal is targeted to developers that have at least a basic understanding of Angular and might already have an Angular Universal implementation up and running. We are going to describe some high-level ideas and concepts, different tools and Angular services that will enable better social sharing for platforms like Facebook, Twitter and LinkedIn.

Angular Universal and SEO problems with SPAs

It’s not a new concept, running Angular on the server with Universal. The first version of this website was built with Angular v2 and Universal concepts back in November 2016. The tooling around it has grown a lot since then and my website evolved together with those tooling updates.

It’s much easier to build an Angular universal website or app today because it’s completely documented and there are a lot of example projects available. Angular CLI also gives us the tools to add this functionality automatically by using simple commands.

The fact that SPAs in general have big problems with indexing and social sharing has been a topic for many discussions, and for many years, ever-since the first JS applications were build. Different approaches have been considered and one of them, like prerendering the application with a headless browser like PhantomJS or Puppeteer and serving different content for different consumers, bots vs real visitors, can be categorized as black hat SEO.

Indexing capabilities of SPAs by search engines

When you generate an Angular application via the angular-cli a polyfills.ts file gets generated. It includes the following content:

// polyfills.ts

/** IE9, IE10, IE11, and Chrome <55 requires all of the following polyfills.
* This also includes Android Emulators with older versions of Chrome
* and Google Search/Googlebot
*/

This means that our Angular app has support for the Google Search / Googlebot, even without SSR, if we include those specific polyfills. It does not mention any other search engines. The other typical search engines, like Bing, Yahoo, DuckDuckGo, etc all have problems parsing JavaScript heavy websites, like SPAs or applications that use frameworks like for example jQuery.

Update (May 2019): The Google bot / crawler is now up-to-date with the latest Chromium and will update regularly. You can read more about this update on the Google Webmasters Blog.

The overview below gives us a good idea which search engines are capable of indexing JavaScript heavy applications, build with different frameworks and libraries.

All credits for this overview go to Bartosz Góralewiczm, see also blogpost linked below

If you want to read more about this topic, I suggest reading https://moz.com/blog/search-engines-ready-for-javascript-crawling. The blogpost is a little bit outdated as the crawlers have most likely evolved in their capabilities, like for example the ability of the Googlebot to parse Angular v2+ applications. The general problem remains: Search engines are, in general, not very capable of rendering JavaScript while indexing.

The 2nd wave of indexing

It would be wrong stating that SPAs are not being indexed. Google does an extra effort in the 2nd wave of indexing, where parsing of JS-heavy applications is done better. More information about this specific topic can be found here.

Google, Bing and others are not the only search engines

Everything that is described above is also true for other crawlers, like those of Facebook, Twitter and LinkedIn, just to name a few of the social media platforms I consider to be important for my cases. These can also be considered search engines, because they parse and cache the content of your website or application when their users share your content.

And just like the other typical search engines, most of them, if not all, are not capable of parsing JavaScript and only rely on the statically generated HTML.

How do social media platforms crawl your page?

Social media platforms use your statically generated HTML to show a preview when sharing one of your pages on their feed. Therefor they parse the title and meta tags that can be found in the <head> tag.

<!-- static-generated.html -->

<!doctype html>
<html lang="en">
   <head>
      <meta charset="utf-8">
      <title>Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer</title>
      <meta name="description" content="Hi there! 👋 Thank you for visiting my website!">
      <meta name="author" content="Sam Vloeberghs">
   </head>
   ...

This seems easy, but it's not that easy to get it right. Because all the different social media platforms expect other meta tags for rendering their previews. They don't even use the same attribute to identify the correct meta tags. Some use the name attribute and others use the property attribute. It means you'll have to support around 15 different meta tags to get the basics like title, image, description and author right (for Facebook, LinkedIn and Twitter).

All of the discussed social media platforms provide us with tools to validate our pages or posts before sharing them. Some platforms, like Facebook and LinkedIn cache all the shared content, so these tools also allow us to invalidate cached content. Super handy if you have already shared your page before, or if you want to invalidate the cache of a set of pages in one go.

Facebook

To get Facebook to show your content correctly you need to implement the following basic meta tags. Please note that Facebook uses the property attribute to define the type of meta value and that those types are always prefixed with og:, which is the prefix for the open-graph protocol.

<!-- static-generated-facebook.html -->

<head>
  <meta property="og:title" content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">
  <meta property="og:description" content="Hi there! 👋 Thank you for visiting my website!">
  <meta property="og:url" content="https://samvloeberghs.be/">
  <meta property="og:image" content="/assets/share_img.png">
  <meta property="og:image:alt" content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">
  <meta property="og:image:height" content="xxx">
  <meta property="og:image:width" content="xxx">
</head>

The open-graph protocol allows for a lot more meta tags to be defined. Especially Facebook supports a lot more, like video. Check the documentation for Facebook sharing to learn more.

Facebook Sharing Debugger

Validating a preview of your post on Facebook can be done via https://developers.facebook.com/tools/debug/sharing. Facebook also provides a way to invalidate a batch of pages, which can also be done programmaticaly via their API.

To get your post to show correctly on LinkedIn you need to implement the following basic meta tags. Please note that LinkedIn also uses the og: prefixes, but not for all values, like the author meta tag.

<!-- static-generated-linkedin.html -->

<head>
  <meta property="og:title" content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">
  <meta property="og:type" content="website">
  <meta property="og:image" content="/assets/share/home.png">
  <meta property="og:description" content="Hi there! 👋 Thank you for visiting my website!">
  <meta name="author" content="Sam Vloeberghs">
</head>

LinkedIn Post Inspector

Validating a preview of your post or website and refreshing the cache on LinkedIn can be done via https://www.linkedin.com/post-inspector/.

The only thing I'm still not sure about is which meta property or name they use to correctly read out the "Publish date". I have asked the question to their support and I'm still waiting for an answer. And sadly there is little to no documentation to be found on this subject.

Update: I did some trial-and-error testing and it seems that the meta tag <meta name="published_date" content="2019-02-24T11:00:00.000Z"> does the trick.

Twitter

To get your post to show correctly on Twitter as a card you need to implement the following meta tags. Please note that, compared to Facebook and LinkedIn, Twitter uses the name attribute to define the type of meta value and that those types are always prefixed with twitter:

<!-- static-generated-twitter.html -->

<head>
  <meta name="twitter:card" content="summary_large_image">
  <meta name="twitter:title" content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">
  <meta name="twitter:description" content="Hi there! 👋 Thank you for visiting my website!">
  <meta name="twitter:image" content="https://samvloeberghs.be/assets/share/home.png">
  <meta name="twitter:image:alt" content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">
  <meta name="twitter:site" content="@samvloeberghs">
  <meta name="twitter:creator" content="@samvloeberghs">
</head>

Twitter Card Validator

Validating the preview of your post on Twitter can be done via https://cards-dev.twitter.com/validator.

Important: The value of twitter:image must be a fully-qualified complete and absolute URL, not a partial/relative one, or the Twitter crawler will be unable to find and display the image.

Using the Meta and Title services in Angular

To achieve all this, the only 2 services we need to use are the Title and Meta service. We use them to dynamically update the title of a webpage and for setting the required meta tags described before.

Using the router we can attach metadata that needs to be coupled to our routes. If we want to set the title for a dynamic page, like for example a blogpost, we need to get the data first and then call the same services to update the meta and title tags. As you can see in the example below, for the about page, we've configured these values and attached them to the route.

// about-routing.module.ts

RouterModule.forChild([{
  path: '',
  component: AboutComponent,
  data: {
    seo: {
      title: `About Sam - ${environment.seo.title}`,
      description: `I'm a 30 year old software engineer living in Belgium.`,
      shareImg: '/assets/share/about.png',
    }
  }
}])

Using a service we can subscribe to route changes to extract this data and update our title and meta tags.

// route-helper.service.ts

@Injectable({
  providedIn: 'root',
})
export class RouteHelper {

  constructor(
    private readonly router: Router,
    private readonly activatedRoute: ActivatedRoute,
    private readonly seoSocialShareService: SeoSocialShareService
  ) {
    this.setupRouting();
  }

  private setupRouting() {
    this.router.events.pipe(
      filter(event => event instanceof NavigationEnd),
      map(() => this.activatedRoute),
      map(route => {
        while (route.firstChild) {
          route = route.firstChild;
        }
        return route;
      }),
      filter(route => route.outlet === 'primary')
    ).subscribe((route: ActivatedRoute) => {
      const seo = route.snapshot.data['seo'];
      // set your meta tags & title here
      this.seoSocialShareService.setData(seo);
    });
  }

}

As explained above, all our social platforms expect other meta tags. Therefore it's best to create a SEO wrapper service, that injects the Meta Service and Title service from Angular. As an example, I will only implement the method to update the title value on our pages. If you want to see the full example, have a look at the full implementation of my SeoService.

// ngx-seo/seo-social-share.service.ts

@Injectable({
    providedIn: 'root',
})
export class SeoSocialShareService {

  constructor(
    private readonly metaService: Meta,
    private readonly titleService: Title
  ) {
  }

  setData(data: SeoSocialShareData) {
    this.setTitle(data.title);
    ... // set all other meta tags
  }

  private setTitle(title: string = '') {
    this.titleService.setTitle(title);
    if (title && title.length) {
      this.metaService.updateTag({name: 'twitter:title', content: title});
      this.metaService.updateTag({name: 'twitter:image:alt', content: title});
      this.metaService.updateTag({property: 'og:image:alt', content: title});
      this.metaService.updateTag({property: 'og:title', content: title});
      this.metaService.updateTag({name: 'title', content: title});
    } else {
      this.metaService.removeTag(`name='twitter:title'`);
      this.metaService.removeTag(`name='twitter:image:alt'`);
      this.metaService.removeTag(`property='og:image:alt'`);
      this.metaService.removeTag(`property='og:title'`);
      this.metaService.removeTag(`name='title'`);
    }
  }

}

Watch out for over-optimization

When generating static pages server side you might consider optimizing the rendered HTML as well. Tools like HTMLMinifier take out all the bloat from your HTML, to keep the bare minimum that the browser needs to be capable of rendering it.

But be careful. Browsers like Chrome, Firefox and others are able to parse your minified HTML, as they are very forgiving and fix errors for us. But parsers that rely heavily on structured HTML, like the ones the social platforms use, might not be able to parse it correctly and get out the information they need.

A simple example

The following HTMLminifier options remove all the stuff the browser does not need to correctly show your page:

// too-much-minify-options.ts

const tooMuchHtmlMinifyOptions = {
    removeComments: true,
    removeCommentsFromCDATA: true,
    collapseWhitespace: true,
    collapseBooleanAttributes: true,
    removeRedundantAttributes: true,
    useShortDoctype: true,
    removeEmptyAttributes: true,
    minifyCSS: true,
    removeAttributeQuotes: true,
    removeOptionalTags: true
};

const minify = require('html-minifier').minify;
const minifiedHtml = minify(html, minifyOptions);

Let's process the following HTML:

<!-- too-much-minify-options-input.html -->

<!doctype html>
<html lang="en">
   <head>
      <meta charset="utf-8">
      <meta property="og:title" content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">
      ...

results in

<!-- too-much-minify-options-output.html -->

<!doctype html><html lang=en><meta charset=utf-8><meta property=og:title content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">

This is too much optimization for the LinkedIn Post Inspector. If you look closely, you'll notice that the optional <head> tag was removed and that the quotes around simple attribute values, with a single non-spaced value, were removed as well. To be able to minify the HTML and keep the LinkedIn Post Inspector happy we need to change our configuration a little bit. We want to keep the attribute quotes and optional tags:

// good-minify-options.ts

const goodHtmlMinifyOptions = {
    …
    removeAttributeQuotes: false,
    removeOptionalTags: false
};

Now let's rerun the minifier again with the same HTML as mentioned above and see what the final result looks like:

<!-- good-minify-options-output.html -->

<!doctype html><html lang="en"><head><meta charset="utf-8"><meta property="og:title" content="Sam Vloeberghs - Freelance Webdeveloper &amp; Software Engineer">

We are keeping the <head> tag and the attribute quotes. And now the LinkedIn Post Inspector is happy 😃.

Conclusion

Not all search engines behave the same way. Not only Google, Bing etc. can be categorized as search engines but we have to think about Social platforms like Facebook, Twitter and LinkedIn, that also crawl and cache external content. Each crawler behaves in a different way and most of them are not capable of parsing JavaScript heavy applications.

This means we have to adjust and configure our applications very specifically for the different crawlers of the social platforms we want to support. By using Angular Universal, we are able to generate a static version of your application on the server and output static HTML. When minifying this static HTML we need to be careful and keep our HTML as strict and structured as possible.

Special thanks to

for reviewing this post and providing valuable and much-appreciated feedback!

Originally published at https://samvloeberghs.be on February 24, 2019

What's with the Subjects in RxJS

Sam Vloeberghs — Thu, 06 Jun 2019 07:42:50 +0000

Originally published at https://samvloeberghs.be on July 14, 2016

RxJS is a great library, it is used more and more in different places & frameworks across the web. Soon it will dominate the asynchronous part of web apps, so that’s why it’s important to know how subjects work. Subjects combine the best of both an Observable and Observer. So you can think of a Subject being an Observable and Observer at the same time.

A Subject is an Observable, but an Observable is unicast. Subjects on the other hand are multicast, which means that they act like EventEmitters that can handle a set of multiple listeners or subscribers. To facilitate this behaviour, new observers are registered in a list of Observers that all get notified. The subscribers are agnostic about what is sending the updates, as it can be a plain Observable or a Subject emitting the values.

A Subject is an Observer, which means it can also send new updates on next(val), emit errors on error(err) and complete() the stream.

Subjects come in different flavours, like a normal Subject, a BehaviourSubject, a ReplaySubject and an AsyncSubject. We’ll try to explain the differences between all of them and support the explanation with a real-world and readable example.

What is a `Subject`?

A Subject can be used to subscribe all the observers of that Subject, which is also an Observable. The Subject can subscribe to another async operation or source like an API backend, serving as a proxy for a collection of subscribers and a source.

Example

In the example below, we have two Observers attached to one Subject, and we feed some values to the Subject. Notice that there are different ways for subscribing. You can use an object that defines a function for the next value, or use the 3 different callback functions for next, error and complete:

import { Subject } from 'rxjs/Subject';
let subject = new Subject<number>();

subject.subscribe(
   (val: number) => { console.log('observer I: ' + val) },
   () => {},
   () => {}
);
subject.subscribe({
    next: (val: number) => console.log('observer II: ' + val)
});

subject.next('test')  // ! compile time error !

subject.next(11);
// log: observer I: 11
// log: observer II: 11

subject.next(22);
// log: observer I: 22
// log: observer II: 22

We are using TypeScript in this example so we can provide a type to the Subject. This way we get type-checking on compile-time if we want to send a next value that does not comply with the type!

See this example in action here: https://plnkr.co/edit/nrXf41PNgUdJTMCjR9gG?p=preview

What is a `BehaviorSubject`?

The BehaviorSubject is different from a normal Subject in the way that it stores the latest emitted value and requires an initial value. When a new subscriber subscribes to the BehaviorSubject, the latest value is immediately send to the new subscriber. If there is no value emitted yet, the default value will be returned. Once a BehaviorSubject is completed it will free up the memory. Newly subscribed Observers will not receive the latest value.

Example

We give the BehaviorSubject an initial value of 11. The first Observer that subscribes to the Subject gets immediately notified of the initial value. The same is true for the next Observer.

import { BehaviorSubject } from 'rxjs/BehaviorSubject';
let behaviorSubject = new BehaviorSubject<number>(11);

behaviorSubject.subscribe(
   (val: number) => console.log('observer I: ' + val)
);
// log: observer I: 11

behaviorSubject.subscribe({
    next: (val: number) => console.log('observer II: ' + val)
});
// log: observer II: 11

From this point on the behavior of the BehaviorSubject is the same as a normal Subject. But if we subscribe a new Observer to the BehaviorSubject it will immediately receive the last emitted value!

behaviorSubject.next(22);
// log: observer I: 22
// log: observer II: 22

behaviorSubject.next(33);
// log: observer I: 33
// log: observer II: 33

behaviorSubject.subscribe({
    next: (val: number) => console.log('observer III: ' + val)
});
// observer III: 33

See this example in action here: https://plnkr.co/edit/HPNSAzFmLipvHQ96ZkFt?p=preview

Real world examples cases

You’ll want to use a BehaviorSubject if you always need an initial value. This initial value can serve as a temporary placeholder while you wait for the first, of a possible sequence, real value. For example:

You are fetching the contents of a file and you want to show an initial placeholder, stating that the file is loading.
Your chatbox shows a welcome message before you effectively start receiving real values.

What is a `ReplaySubject`?

A ReplaySubject is kind of similar to a BehaviorSubject. It can send previous values to new attached Subscribers, but it can also record a part of the Observable execution. The ReplaySubject records multiple values from the Observable execution and replays them to any new Subscribers attached.

The first parameter in the constructor when creating a ReplaySubject defines how many values to replay. The second parameter specifies the window time in miliseconds that, besides the buffer size, determines the maximum lifetime of recorded values.

Example

In the following example we use a rather big buffer size of 25, but the window time parameter invalidates the recorded items in the buffer after 1000ms or 1 second. So if we emit some values using a specific timeline of 3 points every 800ms, only the values emitted in the last timeframe are replayed, as all other values were invalidated after 2000ms. The first batch after 1000ms and the second batch after 2000ms in the complete timeframe.

import { ReplaySubject } from 'rxjs/ReplaySubject';
let replaySubject = new ReplaySubject<number>(25, 1000);

setTimeout(() => {
    replaySubject.next(1);
    replaySubject.next(2);
    replaySubject.next(3);
}, 0);

setTimeout(() => {
    replaySubject.next(4);
    replaySubject.next(5);
    replaySubject.next(6);
}, 800);

setTimeout(() => {
    replaySubject.next(7);
    replaySubject.next(8);
    replaySubject.next(9);
}, 1600);

setTimeout(() => {

    replaySubject.subscribe(
      (val: number) => console.log('observer: ' + val)
  );

}, 2000);
// log: observer: 7
// log: observer: 8
// log: observer: 9

See this example in action here: https://plnkr.co/edit/dfCLm3Eh2c39JKOlGLZy?p=preview

Real world examples cases

You’ll mostly need a ReplaySubject if you need to replay a set of actions or values every time you invoke a specific part of your application. For example:

You could use a ReplaySubject in your application to show the last 5 notifications, every time the user goes back to his dashboard.
You could use a ReplaySubject in your chat application to show the last 5 messages of a specific user, every time you open up his chatbox.

What is an `AsyncSubject`?

The AsyncSubject is an alternate implementation of a Subject where only the last value is sent to its observers, and only when the execution completes. This last value is cached forever and every new Observer subscribing to the AsyncSubject will immediately receive the cached value.

AsyncSubjects can be used as a replacement for Promises. The only difference is that a Promise handles a single value, and AsyncSubjects handle multiple values, only caching and emitting the last value. Remember to complete the AsyncSubject, as this is the caching-enabler!

In the following example the only emitted value to the next function of the observer will be 3, as we complete the subject after the last call to next.

import { AsyncSubject } from 'rxjs/Subject';
let asyncSubject = new AsyncSubject<number>();

asyncSubject.subscribe(
  (val: number) => console.log('observer: ' + val)
);

asyncSubject.next(1);
asyncSubject.next(2);
asyncSubject.next(3);

asyncSubject.complete();
// log: observer: 3

See this example in action here: https://plnkr.co/edit/3QAbza00V36EQv7N5seO?p=preview

Real world examples case

An AsyncSubject can be used in situations where you do multiple requests to a resource. If your remote service/function uses and returns a singleton implementation of the AsyncSubject, subsequent calls to this resource will only invoke one call to the backend. All the subsequent calls (and subscribers) will be notified with the same result of the singleton AsyncSubject.

More details and examples?

If you know of any other related examples for any type of Subject, let me know! I’ll be happy to add them in the list of examples. If you want to read more about Subjects or RxJS in general you can read trough the extensive readme files on the project’s Github repository: https://github.com/ReactiveX/rxjs/tree/5.0.0-beta.10/doc or on the official website’s documentation: http://reactivex.io/rxjs/identifiers.html.

Scroll to top on Angular Router navigation

Sam Vloeberghs — Thu, 06 Jun 2019 07:13:48 +0000

Originally published at https://samvloeberghs.be on November 19, 2016

Update (December 2018): This article has been updated to represent the newly available ViewportScroller class, available from Angular v7+. This class implementation wraps around the window object and only executes if the window object is available.

When I was creating this blog and optimising it for mobile I experienced some default but not so user-friendly behaviour when navigating from one route to the other with Angular.

The problem is that content on mobile can go very deep below the initial viewport height. So when you're scrolling down and you press an internal link to another page, you'll be stuck at that height.

This is somewhat different from standard navigation between pages in a normal web-application, where the page reloads and you start from the top by default. In a S.P.A. this can easily be solved by scrolling to the top on navigation by using the native window.scroll function:

window.scroll(0,0)

A navigation in routing in Angular 1 and ngRoute or even the ui-router can easily be detected by listening to the event $routeChangeSuccess or $stateChangeSuccess. So combining these 2 essentials gives us:

// ngRoute:
$rootScope.$on('$routeChangeSuccess', () => {
    $window.scroll(0,0);
});

// ui-router:
$rootScope.$on('$stateChangeSuccess', () => {
    $window.scroll(0,0);
});

I did not find anything similar in the documentation of the router of Angular so I went digging. The fact is that I'm using the Angulartics2 plugin by @luisfarzati to track you guys' behavior :). This is also done on navigation so there must be something similar going on at that plugin. The plugin BTW works great!

Listening to navigation events in Angular

It seems that the Angular v2+ router has an events Observable property which you can subscribe on. Yes, it is as simple as that. Those events can be of any predefined type NavigationStart, NavigationCancel, NavigationEnd or NavigationError. In my case I only needed the NavigationEnd.

In the component that holds your navigation router-outlet you just need to setup the listener, something like this:

import { Component } from '@angular/core';
import { Router, NavigationEnd } from '@angular/router';
import { ViewportScroller } from '@angular/common';

@Component({
  selector: 'sv-app',
  template: '<router-outlet></router-outlet>'
})
export class AppComponent {

  constructor(
    readonly router: Router,
    readonly viewportScroller: ViewportScroller
  ) {

    router.events
      .filter(event => event instanceof NavigationEnd)
      .subscribe((event: NavigationEnd) => {
        // Angular v2-v6
        window.scroll(0, 0);
        // Angular v7+
        this.viewportScroller.scrollToPosition([0, 0]);
      });

  }
}

And that was it! Be aware that the window object might not be available in every context, except the browser. Check this awesome article by @juristr to read more about why you might want to wrap your window object reference!

Please also be careful not to use these events to do business logic, like for example checking if you can navigate to a specific route based on some authentication rules. For those cases you may want to implement guards! More information about guards can be found in this splendid article by @PascalPrecht of Thoughtram.

Originally published at https://samvloeberghs.be on November 19, 2016

Selecting subtitles in Vimeo using JavaScript

Sam Vloeberghs — Thu, 06 Jun 2019 05:52:55 +0000

Originally published at https://samvloeberghs.be on March 1, 2018

At SponsorMatch we deploy an Angular Universal website & application that is shown in different languages. Currently we support Dutch (nl) and English (en). On the homepage we use a simple introduction video / explainimation that is hosted on Vimeo to show how the product works. This animation video is made in English but for the Dutch-speaking market we provided Dutch subtitles.

Problem

It seems that it's not possible to set the default selected subtitle using the Vimeo iframe embed solution. It's probably possible but anyway, I couldn't find any information on Google or on the Vimeo documentation. There are even some questions on StackOverflow that remained unanswered for years:

Solution

There are several solutions possible and one is moving completely to the JavaScript API Vimeo provides. More information can be found here: https://github.com/vimeo/player.js

The second solution I used was combining the iframe embed solution with the JS API. Basically we just use the iframe that was used for embedding as parameter in the Player constructor:

Using it in Angular

Using it in Angular is fairly easy.. We need to install the @vimeo/player package using yarn or npm:

npm i -s @vimeo/player

Next we can import the Player directly and by using the enableTextTrack function we can easily set the language. As developers / implementors I believe we should be responsible in some way for knowing which languages are available to use. Meaning: which subtitles / texttracks are implemented on the embedded video.

import Player from '@vimeo/player';
const player = new Player(document.getElementById('intro_explanimation'));

player.enableTextTrack(this.language, 'subtitles').then(function (track) {
    // track contains language, kind and label
}).catch(function (error) {
    // error holds what happened in detail
});

Please take a look at the full example below. The end result is implemented on the homepage of SponsorMatch.

angular-vimeo-set-language-subtitles

DEV Community: Sam Vloeberghs

Lessons learned on offline capabilities with service workers using Workbox - The sequel

Target audience

What will we discuss in this post?

Better detection for a first-time install of a service worker

Using Workbox Window to solve the same problem

Visual representation

Updating the application while it's in use

Update application by user-intent

Handling the new but waiting service worker

Listening for a new service worker

Visual representation

Update application when it becomes visible again

Update application by setting up an interval

Conclusion

Further reading

Special thanks to

Angular v9 & Universal: SSR and prerendering out of the box!

Target audience

Example project

Adding Server-side-rendering (SSR) to your application

Getting started

The server run option

The serve-ssr run option

The prerender run option

Rendering more (dynamic) routes

Conclusion

Further reading

Custom plugins for Scully - Angular Static Site Generator

Target audience

What is Scully?

Getting started with Scully

Support for plugins

Example project

Custom router plugin

Configuring the new plugin

Custom renderer plugin

Configuring the new plugin

Using the scully-minify-html plugin

Conclusion

Further reading

Special thanks to

Creating a simple memory cache for your Angular Universal website or application

Target audience

Our example application

Simple diagram for basic rendering

Code for basic rendering

Why cache?

A simple memory-cache implementation

Our simple in-memory caching mechanism

Code for cached rendering

Cache only the existing pages

Possible extensions and improvements

Generate your static versions at build time

Create an interface for your cache

Use a separate caching layer

Server-side render only specific pages

Use a file based cache instead of memory cache

Conclusion

Further reading

Special thanks to

Applications or websites build with Angular Universal

Lessons learned on offline capabilities with service workers using Workbox

Target audience

What will we discuss in this post?

Serve the index.html for all deeplinks (while offline)

Shortcut available

Don't just blindly use skipWaiting() or clients.claim()

What could possibly go wrong?

Service worker active during first-time usage of your application

Be careful with what you cache in the browser

Avoid using global state in the service worker

Now what was going wrong?

So how to correctly share the OAuthToken with the service worker?

Conclusion

Further reading

Special thanks to

Outputting JSON-LD with Angular Universal

Target audience

First of all; What is JSON-LD?

The `server` run option

The `serve-ssr` run option

The `prerender` run option

Serve the `index.html` for all deeplinks (while offline)

Don't just blindly use `skipWaiting()` or `clients.claim()`

So how to correctly share the `OAuthToken` with the service worker?

What is a `Subject`?

What is a `BehaviorSubject`?

What is a `ReplaySubject`?

What is an `AsyncSubject`?