DEV Community: Sam Thorogood

Web Font Loading & The Status Quo

Sam Thorogood — Tue, 16 Feb 2021 00:00:00 +0000

Let's start with the obvious: there's lots of great posts out there on font loading (which all tend to be 27 pages long for some reason) and using the font-display CSS property, and… you get the idea. These all accept the status quo—that fonts cannot load synchronously like your CSS—and just describe ways to mask that.

But, it's my website, and I know exactly what fonts the user is going to need. So why can't I ask the browser to put a small font onto the critical path before a page displays at all? As an engineer, I find the lack of choice frustrating. 😠

I don't have a perfect soution, but this post lays out my gripes, a fallback solution via base64 encoding your fonts, and platform suggestion. To start, here's the fundamental issue, shown via animation.

While there's variants on this problem, there's two things happening here:

"Emojityper" displays with the system font first
The loaded font is bigger than the system font—we see layout shift, which I'm paid by my employer to tell you is bad (it is bad, but I'm also paid to tell you)

The status quo solution is to use the font-display CSS property (and some friends). And to be fair, traditional CSS can solve both of these problems. However, these issues are typically solved by not displaying the offending text until its font arrives—even though the rest of your page is rendered.

The most frustrating issue here is that this "flash" takes all of a few frames—maybe 50-60ms. This is the choice I'd like: to delay rendering by a small amount of time. My opinion on this UX is that users are going to be more delighted by a page ready-to-go rather than one effected by a flash that confuses a user's eyes for mere milliseconds. 👀

Case Study

On developer.chrome.com, we actually inline all of our stylesheets and images (largely SVGs) into each page's HTML in order to reduce the number of requests and make the page load faster. We're really happy with this solution, because for most users, their network is going to deliver that whole single payload incredibly quickly.

Despite this sheer duplication of assets across every HTML page, our fonts still go to the network, and new users will still see a flash.

Loading in general

For background on loading, see my recent interactive post. The TL;DR from that post is that the only thing that can block a page from rendering is loading external CSS. And for fonts, your browser will asynchronously load a font when glyphs from it are needed—e.g., for the heading font of this blog, that's immediately, but only once the stylesheet has first arrived.

Here, I'm actually using two tricks to get you the font earlier (although neither prevents the flash and layout shift):

I use <link rel="preload" ... /> to request the font early, although this only helps if you have an external CSS file (if it's inlined in <style>, the font URL is right there)
I also send the font via HTTP2 Server Push before any HTML goes to the user, although it seems like browser vendors are removing support for this due to misuse

Regardless of what you think this post, preloading your font is a good idea. Modern HTTP is very good at sending you lots of files at once, so the earlier your user's font can get on that train, the better. 🚂🚋🚋

Font files should also be fingerprinted and cached forever for future loads. I digress, but this loading issue—like so many—is only about the user's 1^st load. With the advent of service workers, we as web developers have almost complete control over the user's 2^nd load.

Solutions, today

This is a tricky one. We can actually include a font inline in your blocking CSS file—by base64 encoding it, which has ~33% space overhead. There's no extra network requests here and decoding is done in a blocking way.

@font-face {
  font-family: 'Carter One';
  src: url('data:application/font-woff2;charset=utf-8;base64,d09GMgABAAAAAG74ABI...') format('woff2');
  font-weight: normal;
  font-style: normal;
}

Many folks argue that base64 is a bad idea. Although, in that case study, the size of the image isn't listed—about 220k—and the author fundamentally disagrees with my assertion that fonts can be critical resources.

There is cost here, both in space and decoding time. If you're going to base64 a font to avoid the flash, how can you minimize the cost?

I find that most Latin custom fonts are about ~20k, and I wouldn't base64 anything substantially larger than that—keep it to a single font at most. (I'd use the system font for body text, and leave a custom font for your headings or hero text.)
Put the font declaration in a unique CSS file that's cached forever. Unlike the rest of your CSS, which you might change, the font is not going to change over time.

<!-- These will be downloaded in parallel -->
<link rel="stylesheet" href="./base64-encoded-font-eeb16h.css" />
<link rel="stylesheet" href="./styles-cakl1f.css" />

Only ship woff2—95%+ of users have support
This is advanced, but if you can control what your user gets on their 2^nd load (e.g., via a Service Worker), then you could serve the user a real, cached woff2 as well and then use only it for repeat loads.

Anti-patterns

There are other ways to ensure users don't see any part of your page before the fonts load. But they're going to involve JavaScript and that's just a rabbit hole that increases your site's complexity real fast. 📈

You could mark every part of your page as hidden via a CSS class, and then only remove it once you see a font arrive. You could do this via the Font Loading API or by literally measuring the rendering size of a test <div> until it changes. These are not good solutions.

(This is something I happily do on Santa Tracker, but we literally have a loading screen, lean in to a slow load, and the entire site requires JS. It's not suitable for sites.)

A standards plea

Last year, a proposal was made to add Priority Hints.
Right now, this proposal is just for hints about the importance of network traffic.

But perhaps it could include a hint choice of critical which informs a browser that this preload may block page rendering—if it arrives quickly, of course.

<!-- Preload this font and block until used, with limited budget -->
<link rel="preload"
    importance="critical"
    href="/carter-one.woff2?v11"
    as="font"
    type="font/woff2"
    crossorigin />

<!-- This could work for as="style", as="fetch" or others -->
<link rel="preload"
    importance="critical"
    href="/important-data.json"
    as="fetch"
    crossorigin />

This would allow for standards-based developer choice, and because it's a purely additive attribute, would have a sensible fallback for unsupported browsers (i.e., not to block the page at all). There's also a wide range of resources you can preload, so it could be a versatile tool. ⚒️

Summary

I find a lack of control over font loading frustrating, and using base64 for small fonts can help you if this problem frustrates you too. And if you find yourself trying to preload similarly-sized images 🖼️ to make your page work, that's actually one of the biggest signs this approach might help you—to me, that font is just as important as that site logo or navigation button. 🍔

To be clear, this can be a footgun—don't block page loading for minutes because 100k of fonts haven't arrived—use base64 sparingly to avoid a flash or layout shift. I don't think it makes sense for every site. I'm not even sure I'm going to implement this strategy on this blog.

Yet, to revisit the developer.chrome.com case study from earlier, where we happily inline images and our stylesheets. I don't think we should inline the fonts directly on the page—they're ~20k files which never change—but moving them to a synchronous, fingerprinted (and cached forever), stylesheet including just the base64 font may be on the cards.

➡️ Let me know what you think on Twitter.

Observing rendered DOM nodes

Sam Thorogood — Wed, 20 Jan 2021 00:00:00 +0000

When building webpages, we create and use HTML elements, but this is often a decidedly one-way interface. And while you can continually request information about how a node is being rendered through methods like Element.getBoundingClientRect() or window.getComputedStyle(), it's difficult to be notified when an element's render changes.

This post will explain how to be notified when:

an element is added or removed from the DOM
the bounding box of an element changes (i.e., resizes)
an element moves around the page for any reason

I personally enjoy having these approaches in my toolbox 🛠️ and I hope you find them useful too!

1. DOM addition and removal notifications

You might, in your toolbox, want to be informed when a particular DOM node is added or removed from the page. As a digression: for Web Components, this is really easy—Web Components provide connectedCallback and disconnectedCallback methods, which are literally called when the WC is added and removed.

Instead, this section is going to talk about doing this for arbitrary 'classic' DOM nodes, such as your friendly neighborhood <div> or <button>. There's actually no perfect solution, but read on 👇

Using ResizeObserver to track appearance

The ResizeObserver interface does what it says on the tin: it informs you whether a target element has changed in size. However, a little-known benefit of this is that it'll also tell you when that element is added or removed from the DOM (see the spec). This works because an element that's off the page has zero size—but this introduces an interesting caveat.

If your node has a style of say, display: none while it's on the page, it already has zero size: so as nothing actually changes when it's added and removed from the DOM. The ResizeObserver won't be triggered.

Instead, we can easily track the appearance of a single DOM node. This looks like:

/**
 * @param {Element} element to track appearance for
 * @param {(appearing: boolean) => void} callback when appearance changes
 */
function observeElementAppearing(element, callback) {
  const ro = new ResizeObserver(() => {
    const r = element.getBoundingClientRect();
    const appearing = Boolean(r.top && r.left && r.width && r.height);
    callback(appearing);
  });
  ro.observe(element);
}

This is very simple and doesn't require knowledge of the target element's context, such as its peers or parent. For some of my projects, I'm actually happy with the caveat: an element I'm interested in is gone because it's either off the page or has zero size. I don't care which one, I'll set up or tear down some code based on that.

Using IntersectionObserver

If you're worried about browser support, It's worth noting that ResizeObserver was only added to Safari in March 2020, in a 13.x release. Another helper, IntersectionObserver, was introduced a year earlier in 12.x, and has slightly wider support among other browsers, too. It's ostensibly for tracking the visibility of elements as they appear in your scroll viewport (to lazy-load images and so on), but it can also be used with arbitrary parent elements.

In this case, we can actually ask the browser whether an element has any of its size within document.body, and be informed when that changes:

/**
 * @param {Element} element to track appearance for
 * @param {(appearing: boolean) => void} callback when appearance changes
 */
function observeElementAppearing(element, callback) {
  const io = new IntersectionObserver(() => {
    const r = element.getBoundingClientRect();
    const appearing = Boolean(r.top && r.left && r.width && r.height);
    callback(appearing);
  }, {root: document.documentElement, threshold: 0.0});
  io.observe(element);
}

This code looks almost the same as above, and works the same way—we're not strictly told about removals, but rather, the apperance.

Using MutationObserver

There's also a helper called MutationObserver to help us track changes to DOM nodes. You can read about it on MDN. It replaces something called Mutation events, a long-since deprecated API with low browser support.

The main downside for MutationObserver is that you get a firehouse of all events, as the only way to reliably be informed about page-level changes is to observe the entire <body> element. (And if you're interested in Shadow DOM changes, you'll also have to observe individual #shadow-root nodes.) 🕴️

You can set up a global MutationObserver like this:

// We don't implement the callback yet.
const mo = new MutationObserver((entries) => { /* ... */ });
mo.observe(document.body, {subtree: true, childList: true});

The callback you receive will tell you about any nodes added and removed from document.body. However, it's important to note that only the "parent" of any addition or removal will trigger the callback.

What this means in practice is that you'll have to check the descendants of any node being changed in case you're interested in their status. If you want to get all individual added and removed nodes, you could traverse them:

/**
 * @param {NodeList} nodes
 * @param {Node[]} out
 * @return {Node[]}
 */
const traverseAllNodes = (nodes, out = []) => {
  out.push(...nodes);
  nodes.forEach((node) => traverseAllNodes(node.children, out));
  return out;
};

const mo = new MutationObserver((entries) => {
  for (const entry of entries) {
    const allAddedNodes = traverseAllNodes(entry.addedNodes);
    const allRemovedNodes = traverseAllNodes(entry.removedNodes);
    // do something with added/removed nodes
  }
});
mo.observe(document.body);

This is correct, but could be slow. If you're only interested in a small number of nodes changing, you could instead ignore entries completely and just check whether a target node .isConnected whenever the callback is fired.

⚠️ To be very clear, you cannot directly observe single nodes status in the DOM with MutationObserver, or even via the childList of a target node's parent. As the animation above shows, an element might disappear from the page because of something that happened to any of its ancestors.

2. Bounding box changes

This is really the smallest section of this post, and in many ways, is a superset of my suggested approach above. You can literally just use ResizeObserver, as—turns out—informing you of an element's resize is its primary goal. This looks like:

/**
 * @param {Element} element to track size
 * @param {(bounds: DOMRect) => void} callback when size changes
 */
function observeSize(element, callback) {
  const ro = new ResizeObserver(() => {
    const r = element.getBoundingClientRect();
    callback(r);
  });
  ro.observe(element);
}

…you could also dispense with the helper method and just use ResizeObserver directly.

Something I've often found useful is that it's valid to observe <body> (or <html>, which works the same way). This can tell you whether the whole whole page has changed size. 📄

Unfortunately, ResizeObserver won't tell you if an element moves—you can reposition the same DOM node around the page, and if its bounds don't change, this callback won't fire. (Read on!)

3. Move observations

As well as resize, you might want to know if an element moves on the page. This can be niché: it's your webpage, so you likely have a good idea if change you make (like a CSS class or maniplating DOM) will cause a move.

For me, like the above notifications, this approach is useful to have around in my toolbox when I'm building something tricky. The example I'll use below is that of a tooltip which exists unrelated in the DOM via position: absolute—I need to keep it positioned adjacent to my button of choice while not sharing any common parts of the element heirarchy.

Using IntersectionObserver

It's possible to overload the IntersectionObserver helper to detect moves. I introduced this above, but it's worth restating: if you were to read about this API, you'd believe it's for tracking element visibility—for lazy-loading, or seeing whether users can see your ads and so on. And the most common use case is to determine what proportion of an element is currently visible on the page, expressed as a ratio of its total size.

But it has a couple of interesting options we can play with:

we can choose a parent element to observe within—by default, IntersectionObserver uses the scroll viewport, not a specific element (we used this above to observe within document.documentElement)
we can set a rootMargin to expand or constrain the physical space being observed
we can set a threshold for callback

By observing document.body and getting creative with its rootMargin, we can construct a bounding box which fits around any specific element. If it moves, and our threshold is set to 1.0, we'll be notified—the element starts intersecting 100% with the target range, but as soon as it moves away from the bounding box we'll be triggered—as its visible ratio will go lower than 1.0.

There's a couple of nuances here. We also have to keep track of the size of the <body> element, because the right and bottom margins in rootMargin cannot use calc() (i.e., we cannot use say, the total width or height minus the offset)—so if it resizes, we have to recreate the IntersectionObserver.

So, with that in mind, the code roughly ends up like this (this has some issues, don't just copy and paste it):

const root = document.documentElement;

// Observe the whole document
const vizObservers = new Set();
const documentResizeObserver = new ResizeObserver(() => {
  vizObservers.forEach((fn) => fn());
});
documentResizeObserver.observe(root);

/**
 * @param {Element} element to observe
 * @param {(rect: DOMRect) => void} callback on move or resize
 */
function vizObserver(element, callback) {
  let io = null;

  const refresh = () => {
    io?.disconnect();

    // Inform the user that the bounding rect has changed.
    // If it's zero, we can't build an IntersectionObserver.
    const rect = element.getBoundingClientRect();
    callback(rect);
    if (!rect.width || !rect.height) { return; }

    // Construct the margin in the form "top right bottom left".
    // This needs to be -ve and always rounded _down_.
    const invertToPx = (value) => `${-Math.round(value)}px`;
    const rootMargin = [
      rect.top,
      root.offsetWidth - (rect.left + rect.width),
      root.offsetHeight - (rect.top + rect.height),
      rect.left,
    ].map(invertToPx).join(' ');

    // Watch for intersection change. Ignore the first update
    // as it should always be 1.0.
    let isFirstUpdate = true;
    io = new IntersectionObserver((entries) => {
      if (isFirstUpdate) {
        isFirstUpdate = false;
      } else {
        refresh();
      }
    }, {root, rootMargin, threshold: 1.0});
  };
  vizObservers.add(refresh);

  // Observe size, since size changes refresh.
  const ro = new ResizeObserver(() => refresh());
  ro.observe(element);
}

This is a pretty long snippet, but I've tried to add some comments. The core of this is building rootMargin: we need to find the insets from the sides of the root element, make them negative, and ensure they're rounded down—IntersectionObserver works on pixel boundaries, but DOM nodes can technically have floating-point size. 📏

⚠️ Due to this rounding, it's also possible that we get an initial callback of intersectionRatio slightly less than one—e.g., 0.9991451 or a very high floating-point value. The snippet above doesn't deal with this, but you actually need to recreate the IntersectionObserver at this point too. Due to the way it works, we're only told once we transition past any specific threshold—and in this case, we've already transitioned past the 1.0 threshold—we won't be called back again—so we need to create it again.

If you'd like to play more with this, I've built a demo ➡️ over on Codepen. I've also pulled out an improved vizObserver function as a small library you can find on GitHub. It's also worth noting that the way we track moves, by necessity, also ends up informing you about element resize and appearance (#1 and #2).

Summary

These raw primitives IntersectionObserver and ResizeObserver are very powerful and help us keep track of new and interesting things in ways that weren't possible before. They're largely supported by evergreens, although as of writing, ResizeObserver has slightly less support—it wasn't available until a 13.x release of Safari. That's about 15% of Safari users you can't support, although personally, I'll be embracing ResizeObserver in my web projects in 2021 anyway.

For me, I'll be using these primitives in a few ways, but I hope you find them useful in others, too. My use-case is mostly my last example: I want to align tooltips to arbitrary elements—which I don't want to directly pierce into, as I've written a good abstraction—even though they have no DOM in common. By keeping track of an element's position and size, I can ensure that the tooltip correctly "follows" the target.

Thanks for reading! Let me know on Twitter what you think. 🐦

Check your JS with TS

Sam Thorogood — Mon, 11 Jan 2021 00:00:00 +0000

Check your JS with TS

TypeScript is great and its static analysis and typechecking can help you be more productive, but swapping to TS wholesale isn't possible for many projects. You may also want to keep your project as pure JS, if, like me, you like not compiling when testing in your browser (just serve your files as ESM and let the browser do it).

So, you want check your JS—and this post uses ES Modules, or ESM—with TS tools. Great! This post has three levels of complexity, so read on.

Basic: Inline types in VSCode

As you hover over symbols in VSCode, you'll see inferred type information: for constants and so on you'll see string and number. This is the type that TS can safely guess. (If you see any, this means that TS can't work out what type you're using.)

You can fill in the gaps here with JSDoc comments which add types. There's a number of ways to specify them:

/** @type {number[]} */
const x = []; // otherwise TS thinks this is 'any[]'

/**
 * @param {Element} bar
 * @param {?Element} barOrNull
 * @return {Promise<void>}
 */
async function fooMethod(bar, barOrNull) {
  // do something with bar/barOrNull
}

/** @type {(arg: number) => void} */
const fn = (arg) => {
  /* ... */
};

// this is a _cast_, not a declaration: you need to wrap in parens ()
const nowIsNumberType = /** @type {number} */ (window['someExternalAny']);

Within any @type {...}, you can use a combination of TypeScript's type system as well as much of JSDoc. The possibilities for types are out of scope of this post.

So—this is fine, but all it gets you is helpful information when you hover over a type or you're trying to autocomplete. Let's get more useful feedback.

Intermediate: Write TSConfig

If you create a custom "tsconfig.json" file in the root of your project, you can enable warnings and errors for your project. The file should look something like this:

{
  "compilerOptions": {
    "checkJs": true,
    "noEmit": true,

    // if you'd like to warn if you're using modern features, change these
    // both to e.g., "es2017"
    "module": "esnext",
    "target": "esnext",

    // configure as you like: these are my preferred defaults!
    "strict": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,

    // "strict" implies this, but you'll want to enable it when you're
    // ready: it's a huge reason your project will start complaining
    "noImplicitAny": false,
  },
  "include": [
    // include the JS files you'd like to check here
    "src/**/*.js",
  ],
}

You can also use a base config but you'll still need to specify include as well as the top two compilerOptions to ensure we just check JS files.

⚠️ Keen observers might also notice that I've included JS-style comments inside the JSON as well as my favourite syntax feature, trailing commas. TypeScript seems to be completely fine with this extended syntax.

For free: VSCode

Once you create your "tsconfig.json" and make sure it matches your source files, you'll notice something amazing: VSCode will now just start warning you about problems.

VSCode can now show JS warnings, not even just about types

To be clear: I didn't install any TypeScript tooling to make this happen, it was just implicitly part of VSCode. Nice! 🎉

Command-line: TSC

You can also now run TypeScript via the command-line to get warnings and errors for your whole project, even if it's not compiling your code. Install the NPM package and run its command-line compiler (which will just check, since we set noEmit above):

$ npm install -D typescript
$ tsc

If your project has errors—and trust me, for any project where you've not typechecked before, you'll have them—this will print all of them and exit with a non-zero status.

Advanced: Write/use types

It's all well and good to use types like number and string[], but what if you'd like to define your own types—such as a complex interface type with many properties? There's actually many ways to do this in TypeScript, and some background is useful:

You can use the triple-slash syntax to bring in or reference other types
You're able to import type files, although this only makes sense to TS: not your browser

While the first approach is useful for say, external types—you might depend on something in NPM's @types repo or a built-in library—the second is my preferred option for your ESM projects.

Import your types

If you create a file like "types.d.ts", you can actually import it as "types.js" (and VSCode can suggest this in an autocomplete). TypeScript actually prevents you from importing the ".d.ts" directly-you must pretend that it is a JS file. But the JS file doesn't actually exist—how can this interop with other tools and loading in your browser?

Turns out, we can just create two files: one "types.d.ts" for types, and one "types.js" that's actually just empty. These two files might look like:

//
// @file types.js
//
// This is an empty file so that browsers and tooling doesn't complain.

//
// @file types.d.ts
//
/**
 * This isn't a real class, it just defines an expected object type.
 */
export interface ArgForSomething {
  foo: string;
  bar?: number;
};

/**
 * We can define functions, too.
 */
export function exportedFunction(arg: ArgForSomething): void;

And to use the code, in a regular JS file:

import types from './types.js';

/**
 * @param {types.ArgForSomething} arg
 */
export function foo(arg) {
  // ...
}

/**
 * If you export a function from your types, you can also just reference it
 * wholesale: this might be useful if you're publishing to NPM.
 *
 * @type {types.exportedFunction}
 */
export function exportedFunction(arg) {
  // ...
}

Voila—type information!

Importantly, when you're bundling or compiling, tools will hide the dummy, empty file. And during development, the file technically exists, but is ignored since it's empty anyway and is only referenced inside your comments.

Other approaches

I mention the classic approach for completeness, but this post is really about treating ".d.ts" files as modules. Skip this section unless you're really interested.

So, you can reference other files in your own project using the triple-slash syntax. However, it doesn't mesh well with modules: you can't see anything that had export on it in that referenced file, and everything else will be brought into the global namespace. (There's exceptions here too, and it's just more complex than treating it as an ES Module.)

Export types for others

If you're not publishing to NPM, you can stop reading. But if you are building something that can be consumed further, then read on.

By default, TypeScript looks for the "index.d.ts" file in your project's root directory to provide types for the users of your package. In the example above, I've intentionally not used that name as I think creating an empty peer "index.js" in the top-level of your project is probably going to lead to confusion. I like specifically calling it "types".

You can specify a path to your types in "package.json". (Turns out, TypeScript recommends you do this anyway, even if the file is the default name.) This looks like:

{
  "name": "your-awesome-package",
  "types": "path/to/types.d.ts",
  "exports": {
    "import": "./main-module.js",
    "require": "./main-require.cjs"
  }
}

This types file should match your top-level export file. It can import further ".d.ts" files (and these don't need a dummy peer JS file) and even re-export them.

⚠️ As of writing, TypeScript doesn't support subpath exports. There's some workarounds in that thread.

Summary

Your JavaScript can benefit from TS' static typechecking and errorchecking abilities. It can also reveal a whole bunch of errors or risky behavior you didn't know you had—but hey, that's progress for you. The tools you regularly use—including VSCode, but command-line "tsc" too—are so close to being incredibly useful, even for pure JS, and by giving them the right config, you can get a lot more data.

And of course, while static analysis is great, it's also not a replacement for good tests. Go forth and check your code!

The System Font

Sam Thorogood — Thu, 06 Aug 2020 00:00:00 +0000

tl;dr: as of August 2020, use the following snippet to get the system font on your webpages, on most platforms, with sane fallbacks.

body {
  font-family: Segoe UI,system-ui,-apple-system,sans-serif;
}

If you'd like to know more about how I got to this line, read on.

Background

There's been a recent, major development in web fonts. The "system" font family is now widely supported—it came on the scene, technically in about 2015, and was supported by most browsers in about 2017. And a few years later, in 2020, it's incredibly common to see this used.

What is the system font? If you use it, your webpage will display like the platform it's being read on. I'm not a designer, but I find it visually appealing and it helps simple webpages fit in well with their environment. If you're reading this article on https://whistlr.info, you are in fact reading it in the system font! 🤯

There's some great articles out there that have comparison screenshots. This does not impact font loading—the point is these fonts are already on your user's machine, and are just a good replacement for classic "Safe Web Fonts", e.g., Arial and Times New Roman.

So: Chrome, Chromium-based browsers and Safari have shipped the system font as system-ui. You use it like this:

body {
  font-family: system-ui; /* don't just do this: read on! */
}

But, like all things on the web, it's never as simple as that.

In Practice

Firefox is still yet to come to the party, instead only supporting the system font on Mac, and under a specific name. Additionally, it's been observed that—surprise surprise—on Windows, the system-ui font can in fact end up resolving to literally anything. I suspect the reason no-one noticed this 'issue' is that on every other platform, this system font is fixed, and designers tend not to use Windows. 🔥

In practice, this means most sites will end up with an ungodly mess of CSS like this:

body {
  /* from GitHub, August 2020 */
  font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,Arial,sans-serif,Apple Color Emoji,Segoe UI Emoji;
}

…which serves mostly to avoid using system-ui, and instead has:

-apple-system for Safari and Firefox; plus BlinkMacSystemFont, the same for Chrome
Segoe UI for Windows (originally included with Windows Vista, so it has great coverage)
some "Safe Web Fonts" for very old browsers: Helvetica and Arial
sans-serif
and finally, support for emoji (this isn't really needed—I'll get to this later, but it could have a whole blogpost on its own).

GitHub's code excludes a pretty default for Linux. Various distributions of Linux have their own fonts: Ubuntu has Ubuntu, and Red Hat has… unsurprisingly, Red Hat. I've included GitHub's code as it's pretty good, but… it could be simpler.

Just Tell Me What To Do

Sorry, I know this is why people read articles. My general advice for system font loading in August 2020 is to do this:

body {
  font-family: Segoe UI,system-ui,-apple-system,sans-serif;
}

This works basically by:

specifying Segoe UI for everyone on Windows: this works for modern browsers as well as IE11 and legacy Edge
using system-ui for most modern browsers on macOS, Android, iOS, Linux, Chrome OS, …
using -apple-system for Firefox on macOS, and a tiny fraction of older Safari versions
finally, falling back to sans-serif for all other cases, including ancient browsers

Hooray! 🎉

Some Notes

So there's always some notes. These are pretty obscure, and I don't think they should stop you using the snippet.

Sans-Serif Fallback

The fallback will apply to ancient browsers. It'll probably map to a font like Arial.

The fallback will also be used by Firefox on Linux and Android. Some guides actually suggest using a longer list of fonts—e.g., Ubuntu, Oxygen—to catch all possible Linux distributions. But, in my limited testing, the default sans-serif provided by Ubuntu and other distributions are quite appealing.

Emoji

On macOS and Linux—both Firefox and Crome—the system font has some problems with old emoji. It doesn't matter if you use -apple-system or system-ui: the same problems occur. If this is a black and white emoji [✈️️], then congratulations, you're expericing the issue!

The airplane emoji above includes what's known as a VS16 character, indicating that it should be rendered in color. The airplane actually predates modern emoji, so without the VS16, it renders in black-and-white. However, the system font—in these cases—isn't respecting the VS16.

Confusingly, emoji like the telephone—which also require a VS16 character—work fine. Here's the phone with a VS16 (☎️), and without (☎). This implies to me that this is a bug in either the effected browsers or the platforms, or both. (Safari has no such issues on Mac.)

In my experience with Emojityper and friends, to guarantee emoji support for all emoji, you need to specify actual fonts and can't rely on the system font.
Sites like GitHub, which embrace the system font, have a secret: they replace all their emoji with images anyway. This is probably just to support even the newest emoji on all platforms—but it has the effect of 'fixing' this underlying issue with system-ui too.

As an aside, if you include Segoe UI Emoji in the font list above, Windows will display text in emoji form even without the VS16 character. This is probably against Unicode's specification.

Segoe UI

Finally, a note about Segoe UI. Technically, as I list it first, it could be possible for it to win out—even on Mac or mobile where it doesn't fit—if a user has installed the font locally. However, it's against Microsoft's font policy to redistribute Segoe UI, so in practical terms it should never be available anywhere but on Windows.

Final Thoughts

Thanks for reading. Some thoughts, including on design:

Use the short blurb of CSS included at the top of the article to use the system font.
Personally, I find the best place for the system font is in simple pages—like this blog!—where there's no huge visual style or theme and you'd like to match the platform.
Don't be afraid to use another font for contrast, even if you're including that font via CSS.
And while it's out of scope of this post, read up on CSS properties like -webkit-font-smoothing and text-rendering to tweak your fonts until they're just right.

Control Loading Spinner in HTML

Sam Thorogood — Tue, 21 Apr 2020 06:48:33 +0000

Control Loading Spinner in HTML

If your web experience involves real HTML pages and actual <a href="..."> links, navigation will happen as a normal browser action: your users will see some kind of loading indicator or spinner that's built-in to the platform.

However, if you've adopted the single-page application idiom, this won't happen for free. You'll typically be loading new data with a fetch() and then hydrating your page with the content. This is pretty common for app-like experiences, and lots of even pure content sites do it too—web.dev to name one.

It's time to show you how to fake this experience. I don't endorse this for production, but it's a cool demo.

The Demo

an early version of this demo for users not on Whistlr

⚠️ This is broken on iOS, but I'm looking into a workaround.

Neat, huh? Let's dissect how it works.

How It Works

Firstly: I'll admit it. I am actually loading a new page that does have a real URL. But instead of your natural instinct to reply to HTTP requests as fast as possible, we instead delay its completion until a point in the future that we control.

When the request finally resolves, reply with a HTTP 204. Importantly, and to quote MDN:

The HTTP 204 No Content success status response code indicates that the request has succeeded, but that the client doesn't need to go away from its current page.

With these two parts holding hands together—like a knife 🔪 and a wrench 🔧—we can create the illusion of loading, while not actually doing anything at all.

Using a Service Worker

The easiest–but not the only—way to integrate this with your existing site is to rely on a Service Worker. We can add a fetch handler that accepts a request to e.g., "/204_delay?key=...", and lets you finish the request later. Something like:

self.addEventListener('fetch', (e) => {
  const u = new URL(e.request.url);
  if (u.pathname === '/204_delay') {
    const key = u.searchParams.get('key');
    const handler = async () => {
      await new Promise((r) => activeLoads[key] = r);
      return new Response(null, {status: 204});
    };
    e.respondWith(handler());
  }
});

We'll also need to define activeLoads and a way to invoke them—perhaps via a message:

const activeLoads = {};

self.addEventListener('message', (e) => {
  if (e.data.command === 'done') {
    const r = activeLoads[e.data.key];
    delete activeLoads[e.data.key];
    r && r();
  }
})

Now, on our foreground page, if we've got a valid service worker, your async loading code can be wrapped like:

async function loadPage(url) {
  const {controller} = navigator.serviceWorker.controller;
  const key = String(Math.random());
  if (controller) {
    // cause a real but fake navigation
    window.location.href = `/204_delay?key=${key}`;
  }

  await loadDataAndUpdatePage();  // ¯‍\‍_‍(‍ツ‍)‍_‍/‍¯

  if (controller) {
    // tell the SW to finish the navigation
    controller.postMessage({command: 'done', key});
  }
}

And, as I mentioned: your browser will actually load a new page, with a real URL. But as it's a 204, nothing will change.

Using a Custom Server

The demo in this blog post talks to a real webserver which acts in exactly the same way as the Service Worker, above. One request "hangs" until another request causes the server to complete the first.

It's much more dangerous: if there's a network error or your client is offline, the request to "/204_delay" will actually fail, showing an error message and closing your site. And by the time a navigation has occured—which includes if a user clicks on a real link, we as JS developers can't actually stop that intent.

Caveats

The loadPage() example above shouldn't be run in parallel, as your browser can only handle one navigation at once.

Some browsers—and of the big three, that's currently just Safari—will treat navigation as a signal to stop active network requests. Navigation does actually cause beforeunload to be fired, and that means fetch() will fail. You can work around this by fetching data you need inside your Service Worker, as postMessage is still allowed. But this isn't a very natural way to fetch content.

Safari will also stop DOM changes from being rendered, even in the same frame as navigation starts. In practice, this means that for the demo above, I create the "Loading..." text and then only cause our faux-navigation to start inside a requestAnimationFrame handler.

Finally, in the example above, our Service Worker might end up leaking memory if say, navigation is cancelled by the user closing their page. This is solvable but is tricky because it seems like clientId isn't available everywhere. Chrome only provides resultingClientId, which will be the client if the navigation completes—except we can never navigate to a 204. In practice, you should probably just implement a timeout.

Parting Thoughts

You could also pretend to load by rapidly animating your favicon. But that wouldn't be nearly as satisfying as triggering a native-like experience, especially as the favicon is not the only source of loading UX.

And as I mentioned all the way at the top, I don't condone this approach. Don't @-me.

Async Generators for User Input

Sam Thorogood — Wed, 15 Apr 2020 00:00:00 +0000

Async Generators for User Input

It's possible to build JS, on the web, with a native-like event loop.This is thanks to async generators and the await keyword—are you sick of a twisty maze of addEventListener, timeouts and global state?Read on.

Background

Generators allow us to suspend normal program flow as the interpreter jumps to/from your yield statements:

function* generator() {
  yield 1;
  console.info('between');
  yield 2;
}
for (const value of generator()) {
  console.info(value);
}

This program will print out "1", "between", and "2".Used at face value, this primitive allows you to create something which vaguely looks like an array.

This isn't all, though: from Chrome 63 and friends, you can perform asynchronous work between each yield statement (to be fair, you could already yield a Promise).The syntax isn't too different, either: just add async to your function.

Event Loop

Most documentation about JS' event loop correctly identifies it as event-driven.This is the normal JS model—if you addEventListener, your handler is called, and is expected to complete synchronously.

Instead, let's aim for something more akin to a native event loop, which you could use like:

(async function run()) {
  for await (const event of loop) {
    if (event instanceof MouseEvent) {
      // If the user clicked on something, wait for their result.
      if (event.name === 'click' && event.target === button) {
        await getUserInput('Rate your experience:');
      }
    } else if (event.type === 'keydown') {
      // Submit the form
      if (event.key === 'Enter') {
        // TODO: ...
      }
    }
  }
}());

This is basically implementing a state machine—a DFA, where the states are controlled by user input.This is especially userful for complex user interactions—like forms, or games.

a train game with two input states: choosing an edge, and drawing a track

There's a few decisions you might have to make about the input, though.

As you're now consuming time—potentially asynchronously via await—to process each event, it's unlikely that your code will be able to handle every event as it arrives.For example, if you're processing click events but you do a network round-trip, a user might generate more clicks before the first event is done.This might be intentional, but you'll need to decide what's important to queue up for processing later.

What does the loop object look like, then?Well, you can build a generator and a helper to push events into it:

export function buildEventManager() {
  let resolve = () => {};
  const queue = [];

  // (there's no arrow function syntax for this)
  async function* generator() {
    for (;;) {
      if (!queue.length) {
        // there's nothing in the queue, wait until push()
        await new Promise((r) => resolve = r);
      }
      yield queue.shift();
    }
  }

  return {
    push(event) {
      queue.push(event);
      if (queue.length === 1) {
        resolve(); // allow the generator to resume
      }
    },
    loop: generator(),
  };
}

This is a bit of code, but it basically just has two parts and a connection between them.First, a push() method, which lets you control what events are handled, and pushes them into the queue.Secondly, a generator—which we run, and return as loop—which waits for events to appear, and uses yield to provide the next available one.

To use it purely to keep a queue of all the pointer events that occur, try this:

const {push, loop} = buildEventManager();

window.addEventListener('pointermove', push);
window.addEventListener('click', push);

(async function run() {
  for await (const event of loop) {
    console.info('mouse now at', event.screenX, event.screenY);
  }
}());

This simple example just queues everything, rather than trying to e.g., only provide the most recent movement event.

Not Just User Events

One of the benefits of a generic event loop is that we can process any sort of event we imagine, not just user-generated ones.For example, we could push some custom events and process them in your event loop inline with everything else:

window.setInterval(() => {
  push(new CustomEvent('tick'));
}, 1000);

const ro = new ResizeObserver(() => {
  push(new CustomEvent('resizeElement'));
});
ro.observe(someElement);

… of course, you're not just limited to custom events (push accepts any object), but this might match the rest of the inputs you're processing.

State Machines, Oh My

I mentioned that this native-like event loop helps us create state machines.If you just have a single event loop, that's not really true, because you might still have to manage global state yourself.Instead, you can actually use the loop many times.

Unfortunately, using for await (...) actually doesn't work here, as you're seemingly not able to use it more than once (I may need to read the ECMAScript spec to find out why).Instead, we can use a generator's .next() method:

(async function run() {
  for (;;) {
    const {value: event} = await loop.next();
    if (event.type !== 'click') {
      continue;
    }
    console.info('start line at', event.screenX, event.screenY);

    for (;;) {
      const {value: innerEvent} = await loop.next();
      if (innerEvent.type === 'click') {
        console.info('draw line to', innerEvent.screenX, innerEvent.screenY);
        break;
      }
    }
  }
}());

In this example, we wait for one click, then another.It's a simple example, but shows how you might build up some interesting state—you've started at the default state, then moved to the "waiting for a second point" state, then you're able to draw a line.

Digression

As a digression, while it's not an error to write something like:

element.addEventListener('click', async (event) => {
  const result = await (fetch('/foo').then((r) => r.json());
  // update the DOM
});

… it is basically an antipattern because the async keyword masks some possible issues:

You could end up handling the event several times in parallel, e.g., submitting a form many times before it's complete
Errors aren't caught anywhere (they would appear as an "unhandled promise rejection")
Importantly, it appears like the event handler is synchronous and blocking, even though it's not

You can work around the 1^st issue with something like cancellable async functions.

Conclusion

What I've covered here is an incredibly basic example of processing events in a queue.You'd want to inspire from these ideas and write something appropriate for your use-case: I know I will for my web-based games.

One key difference from actual native event loops is that on the web, we can't (and probably don't want to) turn off every bit of built-in behavior your browser provides for you.I'm not suggesting that you handle every keystroke in an <input type="text">, or overload every click event.But this lets you control perhaps a tricky component which transitions through a bunch of states, especially if it has complex user interactions—keyboard, mouse, and so on.

Finally, there's a lot of reading out there about state machines via async generators—that's great, and I'm glad to see many folks taking advantage of this idea.I've not seen a huge amount of writing about processing user events, though, and that's where I want you to consider the possibilities.

Node.js Streams & Object Mode

Sam Thorogood — Fri, 10 Apr 2020 00:00:00 +0000

Node.js Streams & Object Mode

Streams in Node.js serve two purposes.The first, more commonly documented use-case is that of reading and processing bytes a 'chunk' at a time: bytes which most commonly come to/from your local disk, or are being transferred over a network.Secondly, you have {objectMode: true}, which I'll explain later.

tl;dr: Streams for bytes are rarely useful, and objectMode can be made better through parallelism (which most libraries don't take advantage of).

Chunks of Data

The classic use of stream works with files or file-like objects directly.For example, you might read, compress and write out a file, which looks a bit like:

import fs from 'fs';
import {createGzip} from 'zlib';

const src = fs.createReadStream('image.tar');
const transform = createGzip();
const dest = createWriteStream('image.tar.gz');

// now connect! \o/
src.pipe(transform).pipe(dest);

Without streams, you'd have to read the whole file at once rather than processing it in, well, a stream of smaller chunks.So, you basically want this for one of two reasons:

a file is enormous so loading it all into memory at once is wasteful
or; your task involves a network in any way (so you can send or use bytes immediately)

Interacting with the network is a natural fit for streams, because the network is slow.You don't want to block until a whole file is a in memory, and only then send it over a relatively slow connection to your users.

However, when you're just processing files as part of a script, streams are rarely useful.Tasks like compression make great examples, as their algorithms just don't need the whole file at once.But anything outside this definition doesn't work this way—think compiling source files, resizing images—in places where the full context is required, streams don't make sense.

To look at a popular build tool, Gulp, only one of its top ten popular plugins supports streaming mode (gulp-replace).As an aside, Gulp is an interesting example, because a lack of support in any given plugin will always throw an error—it's actively hostile to streams.

Speed Comparison

fs.readFileSync is much faster than streaming data

If your goal is to just read a whole file into memory as quickly as possible, fs.readFileSync is the champion, with fs.readFile (the callback-based version) slightly behind.Building the most simple of streams—just getting every chunk and putting it into an array—is about 2-3x slower.

Interestingly enough, is that using fs.promises.readFile is about 2-3x slower again.If you're worried about performance, maybe avoid it for now (Node v13).

Object Mode

Where streams come into their own, however, is when they're used in {objectMode: true}.Instead of transferring 'chunks' of a file, you can use them to move literally any object.

Let's start with an example—again using Gulp.Gulp is described as a "streaming build system", but 'streaming' here refers specifically to the Vinyl file objects it generates—which are passed through plugins with objectMode.

Gulp's tasks look a bit like this:

gulp.task('images', () => {
  return gulp.src('images/*.{png,gif,jpg}')
      .pipe(imagemin())
      .pipe(gulp.dest('dist/images/'));
});

The first call, gulp.src, starts a stream that reads files matching a certain glob.It then pipes the stream to a transform which modifies each file (in this case, minifies them) before finally to a writer that puts them on your disk again.

This is a powerful primitive, but it has some serious caveats.Some of them seem like poor design choices in Node which would be impossible to change after so many years.

Caveats

Streams, and the way they pipe together, aren't useful for constructing some permanent structure or scaffold.

In the following example, we generate helper streams that emit some numbers.Once the first Readable is complete it will call the .end() method of the thing you've piped through to.Have a read:

import stream from 'stream';
import * as transforms from 'async-transforms';

const doSomething = transforms.map((each) => console.info(each + 1));

stream.Readable.from([1, 2, 3]).pipe(doSomething);
setTimeout(() => {
  // the setTimeout shows that the above Readable
  // is closing the stream after a frame
  stream.Readable.from([4, 5, 6]).pipe(doSomething); // won't run, won't crash
}, 0);

The program will only output "2, 3, 4" (the values passed in the first stream).To be fair, we can fix this behavior by passing {end: false} as an option to the .pipe call—but this is something the user of a stream decides on, not the author.

To put it another way, if you're writing a library that exposes a stream target, any part of a program that uses your stream can cause you to stop receiving input.This is a challenging place for library authors to be in.

Streams also historically don't do anything sensible with errors.The .pipe() command doesn't forward errors—check out a bunch of other posts about why this is bad.

The modern solution to errors, and which also makes streams a bit nicer to write code for, is the stream.pipeline static method.This was added in Node v10, seemingly as a concession that the previous model hasn't worked that well.

Paralellism in Object Mode

One of the great reasons to use objectMode streams is part of a build process or similar pipeline.The reason I've used Gulp as an example is that it pioneered this approach.

However, not every implementer of a transform or other parts of a stream gets parallelism right.This is the relevant line from Node.js' docs:

transform._transform() is never called in parallel; streams implement a queue mechanism, and to receive the next chunk, callback must be called, either synchronously or asynchronously.

If you were to implement a transform as the docs suggest:

const t = new stream.Transform({
  transform(object, encoding, callback) {
    doComplexTask(object, (result, err) => {
      callback(err, result);
    });
  }
});

... then only a single chunk will be processed at once.This probably makes sense for chunked binary data, but rarely for objects.Let me explain.

If you're writing a transform that compiles SASS or resizes a number of images on-disk, and each object is a filename or file contents, then it's unlikely that the order of the output matters, or that each task is related in any way.

So, naïvely, you can run all your tasks in parallel by:

const t = new stream.Transform({
  transform(object, encoding, callback) {
    callback(); // "done" already, send me more please
    doComplexTask(object, (result, err) => {
      err ? this.emit('error', err) : this.push(result);
    });
  }
});

The tradeoffs of this simple design are basically:

tasks will complete in any order
you'll run all the tasks as fast as they arrive—maybe they shouldn't all run at once, even for CPU or memory-related reasons

(The second point actually doesn't matter too much in Node.js, as it's single-threaded: at most, you could burn through one of your CPUs, unless your "complex task" is actually spawning another thread.)

Async Transforms Library

Yes, this post is mostly advertising for a library I've written.The async-transforms package has a number of stream helpers which implicitly work in parallel, and allow you control the number of tasks to use.

import * as transforms from 'async-transforms';

const compileTransform = transforms.map(async (file) => {
  const {code} = await expensiveCompileTask();
  return code;
}, {tasks: 4});

It's also got a number of other helpers, including farming work out to a Worker inside Node.js (so CPU-bound tasks can run on their own thread).I'll let you take a read of its documentation, but it works around both tradeoffs of the naïve design, above.

_As a fun aside, one of the most egregiously poor bits of JavaScript advice I've seen on the internet is that forEach and map on your stock-standard Array type work in parallel.Nothing could be further from the truth—these are clearly defined to work step-by-step._Proper use of streams, however, can get you close to that vision.

Summary

Streams are confusing.You probably don't need them for reading or writing bytes, unless you have a niche use-case (including interacting with the network).

Streams in objectMode have more utility, but know that they're not a silver bullet to all types of program flow, and that most folks implementing transforms build them in a completely serial way.

Logging with App Engine and Stackdriver

Sam Thorogood — Wed, 08 Apr 2020 00:00:00 +0000

Logging with App Engine and Stackdriver

In days of old, App Engine's logging statement looked something like this:

c := appengine.NewContext(r)
log.Infof(c, "You made a log! Here's a thing: %v", thing)

This appeared in Stackdriver's logging page as a line correctly attributed to the corresponding HTTP request.And if you call e.g., Warningf, or Errorf, the severity of the request itself is increased to match (to the maximum level of severity you log).

Easy, right? Well, not anymore.

⚠️ While this post uses Go, it applies to all App Engine runtimes.This is especially true as the reason behind App Engine's massive changes is that by removing functionality, more languages are supported 'for free'.

New App Engine

Since the go112 runtime, App Engine, as we knew it for 10+ years, has been effectively deprecated.It's now a place to run a HTTP server with a bit of magic around the edges.

Because of this, you now have a couple of points to consider when logging. For background, remember that App Engine still generates a default HTTP log for every request.

If you simply follow the documentation on how to log for go112 and above, you'll encounter two fundamental issues:

the logs you generate won't be associated with the current HTTP request
and, each log event you generate will appear inside Stackdriver on its own line , not grouped together

Of course, you can see these contextless log messages adjacent to incoming HTTP requests, which could be useful for long-lived tasks.But it's now difficult to 'at-a-glance' see logs generated due to a HTTP request in context.

How To Log

For background, to log from user code, you can either:

Log to stdout (via the built-in log package) or by literally printing to stdout or stderr
or, log to a named log via Cloud Logging
- Confusingly, the named log you choose can also be named stdout or stderr.
- The name doesn't really matter except that Stackdriver's Log Viewer will show stdout, stderr and your App Engine logs by default.

Additionally, if you print JSON to stdout, it will be treated as a structured log (as if you called Cloud Logging with "stdout" as the log name).This is badly documented, but there are third-party loggers that can help.

Associate Your Logs

App Engine's documentation is vague on how you associate log events with the current HTTP request.Let's go through the steps.To associate your logs with the top-level, default App Engine log for the HTTP request, you'll need to:

Parse the X-Cloud-Trace-Context header, which include the Trace ID, Span ID and an optional flag
Insert the Trace ID into a string like "projects//traces/"
Use this string as the Trace field in a structured log
Ensure you're passing a valid mrbp.MonitoredResource that describes the log as Type: "gae_app"

This will ensure that your log is nested with the HTTP request, based on Trace and Type.

shows the nested log statement

However, this log will still exist at the top-level—it's just being nested by Stackdriver's UI.A simple workaround here is to use a log name that's not shown by default (only stderr, stdout and the App Engine default logs are shown), so it won't clutter your view.

⚠️ You can set the HTTPRequest field of the log entry.But this will appear as if a whole other HTTP request has occured (as it'll use the structured log format and display "GET", the path, etc) for every individual line you log.

Putting It Together

The code looks roughly like this:

import (
  "fmt"
  "os"
  "strings"
  "cloud.google.com/go/logging"
  mrpb "google.golang.org/genproto/googleapis/api/monitoredres"
)

const (
  projectName = "your-project-name"
  // you can also use os.Getenv("GOOGLE_CLOUD_PROJECT") for this in prod
)

var (
  lg *logging.Logger
)

func init() {
  ctx := context.Background()
  loggingClient, _ := logging.NewClient(ctx, fmt.Sprintf("projects/%s", projectName))
  resource := logging.CommonResource(&mrpb.MonitoredResource{Type: "gae_app"})
  lg := loggingClient.Logger("your-app-appengine-client-log", resource)
}

func httpHandler(w http.ResponseWriter, r *http.Request) {
  traceId := strings.Split(r.Header.Get("X-Cloud-Trace-Context"), "/")[0]

  lg.Log(logging.Entry{
    Trace: fmt.Sprintf("projects/%s/traces/%s", projectName, traceId),
    Payload: "Yes, your log message finally goes here",
    Severity: logging.Info,
  })
  defer lg.Flush()
}

Of course, you probably want to write a helper.Simple. 🤨

Caveats

You can't modify the severity of the default App Engine HTTP log.While this is mentioned in the docs, it's actually an error—there's no way to do this.

You also can't really test this locally, as App Engine no longer runs via the dev_appserver, so no magic headers are provided to you.Local code just won't see the X-Cloud-Trace-Context header.A quick way to test if you're in production or not is:

  projectName := os.Getenv("GOOGLE_CLOUD_PROJECT")
  isProd := projectName != ""

Alternatives

App Engine is no longer designed to help you associate simple log output with its default HTTP logging and provide helpful 'at-a-glance' information.So, let's not work against it: another option is to write our own logs.

Parallel To App Engine Logs

As we know, App Engine generates default HTTP logs.They can't be disabled, which means if you insert additional log statements, you might be fooled into thinking that your application has twice the number of requests.

However, if you create logs under a different log name, and aggressively use a different search inside Stackdriver (as you can't set a default), it's possible to see just your own log lines.

You'll need to create two different types of logs.

The parent log (this maps to the App Engine log we're trying to replicate)
Any individual log statement (generated from a classic Logf-style function)

Confusingly, you should create the parent entry last, because it contains information you only know at request completion—e.g., the response size and the request latency.You don't have to specify this data, but Stackdriver will show "undefined" for several fields without it (Stackdriver has a UI for custom fields, but it aggressively tries to include an undocumented number of HTTP-related fields regardless).

As I mentioned before, Stackdriver will associate requests with the same Trace ID.Since we're not logging a real request, you can just make one up.I suggest deriving something from the real ID.

Here's how you might log individual log lines (part 2, above):

res := logging.CommonResource(&mrpb.MonitoredResource{
  Type: "gae_app",
})
fakeTraceID := "_"+r.Header.Get("X-Cloud-Trace-Context") // derived by adding a char
clientLogger, err := loggingClient.Logger("events", res)
err := logger.LogSync(r.Context(), logging.Entry{
  Payload: "I'm a log message!",
  Severity: logging.Info, // you decide the level
  Trace: fakeTraceID,
})

Next, you can log information about the whole request (part 1, again, typically after your request is complete):

parentLogger, err := loggingClient.Logger("sane_requests", res)
err := logger.LogSync(r.Context(), logging.Entry{
  HTTPRequest: &logging.HTTPRequest{
    Request: r, // use incoming *http.Request from http handler
    RemoteIP: r.Header.Get("X-Appengine-User-Ip"), // not in App Engine's *http.Request

    // nb. These can only be found by wrapping your handler.
    ResponseSize: 1234,
    Latency: time.Millisecond * 1234,
    Status: 200,
  },
  Payload: payload, // the top-level payload is totally freeform (JSON or text)
  Severity: logging.Warning, // you decide what this is
  Trace: fakeTraceID // from previous example
})

... phew.There's definitely room for a library to help you here, and then as a reminder, you'll have to ask Stackdriver to show you the "sane_requests" log.

Orthogonal to App Engine logs

App Engine is going to continue generating its own logs.Many of these logs are likely completely boring: requests for static assets, redirections, etc.

Rather than trying to replace the built-in behavior, another suggestion is to just create logs for the most interesting of your handlers.You can follow the above guidance to insert HTTP requests but remember that the request is mutable and something you can fake—or even not provide at all.

While I mentioned the list was undocumented (it is), I've observed that Stackdriver will show the following fields in its default view:

httpRequest.requestMethod: usually 'GET' or 'POST', but could be 'FOO'
httpRequest.status
httpRequest.responseSize: parsed as bytes
httpRequest.latency: parsed as a time
httpRequest.userAgent
httpRequest.remoteIp: only displayed when the log event is expanded

If any of these fields exist, then Stackdriver will try to display all of them.So the choice is up to you: add single text logging events for the progress of your loggable event, and then provide the top-level logger, which can contain a real, fake or no HTTP request.

Troubleshooting

When writing this blogpost, I found that my App Engine's Service Account (in the form appid@appspot.gserviceaccount.com) didn't have permissions to write logs.I think this is because my app is actually quite old–it predates the Cloud Console.

adding a service account to the right groups

If you see security or other errors, you might need to add the service account to your project (it doesn't always show on the IAM page) and give it "Logs Writer" access.

Parting Thoughts

None of these solutions are ideal.There is an official bug from 2017 which I referenced to write this post.Since this behavior remains the same in early 2020, I don't think there's any plans to simplify logging again.

Progress Indicator With Fetch

Sam Thorogood — Thu, 20 Jun 2019 13:50:14 +0000

A quick tip: in a previous demo, I showed how we can download a large file to seed the content for a Service Worker. If you look fast enough, you'll see a progress indicator. (Although for a small file, blink and you'll miss it!) 👀

The code is pretty simple. Let's start with a simple async fetch:

async function downloadFile(url) {
  const response = await fetch(url);
  const arrayBuffer = await response.arrayBuffer();
  const bytes = new Uint8Array(arrayBuffer);
  // do something with bytes
}

The arrayBuffer call waits until the entire target has downloaded before returning the bytes. Instead, we can consume 'chunks' of the file (since we'll get parts of the file over time) at a time, to get a sense of percentage.

Check The Header

Firstly, we read the "Content-Length" header of our response: this is something the server sends us before the data, so we can actually work out how far along we've gone:

  const response = await fetch(url);
  const length = response.headers.get('Content-Length');
  if (!length) {
    // something was wrong with response, just give up
    return await response.arrayBuffer();
  }

If there's no valid header, then either there's something wrong with the response, or the server hasn't told us how long it is. You can just fall back to whatever you were doing before.

Chunks

Your browser is receiving chunks of bytes from the remote server as the data arrives. Since we know how long the total response will be, we can prepare a buffer for it:

  const array = new Uint8Array(length);
  let at = 0;  // to index into the array

And grab the reader, which lets us get chunks:

  const reader = response.body.getReader();

Now, we can store where we're up to (in at), and insert every new chunk into the output:

  for (;;) {
    const {done, value} = await reader.read();
    if (done) {
      break;
    }
    array.set(value, at);
    at += value.length;
  }
  return array;

Within the loop above, we can log the progress as a percentage, something like:

    progress.textContent = `${(at / length).toFixed(2)}%`;

Then as above, just return the array: we're done.

Fin

20 👋

Graceful Shutdown Is A Lie

Sam Thorogood — Wed, 19 Jun 2019 13:12:15 +0000

If you're writing backend code, you should always assume that your service is going to crash. 💥

This was something I learned when I was designing reliable code that was performing (effectively) a transaction using Megastore. This service needed to be live, always: our failure cases were that it crashed, or we were upgrading the binary.

This is fairly understood for webservers, where e.g. a Node server can be shut down only by Ctrl-C, or a Go server can only "fail":

func main() {
  http.Handle("/foo", fooHandler)
  log.Fatal(http.ListenAndServe(":8080", nil))
}

But it's less clear what a crash means while performing some operation, like database or file operations. When can your server crash awkwardly?

when you're writing a file to disk over a previous file: for a time, neither file will exist
- ... solve with writing a temporary file and replacing the previous one
updating an aggregate count (e.g., when you insert a row, add one to the 'total'): if the second operation fails
- ... solve with database transactions
marking an email as being sent: what if the database fails?
- ... sending a second email is probably fine ¯\_(ツ)_/¯

In all these cases, it pays to code defensively. This isn't a long post, but just something to think about: at what point could the power 🔌 be unplugged from your provider? What's the risk?

Thanks!

19 👋

Divert Vertical Scroll To The Side ↔️

Sam Thorogood — Tue, 18 Jun 2019 10:59:58 +0000

Let's try a combined blog post today: one tip for helping your users get around when you have horizontal scroll areas (for effect, or whatever), and one tip for navigating the web.

Vertical To Horizontal

If you have a great, amazing UI feature which just happens to scroll horizontally: think a big long list of products which go on forever, users can use say, a trackpad or a scroll bar to get there.

But if your user has a regular mouse with only a vertical scroll wheel, how can they navigate that area? 🤷

We can actually redirect catch the wheel event, and just modify the scrollLeft property (which controls horizontal navigation) based on deltaY (the up and down motion of the wheel). We also need deltaX, as otherwise we prevent real horizontal scrolling. Take a look:

// handle up/down scrollwheel on the scroller, as most folks don't have horizontal scroll
scrollableElement.addEventListener('wheel', (ev) => {
  ev.preventDefault();  // stop scrolling in another direction
  scrollableElement.scrollLeft += (ev.deltaY + ev.deltaX);
});

This is something we used on Santa Tracker in the 🧝 Elf Builder game.

The Tip (aka Today I Learned)

If you scroll vertically on a page (in nearly any browser) but hold down shift, you'll actually scroll horizontally.

Thanks!

That's all for today!

18 👋

What To Expect When You're Expecting To Drop IE11 🗑️

Sam Thorogood — Mon, 17 Jun 2019 10:04:53 +0000

So you've decided to drop support for IE11 and move onto evergreen browsers only (IE11 is only about ~2% globally). That's great! 🌲

With that in mind, here's a giant list of the features you should use, today (today being mid-2019), safely, without polyfills or feature detection. 📃

Before we start, of course, there'll always be old browsers. And, to be fair, browsers in emerging markets are more complex: like UC, KaiOS (based on an older Firefox), and Opera Mini. In these cases, I suggest serving no JS whatsoever (if possible), or encouraging users to upgrade. 🤷

Let's go! ⬇️

The DOM

Choose image URL based on resolution 📽️ (via <img srcset> and <picture>)
Frames can load from a Blob
Disable form elements with <fieldset disabled>, useful for in-progress forms
HTML input types color and various date/time options
HTML templates and the <template> element (this is also in JS, but you can specify them in your pages)
The <meter> element (goes along with <progress>)

JavaScript Language

ES Modules, through <script type="module"> and import/export 🎉
Template literals (with backticks)
Classes like class Foo { constructor() { ... } }
Functions! Arrow functions, rest parameters, async functions that allow await, generators which can yield

JavaScript Library

Promise and fetch (no need for XMLHttpRequest anymore 🚫)
- ... XHR's responseType can also now be set safely to "json", but why would you bother? 🤷
Methods on Array: find, includes; and on String: includes, padStart and padEnd
The Proxy object, allowing for interesting approaches
Methods on Object: entries and values, for iteration (like Object.keys)
The URL object (useful to check for query params and work with URLs)
The currentScript property ("what file am I")
You can safely dispatch a new CustomEvent('....') rather than dealing with weird intializers
Symbol and friends

JavaScript + The DOM

Better convenience methods for HTML
The third argument to addEventListener, allowing you to set {once: true} and other options
IntersectionObserver, allowing you to tell whether DOM nodes are visible
The navigator.sendBeacon method, to send POST messages even if a page closes
Find the closest matching element with closest
The 2nd argument to classList.toggle, allowing you to set or remove a class via parameter (also, the .relList property on links)
Canvas blend modes (this is the .globalCompositeOperation property)
Determine whether a CSS feature is supported via CSS.supports (but this only helps future features)

Whole New APIs

Service Workers 🥳
Web Assembly 👩‍💻
Gamepad API 🎮
Web Audio API 📣
Pointer Lock API: useful for HTML games and rich experiences 🐁🔒
Constraint Validation API (improved form validation) 📏
WebRTC 📽️
getUserMedia to get access to video, audio streams 🙏

CSS

Grid 🎉
CSS Variables, such as --foo: blue;, used with color: var(--foo)
Sticky Position
CSS filters, allowing for visual effects like invert, drop shadow and hue changes
Image object-fit (Edge only supports it on <img>), allowing you to make an image contain or cover its contents rather than stretch
Improved media queries for pointer or mouse access Fun fact: this was one of the first demos I wrote working on Chrome.
New CSS cursors 'grab', 'zoom-in', 'zoom-out'
The ::placeholder pseudo-element, for styling the placeholder text inside an <input>
Using initial or unset as CSS values
The vmax unit, which is a percent of whichever's larger: width or height
Going along with the JS method, the CSS @supports at-rule
Read-only and read-write pseudo-class selectors (:read-write seems the more useful of the two)
Stroke and fill on text
- ... although supported on all evergreens, you'll need to include the -webkit- prefixes: yes, even for Edge and Firefox
Risky bugs in IE11 are no longer an issue:
- You can now safely put calc(...) inside a CSS animation
- CSS display: flex had a variety of issues

Phew!

You got this far! Congratulations! 🎉

If there's any I've missed, or good resources for any of these features, let me know below.

17 👋