DEV Community: Zack Sheppard

I Promise you won't have to await long to understand async in Javascript

Zack Sheppard — Tue, 17 Aug 2021 10:31:31 +0000

As you're poking around with modern Javascript, it won't take you long to encounter one of the main asynchronous keywords: Promise, await, or async. So, how do these they work, and why would you want to use them? (And then at the end, some pro-tips for getting the most out of them.)

As with all things in asynchronous programming, we'll answer those questions eventually but the order in which we'll do so is not defined.

async function writeBlogPost() {
  await Promise.all([
    writeHowAsyncWorks(),
    writeWhyAsync().then(() => writeAsyncIsNotMultithreading())
  ])
    .then(() => writeProTips())
    .finally(() => writeConclusion());
}

Why Async?

Since the beginning, Javascript has lived on the internet. This necessarily means that it has had to deal with tasks that could take an indeterminate amount of time (usually calls from your device out to a server somewhere). The way that Javascript dealt with this traditionally has been with "callbacks":

function getImageAndDoSomething() {
  // This is a simplified example, of course, since arrow functions
  // didn't exist back in the day...
  loadDataFromSite(
    // Function argument 1: a URL
    "http://placekitten.com/200/300",
    // Function argument 2: a callback
    (image, error) => {
      // Do something with `image`
    }
  );
}

Callbacks are references to functions that get called when the work is done. Our loadDataFromSite function above will call our callback with image defined if and when it has successfully loaded the data from the target URL. If it fails, it will call our callback with image set to null and, hopefully, error defined.

This works fine when you're dealing with simple "get it and do one thing" loops. However, this can quickly enter callback hell if you need to do multiple chained calls to a server:

function apiCallbackHell() {
  loadData((data, error) => {
    data && transformData(data, (transformed, error) => {
      transformed && collateData(transformed, (collated, error) => {
        collated && discombobulateData(collated, (discombobulated, error) => {
          // And so on...
        })
      })
    })
  })
}

This is a mess! Callback hell like this was the motivation behind the Promise API, which in turn spawned the async/await API. In a moment we'll break down what this is doing, but for now let's just enjoy how clean our function looks with async/await:

async function notApiCallbackHell() {
  const data = await loadData();
  const transformed = await transformData(data);
  const collated = await collateData(transformed);
  const discombobulated = await discombobulateData(collated);
  // And so on...
}

Side Quest: Async is not Multithreaded Javascript

Before we break that down, though, let's clarify one common misconception: async code is not the same as multi-threaded code. At its core, Javascript remains a single-threaded environment.

Under the hood of the language is something called the "event loop", which is the engine responsible for reading in a single instruction and performing it. That loop remains a single threaded process - it can only ever read in one instruction at a time and then move on.

Callbacks and Promises make it look like this loop is doing multiple things at once, but it isn't. Let's imagine the instructions in our code as a pile of cards and the event loop is a dealer, pulling them off the top one-at-a-time and stacking them into a neat deck. If we have no callbacks or Promises then the pile our dealer can pull from is clear: it's just what we have in the program, reading through the lines of code top to bottom.

Adding async code to the mix gives our dealer another pile to pull from - the code in our callback or Promise can be read independently from the instructions in the global scope of our program. However, there is still only one dealer (one thread) and they can still only read through one instruction at a time. It's just that now they share their efforts between the different piles. This means that if you put some very difficult work into a Promise, you'll be creating a very big new pile for your dealer to pull from. This will slow down the execution of your other code, so interactive UI on your screen might get verrrrrry slow as a result.

The solution to this is to move your intense work to another thread - in our metaphor this would be the same as hiring a second dealer to sort through the intense pile of instructions separately from our main dealer. How to do that is beyond the scope of this post, but if you're curious check out Node's Worker Threads or the browser's Web Workers.

What are the pieces here?

So, we've heard of the main three tools in the async/await landscape, but what do they actually do and how do they work?

Promise

The backbone of the async/await toolkit is the Promise type. Promises are objects. They wrap code that does something. Their original purpose was to make it easier to attach callbacks and error handlers to that code. There are several ways to create a promise, but the most basic one is:

new Promise((resolve, reject) => {
  // Do something
  if (itSucceeded) {
    resolve(successResult);
  } else {
    reject(failureReason);
  }
});

Here you can see the core feature of a Promise - it is just a wrapper around callbacks! Inside of the execution block for our new Promise we simply have two callbacks - one we should call if the promise successfully did its work (the resolve callback) and one we should call if it failed (the reject callback).

We then get two functions on the Promise that are the most important:

const somePromise = getPromise();

somePromise
  .then((result) => {
    // Do something with a success
  })
  .catch((rejection) => {
    // Do something with a rejection
  });

then and catch are extremely useful if you've been handed a Promise from some other code. These are how you can attach your own callbacks to the Promise to listen for when it resolves (in which case your then callback will be called with the resolved value) or to handle a failure (in which case your catch callback will be called with the rejection reason, if any).

(Side note there is also a finally which, as you might guess, runs after all the then and catch handlers are finished.)

Then and catch are also useful because they themselves return a Promise now containing the return value of your handler.

So, you can use .then to chain together multiple steps, partly escaping callback hell:

function promisePurgatory() {
  loadData(data)
    .then(data => transformData(data))
    .then(transformed => collateData(transformed))
    .then(collated => discombobulateData(collated))
    .then( /* and so on */ );
}

Async/Await

You might have noticed, though, that Promise doesn't completely get us out of needing a huge stack of callbacks. Sure they are now all on the same level, so we no longer need to tab into infinity. But, the community behind Javascript were sure they could do better. Enter async and its partner await. These two simplify Promise programming tremendously.

First of all is async - this is a keyword you use to annotate a function to say that it returns a Promise. You don't have to do anything further, if you mark a function as async, it will now be treated the same as if you'd made it the execution block inside a promise.

async function doSomeWork() {
  // Do some complicated work and then
  return 42;
}

async function alwaysThrows() {
  // Oh no this function always throws
  throw "It was called alwaysThrows, what did you expect?"
}

const automaticPromise = doSomeWork();
// Without having to call `new Promise` we have one.
// This will log 42:
automaticPromise.then((result) => console.log(result));

const automaticReject = alwaysThrows();
// Even though the function throws, because it's async the throw
// is wrapped up in a Promise reject and our code doesn't crash:
automaticReject.catch((reason) => console.error(reason));

This is by itself pretty useful - no longer do you have to remember how to instantiate a Promise or worry about handling both the reject case and also any throw errors. But where it really shines is when you add in await.

await can only exist inside of an async function, but it gives you a way to pause your function until some other Promise finishes. You will then be handed the resolved value of that Promise or, if it rejected, the rejection will be thrown. This lets you handle Promise results directly without having to build callbacks for them. This is the final tool we need to truly escape callback hell:

// From above, now with error handling
async function notApiCallbackHell() {
  try {
    const data = await loadData();
    const transformed = await transformData(data);
    const collated = await collateData(transformed);
    const discombobulated = await discombobulateData(collated);
    // And so on...
  } catch {
    // Remember - if the Promise rejects, await will just throw.
    console.error("One of our ladders out of hell failed");
  }
}

A couple Pro(mise) Tips

Now that you understand the basics of Promise, async, and await a little better, here's a few Pro Tips to keep in mind while using them:

async and .then will flatten returned Promises automatically. Both async and .then are smart enough to know that if you return a Promise for some value, your end user does not want a Promise for a Promise for some value. You can return either your value directly or a Promise for it and it will get flattened down correctly.
Promise.all for joining, not multiple awaits. If you have several Promises that don't depend on each other and you want to wait for all of them, your first instinct might be to do:

async function waitForAll() {
  // Don't do this
  const one = await doPromiseOne();
  const two = await doPromiseTwo();
  const three = await doPromiseThree();
}

This is going to cause you problems, though, because you're going to wait for promise one to finish before you start promise two, and so on. Instead, you should use the built-in function Promise.all:

async function waitForAll() {
  const [one, two, three] = await Promise.all([
    doPromiseOne(), doPromiseTwo(), doPromiseThree()
  ]);
}

This way your code will create all three promises up front and run through them simultaneously. You're still going to await all three finishing, but it will take much less time because you can spend downtime on promiseOne working on promiseTwo or Three.

Promise.allSettled if failure is acceptable. The downside of Promise.all or serial awaits is that if one of your Promises reject, then the whole chain is rejected. This is where Promise.allSettled comes in. It works the same as Promise.all except that it will wait until all the arguments have resolved or rejected and then pass you back an array of the Promises themselves. This is useful if you're trying to do some work but it's ok if it fails.
Arrow functions can be async too. Last but most certainly not least, it's important to keep in mind that arrow functions can be marked as async too! This is really really useful if you're trying to create a callback handler where you'll want to use await, such as for an onSubmit for a form:

// Imagining we're in react...
return <Form onSubmit={
  async (values) => {
    const serverResponse = await submitValuesToServer(values);
    window.location.href = "/submitted/success";
  }
}>{/* Form contents */}</Form>

.finally(...)

Let me know in the comments down below what questions you now have about Promise, async, and await. Even though I use these three in every Node and React app I write, there are still tons of nuances to learn about them.

If you enjoyed this, please leave me a like, and maybe check out my last "back to basics" article on the ins and outs of this in JS.

Your Product Shouldn't Be an App

Zack Sheppard — Mon, 19 Jul 2021 15:48:57 +0000

Recently, I've run into a few big ideas that make me feel like it's 2010 all over again. In the worst way. They remind me that back in the heady days of 2010, we were fools for apps.

Need to know what song is playing on the radio right now? App! Are new-hires lost in your office and you want to give them a map? Enterprise app! Need to order a sandwich from the independent deli on the corner? Whitelabel app!

Ultimately these all fail the core test of making a product: they have identified a real need but fail to fulfill it better than the competition.

What do I mean by that? Well, let's look at our examples: if I want to know what's on the radio right now, does it make more sense for me to open an app and see it or should I just turn on the radio? If I need to help new-hires orient themselves in my office, should I hire someone to make a map app, get an enterprise distribution certificate for the app store, and debug issues people have installing that app, or should I draw a map and give them a photocopy? If I want to order a sandwich from the deli, do I install their one-off app, make an account, and place an order or do I just walk down the street?

In all of these cases, the competition isn't another app, it's a better way of solving the problem. Even though the wild crappy-app days of the 20-teens are behind us, this lesson still holds true when we think about designing products today.

What's wonderful about this is that you can usually find a way to achieve your goals far quicker and cheaper than by building a full app. Here are some alternatives you need to consider before you dive into the world of designing and developing a full-fledged app:

1. Info-products

If the goal of your app is to communicate something to your users, the first question to ask yourself is whether that could be better accomplished by written words or videos. Simplifying your product to an info-product like this has two big advantages. First, larger media platforms like Substack, Skillshare, Teachable, Gumroad, or even Youtube are almost guaranteed to reach a larger market than a purpose-built app or site. Second, you can often communicate your ideas and make a sellable product in drastically less time than if you were trying to capture those same ideas in the framework of an app.

2. Communities

If your app is trying to bring people together but not necessarily to form a market, you might be better off hosting a private Discord or building a community site on top of a platform like Circle, Tribe, or Dev.to's own Forem. Communities especially are an interesting opportunity when added on top of info-products, as they give you the chance to keep your customers engaged with you between releases of new content.

3. Webapps

While still kind of an app, web apps are much easier to build, maintain, and distribute to users than a native app (especially given the wealth of Nocode tools for building them). Let me stress this point, the simplicity and speed of launching a website makes the stress of launching a true native app feel like hiking Everest. Web based apps can also make much more sense depending on the context you expect your users to be in when they need whatever solution you provide. If you think a user is going to be at work on their computer, for example, it makes more sense for your product to be a website rather than an app that will require they switch contexts to use it.

4. Non-tech Products

Finally, does your product need to be a tech product at all? Let's take a hypothetical product of "COVID-friendly activities for office bonding" for example. For version one, you could jump straight to building this as a productized digital app experience that can lead customers on group tours or scavenger hunts. But, you will be able to execute more quickly, get more feedback, and generate revenue immediately if you instead take the old-school route and lead your first few experiences personally. Once you have run a couple sessions, you'll have money in the bank and a much better sense for the pacing and content that leads to a good experience, and from there you'll be in a much stronger position to think about app-ifying your product.

And therein lies the last, most important lesson here. Choosing not to app-ify your product from day one doesn't mean you'll never turn it into an app or other digital product. Rather, this is about developing your product intelligently. If you jump straight to an app without validating your guesses about what will solve your users' problems most effectively, it's likely you'll build the wrong thing and wind up wasting all that effort. The App Store is already littered with the abandoned corpses of apps that failed to learn this lesson, don't be another one of them.

Speedy and Stylish - How to style NextJS with Bootstrap 5, SASS, and PurgeCSS

Zack Sheppard — Thu, 10 Jun 2021 15:33:05 +0000

Quick how-to for you today. I'm setting up a new web app project with these parameters:

I want it to be based on Next.JS because it's my favorite React framework.
I want to style it with Bootstrap 5 because I need more help setting up a consistent theme than Tailwind would give me.
I want to write the styles in SASS because that's what Bootstrap uses under the hood, which makes customizing the Bootstrap theme easier.
I want to run the styling through PurgeCSS to avoid shipping unnecessary Bootstrap bloat.

So how do you do that?

1. NextJS

Setting up a new NextJS project is foolishly easy. First, pick an example to base your project on. I personally chose with-typescript-eslint-jest because I like writing in Typescript much more than vanilla JS and I like having ESLint in VSCode to automatically style my files when I hit save.

Second, run one of these (depending on whether you want to use npm or yarn), substituting in your chosen example template and replacing your-project with the name for your project:

npx create-next-app -e with-typescript-eslint-jest your-project

yarn create next-app -e with-typescript-eslint-jest your-project

Note that from here onward I'll be writing everything in yarn commands, so be sure to substitute npm if that's what you're using.

And that's it, you're done. You have a basic Next.JS app ready to go. Start it up with yarn dev and see your site running at http://localhost:3000.

2. Bootstrap 5 in SASS

Next, let's add Bootstrap 5. If you want just a few Bootstrap elements, you can get away with using the react-bootstrap package. Their 2.0 build (based on Bootstrap 5) is in beta release right now. But, I want to build larger, more personalized themes, so I'm going to install Bootstrap directly.

yarn add bootstrap@^5.0.1

This will add Bootstrap's raw source to your node_modules. Now Bootstrap has two parts to it - the styling and the javascript - that combine to create the pretty and pretty functional components.

First, let's add the styling. We're going to write our styles in SASS because it allows us to extend and override values in Bootstrap's themes. You can read more about customizing Bootstrap in SASS here.

Luckily, NextJS ships with built-in support for SASS. So it's as easy as adding the sass package and creating styles/index.scss:

yarn add sass

// Example: override to give us a salmon body background
$body-bg: #ffbbbb;

@import "../node_modules/bootstrap/scss/bootstrap";

and then importing it in pages/_app.tsx:

import type { AppProps } from "next/app";
import "../../styles/index.scss";

export default function MyApp({ Component, pageProps }: AppProps) {
  return <Component {...pageProps} />;
}

Refresh your development site and you should find the basic Bootstrap "reboot" styles are applied and the background is now a nice salmon color.

Next, the Javascript. We only have to do this if we use one of the dynamic components like Alert. If so, we first have to install the popper dependency:

yarn add @popperjs/core

Then, we use Bootstrap's recommendations on importing their JS into a Webpack environment. In _app.tsx pick one of these:

// If you want to pull in the whole Bootstrap JS library:
import "bootstrap";

// If you only need a single component:
import Alert from "bootstrap/js/dist/alert";

These each tell Webpack to include the relevant source file(s) in the final bundle it makes when we build our app for production or export.

3. Keep it trim with PurgeCSS

One of the advantages of TailwindCSS over Bootstrap has been that it automatically trims out the bloat of the framework that you don't use, so you don't wind up shipping styling for <table> elements if you don't use any on your site. It accomplishes this through PurgeCSS, which is trivial for it because it is based on PostCSS and dynamic style generation in the first place.

NextJS ships with PostCSS already, so we can give Bootstrap this same superpower by adding PurgeCSS to Next. Unfortunately, as the PurgeCSS page on this notes, customizing PostCSS turns off NextJS's built in optimizations, so we have to recreate them. Luckily, this isn't hard. First, install the packages we need:

yarn add -D autoprefixer postcss @fullhuman/postcss-purgecss postcss-flexbugs-fixes postcss-preset-env

Now that the pieces are in place, we put them all into our build pipeline by creating a custom postcss.config.js file in our project's root folder:

module.exports = {
  plugins: [
    "postcss-flexbugs-fixes",
    [
      "postcss-preset-env",
      {
        autoprefixer: { flexbox: "no-2009" },
        stage: 3,
        features: { "custom-properties": false }
      }
    ],
    [
      "@fullhuman/postcss-purgecss",
      {
        content: ["./src/**/*.{js,jsx,ts,tsx}", "./pages/**/*.{js,jsx,ts,tsx}"],
        defaultExtractor: (content) => content.match(/[\w-/:]+(?<!:)/g) || [],
        safelist: ["html", "body"]
      }
    ]
  ]
};

The first half of this file is pulled directly from NextJS's documentation on customizing PostCSS. The second half adds in PurgeCSS and configures it to remove all classes whose names do not appear in the src or pages directories. It also tells it to manually to leave any html or body styles alone, since those two elements are rendered outside of the scope of our Next project's source files.

Ok, so refresh your local dev build again and... nothing changes! This is intentional - PurgeCSS shouldn't be purging any classes that are used. This also includes the styles we need in Reboot - Bootstrap's cross-browser standardizing stylesheet. If we use an element type that Reboot standardizes, then PurgeCSS will recognize that and won't touch the corresponding style.

That's it

There you go - an easy 1, 2, 3 and now you have a new NextJS project set up with efficiently and intelligently trimmed-down Bootstrap 5 styling. Hope this is helpful for you while you're setting up a project.

How To: Deploy Next.js Apps with Docker Containers (Efficiently!)

Zack Sheppard — Mon, 07 Jun 2021 10:54:16 +0000

So let's say you've written an awesome app in Next.js and you want to deploy it to a nifty containerized platform like Digital Ocean or Fly.io. But let's say that you, like me at the start of last week, have never containerized a Node app before and need a crash course in how to do that?

Here's what I learned going through this process to deploy Tweet Sweep to fly.io - both the naive first steps for making a container work at all and then also some necessary optimizations for it.

Starter Instructions - How to Dockerfile
Optimize It - Make it Production Ready
Just give me a Dockerfile I can blindly copy paste

Follow Along

If you want to follow along, you will need Docker Desktop and Yarn installed. To keep things replicable, I'm using the Next.js Blog-Starter-Typescript example in these instructions. You can set that up locally with this command:

yarn create next-app --example blog-starter-typescript blog-starter-typescript-app

As a side note, the tips and tricks in here are generic for all containerized Node apps, but the Dockerfiles themselves will only work as an untweaked copy-paste if you're using Next.js. So, if you're using a different platform you might have to tweak which files get retained in your final container.

The Basics - Just make it work

So let's start with the 101 - what is Docker and why you want to use it. At its core, Docker Containers are tiny virtual computers serialized to disk in a standardized format. To make them, you need three ingredients:

A starter image to build upon - usually this is a full operating system image with some pre-installed software from Docker Hub.
New files to add - in this case the code for your app.
The steps to combine those first two components. This is what is stored in a Dockerfile and a .dockerignore file.

Using these three components you can wrap up your software into a standardized container that can be run on any machine that has the Docker software installed. (Note that this has a big "in theory" caveat attached - if you are doing complex, advanced operations then you might run into the limits of Docker's capabilities. However, for a straight-forward Next.js app like the one I'm using here, it works very well.)

The Naive Dockerfile

So what do these instructions look like for our Next.js application?

# Naively Simple Node Dockerfile

FROM node:14.17-alpine

RUN mkdir -p /home/app/ && chown -R node:node /home/app
WORKDIR /home/app
COPY --chown=node:node . .

USER node

RUN yarn install --frozen-lockfile
RUN yarn build

EXPOSE 3000
CMD [ "yarn", "start" ]

Put these in a file named Dockerfile in the root folder of your app.

Understanding the Dockerfile

So what does this do? Well, Docker will step through these instructions one by one and do the following:

FROM node:14.17-alpine

This tells Docker that your app is building on a container that has Alpine Linux and Node 14.17 (with npm and yarn) preinstalled.

RUN mkdir -p /home/app/ && chown -R node:node /home/app
WORKDIR /home/app
COPY --chown=node:node . .

USER node

These are our first real instructions - we make a directory called /home/app, give ownership of it to a user named node, make it the "working directory" for our container (where Docker expects our main program files to live), and copy the files in the directory where we ran docker build into the container. Remember the container is basically a virtual little computer, so we have to copy our files in there to access them!

We then become that node user. By default Docker runs as root on the contained machine. But that is pretty dangerous since it gives root privileges to whatever code we run, meaning a little security flaw in Node or one of our NPM dependencies could potentially give access to our whole server. So, to avoid that, we switch to a non-root user.

RUN yarn install --frozen-lockfile
RUN yarn build

We install our NPM dependencies and build our Next.js server in production mode.

EXPOSE 3000
CMD [ "yarn", "start" ]

And finally these two commands give Docker instructions it will use when it tries to run this software. The first tells Docker that this container expects connections on port 3000, so it should expose that leaving the container (we'll wire it up in a moment with the -p flag). The second tells Docker that the command to run to start this container is yarn start.

Build and Run!

Now it's time to execute those steps and make your container. Run the following command in a terminal in your project directory (you can replace some-name with a personal tag like zacks-blog-1.0):

docker build -t some-name .

Your built image, containing the virtual machine ready to run your web app, will now show up locally if you check docker image ls:

$ docker image ls
REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
some-name     latest    4c73a8c8d35c   2 minutes ago    622MB

Let's start it up:

docker run -p 3000:3000 some-name

(You can add the -d flag after run to run the server in the background instead.)

You'll see logs same as if you'd run yarn start normally. And, due to the -p 3000:3000 flag, your container will now be connected to your local port 3000, so if you visit http://localhost:3000 you'll see your blog template:

Optimize it - Getting this production ready

Great! You have now containerized your app. But before you go deploying it to your favorite hosting platform, there are a few things we need to do.

You might have noticed above that the size of our built image is over 600MB - that's over 4x the size of our project on disk outside of the container! This problem only compounds as your apps get more complex - the built versions of the Tweet Sweep Frontend container were more almost 5GB at this point! That's a lot of data to upload to your servers!

Almost all of this size issue is related to one particular quirk of Docker - almost every line in the Dockerfile creates a new "layer" in your final Docker image. Each layer captures the changes made to the virtual machine after that line runs. This is a powerful optimization tool because it allows Docker to reuse work it's already done - for example if you have some setup that never changes like our mkdir line, Docker can compute that layer once and reuse it for all subsequent builds. However, it can also lead to image size issues (since lots of unneeded files might wind up being stored in those layers) and security issues (since you might capture secret values in those layers that could be siphoned off by someone who gets access to your final image).

You can see the layers and their respective sizes using this command (credit to this post where I got it from):

docker history --human --format "{{.CreatedBy}}: {{.Size}}" some-name

CMD ["yarn" "start"]: 0B
EXPOSE map[3000/tcp:{}]: 0B
RUN /bin/sh -c yarn build # buildkit: 10.6MB
RUN /bin/sh -c yarn install --frozen-lockfil…: 340MB
USER node: 0B
COPY . . # buildkit: 155MB
WORKDIR /home/app: 0B
RUN /bin/sh -c mkdir -p /home/app/ && chown …: 0B
/bin/sh -c #(nop)  CMD ["node"]: 0B
/bin/sh -c #(nop)  ENTRYPOINT ["docker-entry…: 0B
/bin/sh -c #(nop) COPY file:238737301d473041…: 116B
/bin/sh -c apk add --no-cache --virtual .bui…: 7.62MB
/bin/sh -c #(nop)  ENV YARN_VERSION=1.22.5: 0B
/bin/sh -c addgroup -g 1000 node     && addu…: 104MB
/bin/sh -c #(nop)  ENV NODE_VERSION=14.17.0: 0B
/bin/sh -c #(nop)  CMD ["/bin/sh"]: 0B
/bin/sh -c #(nop) ADD file:282b9d56236cae296…: 5.62MB

From this we can see that about 117MB of the image size happen before our first command - this the base size of the Alpine-Node image we're building on so there isn't much we can do about that. But let's focus on the two main optimizations we can do after that point:

Easy: Ignore Stuff

In our naive Dockerfile we run the command COPY --chown=node:node . .. This copies all the files in our current directory into the Docker container. This is almost always not what you want! For example, you might have an .env file with secrets in it that will wind up in plain-text in the final Docker image. (You should use the env secrets feature on your hosting platform instead.)

In this app's case this unnecessarily copies the node_modules folder (since we then yarn install it again) and .next folder (since we rebuild the app inside the container). We can fix this with a .dockerignore file. This file, in the root of our project, tells Docker to skip certain files and folders when running COPY.

# .dockerignore file
.DS_Store
.next
node_modules

Advanced: Get your Container a Container

Now the galaxy brain move here is to use containers for our container. We're going to create two that are used only to build the application separately from the one that's uploaded to the server. This saves us from having to upload the layers containing all the files used or created en route to that destination. Here's the Dockerfile for that (with comments explaining along the way what each block does):

(Edit: After I posted this, Vercel got in touch to point out they have their own post with a sample Dockerfile. I've now incorporated some tips from theirs into this one.)

# Double-container Dockerfile for separated build process.
# If you're just copy-pasting this, don't forget a .dockerignore!

# We're starting with the same base image, but we're declaring
# that this block outputs an image called DEPS that we
# won't be deploying - it just installs our Yarn deps
FROM node:14-alpine AS deps

# If you need libc for any of your deps, uncomment this line:
# RUN apk add --no-cache libc6-compat

# Copy over ONLY the package.json and yarn.lock
# so that this `yarn install` layer is only recomputed
# if these dependency files change. Nice speed hack!
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

# END DEPS IMAGE

# Now we make a container to handle our Build
FROM node:14-alpine AS BUILD_IMAGE

# Set up our work directory again
WORKDIR /app

# Bring over the deps we installed and now also
# the rest of the source code to build the Next
# server for production
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN yarn build

# Remove all the development dependencies since we don't
# need them to run the actual server.
RUN rm -rf node_modules
RUN yarn install --production --frozen-lockfile --ignore-scripts --prefer-offline

# END OF BUILD_IMAGE

# This starts our application's run image - the final output of build.
FROM node:14-alpine

ENV NODE_ENV production

RUN addgroup -g 1001 -S nodejs
RUN adduser -S nextjs -u 1001

# Pull the built files out of BUILD_IMAGE - we need:
# 1. the package.json and yarn.lock
# 2. the Next build output and static files
# 3. the node_modules.
WORKDIR /app
COPY --from=BUILD_IMAGE --chown=nextjs:nodejs /app/package.json /app/yarn.lock ./
COPY --from=BUILD_IMAGE --chown=nextjs:nodejs /app/node_modules ./node_modules
COPY --from=BUILD_IMAGE --chown=nextjs:nodejs /app/public ./public
COPY --from=BUILD_IMAGE --chown=nextjs:nodejs /app/.next ./.next

# 4. OPTIONALLY the next.config.js, if your app has one
# COPY --from=BUILD_IMAGE --chown=nextjs:nodejs  ./

USER nextjs

EXPOSE 3000

CMD [ "yarn", "start" ]

The Results

Now if you build that (again with docker build -t some-name-optimized .) and run it (docker run -p 3000:3000 some-name-optimized) you'll be able to connect to it on localhost:3000 same as before.

What has changed, then? Well, if we list our images:

$ docker image ls                      
REPOSITORY           TAG      IMAGE ID       CREATED       SIZE
some-name-optimized  latest   518ed80eae02   1 hour ago    243MB
some-name            latest   4c73a8c8d35c   2 hours ago   622MB

You can see we've reduce our final build image's size by almost a factor of 3! That's a lot less data we'll need to upload to our server with every deploy! I saw similar results when I employed this strategy on Tweet Sweep's containers, saving me gigabytes of upload bandwidth with every deploy.

The Actual Deploy

Ok, so now that we have our app containerizing successfully, how do we actually deploy? For this, I've been using fly.io because their support for Docker is strong and their service has a generous free tier. But if you'd rather use Heroku or Digital Ocean they have strong support for Docker as well.

With Fly, I'd recommend just following their step by step instructions for deploying Docker. TLDR; you have to create an app on your account and a corresponding fly.toml file locally, then the command flyctl deploy will automatically run your Dockerfile build, upload all the resulting layers to their service (this is why it's important to optimize their size!), and then start them on a VM server. After that, deploys really are as easy as running flyctl deploy again thanks to the compartmentalization of containers!

More Optimizations?

I'm still learning Docker so these optimizations are just the first I've come across. If you've played around with it and know some more ins-and-outs one should include while containerizing a NodeJS app, please do let me know down in the comments.

App 101: What is a Work Queue and how to make one in Node

Zack Sheppard — Mon, 17 May 2021 15:16:00 +0000

A lot of the details about building an app are mysterious, even if you've been a programmer for years. So, I'm going to bundle up some of the discussions I've had with friends who are starting their own apps into these App101 posts. Follow along and we'll bring you up to speed on the decisions that you'll have to make when building a web-based app today.

If you boil it down far enough, most internet programming follows this cycle:

Receive HTTP Request -> Do Work -> Deliver HTTP Response

This works fine 99% of the time because usually the work we're doing is simple: update a record in the database, pull some data and return it as json, and so on. However, once in a rare while this cycle breaks down. Maybe you find yourself in a situation where the "do work" step is too time- or resource-consuming to finish before the requester times out. Or maybe the actual work is only being scheduled by this request and can't start until its scheduled time. These are classic cases where you might want to build a work queue on your backend.

What is a Work Queue?

A work queue is like a todo list for your server. You put messages into the queue, which are often no more than simple dictionary objects. You assign one or more workers to the top of the queue. Your workers then peel a message off the queue one-by-one, does some work, and then marks them as processed.

You can process these messages in whatever order you want. Most queues start with first-in, first-out processing (FIFO) processing, where the oldest message in the queue is the next to be processed. However, many queue tools support additional features like scheduling messages - so they can't be processed before a certain time - and adding priority values - so that higher priority messages can jump the queue like a first class passenger at the airport.

So Why Use Queues?

So now we know how a queue works, let's get back to the why of using one. The main benefit of queues is that they allow you to constrain resources and not constrain time.

What do I mean by that? Well, when you are handling an HTTP request, you are time constrained - you only have so many seconds to return a value before the connection to your client times out or breaks. You're not, though, resource constrained - if you're using a serverless architecture or a load balancer your host might just spin up ever more instances of your API server to handle additional traffic. This is bad in both directions when you have heavy work to do, though.

Let's imagine you have a request that is really hard to process and takes 60 seconds, for example calculating some complex AI response. As the time needed to handle that request goes up then not only does your failure rate go up (since connections time out while your working), but also your infrastructure costs go up because you need more web servers to handle your traffic (since your existing instances are spending all their time doing this heavy work).

Queues solve this problem in two ways. First, they escape your time constraint by moving the work to another process. Your web server can now return a "working on it" message almost immediately. Plus, because you pulled the processing out of your web server and into the queue worker, this heavy work no longer runs the risk of slowing or crashing your server. The second way they solve the constraint issue is that they allow you to constrain your infrastructure costs. Now you can control the number of workers who are working on the queue without causing failures. If your traffic goes up, it just means the queue takes longer to do the work rather than necessitating more infrastructure to handle that traffic.

That is, though, also why you may not want to use a queue. When you have more work coming in, the time to handle each job goes up because the queue gets longer. Additionally, you've added complexity to your API - now your client is dealing with an asynchronous process, so it will need to call back later to check "is my job done yet?" However, if you have heavy work that necessitates this, queues are often the better solution than just hoping the connection doesn't time out.

How to Queue

It can take years to truly master the administration and tuning of queues. (Indeed, I am far from a master here.) However, it takes only minutes to start learning those skills. I found two easy ways to start experimenting with queues today:

Option 1: Serverless Queues

The quickest way to start experimenting with queues is with a serverless queue manager like AWS SQS or GCP Cloud Tasks.

These services are cheap (both will handle billions of messages per month for free), and both can be configured to run your worker a serverless FaaS process (Amazon's documentation for that is here, and Google's is here).

They have two downsides, though. First, like many GCP and AWS products, they both have a bit of a painful learning curve as you learn to configure the tool. The main one, though, is that both of these have time limits to how long your worker can run. They are long - Google's is 10 minutes and Amazon's is 15 - but you should consider this if your work might approach that threshold. That was, eventually, why I went with Option 2 during my own experimentation.

Option 2: Roll it Yourself

The other choice is to roll your queue yourself. This is the route I took for my current side project, Tweet Sweep, because I really wanted to learn how the system worked under the hood and I was worried my queue workers might take longer than 15 minutes.

If you're working in Node, I'd recommend using one of the two most maintained open source projects: bee-queue and BullMQ. Both have the features I've mentioned here, so you can get started with priorities and scheduling. Personally, I chose bee-queue because its api for creating parallelized workers was more straight-forward.

With these packages (and, most likely, with any similar package in your preferred language), setting up the queue has two steps. First, you need a location for the queue messages to be stored while they're awaiting processing. Usually, this is a Redis datastore, since it's very fast and it is generally accepted as OK that you might lose queue messages if your infrastructure fails and needs to restart. Second, you will need a computer acting as your worker. Usually, this is process on a server that attaches to your queue(s) in the datastore and, when a new message is published to one, pulls it down and processes it.

A hot tip: while learning to build my own queue, I found fly.io really useful. Its free tier comes with both a preconfigured Redis store and enough CPU budget to run two processes full-time, so you can run your API server and queue worker side by side.

Go out and get in line

I've used queues at every company I've worked at. Indeed, Foursquare had tens of queues handling tasks like score calculation on a check-in or resizing profile images that were uploaded. Even this month, I'm helping a current client explore them for a long application processing step during onboarding. They're a tremendously useful tool and they can, when appropriately used, take a huge amount of strain off of your API servers and network hardware. But the first step to being able to use them in your own apps is learning the basics - let me know how you get on with queues down in the comments.

App 101: Jamstack vs Traditional Webapps

Zack Sheppard — Tue, 04 May 2021 13:56:17 +0000

My friend is a data scientist and I've helping him start making his first web app during lockdown. It's reminded me that there is a lot of App101 that you might not know even if you're already a programmer. So, I'm going to bundle up some of those discussions into these App101 posts to bring you up to speed on the decisions that you'll have to make when building a web app today.

Tales of Yore: Old Webapps and the Invention of the API

Let's start with a little bit of (oversimplified) history.

For the first 20 years the web existed, there was basically one way to build a web app that did more than just serve static content. You would get a database (often MySQL) and you would write a bit of software (often PHP or Perl) that would receive HTTP requests, interact with the database, and then return HTML with the next page to show the user. Basically the whole internet worked like Amazon.com: you'd click links or submit <form>s, those actions would cause HTTP GET or POST requests, and the server would return fully formed pages back to your browser.

That all started to change around 2000 with the invention of AJAX. AJAX used Javascript to move some of the work off the server and onto the user's computer. Now the browser could perform single actions without requiring a whole HTML page be sent over the wire. Instead, your code would call a special URL to get back just a snippet of HTML to substitute onto the page. This snippet later became structured data, first XML and then JSON, that would be used to calculate the HTML changes locally. This was the evolution of the modern web API - a structured set of URLs for loading or changing data on a server.

Monolith Webapps: The old school architecture

By 2015 the rise of mobile apps, a new class of non-html travelers on the internet, solidified the API as part of standard web app architecture. This meant the way we now build traditional webapps had been set:

This is the way that many modern app frameworks still work: Django, Ruby on Rails, etc. Many of these frameworks exist to try to simplify this setup by making the API and Web codebases easier to manage, or by automatically generating the Javascript for the website so you don't have to code your site twice.

Traditional apps built like this have their advantages. They are well-understood and traditional, they have decades of solid tooling built for them, and they generate websites that are roughly what browsers and search engines expect.

However, this architecture has a couple of clear issues as well. A minor one is that both the API and the web renderer have to interpret the data coming from your database and an inconsistency between those interpretations can cause some pretty weird bugs for your users.

A much more serious risk, though, is that both your web renderer and your API have direct connections to your database. This can create serious headaches because the database is often the most fragile and resource-limited part of your architecture, while your website is often your most exposed. That combination opens up a lot of surface area for bugs or hacks to appear. One of the most common is that heavy traffic might overload your database, as was often the case with Twitter in its early days. Meanwhile, a security issue with one of your pages might allow direct access to the data, as has happened many times with Wordpress over the years. While these risks are today often mitigated by well designed frameworks, the connection to the database remains a vulnerability.

Jamstack: The upstart

Around 2015 engineers began to ask, "what if we solve those issues by cutting the link between the web server and the database?" Since many of the larger apps already had full-featured APIs, it was a pretty natural leap to have the website consume data solely via the API. This takes the spider's web of dependencies above and changes it into a linear conga of data:

Once you do this, you find you get a whole host of benefits. Now, the only code you need to worry about having direct access to your database is your API. That gives you one location where proper escaping and rate limiting can protect your uptime. Plus, your API code is now the one source of truth for all your data, so you no longer need to worry about those pesky interpretation bugs.

What I love about the Jamstack is that from this starting point the community began to find even more advantages. Around this time Node.js was skyrocketing in popularity and having Javascript on the server meant that your server-side renderer could literally run the same code as the browsers would. Going even further, though, people realized the server-side renderer could run no code at all! Instead, you could render all your pages during your deploy and serve purely static files through a service like Netlify or a CDN like Cloudfront. This gave rise to a whole generation of tools like Gatsby, Hugo, and Jekyll.

However, JAMStack does come with its own issues and complexities. The main issue is that you have to now handle your API's code and deployment in addition to your webapp's. For lightweight projects, this is often unnecessary overhead. Even if you ignore the extra code, handling the deploys of the static files, the API, and potentially also the server-side renderer is clearly more work than a traditional single-deploy app would have been. But, for apps that might have to handle a lot of requests for static content (like a blog or marketing site) or web+mobile apps that will be dealing with similar requests from multiple different sources, the advantages you gain in simplifying your architecture outweighs the complexity you add to your deploy.

So which is right for you?

The JAMStack is still a very new development in the web world, even if it has its own conference and fancy .org website now. This means there's still a lot of innovation and improvement to be reaped there. You can see this in the development of frameworks that make generating Jamstack applications easier and in the roll-out of server products that make it seamless to deploy snippets of dynamic server-side rendering code alongside large chunks of static code.

However, the traditional form of webapp is far from dead. Single-deploy frameworks like Rails, Laravel, and Django are often still the right choice if you'd like to keep your code and deploy process simple at the expense of some hidden complexity in your architecture.

As for my friend who's working through App 101 right now, we decided the right choice was to work with Django for his project. It gave him a simple, single codebase in a language he was familiar with where he could build his API, admin pages, and database schema at once. I may love Jamstack, but it wasn't right for him because of the added complexity managing and deploying it. That may be the case for you too, or you might take to it as enthusiastically as I have. I'd love to hear what you decided to use for your projects and why.

Let life get in the way

Zack Sheppard — Tue, 20 Apr 2021 10:26:08 +0000

I made zero progress on my side-projects this past week. Indeed, I just barely got the work done for my freelance contracts. I was the opposite of a 24/7 hustler. And that's ok, because I was spending that time letting life get in the way of work.

Specifically one very small bit of life: my new four-legged side project, Islay.

But this post is not (just) an excuse to share adorable photos of this tiny puppy now living in my house. This post is about taking a breath after months of quarantine hustle and realizing I lost sight of that quasi-mythical goal: "work-life balance."

For myself, and I assume I'm not alone in this, it can be easy to get sucked into games about making numbers go "up and to the right" - whether that's getting obsessed with the performance of an API server, maximizing finances, or driving up the engagement stats on a blog or Twitter. These are games with "high scores" where it's easy to monitor whether we're winning and every time the numbers go up, a little dopamine burst whispers "good job." But the tightly-coupled simplicity of that system makes it addictively, dangerously easy to slip into the habit of working almost every waking moment.

I don't have the be-all and end-all answer to perfect work-life balance and in no way would I - someone who has happily plugged away on projects for 60+ hours a week for basically all of 2021 until this week - pretend that I do. All I can share is what has worked for me looking backwards. A decade ago, finding work-life balance was about unplugging completely and sitting in a park or a bar with people. In 2019 work-life balance was becoming a deep-dive nerd about photography.

This week, work-life balance was teaching this little furry destroyer-of-toys how to sit.

The common factor to these was a living goal. One that isn't measured in numbers like commits or reps or engagement but rather in moments like stories told or photos made or lessons taught. I don't know if that's the recipe to success. What I do know is I've signed up for a new big project in raising Islay. The payoff for this project will be a little spitfire who will come demand I stop working if I've been grinding too long, because she's a dog and she knows nothing about numbers or making them go up, right, or in any direction at all.

Back to Basics: Understanding and Conquering "this" in Javascript

Zack Sheppard — Mon, 29 Mar 2021 11:36:43 +0000

I've been thinking about this a lot recently because I've been messing around with a lot of chained callback functions in my web code. This is a good opportunity to go back to basics and recap how this works in Javascript and what tools exist to tame its quirks.

For new developers coming from a more typically object-oriented language like Java or Swift, Javascript's weird use of the this keyword is a trap waiting to crash your code at any moment. This is especially dangerous if you're using React's class components, where you're often defining methods on your class to act as a callback handler. If you blindly assume that this is going to behave the way you've come to expect, you're gonna have a bad time. So, let's understand this enemy so we can learn how to fight it:

What is `this`

Let's start with the basics of how we expect this to work under the best circumstances:

'use strict';

class Person {
  name;

  constructor(theirName) {
    this.name = theirName;
  }

  introduce() {
    console.log("Hello I'm " + this.name);
  }
}

const william = new Person("Bill");
william.introduce(); // Prints out "Hello I'm Bill"

This is pretty straightforward: there is a class of object called Person. Each Person remembers a variable called name and has a method called introduce. When you call introduce on a person it looks at that person's name and prints an introduction. So, this is a reference to the object whose instance of introduce we're looking at, right?

Well, not quite. Take a look at this:

// Continued from above

// This doesn't RUN william's introduce function,
// it makes a REFERENCE to it
const introduceWilliam = william.introduce;

// Because it's a reference to a method that worked,
// we might assume the reference will also work but...
introduceWilliam();
// Uncaught TypeError! Cannot read property 'name' of undefined

Now we've delved below the calm surface into the dark depths of a functional programming language written in the 90's.

You have to remember that as far as Javascript is concerned functions are just another kind of object. They can be stored, passed around, and executed anywhere.

When you call someThing.someFunc(), Javascript parses that you want to execute the instructions in someFunc in the context of someThing. That is to say, set this to someThing and then execute the instructions.

But if you make a reference to someFunc, you could execute it anywhere. Above, we called it in the global context, which leaves this as undefined when you're in strict mode. You can even use the function's call or apply methods (functions on a function!) to provide any context and args you desire.

Let's write some mildly horrifying code to demonstrate this:

// Still using william from above
const william = new Person("Bill");
// Make a reference to william's introduce method
let introduce = william.introduce;

// Make an unrelated object - Bagel the Beagle
const puppy = { name: "Bagel", breed: "Beagle" };
// Run function with manual `this` - Dogs can talk now
introduce.call(puppy); // Prints "Hello I'm Bagel"

Taming `this` Beast

This this is incredibly, and often unnecessarily, powerful. Like many incredibly powerful things, it is also incredibly dangerous. Because of how often we pass around references to functions - to use as callbacks for buttons or forms, for example - the unbound nature of this is just lying in wait to trip you up.

So how do we tame this? I could shake my cane at you and croak "Well, back in **my* day..."* but the truth is that the ES5 and ES2015 revisions to Javascript gave us everything we need to clamp down wandering this values:

Function.prototype.bind()

Added in ES5, the first tool we got was the bind() function, a standardization of this hacks that the various utility libraries of the 2000's had innovated.

// Bind this reference to introduce so this is ALWAYS william.
let alwaysIntroduceWilliam = william.introduce.bind(william);

alwaysIntroduceWilliam(); // Prints "Hello I'm Bill"
alwaysIntroduceWilliam.call(puppy); // Prints "Hello I'm Bill"

bind does what it says on the tin. It binds the function to a chosen this - ensuring that the instructions inside are always run in the context we choose. Here you can see that even if we try to use call to set a different this, the bind overpowers and we're always introducing william. This was a great first step towards fixing this, but these days is less commonly used because of...

Arrow'd =>

Added in ES2015, arrow functions gave us (almost accidentally) the most common way of fixing this to the value that we expect. This is because an arrow function creates a closure over the context in which it was defined. What that means is that all the variables referenced inside the arrow will always reference the same points in memory as when the arrow was first parsed.

This is incredibly useful for capturing local variables so that they can be used later, but it has the added benefit of capturing the value of this that was set when the arrow was defined. And, since this is (basically) always going to be the object being created during construction, we can use arrow functions to make methods where this will behave exactly like we expect:

// Rewriting Person with arrows
class ArrowPerson {
  name;

  constructor(theirName) {
    this.name = theirName;
  }

  introduce = () => {
    // The arrow captures `this` so it is actually a
    // reference to THIS Person.
    console.log("Hello I'm " + this.name);
  }
}

const arrowBill = new ArrowPerson("Arrow Bill");
arrowBill.introduce(); // "Hello I'm Arrow Bill"

// Now `this` is fixed even as we pass the function around:
const introduceRef = arrowBill.introduce;
introduceRef(); // "Hello I'm Arrow Bill"
introduceRef.call(puppy); // "Hello I'm Arrow Bill"

`this` all makes more sense now

I hope you understand this a little bit better now. To be honest, I think I understand it better just from writing this all out. And, because the Javascript this can affect all your code that transpiles into Javascript, hopefully this will also help you understand the twists and turns of function context in other languages like Typescript.

If you have any questions about this, drop them in the comments below. Even after years writing for the web, I'm still learning so I'm sure there are terrible dangers and cool facts about this I forgot or don't yet know.

Programming as the Means of Last Resort

Zack Sheppard — Mon, 22 Mar 2021 14:37:12 +0000

There's a coming-of-age moment for programmers where you're checking out a product's website, you switch to their /pricing page, and immediately scoff "no way - I'll just build it myself."

Let's be honest - it's a little bit of a rush. You look at something that costs $20, $200, even $2000 and feel a surge of confidence. You know your field and your abilities well enough to understand (or at least guess) how this thing was built, and you feel powerful enough to do that work yourself.

Congratulations. Now stop.

It's a trap. You've heard the saying, "when all you have is a hammer, everything looks like a nail." Programming is like that but also you have the superpower to actually transform anything into a nail. But even if you are that strange superhero, sometimes it's still cheaper and easier to just buy some glue.

It's time to internalize that programming is the means of last resort. This is the crux of what "no-coders" have figured out - you can actually build a reasonably good prototype for a huge majority of digital products without writing a line of code. Even as someone who loves to code and wouldn't personally go zero-code, it's time I start asking before I git init a new project: can I borrow this from somewhere or does it make sense to buy it?

"Borrow", if you can

Of course, if you can get exactly what you need for free, go for that. Your first thought here is probably to look for a snippet you can copy-paste off Stack Overflow, or an open-source Github project you can pull in via npm. Those are great instincts, but remember that sometimes free closed-source software can fit the bill as well. If you are starting a new mailing list, ConvertKit has a generous free tier. If you need to host a website, Netlify's free tier can cover you up to literally tens of thousands of visitors. Etc.

But this isn't especially new advice - so much of the web is built on the back of open-source packages from Apache to basically everything on npm. It's once you pass the limits of what is available for free that your engineer brain starts to trap you.

Buy, if you are able

There are so many small, repetitive tasks hidden within building a digital product that are trivial - both in terms of difficulty and price - that you can now solve with SaaS and PaaS tools thanks to the explosion in No Code. But first you need to to have a conversation with that voice in your head goading you into building it yourself. Remind it that building it yourself isn't the same as it being free. You have limited hours in your day, the same way you have limited dollars in your bank account.

As an example, last week I was writing a blog post about a lesson I learned while working on Tweet Sweep and I needed a simple website to link to for that work-in-progress. I know how to build websites, so I started on the ~2 hour journey of spinning up a simple Next.js site on Netlify. But then I remembered this post draft and I put down my hammer.

Instead, I signed up for Carrd - a No Code darling - for $14 / year. It took me 10 minutes, I made a website about 80% as good as the one I would have made from scratch, and I saved over an hour of my life. For me, this tiny transaction of 90 minutes of freedom for less than the price of a take-out dinner felt like the biggest win of my week.

Build, if you must

The cost-benefit analysis of buying a tool could go the other way, though. There are SaaS tools out there that precisely solve problems I've faced, but which unconscionable amounts of money (or who I would feel morally icky working with). While paying $1000 to save 50 hours is the same ROI as I was happy to take with Carrd, it's not one I would take for some simple reason: I don't have $1000 to stake on a side-project.

Worse, you might find that there is nothing out there that does quite what you want. For a number of reasons, I've moved away from Google Analytics for my projects to Plausible, but there's no reasonably-priced Google Analytics alternative for native applications (that I know about). So I'm out of luck if I want an affordable solution for native analytics.

Situations like these are why it's good to still have the hammer of being able to "make it yourself" in your back pocket. Building still makes sense if you don't have the budget to buy a premade solution or if a premade solution doesn't even exist to be bought. In those cases, you have the power as a coder to either find the time to build your own workaround.

But I know myself well enough to know that I can afford one, maybe two, of these construction detours in the course of a project. That is why I need to make programming my means of last resort. Every time I take the long way around, I risk coming to dread the project or getting so engrossed in one of those detours I'll wind up making it instead of the thing I was working on initially. So I have to be wise, ration out my attention, and keep my eye on the prize, as it were. I think that's the secret to how the No-Code community seems to turn out prototypes with such terrifying speed - if you're stringing together premade links into a chain you can move so much faster than if you get feel compelled to craft each link by hand.

Intro photo by Jurriaan Persyn - Licensed CC-BY-NC

Understanding Suspense-ful coding in React

Zack Sheppard — Mon, 08 Mar 2021 18:02:01 +0000

TLDR; <Suspense> is cool and a useful way to handle async loads in your React apps, but it has some tricky gotchas about data flow and caching if you try to use it with bare Promises. I wrote an npm package to avoid these pitfalls that I've open-sourced - suspension.

I've been working this week on a little webapp to practice integrating React apps with Firebase backends. As part of this project, I pulled in reactfire, which is the first module I've used that had first class support for the new React <Suspense> component. I'd heard about this component before but it was finally time to do a deep dive into what it was, how it worked, and how I could integrate it more deeply into my React apps going forward.

What is Suspense?

Suspense was the first component from React's experimental Concurrent mode to be merged into the non-experimental release (way back in 16.6). Suspense's job is to detect the need for an async load and render a fallback loading UI.

function CalendarApp() {
  const [viewedDay, setViewedDay] = useState(new Date());
  // Assuming that CalendarDayView is ready to work with Suspense,
  // this renders your loading spinner while today's data is loading.
  return (<main>
    <Suspense fallback={<LoadingSpinner />}>
      <CalendarDayView date={viewedDay} />
    </Suspense>
  </main>);
}

If that was all it did, it would be basically syntactic sugar over the tried-and-true pattern of if (callStatus === "loading") { return <LoadingSpinner />; }. But Suspense has a superpower that very few people are talking about, but to understand it we have to first understand how this component works.

How does Suspense work?

Suspense works by mildly abusing the throw statement. A component or hook that wants to indicate that it is still loading and needs more time should throw a Promise that will resolve when the component is ready for its render to be reattempted.

function CalendarDayView({ date }) {
  // Let's imagine our ORM has a cache of days' agendas we can check
  const cacheResult = calendarDB.cachedValue({ date });

  // To hook into Suspense, we recognize if we need to load and
  // throw a Promise that resolves when we're ready to try again.
  if (!cacheResult) {
    const loadingPromise = calendarDB.load({ date });
    loadingPromise.then((result) => {
      calendarDB.cache({ date, value: result });
    });
    throw loadingPromise;
  }

  // Otherwise do the render
  return (
    <h1>Calendar for {cacheResult.dayString}</h1>
    // ... and so on
  );
}

When we throw a Promise like this, React climbs the virtual DOM to find the nearest <Suspense> component and hands it the Promise.

This removes the whole tree under that Suspense from the rendered DOM and replaces it with the fallback.

This is how Suspense can give us superpowers. Because the throw interrupts our component's render process, we are guaranteed that if we get past it we are not loading. In the Calendar example above, we can be certain that if we get to the JSX at the bottom of the page then cacheResult is non-null and defined so we no longer have to guard against it being a missing value during a load. When the Promise that we threw resolves or rejects the <Suspense> will automatically try to re-render its children, giving us another chance to draw our calendar.

Gotcha 1 - Handling Errors

So one small gotcha here is that we've nicely separated out the "loading" case, but our component would still have to deal with the "API failed" case itself. Well, the React team have a suggestion for that too - again just throw your Errors and catch them with an error-boundary higher up in the tree. If you're committing to use Suspense, this is almost always the right answer as well since it neatly separates your components into loading, failed, and success cases. This is especially easy thanks to the react-error-boundary package.

Gotcha 2 - Avoiding Infinite Loading

There is a big gotcha with this system: how do you make sure you have your result when Suspense tries again? Since Suspense throws away the tree under it, the state of the component that threw the Promise (and by extension your hooks' state) will be destroyed during the load.

This is fine if you're loading from an API like our imaginary ORM above, where you can easily get the value if it's already cached. But if you're loading something from an API that always returns a Promise, like fetch, how do you get the result when you are told to retry? If you just naively call again, you can get stuck in an infinite load where every retry kicks off another call.

To escape this spiral, you need a cache that exists outside of your <Suspense>. This can be as complex as a fully cached data layer like Firestore or Apollo or it can be as simple as a stateful hook outside of your <Suspense>.

How do I use this with my Promises today?

So, to recap:

<Suspense> components catch Promises that their children throw if they're not ready to render.
They remove their children from rendering and display the Fallback instead. This destroys the children's state.
Because of this, you're almost always going to want a cache for the data so it's accessible when you get asked to re-render.

Putting all this together, the easiest way to convert existing Promise-based accesses to ones ready for Suspense would be to have a top-level cache that your components could send Promises to and later access the results synchronously. If you are already using a heavy datastore layer like Apollo or Redux, then you can use that. If you weren't using one of those, you could use a stateful hook tool like @andreiduca 's use-async-resource package. But I wanted a hook I could use inside the target component so I didn't have to prop-drill my reader function through the <Suspense>, so I built that:

Suspension - hook any async API to Suspense

Suspension uses the cache and call setup described above. You wrap your components in the <SuspensionRig> cache provider, which can also act as both a Suspense and/or an error boundary. Then, whenever you need data from a Promise, you pass it to suspension via a hook and it handles the logic of deciding whether to load, throw, or return a value for you.

Here's how we'd rewrite our Calendar app from above to use Suspension. First we swap out our base Suspense for a SuspensionRig:

import { SuspensionRig } from 'suspension';

function CalendarApp() {
  const [viewedDay, setViewedDay] = useState<Date>(new Date());
  return (<main>
    <SuspensionRig fallback={<LoadingSpinner />}>
      <CalendarDayView date={viewedDay} />
    </SuspensionRig>
  </main>);
}

And then we rip out our cache-or-load logic from above and replace it with one call to the useSuspension hook:

import { useSuspension } from 'suspension';

function CalendarDayView({ renderDay }: { renderDay: Date }) {
  // useSuspension takes a function that triggers your async work,
  //  a cache key to track the result, and the dependencies that
  //  trigger a new load (passed as args to your load function).
  const today = useSuspension(
    (date: Date) => calendarDB.load({ date }),
    'load-day-view',
    [renderDay]
  );

  // The hook guarantees that `today` will always be defined.
  return (
    <h1>Calendar for {today.dayString}</h1>
    // ... and so on
  );
}

All our logic about caching and trigger loads and throwing values gets collapsed into the hook and Suspension handles it all for us.

await React.future()

Learning about <Suspense> this past week has reignited my excitement about React. The whole experimental concurrent feature set feels like a new, simplified mental model for understanding concurrent loads in our UI.

Please check out Suspension - npm install suspension is ready to go. I hope it helps you dive into <Suspense> sooner and with more confidence - let me know if you find it useful or run into issues. The project's Issues and PRs are open for requests and contributions.

View Suspension on Github to read more about how to use it.