DEV Community: Matt Kane

How to do ISR and advanced caching with Remix and Netlify

Matt Kane — Wed, 14 Feb 2024 12:37:00 +0000

TL;DR

ISR is a powerful pattern for rendering pages on the web. You can use it in any framework that supports SSR, but Remix gives you some tools that make it particularly well suited, and it does it all using standard HTTP headers. You don't need anything proprietary to do this as long as your CDN supports the right primitives. You can then take this further with more advanced caching tools, giving you the best of both worlds: a fast, cached site that's always fresh.

What is ISR?

Incremental Static Regeneration or ISR is an approach to page rendering that lets you cache a page when first requested, and then control when that page is regenerated - either after a specific period, or by manually revalidating it. It's great for pages that don't change very often, or which don't need to be absolutely fresh when they are requested but which may still change between deploys. It was popularized by Next.js, which implements it internally within the framework, but it's a pattern that can be implemented with any framework in a standards-based way using HTTP headers with a CDN that supports them.

How to implement ISR using Remix

Remix uses native HTTP caching for ISR, where the pages are cached at the CDN level using Cache-Control headers. The team behind Remix built it with the philosophy of "use the platform" – meaning that whenever possible, you should use the tools that the web gives you rather than inventing something proprietary.

Netlify has a powerful set of caching features that let you do any kind of ISR on Remix. This guide will show the general principles of ISR and advanced caching with Remix, as well as the details of implementing them with Netlify's edge network.

Caching on Remix

To cache a Remix page, you use the headers function, which lets you define headers at the route level. Because Remix uses nested routes, it uses the headers returned by the deepest route, so if you want to define routes at different levels you need to make sure you merge them. Here is a basic headers function, with no merging:

export const headers: HeadersFunction = () => ({
  "Cache-Control": "public, max-age=300, s-maxage=3600",
  // Sets the cache time for browser ☝️ and CDN ☝️
});

This caches the reponse in the browser (max-age) for 5 minutes, and in the CDN (s-maxage) for 1 hour.

Careful!

Notice the difference in the use of a dash in max-age, vs no dash in s-maxage. This inconsistency in the standard is a source of much pain!

This is basic caching: if a user loads the page within an hour they will get the cached response, but if it's more than an hour then it will block while it re-renders.

If the data is unlikely to change, you can set a long expiry time. Don't worry if you find you need to change it later – any new deploy will clear the CDN cache instantly and your visitors will get the fresh content, without broken asset links. It's important that when you do this it's only for the CDN cache, not the browser cache, because otherwise your users may be left with stale content in their cache and you would have no way to invalidate it.

Using `stale-while-revalidate` for faster loading

But what if your page is slow to render? Maybe it needs to load data from a slow API or database. You can still get fast loading for revalidated pages though using the stale-while-revalidate directive. This does what it says: if the cached page has expired, then it will send a stale version while it revalidates the page in the background. The page load is never blocked for the user, though it won't be perfectly fresh for everyone.

We can add the directive to the Cache-Control header, but there is a risk here because this will also be cached in the browser, so you may end up with conflicting versions or double-revalidations trying to take place. Luckily there is a workaround for this which is the CDN-Cache-Control header. This is not used by the browser, only by the CDN, so there's no risk of conflicting directives. But what if there's another CDN cache that sees this header? This won't apply in most cases unless you're running one CDN in front of another, we can remove the risk entirely by using a CDN-specific cache header. Netlify uses the Netlify-CDN-Cache-Control header, which has the same semantics as normal Cache-Control headers, but is ignored by the browser and other CDNs, so there's no risk of it caching in the wrong place.

Once we are using the CDN-specific header, we can change the regular Cache-Control header to ensure it always checks the freshness of its cache. This doesn't mean it will download it every time, because the browser will do a conditional request that ensures it only downloads it if it has changed. The reason we need to do this is if for example you have deployed a new version of the site, which will mean the CDN cache is invalidated. If your browser cache has an old version of the page, it will be requesting assets with hashed filenames that no longer exist. Worse, it may be requesting assets that do exist, but are not the ones you want. This is why it's important to always check the freshness of the cache in this scenario.

Here's how to use it:

export const headers: HeadersFunction = () => ({
  // Tell the browser to always check the freshness of the cache
  "Cache-Control": "public, max-age=0, must-revalidate",
  // Tell the CDN to treat it as fresh for 5 minutes, but for a week after that return a stale version while it revalidates
  "Netlify-CDN-Cache-Control": "public, s-maxage=300, stale-while-revalidate=604800",
});

This header says that the page is stale after 5 minutes, but tells the CDN to return the stale response and regenerate it in the background, unless it's over a week old. A popular page will always be fresh, but a rarely-visited one will not keep re-rendering.

On-demand revalidation with cache tags

While max age and stale-while-revaidate are a nice and easy way to implement ISR on Remix, we're not really using the power of Remix and the CDN. Up until now we've just been using a headers function, but Remix loaders can also return headers. This is a powerful pattern, because it keeps the cache logic next to the thing that probably drives it - the data. Handling caching in the loader gives us the power to scope our revalidations to exactly the pages that use the data that may have changed. This is because Netlify supports the Cache-Tag header, with fine-grained revalidation by tag. Netlify also supports the Netlify-Cache-Tag header with the same meaning but specific to just Netlify's CDN, and which will not be visible to the browser.

To see how cache tags work, let's look at an example. Imagine you are creating a store. You will have product pages, listings and categories. On the listing page you need to fetch all products, in the category page you need the products in one category, and on the product page you just need one product's data. Latency is everything in e-commerce, so you want to cache your pages as efficiently as possible. For this reason you will use advanced caching on Netlify, with on-demand revalidation. When a product is edited, you only want to invalidate routes that use that data, and keep everything else cached for a long period. Let's look at a simplified loader for the listing route.

export const loader = async () => {
  const products = await myStore.products.findMany();
  return json(products);
};

Now let's add some cache tags. Remix lets you return a Response object from a loader, or you can use the json helper to simplify it, passing the headers in the second argument.

export const loader = async () => {
  const products = await myStore.products.findMany();
  return json(products, {
    headers: {
      // Always revalidate in the browser
      "Cache-Control": "public, max-age=0, must-revalidate",
      // Cache for a year in the CDN
      "Netlify-CDN-Cache-Control": "public, s-maxage=31536000",
      // Purge from the cache whenever the products change
      "Cache-Tag": "products",
    },
  });
};

// Just return the loader headers. Merging logic can go here if you need it.
export const headers: HeadersFunction = ({ loaderHeaders }) => loaderHeaders;

By adding the cache tag "products", we can revalidate the listings page whenever the products change. The tag name has no semantic meaning for the CDN, but choosing the model name makes it more readable for you.

We'll get onto the detail of the actual revalidation in a bit. First let's look at how we can take this further. For our category pages we probably don't need to invalidate all of them if only one product has changed. We can handle this with cache tags too. Here's a loader for the category page:

export const loader = async ({ params }) => {
  const products = await myStore.products.findMany({
    where: {
      "category": params.category,
    },
  });

  return json(products, {
    headers: {
      "Cache-Control": "public, max-age=0, must-revalidate",
      "Netlify-CDN-Cache-Control": "public, s-maxage=31536000",
      // Tag with the category id
      "Cache-Tag": `products:category:${params.category}`,
    },
  });
};

This lets us revalidate the route when only products in that category have changed.

We can be even more granular for the product page. Here we're loading just one product, and tagging it with just that product's id.

export const loader = async ({ params }) => {
  const product = await myStore.products.get(params.id);

  return json(product, {
    headers: {
      "Cache-Control": "public, max-age=0, must-revalidate",
      "Netlify-CDN-Cache-Control": "public, s-maxage=31536000",
      // Tag with the product id
      "Cache-Tag": `products:id:${product.id}`,
    },
  });
};

The cache tags we're using here can be any string. I've chosen this scheme of separating the model, field and id with colons, but you can use whatever scheme you want. You can also send more than one tag. Just separate them with commas:

export const loader = async ({ params }) => {
  const product = await myStore.products.get(params.id);
  const offers = await myStore.offers.findMany();

  return json(
    { product, offers },
    {
      headers: {
        "Cache-Control": "public, max-age=0, must-revalidate",
        "Netlify-CDN-Cache-Control": "public, s-maxage=31536000",
        // Tag with the product id and all offers
        "Cache-Tag": `offers,products:id:${product.id}`,
      },
    }
  );
};

This says that this route should be revalidated when either that one product is updated, or if any of the offers change.

Remember!

Remix always uses the headers from the deepest route, so if you're using loaders in parent routes, make sure to merge them. See the docs for headers for more details.

How to revalidate your pages on demand

Now you're returning your pages with all the right cache headers, you need to revalidate them when the data actually changes. Netlify gives you two ways to do this. You can either hit the API directly, using a Netlify access token, or you can use a helper inside a serverless function. I'm going to show you how to do on-demand revalidation inside a Remix route action, but you could also do this in a resource route that's called by something like a CMS webhook.

Imagine this is on your admin page where you can edit the products. When the store owner edits a product or adds a new one, you want to invalidate any pages that use that data. We can do this using the purgeCache helper from the @netlify/functions library.

import { purgeCache } from "@netlify/functions";

export async function action({ request }: ActionFunctionArgs) {
  const body = await request.formData();
  const product = await myStore.products.update({
    where: {
      id: body.get("id"),
    },
    data: {
      title: body.get("title"),
    },
  });
  await purgeCache({
    tags: ["products", `product:id:${product.id}`, `products:category:${product.category}`],
  });
  return redirect(`/products/${product.slug}`);
}

You can see that in this case we're purging the products tag, which will revalidate the listings page, the specific category for that product (which will purge that category), and finally the tag for that individual product.

The beauty of this approach over purging by route is that you don't need to keep track of which routes need which data – each route can just return the cache tag header matching the data they use.

Fine-grained cache control

So far we've been caching each page as the same for each visitor. This is probably what you want for most public pages, but not always. The Vary header can be used to store different versions of a page according to the request headers. However this can be quite an inflexible instrument, so Netlify extended this with the Netlify-Vary header. A common case will be to control how query parameters are treated. The strict spec-compliant way that a cache will handle these is to treat the whole URL complete with query as the cache key, meaning that ?foo=bar and ?foo=baz will be treated as different pages. While this is what you want in some cases, there are a lot of times when this will break caching entirely. A very common example for this is social sharing. Links from social media will often have tracking parameters added automatically, and you don't want to cache a different copy of the page for each visitor. This is also the case if you are using ads with tracking IDs. Should ?fbclid=nnnn... be stored as a separate page than ?fbclid=mmmm...? Probably not. However you might be using the query param for something that you do want to use for the cache. A common example might be something like sorting, filtering or pagination. Netlify lets you control exactly which query params are used for the cache key using the Netlify-Vary header. Here's how you can use it:

export const headers: HeadersFunction = () => ({
  "Cache-Control": "public, max-age=0, must-revalidate",
  "Netlify-CDN-Cache-Control": "public, s-maxage=300, stale-while-revalidate=604800",
  "Netlify-Vary": "query=sort|page|per_page",
});

In this example it will treat pages with different values for sort, page and per_page as different pages, but will ignore any other query params. This is a powerful tool for controlling your cache, and can be used in combination with cache tags for even finer control.

Cookies and caching

If there are user preferences that change how a page is displayed they needn't break caching. You probably shouldn't vary on the whole Cookie header, because this will mean that every user need their own version of the page cached, but you can use the Netlify-Vary header to control exactly which cookies are used for the cache key. Here's an example using a cookie to control the theme:

export const headers: HeadersFunction = () => ({
  "Cache-Control": "public, max-age=0, must-revalidate",
  "Netlify-CDN-Cache-Control": "public, s-maxage=300, stale-while-revalidate=604800",
  // Cache each theme separately
  "Netlify-Vary": "cookie=theme",
});

You can use this to choose cookies with low cardinality (i.e. ones that have a small number of possible values) to vary the cache on, which means you can have customized pages while still caching them. If there is a light theme and a dark theme, each of these will then be cached separately.

i18n and caching

If you have an internationalized site, you might want to cache different versions of the same page for different browser languages. If you are using different routes for different languages then there's nothing to worry about, but what if you are detecting the language from the user's request headers? You can use the Netlify-Vary header to control this, with the language directive. Here's an example using react-i18next:

import { useTranslation } from "react-i18next";

export const headers: HeadersFunction = () => ({
  "Cache-Control": "public, max-age=0, must-revalidate",
  "Netlify-CDN-Cache-Control": "public, s-maxage=300, stale-while-revalidate=604800",
  // Cache each language separately
  "Netlify-Vary": "language",
});

export let handle = { i18n: "home" };

export default function Component() {
  let { t } = useTranslation("home");
  return <h1>{t("title")}</h1>;
}

You can combine this with cookie-based language settings, by adding a cookie directive to the Netlify-Vary header.

Debugging your cache

Advanced cache tools like this can be tough to debug. The Cache-Status header can help, by showing whether the CDN had a had a hit, miss or stale response. Netlify supports this header, and you can use it to see what's happening with your cache.

Remember!

When running netlify dev it will strip all CDN cache headers, but the cache itself is not used. You need to deploy your site to see the cache in action.

Remix gives you powerful tools to control your cache, when combined with a platform like Netlify that supports advanced caching features. This gives you the power to control your cache in a way that's standards-based and uses the web platform.

Get started

If you'd like to see a real example, take a look at this contacts demo, which is based on the main Remix tutorial, but with the addition of advanced cache control.

To create your own Remix site to deploy to Netlify, you can get started with this command:

npx create-remix@latest --template netlify/remix-template

Or if you prefer, you can also deploy an example directly to Netlify from here by clicking the button below.

For more details, and to see how to migrate an existing site, see the Remix on Netlify docs.

Building an astrological GPT

Matt Kane — Thu, 28 Dec 2023 18:42:07 +0000

I'm not a believer in astrology in the slightest, but I've long been fascinated by it as a coder. When I was a kid just learning to code for the first time, a friend of the family was a professional astrologer, writing the horoscopes for a well known weekly women's magazine in the UK. Contrary to popular belief, he didn't just make them up, but actually carefully calculated the current celestial map using ephemeris software, which he then interpreted. I was fascinated with the charts that he'd noisily print off on the his dot matrix printer. Fast forward to today and this intersection of highly specific calculations and fantastical storytelling seems like a perfect candidate for a custom GPT using the OpenAI GPT API. In this post I'm going to show how to build an ephemeris API, and a custom GPT that uses it for a custom action. I'll also show how to use instructions to give responses in the proper astrological style.

If you have ChatGPT Plus, you can try the end result in ChatGPT. See the finished code for the API in GitHub.

To perform their readings, astrologers use an ephemeris to show the locations of all of the celestial bodies visible in the sky at a certain time and place. These used to be huge tables of pre-calculated coordinates, but even in the 1980s they were computerised. Today they can be easily calculated with any of many libraries. We can create an Action to provide this data to the GPT.

I'm going to create a Netlify Edge Function for this, because I like the API and it can give quick responses close to the OpenAI server. I'm using the Moshier ephemeris library because it doesn't need pregenerated data and performs all of the calculations locally. Netlify Edge Functions are based on Deno, so we can load it directly from a URL import using esm.sh:



// netlify/edge-functions/ephemeris.ts
import Ephemeris from "https://esm.sh/gh/0xStarcat/Moshier-Ephemeris-JS/src/Ephemeris.js";

export default async function handler(request: Request) {
    // do stuff
}

export const config = {
    path: "/ephemeris",
};

To get the ephemeris results, we need the location, date and time, so we'll these in the request. The ephemeris library needs the date and time parts to be passed in individually, so we'll parse the datetime string and then extract these values. If the user doesn't pass in the date then we will default to now.



export default async function handler(req: Request) {

    const url = new URL(req.url);
    const datetimeString = url.searchParams.get("datetime");
    const latitude = parseFloat(url.searchParams.get("latitude")!);
    const longitude = parseFloat(url.searchParams.get("longitude")!);

    if (isNaN(latitude) || isNaN(longitude)) {
        return new Response("Invalid parameters", { status: 400 });
    }

    const datetime = datetimeString ? new Date(datetimeString) : new Date();
    const year = datetime.getUTCFullYear();
    const month = datetime.getUTCMonth();
    const day = datetime.getUTCDate();
    const hours = datetime.getUTCHours();
    const minutes = datetime.getUTCMinutes();

By default, the ephemeris library returns values for all of the celestial bodies that it knows about, but we can let ChatGPT choose the ones it's interested in, while defaulting to returning all of them:



const ALL_BODIES = [
    "mercury",
    "venus",
    "mars",
    "jupiter",
    "saturn",
    "uranus",
    "neptune",
    "pluto",
    "sun",
    "moon",
    "chiron",
];


export default async function handler(req: Request) {

    const url = new URL(req.url);
    let celestialBodies = url.searchParams.get("bodies")?.split(",");
    if (!celestialBodies || celestialBodies.length === 0) {
        celestialBodies = ALL_BODIES;
    }

Now we have all of the details we need, we can pass these to the library and return the results:



    const ephemeris = new Ephemeris({
        year,
        month,
        day,
        hours,
        minutes,
        latitude,
        longitude,
        calculateShadows: false,
    });

    const results: Record<string, unknown> = {};
    for (const body of celestialBodies) {
        if (ephemeris[body]) {
            results[body] = ephemeris[body].position;
        } else {
            results[body] = "Celestial body not found";
        }
    }

    return Response.json(results);

This gives us the data we need, and returns it. See the full code here.

Creating a repo with just that file is enough to deploy to Netlify, but I've added a static index.html file too, as well as a privacy policy because we'll need that later. Thanks ChatGPT for both. You can deploy your own copy by clicking this:

We're almost ready to start creating the custom GPT. To register an API as an action for a GPT, we need an OpenAPI spec for it. These are really verbose an annoying to write, so I usually just ask ChatGPT to do it for me. There's no need to go into detail typing the full response - ChatGPT can work it out for itself. You can see what we generated here. I've deployed it as part of the site, but that's not needed. Remember to change the URL to that of your own site!

Now we're ready to go. Let's create a GPT. Go to https://chat.openai.com/gpts/editor to start. You can find it linked from "Explore" in the ChatGPT sidebar.

The editor is quite simple to use, because it does most things for you via the chat interface. However, we should start by defining the Action. Click on "Configure", then scroll down to the bottomn, enable "Code Interpreter", the click "Create new action".

You then need to define your schema. If you deployed it earlier then you can just choose "import from URL". Otherwise paste it in. You might get an error message - if so, just share it with the ChatGPT session where you created the schema and it should fix it for you.

If you get no errors, then you can click on "Test" to try it out.

Now we get to the fun bit. Go back to the "Create" tab and just tell it what you want. You should include all sorts of detail, including the tone for the responses and how exactly to use the APIs.

ChatGPT will then use this to customise the GPT. That prompt gave these results:

You can try this out, and adjust the details as you wish. You can ask for a profile pic. As long as you included the privacy policy when creating the action, you can then publish your GPT. Enjoy! Try it out here

This GPT talks to your GraphQL server and will write your queries for you

Matt Kane — Wed, 22 Nov 2023 09:56:26 +0000

I've built a GPT that can talk to your GraphQL server, learn the schema, then build queries that return exactly the data you need. If you have ChatGPT Plus you can try it here. Read on to see what it can do and how I built it.

Some people love GraphQL and some people hate it, but both the lovers and the haters will agree that writing a query can be frustrating. I'm a big fan of GraphQL: it lets me get just the data I need for a page or app without loads of API calls. However even with great tools like Graphiql, constructing complex queries can be very laborious and error-prone. I'm sure I'm not alone in trying to get ChatGPT or Copilot to build queries for me, with mixed results. The problem is that it doesn't know your real API, and tends to guess and confidently generate broken queries.

The newly-launched custom GPTs let us get around that, and it's worked far better than I could ever imagine.

How to use it

To use it, you first tell it about your server. You can let it know the sort of data you need, or you can start by just asking it to explore the API.

Once you tell it the data you need, it will help build a query.

You can then ask it to try the queries. If there are errors it can fix them.

It can then display the returned results, even showing images inline.

How I made it

Regular ChatGPT can browse the web and read docs, but it can't directly talk to APIs. This means that it tends to make mistakes when generating queries and can't easily correct itself. Now custom GPTs can use "Actions" that let them perform other tasks. I built an action that allows ChatGPT to contact a GraphQL server, get lists of available types, load a schema, and validate and run queries. By giving it direct access to the server it can be sure that the queries are correct and return exactly the data you need.

Tips

Because chats are persistent, you can keep one chat per server that you work with, and it will remember the schema. You can iterate on queries and add and remove fields. If you update the schema, just ask it to refresh and it will do so.

It supports authenticated servers too: just tell it your API token. Be aware that you will be pasting a plain text key into the chat. The action does not store or log the key or any of your data, and just uses it to execute your requests, but you do need to trust ChatGPT and me, so don't use it for anything critical.

Try it out and let me know what you think.

🏎️💨 Turbocharge your builds with a Turborepo remote cache in a single edge function

Matt Kane — Wed, 15 Nov 2023 13:55:08 +0000

Monorepos are a great tool for organizing projects, but they can lead to slow build particularly in complex projects with lots of interdependent packages. Turborepo is a great solution for improving your build times which has become particularly popular for Next.js sites. It uses lots of clever tricks to ensure you only build the absolute minimum needed, using lots of caching and dependency tracking. I'm going to share how I built a custom Turborepo remote cache in a single edge function.

The secrets of Full Turbo

One of Turborepo's neatest tricks is a global remote cache, which lets you share build artifacts between your local development environment and production builds and between different members of your team. It works in the background to speed up your builds everywhere.

Turborepo is owned by Vercel, so understandably they want you to use their cache service. However Turborepo does support custom remote caching, and there is already a great open source Turborepo remote cache project, but I had a feeling I could build something simpler.

A cache server in 100 lines of code

While the remote cache server API is undocumented, it is fortunately quite simple. I had early access to the new Netlify Blobs product, which seemed perfect for this. With an Edge Function handling the API and Netlify Blobs for the storage, I implemented a Turborepo remote cache in under 100 lines of code.

If you just want to start speeding up your builds, you can check out the repo or just click the button:
If you want to see how I built it, read on.

The Turborepo remote cache API

The docs for the Turborepo custom cache currently consists of a link to the Go source of the client, but the API is fortunately very simple. A Turborepo remote cache needs an endpoint at /v8/artifacts/:hash that accepts PUT requests to add an item, and GET requests to return it. Authentication is via a shared bearer token, and the teamId is passed as a query param. This is super simple to set up as a Netlify edge function:

import type { Config } from "@netlify/edge-functions";

export default async function handler(request: Request, context: Context) {
   // Do cool stuff here
}

export const config: Config = {
    method: ["GET", "PUT"],
    path: "/v8/artifacts/:hash",
    // This lets us handle our own cache rules
    cache: "manual",
};

This gives us a function that runs on the right path, with the right methods.

Now add some auth. Turborepo sends the token in an Authorization header, which we can store in a shared environment variable. You can use anything for the token value. I like to use a UUID, which you can generate on a Mac by running uuidgen in the terminal. You'll need to store the same token on the server site and any repos that use it.

// snip
export default async function handler(request: Request, context: Context) {
  const bearerHeader = request.headers.get("authorization");
  // Remove the leading "Bearer " from the token
  const token = bearerHeader?.replace("Bearer ", "");
  // Compare it with the stored value, and return a 401 if it doesn't match
  if (!token || token !== Netlify.env.get("TURBO_TOKEN")) {
    console.log("Unauthorized");
    return new Response("Unauthorized", { status: 401 });
  }
}

We're going to use Netlify Blobs to store our artifacts. These don't need any setup - you can just start using them. We'll use a store based on the team ID:

import { getStore } from "@netlify/blobs";
import type { Context } from "@netlify/edge-functions";

export default async function handler(request: Request, context: Context) {
  // Auth stuff goes here...

  const url = new URL(request.url);

  const teamId = url.searchParams.get("teamId") ?? "team_default";

  // Get a store
  const store = getStore(`artifacts-${encodeURIComponent(teamId)}`);

Now we just need to handle getting and putting the artifacts. First we generate the cache key for the object and then handle uploads:

  const hash = context.params?.hash || url.pathname.split("/").pop();

  if (!hash) {
    console.log("Missing hash");
    return new Response("Not found", { status: 404 });
  }

  const key = encodeURIComponent(hash);

  if (request.method === "PUT") {
    // Get the uploaded file as binary
    const blob = await request.arrayBuffer();
    if (!blob) {
      console.log("No content");
      return new Response("No content", { status: 400 });
    }
    // ...then put it in the store. That's all!
    await store.set(key, blob);
    return new Response("OK");
  }

Now we've implemented uploads, we can implement downloads.

  try {
    // Try to get the blob from the store as a binary buffer
    const blob = await store.get(key, {
      type: "arrayBuffer",
    });
    if (!blob) {
      // Oh no, return a 404
      return new Response(`Artifact ${hash} not found`, { status: 404 });
    }

    const headers = new Headers();
    // This content-type is required by Turborepo
    headers.set("Content-Type", "application/octet-stream");

    headers.set("Content-Length", blob.byteLength.toString());

    // We can set some useful cache headers, using Netlify's new cache features
    headers.set(
      "Netlify-CDN-Cache-Control",
      "public, s-maxage=31536000, immutable"
    );
    headers.set("Netlify-Vary", "header=Authorization,query=teamId");

    return new Response(blob, { headers });
  } catch (e) {
    // Catch any errors and return a 500
    console.log(e);
    return new Response(e.message, { status: 500 });
  }

That's it!

If you add that to your site at netlify/edge-functions/turbofan.ts then it will run and handle requests to the cache URL.

Easy mode

Instead of manually writing or copying the code, you can just use the version I've published to deno.land:

import type { Config } from "@netlify/edge-functions";

export { handleRequest as default } from "https://deno.land/x/turbofan/mod.ts";

export const config: Config = {
    method: ["GET", "PUT"],
    path: "/v8/artifacts/:hash",
    cache: "manual",
};

That's all you need. Make sure you've set the TURBO_TOKEN env var.

Or, even easier, just click this button to deploy a standalone remote cache:

Setting up your repo

You now need to tell turborepo to use the remote cache. You can do this by creating a config file. This is not the turbo.json in the root of your repo! You create it at .turbo/config.json, which may have been excluded in your .gitignore You will need to change that. Here's what you need to create:

{
    "teamid": "team_anything_you_want_here",
    "apiurl": "https://your-turboprop-server-name-here.netlify.app"
}

If you create that and set the TURBO_TOKEN env var, Turborepo will know to use the remote cache.

You can see it's working because Turborepo will print "Remote caching enabled" in the logs.

Have fun, and may all your builds be FULL TURBO.

Bring back the Twitter link preview

Matt Kane — Wed, 25 Oct 2023 23:11:09 +0000

In September 2023, Twitter (sorry, X) changed links to no longer show a preview of the link. They just show an image. This was done to make things look nicer, but makes it hard to see what is a link and what is just an image. You also can't see the title or description anymore. Many people don't want to click links if they don't know what the target is, so this is a problem.

Luckily there is a workaround. This post will show how you can use dynamic Open Graph images to bring back the Twitter link preview, while keeping other social media platforms happy.

The problem

Look at this tweet with a previous post of mine. It has a lovely photo, but aside from the little domain label there's no indication that it's a link or what the linked page is about.

This is what links used to look like. Now, you may agree that the other one looks nice, but I'm going to guess that this one will get more clicks.

Dynamic Open Graph images

I am going to use og_edge to generate dynamic Open Graph images. This is a fork of @vercel/og that makes it cross-platform so you can use it with frameworks other than Next.js.

I am going to deploy this to Netlify Edge Functions, and will show it with an Astro site, but the principles are the same for any framework.

og_edge uses Satori, which lets you use JSX to generate images, and runs in edge functions. This means that the image is generated on the fly, and you can pass in dynamic data to generate the image.

The code

I'm starting with the default Astro blog template. To create a Netlify edge function, you need to create a file in netlify/edge-functions in the root of your site. I'm going to call mine og.tsx.

This is the basic code to use og_edge:



// Just need something that renders JSX. You can use React, but Preact is fine.
/** @jsxImportSource https://esm.sh/preact */

// You don't need to install this, as it's loaded from the URL
import { ImageResponse } from "https://deno.land/x/og_edge/mod.ts";

export default function handler() {
  return new ImageResponse(<div>Hello world!</div>, {
    width: 960,
    height: 480,
  });
}

export const config = {
  path: "/og/twitter,
};

Satori supports basic styling, including flexbox. I want to create a design like the old Twitter link preview.



return new ImageResponse(
    (
      <div
        style={{
          color: "black",
          flexDirection: "column",
          justifyContent: "center",
          alignItems: "center",
          display: "flex",
          fontSize: 28,
          width: 960,
          height: 480,
        }}
      >
        <div
          style={{
            width: "100%",
            flex: 1,
            backgroundSize: "cover",
            background: `url(https://images.unsplash.com/photo-1601805825132-5d7bbab226d3?auto=format&fit=crop&w=960&h=380)`,
          }}
        ></div>
        <div
          style={{
            backgroundColor: "white",
            textAlign: "left",
            display: "flex",
            flexDirection: "column",
            width: "100%",
            padding "20px 48px",
          }}
        >
          <p
            style={{
              margin: 0,
            }}
          >
            Bring back the Twitter link preview
          </p>
            <p
              style={{
                marginTop: 10,
                marginBottom: 10,
              }}
            >
              Use dynamic OG images to bring back the Twitter link preview
            </p>

          <p
            style={{
              color: "#666666",
              margin: 0,
              fontSize: 26,
            }}
          >
            mk.gg
          </p>
        </div>
      </div>
    ),
    {
      width: 960,
      height: 480,
    }
  );
}
// This is the URL where the image will be available
export const config = {
  path: "/og.png",
};

I want to change the text to have bold headers, which means I need custom fonts. Right now I'm using the default font which is Noto Sans Regular. Satori allows us to load custom fonts, so I'm going to load the weights I need.



const notoBold = await loadFont(
  "https://cdn.jsdelivr.net/fontsource/fonts/noto-sans@latest/latin-800-normal.ttf"
);

const notoSans = await loadFont(
  "https://cdn.jsdelivr.net/fontsource/fonts/noto-sans@latest/latin-500-normal.ttf"
);

function loadFont(name: string) {
  return fetch(name).then((res) => res.arrayBuffer());
}

I've added a little helper to load the fonts, which just loads the font and returns the array buffer. I can then pass this to the constructor:



{
    width: 960,
    height: 480,
    fonts: [
    {
        name: "noto-sans",
        style: "normal",
        data: notoBold,
        weight: 800,
    },
    {
        name: "noto-sans",
        style: "normal",
        data: notoSans,
        weight: 500,
    },
    ],
}

Now let's add it all together by adding the styles to the JSX:



/** @jsxImportSource https://esm.sh/preact */

import { ImageResponse } from "https://deno.land/x/og_edge/mod.ts";

const notoBold = await loadFont(
  "https://cdn.jsdelivr.net/fontsource/fonts/noto-sans@latest/latin-800-normal.ttf"
);

const notoSans = await loadFont(
  "https://cdn.jsdelivr.net/fontsource/fonts/noto-sans@latest/latin-500-normal.ttf"
);

function loadFont(name: string) {
  return fetch(name).then((res) => res.arrayBuffer());
}

export default function handler(req: Request) {
  return new ImageResponse(
    (
      <div
        style={{
          color: "black",
          flexDirection: "column",
          justifyContent: "center",
          alignItems: "center",
          display: "flex",
          width: 960,
          height: 480,
        }}
      >
        <div
          style={{
            width: "100%",
            flex: 1,
            backgroundSize: "cover",
            background: `url(https://images.unsplash.com/photo-1601805825132-5d7bbab226d3?auto=format&fit=crop&w=960&h=380)`,
          }}
        ></div>
        <div
          style={{
            backgroundColor: "white",
            textAlign: "left",
            display: "flex",
            flexDirection: "column",
            width: "100%",
            fontSize: 28,
            padding "20px 48px",
          }}
        >
          <p
            style={{
              margin: 0,
              fontFamily: "noto-sans",
              fontWeight: 800,
            }}
          >
            Bring back the Twitter link preview
          </p>
          <p
            style={{
              fontSize: 28,
              marginTop: 10,
              marginBottom: 10,
            }}
          >
            Use dynamic OG images to bring back the Twitter link preview
          </p>
          <p
            style={{
              color: "#666666",
              margin: 0,
              fontSize: 26,
            }}
          >
            mk.gg
          </p>
        </div>
      </div>
    ),
    {
      width: 960,
      height: 480,
      fonts: [
        {
          name: "noto-sans",
          style: "normal",
          data: notoBold,
          weight: 800,
        },
        {
          name: "noto-sans",
          style: "normal",
          data: notoSans,
          weight: 500,
        },
      ],
    }
  );
}

export const config = {
  path: "/og/twitter",
};

Now this is all well and good, but you don't want to manually create this for every page. Now we can make it dynamic. I'm going to get the title, description and image URL from the query string.



// netlify/edge-functions/og.tsx
// Just need something that renders JSX. React works too.
/** @jsxImportSource https://esm.sh/preact */

import { ImageResponse } from "https://deno.land/x/og_edge/mod.ts";

const notoBold = await loadFont(
  "https://cdn.jsdelivr.net/fontsource/fonts/noto-sans@latest/latin-800-normal.ttf"
);

const notoSans = await loadFont(
  "https://cdn.jsdelivr.net/fontsource/fonts/noto-sans@latest/latin-500-normal.ttf"
);

function loadFont(name: string) {
  return fetch(name).then((res) => res.arrayBuffer());
}

// I've hard coded the domain here, but you could get it from the request
const domain = "mk.gg";

export default function handler(req: Request) {
  const url = new URL(req.url);

  const title = url.searchParams.get("title");
  const description = url.searchParams.get("description");
  const image = url.searchParams.get("image");

  let imageUrl;
  try {
    // Try to parse the image URL. Fail if it's not a valid URL.
    imageUrl = new URL(image);
  } catch (e) {}

  return new ImageResponse(
    (
      <div
        style={{
          color: "black",
          flexDirection: "column",
          justifyContent: "center",
          alignItems: "center",
          display: "flex",
          width: 960,
          height: 480,
        }}
      >
        <div
          style={{
            width: "100%",
            flex: 1,
            backgroundSize: "cover",
            // If there's no image, use a background color
            background: image ? `url(${imageUrl.toString()})` : "#000033",
          }}
        ></div>
        <div
          style={{
            backgroundColor: "white",
            textAlign: "left",
            display: "flex",
            flexDirection: "column",
            width: "100%",
            fontSize: 28,
            padding "20px 48px",
          }}
        >
          <p
            style={{
              margin: 0,
              fontFamily: "noto-sans",
              fontWeight: 800,
            }}
          >
            {title}
          </p>
          {description ? (
            <p
              style={{
                fontSize: 28,
                marginTop: 10,
                marginBottom: 10,
              }}
            >
              {description}
            </p>
          ) : null}
          <p
            style={{
              color: "#666666",
              margin: 0,
              fontSize: 26,
            }}
          >
            {domain}
          </p>
        </div>
      </div>
    ),
    {
      width: 960,
      height: 480,
      fonts: [
        {
          name: "noto-sans",
          style: "normal",
          data: notoBold,
          weight: 800,
        },
        {
          name: "noto-sans",
          style: "normal",
          data: notoSans,
          weight: 500,
        },
      ],
    }
  );
}

export const config = {
  path: "/og/twitter",
};

Now we have a dynamic image that we can use for any page. We can dynamically generate it by passing in the title, description and image URL as query parameters in the URL. I'm going to show how to do this with Astro, but most frameworks will have somethign similar. If the page has no description then that line is hidden, and if not image is provided then we use a background color.

I'm using the default Astro blog starter template which already allows you to set the title, description and OG image. We just need to change them a bit to use our edge function. Here's the original version in src/components/BaseHead.astro:



---

// Snip

const { title, description, image = '/blog-placeholder-1.jpg' } = Astro.props;
---
<!-- Snip -->
<!-- Open Graph / Facebook -->
<meta property="og:type" content="website" />
<meta property="og:url" content={Astro.url} />
<meta property="og:title" content={title} />
<meta property="og:description" content={description} />
<meta property="og:image" content={new URL(image, Astro.url)} />

<!-- Twitter -->
<meta property="twitter:card" content="summary_large_image" />
<meta property="twitter:url" content={Astro.url} />
<meta property="twitter:title" content={title} />
<meta property="twitter:description" content={description} />
<meta property="twitter:image" content={new URL(image, Astro.url)} />

If we just change the Twitter image then we can let other sites such as Facebook use the regular image, while Twitter gets the version with the embedded title and description.



---

// Snip

const { title, description, image = '/blog-placeholder-1.jpg' } = Astro.props;

// This is the path set in the edge function config
const og = new URL("/og/twitter", Astro.url);

og.searchParams.set("title", title);
og.searchParams.set("description", description);
og.searchParams.set("image", new URL(image, Astro.url).toString());

---
<!-- Snip -->
<!-- Open Graph / Facebook -->
<meta property="og:type" content="website" />
<meta property="og:url" content={Astro.url} />
<meta property="og:title" content={title} />
<meta property="og:description" content={description} />
<meta property="og:image" content={new URL(image, Astro.url)} />

<!-- Twitter -->
<meta property="twitter:card" content="summary_large_image" />
<meta property="twitter:url" content={Astro.url} />
<meta property="twitter:title" content={title} />
<meta property="twitter:description" content={description} />
<meta property="twitter:image" content={og.toString()} />

This will generate a dynamic image for Twitter, but use the regular image for other sites. Now we can see the title and description again.

You can of course customise the way the image is displayed, and if Twitter ever starts allowing links again then you can just remove the edge function and use the regular image.

You can check out the demo repo and the demo page

Level-up your web images with Unpic picture tags

Matt Kane — Fri, 06 Oct 2023 10:40:09 +0000

Responsive images are among the most important tools for improving the performance and user experience of a site. At its most basic, this means helping the browser load images with different resolutions according to the device's window size, and fluidly resizing the image to fit. This is done by including a srcset and sizes in your image tags, and using CSS for responsive resizing. The Unpic <Image> component is the easiest way to do this. It works automatically with your CMS, e-commerce provider or image CDN to deliver the right size and format for the device and has support for ten different frontend frameworks, including React, Astro, Svelte and Vue. However until now it has only supported this core use case. Now, I have added support for <picture> tags, which unlocks many powerful new features including art direction and even loading different images in dark mode.

Note: the images embedded in this post don't use Unpic, because dev.to doesn't support it.

Art direction

While normally you just want to change the size of an image for smaller screens, sometimes you want to get more creative. A beautiful panoramic shot might be best for a high resolution desktop, but a tight portrait or square crop might be better on a phone. Sometimes you might want to use an entirely different image. Art direction with a picture tag lets you do this, with different sources selected using media queries. Let's look at an example.

This dramatic lighthouse shot would be a great hero image, and can be shown in massive resolution on desktop

Photo by Karthik Sreenivas on Unsplash

Unsplash is a supported image CDN, so embedding it is super easy with Unpic. The example here is Astro, but it's similar in all supported frameworks.

<Image
  src="https://images.unsplash.com/photo-1654099749558-84ac187eb292"
  layout="fullWidth"
  alt="Lighthouse"
/>

The Unpic <Image> component generates the following html:

<img
        alt="Lighthouse"
        loading="eager"
        fetchpriority="high"
        style="object-fit:cover;width:100%"
        srcset="https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=640&fit=min&auto=format 640w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=750&fit=min&auto=format 750w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=828&fit=min&auto=format 828w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=960&fit=min&auto=format 960w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1080&fit=min&auto=format 1080w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1280&fit=min&auto=format 1280w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1668&fit=min&auto=format 1668w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1920&fit=min&auto=format 1920w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=2048&fit=min&auto=format 2048w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=2560&fit=min&auto=format 2560w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=3200&fit=min&auto=format 3200w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=3840&fit=min&auto=format 3840w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=4480&fit=min&auto=format 4480w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=5120&fit=min&auto=format 5120w,
https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=6016&fit=min&auto=format 6016w"
        src="https://images.unsplash.com/photo-1654099749558-84ac187eb292?fit=min&auto=format"
      />

Not something you want to generate by hand! It gives a responsive image that works on the largest displays and smallest phones, loads eagerly to keep LCP low and includes sources for all common screen sizes. However with a picture tag we can do something more interesting.

Here's another image of the same lighthouse, but a closer cropped portrait view. We can use a <picture> tag to display it on mobile, while keeping the hero image for larger screens.

Photo by Y S on Unsplash

The way you do it is with a normal <picture> tag, and the new Unpic <Source> component.

<picture>
  <!-- Hero image, displayed on screens 768px or wider -->
  <Source
    src="https://images.unsplash.com/photo-1654099749558-84ac187eb292"
    media="(min-width: 768px)"
    layout="fullWidth"
  />
  <!-- Portrait image for screens below that size -->
  <Source
    src="https://images.unsplash.com/photo-1601962468178-9f84128bd046"
    media="(max-width: 767px)"
    width={600}
    height={800}
  />
  <!-- Fallback image, also used for shared alt and loading props -->
  <Image
    src="https://images.unsplash.com/photo-1654099749558-84ac187eb292"
    layout="fullWidth"
    alt="Lighthouse"
    priority
    unstyled
  />
</picture>

We've added the unstyled prop to the Image because otherwise the inline styles will override our responsive styling. This is the generated HTML:

<picture>
  <!-- Hero image, displayed on screens 768px or wider -->
  <source
    srcset="https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=640&fit=min&auto=format 640w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=750&fit=min&auto=format 750w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=828&fit=min&auto=format 828w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=960&fit=min&auto=format 960w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1080&fit=min&auto=format 1080w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1280&fit=min&auto=format 1280w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1668&fit=min&auto=format 1668w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1920&fit=min&auto=format 1920w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=2048&fit=min&auto=format 2048w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=2560&fit=min&auto=format 2560w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=3200&fit=min&auto=format 3200w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=3840&fit=min&auto=format 3840w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=4480&fit=min&auto=format 4480w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=5120&fit=min&auto=format 5120w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=6016&fit=min&auto=format 6016w"
    media="(min-width: 768px)"
  />
  <!-- Portrait image for screens below that size -->
  <source
    sizes="(min-width: 600px) 600px, 100vw"
    srcset="https://images.unsplash.com/photo-1601962468178-9f84128bd046?w=600&h=800&fit=min&auto=format 600w,
  https://images.unsplash.com/photo-1601962468178-9f84128bd046?w=640&h=853&fit=min&auto=format 640w,
  https://images.unsplash.com/photo-1601962468178-9f84128bd046?w=750&h=1000&fit=min&auto=format 750w,
  https://images.unsplash.com/photo-1601962468178-9f84128bd046?w=828&h=1104&fit=min&auto=format 828w,
  https://images.unsplash.com/photo-1601962468178-9f84128bd046?w=960&h=1280&fit=min&auto=format 960w,
  https://images.unsplash.com/photo-1601962468178-9f84128bd046?w=1080&h=1440&fit=min&auto=format 1080w,
  https://images.unsplash.com/photo-1601962468178-9f84128bd046?w=1200&h=1600&fit=min&auto=format 1200w"
    media="(max-width: 767px)"
  />
  <!-- Fallback image, which is also used for shared alt and loading props -->
  <img
    alt="Lighthouse"
    loading="eager"
    fetchpriority="high"
    srcset="https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=640&fit=min&auto=format 640w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=750&fit=min&auto=format 750w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=828&fit=min&auto=format 828w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=960&fit=min&auto=format 960w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1080&fit=min&auto=format 1080w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1280&fit=min&auto=format 1280w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1668&fit=min&auto=format 1668w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=1920&fit=min&auto=format 1920w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=2048&fit=min&auto=format 2048w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=2560&fit=min&auto=format 2560w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=3200&fit=min&auto=format 3200w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=3840&fit=min&auto=format 3840w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=4480&fit=min&auto=format 4480w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=5120&fit=min&auto=format 5120w,
  https://images.unsplash.com/photo-1654099749558-84ac187eb292?w=6016&fit=min&auto=format 6016w"
    src="https://images.unsplash.com/photo-1654099749558-84ac187eb292?fit=min&auto=format"
  />
</picture>

A normal Unpic <Image> component handles all the styling for you, but we had to use unstyled here because we can't use responsive breakpoint with inline styles. Instead we'll be adding CSS to do the styling. In Astro we're using a regular <style> tag, but you can use your favourite styling solution.

We need a breakpoint for each size, but the actual CSS is quite simple:

<picture>
  <!-- Hero image, displayed on screens 768px or wider -->
  <Source src="https://images.unsplash.com/photo-1654099749558-84ac187eb292" media="(min-width: 768px)"
    layout="fullWidth" />
  <!-- Portrait image for screens below that size -->
  <Source src="https://images.unsplash.com/photo-1601962468178-9f84128bd046" media="(max-width: 767px)"
    layout="constrained" width={600} height={800} />
  <!-- Fallback image, which is also used for shared alt and loading props -->
  <Image src="https://images.unsplash.com/photo-1654099749558-84ac187eb292" layout="fullWidth" alt="Lighthouse" priority
    unstyled />
</picture>

<style>
  /* Style for all layouts */
  .hero img {
    object-fit: cover;
    width: 100%;
  }

  @media (min-width: 768px) {
    /* Optionally add styles you want for full width here, e.g. max-height */
  }

  @media (max-width: 767px) {
    /* Style for constrained layout */
    .hero img {
      max-width: 600px;
      aspect-ratio: 3/4;
    }
  }
</style>

This gives a full width hero image for large screens which uses the image's intrinsic size to set the aspect ratio, and smaller screens get a portrait image that keeps the 3:4 (i.e. 600x800) aspect ratio, while being constrained to 600px wide. You can use as many Source images and breakpoints as you need to work with your design.

Dark mode

While the <source> media attributes only supports a subset of media queries, it covers most of the ones you might need. Another interesting option is prefers-color-scheme, which lets you have different images for light and dark mode. This offers some creative possibilities. Let's find another photo of the lighthouse, this time a night shot.

Photo by Casey Horner on Unsplash

We can create a picture tag which switches the image according to the current color scheme.

<picture>
  <!-- Daytime, for light mode -->
  <Source
    src="https://images.unsplash.com/photo-1654099749558-84ac187eb292"
    media="(prefers-color-scheme: light)"
    width={800}
    height={600}
  />
  <!-- Nighttime, for dark mode -->
  <Source
    src="https://images.unsplash.com/photo-1522303099041-44f71373af66"
    media="(prefers-color-scheme: dark)"
    width={800}
    height={600}
  />
  <!-- Always include the <Image> last -->
  <Image
    src="https://images.unsplash.com/photo-1654099749558-84ac187eb292"
    width={800}
    height={600}
    alt="Lighthouse"
  />
</picture>

We can use the automatic styling here, because the layouts and sizes are all the same.

Formats

All current browsers support modern image formats such as WebP and AVIF, which can offer much better quality and compression than old formats like JPEG and PNG. Most image CDNs support content negotiation, which means they will deliver the best format supported by the browser. This is handled automatically by Unpic, so normally you don't need to worry about it. Just put in your image URL and the user will be served AVIF, WebP or JPEG according to their browser. However not all image CDNs can do this. In these cases you need to specify the generated format, which means you need to use a <picture> tag if you want to deliver the best images to your users. Probably the most prominent example of a CDN that does this is Contentful. We can use this method to deliver AVIF, WebP or JPEG even though it doesn't support content negotiation.

---
import { Image } from "@unpic/astro"

const toy =
 "https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg";
---
<picture>
  <Source src={toy} type="image/avif" width={800} height={600} />
  <Source src={toy} type="image/webp" width={800} height={600} />
  <Image src={toy} width={800} height={600} alt="Toy" />
</picture>

This time you set type instead of media. This is the mimetype of the image, which tell the browser which image to request. You can see we're using the same src for each – Unpic will automatically trasform the URL to request the correct format for each. This is the generated HTML. You can see that the URLs include the fm parameter to specify the format. Unpic handles this automatically for all supported image CDNs.

<picture>
  <source sizes="(min-width: 800px) 800px, 100vw"
    srcset="https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=640&h=480&fm=avif&fit=fill 640w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=750&h=563&fm=avif&fit=fill 750w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=800&h=600&fm=avif&fit=fill 800w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=828&h=621&fm=avif&fit=fill 828w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=960&h=720&fm=avif&fit=fill 960w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1080&h=810&fm=avif&fit=fill 1080w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1280&h=960&fm=avif&fit=fill 1280w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1600&h=1200&fm=avif&fit=fill 1600w"
    type="image/avif" />
  <source sizes="(min-width: 800px) 800px, 100vw"
    srcset="https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=640&h=480&fm=webp&fit=fill 640w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=750&h=563&fm=webp&fit=fill 750w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=800&h=600&fm=webp&fit=fill 800w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=828&h=621&fm=webp&fit=fill 828w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=960&h=720&fm=webp&fit=fill 960w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1080&h=810&fm=webp&fit=fill 1080w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1280&h=960&fm=webp&fit=fill 1280w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1600&h=1200&fm=webp&fit=fill 1600w"
    type="image/webp" /><img alt="Toy" loading="lazy" decoding="async" sizes="(min-width: 800px) 800px, 100vw"
    style="object-fit:cover;max-width:800px;max-height:600px;aspect-ratio:1.3333333333333333;width:100%"
    srcset="https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=640&h=480&fit=fill 640w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=750&h=563&fit=fill 750w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=800&h=600&fit=fill 800w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=828&h=621&fit=fill 828w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=960&h=720&fit=fill 960w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1080&h=810&fit=fill 1080w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1280&h=960&fit=fill 1280w,
  https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=1600&h=1200&fit=fill 1600w"
    src="https://images.ctfassets.net/yadj1kx9rmg0/wtrHxeu3zEoEce2MokCSi/cf6f68efdcf625fdc060607df0f3baef/quwowooybuqbl6ntboz3.jpg?w=800&h=600&fit=fill" />
</picture>

Remember: you don't need this for most hosts, just for ones that don't support content negotiation.

Try it out

This is just a taste of the things you can do with the <picture> tag in Unpic. You can combine media queries to give all kinds of combination. Try it out - it almost certainly supports your favourite frontend framework. Note that at time of writing, <picture> in Unpic is in beta and does not support web component frameworks (Lit and WebC).

A minimal, multi-framework, responsive image component

Matt Kane — Sat, 11 Feb 2023 16:31:08 +0000

Doing web images right can be hard. The <img> tag is just the starting point. In 2023, if you want the best performance you should be:

using srcset to deliver multiple resolutions for different device and screen sizes
using sizes so that the browser knows which image resolution to download
delivering modern image formats such as AVIF and WebP if the browser supports them
ensuring that the image resizes responsively, maintaining aspect ratio
avoids layout shift when the images has loaded
use native lazy-loading and async decoding for offscreen images
use high priority fetching for critical images
supports placeholders for lazy-loaded images

This isn't realistically something that can be done manually, but luckily many web frameworks provide tools to handle this. Depending on the framework, these may handle image resizing at build time or runtime, and may provide a component that makes it easy to embed the images. These all have drawbacks though - resizing at build time is slow, and the components often generate complex markup that is hard to style.

Let the CDN do the work

A lot of the trouble with embedding images is generating all the different sizes. A great way to solve this is with an image CDN, which resizes the image on the fly. You may have heard of the big names Cloudinary and Imgix, but what you might not know is that lots of other images that you're using are already on image CDNs. For example, CMSs such as Contentful, Sanity, Prismic and WordPress.com all deliver their images from a CDN that can resize on the fly. Shopify does too, as well as Unsplash. If your framework is downloading and resizing these then it is a huge waste. Next.js is a particularly egregious one here. I was curious about this and ran some queries on the data at Netlify – more than half of all next/image requests served by Netlify were for images from CDNs that could handle their own resizing.

Inspired by this, I built unpic, a library for detecting, parsing and generating image CDN URLs. The next step from that was to use this to create an image component that take any image CDN URL and generates all of the correct source images.

Unpic img: a simpler image component for every framework

I have created Unpic img, a minimal image component that makes it easy to do images well. It has some features that make it stand out:

It's just an <img> tag! No wrappers, no spacers. It doesn't even need a <picture> tag.
Just HTML and CSS. If it's pre-rendered there is no runtime JS at all.
Best practices by default. Large image, above the fold? Pass priority and it will ensure it's loaded with high priority fetch to keep your LCP low. Otherwise it will lazy-load it and use async decoding.
Choice of layouts. By default it uses constrained layout, which has a maximum image size but will scale down for smaller screens, maintaining aspect ratio. The fullWidth layout is designed for hero images, and has a default set of breakpoints based on all popular screen widths. The fixed layout is what it sounds like, but ensures it still generates the right sources for Retina displays.
Multi-framework. Currently it supports React, Vue, SolidJS and Svelte. Because there is no runtime script, it's simple to support multiple frameworks and PRs are welcome to add more. All the logic is in a shared @unpic/core library.
Simple API. It's an img tag, but better. Accepts any <img> attribute.

Here's what it looks like in React:

import { Image } from "@unpic/react";

function MyComponent() {
  return (
    <Image
      src="https://cdn.shopify.com/static/sample-images/bath_grande_crop_center.jpeg"
      layout="constrained"
      width={800}
      height={600}
      alt="A lovely bath"
    />
  );
}

This generates the following HTML:

    <img alt="A lovely bath" loading="lazy" decoding="async" sizes="(min-width: 800px) 800px, 100vw"
        style="object-fit: cover; max-width: 800px; max-height: 600px; aspect-ratio: 1.33333 / 1; width: 100%;" 
        srcset="https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=1080&amp;height=1440 1080w,
https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=1280&amp;height=1707 1280w,
https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=1600&amp;height=2133 1600w,
https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=640&amp;height=853 640w,
https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=750&amp;height=1000 750w,
https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=800&amp;height=1067 800w,
https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=828&amp;height=1104 828w,
https://cdn.shopify.com/static/sample-images/bath.jpeg?crop=center&amp;width=960&amp;height=1280 960w"
        src="https://cdn.shopify.com/static/sample-images/bath.jpeg?width=800&amp;height=600&amp;crop=center">

I know which one I'd rather write!

The equivalent code for Vue:

<script setup lang="ts">
import { Image } from "@unpic/vue";
</script>

<template>
  <Image
    src="https://cdn.shopify.com/static/sample-images/bath_grande_crop_center.jpeg"
    layout="constrained"
    width="800"
    height="600"
    alt="A lovely bath"
  />
</template>

Svelte:

<script lang="ts">
  import { Image } from "@unpic/svelte";
</script>

<Image
  src="https://cdn.shopify.com/static/sample-images/bath_grande_crop_center.jpeg"
  layout="constrained"
  width={800}
  height={600}
  alt="A lovely bath"
/>

...and SolidJS:

import type { Component } from "solid-js";
import { Image } from "@unpic/solid";

const MyComponent: Component = () => {
  return (
    <Image
      src="https://cdn.shopify.com/static/sample-images/bath_grande_crop_center.jpeg"
      layout="constrained"
      width={800}
      height={600}
      alt="A lovely bath"
    />
  );
};

It's very much a work in progress right now, but give it a try and let me know what you think. If you'd like to contribute a supported framework then PRs are welcome.

I'm Matt Kane, and I'm a principal engineer at Netlify, where I work on framework integrations. Previously I helped built the Gatsby image plugin

Benchmarking web dev tools on the Mac mini M1

Matt Kane — Sun, 13 Dec 2020 07:44:23 +0000

In the weeks after the new Apple Silicon Macs were launched there were plenty of reviews with benchmarks for tasks like video encoding, 3D rendering and so on. However I couldn't find anything testing the web development tools that I use every day, such as Node, Babel and TypeScript. A lot of the M1's speed comes from specialised cores, so there was a risk that it would not be as fast for web development. In the end I decided to risk it and order one and run some benchmarks myself.

I'd been looking for something to use as a headless compilation machine for a while. I'm an engineer on the Gatsby open source team, and have to compile the massive monorepo many times a day. We do a lot of a pair programming over Zoom, and that's particularly difficult when compiling. VS Code remote development is very slick, so when the M1 Macs were released, a Mac mini seemed just the job. I needn't have worried: the Mac mini M1 comfortably beats my Macbook Pro in every task I threw at it. This includes benchmarks, but also real world development tasks.

Test setup

I ordered the entry level Mac mini with 8GB of RAM, and my comparisons are with my work machine: a Late 2019 16" Macbook Pro 2.3GHz core i9 with 16GB of RAM. For my initial tests I used the V8 Web Tooling Benchmark. This uses Node to run several realistic tasks, using the sort of tools web developers use every day: Babel transpilation, and various other parsers, TypeScript, PostCSS, Prettier, ESLint and more. The benchmark itself is three years old, and uses old versions of these tools. It can run on the latest Node though, and should still be representative of the relative speeds.

To run the tests, I first installed Node 15.4.0 using Homebrew on the mini and using nvm on the MBP. I made sure that it was running natively by using the Node REPL to check process.arch. It will be arch64 if it's a native build, and x64 if running the Intel version using Rosetta2. I ran the benchmarks according to the instructions: npm install, npm build then node dist/cli.js. I ran the benchmarks several times to be sure they weren't outliers.

Results

V8 Web Tooling Benchmark. Higher scores are better

Mac mini M1

Running Web Tooling Benchmark v0.5.3…
-------------------------------------
         acorn: 21.29 runs/s
         babel: 14.64 runs/s
  babel-minify: 20.82 runs/s
       babylon: 21.11 runs/s
         buble: 10.25 runs/s
          chai: 27.62 runs/s
  coffeescript: 14.24 runs/s
        espree:  4.07 runs/s
       esprima: 15.86 runs/s
        jshint: 18.65 runs/s
         lebab: 18.26 runs/s
       postcss: 10.97 runs/s
       prepack: 15.17 runs/s
      prettier: 11.85 runs/s
    source-map: 16.93 runs/s
        terser: 35.89 runs/s
    typescript: 17.13 runs/s
     uglify-js: 11.56 runs/s
-------------------------------------
Geometric mean: 15.57 runs/s

2019 Macbook Pro 2.3GHz core i9

Running Web Tooling Benchmark v0.5.3…
-------------------------------------
         acorn:  9.80 runs/s
         babel:  9.04 runs/s
  babel-minify: 12.95 runs/s
       babylon: 12.37 runs/s
         buble:  6.21 runs/s
          chai: 18.01 runs/s
  coffeescript:  9.25 runs/s
        espree:  4.32 runs/s
       esprima: 10.14 runs/s
        jshint: 12.10 runs/s
         lebab: 13.86 runs/s
       postcss:  8.55 runs/s
       prepack:  9.06 runs/s
      prettier:  8.69 runs/s
    source-map: 11.68 runs/s
        terser: 20.48 runs/s
    typescript: 10.34 runs/s
     uglify-js:  6.61 runs/s
-------------------------------------
Geometric mean: 10.10 runs/s

The Mac mini M1 is consistently around 50% faster than the Macbook Pro in the Web Tooling Benchmark.

These are native speeds, with an arm64 build of Node. However neither Node.js nor Homebrew offically support Apple Silicon yet, and many native npm packages need to compile from source. There are many cases where you may need to run x64 tools using Rosetta2. This is inevitably going to be slower than native, but the good news is that it's still faster than the Macbook Pro.

Mac mini M1, via Rosetta2

Running Web Tooling Benchmark v0.5.3…
-------------------------------------
         acorn: 15.34 runs/s
         babel: 11.84 runs/s
  babel-minify: 15.46 runs/s
       babylon: 14.56 runs/s
         buble:  7.93 runs/s
          chai: 22.66 runs/s
  coffeescript: 11.60 runs/s
        espree:  5.51 runs/s
       esprima: 14.44 runs/s
        jshint: 15.92 runs/s
         lebab: 15.15 runs/s
       postcss: 10.44 runs/s
       prepack: 12.23 runs/s
      prettier:  9.76 runs/s
    source-map: 16.78 runs/s
        terser: 24.10 runs/s
    typescript: 10.36 runs/s
     uglify-js:  7.94 runs/s
-------------------------------------
Geometric mean: 12.64 runs/s

Even via Rosetta the Mac mini is around 25% faster than the Macbook Pro.

Real-world tests

While the benchmark seems to be representative of real-world web development tasks, I still needed to compare with my day-to-day tasks. If anything, the lead is even more there.

Running lerna bootstrap in a clean gatsby monorepo. This runs yarn install in the root of the monorepo, then yarn install across all 112 packages, and then yarn prepare which builds each of them and prepares it for publishing. Most packages are compiled with Babel, but some are compiled with the TypeScript compiler, and some of the Babel ones have TypeScript types generated. Some packages are bundled with microbundle or rollup. There's even a Gatsby site in there (the gatsby-admin app). It's a generally tough but representative sample. Lower times are obviously better!

Macbook Pro: 4 mins, 33 seconds
Mac mini M1: 2 mins, 45 seconds

This was run simultaneously on both machines (using iTerm input broadcasting), with both on the same network.

One thing I noticed while running it was that the yarn install was noticably faster on the mini, which I put down to faster disk IO. To see the other side of disk IO that most web developers need to do, I checked this by running rm -rf node_modules!

I tried several other tasks, such as running the gatsby-dev-cli which publishes all gatsby packages to a local npm server, and then installs those in a site for testing. Each of them was considerably faster with the Mac mini.

Conclusion

As I'm using mine as a remote compilation machine I haven't run tests on other apps, but there are plenty of other reviews that have. Based on my tests, the M1 Macs are incredibly fast for web development, and are an absolute bargain too. This was a bottom of the range Mac mini comfortably beating a top of the range Macbook Pro that's just a year old and cost four times as much.

Top 6 reasons Gatsby incremental builds can be slow (and how to make them fast)

Matt Kane — Sun, 03 May 2020 14:21:34 +0000

Note: I'm an engineer on the open source team at Gatsby, but this isn't an official post

You've heard Gatsby Incremental Builds are Blazing Fast. They can build a site in under 5 seconds! So why does your site still take over a minute to build? I worked on the incremental builds feature at Gatsby, and this is a question I've heard several times since the launch last month. In this post I'll share the top reasons that site builds can be slow with incremental builds enabled, and how to fix this. And to be clear: by slow, I mean more than about 10 seconds.

1. Your site doesn't use a CMS

First up, let's define our terms.

An incremental build is where only the pages, JavaScript and assets affected by a data change are rebuilt.
– Me

You may have heard that currently Gatsby incremental builds only support data changes, not code changes. What you might not have realised is that code changes are any changes in Git. That includes Markdown files! The reason for this is that right now we can't tell if a .md file is just Markdown or MDX with code in it until we parse it. As we only support data changes, we have to do a regular, non-incremental build. When you push a new Markdown file to GitHub, Gatsby will rebuild the whole site. This will change, because we plan to support code changes (I'm working on it), but right now Markdown sites won't get the speedy builds that you get from a supported CMS like Contentful, Dato or the beta WordPress plugin.

2. It's the first build, or the code changed

Even with incremental builds enabled, the first build is a full build so takes longer. Gatsby also currently does a full build if the code has changed, so builds triggered by GitHub will be slower.

3. Static queries that reference pages

OK you're using a supported CMS. You're editing just one page. So why is Gatsby rebuilding them all? We have to rebuild a page whenever the data in it changes. However we also have to rebuild all pages if any static query references a changed page. For example, if you have a static query that gets the total number of blog posts, then whenever you edit any post Gatsby will need to re-run that query. This means every page has to be rebuilt. We will be improving this by building a dependency graph that means we can be smarter at finding unchanged pages, but even then it will be easy to accidentally invalidate lots of pages with careless of of static queries. Try to minimise the use of static queries, and try to not use any that reference node types that change every time (such as blogPost or equivalent). Move the query into a page query if possible, as these don't cause every page to be rebuilt.

4. Templates that change on every build

Related to the previous point, if you have templates or other shared components that change a lot then you'll need to keep rebuilding every page that uses them. Do you really need a "latest post" link on every page that includes the post title? Even if you've managed to avoid having to re-run every page query, if you have something like a timestamp or other changable data in a component, then it means that the HTML will be different on every build, meaning even otherwise unchanged pages will need to be re-uploaded. Put things like that into a useEffect hook and they won't be run at build time.

5. Paginated archives

Do you have loads of paginated archive pages? If you do, then whenever you add a post, every page will need to be rebuilt as the posts shuffle back by one. If you have lots of posts then that could end up being a large number of pages. Instead of paginating them all, have a separate archive per day or month. This way only the most recent archive will need to change when you add a post.

6. You're not using Gatsby Cloud

Sorry to get all corporate (and to reiterate the disclosure above: I worked on incremental builds at Gatsby), but right now if they're not on Gatsby Cloud, they're not proper incremental builds. Some people have been confused by an experimental feature in Gatsby that let's you skip part of page generation when the data hasn't changed. This is kinda-incremental, but is absolutely not the same thing as the feature that Gatsby just launched. They don't use the same code, and you can't expect the same results. Importantly, the flag just skips one part of the build (which is why we don't refer to it as incremental builds). It's an important part, but is far from all of it. There are still lots of things that need to run every time, even if no data has changed. For that reason, you should still expect builds to take a minute or more. This is a great improvement if you have a large site that otherwise takes many minutes to build (and something like it will probably become the default), but it's not the same as the 5 second builds that you can get on Gatsby Cloud. Sorry! This isn't us holding back performance improvements from open source Gatsby: those kind of speeds are only possible when you have a full build and deployment pipeline that has been made just for Gatsby builds. We've been making loads of improvements to the open source project that are speeding up builds for everyone, sometimes by massive amounts (like 1000x), but for sites that have built and deployed before you've switched tabs you need to use Gatsby Cloud.

If you have any more questions about incremental builds, feel free to ask in the comments.

How to build a Gatsby plugin to display your DEV posts

Matt Kane — Tue, 28 Jan 2020 13:48:00 +0000

Note: Since my last Gatsby post I got a job at Gatsby on the open source team! This isn't an official post though.

DEV has a simple API that means you can use it as a basic CMS. It's also a good way to cross-post your DEV posts to your own site. There are already plugins that let you do this, but they don't support everything I need, and in any case it's a great excuse to learn how to write a Gatsby source plugin. Did you know that Gatsby is the #1 hottest tech skill to learn in 2020, according to Udemy?

First up, you should have a working Gatsby site. It's super quick to get started if you already have Node etc set up, but it's worth looking at the quick start. In this post we'll be using the default blog starter, so use the command:

gatsby new my-gatsby-blog https://github.com/gatsbyjs/gatsby-starter-blog

If you need more help getting set up, check out the step-by-step tutorial. When you have a site that runs, come back here and you'll build a plugin.

You can do this directly in your site's gatsby-node.js file, but the tidiest way to add a data source in Gatsby is to create a custom plugin. You don't need to publish it to NPM, but you can if you want to share your creation with the world. You just need to create a folder called plugins in the root of your site. Then create a folder with a name that matches your plugin. The only required file is a package.json.

Call yours gatsby-source-dev, so create the file /plugins/gatsby-source-dev/package.json

{
    "name": "gatsby-source-dev"
}

A Gatsby plugin can do all sorts of things, but you're building a source plugin, which adds a new data source. You can use this data like any other in your site. You'll be using it to create pages.

Create the file gatsby-node.js in the same folder, which does the actual processing. This file will be loaded and run by Node when Gatsby builds your site. If you export a function called sourceNodes then it will be called at the stage when the data is being loaded and created.

This is the most basic sourceNodes function. It doesn't need to be async, but yours should be because you'll be loading data from an API. The first argument passed to the function is an object with lots of useful properties for performing the creation of nodes. The most important is the actions property, which lets you do the actual node creation.

// /plugins/gatsby-source-dev/gatsby-node.js

exports.sourceNodes = async ({ actions }) => {
// Do cool stuff
}

Now, you're not creating anything yet, but this is a valid (but useless) plugin. If you add it to your site config (we'll get to that later), then it will be called whenever you build the site. Now it's time to do something useful with it.

The first thing you're going to want to do is load the data from the DEV API. As this isn't running in a browser, you can't use fetch etc, but you can use axios, which is included with Gatsby.

// /plugins/gatsby-source-dev/gatsby-node.js
const axios = require(`axios`);

exports.sourceNodes = async ({ actions }) => {
    const result = await axios.get(`https://dev.to/api/articles/me/published`);

}

If you try loading this now, you will get a "401 unauthorized" error. To fix this you'll need to get an API key from DEV.

You'll be passing this key as a header, but first you need to look at how you configure a plugin. You don't want to hard-code the key in your plugin, so you need to pass it in to the plugin from your config.

To tell Gatsby about you plugin you need to include it in the plugins array in your gatsby-config.js, and it is also where you set the options.

module.exports = {
// rest of your config here
    plugins: [
        {
            resolve: `gatsby-source-dev`,
            options: {
                apiKey: `your-key-here`
            }
        },
    ]

}

If you're using the blog starter you'll see a couple of entries for gatsby-source-filesystem. You can remove the one with name: "blog" because you'll be getting your posts from DEV instead. You'll also need to remove gatsby-plugin-feed for now, as it'll need extra configuration to understand the posts once you're done.

Now, this will pass the key through to the plugin but it's still hard-coded in the source, which you should not do as this key will let anyone post to DEV in your name. The answer is environment variables, which can be imported from a local file or set in your build environment such as Netlify or Gatsby Cloud. The local variables are loaded by the package dotenv, which is already installed with Gatsby.

First create the .env file in the root of your project:

GATSBY_DEV_API_KEY=your-key-here

...then update your gatsby-config.js to load and use it:

// /gatsby-config.js
require("dotenv").config();

module.exports = {
// rest of your config here
    plugins: [
        {
            resolve: `gatsby-source-dev`,
            options: {
                apiKey: process.env.GATSBY_DEV_API_KEY
            }
        },
    ]

}

The plugin options are passed to all of the gatsby-node functions, so you need to update your function to use the API key:

// /plugins/gatsby-source-dev/gatsby-node.js
const axios = require(`axios`);

exports.sourceNodes = async ({ actions }, { apiKey }) => {
    const result = await axios.get(`https://dev.to/api/articles/me/published`, {
        headers: { "api-key": apiKey }
    });
}

The second argument passed to the function is the options object, and you're getting the apiKey option. You're then passing that as a header to the DEV API.

Now it's time to make some nodes. Nodes are the building blocks of the Gatsby data layer, and you can query them to create pages and display data etc.

You need to loop through the results and create a node for each. The nodes that you create will mostly be a 1:1 copy of the objects returned from the DEV API. However you do need to do a couple of small tweaks to make it work properly. The best way to grab a couple of properties and pass through the rest is to destructure the object:

// ... etc 
    result.data.forEach(post => {
        const { id, body_markdown, ...data } = post;
        // do stuff
    });
};

Next you're going to need some more of the helper functions that Gatsby provides, so add those to the arguments:

exports.sourceNodes = async (
    { actions, createNodeId, createContentDigest },
    { apiKey }
) => {
    const { createNode } = actions;

    const result = await axios.get(`https://dev.to/api/articles/me/published`, {
        headers: { "api-key": apiKey }
    });

    result.data.forEach(post => {
        const { id, body_markdown, ...data } = post;

        // do stuff
    });
};

Now to create the actual nodes. The two changes you need to make are to create an proper node id, and to tell Gatsby to convert the markdown in the post. The clever thing is that all you need to do is set the media type and content, and the built-in markdown plugin will do the rest. It will even let you use plugins to do custom things with the markdown. The default blog starter includes plugins to do syntax highlighting, copying linked files and more. Any images used in the markdown will be automatically downloaded and processed.

exports.sourceNodes = async (
    { actions, createNodeId, createContentDigest },
    { apiKey }
) => {
    const { createNode } = actions;

    const result = await axios.get(`https://dev.to/api/articles/me/published`, {
        headers: { "api-key": apiKey }
    });

    result.data.forEach(post => {
        // Destructure two fields and assign the rest to `data`
        const { id, body_markdown, ...data } = post;

        // Create the node object
        const node = {
            // Create a node id
            id: createNodeId(id),
            internal: {
                // Tell Gatsby this is a new node type, so you can query it
                type: `DevArticle`,
                // Set the markdown content
                mediaType: `text/markdown`,
                content: body_markdown
            },
            // Spread in the rest of the data
            ...data
        };
        const contentDigest = createContentDigest(node);
        node.internal.contentDigest = contentDigest;

        createNode(node);
    });
};

That's the plugin done! Run gatsby develop (or restart it if it's already running) and open GraphiQL and you'll find devArticles and allDevArticles in the explorer.

You might get markdown errors when you run the build. The parser in Gatsby is less forgiving than the DEV one, so you may need to fix some errors. e.g. I had to remove a colon from the frontmatter on one of my posts.

You can then use these to create pages. If you've started with the Gatsby blog starter, as shown in the docs, you'll have a gatsby-node.js file in the root of your project with a createPages function already in it. We just need to change it a little to create pages for your DEV posts.

First change the createPages function to match the one below. It's almost the same, but has a few changes to the data structure:

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions

  const result = await graphql(`
    query DevArticleQuery {
      allDevArticle(filter: { childMarkdownRemark: { id: { ne: null } } }) {
        nodes {
          slug
        }
      }
    }
  `)

  if (result.errors) {
    throw result.errors
  }

  // Create blog posts pages.
  const blogPost = path.resolve(`./src/templates/blog-post.js`)

  const posts = result.data.allDevArticle.nodes
  posts.forEach((node, index) => {
    const previous = index === posts.length - 1 ? null : posts[index + 1]
    const next = index === 0 ? null : posts[index - 1]

    createPage({
      path: `posts/${node.slug}`,
      component: blogPost,
      context: {
        slug: node.slug,
        previous,
        next,
      },
    })
  })
}

The default blog also has an onCreateNode function that adds a slug based on the source filename. You need to remove because it gets confused by our markdown which isn't created from files and has a slug defined already.

This is enough to create the pages, however if you run gatsby develop now you'll get a load of GraphQL errors, because the page queries are still expecting the old markdown node types. You'll need to change anywhere that queries the pages.

First go to /src/pages/index.js and change the page query to:

  query ArticleQuery {
    site {
      siteMetadata {
        title
      }
    }
    allDevArticle {
      nodes {
        title
        slug
        published_at(formatString: "MMMM DD, YYYY")
        description
      }
    }
  }

...Then update the component to:

export const BlogIndex = ({ data, location }) => {
  const siteTitle = data.site.siteMetadata.title
  const posts = data.allDevArticle.nodes

  return (
    <Layout location={location} title={siteTitle}>
      <SEO title="All posts" />
      <Bio />
      {posts.map((node) => {
        const title = node.title || node.slug
        return (
          <article key={node.slug}>
            <header>
              <h3
                style={{
                  marginBottom: rhythm(1 / 4),
                }}
              >
                <Link style={{ boxShadow: `none` }} to={`posts/${node.slug}`}>
                  {title}
                </Link>
              </h3>
              <small>{node.published_at}</small>
            </header>
            <section>
              <p
                dangerouslySetInnerHTML={{
                  __html: node.description,
                }}
              />
            </section>
          </article>
        )
      })}
    </Layout>
  )
}

export default BlogIndex

If you run gatsby develop now you should be able to load the site and see a list of your posts on the front page. If you don't, carefully check the queries and error messages.

Now you need to update the post template. There's also not too much to change here.

Go to src/templates/blog-post.js and update the page query to match this:


  query BlogPostBySlug($slug: String!) {
    site {
      siteMetadata {
        title
      }
    }
    devArticle(slug: { eq: $slug }) {
      id
      title
      description
      published_at(formatString: "MMMM DD, YYYY")
      childMarkdownRemark {
        html
      }
    }
  }

...and then edit the component to change the data structures:

export const BlogPostTemplate = ({ data, pageContext, location }) => {
  const post = data.devArticle
  const siteTitle = data.site.siteMetadata.title
  const { previous, next } = pageContext

  return (
    <Layout location={location} title={siteTitle}>
      <SEO title={post.title} description={post.description} />
      <article>
        <header>
          <h1
            style={{
              marginTop: rhythm(1),
              marginBottom: 0,
            }}
          >
            {post.title}
          </h1>
          <p
            style={{
              ...scale(-1 / 5),
              display: `block`,
              marginBottom: rhythm(1),
            }}
          >
            {post.published_at}
          </p>
        </header>
        <section dangerouslySetInnerHTML={{ __html: post.childMarkdownRemark.html }} />
        <hr
          style={{
            marginBottom: rhythm(1),
          }}
        />
        <footer>
          <Bio />
        </footer>
      </article>

      <nav>
        <ul
          style={{
            display: `flex`,
            flexWrap: `wrap`,
            justifyContent: `space-between`,
            listStyle: `none`,
            padding: 0,
          }}
        >
          <li>
            {previous && (
              <Link to={`posts/${previous.slug}`} rel="prev">
                ← {previous.title}
              </Link>
            )}
          </li>
          <li>
            {next && (
              <Link to={`posts/${next.slug}`} rel="next">
                {next.title} →
              </Link>
            )}
          </li>
        </ul>
      </nav>
    </Layout>
  )
}

export default BlogPostTemplate

Now open the site, click through to the link and you should see the post!

You could leave it there, but right now if you're using the starter, Kyle is getting all the credit for your posts. You can change that by using the data from your DEV profile.

Open the bio component in src/components/bio.js and edit it to get the data from one of the DEV posts:

const Bio = () => {
  const {devArticle} = useStaticQuery(graphql`
    query {
      devArticle {
        user {
          name
          profile_image_90
          twitter_username
        }
      }
    }
  `)

  const user = devArticle.user;

  return (
    <div
      style={{
        display: `flex`,
        marginBottom: rhythm(2.5),
      }}
    >
      <img
        width={45}
        height={45}
        alt={user.name}
        src={user.profile_image_90}
      />
      <p>
        Written by <strong>{user.name}</strong>
        {` `}
        <a href={`https://twitter.com/${user.twitter_username}`}>
          Follow them on Twitter
        </a>
      </p>
    </div>
  )
}

export default Bio

Now you should be able to run it and see your own profile.

When you deploy this you should be aware that it won't rebuild automatically when you add new posts to DEV. You'll need to manually trigger a rebuild when you post, or set your site to rebuild automatically at regular intervals.

You can check out the source code for this demo and view the result, built for free on Gatsby Cloud and hosted on Netlify.

How to use Drupal Paragraphs in your Gatsby site

Matt Kane — Wed, 28 Aug 2019 20:32:36 +0000

If you're using Drupal as a data source for Gatsby, you really should be using the Paragraphs module. It's one of the most powerful ways of building a site. It allows you to create "Paragraph Types" that an editor can add and reorder to assemble a page. Paragraphs can be simple types such as a block of HTML or a list of images, or more complex types with many different fields. Most of the documentation is related to using it within Drupal themes, but it also maps really well to React components. You can create custom components, then define matching paragraph types to let editors add them to a page. This can give editors a lot more flexibility than they would otherwise get with a headless CMS.

It can be a bit tricky to work out how to set this up, but hopefully this post will help you build beautiful component-based sites with Drupal Paragraphs and Gatsby.

I am not going to go into detail about how to set up Drupal. I will assume that you have installed Paragraphs and enabled the paragraphs and paragraphs_demo modules. I have created a content type called "Paragraphed article" with a single field called field_paragraphs_demo which I'll be using here.

Once you have Drupal set up and populated with some content, add and enable gatsby-source-drupal in Gatsby. Start the Gatsby server, and take a look at GraphiQL. You should see your paragraph page type in the explorer. Try a simple query to see what you get:

{
    allNodeParagraphedContentDemo {
        nodes {
            relationships {
                field_paragraphs_demo {
                    __typename
                }
            }
        }
    }
}

You should see an array of pages, with paragraphs of various types. In my example I get the following:

{
    "data": {
        "allNodeParagraphedContentDemo": {
            "nodes": [
                {
                    "relationships": {
                        "field_paragraphs_demo": [
                            {
                                "__typename": "paragraph__image_text"
                            },
                            {
                                "__typename": "paragraph__images"
                            },
                            {
                                "__typename": "paragraph__text"
                            }
                        ]
                    }
                },
                {
                    "relationships": {
                        "field_paragraphs_demo": [
                            {
                                "__typename": "paragraph__image_text"
                            },
                            {
                                "__typename": "paragraph__text"
                            },
                            {
                                "__typename": "paragraph__text"
                            },
                            {
                                "__typename": "paragraph__image_text"
                            }
                        ]
                    }
                }
            ]
        }
    }
}

That's two pages, the first of which has three paragraphs and the second of which has four. You should note that there's more than one of each paragraph type.

In gatsby-node.js you first need to create the pages:

const path = require("path");

exports.createPages = async ({ graphql, actions }) => {
    const { createPage } = actions;
    const tpl = path.resolve(`src/templates/paragraph.js`);

    //   Adjust these field names as needed
    const result = await graphql(`
        {
            paragraphPages: allNodeParagraphedContentDemo {
                edges {
                    node {
                        fields {
                            slug
                        }
                        drupal_internal__nid
                    }
                }
            }
        }
    `);
    result.data.paragraphPages.edges.forEach(({ node }) => {
        createPage({
            path: node.fields.slug,
            component: tpl,
            context: {
                slug: node.fields.slug
            }
        });
    });
};

exports.onCreateNode = ({ node, getNode, actions }) => {
    const { createNodeField } = actions;
    // Use the type of your own paragraph page
    if (node.internal.type === `node__paragraphed_content_demo`) {
        const slug = `/pages/${node.drupal_internal__nid}/`;
        createNodeField({
            node,
            name: `slug`,
            value: slug
        });
    }
};

So far this is similar to the using-drupal demo. The interesting bit starts in the page template. I called mine paragraph.js. You can use the page query inside that.

// paragraph.js
import React from "react";
import { graphql } from "gatsby";
import Layout from "../components/layout";

const PageTemplate = ({ data }) => {
    const paragraphs = data.page.relationships.paragraphs.map(paragraph => (
        <li>{paragraph.type}</li>
    ));

    return (
        <Layout>
            <h1>{data.page.title}</h1>
            <ul>{paragraphs}</ul>
        </Layout>
    );
};

export default PageTemplate;

export const pageQuery = graphql`
    query($slug: String!) {
        page: nodeParagraphedContentDemo(fields: { slug: { eq: $slug } }) {
            id
            title
            relationships {
                paragraphs: field_paragraphs_demo {
                    type: __typename
                }
            }
        }
    }
`;

Before you load a page, you can create the index page to give an easy list of links:

// index.js
import React from "react";
import { Link, useStaticQuery, graphql } from "gatsby";

import Layout from "../components/layout";
import SEO from "../components/seo";

const IndexPage = () => {
    const data = useStaticQuery(graphql`
        query PageQuery {
            paragraphPages: allNodeParagraphedContentDemo {
                nodes {
                    title
                    fields {
                        slug
                    }
                }
            }
        }
    `);
    return (
        <Layout>
            <ul>
                {data.paragraphPages.nodes.map(node => (
                    <li>
                        <Link to={node.fields.slug}>{node.title}</Link>
                    </li>
                ))}
            </ul>
        </Layout>
    );
};

export default IndexPage;

If you load the front page it should give you a nice list of page links. Try clicking on one to view the page.

Right now the page just shows a list of paragraph types. The interesting part is to map that to components.

My example has the following paragraph types:

paragraph__image_text
paragraph__images
paragraph__text

These match the paragraph types created in Drupal. You need to to create a component for each paragraph type. Start with the Text paragraph. Drupal includes sanitised HTML for this, so we can set that directly (but "dangerously"). You need to update the query in the page template to get the data for the paragraph. It's a lot easier to keep track of this if you keep the query alongside the component by using a GraphQL fragment.

// TextParagraph.js
import React from "react";
import { graphql } from "gatsby";

export const TextParagraph = ({ node }) => (
    <div
        style={{
            borderStyle: "solid",
            marginBottom: 2
        }}
    >
        <div dangerouslySetInnerHTML={{ __html: node.text.processed }} />
    </div>
);

export const fragment = graphql`
    fragment ParagraphText on paragraph__text {
        id
        text: field_text_demo {
            format
            processed
            value
        }
    }
`;

You can now use this fragment in the page query in your template:

export const pageQuery = graphql`
    query($slug: String!) {
        page: nodeParagraphedContentDemo(fields: { slug: { eq: $slug } }) {
            id
            relationships {
                paragraphs: field_paragraphs_demo {
                    type: __typename
                    ...ParagraphText
                }
            }
            title
        }
    }
`;

If you log the data in your page you should see that the Text paragraph now has an object with the text fields.

As an aside, this pattern of keeping GraphQL fragments alongside the component that uses the data can be used elsewhere too, not just in these paragraphs. For example, if you have a component that needs images of a specific size, you can include a fragment alongside it that has the ImageSharp query to return the correct image size which you can then use in the page query.

You now have the right data for the Text paragraph, as well as a component to display it. Now you can create a similar component for the Text + Image paragraph type. This time we'll need to use an ImageSharp query to get the image too.

// ImageAndTextParagraph.js
import React from "react";
import Img from "gatsby-image";
import { graphql } from "gatsby";

export const ImageAndTextParagraph = ({ node }) => (
    <figure>
        <Img fixed={node.relationships.image.localFile.childImageSharp.fixed} />
        <figcaption dangerouslySetInnerHTML={{ __html: node.text.processed }} />
    </figure>
);

export const fragment = graphql`
    fragment ParagraphImageText on paragraph__image_text {
        id
        image: field_image_demo {
            alt
        }
        text: field_text_demo {
            format
            processed
            value
        }
        relationships {
            image: field_image_demo {
                id
                localFile {
                    childImageSharp {
                        fixed(width: 400) {
                            ...GatsbyImageSharpFixed_withWebp
                        }
                    }
                }
            }
        }
    }
`;

I've left off all the null-checking in these components for brevity's sake, but you should check that each of the fields exists (I recommend optional chaining if your environment supports it). This is particularly important for the images.

Add the ParagraphImageText fragment to the page query:

// paragraph.js
export const pageQuery = graphql`
    query($slug: String!) {
        page: nodeParagraphedContentDemo(fields: { slug: { eq: $slug } }) {
            id
            relationships {
                paragraphs: field_paragraphs_demo {
                    type: __typename
                    ...ParagraphText
                    ...ParagraphImageText
                }
            }
            title
        }
    }
`;

The Images paragraph is similar, but has an array of several images. You could create a similar paragraph for something like a carousel.

// ImagesParagraph.js
import React from "react";
import Img from "gatsby-image";
import { graphql } from "gatsby";

export const ImagesParagraph = ({ node }) => {
    const { images } = node.relationships;
    return (
        // Carousel like it's 1999! (Seriously though, never do this)
        <marquee>
            {images.map(image => (
                <Img fixed={image.localFile.childImageSharp.fixed} />
            ))}
        </marquee>
    );
};

export const fragment = graphql`
    fragment ParagraphImages on paragraph__images {
        id
        relationships {
            images: field_images_demo {
                id
                localFile {
                    childImageSharp {
                        fixed(width: 200, height: 200) {
                            ...GatsbyImageSharpFixed
                        }
                    }
                }
            }
        }
    }
`;

Now you have components for all of your paragraph types, you just need to work out how to display the correct one. I use a little helper function for this:

// paragraphHelpers.js
import React from "react";
import { ImageAndTextParagraph } from "./components/ImageAndTextParagraph";
import { TextParagraph } from "./components/TextParagraph";
import { ImagesParagraph } from "./components/ImagesParagraph";

const components = {
    paragraph__image_text: ImageAndTextParagraph,
    paragraph__text: TextParagraph,
    paragraph__images: ImagesParagraph
};

export const getParagraph = node => {
    if (components.hasOwnProperty(node.type)) {
        const ParagraphComponent = components[node.type];
        return <ParagraphComponent key={node.id} node={node} />;
    }
    return <p key={node.id}>Unknown type {node.__typename}</p>;
};

The components object maps the paragraph name from Drupal to the component. In this example I'm returning a warning if there's unrecognised type, but in production you'd probably just want to log something.

Now you can use this function to convert your data into components:

import React from "react";
import { graphql } from "gatsby";
import Layout from "../components/layout";

import { getParagraph } from "../paragraphHelpers";

export const PageTemplate = ({ data }) => {
    const paragraphs = data.page.relationships.paragraphs.map(getParagraph);

    return (
        <Layout>
            <h1>{data.page.title}</h1>
            {paragraphs}
        </Layout>
    );
};

export default PageTemplate;

export const pageQuery = graphql`
    query($slug: String!) {
        page: nodeParagraphedContentDemo(fields: { slug: { eq: $slug } }) {
            id
            relationships {
                paragraphs: field_paragraphs_demo {
                    type: __typename
                    ...ParagraphText
                    ...ParagraphImageText
                    ...ParagraphImages
                }
            }
            title
        }
    }
`;

This will give you all of your paragraphs with the correct component, in the correct order.

These examples aren't anything special, because we're using the basic default paragraph types, which you could easily add to a normal page. This is most powerful when you create custom paragraph types to match specific custom components that you would like your editors to be able to add to a page. How about a map component, with a custom paragraph to set the bounds? Or a newsletter signup, with a custom paragraph to select the mailing list?

What we're not doing here is handling nested paragraphs, or using the paragraphs library module. These get a bit more complicated, but can still be modelled using the same system: you just need to go a level deeper to get at the actual data.

A more realistic HTML canvas paint tool

Matt Kane — Tue, 13 Aug 2019 00:00:00 +0000

Creating a basic canvas drawing tool is a simple job in JavaScript, but the result is more MS Paint than Monet. However with a few changes you can make a tool that gives much more realistic result. Read on to learn how to build a canvas paint brush, bristle by bristle.

Let's start with the most basic implementation. First you need to set up a simple canvas element in the page.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, user-scalable=no" />
        <meta http-equiv="X-UA-Compatible" content="ie=edge" />
        <title>Drawing tools</title>
        <style>
            body {
                margin: 0;
            }
            canvas {
                border: 2px solid black;
            }
        </style>
        <script src="src/index.js" defer></script>
    </head>
    <body>
        <canvas id="canvas" height="600" width="800"></canvas>
    </body>
</html>

The basic procedure is to watch for mousedown or touchstart events, at which point you start drawing. Then on touchmove or mousemove you draw a line from the previous brush location to the current location. You add several listeners to handle ending the drawing.

Here's the basic drawing handler for mouse events:

// Brush colour and size
const colour = "#3d34a5";
const strokeWidth = 25;

// Drawing state
let latestPoint;
let drawing = false;

// Set up our drawing context
const canvas = document.getElementById("canvas");
const context = canvas.getContext("2d");

// Drawing functions

const continueStroke = newPoint => {
    context.beginPath();
    context.moveTo(latestPoint[0], latestPoint[1]);
    context.strokeStyle = colour;
    context.lineWidth = strokeWidth;
    context.lineCap = "round";
    context.lineJoin = "round";
    context.lineTo(newPoint[0], newPoint[1]);
    context.stroke();

    latestPoint = newPoint;
};

// Event helpers

const startStroke = point => {
    drawing = true;
    latestPoint = point;
};

const BUTTON = 0b01;
const mouseButtonIsDown = buttons => (BUTTON & buttons) === BUTTON;

// Event handlers

const mouseMove = evt => {
    if (!drawing) {
        return;
    }
    continueStroke([evt.offsetX, evt.offsetY]);
};

const mouseDown = evt => {
    if (drawing) {
        return;
    }
    evt.preventDefault();
    canvas.addEventListener("mousemove", mouseMove, false);
    startStroke([evt.offsetX, evt.offsetY]);
};

const mouseEnter = evt => {
    if (!mouseButtonIsDown(evt.buttons) || drawing) {
        return;
    }
    mouseDown(evt);
};

const endStroke = evt => {
    if (!drawing) {
        return;
    }
    drawing = false;
    evt.currentTarget.removeEventListener("mousemove", mouseMove, false);
};

// Register event handlers

canvas.addEventListener("mousedown", mouseDown, false);
canvas.addEventListener("mouseup", endStroke, false);
canvas.addEventListener("mouseout", endStroke, false);
canvas.addEventListener("mouseenter", mouseEnter, false);

We need to add some extra handlers to deal with touch events.

const getTouchPoint = evt => {
    if (!evt.currentTarget) {
        return [0, 0];
    }
    const rect = evt.currentTarget.getBoundingClientRect();
    const touch = evt.targetTouches[0];
    return [touch.clientX - rect.left, touch.clientY - rect.top];
};

const touchStart = evt => {
    if (drawing) {
        return;
    }
    evt.preventDefault();
    startStroke(getTouchPoint(evt));
};

const touchMove = evt => {
    if (!drawing) {
        return;
    }
    continueStroke(getTouchPoint(evt));
};

const touchEnd = evt => {
    drawing = false;
};

canvas.addEventListener("touchstart", touchStart, false);
canvas.addEventListener("touchend", touchEnd, false);
canvas.addEventListener("touchcancel", touchEnd, false);
canvas.addEventListener("touchmove", touchMove, false);

This is the working example.

You can change strokeWidth and colour, but it doesn't look much like a paintbrush. Let's start to fix that.

The first problem with this is that it uses a single line. A real paintbrush is made up of many bristles. Let's see if we can improve our brush by adding bristles.

First we'll change our stroke function to one that draws a single bristle, then when we draw a brush stroke, we'll draw several bristles at once.

const strokeBristle = (origin, destination, width) => {
    context.beginPath();
    context.moveTo(origin[0], origin[1]);
    context.strokeStyle = colour;
    context.lineWidth = width;
    context.lineCap = "round";
    context.lineJoin = "round";
    context.lineTo(destination[0], destination[1]);
    context.stroke();
};

const continueStroke = newPoint => {
    const bristleCount = Math.round(strokeWidth / 3);
    const gap = strokeWidth / bristleCount;
    for (let i = 0; i < bristleCount; i++) {
        strokeBristle(
            [latestPoint[0] + i * gap, latestPoint[1]],
            [newPoint[0] + i * gap, newPoint[1]],
            2
        );
    }
    latestPoint = newPoint;
};

Here's the result:

Now, this is an improvement, but it looks more like a comb than a paintbrush. Each bristle is exactly the same width and position, which isn't much like a real brush. We can improve that with some randomness. Instead of drawing the bristles at exact intervals from each other we can randomly vary the width and position of each one. We'll do this at the start of the stroke, so that it remains the same for the length of the stroke, but varies the next time.

First we'll create a helper function to generate the brush, which we'll store as an array of "bristle" objects.

const makeBrush = size => {
    const brush = [];
    strokeWidth = size;
    let bristleCount = Math.round(size / 3);
    const gap = strokeWidth / bristleCount;
    for (let i = 0; i < bristleCount; i++) {
        const distance =
            i === 0 ? 0 : gap * i + Math.random() * gap / 2 - gap / 2;
        brush.push({
            distance,
            thickness: Math.random() * 2 + 2
        });
    }
    return brush;
};

let currentBrush = makeBrush();

This uses objects that specify the width and position of each bristle, which we can then use to draw the strokes.

const strokeBristle = (origin, destination, width) => {
    context.beginPath();
    context.moveTo(origin[0], origin[1]);
    context.strokeStyle = colour;
    context.lineWidth = width;
    context.lineCap = "round";
    context.lineJoin = "round";
    context.lineTo(destination[0], destination[1]);
    context.stroke();
};

const drawStroke = (bristles, origin, destination) => {
    bristles.forEach(bristle => {
        context.beginPath();
        const bristleOrigin = origin[0] - strokeWidth / 2 + bristle.distance;

        const bristleDestination =
            destination[0] - strokeWidth / 2 + bristle.distance;
        strokeBristle(
            [bristleOrigin, origin[1]],
            [bristleDestination, destination[1]],
            bristle.thickness
        );
    });
};

const continueStroke = newPoint => {
    drawStroke(currentBrush, latestPoint, newPoint);
    latestPoint = newPoint;
};

const startStroke = point => {
    currentBrush = makeBrush(strokeWidth);
    drawing = true;
    latestPoint = point;
};

Here's the result:

This is staring to look a lot better. The bristles are already looking more natural. However it still looks more uniform than a real brush. The problem is the colours are too flat. A real stroke will have colours that vary slightly according to the thickness of the paint, and the angle of the light. We can emulate this by varying the colour slightly in the same way we varied the thickness and position. For this we're going to use a library called TinyColor. The package name is tinycolor2, so npm install it and include it in your file, or if you're not transpiling you can include it from a CDN.

First create a helper to randomly vary the brightness of a colour.

import tinycolor from "tinycolor2";

const varyBrightness = 5;

const varyColour = sourceColour => {
    const amount = Math.round(Math.random() * 2 * varyBrightness);
    const c = tinycolor(sourceColour);
    const varied =
        amount > varyBrightness
            ? c.brighten(amount - varyBrightness)
            : c.darken(amount);
    return varied.toHexString();
};

Now we can extend the makeBrush method to add a colour property.

const makeBrush = size => {
    const brush = [];
    let bristleCount = Math.round(size / 3);
    const gap = strokeWidth / bristleCount;
    for (let i = 0; i < bristleCount; i++) {
        const distance =
            i === 0 ? 0 : gap * i + Math.random() * gap / 2 - gap / 2;
        brush.push({
            distance,
            thickness: Math.random() * 2 + 2,
            colour: varyColour(colour)
        });
    }
    return brush;
};

...and then modify the drawing functions to use the bristle colour:

const strokeBristle = (origin, destination, bristle) => {
    context.beginPath();
    context.moveTo(origin[0], origin[1]);
    context.strokeStyle = bristle.colour;
    context.lineWidth = bristle.thickness;
    context.lineCap = "round";
    context.lineJoin = "round";
    context.lineTo(destination[0], destination[1]);
    context.stroke();
};

const drawStroke = (bristles, origin, destination) => {
    bristles.forEach(bristle => {
        context.beginPath();
        const bristleOrigin = origin[0] - strokeWidth / 2 + bristle.distance;

        const bristleDestination =
            destination[0] - strokeWidth / 2 + bristle.distance;
        strokeBristle(
            [bristleOrigin, origin[1]],
            [bristleDestination, destination[1]],
            bristle
        );
    });
};

Here's the result:

I'm happy with the look of those strokes now, but the problem now is the action. The brush here has a fixed angle which is more like a marker pen. A real brush changes angle as you move. To do this we can make the angle match the direction in which we're moving. This requires some maths.

In our move handler, we know the previous point and the new point. From this we can work out the bearing, which gives us the new angle for the brush. We then draw a line for each bristle from its old posotion and angle to its new position and angle.

First we'll add some helpers that do the trigonometry to work out these angles.

const rotatePoint = (distance, angle, origin) => [
    origin[0] + distance * Math.cos(angle),
    origin[1] + distance * Math.sin(angle)
];

const getBearing = (origin, destination) =>
    (Math.atan2(destination[1] - origin[1], destination[0] - origin[0]) -
        Math.PI / 2) %
    (Math.PI * 2);

const getNewAngle = (origin, destination, oldAngle) => {
    const bearing = getBearing(origin, destination);
    return oldAngle - angleDiff(oldAngle, bearing);
};

const angleDiff = (angleA, angleB) => {
    const twoPi = Math.PI * 2;
    const diff =
        (angleA - (angleB > 0 ? angleB : angleB + twoPi) + Math.PI) % twoPi -
        Math.PI;
    return diff < -Math.PI ? diff + twoPi : diff;
};

We can then update our drawing functions to use the angles.

let currentAngle = 0;

const drawStroke = (bristles, origin, destination, oldAngle, newAngle) => {
    bristles.forEach(bristle => {
        context.beginPath();
        const bristleOrigin = rotatePoint(
            bristle.distance - strokeWidth / 2,
            oldAngle,
            origin
        );

        const bristleDestination = rotatePoint(
            bristle.distance - strokeWidth / 2,
            newAngle,
            destination
        );
        strokeBristle(bristleOrigin, bristleDestination, bristle);
    });
};

const continueStroke = newPoint => {
    const newAngle = getNewAngle(latestPoint, newPoint, currentAngle);
    drawStroke(currentBrush, latestPoint, newPoint, currentAngle, newAngle);
    currentAngle = newAngle % (Math.PI * 2);
    latestPoint = newPoint;
};

This gives the following:

This is a more natural action than before, but the turns are a little strange. This is because it's making sharp changes in angle. We can improve this using bézier curves.

First, update drawStroke to calculate a control point for the curve. We're using the position of the origin point, rotated to the new angle.

const drawStroke = (bristles, origin, destination, oldAngle, newAngle) => {
    bristles.forEach(bristle => {
        context.beginPath();
        const start = bristle.distance - strokeWidth / 2;

        const bristleOrigin = rotatePoint(start, oldAngle, origin);
        const bristleDestination = rotatePoint(start, newAngle, destination);

        const controlPoint = rotatePoint(start, newAngle, origin);

        strokeBristle(bristleOrigin, bristleDestination, bristle, controlPoint);
    });
};

We then update strokeBristle to use a curve instead of the straight line:

const strokeBristle = (origin, destination, bristle, controlPoint) => {
    context.beginPath();
    context.moveTo(origin[0], origin[1]);
    context.strokeStyle = bristle.colour;
    context.lineWidth = bristle.thickness;
    context.lineCap = "round";
    context.lineJoin = "round";
    context.shadowColor = bristle.colour;
    context.shadowBlur = bristle.thickness / 2;
    context.quadraticCurveTo(
        controlPoint[0],
        controlPoint[1],
        destination[0],
        destination[1]
    );
    context.stroke();
};

This works great, except when we first start a stroke it tries to curve from whatever the previous angle of the brush was, which gives some unnatural results. Our final changes will be to not use the curve when starting a stroke.

let currentAngle;

const getNewAngle = (origin, destination, oldAngle) => {
    const bearing = getBearing(origin, destination);
    if (typeof oldAngle === "undefined") {
        return bearing;
    }
    return oldAngle - angleDiff(oldAngle, bearing);
};

// ...

const startStroke = point => {
    currentAngle = undefined;
    currentBrush = makeBrush(strokeWidth);
    drawing = true;
    latestPoint = point;
};

Here's the final version:

Now, as much as I like purple, you might want to use some other colours. This is a simple addition, with the rarely-used <input type="color">:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, user-scalable=no" />
    <meta http-equiv="X-UA-Compatible" content="ie=edge" />
    <title>Drawing tools</title>
    <style>
      body {
        margin: 0;
      }
      canvas {
        border: 2px solid black;
      }

      #colourInput {
        position: absolute;
        top: 10px;
        left: 10px;
      }
    </style>
    <script src="src/index.js" defer></script>
  </head>
  <body>
      <canvas id="canvas" height="450" width="800"></canvas>
      <input type="color" id="colourInput" value="#3d34a5" />
  </body>
</html>

You then read this value when you start each stroke:

const startStroke = point => {
    colour = document.getElementById("colourInput").value;
    currentAngle = undefined;
    currentBrush = makeBrush(strokeWidth);
    drawing = true;
    latestPoint = point;
};

You could do similar with brush size. You could also try something like brush presets, which change the bristle size and count.

This is the final version with colour picker included:

Try the full-screen version. If you have some suggestions, open a PR on the GitHub repo

DEV Community: Matt Kane

How to do ISR and advanced caching with Remix and Netlify

TL;DR

What is ISR?

How to implement ISR using Remix

Caching on Remix

Using stale-while-revalidate for faster loading

On-demand revalidation with cache tags

How to revalidate your pages on demand

Fine-grained cache control

Cookies and caching

i18n and caching

Debugging your cache

Get started

Building an astrological GPT

This GPT talks to your GraphQL server and will write your queries for you

How to use it

How I made it

Tips

🏎️💨 Turbocharge your builds with a Turborepo remote cache in a single edge function

The secrets of Full Turbo

A cache server in 100 lines of code

The Turborepo remote cache API

Easy mode

Setting up your repo

Bring back the Twitter link preview

The problem

Dynamic Open Graph images

The code

Level-up your web images with Unpic picture tags

Art direction

Dark mode

Formats

Try it out

A minimal, multi-framework, responsive image component

Let the CDN do the work

Unpic img: a simpler image component for every framework

Benchmarking web dev tools on the Mac mini M1

Test setup

Results

Mac mini M1

2019 Macbook Pro 2.3GHz core i9

Mac mini M1, via Rosetta2

Real-world tests

Conclusion

Top 6 reasons Gatsby incremental builds can be slow (and how to make them fast)

1. Your site doesn't use a CMS

2. It's the first build, or the code changed

3. Static queries that reference pages

4. Templates that change on every build

5. Paginated archives

6. You're not using Gatsby Cloud

How to build a Gatsby plugin to display your DEV posts

How to use Drupal Paragraphs in your Gatsby site

A more realistic HTML canvas paint tool

Using `stale-while-revalidate` for faster loading