Reducing Gatsby's page preloading bandwidth

Gersom van Ginkel — Sun, 05 Apr 2020 23:01:23 +0000

This post was originally published on gersom.nl.

Gatsby's component automatically prefetches page data whenever links are scrolled into view. But what if you have big pages and a long list of links?

This post is about solving a particular issue that I encountered while rebuilding my mother's website Kleuteridee.nl, on which Dutch kindergarten teachers can find various downloads. The website experienced a spike in homeschool visitors during the COVID-19 school closures, exceeding the limits of the managed Wordpress hosting.

In order to make the website faster, enable it to further scale, and simplify the hosting, I decided to rebuild the website using Storyblok as CMS and Gatsby for the front end. The end product needed to be easy to use for my mother and cheap enough in order to not burden her with high monthly bills.

Check out the free and awesome Excalidraw if you want to make similar sketches.

My main two goals were:

Don't exceed the 1TB bandwidth limit of Storyblok, since the next TB costs $170.
Stay within the 400GB bandwidth limit of Netlify's pro plan.

For Storyblok I hope to manage this by using Gatsby Image's responsive image component, combined with Storyblok's image resizing service. Gatsby loads the smallest image needed for display on the visitor's device and doesn't load the image before it is scrolled into view. If there would be future spikes in traffic, I can quickly turn on more aggressive image compression.

For Netlify it was a bit more tricky. Gatsby generates a page-data.json file for every page, that is prefetched whenever a link to that page scrolls into view. My mother's website has a gigantic side menu, showing all the 150+ pages, and those pages are long, meaning large page-data.json files.

If a visitor scrolls down, past the side menu, the browser would download 4.5MB worth of page data. Last month my mother had 100.000 visitors on her website, potentially leading to 450GB worth of bandwith just for preloading some pages. I was left with a choice: not using Gatsby's <Link> and therefore not having any preloading or trying to compress the page data. I chose the latter.

The solution

After playing around with several JSON compression libraries, I found that compressing a stringified version of the data using pako worked best. This got me down from 4.5MB to 1.7MB of page data, saving a potential 280GB of monthly bandwidth.

// gatsby-node.js

const pako = require('pako');

const page = {}; // Page data derived from Storyblok's API

const context = {
  compressed: pako.deflate(JSON.stringify(page), { to: 'string' }),
};

actions.createPage({
  path: `/${page.path}`,
  component: PageComponent,
  context,
});

// PageComponent.js

import pako from 'pako';

const PageComponent = props => {
  const content = React.useMemo(
    () =>
      JSON.parse(pako.inflate(props.pageContext.compressed, { to: 'string' })),
    [props.pageContext.compressed]
  );

  // Return the page JSX
};

export default PageComponent;

I'm wondering if there are other people who had a similar challenge. Please let me know if this was you and how you solved it! You'll find me on Twitter.

Generating Open Graph images during Netlify deploy

Gersom van Ginkel — Thu, 16 Jan 2020 16:11:03 +0000

> Original article <

Every post on my website now has an auto-generated og:image. I was curious if this could be done all from within the Netlify build phase. Here are my findings.

If you don't know what an og:image is, just have a look at the picture below. Open Graph images show up the moment you share a link to Facebook, LinkedIn, Twitter, WhatsApp, etc. As you can see, I chose to compile the image from the post title, website URL and a personal photo.

There is some prior art on auto-generating og:image, for example Open Graph Image as a Service by Zeit. They've built an API endpoint that generates the image on every request. I didn't want to have a separate API function because it feels like a waste of resources if the images don't change after deploy anyway. It turns out to work wonderfully in Netlify's build step!

The steps involved in generating the images are:

Initiating the script during Gatsby's onPostBuild
Booting up puppeteer (Chrome)
Rendering an HTML template
Updating the H1 element with the post title
Making a screenshot and saving it to /public
Repeating steps 4 and 5 for all posts.
Configuring the Open Graph / Twitter meta tags

Initiating the script during Gatsby's onPostBuild

The Gatsby onPostBuild API only runs when building for production and not during development. For this reason we will be using it, since we don't need og:image during development, and generating them slows down compilation.

Here is an example of how I take all the posts from the Gatsby data layer and send them to the function that generates the images. The query might look different for your use case, since I use .mdx files as a data source, with frontmatter for metadata.

// gatsby-node.js
import generateOgImages from './src/scripts/generate-og-images';

exports.onPostBuild = async ({ graphql }) => {
  const result = await graphql(`
    query {
      posts: allMdx {
        nodes {
          fields {
            slug
          }
          frontmatter {
            title
          }
        }
      }
    }
  `);
  await generateOgImages(result.data.posts.nodes);
};

Booting up puppeteer (Chrome)

We'll render HTML in puppeteer and screenshot it, in order to get our image. Puppeteer is a bit too heavy for the Netlify build step and won't work. We'll work around this by using a package called chrome-aws-lambda that is made to work within the limitations of a AWS Lambda function.

npm i chrome-aws-lambda puppeteer-core
npm i -D puppeteer

By accessing puppeteer through the chrome-aws-lambda instance, we'll automatically get the right chromium for our context (local development vs. Netlify).

// generate-og-images/index.js
const chromium = require('chrome-aws-lambda');

module.exports = async posts => {
  const browser = await chromium.puppeteer.launch({
    args: chromium.args,
    executablePath: await chromium.executablePath,
    headless: chromium.headless,
  });

  const page = await browser.newPage();

  // Here we'll add the screenshot logic

  await browser.close();
};

Ok, I'm kidding, it does not work outside of the box. The Netlify build step runs with NODE_ENV=development, which confuses chrome-aws-lambda. We could work around this by setting the NODE_ENV to production, but this has unintended side effects. Another way to force using the serverless chrome instance is by setting the AWS_LAMBDA_FUNCTION_NAME environment variable (docs).

Netlify does let us set this environment variable through their UI, so we'll just hack it into our netlify config.

# netlify.toml
[build]
  command = "AWS_LAMBDA_FUNCTION_NAME=trickpuppeteer npm run build"

Screenshotting HTML templates

In the same folder as the script, we'll create an HTML template file.

<!-- generate-og-images/template.html -->
<html>
  <head>
    <meta charset="utf-8" />
    <style>
      /*SEE THE REPOSITORY FOR ALL THE STYLES*/
    </style>
  </head>

  <body>
    <section>
      <img src="./avatar.jpeg" />
      <h1 id="title">
        Lorem ipsum dolor sit amet
      </h1>
      <p>Gersom.nl</p>
    </section>
  </body>
</html>

Then we'll load this HTML file into our script and inject it into puppeteer to make screenshots.

// generate-og-images/index.js
const chromium = require('chrome-aws-lambda');
const fs = require('fs');
const path = require('path');

module.exports = async posts => {
  const browser = await chromium.puppeteer.launch({
    args: chromium.args,
    executablePath: await chromium.executablePath,
    headless: chromium.headless,
  });

  const page = await browser.newPage();

  // Load html from template
  const html = fs
    .readFileSync(path.resolve(__dirname, './template.html'))
    .toString();

  // Render html
  await page.setContent(html, {
    waitUntil: ['domcontentloaded'],
  });

  // Wait until the document is fully rendered
  await page.evaluateHandle('document.fonts.ready');

  // Set the viewport to your preferred og:image size
  await page.setViewport({
    width: 1200,
    height: 632,
    // My macbook is retina, so this should be 2 while testing locally
    deviceScaleFactor: process.env.NETLIFY === 'true' ? 1 : 2,
  });

  // Create an og-images directory in the public folder
  const dir = path.resolve(__dirname, '../../../public/og-images');
  if (!fs.existsSync(dir)) fs.mkdirSync(dir);

  // Go over all the posts
  for (const post of posts) {
    // Update the H1 element with the post title
    await page.evaluate($post => {
      const element = document.querySelector('#title');
      element.innerHTML = $post.frontmatter.title;
    }, post);

    // Save a screenshot to public/og-images/slug-of-post.jpeg
    await page.screenshot({
      path: `${dir}/${post.fields.slug.replace(/\//g, '')}.jpeg`,
      type: 'jpeg',
      quality: 100,
      clip: { x: 0, y: 0, width: 1200, height: 632 },
    });
  }

  await browser.close();
};

This should run without error, but if you try this yourself, you'll find that the relative paths to the avatar (and font-face declarations in CSS) won't resolve in puppeteer. I ended up base64-injecting them:

// generate-og-images/index.js
let html = fs
  .readFileSync(path.resolve(__dirname, './template.html'))
  .toString();

let avatar = fs.readFileSync(path.resolve(__dirname, './avatar.jpeg'), {
  encoding: 'base64',
});
html = html.replace(
  './avatar.jpeg',
  `data:image/jpeg;charset=utf-8;base64,${avatar}`
);

let font = fs.readFileSync(path.resolve(__dirname, './Roboto-Bold.ttf'), {
  encoding: 'base64',
});
html = html.replace(
  "'./Roboto-Bold.ttf'",
  `data:application/x-font-ttf;charset=utf-8;base64,${font}`
);

Configuring the Open Graph / Twitter meta tags

Lastly, we organize our meta tags, so Twitter, WhatsApp and others know where to find our image.

import Helmet from 'react-helmet';

// DEPLOY_URL is a Netlify environment variable.
// You can set this to localhost in a .env.development file, for dev testing.
const image = `${process.env.DEPLOY_URL}/og-images/${slug}.jpeg`;

<Helmet>
  <meta property="og:image" content={image} />
  <meta name="twitter:image" content={image} />
</Helmet>;

That's it, thanks for reading! Let me know here or on Twitter if this worked for you.