Note: I'm an engineer on the open source team at Gatsby, but this isn't an official post
You've heard Gatsby Incremental Builds are Blazing Fast. They can build a site in under 5 seconds! So why does your site still take over a minute to build? I worked on the incremental builds feature at Gatsby, and this is a question I've heard several times since the launch last month. In this post I'll share the top reasons that site builds can be slow with incremental builds enabled, and how to fix this. And to be clear: by slow, I mean more than about 10 seconds.
1. Your site doesn't use a CMS
First up, let's define our terms.
An incremental build is where only the pages, JavaScript and assets affected by a data change are rebuilt.
– Me
You may have heard that currently Gatsby incremental builds only support data changes, not code changes. What you might not have realised is that code changes are any changes in Git. That includes Markdown files! The reason for this is that right now we can't tell if a .md file is just Markdown or MDX with code in it until we parse it. As we only support data changes, we have to do a regular, non-incremental build. When you push a new Markdown file to GitHub, Gatsby will rebuild the whole site. This will change, because we plan to support code changes (I'm working on it), but right now Markdown sites won't get the speedy builds that you get from a supported CMS like Contentful, Dato or the beta WordPress plugin.
2. It's the first build, or the code changed
Even with incremental builds enabled, the first build is a full build so takes longer. Gatsby also currently does a full build if the code has changed, so builds triggered by GitHub will be slower.
3. Static queries that reference pages
OK you're using a supported CMS. You're editing just one page. So why is Gatsby rebuilding them all? We have to rebuild a page whenever the data in it changes. However we also have to rebuild all pages if any static query references a changed page. For example, if you have a static query that gets the total number of blog posts, then whenever you edit any post Gatsby will need to re-run that query. This means every page has to be rebuilt. We will be improving this by building a dependency graph that means we can be smarter at finding unchanged pages, but even then it will be easy to accidentally invalidate lots of pages with careless of of static queries. Try to minimise the use of static queries, and try to not use any that reference node types that change every time (such as blogPost or equivalent). Move the query into a page query if possible, as these don't cause every page to be rebuilt.
4. Templates that change on every build
Related to the previous point, if you have templates or other shared components that change a lot then you'll need to keep rebuilding every page that uses them. Do you really need a "latest post" link on every page that includes the post title? Even if you've managed to avoid having to re-run every page query, if you have something like a timestamp or other changable data in a component, then it means that the HTML will be different on every build, meaning even otherwise unchanged pages will need to be re-uploaded. Put things like that into a useEffect hook and they won't be run at build time.
5. Paginated archives
Do you have loads of paginated archive pages? If you do, then whenever you add a post, every page will need to be rebuilt as the posts shuffle back by one. If you have lots of posts then that could end up being a large number of pages. Instead of paginating them all, have a separate archive per day or month. This way only the most recent archive will need to change when you add a post.
6. You're not using Gatsby Cloud
Sorry to get all corporate (and to reiterate the disclosure above: I worked on incremental builds at Gatsby), but right now if they're not on Gatsby Cloud, they're not proper incremental builds. Some people have been confused by an experimental feature in Gatsby that let's you skip part of page generation when the data hasn't changed. This is kinda-incremental, but is absolutely not the same thing as the feature that Gatsby just launched. They don't use the same code, and you can't expect the same results. Importantly, the flag just skips one part of the build (which is why we don't refer to it as incremental builds). It's an important part, but is far from all of it. There are still lots of things that need to run every time, even if no data has changed. For that reason, you should still expect builds to take a minute or more. This is a great improvement if you have a large site that otherwise takes many minutes to build (and something like it will probably become the default), but it's not the same as the 5 second builds that you can get on Gatsby Cloud. Sorry! This isn't us holding back performance improvements from open source Gatsby: those kind of speeds are only possible when you have a full build and deployment pipeline that has been made just for Gatsby builds. We've been making loads of improvements to the open source project that are speeding up builds for everyone, sometimes by massive amounts (like 1000x), but for sites that have built and deployed before you've switched tabs you need to use Gatsby Cloud.
If you have any more questions about incremental builds, feel free to ask in the comments.
Latest comments (7)
Hi! Great article! I was just wondering, do you know if Gatsby Incremental Builds also support Markdown and Markdown based CMSs like Forestry, NetlifyCMS etc. in the meantime since 2020?
I have been trying to find any information regarding it, but had not much luck. Thanks!
How to enable Incremental builds if I use a custom source i.e. I have my custom GraphQL API?
I guess I need to make requests with a changed data to Gatsby Cloud (as a webhook). Is there any documentation for this?
Thanks for the info Matt. Could you please elaborate the point no. 4? What sort of time stamp are you highlighting here? Are u asking to do any time calculations on the client side?
I'm saying that anything which displays the current time or date should be done only on the client (so in a useEffect hook, for example), because otherwise it will mean that the page is different on every build so can never be cached.
Gatsby Cloud is good if we can use it with custom domain.
Gatsby Cloud doesn't doesn't include hosting, so you can deploy it to a hosting service that offers custom domains, such as Netlify or AWS
Excellent article - I suspect the first issue will stump most folks with code or markdown changes triggering full builds.
Also, just wanted to note that Agility CMS supports incremental builds as a first class citizen in Gatsby. Works awesome!