DEV Community: Atte Juvonen

How to update Gatsby dependencies through major version upgrades

Atte Juvonen — Wed, 20 Jul 2022 18:30:46 +0000

I recently went through the pain of updating my Gatsby starters from Gatsby v2 to v4, which didn't turn out to be as easy as the internet would have you believe. With the help of some kind souls from Gatsby Discord, I eventually got through it, and I figured I would document the process in this article.

Setting expectations

According to the official docs, you should be able to update both Node version and Gatsby version without having to do code changes (as long as your code isn't using any of the documented breaking changes). Other advice on the internet is similar. Unfortunately, when you have a moderately complex project, major version upgrades are likely to break everything in a myriad of ways. Here's a non-exhaustive list of issues that I experienced:

Got stuck in dependency hell where npm install fails due to dependency version conflicts, but any attempt at resolving them by upgrading or downgrading packages merely shifts the conflict to a different set of dependencies.
Running gatsby develop was stuck in an infinite loop due to a bug in Gatsby PostCSS plugin that only occurs with specific Node versions. The community plugin still isn't fixed and the workarounds involve either removing the plugin, downgrading Node version, or refactoring CSS in a way that doesn't trigger the bug.
My CSS broke — not in a clear and obvious manner, but rather in a subtle way affecting only lines where the CSS is nested, the line includes the & character, and some value within the line is resolved from a variable. To clarify, my CSS broke only for lines where all 3 of those conditions hold.
Another subtle breakage was related to my image placeholders, which were misaligned due to an internal change in gatsby-image. This is one of those things that happen when you venture off the beaten path. Internals of Gatsby were never guaranteed to be stable, so you could say that I brought this upon myself by relying on an internal API. This bug was particularly subtle because the placeholders are only visible while the image is loading, so you typically don't even see them when you're developing locally. A real user with a slow internet connection will see them, though, and the misalignment looks quite jarring.

I encountered issues that only affect the production build but don't affect the development build, issues that only affect the development build but don't affect the production build, and issues that only affect hot reload within the development build. Catching everything required extensive manual testing, and I can't imagine a suite of automated tests that would catch things like image placeholder misalignment.

How to update

The following instructions are written for npm, but you should be able to follow along even if you are using yarn.

General tips:

In order to avoid dependency hell, you should update everything at the same time instead of updating things one by one.
Inscrutable errors are often resolved by deleting .cache, node_modules and package-lock.json before running npm install.
Don't stop at the point where the build completes successfully and your automated tests pass; you'll probably need to do manual testing to catch all the subtle issues.

Step by step instructions:

Switch to desired Node version (e.g. nvm use v16)
Ensure desired version of gatsby-cli is installed in this particular Node environment (e.g. npm install -g gatsby-cli@latest-v4)
Run ncu -u to bump package.json dependencies to newest versions (requires npm-check-updates).
Manually edit package.json to downgrade packages that need to be downgraded for compatibility. For example, at the time of writing this, I needed to downgrade React from v18 to v17 in order to avoid hydration errors (Gatsby doesn't fully support v18 at the time of writing).
Delete .cache
Delete node_modules
Delete package-lock.json
Hammer on npm install until it runs without errors. You can expect to see many errors related to version conflicts in your dependencies. A common source of issues is unmaintained plugins created by the community. In some cases the conflicts can be resolved by downgrading some of the dependencies you just upgraded. In other cases you might want to remove or replace incompatible plugins. As a last resort, you might want to try npm install --legacy-peer-deps to force an installation despite version conflicts. Unmaintained plugins often work with newer dependencies in practice.
Hammer on gatsby develop until it runs without errors. Resolving errors at this step often requires changes to dependencies, in which case goto step 5.
When you get the dev environment running, it's time to run your automated tests and fix everything that broke according to your tests.
After your tests pass successfully, it's time to manually test your site in dev environment and fix everything that's broken (malformed CSS etc. that won't be caught by your automated tests).
Additionally, you should manually test that hot reload is not broken: First make some code changes. Then check that you see the changes applied in browser without any action inside the browser. Then try refreshing the site in browser. Then try navigating across different pages.
Once your dev environment is fully working, it's time to manually test the production build (gatsby build && gatsby serve). For example, some routing issues only exist in the production build.
After everything seems to work locally, it's time to update your CI configuration to use the same node version and gatsby-cli as you use locally. This is also a good time to update the build image and other artifacts, if needed.
Update your README set-up instructions. (I do this even for my personal projects because I want to know how I can run the project 2 years from now after I have forgotten literally everything about setting it up. It's good to write things like Node version and gatsby-cli version in the README because those won't be contained in your package.json.)
If you still have remaining energy, you can continue working through the numerous npm warnings and Gatsby warnings that have appeared as a result of the upgrades.

Don't pick the best tool for the job

Atte Juvonen — Mon, 13 Jun 2022 09:29:03 +0000

Tech stack choices are one of the more significant decisions when creating a new digital product — entire teams can benefit or suffer from these decisions for years to come.

Should we build our app as a Ruby monolith or as NodeJS microservices? Should our data live in a traditional SQL database like Postgres or a NoSQL database like DynamoDB? Should we use a React framework like NextJS, or pass on React and go with Svelte or whatever the cool kids prefer this summer? (I'm too old to know.)

When we compare the pros and cons of different options, we often focus on the technical merits. You may have heard the old adage: “pick the best tool for the job”. It's great as an expression of technology agnosticism — a fun way of saying “we don't have an Oracle license to sell you” — but don't take it too literally. The saying implies that you should rank your options by some metric of technological goodness and pick whatever is “best”. As if second-best just won't cut it.

Sometimes indeed, second-best won't cut it. For example, data science projects are commonly built on the Python stack because data science libraries available for Python far exceed what's available for other languages (with the possible exception of R). It's perfectly reasonable to choose Python for a data science project on this basis alone. But this is a rare example. A more common scenario is that multiple tools are perfectly suitable for the job, and the decision comes down more or less to personal taste. We've all been in discussions where numerous developers advocate why their favorite tech just happens to be a perfect fit for a particular task.

When you find yourself comparing several viable options, take a step back from technical considerations and consider your options from the following 3 viewpoints: competency, continuity, and consistency.

1. Competency

Play to your strengths. When your team is working with familiar techs, they will be able to build a better product and build it faster.

Where is your team's proficiency? If you have a couple of experienced Java developers who have minimal experience with NodeJS, maybe that new backend service doesn't have to be implemented with NodeJS (no matter how fashionable it would be). Forgo NodeJS if the team is happy to work with boring old Java.

If the team wants to take the opportunity to learn some new techs, how do people feel about other JVM-based languages like Scala or Kotlin? The jump from Java to Kotlin is much smaller than that from Java to NodeJS, so you can better leverage your team's competency if you choose Kotlin over NodeJS.

2. Continuity

You won't be here forever, and your team won't be here forever. As people move on, there will be a need to find new talent to continue development or, at the least, keep the lights on. This is why it's not enough to consider your team's competency: you need to consider market availability for different skills to ensure continuity.

It's easier to find developers for popular techs and more challenging for esoteric techs. Nowhere is this more apparent than the current frontend landscape. There is a large pool of React developers, and that pool is guaranteed to grow over the next few years (because React is chosen for the majority of new frontend projects, and those projects will need continued development for years).

If you decide to build your new frontend with something esoteric, such as Elm, there is a massive risk that the project will be abandoned or rewritten prematurely. No matter how proficient you are with Elm, no matter how good the front end will be, who will continue development after you've moved on? I've seen even business-critical software left to rot, keeping the lights on with duct tape — just because the original developers made some esoteric choices. Of course, you could argue that the corporation should simply invest in their developers to ensure continuity of their business-critical software, but that often doesn't happen in practice. So let's make it easier for them.

There are differences in degree here. The step from Gatsby to NextJS is small. The jump from React to Angular is significant. The leap from React to Elm is vast. The more esoteric you go, the more you risk continuity.

3. Consistency

Even if we're building a new digital product from scratch, it's unlikely to be the first digital product in this company. How were the other codebases built? We should lean towards using the same techs in this new product (to the extent that makes sense). Consistency across codebases is a boon to development and recruitment.

Let's consider inconsistency in codebases from the perspective of recruitment. Suppose you have a job ad that requires experience in a long list of non-complementary techs. If you need experience with both React and Angular, and also with both AWS and Azure, and also with both Ruby and NodeJS, you're going to have a bad time. You're narrowing down the pool of candidates to a small fraction, as very few candidates fulfill your requirements. And if we look at the candidates who pass the bar, the position might not be appealing to them because it's not very focused on anything.

Let's consider inconsistency in codebases from the perspective of development. If every codebase is a unique snowflake, learning one will not help you learn the next one. Onboarding times will be long and the velocity of work will be slow even for your seasoned developers if they have to constantly chart unfamiliar territory.

In some ways, the consistency viewpoint is even more critical than the competency and continuity viewpoints. For example, suppose you have a multi-vendor team of 40 devs working on your legacy Ruby codebases. You're planning to create a new digital product, and the developers who are going to work on it have more experience with JavaScript than they do with Ruby. Should you make the new product with Ruby or with JavaScript? Although the developers have more competency with JavaScript, and although there is a larger pool of candidates available for JavaScript, you're still going to have those legacy Ruby codebases at the end of the project! You're going to have trouble recruiting Ruby developers anyway, so don't make it harder for yourself by requiring those Ruby developers to have JavaScript skills as well.

Recap

When you choose techs for a digital product, consider the following aspects in addition to technical considerations:

Competency: techs familiar to your team
Continuity: tech skills available in the market
Consistency: techs you're already using in your other codebases