DEV Community: SitePoint

Why We Moved a 20-Year-Old Site to Gatsby

Stuart Mitchell — Mon, 13 Jul 2020 06:11:45 +0000

We knew we had a problem.

In 2019, SitePoint was getting Lighthouse Speed scores under 10 on mobile, and between 20 and 30 on desktop.

Our efforts to control UX bloat were failing in the wake of a publishing business environment that sprang new leaks just as we’d finished temporarily plugging the last one. Our reliance on advertising, controlled by external parties, was a major obstacle to improved site performance. Our traffic growth had turned into decline.

On a site that provided people with a place to come and learn to code with best practices, this was not a good look. And it wasn't a site we could feel proud of, either.

To make matters worse, operational bottlenecks had arisen that made adaptation a tricky logistical business. Our team was struggling to make changes to the site: having focused on our Premium experience for several years, we were down to one developer with WordPress and PHP experience. To test out code changes, the team would have to wait in a queue to access our staging server.

It wasn’t energizing work for anyone, and it certainly wasn't efficient.

It was time to make some changes, and we set out to look for a solution. After a lot of research, we decided that Gatsby would be a great fit for our team. It would play to our talent strengths, help us solve all of the issues we had identified, and allow us to keep using WordPress for the backend so the editorial process wouldn’t need to change.

Why We Moved to Gatsby

Early in the research process, Gatsby started to look like a serious frontrunner. SitePoint isn't a small site, so we knew that the tech we chose had to be able to handle some pretty intense demands. Gatsby checked all of our boxes:

We could code everything in React, a tech that every member of the front-end team knows and uses daily.
Gatsby is super fast at its core — performance was at the heart of this project, and we could start from a good footing.
The entire site is rendered as static, which would be great for SEO.
We could build it as a new project, which meant no worrying about the existing codebase, which brought a huge amount of legacy code with it.
We could use Gatsby Cloud, allowing the team to get feedback on the build at any time just by pushing the branch to GitHub.
DDoS attacks on WordPress wouldn’t cause us issues, as the front-end is completely stand-alone.

More Maintainable CSS with `styled-components`

Since we were going to rebuild the site from scratch, we planned to make some design changes at the same time. To help with this work we decided to use styled-components.

styled-components keeps the site's styling easy to maintain, and we know where to look when we want to change the style of something — the style is always with the component.

How We Made the Build Happen

We started by following Gatsby's basic docs and pulling in our posts with the gatsby-source-wordpress plugin.

This was a big initial test for us: we had to see if it was even possible to use Gatsby for our site.

After 20 years of blogging, we have over 17,000 posts published. We knew the builds would take a long time, but we had to find out if Gatsby could deal with such a massive amount of content. As you've probably figured, the test delivered good news: Gatsby works.

A quick tip for other teams working with large sites: to make development a better experience, we used environment vars to prevent Gatsby from fetching all of the site's posts in development. There's nothing quite like a 60 minute hot reload to slow progress.

if (hasNextPage && process.env.NODE_ENV != "development") {
  return fetchPosts({ first: 100, after: endCursor });
}

From this point, we ran into some limitations with the WordPress source plugin. We couldn't get all the data we needed, so we moved to the WordPress GraphQL plugin.

We use Yoast to set our metadata for SEO, and had to ensure we were pulling in the correct information. We were able to do this with WordPress GraphQL. By doing it this way, the content team could still edit metadata the same way, and the data would still be dynamic and fetched on each build.

During the build, we would have three or four people in the team working on parts of the new blog. In the past, if they wanted to get feedback they'd have to push to our staging server and make sure nobody was already using it.

We found that Gatsby Cloud was a great solution to this issue. Now when someone pushes to a branch in GitHub, it creates a build in Gatsby Cloud along with a preview link. Our developers could share this link and get immediate testing and feedback much more effectively than before.

This faster feedback cycle made it easy to have multiple people on the team working on the build and put an end to a major bottleneck.

Launch Day Fun

On the big day, we launched the new site and ran through our initial tests. The new blog was flying — every page load felt instant.

We ran into some problems on SitePoint Premium, which started running into slows and even crashes. The culprit was a new element on blog pages that pulled in the popular books people were currently reading. It would do this via a client-side API call, and it was too much for Premium to handle due to the amount of traffic we get on the blog side.

We quickly added some page caching to the API to temporarily solve the issues. We realized we were doing this wrong — we should have been sourcing this data at build time, so that the popular books are already loaded when we serve the page to the user.

This is the main mindset shift you need to make when using Gatsby: any data that you can get at build time should be fetched at build time. You should only use client-side API calls when you need live data.

Once we'd re-written the API call to happen during the build, the first load of a blog page was even quicker — and Premium stopped crashing.

What We Still Need to Solve

While it's hard to overstate how much better our on-site experience is today, there are still a few pain points we need to solve.

If a new article is published, or if content is updated — as it is multiple times per day — we need to re-run the Gatsby build before these changes show up.

Our solution for that right now is a simple cron job that runs at pre-scheduled times over the course of a day. The long-term solution to this is to add a webhook to the WordPress publish and update button, so that a new build is triggered once pressed.

We also need to get incremental builds running. Right now, the entire site needs to be rebuilt each time, and given our content archive, this can take a while. Gatsby just introduced incremental builds as we went live, and we're working on implementing this on our site. Once that's set up our builds will be much faster if the only thing that has changed is content.

Our speed score is still not where we want it to be. While the site feels subjectively very fast, we are still not getting consistent scores in Lighthouse. We want to get both mobile and desktop into the green zone (scores of 90+) for optimal user experience and SEO.

Would We Do It Again?

A launch of this type would normally be a pretty nerve-wracking event, and take a lot of work from the team on launch day.

With Gatsby, our launch was really easy. We just had to move WordPress onto a new domain, and point sitepoint.com at the Gatsby version of the site.

Then we sat back and watched the numbers to see what happened to our traffic. Within a few days, the data was starting to come in and we were seeing a 15% increase in traffic. User engagement metrics were up across the board. And we hadn't even removed our ads yet (which, you may have noticed, we've since done).

It's not hard to figure out why the effects were so immediate. We had better SEO running on static HTML and CSS pages, and massive speed improvements made possible by the move to Gatsby.

Since we made the move, we've increased our Lighthouse speed scores from 6-15 on mobile to the 50-60 range, and from the 30s on desktop into the 70s. We wanted to ensure speed remained top of mind with this change, so we're using a great tool called Calibre that runs speed tests over a number of top pages each day and alerts us to the scores. We are using this tool to continue to improve our score, so I hope to have another article for you in three months when we get everything to stay in the 90+ range.

The team loves working in Gatsby. The blog codebase was something that nobody wanted to work on. Now, everyone wants to take those cards thanks to the great developer experience.

If you've been eyeing a move to Gatsby and wondering if it's ready for prime time, take our advice — it's worth the switch.

Build a Chrome Extension to Streamline Your Workflow

SitePoint — Thu, 14 May 2020 07:56:06 +0000

By James Hibbard. This article was originally published on SitePoint.

When I began as a JavaScript editor at SitePoint, our submission process was something of a mess — articles coming from every direction in a variety of formats. So we decided to standardize things and settled on submission via GitHub in Markdown.

This was a step forward, but we still needed to convert the Markdown into HTML for our WordPress back end. The powers that be at SitePoint HQ at the time had vetoed the installation of any WordPress plugins, which made us consider if we could accomplish this task with a browser extension. Luckily we could!

In the following post, I'm going to demonstrate how you can build your own Chrome extension to add additional functionality to WordPress. I'll also introduce you to SP-Tools, the extension we use at SitePoint to make life as an editor that little bit easier.

The Anatomy of a Chrome Extension

Despite what you might think, building a Chrome extension isn't difficult. Let's start by looking at the various components.

Parts of the following section are borrowed from another tutorial I wrote about building a Chrome extension using Vue.js.

The core piece of any Chrome extension is a manifest file. This is in a JSON format and provides important information about an extension, such as its version, resources, or the permissions it requires.

A manifest file won't do much on its own, so we can use a content script to add some functionality. Content scripts are files that "run in the context of web pages". That is to say, you specify the URL in your manifest file, then when your browser visits a page whose address matches the URL you specified, the content script is injected into the page and run.

To demonstrate these concepts, let’s start by writing a Chrome extension to do something on the SitePoint main site.

Make a new folder called my-extension and two files, manifest.json and main.js:

mkdir my-extension
cd my-extension
touch manifest.json main.js

Open up manifest.json and add the following code:

{
  "name": "My Extension",
  "version": "0.0.1",
  "manifest_version": 2,
  "content_scripts": [
    {
      "matches": [ "*://*.sitepoint.com/*" ],
      "js": [ "main.js" ]
    }
  ]
}

The name, version and manifest_version are all required fields. The name and version fields can be whatever you want; the manifest version should be set to 2 (as of Chrome 18).

The content_scripts key allows us to register a content script (main.js), which will be run whenever we visit SitePoint. Notice how we can use match patterns to specify parts of the URL, such as the protocol.

Now let’s add the following code to main.js to make the browser say hello whenever we visit SitePoint:

alert('Hello there!');

Finally, let’s install the extension. Open Chrome and enter chrome://extensions/ in the address bar. You should see a page displaying the extensions you’ve installed.

As we want to install our extension from a file (and not the Chrome Web Store) we need to activate Developer mode using the toggle in the top right-hand corner of the page. This should add an extra menu bar with the option Load unpacked. Click this button and select the my-extension folder you created previously. Click Open and the extension will be installed.

Now when you visit SitePoint, this will happen:

Congratulations! You just made a Chrome extension.

Background Scripts and Message Passing

So, that dialogue box is pretty annoying right? To finish off this section, let's add a context menu entry to fire it off manually, instead of having it appear on every page load.

This introduces us to another important component of Chrome extensions — background scripts. These scripts can react to browser events (such as a user clicking a context menu entry) and they have full access to Chrome's APIs. However, they don’t have access to the current page, and rely on message passing to communicate with content scripts.

Update the manifest like so:

{
  "name": "My Extension",
  "version": "0.0.1",
  "manifest_version": 2,
  "permissions": [ "contextMenus" ],
  "content_scripts": [
    {
      "matches": [ "*://*.sitepoint.com/*" ],
      "js": [ "main.js" ]
    }
  ],
  "background": {
    "scripts": ["background.js"],
    "persistent": false
  }
}

Notice that we’re requesting the contextMenus permission, as we want to add something to the context menu, and that we’ve registered a non-persistent background script. Making the background script non persistent allows it to be unloaded when it’s not needed.

Next, create a background.js file and add:

chrome.runtime.onInstalled.addListener(() => {
  chrome.contextMenus.create({
    id: 'greet',
    title: 'Say hi',
    contexts: ['page'],
    documentUrlPatterns: ['*://*.sitepoint.com/*'],
  });
});

chrome.contextMenus.onClicked.addListener((info, tab) => {
  if (info.menuItemId === 'greet') {
    chrome.tabs.sendMessage(tab.id, { text: 'greet' }, (res) => {
      console.log(res);
    });
  }
});

We register the context menu entry when the extension is installed, then add an event listener to send a message to our content script whenever the entry is clicked.

Change main.js like so:

chrome.runtime.onMessage.addListener((msg, sender, sendResponse) => {
  if (msg.text === 'greet') {
    alert('hi');
    sendResponse('I greeted the user');
  }
});

Here, we listen for a message from our background script. If it has a text of "greet", we then fire off the alert and send back a message informing the background script that we did what was requested.

To try this out, head back to the extensions page (at chrome://extensions/), then click the reload icon and reload any SitePoint page. When you right click, you should now see a context menu entry.

Enhancing WordPress' Functionality with a Chrome Extension

Now that we’ve familiarized ourselves with the basic components of a Chrome extension, let's look at how we can make some additions to WordPress' functionality.

To follow along with this section, you’ll need a working installation of WordPress. I installed mine locally. It’s running on an Apache server at http://localhost/wp.

The code for this extension can be found on GitHub.

Adding a Markdown Converter

Let's start by adding a Markdown converter to the WordPress editor. True to the experience on SitePoint, I'll be using the "classic" editor (achieved by installing the Disable Gutenberg plugin) and the Text view.

To begin, create the following folder structure for our new extension:

wp-enhance
├── lib
│   ├── jquery.min.js
│   └── showdown.min.js
├── manifest.json
└── scripts
    └── main.js

On the command line:

mkdir wp-enhance
cd wp-enhance
mkdir lib scripts
touch lib/showdown.min.js lib/jquery.min.js
touch scripts/main.js
touch manifest.json

As you can see, we’ll be using the Showdown markdown converter and jQuery (because I'm lazy).

The first order of business is to grab the latest minified version of these libraries (Showdown and jQuery) and add the contents to the appropriate files.

Next, add the following code to manifest.json:

{
  "manifest_version": 2,
  "name": "WP Enhance",
  "description": "A Chrome extension to enhance WordPress' functionality",
  "version": "0.0.1",
  "content_scripts": [
    {
      "matches": [ "http://localhost/wp/wp-admin/post-new.php" ],
      "js": [
        "lib/jquery.min.js",
        "lib/showdown.min.js",
        "scripts/main.js"
      ]
    }
  ]
}

There's nothing spectacular going on here. The extension is set to run when we visit http://localhost/wp/wp-admin/post-new.php, and we're including the two libraries we just downloaded.

Finally, in scripts/main add the following:

'use strict';

const $editorToolbar = $('#ed_toolbar');
const $mainTextArea = $('#content');

function getShowdownConverter() {
  const converter = new showdown.Converter();

  // Set any options here, for example to add table support
  converter.setOption('tables', true);

  return converter;
}

function addMDButton() {
  const mdConverter = getShowdownConverter();
  const $convertButton = $('<input />', {
    type: 'button',
    value: 'MD',
    class: 'ed_button button button-small',
    title: 'Convert MD to HTML',
    click() {
      const md = $mainTextArea.val();
      const html = mdConverter.makeHtml(md);
      $mainTextArea.val(html);
    },
  });
  $editorToolbar.append($convertButton);
}

addMDButton();

Here, we’re creating a new button and appending it to the WordPress editor's toolbar. When it’s clicked, we’re calling Showdown's makeHtml method, which we pass the contents of the content area. This returns us some HTML, which we then insert back into the editor.

Load the extension and visit the new post page. You should see something like this:

I'm sure you'll agree, that's a reasonably impressive result in just a few lines of code.

Adding a Date Picker to the Publish Widget

Next, we're going to enhance the publish widget with a datepicker. This will replace the series of drop-downs and input boxes that you normally see when you click the Edit link next to the "Publish immediately" message in WordPress' Publish widget.

The first thing we’ll need to do is to download a datepicker. For this demo I'll be using this one. You can download the necessary files from here. Unzip that file and place build/jquery.datetimepicker.full.min.js into our lib folder. Then create a new css folder in the extension and place build/jquery.datetimepicker.min.css into it.

Our extension should now look like this:

wp-enhance
├── css
│   └── jquery.datetimepicker.min.css
├── lib
│   ├── jquery.datetimepicker.full.min.js
│   ├── jquery.min.js
│   └── showdown.min.js
├── manifest.json
└── scripts
    └── main.js

Now include these files in the manifest:

{
  "manifest_version": 2,
  "name": "WP Enhance",
  "description": "A Chrome extension to enhance WordPress' functionality",
  "version": "0.0.1",
  "content_scripts": [
    {
      "matches": [ "http://localhost/wp/wp-admin/post-new.php" ],
      "js": [
        "lib/jquery.min.js",
        "lib/showdown.min.js",
        "lib/jquery.datetimepicker.full.min.js",
        "scripts/main.js"
      ],
      "css": [ "css/jquery.datetimepicker.min.css" ]
    }
  ]
}

Finally, we need to alter our content script (main.js) to look like this:

const $editorToolbar = $('#ed_toolbar');
const $mainTextArea = $('#content');
const $timeStampDiv = $('#timestampdiv');
const $wpSchedulePostDropdown = $('.timestamp-wrap');
let $datepicker;

const $dd = $('#jj');
const $mm = $('#mm');
const $yyyy = $('#aa');
const $hh = $('#hh');
const $mn = $('#mn');

function getShowdownConverter() { ... }

function addMDButton() { ... }

function addDatePicker() {
  $datepicker = $('<input />', {
    id: 'bandaid-datepicker',
    type: 'text',
    placeholder: 'Date and time',
  });

  $datepicker.datetimepicker();
  $timeStampDiv.prepend($datepicker);
}

addMDButton();

$wpSchedulePostDropdown.hide();
addDatePicker();

$datepicker.on('change', function updateDateFields() {
  // String in format yyyy/mm/dd hh:mm
  const dateString = this.value;

  $yyyy.val(dateString.substr(0, 4));
  $mm.val(dateString.substr(5, 2));
  $dd.val(dateString.substr(8, 2));
  $hh.val(dateString.substr(11, 2));
  $mn.val(dateString.substr(14, 2));
});

What we’re doing is getting a reference to the input elements that WP uses to manage the time and date of the scheduled post. We’re then hiding these elements and initializing the datepicker. Whenever a user selects a date, the hidden field is updated and the post can be scheduled.

Reload the extension, then refresh the WordPress new post page. What you have now should look like this:

Again, an impressive result for not much code.

Testing the Extension

One of the things I noticed early on with our SP-Tools extension was that, when WordPress got updated, things would break. So, I got to thinking how I could test the extension and decided that some end-to-end tests with Nightwatch would make sense.

In the following section, I'll demonstrate how we can test our extension in the same way.

First, we'll need to generate a package.json file. In the extension root, run npm init -y. Next, let's install Nightwatch and the ChromeDriver as dev dependencies:

npm install --save-dev nightwatch chromedriver

Now make a testdirectory and add a nightwatch.config.js file, as well as a wp.js file for our test code:

mkdir test
touch test/nightwatch.config.js test/wp.js

Add the following to the config file:

module.exports = {
  src_folders: 'test',
  output_folder: 'test',
  page_objects_path: '',
  custom_commands_path: '',
  custom_assertions_path: '',

  webdriver: {
    start_process: true,
    port: 9515,
    server_path: 'node_modules/.bin/chromedriver',
    log_path: false,
    cli_args: [],
  },

  test_settings: {
    default: {
      desiredCapabilities: {
        browserName: 'chrome',
        chromeOptions: {
          args: [
            'load-extension=./',
            '--test-type',
          ],
        },
      },
    },
  },
};

The important part is 'load-extension=./',, which tells Nightwatch to load our extension into the test browser.

And add the following to wp.js (replacing my login credentials with your own):

module.exports = {
  'Test WordPress Mods': (browser) => {
    browser
      // Login to WP Dashboard
      .url('http://localhost/wp/wp-login.php')
      .setValue('#user_login', 'jim')
      .setValue('#user_pass', 'secret')
      .click('#wp-submit')

      // Go to New Post Page
      .url('http://localhost/wp/wp-admin/post-new.php')

      // Test MD > HTML conversion
      .setValue('#content', '## level 2 heading\n### level 3 heading')
      .click('input[value="MD"]')
      .assert.valueContains('#content', '<h2 id="level2heading">level 2 heading</h2>')
      .assert.valueContains('#content', '<h3 id="level3heading">level 3 heading</h3>')

      // This is here so that you can take a look at the browser
      .pause(5000)
      .end();
  },
};

Now run the tests using:

node_modules/.bin/nightwatch --config test/nightwatch.config.js

You should see an instance of the Chrome browser open and Nightwatch perform the tests we’ve specified. The result of the tests is output to the terminal.

Hopefully Nightwatch's DSL is pretty self explanatory. You can read more about it in their documentation. If you fancy a challenge, try adding tests for the datepicker.

Note that I've hardcoded my credentials here. If you use this for anything other than demonstration purposes, it'll be a good idea to move these to a config file that’s not committed to GitHub.

And don't forget you can find the code for everything I’ve demonstrated so far on GitHub.

Notable Features of SitePoint's Chrome Extension

As I'm sure you've realized, your mileage will vary regarding how useful you find such a browser extension. Most people will have (slightly) different needs and will be able to install WordPress plugins to solve most of the problems they encounter.

Nonetheless, in this final section, I'd like to outline some of the features we have added to our SP-Tools extension in the hope that they might inspire or even be useful for others.

A Capitalize and Check button. This converts the post title to title case.
A headline analysis tool, which gives you a score out of 100 for your title and offers suggestions for improvements.
A Capitalize Subheadings button, which checks the remaining headings in the article for title capitalization.
A Copy Link button, which copies the post's current permalink to the clipboard.
A Rebuild Link button, which rebuilds the post's permalink. This is useful, for example, when WordPress creates a permalink based on a draft heading which subsequently changes.
An extensible molly-guard, which performs a number of checks and disables/enables the publish button accordingly. Among other things, it checks for:
- a sensible post permalink
- the presence of relative URLs in the editor pane
- the presence of empty links in the editor pane
- the presence of <h1> tags in the editor pane
- the presence of [special] shortcode tags in the excerpt
A Copy Tags button, which gives you a comma-separated list of tags copied to the clipboard.
A rel="sponsored" button, which toggles the rel attribute of all links in a post as being sponsored.

If you'd like to check it out, you can find our extension on GitHub. There are a few other goodies in there, such as context menu entries, some code to turn off the infinite scroll on our main site and, of course, tests with Nightwatch.

Conclusion

In this tutorial, we’ve looked at the various components that make up a Chrome extension. I’ve demonstrated how we can build and test our own Chrome extension to enhance the basic functionality of a WordPress install. I’ve also introduced you to SP-Tools, SitePoint's own Chrome extension, which we use to make various editing tasks somewhat easier.

If you find our extension useful, or adapt it to do anything else, I'd love to hear from you on Twitter.