How to schedule JAMstack deploys with Netlify and GitHub

One thing that got me excited about the JAMstack movement is the idea of a content mesh. I heard the term first when the Gatsby folks talked about combining multiple data sources into static JAMstack sites, which became one of Gatsby's core strengths.

The idea behind a content mesh is that you use APIs to collect and display third-party-data during build time.

My site runs on Netlify and is currently a content mesh of:

• Buttondown's API to show a newsletter count in the footer
• Feedly to display a webring of RSS feeds that I'm subscribed to
• Contentful to write and render my blog posts
• Netlify Analytics to fetch and render my "popular articles" page
• GitHub to automatically enrich my projects with stargazer data

Honestly, I think that's pretty cool. But suppose you're running a JAMstack site fed by many data sources; when and how do you rebuild the site and refresh the data?

Trigger deploys with Netlify build hooks

To trigger deploys on Netlify, you can define build hooks. These are HTTP endpoints that will start a new deploy whenever an HTTP request comes in.

I'm currently running with four different hooks starting Netlify builds. To make sure you can analyze what's starting your builds, I recommend defining each build hook depending on its functionality or trigger.

You see that I defined a REBUILD_CRON hook. This URL endpoint is supposed to be called within a time interval – via a cronjob, so to say.

If you wonder how do you set up a cronjob in the cloud, you're not alone. Many people asked for scheduled deploys and cronjobs on Netlify's platform. The recommendation so far has been to use Zapier or another third-party tool.

I never liked this approach because it adds another service dependency to my setup.

But now that we have GitHub actions, and these turned out to be schedulable, I finally found the perfect solution to keep my website data fresh without bringing in another tool.

A scheduled GitHub action to build and deploy your Netlify site

To dive into GitHub actions, I recommend reading the documentation. If you're only looking for a snippet that works, here's the tl;dr:

• add a workflow file to your repository (/.github/worfklows/main.yml)
• paste the snippet below and update it to use your build hook URL
• add, commit and push the new file to GitHub

# /.github/worfklows/main.yml
name: Trigger Netlify Build at Midnight
on:
  schedule:
    # Every day at midnight
    # If you're also puzzled by 
    # the cron syntax check
    # -> https://crontab.guru
    - cron: '0 0 * * *'
jobs:
  build:
    name: Netlify Midnight Cron
    runs-on: ubuntu-latest
    steps:
      - name: Curl request
        run: curl -X POST -d {} https://api.netlify.com/build_hooks/xxx

main.yml defines a scheduled GitHub workflow that curls a Netlify build hook URL every midnight.

That's beautiful! And setting up GitHub actions feels like a perfect solution to refresh my site data because it's just another way GitHub interacts with Netlify; I love that!

Comments

[kontrollanten]

Thanks for sharing! Where in the process are you running your tests? In Netlify? Personally I haven't found any beautiful way of solving that.

[Stefan Judis]

I'm not running any tests at the moment. Did you have a look a build plugins? postbuild might be what you're lookin for?

[kontrollanten]

I checked postBuild, but it doesn't really feel like it's the right place to run tests. I mean I want to have CI pipelines triggering different jobs depending on whats going on. Netlify is a build environment, so putting in the tests inside there just feels ugly (to me).

Actually I'm thinking about building something like Netlify, but instead of building the whole site, you just send all your static files. Then the flow would look like push > trigger CI > run linting, tests, etc > build and deploy static files > run e2e tests or whatever. That would mean that your CI env is responsible for all the actions. It would be interesting to hear your opinion, would that make sense?

[Stefan Judis]

What you're saying makes sense to me. :) For projects that are more complex than my own site, I think going with a "classic" CI provider makes more sense. The engineering teams at my company use CircleCI paired with Netlify and they're pretty happy with that. That said...

Actually I'm thinking about building something like Netlify, but instead of building the whole site, you just send all your static files

I'm not sure if that's needed if you can chain and combine CI providers with Netlify etc. :)

[kontrollanten]

Nice with someone with a different opinion! =D In my experience combining Netlify and CI environments works mostly fine, but I've had problems with

• Multiple build environments. Sometimes the build works in CI, but not in Netlify or vice versa. Bumping node version, etc requires double config change.
• Running e2e, lighthouse or something else after the site has been deployed isn't possible. I've created a small PoC of a bridge to solve that, but it shouldn't be that tricky.
• It's illogical that the site is built two times. I'm building my site in the CI env to run packtracker and then it's built again in Netlify. Of course I can run packtracker inside Netlify, but then it starts getting messy with no separation of concerns.

[Stefan Judis]

I see. :)

Yeah, I'm not very deep into the topic, but I can def imagine that there are many little details that make it trickier after a first look. I bet your three problems are only a few. 🙈