Speeding up GitHub Actions with npm cache

GitHub Actions is a feature of GitHub that allows you to run various actions based on certain triggers within a secure VM that GitHub host for you. It has many uses, but it's primarily a CI/CD tool used to build, test and deploy your apps.

I love it. We use it on all of our projects. It's free (well, you get 2,000 minutes of VM run time for free each month), it works great, and it's all integrated into GitHub already.

We've also talked about how to cache npm effectively on GitHub Actions over on our website, be sure to check it out as it includes a few extra tips.

How we use GitHub Actions

There are tons of applications for GitHub Actions, but I'm going to be covering what we use it for every day here.

Our core use-case for it is CI/CD. We have something like the following:

1. A push is done to master (or main, or whatever your central branch is).
2. GitHub Actions triggers our workflow.
3. GitHub spins up a VM running the latest version of Ubuntu.
4. It installs a bunch of pre-requisites for our app to run (in our case at accreditly.io that's things like PHP, Imagick and node).
5. It already has access to our code, so it then sets up the project (copying our .env file, generating a fresh key, setting up some permissions, etc).
6. Installs our dependencies via composer, the PHP package manager.
7. Creates a local sqlite database, and configures the app to use it.
8. Runs migrations to set up our database, and then runs seeders to populate it with fake data.
9. Runs npm install to install our dependencies.
10. Runs our full test suite.
11. Assuming everything passes, it then deploys.

This all gets defined in our .github/workflows/laravel.yml file.

It works really well. From the second a commit is pushed to when a deployment commences takes about 5 minutes. That's not bad for starting a VM, installing an OS, setting up a project with tons of dependencies and running a test suite.

But, we can do better than that.

Optimising npm install

By far the slowest part of our GitHub Action workflow is npm install and npm run prod (which builds our assets), coming in at around 3min 30sec. It differs by about ~40seconds each side of that, so it must depend on available bandwidth or where the VM is located at any given time. Either way, that's a whopping amount of our ~5min 30sec total build time.

On my local machine it takes around 30 seconds to install and build. Granted I have quite a decent machine, but that's a huge disparity. The main reason for this is that we have a large number of packages that we use, each with a chain of dependencies attached. That's a big overhead.

So let's cache it.

Actions Cache

GitHub maintain a set of repos called actions. One of which is called cache.

Let's start by going through our full file.

# Name of our workflow
name: Laravel

# When to run it
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

# Define our jobs
jobs:
  laravel-tests:

    runs-on: ubuntu-latest
    # Setup
    steps:
    - uses: actions/checkout@v3
    - name: Update apt
      run: sudo apt-get update
    - name: Install imagick
      run: sudo apt-get install php-imagick
    - uses: actions/setup-node@v3
      with:
        node-version: 16
    - uses: nanasess/setup-php@master
      with:
        php-version: '8.1'

    - name: Copy .env
      run: php -r "file_exists('.env') || copy('.env.example', '.env');"

    - name: Install Dependencies
      run: composer install --no-ansi --no-interaction --no-scripts --no-progress --prefer-dist --ignore-platform-req=ext-imagick

    - name: Generate key
      run: php artisan key:generate

    - name: Directory Permissions
      run: chmod -R 777 storage bootstrap/cache

    - name: Create Database
      run: |
        mkdir -p database
        touch database/database.sqlite
    - name: Migrate
      env:
        DB_CONNECTION: sqlite
        DB_DATABASE: database/database.sqlite
      run: php artisan migrate
      run: php artisan db:seed

OK that looks like a lot is going on, but it's just following points 1-8 in the above list. It sets up our VM, installs some dependencies, packages, and sets up our database.

Now we need to handle npm. What we're going to do next is:

1. Install the cache Action module.
2. Create a cache in ~/.npm containing our packages as a hash.
3. Check if there is a cache hit (eg. do we already have a cached version of our packages?), if so continue, if not install them.
4. Continue with compilation.

    - name: Cache node modules
      id: cache-npm
      uses: actions/cache@v3

      env:
        cache-name: cache-node-modules
      with:
        # npm cache files are stored in `~/.npm` on Linux/macOS
        path: ~/.npm
        key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
        restore-keys: |
          ${{ runner.os }}-build-${{ env.cache-name }}-
          ${{ runner.os }}-build-
          ${{ runner.os }}-

    - if: ${{ steps.cache-npm.outputs.cache-hit != 'true' }}
      name: List the state of node modules
      continue-on-error: true
      run: npm list

    - name: Compile assets
      run: |
        npm install
        npm run production

Then we continue running our tests and deploy (we use Forge to deploy, so it's just a case of hitting a webhook):

    - name: Execute tests (Unit and Feature tests) via PHPUnit
      run: vendor/bin/phpunit
    - name: Deploy to Laravel Forge
      run: curl ${{ secrets.FORGE_DEPLOYMENT_WEBHOOK }}

Putting it all together:

# Name of our workflow
name: Laravel

# When to run it
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

# Define our jobs
jobs:
  laravel-tests:

    runs-on: ubuntu-latest
    # Setup
    steps:
    - uses: actions/checkout@v3
    - name: Update apt
      run: sudo apt-get update
    - name: Install imagick
      run: sudo apt-get install php-imagick
    - uses: actions/setup-node@v3
      with:
        node-version: 16
    - uses: nanasess/setup-php@master
      with:
        php-version: '8.1'

    - name: Copy .env
      run: php -r "file_exists('.env') || copy('.env.example', '.env');"

    - name: Install Dependencies
      run: composer install --no-ansi --no-interaction --no-scripts --no-progress --prefer-dist --ignore-platform-req=ext-imagick

    - name: Generate key
      run: php artisan key:generate

    - name: Directory Permissions
      run: chmod -R 777 storage bootstrap/cache

    - name: Create Database
      run: |
        mkdir -p database
        touch database/database.sqlite
    - name: Migrate
      env:
        DB_CONNECTION: sqlite
        DB_DATABASE: database/database.sqlite
      run: php artisan migrate
      run: php artisan db:seed

    - name: Cache node modules
      id: cache-npm
      uses: actions/cache@v3

      env:
        cache-name: cache-node-modules
      with:
        # npm cache files are stored in `~/.npm` on Linux/macOS
        path: ~/.npm
        key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
        restore-keys: |
          ${{ runner.os }}-build-${{ env.cache-name }}-
          ${{ runner.os }}-build-
          ${{ runner.os }}-

    - if: ${{ steps.cache-npm.outputs.cache-hit != 'true' }}
      name: List the state of node modules
      continue-on-error: true
      run: npm list

    - name: Compile assets
      run: |
        npm install
        npm run production

    - name: Execute tests (Unit and Feature tests) via PHPUnit
      run: vendor/bin/phpunit
    - name: Deploy to Laravel Forge
      run: curl ${{ secrets.FORGE_DEPLOYMENT_WEBHOOK }}

Our new full build time is around 3 mins, which is a massive saving for simply caching the npm dependencies.

We could further improve this by doing the same thing with composer, however the composer dependencies already install very quickly.

We're currently considering moving to Pest for our tests, which is substantially quicker, particularly when running in parallel mode.