Testing npm packages against multiple versions of their peer dependency

If you're maintaining an npm package you know how automated testing takes away much of the pain that comes with it. Especially regression tests help you release with confidence by preventing you from accidentally introducing breaking changes. For most packages, testing the API and running the tests with all supported node versions is enough. Maintaining a plugin, however, comes with further challenges. Typically a plugin supports multiple versions of its host package. An Express middleware, for example, is supposed to work with a range of Express versions. The only way to ensure this is to actually test a plugin against different versions of its host package. I tried to achieve that and with two small tools - package aliases and job matrices - it required only small changes to my existing test setup.

I came across this issue when maintaining the package hapi-auth-any, an authentication strategy for hapi. You don't need to know anything about hapi or hapi-auth-any. I'll just use it as an example of how I adapted my tests after I added support for another major version. I originally wrote the plugin to work with hapi 18. At that point I decided testing against the latest minor version would be sufficient, so my package.json file looked somewhat like that:

{
 ...
 "peerDependencies": {
    "@hapi/hapi": "18.x"
  },
  "devDependencies": {
    "@hapi/hapi": "^18.3.1",
  }
}

By defining hapi 18.x as a peer dependency, I was telling the user of my plugin that they needed to have any version of hapi 18 installed. That wouldn't automatically install hapi when installing the plugin but it would warn them if they didn't meet this requirement. Since peer dependencies aren't automatically installed I also needed to define hapi as a dev dependency so I could import it in my tests like that:

const hapi = require('@hapi/hapi');

Then, I set up a hapi instance, registered the hapi-auth-any authentication strategy and wrote some test cases. All was set.

Some months later, hapi 19 was released. I checked the breaking changes and didn't find any compatibility issues. To ensure that with further development I wouldn't break the compatibility to either version I decided to run the tests against both hapi 18 and hapi 19.

Instead of involving some black pipeline magic I chose a solution that works independent of the pipeline, so any contributor could run the tests against the version of hapi as they like. For this I needed to install two versions of the same package and find a way to tell the test which one it should use. The solution for this problem was package aliases. They were introduced in npm 6.9.0 and allow you to install packages under a different name. That way you can install the same package multiple times and import it using the alias. So all I needed to do was to change the package.json file to look like this:

{
  ...
  "devDependencies": {
    "hapi18": "npm:@hapi/hapi@^18.3.1",
    "hapi19": "npm:@hapi/hapi@^19.0.5",
  },
  "peerDependencies": {
    "@hapi/hapi": "18.x || 19.x"
  }
}

I decided to use an environment variable to tell the tests which version it should use so I replaced the first line of my test file with the following:

const hapi = require(process.env.HAPI_VERSION === '19' ? 'hapi19' : 'hapi18');

If the environment variable HAPI_VERSION is set to 19 it'll use hapi 19, otherwise hapi 18. That makes it easy to use in the pipeline. In order to make it easier for contributors to run the tests I added another test script to the package.json file that uses cross-env to set the environment variable:

{
  ...
  "test": "nyc ava",
  "test:hapi19": "cross-env HAPI_VERSION=19 nyc ava",
}

Now, npm test with no env variable set runs the tests against hapi 18, npm run test:hapi19 runs them against hapi 19.

That was all I needed to do to test against multiple versions of the same peer dependency. Now I only needed to change my GitHub Action so it would test against both versions.

Here, something called a job matrix came to my help. With the help of a job matrix you can generate jobs by substituting specific parts of a job definition. I was already using a job matrix to run my tests with different node versions.

name: Node CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [8.x, 10.x, 12.x]
    steps:
    - uses: actions/checkout@v1
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: install, lint, and test
      run: |
        npm install
        npm run lint
        npm test
      env:
        CI: true

This creates a job for each node version. We can use such a matrix to set the environment variable HAPI_VERSION to 18 or 19 respectively.

name: Node CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version:
          - 8.x
          - 10.x
          - 12.x
        hapi-version:
          - 18
          - 19
    env:
      HAPI_VERSION: ${{ matrix.hapi-version }}
    steps:
    - uses: actions/checkout@v1
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: install, lint, and test
      run: |
        npm install
        npm run lint
        npm test
      env:
        CI: true

Now, the workflow creates a job for each combination of node and hapi version. This seems like what we want but there's a small issue with that: hapi 19 dropped support for both Node 8 and 10. In order to exclude those specific combinations, we can use the exclude key.

name: Node CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version:
          - 8.x
          - 10.x
          - 12.x
        hapi-version:
          - 18
          - 19
        exclude:
          - node-version: 8.x
            hapi-version: 19
          - node-version: 10.x
            hapi-version: 19
    env:
      HAPI_VERSION: ${{ matrix.hapi-version }}
    steps:
    - uses: actions/checkout@v1
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: install, lint, and test
      run: |
        npm install
        npm run lint
        npm test
      env:
        CI: true

That's it. All the tests run now against all the hapi versions and the node versions they support. Even for my small package, there's a benefit in that; for packages with a lot of contributors and bigger releases, this might be even more helpful. With such a setup you can finally release with confidence.

Comments

[Andrei Gatej]

Very interesting! Thanks for sharing!

I think an alternative to cross-env(for unix systems at least) is to use the “env” command.

For example

env $(cat dev.env) command$

[Jonas Scheffner]

Thanks for the tip. I chose cross-env to make it easier for windows users to contribute. If it's a seperate script that runs only in the pipeline, the env command seems like a legit way to do it.

[Christian Bewernitz]

Thx for this, exactly what I was looking for.

What do you think of having the "current" version as hapi in your devDependencies and only old versions with the numeric postfix? This way if tests need to be different for different versions they can be written for the specific versions and you don't need to replace anything, right?

[Jonas Scheffner]

I can see some advantages of not using a prefix for the current version. I'm using dependabot for automatic dependency updates and without a prefix it could create a MR to update the hapi version after a new release. In that case, my tests would fail if there was a compatibility issue and I would know about it much faster than I do now.

I'm not so sure about writing tests for specific versions of the dependency. That makes it harder to find out if the plugin works the same for all versions. I had a test that wasn't working for hapi 19 because it accessed some internal property that was made private in the newer release. I decided to remove the test case because I thought it wasn't worth it. But maybe there are some cases when it's just not possible to run the same tests against all the supported versions.