Fast and reliable end-to-end tests with Playwright on GitHub Actions

#webdev #github #devops #testing

Playwright is a powerful web testing tool supporting Chromium, Firefox and WebKit engines. The project has excellent documentation and many examples.

I use Playwright for E2E tests on the DEV environment. Today I'm going to show how you can integrate Playwright on GitHub Actions more efficiently.

The documentation suggests using the following snippet to run Playwright inside GitHub Actions:

steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-node@v3
    with:
      node-version: '18'
  - name: Install dependencies
    run: npm ci
  - name: Install Playwright
    run: npx playwright install --with-deps
  - name: Run your tests
    run: npx playwright test
  - name: Upload test results
    if: always()
    uses: actions/upload-artifact@v3
    with:
      name: playwright-report
      path: playwright-report

What is notable here?

It's supposed to run all tests on the same type of GitHub Runner. ubuntu-latest by default.
All supported browser engines (chromium, firefox, WebKit) will be installed on the Linux runner.
Playwright will execute tests for all browser engines from the same runner.
The last step will prepare an archive with an HTML report.

After several months of experiments I come up with the following configuration:

# ...

jobs:
  test:
    name: 🧪 ${{ matrix.project }} E2E Tests
    runs-on: ${{ matrix.os }}
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        include:
          - project: chromium
            os: ubuntu-latest
            cache_dir: ~/.cache/ms-playwright
          - project: firefox
            os: ubuntu-latest
            cache_dir: ~/.cache/ms-playwright
          - project: webkit
            os: macos-12
            cache_dir: ~/Library/Caches/ms-playwright
    steps:

    - uses: actions/checkout@v3

    - uses: actions/setup-node@v3
      with:
        node-version-file: '.nvmrc'
        cache: 'npm'

    - name: ⚡️ Cache NPM dependencies
      uses: actions/cache@v3
      id: cache-primes
      with:
        path: node_modules
        key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}

    - name: ⚡️ Cache playwright binaries
      uses: actions/cache@v3
      id: playwright-cache
      with:
        path: ${{ matrix.cache_dir }}
        key: ${{ runner.os }}-${{ matrix.project }}-pw-${{ hashFiles('**/.playwright-version') }}

    - name: 📥 Install deps
      if: steps.cache-primes.outputs.cache-hit != 'true'
      run: make install

    - name: 📥 Install ${{ matrix.project }} with Playwright
      if: steps.playwright-cache.outputs.cache-hit != 'true'
      run: npx playwright install --with-deps ${{ matrix.project }}

    - name: 🎭 Playwright tests
      run: npx playwright test --project=${{ matrix.project }}
      env:
        DEBUG: pw:api,pw:browser*

    - name: 📊 Upload test results
      if: failure()
      uses: actions/upload-artifact@v3
      with:
        name: playwright-report-${{ matrix.project }}
        path: playwright-report

What I've changed in GitHub Workflow for Playwright tests?

I use a matrix with three browser engines to run the jobs simultaneously. Instead of ubuntu-latest, I run WebKit tests on macos-12 runner. I had to make the change to avoid fragile tests. I noticed that the page.goto() API works differently on Linux and macOS for the WebKit engine. WebKit didn't wait for load event, which causes failures from time to time. At the same time, I've never caught such a problem locally on my MacBook. If you're interested, the ticket is still open in the Playwright bug tracker.
The fail-fast parameter is set to false to wait for all testing jobs to complete, even if one of them fails for some reason.
I believe it is essential to specify timeout-minutes for E2E tests so that the tests will crash if the infrastructure hangs or behaves incorrectly.
I use the .nvmrc file inside the repository to specify the NodeJS version. DRY all over the place!
Configured aggressive cache for npm dependencies and binary files with browsers. I added a special target to Makefile to calculate the Playwright browsers hash:

bump-playwright:
  npm install --save-dev @playwright/test
  npx playwright --version > .github/.playwright-version

After executing make bump-playwright command, the latest PW version will be installed and the value of the installed release will be written to the .github/.playwright-version file. GitHub Actions calculate the hash for the cache from the file content. Example file:

$ cat .github/.playwright-version
Version 1.29.2

Each runner installs only one browser and it's pretty handy cause the HTML report will have the results for that browser — easy to navigate and spot the issues. This distinction also allows you to have an idea of the test execution speed in each browser and to notice flickering tests specific to a particular engine.
Playwright tests run with enabled DEBUG, so that in case of a crash, you can quickly get additional information for diagnostics.
In contrast to the recommended configuration from the documentation, I prefer to upload artifacts with the report only in case of test failures. I don't see the point in storing those artifacts on the cloud, given how often the tests are run.

I shared my findings for configuring E2E playwright tests in GitHub Actions. Feel free to ask questions about the example GitHub workflow.

What Playwright tricks do you use in your daily work? Would love to hear about your hidden gems 💎.

Cover photo by Denny Müller.