DEV Community: Anton Yakutovich

Fast and reliable end-to-end tests with Playwright on GitHub Actions

Anton Yakutovich — Tue, 24 Jan 2023 17:48:30 +0000

Playwright is a powerful web testing tool supporting Chromium, Firefox and WebKit engines. The project has excellent documentation and many examples.

I use Playwright for E2E tests on the DEV environment. Today I'm going to show how you can integrate Playwright on GitHub Actions more efficiently.

The documentation suggests using the following snippet to run Playwright inside GitHub Actions:

steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-node@v3
    with:
      node-version: '18'
  - name: Install dependencies
    run: npm ci
  - name: Install Playwright
    run: npx playwright install --with-deps
  - name: Run your tests
    run: npx playwright test
  - name: Upload test results
    if: always()
    uses: actions/upload-artifact@v3
    with:
      name: playwright-report
      path: playwright-report

What is notable here?

It's supposed to run all tests on the same type of GitHub Runner. ubuntu-latest by default.
All supported browser engines (chromium, firefox, WebKit) will be installed on the Linux runner.
Playwright will execute tests for all browser engines from the same runner.
The last step will prepare an archive with an HTML report.

After several months of experiments I come up with the following configuration:

# ...

jobs:
  test:
    name: 🧪 ${{ matrix.project }} E2E Tests
    runs-on: ${{ matrix.os }}
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        include:
          - project: chromium
            os: ubuntu-latest
            cache_dir: ~/.cache/ms-playwright
          - project: firefox
            os: ubuntu-latest
            cache_dir: ~/.cache/ms-playwright
          - project: webkit
            os: macos-12
            cache_dir: ~/Library/Caches/ms-playwright
    steps:

    - uses: actions/checkout@v3

    - uses: actions/setup-node@v3
      with:
        node-version-file: '.nvmrc'
        cache: 'npm'

    - name: ⚡️ Cache NPM dependencies
      uses: actions/cache@v3
      id: cache-primes
      with:
        path: node_modules
        key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}

    - name: ⚡️ Cache playwright binaries
      uses: actions/cache@v3
      id: playwright-cache
      with:
        path: ${{ matrix.cache_dir }}
        key: ${{ runner.os }}-${{ matrix.project }}-pw-${{ hashFiles('**/.playwright-version') }}

    - name: 📥 Install deps
      if: steps.cache-primes.outputs.cache-hit != 'true'
      run: make install

    - name: 📥 Install ${{ matrix.project }} with Playwright
      if: steps.playwright-cache.outputs.cache-hit != 'true'
      run: npx playwright install --with-deps ${{ matrix.project }}

    - name: 🎭 Playwright tests
      run: npx playwright test --project=${{ matrix.project }}
      env:
        DEBUG: pw:api,pw:browser*

    - name: 📊 Upload test results
      if: failure()
      uses: actions/upload-artifact@v3
      with:
        name: playwright-report-${{ matrix.project }}
        path: playwright-report

What I've changed in GitHub Workflow for Playwright tests?

I use a matrix with three browser engines to run the jobs simultaneously. Instead of ubuntu-latest, I run WebKit tests on macos-12 runner. I had to make the change to avoid fragile tests. I noticed that the page.goto() API works differently on Linux and macOS for the WebKit engine. WebKit didn't wait for load event, which causes failures from time to time. At the same time, I've never caught such a problem locally on my MacBook. If you're interested, the ticket is still open in the Playwright bug tracker.
The fail-fast parameter is set to false to wait for all testing jobs to complete, even if one of them fails for some reason.
I believe it is essential to specify timeout-minutes for E2E tests so that the tests will crash if the infrastructure hangs or behaves incorrectly.
I use the .nvmrc file inside the repository to specify the NodeJS version. DRY all over the place!
Configured aggressive cache for npm dependencies and binary files with browsers. I added a special target to Makefile to calculate the Playwright browsers hash:

bump-playwright:
  npm install --save-dev @playwright/test
  npx playwright --version > .github/.playwright-version

After executing make bump-playwright command, the latest PW version will be installed and the value of the installed release will be written to the .github/.playwright-version file. GitHub Actions calculate the hash for the cache from the file content. Example file:

$ cat .github/.playwright-version
Version 1.29.2

Each runner installs only one browser and it's pretty handy cause the HTML report will have the results for that browser — easy to navigate and spot the issues. This distinction also allows you to have an idea of the test execution speed in each browser and to notice flickering tests specific to a particular engine.
Playwright tests run with enabled DEBUG, so that in case of a crash, you can quickly get additional information for diagnostics.
In contrast to the recommended configuration from the documentation, I prefer to upload artifacts with the report only in case of test failures. I don't see the point in storing those artifacts on the cloud, given how often the tests are run.

I shared my findings for configuring E2E playwright tests in GitHub Actions. Feel free to ask questions about the example GitHub workflow.

What Playwright tricks do you use in your daily work? Would love to hear about your hidden gems 💎.

Cover photo by Denny Müller.

Aggressive dependency caching in GitHub Actions

Anton Yakutovich — Tue, 18 Oct 2022 15:54:52 +0000

There are three things you can watch forever: fire burning, water falling, and how the build passes the stages in Pipeline after the next commit. To make the wait less tedious, it's best to take care of the CI setup from the beginning.

GitHub Actions has a cache that gets to the runner's virtual machine in seconds. In this article I'd like to share examples of how to set up aggressive dependency caching. Why did I call this approach "aggressive caching"? Because we will be caching not only the packages archives but also the state of the environment after installation.

For Node.js it will be the node_modules directory, and for Python it will be the virtualenv directory with installed dependencies.

Node.js Example

Let's take the typical setup for dependency caching example mentioned in the documentation. If you don't need any exotics, you can use the standard actions/setup-node action, specifying a package manager.

steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
  with:
    node-version: 16
    cache: 'npm'
- run: npm ci
- run: npm test

This will save the .npm directory with the global package cache. Sounds great! Remember that if we have several workflow jobs requiring npm ci inside, this will also be time-consuming.

Let's imagine a pipelining with several jobs:

                         │
          Create         │                        Reuse
          Dependencies   │                        Dependencies
          Cache          │
                         │  ┌────────────────────┐
                ┌────────┼──►     Lint  Job      ├────────────────────┐
                │        │  └────────────────────┘                    │
                │        │                                            │
                │        │                                            │
┌───────────────┴────┐   │  ┌────────────────────┐                 ┌──▼─────────────────┐
│     Build Job      ├───┼──►     Test  Job      ├─────────────────►     Deploy Job     │
└───────────────┬────┘   │  └────────────────────┘                 └──▲─────────────────┘
                │        │                                            │
                │        │                                            │
                │        │  ┌────────────────────┐                    │
                └────────┼──►     E2E   Job      ├────────────────────┘
                         │  └────────────────────┘
                         │
                         │
                         │
                         │

Ideally, we want to install dependencies only in the first job and get a state with available dependencies in all subsequent jobs.
I'll show how to achieve it using a sample repo — redux-react-realworld-example-app.

The first (build) job might look like this:

steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
  with:
    node-version-file: '.nvmrc' # (1)
    cache: 'npm'

- name: Cache NPM dependencies # (2)
  uses: actions/cache@v3
  id: cache-primes
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}

- name: Install dependencies # (3)
  if: steps.cache-primes.outputs.cache-hit != 'true'
  run: npm ci

- name: Build
  run: npm run build

Line #1 specifies the node version using .nvmrc file. That's the alternative way to specify the version and it helps follow the DRY - Don't Repeat Yourself principle.

In line #2 we use actions/cache to cache the node_modules directory. We use the hash from the package-lock.json file as the key.

In line #3 we only install dependencies if the cache is invalidated.

To automatically retrieve node_modules in subsequent jobs, you must declare actions/cache with the same key. For example the test job can be configured as:

steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
  with:
    node-version-file: '.nvmrc'
    cache: 'npm'

- name: Cache NPM dependencies
  uses: actions/cache@v3
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }} # (1)

- name: Tests
  run: npm run test # (2)

Line #1 specifies the cache key. The key must be the same as in the build job. After the actions/cache step we consider that the dependencies are installed and run the tests in line #2.

Check out the complete workflow on GitHub.

Python Example

Standard scenario from actions/setup-python docs:

steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
  with:
    python-version: '3.9'
    cache: 'pip' # caching pip dependencies
- run: pip install -r requirements.txt

This workflow will cache pip packages in ~/.cache/pip, but the installation step will always be performed, as in the previous example with npm ci.

Let's see how we can optimize the installation of dependencies. I'll use the Django-based education-backend repo.

Let's dive into the build job:

steps:
- uses: actions/checkout@v3

- uses: actions/setup-python@v4
  id: setup-python
  with:
    python-version-file: '.python-version'

- uses: actions/cache@v3
  with:
    path: venv
    key: ${{ runner.os }}-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/*requirements.txt') }} # (1)

- name: Install dependencies # (2)
  if: steps.cache-primes.outputs.cache-hit != 'true'
  run: |
    python -m venv venv
    . venv/bin/activate
    pip install --upgrade pip pip-tools
    pip-sync requirements.txt dev-requirements.txt

- name: Run the linter
  run: |
    . venv/bin/activate # (3)
    cp src/app/.env.ci src/app/.env
    make lint

As you can see, we use the same idea for caching as for the Node.js project. There are some minor changes which are quite important. We need to specify the cache key for each python version involved in the workflow. Line #1 has steps.setup-python.outputs.python-version variable exactly for this purpose.

Dependencies installation from line #2 is tricky. For python we use a virtual environment created with the module venv. The environment directory venv will be cached. You can think about it as node_modules for node.

Line #3 has one more trick. After the cache is warmed up it's necessary to initialize virtualenv in the future steps. Otherwise, the python interpreter will not be able to detect the necessary libraries to import.

The simplified test job may look as follows:

- uses: actions/checkout@v3

- uses: actions/setup-python@v4
  id: setup-python
  with:
    python-version-file: '.python-version'

- uses: actions/cache@v3
  with:
    path: venv
    key: ${{ runner.os }}-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/*requirements.txt') }} # (1)

- name: Run the tests
  run: |
    . venv/bin/activate # (2)
    cp src/app/.env.ci src/app/.env
    make test

Ensure you use the same key for the caching step (line #1) and remember to activate the virtual environment before running the tests (line #2).

Check out the complete workflow on GitHub.

Summary

We've practiced "aggressive caching" with Node.js and Python examples. As far as you have a significant number of dependencies the changes can speed up your GitHub workflow sensibly. I recommend trying to set up workflows for your project using the references I've mentioned:

If you still have questions about caching in GitHub Actions, don't hesitate to ask in the comments. I'll try to help.
I would be grateful if you share your tips on how to speed up workflows on GitHub too.

Operation Pact or: How I Learned to Stop Worrying and Love Contract Testing

Anton Yakutovich — Tue, 20 Sep 2022 08:44:37 +0000

I heard about Contract Tests in detail at the Ministry of Testing Meetup. Abhi Nandan made an excellent introduction talk Contract testing in Polyglot Microservices.

I liked the idea and decided to try this approach on my current project. I must say that the devil is in the details. In theory, everything is simple until you try the integration in practice on an existing project.
It is the same with algorithms: when you are watching how the developer on YouTube solves the task, everything seems straightforward. You immediately hit a snag while sitting down to solve the problem alone.

Today I want to focus on practical examples, as much as possible close to real-world conditions. I encourage you to try it on your own to gain experience and see if these approaches are worth applying to your projects. We will use the realword project on GitHub, an Exemplary Medium.com clone powered by different frameworks and tools.

I like using this project for the demo because it's much more complex than HelloWorld, where everything works most of the time. True-to-life problems give your extensive experience.

What is so unique in contract tests?

Contract tests assert that inter-application messages conform to a shared understanding documented in a contract. Without contract testing, the only way to ensure that applications will work correctly together is by using expensive and brittle integration tests.

You can hear opinions: Contract tests often make sense when you have a microservice architecture with 10-50 services.
I'm afraid I have to disagree with that. If you have at least one Provider (back-end with API) and Consumer (front-end), it already makes sense to try this testing approach.

I suggest starting exploration with bi-directional contract tests.

Bi-Directional Contract Testing is a type of static contract testing.
Teams generate a consumer contract from a mocking tool (such as Pact or Wiremock) and API providers verify a provider contract (such as an OAS) using a functional API testing tool (such as Postman). Pactflow then statically compares the contracts down to the field level to ensure they remain compatible.

Why is it better to start with BDCT? Bi-Directional Contract Testing (BDCT) allows you to use existing Provider Open API. If your current project has a back-end service with Open API specification, you are lucky to leverage it as a Provider contract.

1. Provider Contract

Let's configure publishing of OAS spec from realworld app to pactflow.

Requirements:

Create a free account on Pactflow.io;
Install Docker on your machine.

We want to be able to publish Open API spec to Pactflow locally and from CI/CD. To do that we will need the following: API key from Pactflow, docker and make commands installed.

Consider the Makefile:

# Makefile
PACTICIPANT ?= "realworld-openapi-spec"

## ====================
## Pactflow Provider Publishing
## ====================
PACT_CLI="docker run --rm -v ${PWD}:/app -w "/app" -e PACT_BROKER_BASE_URL -e PACT_BROKER_TOKEN pactfoundation/pact-cli"
OAS_FILE_PATH?=api/openapi.yml
OAS_FILE_CONTENT_TYPE?=application/yaml
REPORT_FILE_PATH?=api/README.md
REPORT_FILE_CONTENT_TYPE?=text/markdown
VERIFIER_TOOL?=newman

# Export all variable to sub-make if .env exists
ifneq (,$(wildcard ./.env))
    include .env
    export
endif

default:
    cat ./Makefile

ci: ci-test publish_pacts can_i_deploy

# Run the ci target from a developer machine with the environment variables
# set as if it was on CI.
# Use this for quick feedback when playing around with your workflows.
fake_ci:
    @CI=true \
    GIT_COMMIT=`git rev-parse --short HEAD` \
    GIT_BRANCH=`git rev-parse --abbrev-ref HEAD` \
    make ci

## =====================
## Build/test tasks
## =====================

ci-test:
    @echo "\n========== STAGE: CI Tests ==========\n"

## =====================
## Pact tasks
## =====================

publish_pacts:
    @echo "\n========== STAGE: publish provider contract (spec + results) - success ==========\n"
    PACTICIPANT=${PACTICIPANT} \
    "${PACT_CLI}" pactflow publish-provider-contract \
    /app/${OAS_FILE_PATH} \
    --provider ${PACTICIPANT} \
    --provider-app-version ${GIT_COMMIT} \
    --branch ${GIT_BRANCH} \
    --content-type ${OAS_FILE_CONTENT_TYPE} \
    --verification-exit-code=0 \
    --verification-results /app/${REPORT_FILE_PATH} \
    --verification-results-content-type ${REPORT_FILE_CONTENT_TYPE} \
    --verifier ${VERIFIER_TOOL}

deploy: deploy_app record_deployment

can_i_deploy:
    @echo "\n========== STAGE: can-i-deploy? 🌉 ==========\n"
    "${PACT_CLI}" broker can-i-deploy --pacticipant ${PACTICIPANT} --version ${GIT_COMMIT} --to-environment test

deploy_app:
    @echo "\n========== STAGE: deploy ==========\n"
    @echo "Deploying to test"

record_deployment:
    "${PACT_CLI}" broker record-deployment --pacticipant ${PACTICIPANT} --version ${GIT_COMMIT} --environment test

## =====================
## Misc
## =====================

.env:
    cp -n .env.example .env || true

Run make .env to create .env file. You will need to update two variables:

PACT_BROKER_BASE_URL
PACT_BROKER_TOKEN

[Optional] We may try to publish the contract from the local machine using make fake-ci. This target under the hood executes the following stages:

ci-test runs functional tests (skipped in our case)
publish_pacts uses pactflow docker image to publish current OAS to Pactflow broker. GIT_COMMIT and GIT_BRANCH environment variables are used to set the version tag and the branch on the broker side.
can_i_deploy verifies that the published contract is compatible with dependent consumers. Hence we are safe to deploy.

And the last piece. Let's create a GitHub workflow to publish OAS on each push to the repository:

# .github/workflows/pact.yml

name: Pact CI

on:
  pull_request:
  push:
    branches:
      - main
    paths-ignore:
      - '*.md'

concurrency:
  # For pull requests, cancel all currently-running jobs for this workflow
  # https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency
  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
  cancel-in-progress: true

env:
  PACT_BROKER_BASE_URL: https://drakulavich.pactflow.io
  PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }}
  GIT_COMMIT: ${{ github.sha }}

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: Inject slug/short variables
      uses: rlespinasse/github-slug-action@v4

    - name: Tests
      run: GIT_BRANCH=${GITHUB_REF_NAME_SLUG} make ci

  deploy:
    runs-on: ubuntu-latest
    needs:
      - verify
    if: ${{ github.ref == 'refs/heads/main' }}
    steps:
      - uses: actions/checkout@v3

      - name: Inject slug/short variables
        uses: rlespinasse/github-slug-action@v4

      - name: 🚀 Record deployment on Pactflow
        run: GIT_BRANCH=${GITHUB_REF_NAME_SLUG} make deploy

GitHub will use the same Makefile. Please, have a look at the Deploy job from the workflow above. We are executing make deploy to explicitly mark the deployed version for the test environment on the Pactflow broker.

We will provide the environment variable to access Pactflow differently. Don't forget to add the secret PACT_BROKER_TOKEN in Settings — Secrets — New repository secret.

Ok. We are ready to commit the files! If everything is fine, the pipeline should be green.

Open your Pactflow workspace and check out the Provider Contract. You should be able to open Open API Swagger spec.

So, we finished with the first part. We have CI/CD for Provider Contract. Let's move on to the second part.

2. Consumer contract

We will use ts-redux-react-realworld-example-app project which implements the front-end in TS for realword app using React/Redux.

If you want to follow the same approach, I suggest you to check pact-workshop-js for better understanding.

1) We'll start by installing @pact-foundation/pact dependency:

npm install --save-dev @pact-foundation/pact

2) Create directory src/services/consumer/
3) Create src/services/consumer/apiPactProvider.ts
Here we'll describe the pact specification format and name of our pacticipants (participants).

// src/services/consumer/apiPactProvider.ts

import path from 'path';
import { PactV3, MatchersV3, SpecificationVersion } from '@pact-foundation/pact';

export const { eachLike, like } = MatchersV3;

export const provider = new PactV3({
  consumer: 'ts-redux-react-realworld-example-app',
  provider: 'realworld-openapi-spec',
  logLevel: 'warn',
  dir: path.resolve(process.cwd(), 'pacts'),
  spec: SpecificationVersion.SPECIFICATION_VERSION_V2,
});

4) Create src/services/consumer/api.pact.spec.ts

Let's start with something tiny and achievable. We would like to check /tags endpoint.

// src/services/consumer/api.pact.spec.ts

describe('API Pact tests', () => {
  describe('getting all tags', () => {
    test('tags exist', async () => {
      // set up Pact interactions

      // Execute provider interaction
    });
  });
});

From pact-workshop-js you will learn that each Pact test contains two parts:

Setup Pact interactions.
Execute the interaction using the API client.

The first part might look like:

// src/services/consumer/api.pact.spec.ts
// ...
      const tagsResponse = {
        tags: ['reactjs', 'angularjs'],
      };

      // set up Pact interactions
      await provider.addInteraction({
        states: [{ description: 'tags exist' }],
        uponReceiving: 'get all tags',
        withRequest: {
          method: 'GET',
          path: '/tags',
        },
        willRespondWith: {
          status: 200,
          headers: {
            'Content-Type': 'application/json',
          },
          body: like(tagsResponse),
        },
      });
// ...

We created a stub for the response tagsResponse. Then we added interaction to the provider object from src/services/consumer/apiPactProvider.ts. The essential parts relate to the expected request and how the provider will reply.

In the second part we will add the test execution:

// src/services/consumer/api.pact.spec.ts
// ...

      await provider.executeTest(async (mockService) => {
        axios.defaults.baseURL = mockService.url;

        // make request to Pact mock server
        const tags = await getTags();

        expect(tags).toStrictEqual(tagsResponse);
      });
// ...

Ultimately we need to change the URL for the API client and execute the request after it. In our case we have to alter axios baseURL, because all API interactions explicitly use axios. For example, getTags() implementation:

export async function getTags(): Promise<{ tags: string[] }> {
  return guard(object({ tags: array(string) }))((await axios.get('tags')).data);
}

We want to verify that response is the same as a prepared stub. After collecting all parts inside the test file we will get the following:

// src/services/consumer/api.pact.spec.ts

import axios from 'axios';
import { provider, like } from './apiPactProvider';
import { getTags } from '../conduit';

describe('API Pact tests', () => {
  describe('getting all tags', () => {
    test('tags exist', async () => {
      const tagsResponse = {
        tags: ['reactjs', 'angularjs'],
      };

      // set up Pact interactions
      await provider.addInteraction({
        states: [{ description: 'tags exist' }],
        uponReceiving: 'get all tags',
        withRequest: {
          method: 'GET',
          path: '/tags',
        },
        willRespondWith: {
          status: 200,
          headers: {
            'Content-Type': 'application/json',
          },
          body: like(tagsResponse),
        },
      });

      await provider.executeTest(async (mockService) => {
        axios.defaults.baseURL = mockService.url;

        // make request to Pact mock server
        const tags = await getTags();

        expect(tags).toStrictEqual(tagsResponse);
      });
    });
  });
});

Let's run make fake-ci to check how is going:

Computer says yes \o/ 

CONSUMER                             | C.VERSION          | PROVIDER               | P.VERSION  | SUCCESS? | RESULT#
-------------------------------------|--------------------|------------------------|------------|----------|--------
ts-redux-react-realworld-example-app | d981433+1663657371 | realworld-openapi-spec | 95dbd23... | true     | 1

Hooray! It's a huge step towards contract testing.
Now let's try to test another endpoint GET /user. We will start with the same ideas: create a stub, prepare interactions and execute API client call:

// src/services/consumer/api.pact.spec.ts
// ...

  describe('getting current user', () => {
    test('user exists', async () => {
      const tUser: User = {
        email: 'jake@jake.jake',
        token: 'jwt.token.here',
        username: 'jake',
        bio: 'I work at statefarm',
        image: 'https://i.stack.imgur.com/xHWG8.jpg',
      };

      const userResponse = {
        user: tUser,
      };
      // set up Pact interactions
      await provider.addInteraction({
        states: [{ description: 'user has logged in' }],
        uponReceiving: 'get user',
        withRequest: {
          method: 'GET',
          path: '/user',
        },
        willRespondWith: {
          status: 200,
          headers: {
            'Content-Type': 'application/json',
          },
          body: like(userResponse),
        },
      });

      await provider.executeTest(async (mockService) => {
        axios.defaults.baseURL = mockService.url;

        // make request to Pact mock server
        const user = await getUser();

        expect(user).toStrictEqual(tUser);
      });
    });
  });

// ...

If you run make fake-ci, you will get an error:

Computer says no ¯_(ツ)_/¯

Follow the generated report link and you will see the details:

Request Authorization header is missing but is required by the spec file

We missed Authorization header that is required to check logged in user. Let's add the stub (1), headers to the expected request (2) and axios defaults (3):

// src/services/consumer/api.pact.spec.ts

      // (1)
      const authToken = {
        Authorization: 'Token xxxxxx.yyyyyyy.zzzzzz',
      };

// ...

        uponReceiving: 'get user',
        withRequest: {
          method: 'GET',
          path: '/user',
          headers: authToken, // (2)
        },
// ...
      await provider.executeTest(async (mockService) => {
        axios.defaults.baseURL = mockService.url;
        axios.defaults.headers.Authorization = authToken.Authorization; // (3)

After next make fake-ci attempt you will get the new error:

FAIL src/services/consumer/api.pact.spec.ts
  ● Console

    console.error
      Error: Cross origin http://localhost forbidden

I found the fix for it on StackOverflow. We need to add the following line on the top of the test file:

axios.defaults.adapter = require('axios/lib/adapters/http');

Next make fake-ci should be successful.

Summary

We have learned how to configure Provider and Consumer for bi-directional contract testing. We used RealWorld project to practice integration with Pactflow broker. And finally, we wrote a couple of consumer tests for ts-redux-react-realworld-example-app. We also learned how to debug the tests using make fake-ci command. All of the examples are available on GitHub. Feel free to try and post your questions if you face the issues.

Materials

RealWorld repo with Open API spec (provider contract)
ts-redux-react-realworld-example-app with pact tests (consumer)
pact js workshop

GitLab CI: Cache and Artifacts explained by example

Anton Yakutovich — Wed, 04 Aug 2021 10:13:46 +0000

Hi, DEV Community! I've been working in the software testing field for more than eight years. Apart from web services testing, I maintain CI/CD Pipelines in our team's GitLab.

Let's discuss the difference between GitLab cache and artifacts. I'll show how to configure the Pipeline for the Node.js app in a pragmatic way to achieve good performance and resource utilization.

There are three things you can watch forever: fire burning, water falling, and the build is passing after your next commit. Nobody wants to wait for the CI completion too much, it's better to set up all the tweaks to avoid long waiting between the commit the build status. Cache and artifacts to the rescue! They help reduce the time it takes to run a Pipeline drastically.

People are confused when they have to choose between cache and artifacts. GitLab has bright documentation, but the Node.js app with cache example and the Pipeline template for Node.js contradict each other.

Let's see what the Pipeline in GitLab terms means. The Pipeline is a set of stages and each stage can have one or more jobs. Jobs work on a distributed farm of runners. When we start a Pipeline, a random runner with free resources executes the needed job. The GitLab-runner is the agent that can run jobs. For simplicity, let's consider Docker as an executor for all runners.

Each job starts with a clean slate and doesn't know the results of the previous one. If you don't use cache and artifacts, the runner will have to go to the internet or local registry and download the necessary packages when installing project dependencies.

What is cache?

It's a set of files that a job can download before running and upload after execution. By default, the cache is stored in the same place where GitLab Runner is installed. If the distributed cache is configured, S3 works as storage.

Let's suppose you run a Pipeline for the first time with a local cache. The job will not find the cache but will upload one after the execution to runner01. The second job will execute on runner02, it won't find the cache on it either and will work without it. The result will be saved to runner02. Lint, the third job, will find the cache on runner01 and use it (pull). After execution, it will upload the cache back (push).

What are artifacts?

Artifacts are files stored on the GitLab server after a job is executed. Subsequent jobs will download the artifact before script execution.

Build job creates a DEF artifact and saves it on the server. The second job, Test, downloads the artifact from the server before running the commands. The third job, Lint, similarly downloads the artifact from the server.

To compare the artifact is created in the first job and is used in the following ones. The cache is created within each job.

Consider the CI template example for Node.js recommended by GitLab:



image: node:latest # (1)

# This folder is cached between builds
cache:
  paths:
    - node_modules/ # (2)

test_async:
  script:
    - npm install # (3)
    - node ./specs/start.js ./specs/async.spec.js

test_db:
  script:
    - npm install # (4)
    - node ./specs/start.js ./specs/db-postgres.spec.js

Line #1 specifies the docker image, which will be used in all jobs. The first problem is the latest tag. This tag ruins the reproducibility of the builds. It always points to the latest release of Node.js. If the GitLab runner caches docker images, the first run will download the image, and all subsequent runs will use the locally available image. So, even if a node is upgraded from version XX to YY, our Pipeline will know nothing about it. Therefore, I suggest specifying the version of the image. And not just the release branch (node:14), but the full version tag (node:14.2.5).

Line #2 is related to lines 3 and 4. The node_modules directory is specified for caching, the installation of packages (npm install) is performed for every job. The installation should be faster because packages are available inside node_modules. Since no key is specified for the cache, the word default will be used as a key. It means that the cache will be permanent, shared between all git branches.

Let me remind you, the main goal is to keep the pipeline reproducible. The Pipeline launched today should work the same way in a year.

NPM stores dependencies in two files — package.json and package-lock.json. If you use package.json, the build is not reproducible. When you run npm install the package manager puts the last minor release for not strict dependencies. To fix the dependency tree, we use the package-lock.json file. All versions of packages are strictly specified there.

But there is another problem, npm install rewrites package-lock.json, and this is not what we expect. Therefore, we use the special command npm ci which:

removes the node_modules directory;
installs packages from package-lock.json.

What shall we do if node_modules will be deleted every time? We can specify NPM cache using the environment variable npm_config_cache.

And the last thing, the config does not explicitly specify the stage where jobs are executed. By default, the job runs inside the test stage. It turns out that both jobs will run in parallel. Perfect! Let's add jobs stages and fix all the issues we found.

What we got after the first iteration:



image: node: 16.3.0 # (1)

stages:
  - test

variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm" (5)

# This folder is cached between builds
cache:
  key:
    files:
      - package-lock.json (6)
  paths:
    - .npm # (2)

test_async:
  stage: test
  script:
    - npm ci # (3)
    - node ./specs/start.js ./specs/async.spec.js

test_db:
  stage: test
  script:
    - npm ci # (4)
    - node ./specs/start.js ./specs/db-postgres.spec.js

We improved Pipeline and make it reproducible. There are two drawbacks left. First, the cache is shared. Every job will pull the cache and push the new version after executing the job. It's a good practice to update cache only once inside Pipeline. Second, every job installs the package dependencies and wastes time.

To fix the first problem we describe the cache management explicitly. Let's add a "hidden" job and enable only pull policy (download cache without updating):



# Define a hidden job to be used with extends
# Better than default to avoid activating cache for all jobs
.dependencies_cache:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm
    policy: pull

To connect the cache you need to inherit the job via extends keyword.



...
extends: .dependencies_cache
...

To fix the second issue we use artifacts. Let's create the job that archives package dependencies and passes the artifact with node_modules further. Subsequent jobs will run tests from the spot.



setup:
  stage: setup
  script:
    - npm ci
  extends: .dependencies_cache
  cache:
    policy: pull-push
  artifacts:
    expire_in: 1h
    paths:
      - node_modules

We install the npm dependencies and use the cache described in the hidden dependencies_cache job. Then we specify how to update the cache via a pull-push policy. A short lifetime (1 hour) helps to save space for the artifacts. There is no need to keep node_modules artifact for a long time on the GitLab server.

The full config after the changes:



image: node: 16.3.0 # (1)

stages:
  - setup
  - test

variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm" (5)

# Define a hidden job to be used with extends
# Better than default to avoid activating cache for all jobs
.dependencies_cache:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm
    policy: pull

setup:
  stage: setup
  script:
    - npm ci
  extends: .dependencies_cache
  cache:
    policy: pull-push
  artifacts:
    expire_in: 1h
    paths:
      - node_modules

test_async:
  stage: test
  script:
    - node ./specs/start.js ./specs/async.spec.js

test_db:
  stage: test
  script:
    - node ./specs/start.js ./specs/db-postgres.spec.js

We learned what's the difference between cache and artifacts. We built a reproducible Pipeline that works predictably and uses resources efficiently. This article shows some common mistakes and how to avoid them when you are setting up CI in GitLab.
I wish you green builds and fast pipelines. Would appreciate your feedback in the comments!

DEV Community: Anton Yakutovich

Fast and reliable end-to-end tests with Playwright on GitHub Actions

Aggressive dependency caching in GitHub Actions

Node.js Example

Python Example

Summary

Operation Pact or: How I Learned to Stop Worrying and Love Contract Testing

What is so unique in contract tests?

1. Provider Contract

2. Consumer contract

Summary

Materials

GitLab CI: Cache and Artifacts explained by example

What is cache?

What are artifacts?

Links