Setting up GitHub Actions Workflows to automate the API tests

In the previous "episode", I presented a quick solution to automate the API Tests using GitLab pipelines. Here, I'd like to discuss the same solution working on GitHub repository instead. GitHub provides built-in support for CI/CD as well as GitLab, so I expect that by applying the same principles I should have no problems (or at least it shouldn't be a complicated task).

What we have at disposal

1. GitHub repository to perform the continuous integration. We are going to use Newman, the command-line collection runner for Postman.
2. The extracted Postman Collection. Here is the same HttpbinForNewman.json file used in the GitLab case. The purpose, is to compare the results obtained previously in GitLab with the one we are going to see in GitHub.

Continuous Integration on GitHub

As described in the GitHub Docs, is possible to create a CI workflow directly in the GitHub repository using the GitHub Actions. A workflow is a collection of jobs that can perform tasks of continuous integration. Each job consist of a small operations called steps. The workflow can be configured to run right after a certain type of event, for instance when a push is made to a specific branch, and it can directly run on GitHub-hosted virtual machines.

Setting up GitHub Actions

A custom workflow can be created opening the Actions pane of the repository. The main page contains a list of popular workflows / CI template, clicking on the Configure a workflow yourself button a new commit with an empty workflow file will be created in the repository.

All workflows must reside inside a specific path of the GitHub repository: .github/workflows. Inside that folder there is the YAML file that describes the jobs. The name of that file has no particular conventions, but the structure is well specified. Below, I will present two different solutions, with different approach:

1. with postman/newman docker image
2. without postman/newman docker image

The structure of the YAML file

As a first step, right after the name of the workflow (CI), is necessary to define the event that will trigger the Action. In that case is set for each push event on the master branch.

name: CI

on:
  push:
    branches: 
    - master

As you can see below, the name of the workflow corresponds to the attribute specified on the yaml file. Each job executed, is linked to the commit message, in that case docker run

Now we need to define the list of jobs that will be executed: the first sub-line is intended for the name of the job (test), the runs-on field specifies the runner selected and for this solution will be a Ubuntu GitHub-hosted virtual machine.

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master

The uses field specifies a predefined task that can run inside the virtual machine, there are thousands of such actions available on GitHub Marketplace for free, later we will meet other examples. In the case shown above the Action is intended to checkout the latest commit on master branch, in order to have an updated version of the source code for the desired branch. This is, in general, the very first step to perform before proceeding with the other steps. Now it's time to approach the implementation of the two solutions mentioned before.

Solution with postman/newman docker image

Super easy, it's possible to run the docker command directly inside the VM, using the run field. When you create the step you just have to be careful to copy inside the container the correct volume, containing the json file with the Postman Collection.

- name: run docker command on VM
  run: |
    docker run -v $(pwd)/test:/etc/newman -t postman/newman:latest run "HttpbinForNewman.json" --reporters="cli"

Solution without postman/newman docker image

In this case the solution is a bit more articulated: to run Newman command, the Node.js >= v10 need to be installed on the VM before and specific GitHub Action exists for that: actions/setup-node@v1. Using the with field you can set options to configure the action. Then you have to install Newman, the easiest way is using NPM. The last step is run the agent: the newman run command allows you to specify a collection to be run. The -r is the flag related to the preferred reporter, in this case through cli.

- name: Install Node
  uses: actions/setup-node@v1
  with: 
    node-version: '12.x'

- name: Install newman
  run: |
    npm install -g newman

- name: Run POSTMAN collection
  run: |
    newman run ./test/HttpbinForNewman.json -r cli

The result

From the Actions pane of the GitHub page, is it possible to easily understand the list of results from the different workflows. The picture below is related to the one just discussed.

Extending the Docker for newman portion, is it possible to investigate the logs and evaluate the success of the tests performed through Newman, as shown below.

This is the YAML file used to obtain that result.

name: CI

on:
  push:
    branches: 
    - main

jobs:
  test-api:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@main

    - name: Docker for newman
      run: |
        docker run -v $(pwd)/test:/etc/newman -t postman/newman:latest run "HttpbinForNewman.json" --reporters="cli"

At the end of these few lines, there are not so many differences between the GitLab pipeline and GitHub Actions, in relation to the case of using Newman to automate the API tests. The only aspect to take into account is the different structure of the YAML file.