Automate the API tests in Gitlab CI with postman/newman docker image

The purpose of this article is to present a simple procedure to test automatically the service API response. The solution leverage the features offered by the Continuous Integration tool built into Gitlab. Along these few lines will be presented the agents involved and how to integrate every part. First of all the Postman Collection and its test tab, then the dockerized Newman command to run all the requests automatically in sequence and finally the integration with Gitlab preparing an ad-hoc gitlab-ci.yml file to execute all the test using the CI pipeline after every change pushed.

That's it, let's start then!

TL;DR

1. Postman: the API desktop tool client. The tool is used to send requests and inspect the responses, in order to easily debug the behavior of the API service during its development.
2. Newman: is a command line agent that allows to run multiple requests sequentially. It takes as input the Postman Collection extracted as a json file and return the responses with the results of the each test, if any.
3. CI: Gitlab has its own section for Continuous Integration. For every change submitted, Gitlab is able to run a pipeline of scripts to build, test and validate the code changes. In this case, the CI will be devoted to execute automatically the requests contained inside the extracted Postman Collection, evaluating the results of each test. The status of each job will be related to the result obtained by the test.

Project reference

To those who are interested, a specific Gitlab repository has been created in order to investigate more thoroughly the solution presented here.
Feel free to take inspiration from that if you find it useful.
generic-api-gitlabci.
Without a real API service to test, the response has been simulated using httpbin as endpoint. In particular the REST request https://httpbin.org/anything returns a predictable response: its payload is created extracting the details of the request received.

First step: postman

For those who like me have used Postman to send a REST request and analyze the response received, will be happy to know that there is much more to discover inside that tool. The test tab has been created for develop a specific test / assertion cases for each request; the guidelines are well documented here, on the Postman Collection SDK. This is a NodeJS module through which the developer can manipulate each request presented on the collection. There are many possibilities and useful functions to preare a full test case and investigate in depth each response; for semplicity I selected the easiest test/assertion cases as you can see in the HttpbinForNewman.json file. The list of test to be performed are included inside that json file, it could be easily downloaded and imported into the Postman tool as a new Collection to evaluate the content. As endpoint I used httpbin, which has the advantage of providing a predictable responses each time a request is sent.

Second step: newman

Newman is a command-line agent that runs the entire list of requests contained inside the Postman Collection executing the tests, if any. The easiest solution could be trigger it using its postman/newman docker image. Be careful to use the proper path and the proper Postman Collection extracted as a json file to pass to the docker command. These two parameters are respectively named pwd_path_collection and name_of_collection_extracted in the command below. In this case, the result will be reported via cli.

docker run -v <pwd_path_collection>:/etc/newman -t postman/newman:latest run "name_of_collection_extracted.json" --reporters="cli"

At this stage you can test the Newman command directly from your local machine: the outcome of the command will be a table with the details of each request launched.

Third step: gitlab-ci

The last step is to integrate what Newman has been performed into Gitlab CI pipeline, again exploiting the docker image. The purpose is to create an automated tested solution, that could be run each time a new change is pushed on the remote repository. This step is achieved by adding the gitlab-ci.yml file into the project's root as you can see here. To run the script with the docker command inside the Gitlab is necessary using docker-in-docker (dind) image. The Newman solution could be easily integrate inside the test stage of the CI. Since this is the core aspect of the solution, the content of the gitlab-ci.yml file will be directly copied here.

image: docker:dind

services:
  - docker:dind

postman_tests:
  script:
  - docker run -v $(pwd)/test:/etc/newman -t postman/newman:latest run "HttpbinForNewman.json" --reporters="cli"
  stage: test

For convenience I have chosen to upload the file inside the test folder of the project, for that reason the docker volume argument mount the directory /test on the host inside the container.

Test cases

I have prepared two requests representing the test cases:

1. GET -> https://httpbin.org/anything?first=a&second=b
2. POST -> https://httpbin.org/anything with a json body

Both the requests have their tailor made list of tests.
Once the Collection has been imported in the Postman tool, is sufficient send the request to see the tests executed. This is valid for both the requests in the Collection.

As soon as the request will be exectued, the result of the test will be presented immediately. Here for example the screenshot obtained launching the GET request.

The outcome is rather simple to understand, but has the limitation of manual execution for each request. The integration with Gitlab CI has been discussed to overcome this. Let's see the outcome of the CI pipeline job.

The image above has been taken from here. From the list of all jobs finished is possible view immediately the status of each job, if passed or failed. The status is related to the outcome of the test.

Conclusions

The great advantage of integrating a solution like the one presented here is related to the creation of an automated process to test the integrity of the API service after each change introduced in the repository. That could save developers time and trouble, with the effort of spending a little time preparing the list of requests that the API service must always satisfy. Thanks to the Gitlab CI pipeline integration the developer could have an immediate feedback if the last change pushed have compromised anything on the response of the service API.

Comments

[Srinivas Kadiyala]

So are you connecting postman to docker image in local system?