Automate your API tests with Jenkins, Postman and Newman

A little background

Nowadays CI pipelines are a must on our development dayjob, helping us to shorten delivery cycles and to bring higher quality software.

As a backend developer, APIs have became a cornestone on most of my applications, so testing them became a must.

Having an API Test Stage in our CI pipeline brings us confidence in the quality of the product and alerts us if something had broken.

Postman

Postman is a great tool that helps developers in every stage of our API lifecycle. It has a great documentation and an increasingly community of developers which use it, so it is easy to find posts explaining cool tips and tricks.

Collections

Postman makes easy to organize our different request into collections. Personally I create a collection for every project I want to test. Then, inside the collection, I create a folder for every entity on the project, trying to cover every endpoint in the API.

Requests are organized by entities

Environments

Postman allows us to create different environments to run our requests. Giving a real example, I would tell you I work on three different environments: my local environment, a staging environment and a production environment, so I create three different environments with variables which represents the changing data between the environments (like the base path for the API requests). This way I can define a single request but target it to my three different environments easily. Think about how easy would be to change or add a new environment!

Extracting base path as environment variable to target different environments

Testing requests

One of the greatest features of Postman is that allows you to code tests that runs every time a request run. They are written in JS and have a cool set of assertions to check things like the response code, response body o response headers.

A request with few tests

Sweet! Once we have written a test suite to cover our API, we use the Collection Runner, a built-in tool provided by Postman to run every request of a collection and output a report of tests results.

How beatiful are the green meadows...

Bonus: Sometimes, we want to run our requests in a certain order and guess what? Postman allows us to do it! Check it at building workflows issue

What do we have so far?

At this point, we have build up a Postman collection with a bunch of request that could target our API in all our environments. Besides that, every request has several tests to check if it is working properly. What a time to be alive!

Newman

Do you remember the Collection Runner I mentioned previously? It could run every request in our collection with a simple click. But what about a command line tool to do the same? Newman does it. Newman is a command line Collection Runner for Postman that allows you to run and test a Postman Collection directly from the command line and integrate it easily in CI servers, that makes it a great tool for our purpose: Automate our API test with Jenkins.

Run a collection

In order to run a collection with Newman, we first have to export it from Postman. Since we have some variables playing around in some of our request, we have to export the suitable environment too. Once we have both files, we are able to run the collection with:

$ newman run our.postman_collection.json -e our.postman_environment.json

and we will see a report of the collection run, similar to others testing frameworks.

Newman CLI test report

Formatting the output

The report can be formatted in different ways. To achieve this, Newman provides a set of reporters to support different output formats like JSON, XUNIT or HTML. Besides that you can create your own reporter to format the report in any format you want.

Jenkins

Cool, we have defined our API test suite with Postman and we are able to run it from command line and obtain a nice report of the results so far with Newman. But now we want to include our awesome new skill in our CI pipeline.

Disclaimer: We are using Jenkins as an automation server to build up our CI/CD pipelines and we run our CI pipeline in a Docker container which have all the necessary tools installed (such Newman).

Mixing things up

Essentially we will need three files to include our API Test inside a CI pipeline:

• Jenkinsfile, which defines our pipeline
• Postman Collection, which defines our set of request and tests
• Postman Environment, which defines the variables of the target environment

We include all of them into project repository, so we can benefit of version control over them and track their changes.

Enough talking, let’s code!

We need to create a new stage in our pipeline to run the API Test. We usually put it after unit and integration tests to be sure everything is running as expected before testing the interface.

pipeline {
    ...
    stage('Test API Rest') {
        steps {
            sh 'newman run tests/Newman/our.postman_collection.json -e tests/Newman/env/test.postman_environment.json -r junit,html --reporter-junit-export var/reports/newman/junit/newman.xml --reporter-html-export var/reports/newman/html/index.html'

            publishHTML([allowMissing: false, alwaysLinkToLastBuild: false, keepAll: false, reportDir: 'var/reports/newman/html', reportFiles: 'index.html', reportName: 'Newman API Test', reportTitles: ''])
        }
    }
    ...
}

As you can see, we invoke the run command of Newman, providing our collection an the test environment file as parameters. Besides that, we also inform which reporters we want to use (JUNIT, HTML) and where the outputs will be stored.

As a bonus, we use HTML PUblisher Jenkins plugin to link the HTML report to the build.

I hope you enjoyed this journey as much as I did writing it and, as always, feedback will be welcomed!

Comments

[Boris Jamot ✊ /]

Very good sum-up!