DEV Community

Cover image for Running Postman Collections in Azure Pipelines
Kherin Bundhoo
Kherin Bundhoo

Posted on

Running Postman Collections in Azure Pipelines

Introduction

Running a set of Postman collections within a dedicated Azure Pipeline provides the ability to automate API testing of Development, Testing and even Staging environments with minimal manual intervention. Furthermore, through the use of regular runs, it encourages the adoption of the shift-left approach:

test early and often

Hence, teams can detect regressions and bugs early in the development lifecycle.

Create Postman Collection

The first step is to gather your Postman API requests under the same Collection. In this example, the collection is named Post-Office. Under the Tests tab, there is a simple assertion which verifies the expected status code returned after the API call.

Note: The endpoint *-foo.com is only an illustrative example and you should replace it with your own in your current context.

Sample Postman Collection

Create environments + variables

Ensure that you have created corresponding Postman Environments for the different contexts in which your code is being deployed to. Environments provide a way to organize variables which are only relevant for a specific context (API keys, external web-services, etc...).

Postman Environments

Export the collection

You can export the relevant Postman Collection by hovering over it on the sidebar and clicking on the "..." (View more actions). Finally click on Export

Export Postman Collection

A file with a default filename of *.postman_collection.json (in this case Post-Office.postman_collection.json) will be downloaded

Export Postman Environment Variables

Go to the Environments tab on the side bar, then click on the "..." and finally select export.

Export Postman Environment Variables

Create a new Pipeline

Once you have exported the collection and environment json files, ensure that you push them to the repository of your web app on Azure DevOps.

Now it is time for some plumbing! The task is to create a new pipeline that will execute your Postman collection scripts automatically.

Go to Pipelines and click on New. Select the classic editor.

Pipeline Classic Editor

Add Tasks

Task 1: Install newman package

For the first job, we will be using the newman package from npm to execute our postman collection.

Newman is a command-line Collection Runner for Postman. It enables you to run and test a Postman Collection directly from the command line.

-- https://learning.postman.com/docs/running-collections/using-newman-cli/command-line-integration-with-newman/

Install newman package

Task 2: Execute Postman Collection

To execute the postman collection with newman, we will need to provide the following command:

newman run sample-collection.json -e env.postman_environment.json

The first argument of the newman run is the file path of your postman collection. The option -e take as input the file path of the Postman Environment file.

Execute Collection Runner

Making use of Pipeline variables

In the previous screenshot, we can see that we have hardcoded the environment filename for the option -e:

newman run sample-collection.json -e env.postman_environment.json
Enter fullscreen mode Exit fullscreen mode

Instead of specifying directly the environment filename within the command, we can leverage the use of Pipeline variables.
We create a variable CUSTOM.ENVIRONMENT_FILE to store the filename.

Pipeline Variables

We can then reference the variable inside the command as follows:

Reference Pipeline Variable

# Navigate to postman directory
DESTINATION='$(System.DefaultWorkingDirectory)/postman'
cd $DESTINATION

# Absolute path for environment file
ENVIRONMENT_FILE='$(System.DefaultWorkingDirectory)/postman/environments/$(CUSTOM.ENVIRONMENT_FILE)'

newman run collections/Post-Office.postman_collection.json -e $ENVIRONMENT_FILE

Enter fullscreen mode Exit fullscreen mode

After the execution of the Postman Collection, the results of all tests and requests can be exported into a custom file.
This is made possible through the use of reporters. In our case, we will use a JUNIT reporter to export the test results to a custom file.

newman run collections/Post-Office.postman_collection.json  -e $ENVIRONMENT_FILE --reporters cli,junit --reporter-junit-export test-results/outputfile.xml
Enter fullscreen mode Exit fullscreen mode

outputfile.xml represents the custom file where you want to collect the test results.

View more about custom reporters here: Newman Custom Reporters

Task 3: Publish Test Results

The final step is to publish the test results and view the results on the Tests tab of the pipeline.

Add the Publish Test Results task.

Publish Test Results

View the test results on the Tests tab

Test Results Tab

That's its folks! Thank you 😃

References

Latest comments (2)

Collapse
 
priteshusadadiya profile image
Pritesh Usadadiya

[[..PingBack..]]
This article is curated as a part of #61st Issue of Software Testing Notes Newsletter.

Collapse
 
techwatching profile image
Alexandre Nédélec • Edited

Interesting article. However one problem I see with running postman collections this way is that your are pushing to your repository environment variables that might be secrets for some of them. So in the end you might have secrets in your source code ...

Postman is quite useful but I don't like it because you have to use huge json files that are not very readable and maintanable. I prefer using tools like rest client (vscode), not as powerful but work on text files following the HTTP convention.