Introduction
Running a set of Postman collections within a dedicated Azure Pipeline provides the ability to automate API testing of Development, Testing and even Staging environments with minimal manual intervention. Furthermore, through the use of regular runs, it encourages the adoption of the shift-left approach:
test early and often
Hence, teams can detect regressions and bugs early in the development lifecycle.
Create Postman Collection
The first step is to gather your Postman API requests under the same Collection. In this example, the collection is named Post-Office. Under the Tests tab, there is a simple assertion which verifies the expected status code returned after the API call.
Note: The endpoint *-foo.com is only an illustrative example and you should replace it with your own in your current context.
Create environments + variables
Ensure that you have created corresponding Postman Environments for the different contexts in which your code is being deployed to. Environments provide a way to organize variables which are only relevant for a specific context (API keys, external web-services, etc...).
Export the collection
You can export the relevant Postman Collection by hovering over it on the sidebar and clicking on the "..." (View more actions). Finally click on Export
A file with a default filename of *.postman_collection.json (in this case Post-Office.postman_collection.json) will be downloaded
Export Postman Environment Variables
Go to the Environments tab on the side bar, then click on the "..." and finally select export.
Create a new Pipeline
Once you have exported the collection and environment json files, ensure that you push them to the repository of your web app on Azure DevOps.
Now it is time for some plumbing! The task is to create a new pipeline that will execute your Postman collection scripts automatically.
Go to Pipelines and click on New. Select the classic editor.
Add Tasks
Task 1: Install newman package
For the first job, we will be using the newman package from npm to execute our postman collection.
Newman is a command-line Collection Runner for Postman. It enables you to run and test a Postman Collection directly from the command line.
Task 2: Execute Postman Collection
To execute the postman collection with newman, we will need to provide the following command:
newman run sample-collection.json -e env.postman_environment.json
The first argument of the newman run is the file path of your postman collection. The option -e take as input the file path of the Postman Environment file.
Making use of Pipeline variables
In the previous screenshot, we can see that we have hardcoded the environment filename for the option -e:
newman run sample-collection.json -e env.postman_environment.json
Instead of specifying directly the environment filename within the command, we can leverage the use of Pipeline variables.
We create a variable CUSTOM.ENVIRONMENT_FILE to store the filename.
We can then reference the variable inside the command as follows:
# Navigate to postman directory
DESTINATION='$(System.DefaultWorkingDirectory)/postman'
cd $DESTINATION
# Absolute path for environment file
ENVIRONMENT_FILE='$(System.DefaultWorkingDirectory)/postman/environments/$(CUSTOM.ENVIRONMENT_FILE)'
newman run collections/Post-Office.postman_collection.json -e $ENVIRONMENT_FILE
After the execution of the Postman Collection, the results of all tests and requests can be exported into a custom file.
This is made possible through the use of reporters. In our case, we will use a JUNIT reporter to export the test results to a custom file.
newman run collections/Post-Office.postman_collection.json -e $ENVIRONMENT_FILE --reporters cli,junit --reporter-junit-export test-results/outputfile.xml
outputfile.xml represents the custom file where you want to collect the test results.
View more about custom reporters here: Newman Custom Reporters
Task 3: Publish Test Results
The final step is to publish the test results and view the results on the Tests tab of the pipeline.
Add the Publish Test Results task.
View the test results on the Tests tab
That's its folks! Thank you 😃
Top comments (2)
Interesting article. However one problem I see with running postman collections this way is that your are pushing to your repository environment variables that might be secrets for some of them. So in the end you might have secrets in your source code ...
Postman is quite useful but I don't like it because you have to use huge json files that are not very readable and maintanable. I prefer using tools like rest client (vscode), not as powerful but work on text files following the HTTP convention.
[[..PingBack..]]
This article is curated as a part of #61st Issue of Software Testing Notes Newsletter.