API provider contract testing for all with Portman, OpenAPI and Postman

Introduction

APIs are everywhere and power almost everything we use today.
OpenAPI has revolutionized the way APIs are designed, documented, deployed... Its kind of a big thing.

Making a great OpenAPI definition is helped with amazing tools such as Stoplight Spectral which can help you ensure that your definition is not only valid against the OpenAPI specification but also your own API standards through custom linting rules.

How do you ensure that what you create and deploy matches your definition? Do you write all your tests by hand? Do you probe with Postman and hope for the best?

Maybe you do today but this blog will help you try something new... Welcome to the world of contract testing.

By the end of this blog you will have completed your first contract test (against my mocked API) and you should be free to use this knowledge to find gaps in your or any API out there subject to permission!

Getting started

Rather than spending too long explaining what contract testers are, where they came from etc... Lets use one! (It's free so what is stopping you?)

We will be using apideck Portman. You are welcome to follow their readme.MD and come back later or follow the instructions here.

Installing Portman

Install Portman

$ npm install -g @apideck/portman

Initialize your CLI

portman --init

Creating a contract test to run in Postman

Make a new directory and download this example portman configuration file and store it there
https://api.oneadvanced.com/portman/configurations/portman-config.adv.json

Download this openAPI definition from SwaggerHub (the unresolved YAML works fine)
Link to SwaggerHub: https://app.swaggerhub.com/apis/AdvancedComputerSoft/Contract-Breaker/1.0.0
Direct link to YAML: https://api.swaggerhub.com/apis/AdvancedComputerSoft/Contract-Breaker/1.0.0

Open the directory that has both files and run the following:

portman -c portman-config.adv.json -l AdvancedComputerSoft-Contract-Breaker-1.0.0-swagger.yaml

Portman will use the configuration file to port the openAPI definition to a Postman collection you can now import into Postman and start running tests!

Run the test

Import the Postman collection file from the tmp/converted folder that Portman created as part of the process.

Open the List Countries request and press send!

Congratulations! You ran a test and you got some failed tests. (Hopefully)
Don't worry, that was supposed to happen.

What happened?

Lets go back and look at the collection.
At first glance, it looks like any other collection. Check the tests tab and you should see something new.

This is where Portman has done its magic and created contract tests for you to run against the API. The OpenAPI definition was used to generate these tests.

You can see Postman will now test to ensure:

• The status code matches 200
• The response is application/json
• The response has a body which is an object
• The response body validates against the schema from the OpenAPI
• Bonus - The response is received within 2000ms (This is not taken from the OpenAPI definition but from the Portman configuration

All of this and you did not need to write anything. No human error, no missed tests. All of it was auto-generated.

Interpreting the responses

Postman has reported that 3 out of 5 tests have failed:

1) [GET]::/countries - Response status code is 200 | AssertionError: expected 201 to equal 200

This error was raised by the status code check as the API incorrectly returned a 201 response code instead of a 200 which it was expecting

2) [GET]::/countries - Schema is valid | AssertionError: expected data to satisfy schema but found following errors: data.data[0] should have required property 'countryId', data.data[0].population should be integer, data.data[1] should NOT have additional properties, data.data[1].countryId should be string, data.data[1].population should be integer, data.pagination should NOT have additional properties, data.pagination.offset should be integer

This error was contains all the json schema validation issues. There are missing required properties, wrong types and additionProperties (these are caused by misspelled properties or those that were not documented)

What you should find is this isn't an API that you would want to release! Contract testing lets you find where your deployed API (or your one that you are developing) doesn't match the OpenAPI definition. Catch things now before you release!

More tests

Portman has lots of other functionality to explore.

I would recommend to first look at overrides to try manipulating the collection. I started by making tests that removed the security and tried to call the API. You can stipulate which response from the OpenAPI you wish to receive. I was expecting a 401 to be returned. The Portman documentation has details of all of the functionality on offer.

Summary

APIs need to match their documentation. Whether or not you start with Design in OpenAPI or start with code, contract testing can ensure the 2 are in sync so your consumers don't get surprises and raise issues.