Consumer-driven Contract Testing using Postman

API behavior is typically described in documentation pages which list available endpoints, request data structures and expected response data structures, along with sample query and responses. These documentation are then used by people building systems that consume those APIs.

But, documentation written separately do not shield the consumers of the APIs from changes in the API. API producers may need to change the response data structure or rename an endpoint altogether to keep up with business requirements.

The onus of incorporating these changes then falls on the consumers of those APIs who have to keep checking the documentation for any changes. This model does not scale well. Consumers often end up with unexpected bugs because the response they were expecting had changed.

This is where having consumers assert on API contracts become useful. Instead of having API producers build a specification on their own, consumers of those APIs can set expectations by letting the producers know what data they want from the API.

An example scenario documenting steps that are usually taken in building consumer-driven contracts for microservices

API design then turns into a negotiation between what the consumers need and what the providers can give.

Make API contracts explicit and executable

Postman conducted a survey in 2018 which showed that the majority of business APIs are internal. These are teams building services within an organization which comes together to build a larger, user-facing product. In such contexts, ensuring that there are no differences between the expectations of the consumers and the data delivered by the API becomes a vital part of the organization’s workflow. Having explicit, shareable and executable contracts shield from much of this confusion and frustration.

Instead of documenting API behavior in a static page, producers can start by creating a specification for their API with any of the popular tools on the market, like RAML, API Blueprint, OpenAPI, Pact or Postman Collections. The last two, Pact and Postman let you implement consumer-driven contracts as a first-class concept. Pact further focuses exclusively on contract testing, while the Postman ecosystem packs a lot more power beyond that.

All of these formats allow you to specify details about your API’s behavior. They let you convey the intent of the API through endpoint names, descriptions, data types and data structure for requests and responses. They also support adding examples to each endpoint.

Once you have your specification in these formats, you can run tools that generate test code or directly send requests to a given service using the structure of endpoints described in the spec. The degree of flexibility you get varies with the toolset you use.

These specifications form the source of contracts for your service — the agreement between what the producer provides and what the consumers can expect. The underlying value of each of these toolsets is that they make your API structure explicit to those collaborating to build the service as well as to the service’s consumers.

Test contracts against multiple consumers

If you are a consumer, having such a specification handy will let you figure out exactly what data you want to fetch from the API. You can write your tests to set expectations and assert on the data that your service will use. There are two major benefits to this approach:

1. You can run your test suite whenever there is a change in the specification and respond to changes in the API specification proactively, and,
2. You can participate in the design process of the API as a consumer and let your needs be known before a contract is formalized.

In consumer-driven contract testing, the contracts are written and managed explicitly by the consumers. If the producer needs to make some changes to their service, like implementing a new feature, producers will get to know which consumers they broke just by looking at which consumer’s contract tests have failed. This frees API providers from worrying about accidentally breaking consumer apps or services every time they make some change to their service.

Independent service testing

This reduces the need to perform end-to-end testing of services. If all the contract tests pass, the API producer can be reasonably certain that their service performs as expected when integrated with other services. This is a huge respite from the complexity of doing end-to-end tests in a microservice environment.

End-to-end tests can be expensive and cumbersome to do for every change to the API. Producers and consumers can move at their own pace, have their own roadmaps and continuously deploy their changes. Consumer teams need to worry only when there is a failure in the contract tests. Coupling this with an updated documentation, generated automatically from the specification, make failures quick to detect without building up complex setups required to run end-to-end tests.

Using Postman for consumer-driven contracts

Postman has all the tools you need in place to start implementing contract testing in your organization. Postman Collections are executable specifications of an API. You can run a collection either on your local machine using the Postman app, on the command line and CI systems using newman and in the cloud using Monitors. In either case, requests in your collection are executed sequentially.

Further, you can add dynamic behavior to your requests with pre-request scripts and assert on responses with tests. The article on transitioning from manual to automated API testing in Postman runs you through details of these steps.

Couple all of these with Workspaces in Postman and you have executable, shared and collaborative contract collections ready for your entire team. You don’t need to manually share these details with your team members.

Let me run you through an example use case on how these come together to implement consumer-driven contracts.

A sample use case: Simple log retrieval service

Let’s say we need to build a hypothetical service that returns a list of log entries from some centralized log storage. This service exposes only one endpoint (for simplicity’s sake) that returns latest 5 log entries, with the name of the service that created the entry, timestamp of the entry and a description string describing the entry.

The endpoint resides at /api/v1/logs. Sending a GET request to this endpoint is supposed to return JSON data in this structure:

{
  "count": Number,
  "entries": Array[Object]
}

The entries array will contain an object for each log entry. Their data structure will look like this:

{
  "serviceName": String,
  "timestamp": Number,
  "description": String
}

Blueprint

First, we create a blueprint collection. A blueprint collection lays down the API structure. This is created by the producer of the service.

You can access the blueprint collection for this example by clicking the button below:

Sample blueprint collection

Then, we add examples for the request. I had already added an example. Examples allow such blueprint collections to describe response data. They show up in documentation generated by Postman. Here is an example of response data for the default output of this service:

{
  "count": 5,
  "entries": [
    {
      "serviceName": "foo",
      "timestamp": 1540206226229,
      "description": "Received foo request from user 100"
    },
    {
      "serviceName": "bar",
      "timestamp": 1540206226121,
      "description": "Sent email to user 99"
    },
    {
      "serviceName": "foo",
      "timestamp": 154020622502,
      "description": "Received foo request from user 10"
    },
    {
      "serviceName": "baz",
      "timestamp": 1540206223230,
      "description": "Activated user 101"
    },
    {
      "serviceName": "bar",
      "timestamp": 1540206222126,
      "description": "Error sending email to user 10"
    }
  ]
}

Each example has a name and specific request path. This is how it looks in the Postman app:

Adding example to sample blueprint collection’s request

With these in place, Postman generates web-based documentation for the blueprint automatically. Below is a screenshot of how the published documentation looks like.

Published documentation generated by Postman from the sample blueprint collection

All members who are part of the workspace in which this collection was created can view the documentation and access the collection, making it an excellent way of collaborating with other members. The service owners can then create a mock server in Postman based on this collection. The examples added in the request are sent as part of the mock server response.

Once a mock server is created in Postman, you can make requests to the mock service by using the same endpoint after the mock server URL that Postman generates for you. So, making a request to https://<mock-server-id>.pstmn.io/api/v1/logs will return the following response:

Response returned from mock server created from sample collection

Writing contract collections

Consumers of the service can build their contract collections based on the blueprint and the mock. Postman tests allow you to assert on every aspect of the response — including response headers, body and response time. So, contracts can make use of all of them and build robust tests.

For this example, let us assume there is only one consumer of this service. To keep things simple, our contract collection example will also have one request and it will assert only on the response data structure. A real-world contract would assert on the data structure as well as the data received in the response.

Consumer contract collection using tests to assert on response data

Here is the test script that the contract collection can use to test the data structure mentioned above. It uses the tv4 library, which is shipped as part of the Postman Sandbox:

// Define the schema expected in response
var responseSchema = {
    "type": "object",
    "properties": {
        "count": {
            "type": "number"
        },
        "entries": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "serverName": {
                        "type": "string"
                    },
                    "timestamp": {
                        "type": "number"
                    },
                    "description": {
                        "type": "string"
                    }
                }
            }
        }
    }
};

// Get response data as JSON
var jsonData = pm.response.json();

// Test for response data structure
pm.test('Ensure expected response structure', function() {
    var validation = tv4.validate(jsonData, responseSchema);
    pm.expect(validation).to.be.true;
});

You can download the contract collection by clicking the button below:

Note the use of the {{url}} variable placeholder in the contract collection. When the service is in its early phase, consumers can use the mock server URL to make the requests. When the service has been built, the environment variable can be switched to point to a hosted instance of the service. This way, development of the consumer apps or services can happen in parallel without remaining blocked for the upstream service to be built.

Continuous testing

Contracts need to be continuously tested to ensure they are valid over time. There are two ways this can be achieved.

If you have an existing continuous integration system: You can export collection files and environments from Postman and run them from the command-line using newman. Refer to newman’s documentation for steps to set up continuous builds for Jenkins and Travis CI. Your build pipelines can be triggered every time there is a change in the specification or a new version of the upstream service.

Along with that, you can also run contract collections using Postman Monitors. Monitors can run periodically, making it an excellent tool to get a long-term overview of contract breakages.

Read more about Continuous Testing of APIs with Postman.

Organizing contract tests

Real-world tests have setup and teardown phases. Contract tests are no different. A common use case for contract tests is to populate the system being tested with some data, then perform operations on it and finally remove those test data.

A neat pattern of emulating this in Postman is to have a Setup folder as the first folder in the collection and a Teardown folder as the last folder of your collection. All contract tests then go as folders in between the Setup and Teardown folders. This will ensure that Postman will always run the requests in the Setup folder in the beginning of a collection run, then run the requests performing actual testing, and end with running all the requests in the Teardown folder.

We make use of this pattern heavily when writing the internal contracts.

Setup and Teardown folders in an actual contract collection used by the Postman engineering team.

This helps in grouping and organizing your tests neatly by abstracting out repetitive tasks in the first and last folders. Ideally, the API producer should provide a collection with the Setup and Teardown requests. Consumers can create a duplicate of that collection and add their contract tests.

---

The complexity of your contract tests will depend on your business case. One of the extra benefits of writing consumer-driven contract tests is that you can easily spot when a service gets too big or has too many dependencies by looking at the number of consumer contract collections that service has.

Overall, consumer-driven contracts help keep the surface area for testing microservices and negotiating changes to a controllable size.

Happy testing!

---

References

• Consumer-Driven Contracts: A Service Evolution Pattern by Ian Robinson
• Building Microservices: Designing fine-grained systems by Sam Newman
• Microservices From Day One by Cloves Carneiro Jr., Tim Schmelmer
• How to be Confident That Your Microservices Can Still Communicate in Production with Pact and Docker by Harry Winser

---

Originally published as Consumer-driven Contract Testing using Postman in Better Practices on Oct 24, 2018.