Consumer Driven Contract Testing

Introduction

At NewDay, we used to create backend services and front-end web apps within the same team (or between teams who worked together closely). By growing fast and becoming more mobile oriented, the system architecture is now developing into BFF (Backends for Frontends). That means front-end developers should trust APIs created by other teams and API teams should feel safe to change their APIs.

Contracts are essential in API based architectures (Microservices or BFF), but what if the contract needs to be changed? Who should be notified of the changes and how should they be notified?

These are the issues Consumer Driven Contract Testing solves.

It’s essential that an API team (Provider) and front-end team (Consumer) agree on a contract and develop their code accordingly. In BFF model, the idea is that there are many Consumers of an API - for example mobile and web app teams. When the Provider changes part of the contract, the traditional way of emailing or putting a message in a Slack/Teams Channel is not practical - each Consumer may not know exactly which part of the contract they are using at their end.

Consumer driven contract testing is an API test approach which doesn’t test if the API is working fine, but checks if the contract is not-broken.

---

---

Example

Let's say we have an API for setting marketing consent and both mobile and web apps are using them; they are consumers. The API accepts an object with 5 Boolean fields:

{
  "email": true,
  "post": true,
  "sms": true,
  "phone": true,
  "push": true
}

The web app only needs to send four of these fields because "push" (push notifications) are not used in the web app. If the API team decides to change one of the fields, "email", "post", "sms" or "phone", they need to notify both the web app team and mobile team; but if they want to change the "push" field, they don't need to notify the web app team because they are not using that field.

In this scenario, mobile and web app teams write their own contract tests and will run these tests in their own CI pipeline.

This approach allows the consuming team to understand contract changes from the provider, and if there is an impact to them; i.e. CI of mobile / web app teams will fail if the contract tests fail, thereby, notifying themselves as consumers of contract changes from the provider.

---

Tooling

There are different tools to write Contract Tests. The most common ones are:

• Postman/Newman
• Pact
• Spring Cloud Contract

At Newday, we use Postman to write contract tests, Newman to run the tests in CI (TeamCity) and the Newman reporter to see reports in TeamCity.

---

Writing contract tests in Postman

Writing API tests in Postman is explained fully here: https://www.postman.com/use-cases/api-testing-automation/

Notice at this point we call our tests, “API tests”, this is because Consumer driven contract tests are only an approach; they are basically API tests.

Here is an example of an API test to validate the schema of our above mentioned consent API scenario. This is the test written by the web app team. Recall they are not interested in the, “push” field, they haven’t put it as part of the schema validation. By clicking, “Send” the test will run and the results are displayed in the, “Test Results” tab.

---

Save API collection

To be able to run the tests in CI/CD they need to be in a repository. First Export the API collection as a json file and push them to your desired repository. If you are using variables in your collection, you should export the variables to a separate json file. The variable file will be used when running the collection in next step.

---

Run the tests locally

The next step is to run the API tests in your local CLI and then we will do the same steps in TeamCity.

Newman is the CLI companion of Postman and it can execute Postman collections.

Install Newman globally like so:

npm install -g newman

and run the saved collection

newman run mycollection.json

add -e flag to use an environment variable file

newman run mycollection.json -e dev_environment.json

Now you can run your tests locally and see the result. hopefully all passed!

---

Run tests in CI (TeamCity)

newman-reporter-teamcity is a newman report for TeamCity. It’s optional but recommended.

Install newman and the above package globally in your build:

npm install -g newman
npm install -g newman-reporter-teamcity

Add another step to run the newman and generate the report.

I use two parameters to locate the postman collection file and postman environment file.

newman run %postman.collection.path% -e %postman.environment.path% --suppress-exit-code --reporters teamcity,cli

Now you can be sure that whenever there is a change in API the contract tests will run and the correct team can be notified if the contract is broken.

---

Big Questions

Who owns the contract tests?
The Consumer driven contract tests can be done in many ways and you should implement the approach best suited to your products and teams.

I believe the Consumer is the owner of the contract tests and Provider should not change these tests.

Where to put the tests (in this example the postman collection)?
I prefer to put them in the same repository as the API (Provider’s repository).

What stage of CI/CD process should contract tests be executed?
The idea is to notify both the Provider and Consumer when the contract is broken. I prefer to notify the Provider as soon as possible (for example a git commit hook to prevent the Provider to push breaking changes). If the Provider is sure that the changes are correct then they need to change the contract tests but that means Consumer might never get notified. So having them in the Provider’s build and setting up an automated notification for Consumer is the earliest time to get the best results.