Integrating Contract Tests into Build Pipelines with Pact Broker and Jenkins

Written by Kristine Jetzke

This blog post is part 3 of a series on consumer-driven contract testing. The other parts are: introduction, writing contract tests and lessons learned.

In the previous blog post we showed you how to write consumer-driven contract tests with Pact. In this blog post we will show you how to extend the build pipelines to automatically ensure that consumers and providers won't break. We will heavily utilize the Pact Broker for this.

The Pact Broker is an application that stores all the contracts in a database. It knows for each consumer version which provider version has - or has not - verified the contract. It can be accessed over a web interface and via a CLI. We will integrate CLI commands in our build pipelines to achieve the following goals:

• Don't deploy a provider if it would break a consumer.
• Don't deploy a consumer if it can no longer consume a provider's API.

For simplicity's sake, we initially assume that all changes get deployed straight to production (a.k.a. Continuous Deployment). We will show in a later section how the workflow can be extended if that's not the case.

The steps are:

• Preparation: Set up an instance of the Pact Broker.
• Step 1: Use it to share contracts and verification results between consumer and provider.
• Step 2: Prevent consumer from being deployed if a contract was not successfully verified by the provider. This requires that new contract verification is executed whenever the contracts change.
• Step 3: Prevent provider from being deployed if not all consumer contracts were successfully verified.

Preparation: Set up an instance of the Pact Broker

Before we introduced Pact, one of the biggest concerns was "But we need to set up a Pact Broker". While it's true that it is an additional piece of infrastructure that you need to maintain, it's actually quite robust and doesn't require much maintenance effort. The setup is quite straightforward: You need to have set up a PostgreSQL and can then use the provided docker image of the Pact Broker. Maintaining it afterwards does not require much effort. However, it does come with some additional costs, which we will cover in detail in the next part of this blog post series.

For this blog post we have prepared a docker-compose file that starts a Pact Broker, PostgreSQL and Jenkins. You can find it here. The examples below assume that the Pact Broker is running on http://pact_broker and Jenkins on http://jenkins:8080.

Note that we use Jenkins because it was the dominant tool at our client. All examples shown can easily be transferred to any other CI server.

Step 1: Use Pact Broker to share contracts and verification results

Consumer: Publish contracts to the Pact Broker

We use the Pact Maven Plugin to accomplish this. Add the plugin to the pom.xml:

<build>
 <plugins>
   <plugin>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-provider-maven_2.12</artifactId>
     <version>3.5.24</version>
     <configuration>
       <pactBrokerUrl>http://pact_broker</pactBrokerUrl>
       <projectVersion>${pact.consumer.version}</projectVersion>
       <tags>
         <tag>${pact.tag}</tag>
       </tags>
     </configuration>
   </plugin>
 </plugins>
</build>

And add a step to the build pipeline that executes the plugin:¹

    stage('Publish Pacts') {
      steps {
        sh './mvnw pact:publish 
          -Dpact.consumer.version=${GIT_COMMIT} 
          -Dpact.tag=${BRANCH_NAME}'
      }
    }

Now every time the job is run, the contract will be published to the Pact Broker with the git commit hash as application version² and it will be tagged with the current branch name. The tag will help the provider in knowing which version to verify. If you open the broker now you will see this entry on the start page:

Provider: Consume contracts from the Pact Broker and publish verification results

In the previous blog post we used the @PactFolder annotation to read the Pact files from the file system. We need to replace it with the @PactBroker annotation so that it fetches them from the Pact Broker. In addition to the broker's host, we add the tag master so that we don't verify contracts that were published by a branch build:

@PactBroker(host = "pact_broker", tags = "master")

The tests will be run as part of the regular build. However, they will not be published to the broker by default (to prevent publishing from local development machines, s. this issue.) To enable the publishing, the flag pact.verifier.publishResults needs to be set to true in the Jenkinsfile. Additionally, the provider version needs to be passed.

stage ('Build') {
  steps {
    sh './mvnw clean verify 
      -Dpact.provider.version=${GIT_COMMIT} 
      -Dpact.verifier.publishResults=true'
  }
}

Now after the job was run successfully we can see in the broker that the contract was successfully verified:

If the provider breaks the contract e.g. by removing a field, its contract unit test will fail, which will cause the entire job to fail. That the contract verification failed will be visible in the broker's UI as well:

What have we achieved? The provider build will fail if it no longer fulfills the latest contracts.

Step 2: Don't deploy consumer if contract was not verified successfully

The next step is to add a check to the consumer build pipeline, which checks that the contracts were successfully verified before it gets deployed.

Let's start with the easy case that the consumer stops consuming one field. The expected outcome is that the contract is still fulfilled and that the consumer build keeps passing.

The field is removed from the contract and the build job publishes the new contract to the Pact Broker. When we check the Pact Broker manually, we see that the latest contract was not yet verified, which means that we have no idea if we can safely deploy:

The Pact CLI (which can be downloaded here.) provides the can-i-deploy command that results in the same outcome. When we integrate it into the build pipeline:

stage('Check Pact Verifications') {
  steps {
    sh "./pact-broker can-i-deploy 
      -a messaging-app 
      -b http://pact_broker 
      -e ${GIT_COMMIT}"
  }
}

the build fails because the changed contract was never verified by the provider:

The provider tests need to run in order to make the consumer build pass. The easiest option is to trigger the provider build every time the contract changes and wait for the result to be published. The Pact Broker allows configuring webhooks that are executed if and only if a contract changes.

In order to create a webhook, go to the Pact Broker home page and click on the "Create" button in the "Webhook" column for the consumer you want to create it for. Select in the HAL browser the "NON-GET" button for "pb:create" and enter the following POST body:

{
 "consumer": {
    "name": "messaging-app"
  },
  "provider": {
    "name": "user-service"
  },
  "request": {
    "method": "POST",
    "url": "http://jenkins:8080/job/<provider-job-name>/build",
    "headers": {
      "Accept": "application/json"
    }
  },
  "events": [
    {
      "name": "contract_content_changed"
    }
  ]
}

The webhook will be triggered after the contract was published to the Pact Broker. However, the consumer build still fails because the can-i-deploy check is executed before the provider finished its build and published the results:

One option is to wait until the provider job has finished by adding the following --retry-... options to the can-i-deploy command:

sh "./pact-broker can-i-deploy 
  --retry-while-unknown=12 
  --retry-interval=10 
  -a messaging-app -b http://broker_app -e ${GIT_COMMIT}"

The build will now pass:

The main downside of this is that the consumer build will take longer because it needs to wait for the provider to finish. If feature branches are used, it's better to already verify the contract in the branch. After it is merged to master, the Pact Broker will detect that the content is the same as the one in the branch and it will know that it was already verified.

To do so we need to extend the provider job to accept the consumer's tag as a parameter and pass it to the unit test.

...
  parameters {
    string(name: 'pactConsumerTags', defaultValue: 'master')
  }

  stage ('Build') {
   ...
     sh "./mvnw clean verify ... 
       -Dpactbroker.tags=${params.pactConsumerTags}"
   ...
  }
}

And the webhook URL needs to be changed to pass the parameter: http://jenkins:8080/job/<provider-job-name>/buildWithParameters?pactConsumerTags=${pactbroker.consumerVersionTags}

The variable ${pactbroker.consumerVersionTags} contains the tags assigned to the contract for which the webhook is triggered and is just one of many variables that can be passed in the webhook.

Now every branch that is being built for the consumer will be pre-verified before it's merged to master.

Example:

Contract from branch remove-field was successfully verified by provider version 9968e19 (number 139).

After the branch is merged to master, the broker detects that the pact's content did not change and thus the verification result is the same as for the branch. It is not necessary to run the tests again.

What have we achieved? The consumer will not get deployed if a contract was not verified at all or not verified successfully by the provider.

Step 3: Don't deploy provider if not all contracts were verified successfully

No additional step is required if the provider tests are part of the main build. However, the main build job might do more things than just building e.g. run static code checks, run long-running integration tests, increase a version… You don't want to perform all these steps each time you run the contract tests.

A better approach is

• Have a separate job that only runs the contract tests.
• Exclude the contract tests from the main build job.
• Excute the can-i-deploy command before deploying the provider to production.

What have we achieved? Services won't get deployed if they would break another service or would break themselves. However, this only holds true if we do continuous deployment. The next section will show how the same results can be achieved if production deployment is a separate step.

Additional steps without Continuous Deployment

Until now we've assumed that all changes go straight through to production i.e. the pact tagged with master is the one that's actually in production (except maybe for a small time frame when the deployment takes place.) If that's not the case, the approach described above is not sufficient. We will illustrate why using two examples and show which steps are necessary to ensure that no one deploys breaking changes.

Example 1: Consumer and provider agree to remove a field

• The provider team removes the field from their API, pushes their changes to master and the master gets build on the CI server. When they try to deploy, the can-i-deploy command fails because the current version does not yet exist on the Pact Broker since it's only created when the contract tests are run.
• The consumer team updates their code to no longer require the field. They remove the field from the contract, push all their changes to master and the master gets build on the CI server. The pact gets published to the broker and is tagged with master. The consumer does not get deployed to production yet.
• Publishing the pact triggers the webhook to execute the provider contract tests. The tests fetch the current pact tagged with master from the broker and run them successfully.
• The consumer team informs the provider team that they have removed the field (but forget to mention that they didn't deploy yet). The provider team starts the production deployment job again. The call to the can-i-deploy command passes this time because the contract test triggered by the webhook passed. Thus, the provider gets deployed to production.

Boom! Now we've caused an incident. The production consumer can no longer consume the provider's API because it still expects the field to exist.

We need to prevent the provider from deploying anything to production if it's not compliant with the consumer production version.

This requires the following steps:

1. The consumer declares which version is currently deployed to production.
2. The provider needs to run the contract tests against this version of the consumer's contract.
3. The provider needs to ensure that it could successfully verify all production versions before deploying to production.

This can be implemented as follows:

1. A new tag needs to be added to the consumer version e.g. prod. The tag can be applied with the create-version-tag command of the CLI. We usually add the tag after the deployment was successful, but you can also add it beforehand if your deployment is very unlikely to fail.
2. The provider needs to verify the prod tag as well by passing it to the test: -Dpactbroker.tags=prod,${params.pactConsumerTags}
3. The can-i-deploy call needs to be changed to check that it's safe to deploy to production. This is done by adding --to prod at the end of the command. What does it change? Previously the can-i-deploy command got all the verification results, and selected the latest entries with the other participants. Now it instead selects the latest entries tagged with prod. If there is a newer entry from master, it will be ignored.

With this setup in place, the provider's contract test job verifies the following consumer versions:

• master: Passes, because field was already removed
• prod: Fails, because it still expects the removed field

The can-i-deploy call fails and prevents the provider from deploying the breaking change:

After the consumer was deployed to production, the prod tag is moved to the same version as the master tag and the verification result is taken over:

The can-i-deploy call succeeds now and the provider can deploy:

Example 2: Consumer and provider agree to add a field

• The provider team adds the new field. They push their changes to master and the master gets build on the CI server.
• The consumer team updates their code to start consuming the new field. They add the new field to the contract, push all their changes to master and the master gets build on the CI server. The pact gets published to the broker and is tagged with master.
• Publishing the pact triggers the webhook to execute the provider contract tests. The tests checkout the provider's master version (that already includes the new field), fetch the consumer's pact tagged with master from the broker and run them successfully. They also verify the pacts tagged with prod. They also run successfully because they don't expect the new field yet.
• The consumer team starts the production deployment job. The call to the can-i-deploy command passes because the contract test triggered by the webhook passed. Thus, the consumer gets deployed to production.

And we've caused another incident. The production consumer can no longer consume the provider's API because it already expects the new field, but the provider does not yet provide it.

To prevent this from happening we need to perform the same steps as before - just to the respective other participant:

• Tag the provider with a prod tag after it got deployed
• Add --to prod to the consumer's can-i-deploy call.

Now the consumer deployment fails because the version about to be deployed (tagged with master) was not yet verified against the provider's prod version:

And thus the call to can-i-deploy fails and prevents the consumer from deploying a breaking change:

After the provider was deployed, the provider's prod tag is moved to its current version and the verification result is taken over:

Now the call to can-i-deploy succeeds and the consumer can deploy:

The full deployment Jenkinsfile looks like this (similar for consumer and provider, just the participant's name given with -a option differs):

stage('Check Pact Verifications') {
  steps {
    sh "./pact-broker can-i-deploy 
      -a messaging-app 
      -b http://pact_broker 
      -e ${GIT_COMMIT} 
      --to prod"
  }
}
stage('Deploy') {
  steps {
    echo "Deploying to prod now..."
  }
}
stage('Tag Pact') {
  steps {
    sh "./pact-broker create-version-tag 
      -a messaging-app 
      -b http://pact_broker 
      -e ${GIT_COMMIT} 
      -t prod"
 }
}

Note: It's important to have unique versions. Tags get attached to a specific version (with -e ${GIT_COMMIT}). If the version number always remains the same, each version has the same tags attached and thus it is impossible to find the latest version with a specific tag.

What have we achieved? No matter how services get deployed - they won't get deployed if they would break another service or would break themselves. However, there is one remaining set of issues. Sometimes deployments are blocked until another participant is deployed. The next section describes how to prevent this.

Don't block deployments unnecessarily

In the first example of the previous section, the provider committed a breaking change to master and could not deploy to production until the consumer was deployed to production. While this is much better than accidentally deploying a breaking change, it comes with a new problem: What if the provider suddenly needs to make a very urgent bugfix? They can either rollback their breaking change or ask the consumer team to deploy. The latter option might just not be possible e.g. because the team is not available, they have issues with their deployment pipeline or they simply haven't made the required changes yet.

The same is true for the second example, where the consumer team wanted to use a new provider feature and had to wait for the provider team to deploy. A third blocking situation arises when the consumer starts consuming a field that the provider already provides.

We will go through each case separately and show how to prevent deployments from being blocked unnecessarily.

Case 1: Provider makes breaking change

The provider can only deploy a breaking change to production after all consumers have adapted their code and got deployed. Thus, the breaking change should stay in a branch until all consumers have deployed. How does the provider know that the branch can get merged and deployed? By executing the contract tests against all consumer's prod versions. If they pass, it's safe to merge (and deploy).

One way to achieve this is by using prod as the default tag in the @PactBrokerannotation: @PactBroker(..., tags = "${pactbroker.tags:prod}")

Note that it's important to only run against prod and not against master during the regular build. Otherwise, the consumer can block the provider build by making a breaking change to their contract and tagging it with master. This is not desired behavior. If a consumer makes a change to their contract that the provider can not (yet) fulfil, the consumer pipeline's should be blocked and not the provider's.

Case 2: Consumer requires new feature

The consumer can only start consuming a new feature after the provider has deployed it to production. Thus, the code to consume the new feature needs to stay in a branch until the provider has deployed it to production. How does the consumer know that it can merge the branch? By running the provider tests for each branch (triggered via webhook) as explained in the beginning of this article and by executing can-i-deploy in the branch. If the command returns successfully, it's safe to merge (and deploy).

Related to this: Publishing the contracts for the branch also helps the provider in developing new API features. The provider can develop new features "test-driven" i.e. the contract test for a new contract will fail until the provider has implemented the new feature and it matches the consumer's requirements given in the contract. We have not yet found a way to automate this. Our best idea so far is to hard-code the consumer's branch name into the provider's contract test and change it back before merge.

Case 3: Consumer starts using existing field

There is in fact one last case that blocked us in the beginning significantly more often than the others: The consumer started consuming a field that the provider already provided and thus the provider actually didn't need to be changed or deployed.

The consumer adds the new field to the contract and the new contract gets published to the broker and tagged with master. We - as people - know that we'll be able to consume this field from the prod provider but the broker does not know. The broker only knows that the current master provider (be2f441) fulfills the contract:

Since this version is not yet deployed to production, the can-i-deploy will prevent the consumer from deploying.

Again, one solution would be to deploy the provider but this might not be possible as already explained in the beginning of this section. Instead we can change the provider's contract verification job to run the current prod version's test instead of the master's.

Two changes are necessary:

1. Find out which version is currently deployed to prod.
2. Checkout this version instead of master.

The CLI provides the describe-version command which can be used to show the latest version tagged with prod:

./pact-broker describe-version -a user-service -b http://pact_broker -l prod

NUMBER                                | TAGS
-----------------------------------------|-----
d87dbfdf126f7d46eb5f7faa3923e753627ff405 | prod

The returned version number is the git hash which can be used to checkout the prod version. Now the consumer's master will get verified against the provider's prod version and the can-i-deploy command passes.

stage ('Get Latest Prod Version From Pact Broker') {
  steps {
    script {
      env.PROD_VERSION = sh(
        script: "./pact-broker describe-version 
          -a user-service -b http://pact_broker 
          -l prod | tail -1 | cut -f 1 -d \\|", 
        returnStdout: true).trim()
      }
    }
    echo "Current prod version: " + PROD_VERSION
  }
}    
stage("Checkout Latest Prod Version") {
  steps {
    sh "git checkout ${PROD_VERSION}"
  }
}
stage ('Run Contract Tests') {
  steps {
    sh "./mvnw clean test 
      -Dpact.provider.version=${PROD_VERSION} ... "
    }
  }
}

Conclusion

In this blog post we have explained the steps required to include consumer-driven contract tests in the build pipelines to prevent breaking changes. In the beginning the sheer amount of additional steps might seem overwhelming. It takes some time for everyone involved to understand the workflow. ³ Additionally, alignment between the teams is necessary in order to agree on tag naming and the jobs to execute via webhook triggers.

But once everything is in place, the Pact Broker is an awesome tool that does not create a lot of extra work and manages the whole workflow for you. It does however have some drawbacks, which will be covered in the next part of this blog post series.

Note that you can find all Jenkinsfiles mentioned in this blog post in the GitHub repo. It also contains a docker-compose file that starts the Pact Broker and Jenkins and initializes the Jenkins with all mentioned build jobs.

Notes

---

1. The environment variable GIT_COMMIT is automatically available in Jenkins. BRANCH_NAME is available if you use a multi-branch pipeline. Otherwise you can calculate it from the available environment variable GIT_BRANCH.↩ ↩

2. Note that it's important that this version changes every time a new application version is created, s. Pact documentation for more background on versioning.↩ ↩

3. A good additional read is the excellent article The steps to reaching pact nirvana from the Pact team. Once you have understood the steps and their purpose it's usually sufficient to follow the set up checklist provided in the Pact Broker wiki.↩ ↩