DEV Community: Kaustav 🏳️‍🌈

APIOps: The art of mixing APIs

Kaustav 🏳️‍🌈 — Tue, 28 May 2019 15:33:23 +0000

Good APIs encapsulate domain knowledge. You may use Twilio to embed multi-channel in-app communications, Stripe to process payments and AWS to host your applications. When you use an API, you let domain experts take care of the nitty-gritties while you get your work done. Much of our daily tasks today depend on calling multiple endpoints across different services in a single workflow.

In this article we will look at what involves APIOps and how you can do it with Postman. I will cover some of the lesser known features of Postman that will upgrade your APIOps game.

API-first Ops is for everyone

APIOps is a nice little term for performing tasks that involve APIs. It becomes significant in API-first organisations. DevOps teams need to spend more time and effort working with APIs to keep systems up and running. Testers need to work with API testing tools to ensure their APIs behave as expected. APIOps is not limited to engineering functions. It can span the entirety of an organisation — in sales, marketing, devrel, success, support. You name it.

Think of it in this way: If you use APIs to get things done, you are doing APIOps.

At Postman, we’re (web)hooked to Slack

The Postman team relies a lot on Slack for internal communications. Just like any organisation of our size and scale, we use an array of tools to get different tasks done. We use JIRA and Aha! for project management, Bitbucket to host code, AWS to host our platform, Sentry to track errors, Zendesk for support, Strikedeck for customer success, Salesforce to keep the sales pipelines alive and Looker for data science. The list goes on. Each function has its own needs and the right tools for the job.

Among all these tools, Slack is the one where everyone in the organisation is present. Slack has organically emerged as the de-facto medium for consuming reports at Postman, made possible by Slack API’s webhooks. It is almost a custom now for a new team member’s first Postman Collection to push the end result to Slack. If we were to run a scan to find the most common request across all of the collections that the Postman team creates, “Post to Slack” will definitely rank the first.

Protip: Postman’s growing collection of Templates has some interesting use cases for APIOps.

Searching for “Slack” in the Templates listing inside the app shows templates that do something with Slack. You can access this by Clicking “New” → “Templates” and typing in “Slack” in the search bar.

This practice has resulted in a rich set of collections that we use internally. We polish and publish some of those as templates in Postman for you to remix and reuse. We have collections which scan AWS IAM for security audit and send those results to Slack. We have a collection that posts every new comment from Postman’s GitHub issue queue to a Slack channel for everyone to see. We have another collection that alerts us of topics that have received no reply in our Discourse community forum. Among the fun ones, we have a collection that scans for nearest Sushi places using the Google Maps API and posts the findings as a Twitter DM.

The best part? All of these are programmable. You can hack on them. Remix them. Adapt them to your use cases.

Four aspects of APIOps

Whether your task at hand is to deploy an application, react to new entries, or to find the nearest Sushi bar, there are few aspects common to workflows involving HTTP APIs:

Interact with multiple APIs. Mix and match APIs across different services to fulfil a business requirement.
Chain API calls. You will want to trigger a group of HTTP requests one after the other. You will need to share data across these requests.
Trigger conditional workflows. Change execution order of requests based on responses. For example, you may want to call a webhook 10 times if you receive an array of 10 items from a previous request.
Make end result consumable. Send and compile reports of each execution to places where they can be consumed by others. This can be a Slack channel, an Airtable base, a remote database or an analytics system.

So, if you find yourself increasingly using APIs in your daily work, how will Postman help you? Postman today packs a lot more power than the simple HTTP client that it started off as. Let us dig in.

APIOps with Postman

Doing APIOps using Postman consists of a few common tasks. At a high level, you create a collection in Postman to send requests to the endpoints that you need; use environments to manage variable data; add scripts to the requests to perform the computations; use some other endpoint to send results to respective services.

Protip: If you are new to Postman, you would want to read through environments in Postman, variable scopes, scripting, pre-request scripts, tests, running collections and monitors.

A template for TL;DR

To make things simpler, I have created a collection called “APIOps 101” that covers all the features you need to know to do the 4 steps mentioned above, with a few examples thrown in. You can directly download the APIOPS 101 collection if you have the Postman app installed by clicking the button below.

Documentation for the APIOps 101 collection

The remaining sections of the article will elaborate on the tips and tricks mentioned in this template. You can download it and follow along, or just continue reading.

Collections — running and sharing them

Postman collections let you reuse workflows. You can add and organise requests in folders and subfolders within a collection. Creating a collection is the first stage of any APIOps you may do in Postman.

When you combine your requests in a collection, you build a workflow that can interact with different APIs. Each collection typically represents one workflow. Each folder in the collection represents a unit of the workflow. Think of it as building recipes for your favourite cuisines.

Collections are executable. When you run a collection, Postman triggers the requests in the collection one by one. This is the base of the automation you need for APIOps, allowing you to mix calls across APIs and chain those calls. You can run a collection from the Postman app or in CLI (using newman) or on a schedule using Monitors.

Once you have a few collections, you can collaboratively work on them by sharing them with your team members in team workspaces. That way, your collections (and, your API workflows) never go out of sync.

Protip: For more information on collections and how to run them, you can follow guided lessons in the app. Click the “Bootcamp” icon in the bottom bar of your Postman app to start a guided lesson.

Bootcamp is an in-built learning tool built into the Postman app. The left sidebar in the image shows the collections organised in the current workspace.

Variables and their scopes

The next thing to know are variable scopes. Variables are the foundation of dynamic behaviour in Postman. You can use variables in requests, environment, collection and global scopes. In requests, variables use a double braces syntax. For example, {{url}} is a variable which you can substitute by declaring their value at any of the scopes. You can also edit them programmatically using scripts.

Variable scopes in Postman are resolved outside in. If you look at the diagram below, local variables have the highest precedence. Global variables have the least. Let’s look at them in brief:

Variable scopes in Postman

Global variables are valid for the entire Postman instance, multiple collections can share a single global variable.
Collection variables apply to all the requests in that collection. The same variable set at a collection level will override the global variable, and will be available only for the scope of that collection.
Environment variables are unique here. They override collection variables, but you can carry around an environment. Think of environments as a shareable container of variables that you can apply to multiple collections. You use pm.environment.get('key') and pm.environment.set('key', 'value') to get and set them through scripts.
Local variables apply to a single request or to a single collection run. They override environment variables. You use pm.variables.get('key') to retrieve them and pm.variables.set('key', 'value') to set them through scripts.

Use local variables to transfer data between requests

Local variables come in very handy when you are building workflows using requests. You can transfer data between requests using either environment or local variables. The upside of using local variables in this context is that you don’t have to worry about your environments bloating up, since these variables will be reset after each run. It is also a more secure alternative to share sensitive data between requests.

The typical flow here is to process the response using scripts in the “Tests” tab of a request, set the required data as a local variable using pm.variables.set() and access it in next requests.

For example, your collection may have a request to fetch an authentication key and then use that same authentication key in subsequent requests. In this case, the first request can set the variable in the “Tests” section using something like pm.variables.set('authKey', authkey) (assuming authKey is the variable in your scripts). Subsequent requests can use the {{authKey}} variable in their headers or query params, wherever needed.

Protip: Variables can only store strings, so a common practice here is to JSON.stringify() the data before setting the variable. You can then use JSON.parse() after getting the variable later on.

Another common use case for local variables is to calculate values at runtime to add dynamic behaviour in a request. For example, you can use the following script to set the structure of a request to post data to Slack in the pre-request tab:

// Prepare the body for the Slack message
const slackBody = {
  "text": "\*Title here\*",
  "mrkdwn": true,
  "attachments": [
    {
      "title": "Field name",
      "text": "Some value"
    },
    {
      "title": "Another field name",
      "text": "Some more value"
    }
  ]
};

pm.variables.set("slackBody", JSON.stringify(slackBody));

You can then set the body of the same request to {{slackBody}}. Like this:

Setting the {{slackBody}} variable in the body of the request

Use environment with Postman API to store states

You would often need to save states across collection runs. Collections runs are stateless. But, you can use environments to store data that you would need in subsequent runs. A common example of this is in processing lists of data, like parsing an RSS feed and sending new entries to Slack. You may need to remember which entries you have already processed so that you don’t trigger a notification for the existing one.

A quick and easy solution for this is to attach an environment to your collection runs and update the environment variables using pm.environment.set(). But, Monitors don't persist environment across monitor runs. This is where the Postman API comes handy.

Postman API documentation page showing the “Update Environment” endpoint.

You can use the “Update Environment” endpoint of the Postman API to update an environment with latest data. This can be a final request in your collection to save any relevant state by calling the Postman API. You can access the environment UID by switching to “Browse” view and opening the environment in the web dashboard.

A pre-request script to convert the current environment keys into the data structure that the Postman API will accept for the “Update Environment” endpoint.

This way, your collections remain portable across Monitors, in-app collection runner and newman.

Control execution order with postman.setNextRequest()

The postman.setNextRequest() method is one of the lesser used, but incredibly useful API that the Postman Sandbox exposes. You can use it in your script to branch out your execution order based on responses. This method takes one argument, the name of a request to execute after the current one.

Here are three very interesting things you can do with it.

1. Loop on the same request

You can pass the current request name as argument to the method to loop on the same request. Like, postman.setNextRequest("Process response array") where your current request name is "Process response array". You will need to ensure that you have some terminating condition to break out of the loop. For example:

// Assuming "results" in a stringified array
const results = JSON.parse(pm.environment.get("results"));

// Get the last item of the array
const currentResult = results.pop();

for (const item of currentResult) {
  // ... perform your action
}

// If there are more entries, loop on the same request.
// This is the terminating condition.
if (results.length) {
  // Stringify the remaining array items
  pm.environment.set("results", JSON.stringify(results));
  // Set the next request to the current one
  postman.setNextRequest("Post to Airtable");
}

2. Dynamically iterate on input data

Extending the previous point, you can loop on the same request to parse JSON or CSV inputs. Amber Race has written a nice article describing how you can loop on the same request to process CSV data inputs.

3. Stop executing further requests

You can stop executing next requests by passing null as argument to postman.setNextRequest(). Taking the previous example about parsing an RSS feed, you won't want to trigger a notification if there are no new items. In that case, the request that parses the RSS feed can set postman.setNextRequest(null), if there are no new entries. If the request to post to Slack is present after the current request, Postman will not stop execution and not trigger that request.

// Assuming `postsForSlack` is an array of entries,
// if there are any new posts to notify via Slack,
// call the next request, otherwise terminate collection run
if (!postsForSlack) {
  postman.setNextRequest(null);
  console.log("No new articles found.");
}

Write tests and set expectations

Finally, you would want to automate expectations. So, write as many tests as you need in the Tests tab of your requests and assert on the responses. That way, you will be able to ensure that all your responses and data structures are as you expect. You can read through the test scripts documentation, or try the quick snippets in the Postman app, or try the Bootcamp for automated testing.

Expectations in tests using pm.test() and pm.expect() methods, with snippets on the right.

With these tools in your toolkit, you will now be able to mix and mash APIs together as you wish. Go ahead and explore it! If you end up building something interesting, share those collections as templates for the rest of the community to remix. Happy APIOps-ing!

Cover Photo by Barn Images on Unsplash

Consumer-driven Contract Testing using Postman

Kaustav 🏳️‍🌈 — Fri, 24 May 2019 10:15:37 +0000

API behavior is typically described in documentation pages which list available endpoints, request data structures and expected response data structures, along with sample query and responses. These documentation are then used by people building systems that consume those APIs.

But, documentation written separately do not shield the consumers of the APIs from changes in the API. API producers may need to change the response data structure or rename an endpoint altogether to keep up with business requirements.

The onus of incorporating these changes then falls on the consumers of those APIs who have to keep checking the documentation for any changes. This model does not scale well. Consumers often end up with unexpected bugs because the response they were expecting had changed.

This is where having consumers assert on API contracts become useful. Instead of having API producers build a specification on their own, consumers of those APIs can set expectations by letting the producers know what data they want from the API.

An example scenario documenting steps that are usually taken in building consumer-driven contracts for microservices

API design then turns into a negotiation between what the consumers need and what the providers can give.

Make API contracts explicit and executable

Postman conducted a survey in 2018 which showed that the majority of business APIs are internal. These are teams building services within an organization which comes together to build a larger, user-facing product. In such contexts, ensuring that there are no differences between the expectations of the consumers and the data delivered by the API becomes a vital part of the organization’s workflow. Having explicit, shareable and executable contracts shield from much of this confusion and frustration.

Instead of documenting API behavior in a static page, producers can start by creating a specification for their API with any of the popular tools on the market, like RAML, API Blueprint, OpenAPI, Pact or Postman Collections. The last two, Pact and Postman let you implement consumer-driven contracts as a first-class concept. Pact further focuses exclusively on contract testing, while the Postman ecosystem packs a lot more power beyond that.

All of these formats allow you to specify details about your API’s behavior. They let you convey the intent of the API through endpoint names, descriptions, data types and data structure for requests and responses. They also support adding examples to each endpoint.

Once you have your specification in these formats, you can run tools that generate test code or directly send requests to a given service using the structure of endpoints described in the spec. The degree of flexibility you get varies with the toolset you use.

These specifications form the source of contracts for your service — the agreement between what the producer provides and what the consumers can expect. The underlying value of each of these toolsets is that they make your API structure explicit to those collaborating to build the service as well as to the service’s consumers.

Test contracts against multiple consumers

If you are a consumer, having such a specification handy will let you figure out exactly what data you want to fetch from the API. You can write your tests to set expectations and assert on the data that your service will use. There are two major benefits to this approach:

You can run your test suite whenever there is a change in the specification and respond to changes in the API specification proactively, and,
You can participate in the design process of the API as a consumer and let your needs be known before a contract is formalized.

In consumer-driven contract testing, the contracts are written and managed explicitly by the consumers. If the producer needs to make some changes to their service, like implementing a new feature, producers will get to know which consumers they broke just by looking at which consumer’s contract tests have failed. This frees API providers from worrying about accidentally breaking consumer apps or services every time they make some change to their service.

Independent service testing

This reduces the need to perform end-to-end testing of services. If all the contract tests pass, the API producer can be reasonably certain that their service performs as expected when integrated with other services. This is a huge respite from the complexity of doing end-to-end tests in a microservice environment.

End-to-end tests can be expensive and cumbersome to do for every change to the API. Producers and consumers can move at their own pace, have their own roadmaps and continuously deploy their changes. Consumer teams need to worry only when there is a failure in the contract tests. Coupling this with an updated documentation, generated automatically from the specification, make failures quick to detect without building up complex setups required to run end-to-end tests.

Using Postman for consumer-driven contracts

Postman has all the tools you need in place to start implementing contract testing in your organization. Postman Collections are executable specifications of an API. You can run a collection either on your local machine using the Postman app, on the command line and CI systems using newman and in the cloud using Monitors. In either case, requests in your collection are executed sequentially.

Further, you can add dynamic behavior to your requests with pre-request scripts and assert on responses with tests. The article on transitioning from manual to automated API testing in Postman runs you through details of these steps.

Couple all of these with Workspaces in Postman and you have executable, shared and collaborative contract collections ready for your entire team. You don’t need to manually share these details with your team members.

Let me run you through an example use case on how these come together to implement consumer-driven contracts.

A sample use case: Simple log retrieval service

Let’s say we need to build a hypothetical service that returns a list of log entries from some centralized log storage. This service exposes only one endpoint (for simplicity’s sake) that returns latest 5 log entries, with the name of the service that created the entry, timestamp of the entry and a description string describing the entry.

The endpoint resides at /api/v1/logs. Sending a GET request to this endpoint is supposed to return JSON data in this structure:

{
  "count": Number,
  "entries": Array[Object]
}

The entries array will contain an object for each log entry. Their data structure will look like this:

{
  "serviceName": String,
  "timestamp": Number,
  "description": String
}

Blueprint

First, we create a blueprint collection. A blueprint collection lays down the API structure. This is created by the producer of the service.

You can access the blueprint collection for this example by clicking the button below:

Sample blueprint collection

Then, we add examples for the request. I had already added an example. Examples allow such blueprint collections to describe response data. They show up in documentation generated by Postman. Here is an example of response data for the default output of this service:

{
  "count": 5,
  "entries": [
    {
      "serviceName": "foo",
      "timestamp": 1540206226229,
      "description": "Received foo request from user 100"
    },
    {
      "serviceName": "bar",
      "timestamp": 1540206226121,
      "description": "Sent email to user 99"
    },
    {
      "serviceName": "foo",
      "timestamp": 154020622502,
      "description": "Received foo request from user 10"
    },
    {
      "serviceName": "baz",
      "timestamp": 1540206223230,
      "description": "Activated user 101"
    },
    {
      "serviceName": "bar",
      "timestamp": 1540206222126,
      "description": "Error sending email to user 10"
    }
  ]
}

Each example has a name and specific request path. This is how it looks in the Postman app:

Adding example to sample blueprint collection’s request

With these in place, Postman generates web-based documentation for the blueprint automatically. Below is a screenshot of how the published documentation looks like.

Published documentation generated by Postman from the sample blueprint collection

All members who are part of the workspace in which this collection was created can view the documentation and access the collection, making it an excellent way of collaborating with other members. The service owners can then create a mock server in Postman based on this collection. The examples added in the request are sent as part of the mock server response.

Once a mock server is created in Postman, you can make requests to the mock service by using the same endpoint after the mock server URL that Postman generates for you. So, making a request to https://<mock-server-id>.pstmn.io/api/v1/logs will return the following response:

Response returned from mock server created from sample collection

Writing contract collections

Consumers of the service can build their contract collections based on the blueprint and the mock. Postman tests allow you to assert on every aspect of the response — including response headers, body and response time. So, contracts can make use of all of them and build robust tests.

For this example, let us assume there is only one consumer of this service. To keep things simple, our contract collection example will also have one request and it will assert only on the response data structure. A real-world contract would assert on the data structure as well as the data received in the response.

Consumer contract collection using tests to assert on response data

Here is the test script that the contract collection can use to test the data structure mentioned above. It uses the tv4 library, which is shipped as part of the Postman Sandbox:

// Define the schema expected in response
var responseSchema = {
    "type": "object",
    "properties": {
        "count": {
            "type": "number"
        },
        "entries": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "serverName": {
                        "type": "string"
                    },
                    "timestamp": {
                        "type": "number"
                    },
                    "description": {
                        "type": "string"
                    }
                }
            }
        }
    }
};

// Get response data as JSON
var jsonData = pm.response.json();

// Test for response data structure
pm.test('Ensure expected response structure', function() {
    var validation = tv4.validate(jsonData, responseSchema);
    pm.expect(validation).to.be.true;
});

You can download the contract collection by clicking the button below:

Note the use of the {{url}} variable placeholder in the contract collection. When the service is in its early phase, consumers can use the mock server URL to make the requests. When the service has been built, the environment variable can be switched to point to a hosted instance of the service. This way, development of the consumer apps or services can happen in parallel without remaining blocked for the upstream service to be built.

Continuous testing

Contracts need to be continuously tested to ensure they are valid over time. There are two ways this can be achieved.

If you have an existing continuous integration system: You can export collection files and environments from Postman and run them from the command-line using newman. Refer to newman’s documentation for steps to set up continuous builds for Jenkins and Travis CI. Your build pipelines can be triggered every time there is a change in the specification or a new version of the upstream service.

Along with that, you can also run contract collections using Postman Monitors. Monitors can run periodically, making it an excellent tool to get a long-term overview of contract breakages.

Read more about Continuous Testing of APIs with Postman.

Organizing contract tests

Real-world tests have setup and teardown phases. Contract tests are no different. A common use case for contract tests is to populate the system being tested with some data, then perform operations on it and finally remove those test data.

A neat pattern of emulating this in Postman is to have a Setup folder as the first folder in the collection and a Teardown folder as the last folder of your collection. All contract tests then go as folders in between the Setup and Teardown folders. This will ensure that Postman will always run the requests in the Setup folder in the beginning of a collection run, then run the requests performing actual testing, and end with running all the requests in the Teardown folder.

We make use of this pattern heavily when writing the internal contracts.

Setup and Teardown folders in an actual contract collection used by the Postman engineering team.

This helps in grouping and organizing your tests neatly by abstracting out repetitive tasks in the first and last folders. Ideally, the API producer should provide a collection with the Setup and Teardown requests. Consumers can create a duplicate of that collection and add their contract tests.

The complexity of your contract tests will depend on your business case. One of the extra benefits of writing consumer-driven contract tests is that you can easily spot when a service gets too big or has too many dependencies by looking at the number of consumer contract collections that service has.

Overall, consumer-driven contracts help keep the surface area for testing microservices and negotiating changes to a controllable size.

Happy testing!

References

Consumer-Driven Contracts: A Service Evolution Pattern by Ian Robinson
Building Microservices: Designing fine-grained systems by Sam Newman
Microservices From Day One by Cloves Carneiro Jr., Tim Schmelmer
How to be Confident That Your Microservices Can Still Communicate in Production with Pact and Docker by Harry Winser

Originally published as Consumer-driven Contract Testing using Postman in Better Practices on Oct 24, 2018.

Continuous Testing of APIs

Kaustav 🏳️‍🌈 — Mon, 06 May 2019 11:50:34 +0000

A strategy to ensure all of your APIs are tested, all the time.

Software composition is increasingly moving towards an API-driven world. If you follow what we write and talk about at Postman, you have probably come across this statement multiple times. We cannot emphasize this enough.

APIs are the building blocks of large software systems that are being built today. More and more companies are moving towards an API-first approach. Building systems as APIs is becoming a business decision, instead of only a technology decision. Ensuring stability, security, performance, and availability are high priorities in this scenario. These shifts have made API testing a first-class objective when building and shipping APIs.

At Postman, we have a unique view of this evolving landscape, thanks to our lovely community. We see API testing strategy becoming a necessary part of the API design lifecycle. Just as designing the interface for a service or product’s API is not an afterthought for an API-first organization like us, so is the need to design a resilient testing system for those APIs. My colleague, Joyce, and I have been talking about this topic since January of 2019 and it is high time that we actually write about it.

A collection of strategies

The Better Practices publication already contains quite a few articles on test automation of APIs. I have talked about how integration testing of APIs changes in the context of a service-oriented architecture. We followed that up with how Postman sets you up on the path to better automation. We have dived into consumer-driven contracts and how they can help you get over microservice dependency hell. We have also written about how snapshot testing improves reliability guarantees of APIs.

There are a few practices common to all of these solutions. At their core, they are workflows around a set of tools, which when used in a certain way solve a set of problems that API producers and consumers face regularly. Each of these testing patterns are not very useful without rigor.

APIs represent business domain requirements. They change as requirements evolve. The evolution of an API’s lifecycle needs to be in harmony with other systems that depend on it and with the systems that it depends on. APIs have to be flexible and still not break things as they grow and scale.

Need for a tight feedback loop

Building APIs with an API-first model in mind takes care of the design aspect of a distributed system. This is especially relevant in the context of microservices. These design and development need to be supported with a resilient testing system that allows you to react to changes in code or business requirements for your APIs.

You need to know when your APIs fail, why they failed, and you need a tight feedback loop to alert you as soon as possible. So, how do you go about building an API testing pipeline that satisfies all these requirements?

Your API testing pipeline needs 3 key stages:

Well-defined tests for your APIs.
Ability to run your tests on-demand and on a schedule.
Report back passes and failures to alerting and analytics systems.

Writing good tests

A testing system is as good as the tests. Everything begins with well-written tests. When it comes to testing APIs, you need to assert on the responses sent by the application.

You can test for the response data-structure, presence (or absence) of specific parameters in the response, response timing, response headers, cookies and response status. If your API is not over HTTP, these semantics may vary. That is a larger discussion, but the core things you would test in a response would still remain the same.

All of these need good test cases. These test cases should map to business requirements. These can be user stories, user journeys or end-to-end workflows. You can have test cases documented as a BDD specs or as epics/stories in your product management platform or Postman Collections.

You can pick any tool of your choice as long as you are able to author such tests, preferably collaboratively, and execute those tests when you need.

Run your tests on-demand or on schedule

This is the key to continuous testing. To get to this phase, you need to have a continuous integration (CI) pipeline in place. Assuming you have one, you will want to run some of your API tests at build time, and some of your tests on a regular schedule. The cadence will vary based on the scale of your systems and the frequency with which code changes are committed.

On-demand runs: You would run contract tests, integration tests and end-to-end tests in your build system. Code changes, merges and release flows are the typical sources that trigger build pipelines. Depending on how your build pipelines are set up, each stage of tests can be run only after the previous stages pass. The illustration below shows how Postman’s continuous deployment pipelines are set up.

An overview of Postman’s deployment pipeline. Source: Continuous Deployment with Postman Collections

Scheduled runs: You would then want to run some tests at regular intervals against your staging and production deployments to ensure everything works as expected. This is where you would run API health checks, DNS checks, security checks, and any infrastructure related checks. For example, you may test that an upstream dependency API responds with the proper data structure, or, your cloud security permissions are in place. You can even test for something as simple as the response time of your API endpoints.

Combining the power of the two: When you do both scheduled and on-demand tests on your APIs, you end up getting complete test coverage of your APIs. On-demand tests prevent broken APIs from shipping out. Scheduled runs ensure they still remain performant and retain quality after being integrated with the larger system or in production.

Analytics and alerting

Now that you have some data generated from the tests, you would want to make use of it. The 3rd stage towards a resilient API testing pipeline is connecting it with alerting and analytics systems.

Alerting systems will let your stakeholders know the moment a system fails — in this case, these should be failed tests. You can use services like Pagerduty or BigPanda here. If you use Slack at work, you can also push notifications to Slack.

Analytics systems give you a view of system health, performance, stability, resiliency, quality and agility over time. If you follow a maturity model for your services, these data will feed there as well. These data can enrich the product design and product management roadmap by giving important metrics on what works and what doesn’t. Piping this data back to product management closes that feedback loop that I mentioned earlier.

Continous Testing with Postman

The philosophy of continuous testing is tool agnostic. But, if you were to implement continuous testing in your organization with Postman, this is how you would go about it. Let’s map the three key stages of an API testing pipeline that I mentioned above to the features Postman gives you.

Writing good tests — Collections: This is where Postman Collections come in. A collection is a group of API requests that can be executed at a go. You can write tests for each request and for group of requests. Postman tells you how many of these tests pass or fail when you run the collection. You would have collections for each of your test suites. For example, you would have a collection that runs contract tests of a given consumer’s expectations against a producer, and you would have a separate collection that runs health checks for that service.
Running tests — newman and Monitors: You would integrate newman, Postman’s command-line collection runner, in your CI systems and run your collections on-demand as part of your CI pipeline. These run on your setup. On the other hand, Monitors in Postman let you schedule collection runs on pre-defined intervals. Monitors can run across multiple regions worldwide. These run on the Postman’s hosted cloud infrastructure.
Analytics and alerting — Integrations & custom requests: Postman has pre-defined integrations with external services. Your monitor runs can integrate with analytics systems and keep a tab of the runs over a period of time. There are also integrations with notification systems which can alert you whenever monitor runs fail. Beyond these, you can always include requests in your collection that push data to 3rd party services. This is useful when you are running your collections on your own infrastructure using newman.

This workflow will enable you to get the rigor that I alluded to earlier. You will have a strong pipeline and a resilient process that will ensure all the systems in your services function well with each other.

Finally, here is a rather large and poster-worthy illustration showing how these items fit in place conceptually.

Flows that can be used to perform continuous tests of APIs with Postman Monitoring and the Newman CLI tool. Image credits: Joyce Lin.

Cover Image credits: Photo by Brooke Lark on Unsplash