Dynamic Test Generation with Open API Spec 3.0

Software development has always relied on APIs. Being able to integrate with an external system has been a requirement of almost all software pretty much since the beginning.

Today, they are more prominent than ever. Everything is API driven. We even have API-first development as a thing now.

The demand is high for APIs, and as developers, we must focus our efforts on getting reliable, strongly-defined, and stable ones out the door as quickly as possible.

But what kind of responsible developer would I be if I didn't talk about testing?

An API needs to be tested. You need to verify that it does what the documentation says it does. You need to make sure you don't break backward compatibility.

Sure, you could do this all manually. But as your API grows, so do the number of test you have to write. And maintain. Eventually you're going to find yourself needing to hire someone whose only job is to keep track of all these API tests.

But what if there was a way we could automate that?

Logo for Open API Specification

Open API Specification

I've written a few times about Open API Specification 3.0 (OAS). In short, it's an industry standard way for describing modern APIs. You can learn more about it on their website.

You should be defining your APIs with this specification. It's industry standard, meaning you can hire people with the skills necessary to come in and immediately start working on your existing software.

There is community support and forums when you have questions. Improvements continuously get added to the spec. The list goes on and on.

I'm not actually here to advertise OAS to you. I just want to make sure you start building your APIs with this in mind, because the rest of what I'm about to say assumes you are.

Logo for Postman

Postman

Another topic I frequently write about, Postman is a fully featured application for API design. It provides you with API design tools, the ability to test requests, automation approaches, source control, and so much more.

But one feature that most people are unaware of is their API. The Postman API allows you to manipulate collections, environments, mock servers, and most importantly -- APIs.

If you design your APIs with Open API Spec and store them in Postman, you have the ability to easily and dynamically create a suite of tests that fully covers your API.

Image by Gerd Altmann from Pixabay

How It Works

Using the Postman API, you have the ability to load the OAS definition of your API. With your definition loaded, you can perform any number of governance validations, security tests, contract tests, even integration tests.

With each request you send in Postman, you have the ability to run custom javascript code before and after execution. The custom code can make use of Postman variables, run tests and assertions, and even set which request to run next - allowing you to build complex workflows or loop on the same request over and over again.

To build a dynamic test generator, you will use Postman to load the OAS definition of your API, load all the defined paths, build request bodies, and determine expected results.

Luckily, I've already done all this for you! You can clone the repository, import the environment and collection to your workspace, fill out the required environment variables, and run it!

What It Does

In the provided GitHub repo, the generator will perform the following actions:

• Load the OAS for the provided API/workspace
• Validate the definition is in the proper format
  • All parameters, schemas, and responses have description and example provided for every field
  • Servers are defined
  • User-defined governance settings configured in the environment
• Build schema test array
• Loop through every test in the array and submit a request to the API
• Validate status code and response body schema against OAS

As of writing this article, the main purpose of this collection is to provide tests that validate required API fields. It will create a request body that contains all the required fields for each endpoint, then create a permutation for each field. One permutation that omits a required field, and another one that passes in a blank value.

It then validates if it receives a success status code in the response if all required fields are present, or a Bad Request (400) status code if a required field is missing.

Schema Test Array

The generator is going to produce an array of tests to run. The tests are defined in the following format:

{
  "path": "", // Combines url from server and path
  "parameters": [ ], // All parameters defined at the path
  "method": "", // Path method
  "allowedRole": "", // If using role based apps, grabs the first allowable role
  "responses": [ // All expected responses for the path
    {
      "statusCode": 200, // Status code defined in responses array
      "$ref": "#/components/responses/Ok" // Grabs either referenced response or inline defined schema
    }
  ],
  "success": true, // Boolean for if this test is expected to be a success
  "description": "Has all required fields", // Generated description for what is being tested
  "body": { }, // Generated request body in JSON
  "name": "" // Generated request name with method, path, description, and success
}

Each test represents a call to your API. Postman will loop over every generated test until the array is empty, hitting your API and validating the response against what was defined in the Open API Specification.

Dynamic Requests

Everything about a request can be set programmatically in a pre-request script (the code that runs before the request executes) in Postman.

Taking things from the generated schema tests like the path, parameters, method, and body and setting those all via code allows us to have a single generic "Request" that is transformed over and over and over again.

For example, if we wanted to make a POST request to the postman echo API, we could simply do the following in the pre-request script:

pm.request.url = 'https://postman-echo.com/post';
pm.request.method = 'POST';

When the request executes, it will POST to the provided url instead of whatever was configured on the request originally. Pretty cool, right??

Dynamic Tests

Part of the generated schema tests include a success field and a responses field. If the success field is set to true, the collection knows to expect a success status code (2XX). If the field is false, it knows to expect a failure status code (specifically 400).

It will also validate the shape of the response body using ajv. The test will look at the status code, load the response schema specific to that code, and provide ajv with the schema definition and response body. A Postman test is then run to make sure validation comes back successful. Ajv is included in the Postman sandbox environment by default, so no external dependencies are required to run these tests.

At the conclusion of the validation, the collection will check to see if there are more generated schema tests to run. If there are, it will use the postman.setNextRequest command to loop back on the request and do it all over again.

Image by Mohamed Hassan from Pixabay

Why It Matters

If you've ever spent time writing and maintaining API tests, you know how burdensome it can be. It's the same thing over and over and over again with minor changes between executions.

It takes a significant amount of developer time (when they should be working on solving business problems), and is a heavily manual process. Plus, you won't get 100% coverage. Meaning every permutation of your API won't be tested.

Any time you have a manual process you introduce an opportunity for error.

People make mistakes. It doesn't matter if you have done it 1 time or 1,000. You will forget something and it will break. That is a fact of life.

So if we automate these API tests, we get multiple benefits:

1. Significant time savings for developers
2. Guaranteed to be built and tested the same way every time
3. 100% coverage

A machine isn't going to get lazy and say "this is good enough." It will do exactly what it is told to do, and build and run tests the same way and cover every permutation.

Image by Amjad Ali Ali from Pixabay

Try It Out

I encourage you to try this out and maybe even include it in your CI pipeline. I have references for both contract tests and security tests (I also wrote a blog on this one).

This is just the beginning. There are many more features to be added as we discover new use cases and prove out the effectiveness of the generators.

If you wish to contribute, please feel free to submit a pull request, add an issue, or just send me a message on Twitter. I'm more than happy to help if you run into issues or have a good idea to include.

Let me know if this works out for you! I'd love to hear your success stories.