Automatically Keep Postman Collections In Sync With OpenAPI Specs

This post will show you how to take an OpenAPI Spec file and automatically generate a Postman Collection of API tests with minimal configuration. This is especially helpful for a CI/CD pipeline that runs automated API tests after a deployment.

My previous post automatically generated an OpenAPI spec from a CDK construct. See that post here:

https://matt.martz.codes/openapi-specs-from-cdk-stack-without-deploying-first

Since this takes an OpenAPI spec as input it doesn't matter if you're automatically generating the OpenAPI spec, manually writing yaml (🤢) or exporting it from something that's already deployed.

Since I've already automated the OpenAPI spec generation I'm going to extend my last post's code See the diff here

For some minor housekeeping, I bumped the CDK dependencies. The old version of this project's @aws-cdk/aws-lambda-nodejs dependency still used parcel to bundle its lambdas. Parcel was PAINFULLY slow, so I'm glad the CDK team switched to esbuild. It is extremely fast, assuming you have it installed in your project. If you don't have it installed it uses a docker image, which somewhat negates the speed benefits.

The rest of the dependencies are straight from postman:

• openapi-to-postmanv2 converts OpenAPI specs to Postman Collections
• postman-collection (docs) is an SDK to help generate/customize Postman Collections
• newman runs the customized postman spec.

Using this project, a CI pipeline flow for API testing would look like:

1. Run the unit tests to generate the OpenAPI Spec
2. Convert the OpenAPI Spec to a Postman Collection
3. Extend the Postman Collection with specific endpoint tests
4. Use the extended Postman Collection to test the API

🚀 Note: None of this has us editing json/yaml by hand! Everything is generated via code! 🚀

Code Walkthrough

src/newman/newman.ts is the main file.

This file uses a list of tests that it iterates through to generate the customized postman collection. It uses the test's path and method to find the corresponding test from the OpenAPI-generated file. If that doesn't exist (for example if you rename an endpoint) the generator will return an error:

> ts-node src/newman/newman.ts

Error: Misconfigured test: Advanced Hello Check - /other/:hello/advanced

This would break a PR's CI prior to getting merged anywhere.

Postman tests are a little unnerving in that they just execute a list of strings as javascript for their test. The benefit of this is that all we need to do is have a bunch of string generators to produce the tests we want to run.

Each endpoint test we run is defined by the ApiTestConfig interface:

export interface ApiTestConfig {
  name: string;
  description?: string;
  path: string;
  method: string;
  status: number;
  pathParams: Record<string, string>;
  body?: any;
  responseValues?: Record<string, string>;
}

For example:

{
    name: 'Basic Hello Check',
    path: '/example/:hello/basic',
    method: 'POST',
    pathParams: {
    hello: 'heyCheck',
    },
    status: 200,
    responseValues: {
    message: 'Hello heyCheck.  How many times have you some string?  12 times.',
    },
    body: {
    someString: 'some string',
    someNumber: 12,
    },
}

This names the test, uses the /example/:hello/basic POST endpoint defined here, makes sure it returns a 200 and the message ultimately matches what the inputs are. This corresponds to the test output of:

Basic Hello Check
  POST https://w9tfo30qa2.execute-api.us-east-1.amazonaws.com/prod/example/heyCheck/basic [200 OK, 566B, 655ms]
  ┌
  │ { message: 'Hello heyCheck.  How many times have you some string?  12 times.' }
  └
  ✓  Status code is 200
  ✓  Value at message is 'Hello heyCheck.  How many times have you some string?  12 times.'

The test interface's status and response values get turned into the string arrays used by newman/postman.. The src/newman/checks.ts file has the generators for those tests, and it's easy to see how these could be extended. For example you could check specific items in a json response using dot notation since they are simply evaluated directly. 🤦‍♂️

Newman Tests

Since the previous project automatically generated the OpenAPI spec, we just need to add two more commands to test the deployed API:

npm run test
npm run newman:convert
npm run newman:test

😍 Notice how much faster npm run test is thanks to esbuild? 😍

But what if we don't want to deploy the API first? No problem! Instead of npm run newman:test use npm run newman:pr. This will make sure the extended Postman Collection is in sync with the generated OpenAPI spec. This only protects against basic endpoint changes such as renames. I talk about some strategies at the bottom of this article for how to potentially make this more robust.

If the full API test is run, the output will look like this:

Import into Postman

👀 But wait... THERES MORE!?! 👀

The newman library also has the option of exporting the new collection with all the tests.

You can take this file and import it into actual Postman and run the tests / hit the APIs in there:

Depending on your level of customization... this file could contain some secrets so it's .gitignore-ed

Possible Next Steps

This doesn't fully automate our API testing (yet), but it does ensure that our API tests are in sync with the OpenAPI Spec file AND we don't have to touch a line of yaml... which is at least an improvement.

There's a lot that could be done to extend this. For example:

• Create the OpenAPI JSON Schema from https://www.npmjs.com/package/class-validator-jsonschema and extend it with examples and more validations
• Write a decorator or util that you can add to jest tests that registers an API test. This way you can make sure that that API test payload/response bodies are in-sync with what's tested in the unit tests. This should keep things more in-sync.

What other ways could this be extended?