DEV Community: ITFabers

Schemathesis vs Postman for generated API tests

ITFabers — Mon, 09 Nov 2026 19:44:12 +0000

I get asked which one to use, and I think the question hides a false choice. Schemathesis and Postman are not two versions of the same tool. One generates requests from your schema while the test runs. The other stores requests you can send, and what you assert on them is up to you. Pick the wrong one for the job and you spend a week building the thing you did not need.

Full disclosure: I build apitestgen, which generates test files from a spec. I have read the docs for both tools below and linked them so you can check me.

What Schemathesis actually does

Schemathesis reads an OpenAPI or GraphQL schema, or a gRPC service descriptor, and generates requests from it at run time. You do not write one test per endpoint. You point it at the schema, and it produces cases, including malformed ones, then checks the responses against what the schema promises.

The docs put it this way: it "automatically generates property-based tests from OpenAPI and GraphQL schemas or gRPC service descriptors, chains operations into realistic workflows, and exercises the edge cases that break your API" (Schemathesis docs).

In pytest the entry point is @schema.parametrize(), which turns one test function into one test per operation, and case.call_and_validate() sends each generated request and runs the built-in checks. Stateful tests are a separate class built with schema.as_state_machine() (pytest tutorial). The CLI runs the full set, stateful testing included, in one command.

The output is failures, not a suite you commit. When something breaks you get a minimal curl or grpcurl command that reproduces it. That is the point. It is a bug finder that runs against a live server.

What Postman actually does

Postman starts from requests. If you import an OpenAPI file, Postman generates "a collection with folders, requests, and response examples based on the specification." Spec Hub supports OpenAPI 2.0, 3.0, and 3.1, and you can import a spec as a collection or as a spec together with a collection (Postman import docs).

The generated collection is scaffolding. The assertions are post-response scripts you add to a request, a folder, or the whole collection. Postman runs them in the Postman Sandbox and gives you pm.test and Chai assertions, so a status check looks like this (Postman test scripts):

pm.test("Status test", function () {
    pm.response.to.have.status(200);
});

Collection and folder scripts run after every request in that scope, which is how people reuse auth and response checks. You can run the collection in the Collection Runner or from the Postman CLI in CI.

So the artefact is a collection that lives in Postman, or exports as JSON, plus the scripts you wrote. The schema import saves typing. It does not decide what a correct response looks like for you.

Side by side

	Where tests come from	What you get	What fails the run
Schemathesis	The schema, generated at run time	Requests plus built-in schema checks	Server errors, undocumented status codes, schema violations
Postman	An imported spec plus your scripts	A collection of requests with assertion scripts	Whatever your `pm.test` assertions check
apitestgen	A spec file you paste	pytest, Jest, or Postman test files	Generated checks for the happy path, auth failures, missing params, bad types, response schema

Where each one wins

Schemathesis is the right pick when you want to find bugs you did not think of, when the API is already running somewhere reachable, when your tests live in pytest, and when multi-step workflows matter. It can carry real identifiers from one operation to the next, so "create a booking, then fetch it" is one flow rather than two disconnected requests.

Postman is the right pick when the team already lives in Postman, when you want a shared collection for manual exploration with assertions attached, when you need to hand a collection to someone who does not run Python, or when you want collection runs in CI through the Postman CLI.

Neither answers a different question: "give me test files I can read, edit, and commit, that cover the boring cases." That is the gap I built into apitestgen. Paste an OpenAPI, Swagger, or Postman file and you get pytest, Jest, or Postman tests covering the happy path, auth failures, missing parameters, wrong types, and response-schema checks. The files are yours to edit and run wherever your CI already runs.

What I would do on a new API

Run Schemathesis against staging first. Server errors and undocumented status codes are the cheapest bugs to find, and they do not need a test suite.
Generate a file-based suite for the checks you want to keep and review. Read the assertions before you commit them. A generated test you do not understand is a liability.
Reach for Postman when a human needs to click through the same requests, or when the team already shares collections there.

If you already have a Postman collection and no tests in the repo, the reverse order works too. Import the collection somewhere that writes code, then keep the parts that still match your API.

The honest summary

Schemathesis is a schema-aware fuzzer. Postman is a request workspace with a scripting layer. Both are good at what they do, and neither is a substitute for the other. Only one of them gives you a test file on disk. If that is the thing you actually need, say so when you are choosing tools, because neither tool will admit the gap for you.

Sources:

If you want this done for you

If you have an OpenAPI 3.x spec and want a runnable contract-test suite in 48 hours, that is $149. You send the spec, YAML or JSON, public URL or shared file. Within 48 hours of purchase you get runnable pytest, Jest or Newman test files, a README with run instructions, and the list of spec gaps I found while generating them. If I miss the 48 hours, you get a refund.

Checkout: https://itfabers.lemonsqueezy.com/checkout/buy/d1cbe20e-1358-466b-8af2-7cbfbff88636

A status-200 check is not a contract test

ITFabers — Mon, 09 Nov 2026 19:43:02 +0000

A test that checks for HTTP 200 and stops there is not testing your API. It is testing that the server is awake.

I have been building a small tool that turns OpenAPI specs into runnable test suites. The hard part was never generating the code. It was deciding what a generated test is allowed to assert.

Here is the trap I keep running into in real public specs: every response is described as type: object, with no properties and no required fields. A generator can write a test against that, watch it return 200, and call it a pass. The suite is green and the contract is untested.

Two fixes, both in the spec:

Give response schemas real properties and required fields. If the body can be anything, nothing can fail.
Document the error responses. When 400, 401 and 404 have no shapes, the negative tests have nothing to check.

A generated suite is a mirror of the spec. A vague spec produces vague tests. So before writing a single test, I read the spec for the holes. That step is boring and it is most of the value.

Why a status code assertion is not enough

A status code assertion passes on an empty body too. Adding a schema check on the response turns a lot of those green tests red, which is the point. If required is missing, the generator has no reason to send the field, and the test proves less than it looks like it does.

Two more things I check before generating anything:

401 versus 403. A missing token should be 401. A valid token without permission should be 403. APIs that return 401 for both hide authorization bugs until production.
Pagination edges: last page, empty page, an expired cursor, a limit larger than the server allows. Cheap to test, expensive to find in production.

None of this is novel. It is the work a fast generator will skip if you let it.

If you want a suite built

If you have an OpenAPI spec and want a runnable contract-test suite in 48 hours, that is what I do for $149. You send an OpenAPI 3.x file, YAML or JSON, public URL or shared file. You get runnable test files (pytest, Jest or Newman), a README with run instructions, and a short list of the spec gaps I found along the way. If I miss the 48 hours, you get a refund.

Details and checkout: https://itfabers.lemonsqueezy.com/checkout/buy/d1cbe20e-1358-466b-8af2-7cbfbff88636

If you want to check fit first, send the spec link in a comment.

I am an independent developer in Yerevan. I am not going to pretend I have a wall of logos. This is the offer, and the spec-gap audit is the part I think is worth paying for.

Your API test suite is green and your contract is still untested

ITFabers — Mon, 09 Nov 2026 19:17:35 +0000