DEV Community: Anton Kirilchuk

How I fuzz my API with one command using Schemathesis (and let Claude triage the bugs)

Anton Kirilchuk — Sat, 27 Jun 2026 20:09:42 +0000

Manual API tests check what you expect to happen. The bugs that take down production are the ones nobody thought to write a test for: an empty string where the code assumed a value, a negative number in a field that only ever saw positives, an emoji in a date.

You can sit and brainstorm broken inputs by hand. You will write ten of them and get bored. The three-hundredth one, the one that actually crashes the server, you will never reach.

Schemathesis writes those for you. You give it your OpenAPI schema, it generates thousands of requests on its own, and it hammers every endpoint looking for the ones that break. Here is how I run it, what it actually finds, and how I deal with the flood of results it produces.

What Schemathesis actually is

Schemathesis is a Python tool built on top of Hypothesis, the property-based testing library. Property-based means you do not write example inputs. You describe the shape of valid data, and the engine generates hundreds of cases that fit (and deliberately break) that shape.

Your OpenAPI schema already describes that shape: every endpoint, every parameter, every type. Schemathesis reads it and turns it into a test generator. No assertions to maintain by hand. When the schema changes, the tests change with it.

Install it:

pip install schemathesis

The first run

One command, pointed at your schema:

st run http://localhost:8000/openapi.json

That is it. It reads the schema, then fires thousands of requests at every endpoint: valid ones, and ones deliberately malformed. By default it runs in all mode, mixing valid and invalid data. You can push it harder with negative-only generation:

st run --mode=negative http://localhost:8000/openapi.json

What it catches

Out of the box, Schemathesis runs a set of checks against every response. Three of them find the bugs that matter most.

The silent 2xx. You send a malformed payload expecting a 400 Bad Request. Instead the server replies 200 OK and quietly accepts the garbage. The bad data either got saved to your database or vanished without a trace. Both are bugs, and neither shows up in a happy-path test.

The unexpected 5xx. Fuzzing string boundaries (huge Unicode strings, null bytes, integers at the edge of their range) drags raw stack traces and unhandled exceptions out of code that unit tests walk right past.

The contract violation. The server returns a status code, or a response body, that its own OpenAPI schema never documented. The status_code_conformance and response_schema_conformance checks catch the gap between what the docs promise and what the API does.

You can run a focused subset:

st run openapi.yaml --checks not_a_server_error,response_schema_conformance

Reading a failure

This is the part that makes Schemathesis worth it. Every failure comes with the exact curl command to reproduce it. No guessing which of the thousand requests broke things.

A real example from a booking API:

- Server error

- Undocumented HTTP status code

    Received: 500
    Documented: 200, 422

[500] Internal Server Error

Reproduce with:

    curl -X POST -H 'Content-Type: application/json' \
      -d '{"guest_name": "00", "nights": 1, "room_type": ""}' \
      http://127.0.0.1:8080/bookings

A guest name of "00" and an empty room_type crash POST /bookings with a 500. Copy the command, paste it in a terminal, and the bug reproduces every time. Hand that line straight to the developer who owns the endpoint.

The noise problem

Run Schemathesis for the first time and it will surface a hundred findings. Do not panic, and do not trust all of them.

The negative_data_rejection check is a known source of false positives. Schemathesis sends a malformed request (say, an empty query parameter) and expects a 4xx. But many frameworks, FastAPI among them, simply ignore the empty parameter and return 200 OK. Schemathesis flags that as a failure, even though nothing is actually broken.

So the first pass is mostly triage: separate the real crashes from the protocol nitpicks. Doing that by hand across a hundred findings is an afternoon gone.

Letting Claude triage the flood

This is where I plug in an AI layer. I take the full Schemathesis report and hand it to Claude with a simple instruction: group the findings, drop the false positives like the empty-parameter ones, and rank what is left by severity.

It collapses the duplicates (one root cause often shows up in twenty different request shapes), strips the framework-level noise, and hands back a short list: these three are real server crashes, fix them first; the rest is the negative_data_rejection pattern, safe to ignore.

What was an afternoon of manual sorting becomes a couple of minutes. The machine generates the attacks, the AI clears the noise, and I am left fixing actual bugs instead of reading stack traces.

Wiring it into CI

A fuzzer in CI is dangerous if you let it stay random. A pipeline that passes today and fails tomorrow, for no reason other than the generator stumbling onto a new case, trains everyone to ignore it.

Fix the seed so runs are reproducible:

uvx schemathesis run --generation-deterministic http://localhost:8000/openapi.json

uvx runs it in an isolated, throwaway environment with no dependency setup, which is exactly what you want in CI. The deterministic flag means the same code gives the same result every time. Save the wild, fully-random run for a nightly cron job, where a new failure is a signal to investigate rather than a broken build.

Schemathesis also emits JUnit XML, so the results drop straight into the dashboards your team already reads.

Where it fits

If you have used Dredd, Schemathesis is the step past it. Dredd validates that your API matches the examples in your docs. It does not generate negative tests or fuzz. Schemathesis actively hunts for inputs that break things.

Compared to writing assertions by hand in pytest or Postman, the trade is clear: those tools make you write and maintain every check. Schemathesis treats the OpenAPI spec as the single source of truth. Update the spec, and the test surface updates with it.

The short version

Manual tests cover the inputs you imagined. Schemathesis covers the ones you did not, generates a reproducible curl for every bug it finds, and an AI pass turns the noisy output into a clean fix list. One command to find the holes, a deterministic seed to keep CI honest, and a nightly chaos run to keep looking.

If you test APIs, point it at your schema once and see what falls out. The first run is usually uncomfortable, in a useful way.

Drop your stack in the comments and I will tell you where to start.

Mock any API response in Postman (and let AI build the collection)

Anton Kirilchuk — Sun, 21 Jun 2026 21:06:29 +0000

The hardest part of frontend testing isn't writing the test. It's getting the backend to return the exact response you need: a 500, an empty list, a malformed payload, right when you want it. On a live server that's painful, and sometimes impossible.

Here's the workflow I use instead. I make the app receive any response I want, without touching the backend at all. Postman mock servers do the heavy lifting, and an AI fills them with data.

The idea: a fake backend that looks real

Your frontend doesn't know where the JSON comes from. It calls a URL and trusts whatever comes back. So you point it at a mock server: a fake address that returns responses you defined in advance. Same endpoints, same shapes, zero real backend.

Why not just DevTools or a proxy?

For a quick one-off, browser tools are fine:

Chrome DevTools, Local Overrides rewrite a response right in the browser.
Charles / Requestly / mitmproxy intercept and swap responses on the fly.

But the override dies when you close the tab, it lives only on your machine, and you can't hand it to a teammate or a CI pipeline. For anything beyond a single check, you want a real, persistent stand.

Step 1: build the collection

In Postman you already (or soon will) have a collection: the same endpoints your real API exposes, with method, URL, body and headers.

Step 2: add example responses

For each request, hit the three dots and pick Add example. An example is a saved response: a status code plus a body. You write it by hand.

Hang several examples on the same endpoint to cover every case:

200 success
404 not found
[] empty list
500 server error

Step 3: spin up the mock server

Three dots on the collection, then Mock collection. Postman gives you an address:

https://xxxx.mock.pstmn.io

Step 4: point the frontend at it

Swap the base URL in your app config:

- const API = "https://api.production.com"
+ const API = "https://xxxx.mock.pstmn.io"

That's it. Your frontend now talks to the mock and never notices the difference.

One endpoint, every scenario

Here's where it gets good. You hung 200, 404 and 500 on the same endpoint. Which one does the mock return? It decides by a request header:

x-mock-response-name: order not found

Send that header from your automated test, and a single endpoint runs through every scenario without touching the server or the data:

await fetch(`${API}/orders/42`, {
  headers: { "x-mock-response-name": "server error" }
})
// the mock returns your 500 example

Responses don't have to be static

Postman supports dynamic variables right inside the example body:

{
  "id": "{{$randomInt}}",
  "name": "{{$randomFullName}}",
  "email": "{{$randomEmail}}"
}

Every call comes back with different data instead of one hardcoded blob, so you catch the bugs that only surface on unexpected input.

The part that saves the most time

Writing dozens of example responses by hand is the boring tax on all of this. So I don't.

I hand the whole collection to Claude through the Postman MCP, and it generates the example responses for every endpoint (success, edge cases, malformed payloads) and wires up the mock. I'm not asking it to write code. I'm asking it to assemble a working test stand while I drink my coffee.

That's the shift: AI stopped being "write me this function" and became "build me the tool".

How do you handle this on your team: Postman mocks, a standalone mock service, or still waiting on the backend?

How I stopped building Postman collections by hand (Claude + Postman MCP)

Anton Kirilchuk — Sat, 20 Jun 2026 10:30:43 +0000

If you test backend services, you know the
ritual. A new service shows up, you need to poke its endpoints, so you open Postman and start building a collection request by request. Method, URL, headers, auth, request body, fix the folder structure, name everything so future-you can read it. On a service with a few dozen endpoints, that's an hour or two gone before you send a single request.

I haven't done that in months.

Now I hand the API definition to Claude, and it builds the whole collection through the Postman MCP server in a few seconds - laid out as a step-by-step runbook, with values already filled in. I just hit Send and read the responses. This post is how I set it up and how I actually use it day to day.

The usual way, and why it's a waste

The information about an API lives in different places depending on the team. Sometimes it's a clean Swagger / OpenAPI page. Sometimes it's a description in a Jira ticket. Sometimes a Confluence page, or just a message from the developer who wrote the endpoint. Wherever it lives, the manual move is the same: read it, then re-type each endpoint into Postman.

It's not hard work. It's just slow, repetitive, and easy to get subtly wrong - a missing header here, a wrong content type there. And none of it is actually testing. It's setup that stands between you and the part of the job that matters.

What you need

Three things:

A Postman account and an API key
Claude (Claude Code or the desktop app) with the Postman MCP server connected
The API definition - from wherever it lives

1. Get your Postman API key

In Postman: Settings -> API keys -> Generate API Key. Copy it somewhere safe - you'll pass it to the MCP server, not paste it into chats.

2. Connect the Postman MCP server to Claude

MCP (Model Context Protocol) lets Claude talk directly to Postman instead of you copy-pasting back and forth. Add the server to your .mcp.json (project root, or ~/.claude/mcp.json for global):

{
 "mcpServers": {
 "postman": {
 "command": "npx",
 "args": ["-y", "@postman/postman-mcp-server", "--full"],
 "env": {
 "POSTMAN_API_KEY": "${POSTMAN_API_KEY}"
 }
 }
 }
}

Set POSTMAN_API_KEY in your environment, restart Claude, and run /mcp to confirm it connected. Now Claude can create and edit collections in your workspace on its own.

3. Point it at the API

If you have Swagger, even better - Spring services expose the full method library as OpenAPI at /v3/api-docs:

curl -s http://your-service/v3/api-docs -o api.json

But the source doesn't have to be Swagger. Paste the Jira ticket text, the Confluence export, or the endpoint description straight into Claude. It pulls the endpoints out either way.

How it actually works

Here's the part I like. I don't just get a flat dump of requests - I ask Claude to build the collection as a runbook, where the order of the requests is the order I run the test:

request names are steps: 1) create threshold, 2) get by id, 3) update, 4) delete and verify
top to bottom is the test sequence
every value (host, ids, request bodies) is hardcoded right into the request - no {{variables}} to fill in by hand

Then testing looks like this: I open the collection and press Send, top to bottom. When a request creates something and returns a new id, Claude reads it from the response and patches it into the next requests through the Postman API. I keep pressing Send.

A quick example

Say I give Claude an endpoint spec like this:

POST /api/threshold - create a detection threshold for a resource
GET /api/threshold/{id} - read it back
PUT /api/threshold/{id} - update it
DELETE /api/threshold/{id}

I ask: "Build a Postman collection in my workspace as a step-by-step runbook to create a threshold, read it, update it, and delete it. Hardcode the values, name each request as a numbered step."

A few seconds later there's a collection with four ordered requests, bodies filled in, auth set at the collection level. I run them in order. After step 1 returns the new threshold id, the GET/PUT/DELETE steps already point at it. I read each response and decide whether the service behaved correctly.

Where this helps, and where it doesn't

The setup work mostly disappears. The thinking doesn't, and that's the point.

AI is good at: turning a spec into a structured collection, naming and ordering requests, filling in boilerplate, patching ids between steps.

AI does not do the actual QA: deciding which scenarios matter, what the edge cases are, whether a 201 actually means the business logic is correct, reading logs to find why something failed. A green response is not a passing test - that judgment is still mine.

So this isn't "AI tests for me". It's "AI clears the busywork so I spend my time on the part that needs a tester".

Wrapping up

If you do REST API testing and haven't tried Claude + Postman MCP yet, it's worth half an hour to set up. The first time a collection you'd have spent an hour building shows up ready in seconds, it clicks.

I wrote up my full setup, prompt templates, and honest notes on where AI breaks down here: github.com/anton-kirilchuk/ai-assisted-qa-mcp

I post more about backend QA and AI-augmented testing on LinkedIn - happy to compare notes.

How much of your testing time still goes to setup before the real testing starts?