What an OpenAPI spec doesn't tell you

Conor Griffin — Sun, 31 May 2026 21:32:13 +0000

An OpenAPI spec is a promise. It says: this field is an integer, this one has a maximum
of 100, this one is required, this one is a string no longer than 50 characters. It reads
like a contract.

The problem is that nobody verifies the contract. The spec describes what the API claims
to accept. It says nothing about what happens when you send it everything it claims to
reject, and that gap is where the interesting bugs live.

I spent a while building a tool that lives entirely inside that gap. This is what I learned
from it.

A spec is a list of things to violate

The moment you stop reading a spec as documentation and start reading it as a list of rules
to break, it becomes generative. Every constraint implies its own violation:

minimum: 1 → send 0, send -1
maxLength: 50 → send 51 characters, send 10,000
required → omit it, send null
type: integer → send "not_an_int", send Integer.MAX_VALUE, send a float
type: boolean → send "true" as a string, send 1, send "yes"

You don't need to imagine test cases. The spec hands them to you. A field with three
constraints generates a dozen payloads mechanically. An API with 60 fields generates
thousands. This is the part manual testing can't keep up with, not because the cases
are hard to think of, but because there are too many to remember and too tedious to write
by hand.

The response tells you more than the status code

A 500 is the obvious finding. But the response body is where the API leaks what it actually
does internally:

A Java stack trace in the body means an unhandled exception reached the client and exposed your package structure while it was at it.
A SQLException or an ORA- error string means a database error propagated all the way out and you're now looking at a potential injection surface.
A 200 on input you know is invalid is the quietest and most dangerous finding of all. The API didn't crash. It accepted bad data and moved on. Nobody gets paged for that. It just sits in your database.

That last category, the silent acceptance is the one I find most interesting, because
it produces no error anywhere. The only way to catch it is to send input you know is wrong
and be suspicious of success.

What I pointed it at

The official Swagger Petstore demo. The reference API. The one the tutorials use.

GET /user/login returns a token for null credentials. Oversized login strings get a 200.
Write endpoints 500 on malformed bodies. Again, it's a demo, none of it matters in
practice. But it's a clean illustration of the central point: being the canonical example
of an API and correctly handling adversarial input are two completely unrelated properties.

The shape of the thing

If you want to build something similar, the pipeline is straightforward:

parse the spec → generate payloads per field → fire them → analyse responses → report

The only non-obvious parts are payload generation (drive it off the declared type and
constraints, fall back to a static list for undeclared fields) and response analysis
(don't just check the status, pattern-match the body for leaks and treat unexpected
successes as findings).

I wrote mine in Java 21 with REST Assured and put the output in an Allure report, mostly
because that's the stack the QA teams I want to work with already use. The code is here
if it's useful as a reference:

https://github.com/ConorGriffin-Dev/chaos-monkey

The takeaway I'd leave you with: you already have a test suite sitting in your repo. It's
called openapi.json. You're just not running it adversarially yet.

DEV Community: Conor Griffin

What an OpenAPI spec doesn't tell you

A spec is a list of things to violate

The response tells you more than the status code

What I pointed it at

The shape of the thing