I fed the same OpenAPI spec into four different test generators last weekend. All four failed on the same five things.
That wasn't supposed to happen.
The generators came from different vendors. They used different parsing engines, different test-generation strategies, and different AI capabilities. Some focused on contract testing. Others emphasized intelligent test creation and edge-case discovery.
Yet all four stumbled on the exact same sections of the specification.
The surprising part wasn't that the tools failed.
The surprising part was that every failure could be traced back to issues inside the OpenAPI document itself.
Over the years, I've learned that most API test generation problems aren't actually tool problems. They're specification quality problems.
A clean OpenAPI specification can generate hundreds of useful tests automatically.
A flawed specification can confuse even the smartest generator.
If you're using OpenAPI to generate tests, SDKs, mocks, documentation, or validation suites, these are the five most common OpenAPI mistakes I've seen repeatedly—and why they cause so much trouble.
1. Missing Required Fields on Nested $ref Objects
This is easily one of the most overlooked issues in OpenAPI specifications.
Consider the following schema:
Customer:
type: object
properties:
id:
type: integer
address:
$ref: '#/components/schemas/Address'
The referenced schema looks like:
Address:
type: object
properties:
street:
type: string
city:
type: string
Looks harmless.
Now imagine the actual API requires:
{
"street": "Main Street",
"city": "London"
}
but the schema never specifies:
required:
- street
- city
The generator now assumes both fields are optional.
Why Test Generators Struggle
When generating positive and negative test cases, the tool needs to know:
- Which fields must exist
- Which fields may be omitted
- Which omissions should trigger validation failures
Without required declarations, generators often create:
{
"address": {}
}
and treat the payload as valid.
The resulting tests provide little value because they don't reflect real application behavior.
Best Practice
Always define required fields explicitly at every schema level.
Even when schemas are reused via $ref, each referenced object should clearly identify mandatory properties.
Never assume the generator will infer business intent.
Schemas should be unambiguous.
2. anyOf With No Discriminator (The One That Hurts Most)
If I had to choose the single most problematic specification pattern, this would be it.
Consider:
Pet:
anyOf:
- $ref: '#/components/schemas/Cat'
- $ref: '#/components/schemas/Dog'
This tells the generator:
"The payload may match Cat or Dog."
Sounds reasonable.
The problem is that the generator has no reliable way to determine which schema should be used for a specific test case.
Why It Breaks Generation
Imagine:
Cat:
properties:
name:
type: string
Dog:
properties:
name:
type: string
Now the payload:
{
"name": "Buddy"
}
matches both schemas.
The generator faces several questions:
- Is this a Cat?
- Is this a Dog?
- Should both test paths be created?
- Which validation rules apply?
Different tools make different assumptions.
Most assumptions are wrong.
The Fix
Use discriminators whenever possible.
Example:
Pet:
oneOf:
- $ref: '#/components/schemas/Cat'
- $ref: '#/components/schemas/Dog'
discriminator:
propertyName: petType
Now responses become:
{
"petType": "Dog",
"name": "Buddy"
}
The ambiguity disappears.
Test generation becomes deterministic.
This single change often improves generated coverage dramatically.
3. Untyped Query Parameters (String vs Integer Ambiguity)
Another surprisingly common issue involves query parameters.
Consider:
parameters:
- name: page
in: query
Looks simple.
Unfortunately, the schema never specifies the data type.
Why This Creates Problems
A generator needs type information to create meaningful tests.
Should it generate:
?page=1
or
?page=abc
or
?page=true
Without explicit typing, every option becomes theoretically valid.
Different generators handle this differently:
- Some assume strings
- Some guess based on naming
- Some generate everything
- Some skip validation entirely
None of these approaches are ideal.
The Hidden Impact
This issue affects more than positive tests.
It also impacts:
- Boundary testing
- Negative testing
- Fuzz testing
- Schema validation
For example:
schema:
type: integer
minimum: 1
maximum: 100
enables generators to automatically create:
- 0
- 1
- 100
- 101
Without typing, those valuable edge cases disappear.
Best Practice
Always define parameter schemas completely.
Example:
parameters:
- name: page
in: query
schema:
type: integer
minimum: 1
The more constraints provided, the better the generated tests become.
4. Example Values That Don't Match the Schema
This problem creates some of the most confusing failures.
Imagine:
type: integer
example: "123"
or
type: boolean
example: "true"
The examples look correct at first glance.
But they violate the schema definition.
Why This Matters
Most generators rely heavily on example values.
Examples help generate:
- Request payloads
- Mock data
- Positive test cases
- Sample assertions
When examples contradict schemas, the generator receives conflicting instructions.
The schema says:
type: integer
The example says:
"123"
Which one should the tool trust?
What Usually Happens
Different generators respond differently:
Generator A
Uses schema definition.
Generator B
Uses example value.
Generator C
Attempts to coerce data types.
Generator D
Fails validation completely.
The result is inconsistent behavior across platforms.
Best Practice
Treat examples as executable documentation.
Every example should validate successfully against its schema.
A useful review process is:
- Validate schema
- Validate examples
- Validate generated payloads
All three should agree.
5. Auth Scheme Defined Globally but Overridden Per-Path
Authentication definitions often create subtle specification issues.
Consider:
security:
- bearerAuth: []
This applies globally.
Everything looks fine.
Later, a specific endpoint introduces:
paths:
/public-data:
get:
security: []
This explicitly removes authentication.
Still valid.
The trouble begins when specifications contain dozens or hundreds of endpoints.
Why Test Generators Fail Here
Generators must determine:
- Which endpoints require authentication
- Which endpoints are public
- Which credentials to attach
- Which negative auth scenarios to generate
Conflicting security definitions make this surprisingly difficult.
I've seen generators:
- Add tokens to public endpoints
- Skip tokens on protected endpoints
- Generate invalid authentication tests
- Ignore overrides completely
The Real Problem
Many teams inherit APIs over several years.
Security definitions evolve.
Documentation evolves.
Endpoints move between versions.
Eventually, the specification contains multiple overlapping security patterns.
Humans can usually understand the intent.
Generators often cannot.
Best Practice
Maintain a clear authentication strategy.
Review:
- Global security settings
- Path-level overrides
- Operation-level overrides
Whenever possible, minimize exceptions.
The fewer special cases you create, the easier automated tooling becomes.
Why These Mistakes Keep Appearing
What's interesting is that none of these issues are technically invalid OpenAPI.
Many specifications containing these patterns will still:
- Render documentation correctly
- Generate SDKs successfully
- Pass schema validation
The problems emerge only when advanced automation enters the picture.
Test generation requires much higher precision than documentation generation.
Documentation can tolerate ambiguity.
Automated testing cannot.
A Simple Validation Checklist
Before feeding an OpenAPI document into any generator, review the following:
Schema Definitions
- Are required fields defined?
- Are nested references complete?
- Are enums constrained properly?
Polymorphism
- Does every
anyOforoneOfinclude a discriminator? - Can payload types be identified deterministically?
Parameters
- Are query parameters typed?
- Are constraints defined?
Examples
- Do examples validate against schemas?
- Are example payloads realistic?
Authentication
- Are security rules consistent?
- Are overrides intentional?
This five-minute review can save hours of debugging later.
Final Thoughts
After testing multiple generators against the same API specification, one conclusion became obvious:
Most test-generation failures begin long before the generator runs.
They begin when the specification is written.
The better your OpenAPI document, the better your generated tests, mocks, SDKs, documentation, and validation suites become.
Tools continue to improve every year.
AI-assisted generation is becoming increasingly capable.
Yet even the most advanced platform struggles when the specification contains ambiguity, conflicting definitions, or incomplete schema information.
If you're planning to generate automated API tests from your OpenAPI specification, spending time on specification quality is often the highest-return investment you can make.
For a deeper walkthrough on how to generate API tests from OpenAPI without these traps, visit:
https://totalshiftleft.ai/blog/how-to-generate-api-tests-from-openapi
A clean specification doesn't just improve documentation.
It becomes the foundation for every automation layer built on top of it.
Top comments (0)