DEV Community

Cover image for What We Learned Generating 12,000 API Tests from OpenAPI Specs
Sushant Joshi
Sushant Joshi

Posted on

What We Learned Generating 12,000 API Tests from OpenAPI Specs

#ai #api #automation #testing

We thought generating API tests from an OpenAPI specification would eliminate the repetitive work of writing test cases. It did. What we didn't expect was how quickly it exposed problems in our API specifications instead of our APIs.

After generating more than 12,000 API tests from OpenAPI specifications across dozens of projects, one thing became obvious:

The biggest bottleneck wasn't the test generator.

It was the API specification itself.

We expected to spend most of our time improving generation algorithms.

Instead, we spent most of it fixing incomplete schemas, ambiguous contracts, and inconsistent specifications.

If you're planning to generate tests from OpenAPI, here's what surprised us the most.

The Promise of OpenAPI Test Generation

Most API teams already have an OpenAPI specification.

Unfortunately, many teams treat it as documentation instead of a contract.

That's a missed opportunity.

A good specification can generate:

  • Request validation
  • Response validation
  • Schema assertions
  • Negative test cases
  • Authentication scenarios
  • Mock servers
  • SDKs
  • Documentation

And, of course, automated API tests.

Generating tests directly from the specification means developers don't have to manually recreate the contract inside Postman collections or code-based frameworks.

The specification becomes the single source of truth.

At least, that's the theory.

What We Actually Generated

Over several months we processed thousands of endpoints from internal projects, public APIs, and customer specifications.

Across all of them we generated approximately:

  • 12,000+ API test cases
  • Hundreds of OpenAPI specifications
  • Thousands of request/response assertions
  • Countless schema validations

The APIs themselves weren't the interesting part.

The specifications were.

Lesson 1: Most OpenAPI Specs Are Only 80% Complete

One pattern appeared immediately.

Developers document the happy path.

Everything else gets skipped.

Typical examples included:

Missing error responses:

responses:
  200:
    description: Success
Enter fullscreen mode Exit fullscreen mode

But no:

400
401
403
404
500
Enter fullscreen mode Exit fullscreen mode

The generator can only create tests for scenarios that exist in the contract.

If the specification ignores validation failures, your generated suite will too.

Lesson 2: Example Values Matter More Than Expected

OpenAPI supports examples.

Many teams ignore them.

Instead of:

name:
  type: string
Enter fullscreen mode Exit fullscreen mode

provide:

name:
  type: string
  example: John Doe
Enter fullscreen mode Exit fullscreen mode

This improves generated:

  • Requests
  • Mock responses
  • Documentation
  • Test readability

Good examples dramatically improve generated tests.

Lesson 3: Nullable Doesn't Mean Optional

This was responsible for an astonishing number of incorrect assertions.

These are different:

nullable: true
Enter fullscreen mode Exit fullscreen mode

and

required: false
Enter fullscreen mode Exit fullscreen mode

One means:

"The property exists but may contain null."

The other means:

"The property might not exist at all."

Generated tests need to distinguish between both.

Otherwise perfectly valid responses fail incorrectly.

Lesson 4: Authentication Is Frequently Under-Specified

Many APIs defined:

security:
  - bearerAuth: []
Enter fullscreen mode Exit fullscreen mode

Globally.

Then individual endpoints silently behaved differently.

Some required API keys.

Others ignored authentication.

The specification never reflected reality.

Test generation immediately exposed those inconsistencies.

Lesson 5: Schemas Drift Faster Than Teams Realize

One endpoint returned:

{
  "userId": 123
}
Enter fullscreen mode Exit fullscreen mode

Another returned:

{
  "id": 123
}
Enter fullscreen mode Exit fullscreen mode

Both represented the same resource.

Both passed manual testing.

Neither matched the documented contract consistently.

Generated tests highlighted schema drift that had accumulated over years.

What the Generator Could Infer

Modern generators are surprisingly capable.

They can infer:

  • Required request bodies
  • Path parameters
  • Query parameters
  • Status codes
  • JSON schema assertions
  • Authentication requirements

From a single specification.

But they cannot infer business rules.

What Still Requires Human Input

For example:

Should:

discount = 110%
Enter fullscreen mode Exit fullscreen mode

be valid?

The schema cannot answer that.

Should creating an order reduce inventory?

The schema doesn't know.

Should duplicate email addresses return 409 or 422?

Only the business can define that.

Generated tests complement—not replace—human-designed scenarios.

The Biggest Surprise

We expected developers to ask:

"How accurate is the generator?"

Instead they asked:

"Why is the generator failing on my specification?"

In most cases the answer was simple.

The specification wasn't actually valid.

Or it was incomplete.

The tool simply surfaced issues that had always been there.

The Generator Became a Spec Validator

One unexpected outcome was that teams started fixing specifications before generating tests.

The generator effectively became a quality gate.

Instead of discovering contract issues in production, developers discovered them during generation.

That feedback loop was much cheaper.

Common Specification Problems We Found

Across hundreds of specifications, the same issues appeared repeatedly:

  • Missing required properties
  • Incorrect data types
  • Invalid examples
  • Inconsistent naming
  • Missing response schemas
  • Undefined authentication
  • Duplicate models
  • Ambiguous nullable fields

Most of these weren't testing problems.

They were specification problems.

What We Changed in Our Workflow

Today our process looks like this:

Design API

↓

Write OpenAPI Specification

↓

Validate Specification

↓

Generate Tests

↓

Add Business Assertions

↓

Run in CI
Enter fullscreen mode Exit fullscreen mode

Notice what's missing?

Manual recreation of test cases.

The contract drives almost everything.

Where Human Testing Still Wins

Automatic generation is fantastic for:

  • Schema validation
  • Required fields
  • Authentication
  • Parameter combinations
  • Status codes

Humans still write tests for:

  • Business rules
  • Complex workflows
  • Performance
  • Security
  • User journeys

The two approaches work best together.

The Biggest Productivity Gain

The biggest benefit wasn't writing fewer tests.

It was eliminating duplicated work.

Previously developers described the API three times:

  • OpenAPI specification
  • Postman collection
  • Automated tests

Now the specification generates much of the repetitive work automatically.

Teams spend more time testing behavior instead of recreating contracts.

What We'd Recommend to Teams Starting Today

If you're adopting OpenAPI-driven testing:

  1. Treat the specification as code.
  2. Validate it continuously.
  3. Keep examples realistic.
  4. Separate nullable from optional fields.
  5. Add business assertions after generation.
  6. Run generated tests in every CI pipeline.

The better the specification becomes, the better the generated tests become.

Final Thoughts

Generating 12,000 API tests taught us something unexpected.

The hardest part wasn't automation.

It was creating clear, accurate API contracts.

OpenAPI isn't just documentation.

Done well, it becomes the foundation for documentation, SDK generation, mock servers, contract validation, and automated testing.

The quality of your generated tests will almost always reflect the quality of your specification.

If you're interested in this approach, we have documented more about OpenAPI test automation

The best API tests don't start with a testing framework.

They start with a well-written contract.

Top comments (0)