From OpenAPI Spec to a Running Test Suite in 90 Seconds - A Walkthrough

Rishi Gaurav — Wed, 17 Jun 2026 09:22:13 +0000

I timed it: 87 seconds from pasting our Petstore-clone openapi.yaml into the tool to seeing 41 generated tests pass against a local mock.

That number surprised me.

Like many engineers, I've spent years building API tests manually. Even with mature frameworks, the process usually involves reading documentation, understanding endpoints, creating test data, writing assertions, handling authentication, and then maintaining everything as APIs evolve.

The promise of openapi test generation has existed for years, but most implementations either produce low-value boilerplate or require so much manual configuration that the time savings disappear.

This walkthrough documents a real-world experiment: taking a simple OpenAPI specification and turning it into a running API test suite in under 90 seconds.

The 4-Line openapi.yaml I Started With

Technically, the complete specification wasn't four lines long. The actual OpenAPI file contained all the endpoint definitions, schemas, and responses.

However, from a user's perspective, the process was essentially this simple:

openapi: 3.0.3
info:
  title: "Petstore Clone"
  version: 1.0.0

I uploaded the complete OpenAPI document into the generator.

The specification contained:

12 endpoints
CRUD operations for pets
User management endpoints
Search and filtering operations
Standard HTTP response definitions
Authentication requirements
JSON schema definitions

Nothing particularly complex.

The goal wasn't to challenge the system with edge cases. Instead, I wanted to evaluate whether modern openapi to tests tooling could generate a practical test suite that a real team would actually use.

The upload process took only a few seconds.

Then the interesting part began.

What the Generator Inferred vs What It Asked Me to Fill In

One of the biggest concerns with automated test generation is configuration overhead.

Many tools advertise automation but immediately present users with dozens of setup screens.

In this case, the generator successfully inferred:

Endpoint Coverage

It automatically identified:

GET endpoints
POST endpoints
PUT operations
DELETE operations
Query parameters
Path parameters

No manual mapping was required.

Request Payloads

The OpenAPI schemas already described valid request bodies.

The generator used those schemas to create realistic payload examples.

For example, instead of generating:

{
  "name": "string"
}

it generated structured test data that more closely resembled real requests.

Response Validation

The tool automatically extracted:

Expected status codes
Response schema definitions
Required properties
Data types

This became the foundation for generated assertions.

Authentication Requirements

The specification included bearer token authentication.

The generator recognized this requirement immediately.

What It Asked Me to Provide

Despite the high level of automation, a few inputs still required human intervention.

Environment URL

The generator couldn't know where the API was running.

I supplied:

http://localhost:4010

for the mock server.

Authentication Values

The specification described authentication mechanisms but not actual credentials.

I entered a sample token.

Dynamic Test Data

A few endpoints depended on resources being created first.

The generator offered sensible defaults, but I reviewed these relationships to ensure consistency.

Overall, the manual configuration took less than one minute.

That balance felt right.

The system handled structural information from the specification while leaving environment-specific details to the user.

The 3 Tests It Generated That I Would Have Forgotten to Write

The most valuable part wasn't generating obvious happy-path tests.

Any engineer can write:

Create Pet
Get Pet
Update Pet
Delete Pet

The real value came from tests that frequently get skipped during manual implementation.

1. Invalid Enum Values

One endpoint accepted pet status values:

{
  "status": "available"
}

The specification defined allowed enum values.

The generator automatically created negative tests that submitted invalid values.

Examples included:

{
  "status": "invalid-status"
}

and verified proper validation responses.

This is exactly the type of test many teams intend to write but rarely prioritize.

2. Missing Required Fields

The schema marked several properties as required.

The generator systematically removed each required property and verified server behavior.

Examples included:

Missing name
Missing category
Missing identifier

Rather than creating one generic validation test, it created multiple targeted scenarios.

This improved coverage significantly.

3. Boundary Condition Testing

One search endpoint accepted page size parameters.

The generator identified numeric constraints from the OpenAPI specification and generated tests around boundaries.

Examples included:

Minimum values
Maximum values
Values just outside accepted ranges

Boundary testing often produces high-value defect discovery, yet it is commonly overlooked because it requires additional effort.

The generated suite included these cases automatically.

These three categories alone justified the experiment.

Running the Suite Against a Mock and Then Against a Real Service

The generated suite was first executed against a mock server.

This provided immediate feedback.

Phase 1: Mock Validation

Against the mock service:

41 tests executed
41 tests passed

This validated that:

Endpoint definitions were correct
Request structures matched specifications
Generated assertions aligned with documented responses

The entire process took seconds.

At this stage, we weren't testing application logic.

We were testing contract compliance.

That's an important distinction.

Phase 2: Real Service Validation

Next, I pointed the same suite at a running implementation.

This is where things became more interesting.

Several tests exposed inconsistencies between implementation and documentation.

Examples included:

Missing response properties
Incorrect status codes
Validation behavior that differed from the specification

None of these were catastrophic issues.

However, they represented exactly the type of drift that accumulates over time.

The generated tests immediately highlighted those differences.

This demonstrated one of the strongest use cases for generate api tests from swagger workflows:

Keeping implementation behavior aligned with documented contracts.

Where OpenAPI Code Generation Helps Most

Many teams already use OpenAPI codegen to generate:

SDKs
Client libraries
Server stubs
Documentation

Test generation is a natural extension of the same philosophy.

The specification already contains:

Endpoint definitions
Parameters
Schemas
Constraints
Authentication models

Why should teams manually recreate that knowledge inside a testing framework?

By generating tests directly from the specification, teams reduce duplication and improve consistency.

The generated suite becomes another artifact derived from a single source of truth.

As the API evolves, regenerating tests becomes significantly easier than maintaining large collections of handcrafted boilerplate.

What the Generator Still Gets Wrong (And the Workaround)

No automation tool is perfect.

This one isn't either.

Here are the biggest limitations I encountered.

Business Logic Understanding

The generator understands structure.

It does not understand intent.

For example:

A discount API may require:

Gold customers receive 20%
Silver customers receive 10%
New customers receive 5%

The OpenAPI specification rarely contains those rules.

As a result, the generator cannot create meaningful business-validation tests automatically.

Workaround

Use generated tests as the baseline layer.

Then add custom business-rule tests on top.

Think of generated suites as coverage for:

Contracts
Validation
Authentication
Data types
Response schemas

Keep domain-specific behavior in manually authored tests.

Complex Workflow Scenarios

Multi-step business journeys remain challenging.

For example:

Create account
Verify email
Purchase subscription
Upgrade plan
Request refund

The generator can create pieces of this flow but may not assemble the complete lifecycle correctly.

Workaround

Generate foundational endpoint tests first.

Then build workflow tests using those generated assets as reusable components.

Test Data Semantics

Generated payloads are structurally valid.

They are not always logically meaningful.

For example:

{
  "firstName": "Test",
  "email": "abc@example.com"
}

is technically valid.

But realistic production scenarios often require more sophisticated datasets.

Workaround

Replace generated fixture data with reusable test data libraries after generation.

This preserves automation while improving realism.

Final Thoughts

After timing the process, reviewing the generated suite, and executing it against both mock and real services, my conclusion is straightforward:

OpenAPI-based test generation is no longer a novelty. It's becoming a practical way to establish baseline API coverage quickly.

Will it replace experienced QA engineers?

No.

Will it eliminate the need for custom testing?

Absolutely not.

But if a specification already exists, manually creating dozens of repetitive contract-validation tests no longer feels like the best use of engineering time.

The strongest outcome wasn't that 41 tests were generated automatically.

The strongest outcome was that those 41 tests immediately surfaced gaps between documentation and implementation while covering several scenarios that I likely would have postponed or forgotten.

If you're exploring openapi test generation for your own APIs, it's worth evaluating modern approaches that transform specifications directly into executable suites.

Learn more on the OpenAPI test automation page: https://totalshiftleft.ai/openapi-test-automation

When the API contract already contains the knowledge, letting automation build the first version of the test suite is increasingly becoming the obvious choice.

DEV Community: Rishi Gaurav

From OpenAPI Spec to a Running Test Suite in 90 Seconds - A Walkthrough

The 4-Line openapi.yaml I Started With

What the Generator Inferred vs What It Asked Me to Fill In

Endpoint Coverage

Request Payloads

Response Validation

Authentication Requirements

What It Asked Me to Provide

Environment URL

Authentication Values

Dynamic Test Data

The 3 Tests It Generated That I Would Have Forgotten to Write

1. Invalid Enum Values

2. Missing Required Fields

3. Boundary Condition Testing

Running the Suite Against a Mock and Then Against a Real Service

Phase 1: Mock Validation

Phase 2: Real Service Validation

Where OpenAPI Code Generation Helps Most

What the Generator Still Gets Wrong (And the Workaround)

Business Logic Understanding

Workaround

Complex Workflow Scenarios

Workaround

Test Data Semantics

Workaround

Final Thoughts