How to Generate Karate API Tests from OpenAPI — Without Creating a Maintenance Nightmare

Generating API tests from an OpenAPI file sounds like an easy automation win.

Parse the specification. Create one test per endpoint. Add a few assertions. Commit the generated files.

That approach works surprisingly well in a demo — and often becomes painful as soon as the API or test suite starts evolving.

The hard problem is not producing Karate syntax. The hard problem is producing tests that fit the way your project already works and remain useful after the first generation run.

The tempting approach

Imagine an API contract containing:

paths:
  /orders:
    post:
      operationId: createOrder
  /orders/{id}:
    get:
      operationId: getOrder

A generator can easily produce something like:

Feature: Orders API

Scenario: Create an order
  Given url baseUrl
  And path 'orders'
  And request { productId: 123, quantity: 1 }
  When method post
  Then status 201

Scenario: Get an order
  Given url baseUrl
  And path 'orders', 123
  When method get
  Then status 200

It is valid Karate. It may even pass.

But several important questions are still unanswered:

• Where did baseUrl come from?
• Does the project already have authentication helpers?
• Is productId: 123 valid test data?
• What should the response schema contain?
• What happens for an invalid product?
• What happens when quantity is missing?
• Is the endpoint already covered by another scenario?
• Does the team have naming, tagging or reusable-feature conventions?

A generated test that ignores those questions creates code, but not necessarily useful coverage.

Treat OpenAPI as a contract, not a test suite

OpenAPI tells us a lot about the API surface:

• paths and HTTP methods
• parameters
• request schemas
• response schemas
• required fields
• enums and formats
• documented response codes
• security schemes

That information is excellent input for test generation.

But the specification usually does not contain all the information needed for an executable test suite. Environment configuration, reusable authentication, realistic data creation and project conventions typically live elsewhere.

A good generator therefore needs two kinds of context:

1. Contract context — what the API says it accepts and returns.
2. Repository context — how this project already executes Karate tests.

Ignoring the second is where many generated suites become difficult to maintain.

Generate deterministic coverage first

I prefer to begin with rules that do not need an LLM.

For each operation, deterministic generation can derive candidate scenarios from the contract.

For example, a required field gives us at least two obvious cases:

quantity:
  type: integer
  minimum: 1

Positive case:

And request { productId: 123, quantity: 1 }
When method post
Then status 201

Boundary or negative candidates can be derived from the same schema:

And request { productId: 123, quantity: 0 }
When method post
Then status 400

and:

And request { productId: 123 }
When method post
Then status 400

This has an important advantage: the reasoning is explainable. We know exactly why each scenario exists.

AI can still improve the suite later, but it should not be the only mechanism deciding what needs to be tested.

Reuse the project before inventing configuration

Before generating a new feature, inspect the repository.

A Karate project may already contain:

src/test/java/
  karate-config.js
  auth/
    token.feature
  common/
    create-user.feature
  orders/
    orders.feature

If authentication already comes from karate-config.js, generated tests should use it.

If the team creates test users through a reusable feature, the generator should reuse that pattern instead of inventing credentials.

If scenarios consistently use tags such as:

@orders @smoke

new tests should follow the same convention.

Repository awareness is the difference between generating a standalone example and adding a maintainable test to an existing suite.

Detect coverage before generating more tests

Another mistake is assuming every OpenAPI operation needs a new feature file.

First compare the API contract with the existing Karate scenarios.

Think of the API surface as a matrix:

POST   /orders         covered
GET    /orders/{id}    covered
DELETE /orders/{id}    missing
PATCH  /orders/{id}    missing

Now generation becomes targeted.

Instead of producing another complete test suite, generate tests only for the gaps.

That makes the workflow:

OpenAPI
   ↓
Existing Karate suite
   ↓
Coverage mapping
   ↓
Missing operations
   ↓
Generate only what is missing

This is much more useful in a real repository than repeatedly regenerating everything.

Use AI as an enhancement layer

There are places where AI is genuinely useful.

For example:

• understanding domain intent from descriptions
• suggesting edge cases that are not encoded in the schema
• adapting generated scenarios to an existing style
• explaining a failed test
• proposing a repair after an API change

But those workflows should sit on top of deterministic evidence.

A useful architecture looks like this:

OpenAPI specification
        ↓
Deterministic parser
        ↓
Coverage + schema rules
        ↓
Repository conventions
        ↓
Candidate Karate scenarios
        ↓
Optional AI enhancement
        ↓
Validation / review

The key word is optional.

You should still be able to generate, execute and analyse tests when an AI provider is unavailable.

Validate what was generated

Generation is not finished when a .feature file exists.

At minimum, generated output should be checked for:

• valid Karate syntax
• valid referenced configuration
• duplicate scenarios
• unresolved variables
• incorrect paths or HTTP methods
• missing required request data
• assertions that are too weak

The last item deserves particular attention.

This test can pass while proving very little:

When method get
Then status 200

Where the contract supports it, stronger assertions might verify:

Then status 200
And match response.id == '#number'
And match response.status == '#string'
And match response.items == '#array'

The goal is not to maximize assertion count. It is to make sure the test can detect the failures it was created to protect against.

This is the workflow I built into Karate Test Management

I originally built Karate Test Management, an open-source VS Code extension, around test generation. As the project grew, the interesting problem became everything around generation.

The current workflow can:

• generate Karate tests from OpenAPI
• import Postman, HAR and GraphQL sources
• index an existing Karate test library
• execute features and individual scenarios
• compare OpenAPI operations with existing test coverage
• identify missing tests
• track quality findings
• analyse failures and flakiness
• optionally use AI for generation and repair

The important architectural decision is that deterministic generation, execution and coverage analysis do not depend on AI.

AI is there to enhance test-engineering work, not to replace the evidence underneath it.

A better mental model

Instead of thinking:

OpenAPI → generated test files

I find this model more useful:

Discover
   ↓
Understand the contract and repository
   ↓
Measure existing coverage
   ↓
Generate missing scenarios
   ↓
Execute
   ↓
Analyse evidence
   ↓
Maintain

That turns test generation from a one-off code-generation trick into part of a sustainable API testing workflow.

If you use Karate and want to experiment with this approach, the project is open source:

• GitHub: https://github.com/mov2day/KaratePlugin
• VS Code Marketplace: https://marketplace.visualstudio.com/items?itemName=MuthuKumarKoodalingam.karate-test-generator

The next article in this series will look at a related problem: your Karate tests may all be green while large parts of your API are still untested.

Comments

[Raknaos]

The "maintenance after the first generation run" framing is the honest one. Generated tests that don't match the repo's existing style get rewritten by hand anyway, so you pay the generator cost and the human cost.

The best ratio I've had was generating at the flow boundary rather than per endpoint: one feature file per domain journey with the spec as input, not as gospel. How does the plugin handle spec drift on regeneration, does it overwrite hand-tuned assertions or merge around them?