DEV Community: Muthu Kumar Koodalingam

Your Karate Test Failed in CI. Should AI Fix It?

Muthu Kumar Koodalingam — Tue, 22 Sep 2026 07:58:49 +0000

Your Karate Test Failed in CI. Should AI Fix It?

A failed API test often creates an awkward handoff.

CI knows that a scenario failed. The Karate report may know the failed step, HTTP status, request and response. The source repository knows the scenario that produced it. But by the time someone investigates, that evidence is scattered across logs, artifacts and source files.

It is tempting to solve this with a simple prompt:

Here is the failing Karate feature and the error.
Fix the test.

That is also a good way to create a test suite that slowly learns to accept broken behavior.

The interesting problem is not whether an LLM can edit Gherkin. It can. The problem is deciding what evidence is strong enough to justify a repair, how narrowly the repair is allowed to operate, and where a human must remain in control.

This article describes the architecture I use in Karate Test Management for evidence-driven CI repair.

A failing test is not proof that the test is wrong

Consider this scenario:

Scenario: Get an existing order
  Given path 'orders', orderId
  When method get
  Then status 200
  And match response.status == 'CONFIRMED'

CI reports:

expected: 200
actual:   404

A repair model could make the build green immediately:

- Then status 200
+ Then status 404

That is syntactically valid and operationally dangerous.

The failure might mean:

the API contract changed;
test data was not created;
an environment reset removed the order;
authentication selected the wrong tenant;
the endpoint is broken;
the test is genuinely stale.

A repair system that starts by editing the assertion has confused observed behavior with expected behavior.

So the first rule is simple:

Treat a CI failure as evidence to investigate, not permission to weaken a test.

Normalize CI failures before asking AI anything

Different pipelines expose failures differently. GitHub Actions may provide downloadable test artifacts and logs. Jenkins may expose console output. A custom runner may already have structured JSON.

The repair layer should not care.

Normalize the useful evidence into one payload first:

{
  "source": "github-actions",
  "featurePath": "src/test/java/orders/orders.feature",
  "scenarioName": "Get an existing order",
  "scenarioLine": 18,
  "failedStep": "Then status 200",
  "errorMessage": "status code was: 404, expected: 200",
  "httpRequest": {
    "method": "GET",
    "url": "https://api.example.test/orders/9821"
  },
  "httpResponse": {
    "status": 404,
    "body": "{\"code\":\"ORDER_NOT_FOUND\"}"
  },
  "runId": "814521:1"
}

This is much more useful than sending an entire CI log to a model.

It gives the repair workflow explicit anchors:

CI run
  ↓
failed job / step
  ↓
feature + scenario identity
  ↓
HTTP evidence
  ↓
source scenario
  ↓
repair candidate

It also makes the process inspectable. If a proposed repair is questionable, you can see exactly which evidence produced it.

Locate the scenario precisely

Scenario identity is an underrated part of automated repair.

Imagine a repository containing:

Scenario: Get user
  ...

in several feature files, or a feature with generated scenarios that have similar names.

A repair engine should use more than a free-text scenario name where possible. Useful coordinates include:

feature path
scenario name
scenario line
scenario tags

The important safety property is unique identification.

If the system cannot identify exactly one source scenario, it should refuse the edit rather than guessing.

This principle generalizes beyond Karate: autonomous code changes become safer when uncertainty causes the system to stop instead of broadening its write scope.

Give the model evidence, not authority

Once the source scenario is located, AI can help reason about the failure. But the prompt should define a narrow repair boundary.

A useful repair context looks like this:

REPAIR CONTEXT
Feature: src/test/java/orders/orders.feature
Scenario: Get an existing order
Failed step: Then status 200
Error: status code was: 404, expected: 200

HTTP EVIDENCE
GET https://api.example.test/orders/9821
Response: 404
Body: {"code":"ORDER_NOT_FOUND"}

CONSTRAINT
Fix only the failed step and its immediate dependencies.
Do not change any other scenario.
Do not invent endpoints or fields not visible in the evidence.

OUTPUT
Return the complete repaired Scenario block only.

Notice what is deliberately absent: "make the test pass."

The goal is to produce a candidate repair, not optimize for a green build at any cost.

Why the complete scenario is a useful repair unit

Returning a single replacement line sounds safer, but failures often involve a small dependency chain.

For example:

* def order = call read('classpath:helpers/create-order.feature')
* def orderId = order.id
Given path 'orders', orderId
When method get
Then status 200

If order.id changed to order.orderId, repairing only the failed HTTP assertion cannot solve the root problem.

On the other hand, returning the entire feature file gives the model unnecessary authority over unrelated tests.

The scenario is a useful middle ground:

single line        → often insufficient context
single scenario    → useful bounded repair surface
entire feature     → unnecessarily broad write scope
repository         → far too broad

The repair system can then replace exactly that scenario in the original feature.

Diff before apply should be the default

Suppose the model proposes:

Scenario: Get an existing order
- * def orderId = order.id
+ * def orderId = order.orderId
  Given path 'orders', orderId
  When method get
  Then status 200

That is the moment where automation should slow down.

Show the original and candidate side by side. Let the engineer inspect whether the change preserves intent.

In Karate Test Management, CI repair defaults to a reviewable diff. Automatic application is an explicit setting rather than the default behavior. A backup can also be created before applying a repair.

That distinction matters because there are really two separate capabilities:

AI can propose a change
        ≠
AI is authorized to modify the test suite

Keeping those permissions separate is one of the simplest guardrails for AI-assisted engineering.

Never let repair erase the oracle

The most dangerous repair is one that makes a test less capable of detecting defects.

Consider:

Then status 200
And match response ==
"""
{
  id: '#number',
  state: 'CONFIRMED'
}
"""

A naive repair could turn this into:

Then status 200

The build becomes green, but the test has lost most of its value.

A practical repair review should therefore ask:

Did the candidate change the expected business behavior?
Did it remove or weaken assertions?
Did it introduce an endpoint, field or status not supported by evidence?
Did it modify anything outside the failed scenario?
Is the failure more likely product behavior, environment behavior or test behavior?

That final question is important. Some failures should produce no repair at all.

Separate deterministic work from AI work

Most of the CI-repair pipeline does not require an LLM.

Deterministic code can:

identify failed jobs;
download artifacts and logs;
extract .feature references;
locate scenario names and source lines;
detect common status/assertion failures;
extract HTTP methods, URLs, statuses and response bodies;
locate the exact scenario in the repository;
generate a diff;
enforce write boundaries.

AI becomes useful after that evidence has been assembled, where the task requires reasoning about how the scenario might need to change.

A safer architecture therefore looks like:

GitHub Actions / CI
        ↓
Failure extraction          deterministic
        ↓
Structured evidence         deterministic
        ↓
Scenario location           deterministic
        ↓
Repair proposal             AI-assisted
        ↓
Scope validation            deterministic
        ↓
Diff / review               human
        ↓
Apply

This is a much stronger pattern than giving an agent a repository and asking it to "fix CI."

Pulling evidence from GitHub Actions

For GitHub Actions, a useful workflow is to inspect the failed run and gather the evidence already produced by the test job.

The current Karate Test Management implementation can pull workflow runs, jobs, artifacts and logs, then search those sources for a feature reference, scenario name, error and HTTP evidence.

The extraction is intentionally best-effort because CI output varies. A Karate failure might expose:

src/test/java/orders/orders.feature:18
Scenario: Get an existing order
status code was: 404, expected: 200

and request logs may contain:

> GET https://api.example.test/orders/9821
response status: 404
{"code":"ORDER_NOT_FOUND"}

From those fragments, the system can build the structured payload shown earlier.

If it cannot infer the feature path reliably, the correct behavior is to stop the automated repair path rather than invent a target.

What about Jenkins or GitLab?

The same architecture does not require every CI provider to expose the same API.

The normalized failure contract can represent several sources:

type FailureSource =
  | 'github-actions'
  | 'jenkins'
  | 'gitlab-ci'
  | 'generic';

That means provider-specific adapters can evolve independently while the repair engine continues to consume one evidence shape.

A Jenkins pipeline, for example, could POST a compact failure payload after a failed Karate run instead of granting a desktop extension broad access to Jenkins itself.

This separation is useful operationally and from a security perspective.

AI repair should be optional

There is another architectural boundary worth keeping: the test suite must remain usable when AI is unavailable.

Generation, execution, coverage analysis and normal test management should not depend on a model provider. Repair is an enhancement layer.

If AI is disabled, quota is exhausted, or a provider is unavailable, the CI evidence is still useful:

Failed scenario
Failed step
Request
Response
Source location
Run identity

An engineer can investigate from that information directly.

This prevents AI availability from becoming a new dependency in the delivery pipeline.

The bigger lesson: automate evidence before decisions

CI repair is one example of a broader pattern I increasingly use for AI-assisted test engineering.

Do not begin with:

failure → AI → code change

Build this instead:

failure
  ↓
collect evidence
  ↓
normalize evidence
  ↓
identify exact change surface
  ↓
reason about candidate change
  ↓
validate boundaries
  ↓
review

The AI step becomes smaller, better informed and easier to audit.

That is important for testing tools because the test suite is itself part of the safety system. An automated repair that silently weakens the oracle can be worse than a failing build.

How this fits into Karate Test Management

Karate Test Management treats CI repair as one part of a larger test lifecycle rather than a standalone "self-healing" trick.

The current project includes:

structured CI failure ingestion;
GitHub Actions run, artifact and log intake;
feature/scenario location;
HTTP evidence extraction;
AI-assisted scenario repair;
bounded scenario replacement;
reviewable diffs;
optional backups and explicit auto-apply;
the same provider-routing guardrails used by other AI-assisted workflows.

The principle behind all of them is the same: AI can help reason over test evidence, but deterministic controls decide what it is allowed to touch.

If you use Karate and want to inspect or experiment with the implementation:

GitHub: https://github.com/mov2day/KaratePlugin
VS Code Marketplace: https://marketplace.visualstudio.com/items?itemName=MuthuKumarKoodalingam.karate-test-generator

The result I want from AI-assisted maintenance is not "tests that heal themselves." It is something less magical and much more useful: failures that arrive with enough evidence to make a small, reviewable repair possible.

Originally published at muthukumarkoodalingam.com.

Your Karate Tests Are Green. But Is Your API Actually Covered?

Muthu Kumar Koodalingam — Tue, 15 Sep 2026 07:35:34 +0000

Your Karate Tests Are Green. But Is Your API Actually Covered?

A green Karate run answers one question well:

Did the tests we executed pass?

It does not answer a different question:

Did we test the API surface that matters?

That distinction becomes painful in mature API suites. You can have 300 passing scenarios, a beautiful CI report, and still have newly added operations, destructive methods, or entire resource paths that no Karate scenario exercises.

The useful coverage metric for an API suite is therefore not simply scenario pass rate. It is the relationship between the contract you expose and the tests you can map back to that contract.

This article shows a practical way to build that map using OpenAPI and an existing Karate suite, including where simple matching fails and what evidence you should retain before generating missing tests.

Pass rate and API coverage are different dimensions

Assume CI reports:

Karate scenarios: 184
Passed:           184
Failed:             0
Pass rate:         100%

That looks excellent.

Now compare the suite with the current OpenAPI contract:

GET    /customers                 covered
POST   /customers                 covered
GET    /customers/{customerId}    covered
PATCH  /customers/{customerId}    missing
DELETE /customers/{customerId}    missing
GET    /customers/{customerId}/orders  missing

The suite is still 100% green. It is simply green over an incomplete slice of the API.

This is why I treat execution health and contract coverage as separate signals:

Execution health = did known tests pass?
Contract coverage = which API operations have test evidence?

Neither replaces the other.

Start with operations, not lines of code

For API testing, an immediately useful unit of coverage is an OpenAPI operation:

HTTP method + normalized path

For example:

GET  /orders/{id}
POST /orders

This is more actionable than application code coverage when the question is whether your externally documented API surface is exercised.

Given this OpenAPI fragment:

paths:
  /orders:
    post:
      operationId: createOrder
      responses:
        '201':
          description: Created

  /orders/{orderId}:
    get:
      operationId: getOrder
      parameters:
        - name: orderId
          in: path
          required: true
          schema:
            type: string

we can derive two contract operations:

POST /orders
GET  /orders/{orderId}

The next job is to find evidence for those operations in Karate.

Mapping Karate scenarios to OpenAPI operations

Consider this feature:

Feature: Order lookup

Background:
  * url baseUrl

Scenario: Get an existing order
  Given path 'orders', orderId
  When method get
  Then status 200
  And match response.id == orderId

A human immediately sees that it probably covers:

GET /orders/{orderId}

A coverage engine has to establish that relationship systematically.

At minimum it needs to extract:

method = GET
path   = /orders/<dynamic-value>

and normalize the dynamic segment against the contract path.

Conceptually:

Karate:  GET /orders/8c12...
OpenAPI: GET /orders/{orderId}
                     ↓
                  match

You do not need an LLM for this basic comparison. It should be deterministic and explainable.

Normalize paths before comparing them

Literal string matching is too weak.

These can all represent the same contract operation:

Given path 'orders', orderId

Given path 'orders/' + orderId

Given url baseUrl + '/orders/' + orderId

while OpenAPI represents it as:

/orders/{orderId}

A useful mapper normalizes both sides into comparable structures.

For example:

Contract segments: [orders, {param}]
Test segments:     [orders, <dynamic>]
Method:            GET == GET

Then coverage can be based on structural compatibility rather than source-code spelling.

Be conservative when evidence is ambiguous. A false covered result is more dangerous than an unknown result because it hides work.

Coverage should retain evidence

A percentage alone is not enough.

Suppose a dashboard says:

API coverage: 82%

The first useful question is: why 82%?

For every covered operation, retain evidence such as:

GET /orders/{orderId}
  -> src/test/java/orders/orders.feature
  -> Scenario: Get an existing order

For uncovered operations, retain the opposite evidence:

DELETE /orders/{orderId}
  -> no mapped Karate scenario

This turns coverage into something engineers can inspect rather than a vanity metric.

A practical result model might look like:

{
  "method": "DELETE",
  "path": "/orders/{orderId}",
  "status": "uncovered",
  "evidence": []
}

versus:

{
  "method": "GET",
  "path": "/orders/{orderId}",
  "status": "covered",
  "evidence": [
    {
      "feature": "orders/orders.feature",
      "scenario": "Get an existing order"
    }
  ]
}

Now the number is reproducible.

Do not confuse endpoint coverage with assertion quality

Finding a mapped scenario proves that an operation is exercised. It does not prove the scenario is strong.

This scenario technically reaches the endpoint:

Scenario: Get order
  Given path 'orders', orderId
  When method get
  Then status 200

A more meaningful test may include contract-relevant assertions:

Scenario: Get order
  Given path 'orders', orderId
  When method get
  Then status 200
  And match response ==
    """
    {
      id: '#string',
      status: '#string',
      total: '#number',
      items: '#array'
    }
    """

I therefore avoid pretending operation coverage is a complete test-quality score.

Think of quality as several independent questions:

Was the operation exercised?
Did the scenario pass?
Are meaningful assertions present?
Are important response/error variants tested?
Is the test stable over repeated runs?

A useful test-management system keeps these signals separate instead of compressing all of them into one opaque score.

Positive coverage is only the first layer

OpenAPI can also expose candidate coverage dimensions inside an operation.

Suppose the contract documents:

responses:
  '200':
    description: Order returned
  '404':
    description: Order not found
  '401':
    description: Authentication required

A single happy-path scenario means the operation is exercised, but the documented behavior is not necessarily well covered.

You can model progressively richer coverage:

Level 1: operation coverage
  GET /orders/{id}

Level 2: documented response coverage
  200 covered
  404 missing
  401 missing

Level 3: schema / boundary coverage
  required fields
  enums
  minimum / maximum
  formats

The important part is to label these dimensions accurately. Do not report Level 1 as though it proves Level 3.

Generate from gaps, not from the whole specification

Once the comparison is trustworthy, generation becomes much safer.

Imagine the analysis produces:

12 uncovered operations

The wrong next step is:

regenerate 86 operations

The better workflow is:

OpenAPI contract
       +
Existing Karate suite
       ↓
Operation mapping
       ↓
Coverage evidence
       ↓
Uncovered operations only
       ↓
Generate candidate tests
       ↓
Review + execute

This matters because mature suites contain engineering intent that does not exist in OpenAPI: authentication helpers, setup flows, domain fixtures, custom assertions, tags and team conventions.

Targeted generation reduces the chance of replacing that intent with generic generated code.

Example: turning one coverage gap into a test

Suppose coverage identifies:

DELETE /orders/{orderId} — uncovered

The contract says:

/orders/{orderId}:
  delete:
    operationId: deleteOrder
    parameters:
      - name: orderId
        in: path
        required: true
        schema:
          type: string
    responses:
      '204':
        description: Deleted
      '404':
        description: Not found

Do not invent a random order ID and call that complete.

First inspect how the repository creates valid orders. If an existing reusable feature exists:

* def created = call read('classpath:helpers/create-order.feature')
* def orderId = created.response.id

then a generated candidate can fit the suite:

Scenario: Delete an existing order
  * def created = call read('classpath:helpers/create-order.feature')
  * def orderId = created.response.id

  Given path 'orders', orderId
  When method delete
  Then status 204

A second candidate can address documented negative behavior:

Scenario: Delete an unknown order
  Given path 'orders', 'does-not-exist'
  When method delete
  Then status 404

That is far more useful than blindly producing one isolated scenario per OpenAPI operation.

What happens when the specification changes?

Coverage becomes even more valuable when it is recomputed after contract changes.

Suppose yesterday's contract contained:

GET  /orders/{id}
POST /orders

and today's contains:

GET   /orders/{id}
POST  /orders
PATCH /orders/{id}

The interesting result is not simply "OpenAPI changed."

It is:

New operation:
PATCH /orders/{id}

Mapped scenarios:
none

Coverage state:
uncovered

That is actionable change impact.

Likewise, if a response schema adds a required field, the tool should distinguish that from a completely new operation. The likely test impact is different.

This is where contract coverage starts becoming a maintenance mechanism rather than a reporting feature.

Where AI helps — and where it should not decide the truth

AI is useful after deterministic evidence exists.

Given:

PATCH /orders/{id} is uncovered

plus the operation schema and nearby repository patterns, an AI assistant can help propose:

realistic domain scenarios
useful boundaries
repository-consistent naming
stronger assertions
reuse of nearby setup helpers

But I would not ask the model to be the source of truth for whether the operation is covered.

The safer split is:

Deterministic layer
  parse OpenAPI
  index Karate
  map operations
  identify gaps
  retain evidence

AI-assisted layer
  interpret domain intent
  suggest scenarios
  adapt to local style
  propose repairs

That separation makes the result inspectable when the model is wrong or unavailable.

How Karate Test Management approaches this

This workflow is now part of Karate Test Management for VS Code.

The current extension can analyze an OpenAPI specification by itself, or combine the specification with Karate feature files for exact scenario-to-endpoint mapping. Coverage gaps enter the Quality workflow, and missing-test generation can start from those gaps rather than regenerating the entire API suite.

The same workspace also keeps execution, run history, flakiness, specification-change findings and repair review alongside coverage. That matters because an uncovered endpoint, a failing endpoint and a flaky endpoint are three different engineering problems.

The implementation deliberately keeps deterministic coverage available without an AI provider. AI enhancement is optional and uses the evidence produced by the underlying workflow.

A practical coverage review for your own Karate repository

Even without tooling, you can apply the same model manually or in a script:

Parse the current OpenAPI document into (method, normalized path) operations.
Index Karate scenarios and extract the HTTP method and resolved/structural path evidence you can establish.
Map tests to operations conservatively.
Mark ambiguous mappings as unknown instead of covered.
Keep the feature and scenario name as evidence for every match.
Review uncovered operations by risk, not alphabetically.
Generate or write tests only for justified gaps.
Re-run the mapping whenever the contract changes.

The key output should look less like a single percentage and more like an engineering queue:

HIGH   DELETE /customers/{id}       uncovered
HIGH   POST   /payments/refunds     uncovered
MEDIUM PATCH  /orders/{id}          uncovered
LOW    GET    /catalog/metadata     uncovered

Now API coverage can influence release decisions.

Green is necessary, not sufficient

A passing suite is good evidence about the tests you ran.

It is not evidence about tests that do not exist.

For contract-driven APIs, comparing Karate scenarios with OpenAPI gives you a practical way to expose that blind spot. Keep the mapping deterministic, retain evidence, distinguish endpoint coverage from assertion quality, and generate from gaps instead of repeatedly generating the whole suite.

That is the difference between asking "are my tests green?" and asking the more useful question:

What part of my API do those green tests actually protect?

Karate Test Management is open source if you want to try the workflow:

GitHub: https://github.com/mov2day/KaratePlugin
VS Code Marketplace: https://marketplace.visualstudio.com/items?itemName=MuthuKumarKoodalingam.karate-test-generator

How to Generate Karate API Tests from OpenAPI — Without Creating a Maintenance Nightmare

Muthu Kumar Koodalingam — Fri, 11 Sep 2026 16:16:06 +0000

How to Generate Karate API Tests from OpenAPI — Without Creating a Maintenance Nightmare

Generating API tests from an OpenAPI file sounds like an easy automation win.

Parse the specification. Create one test per endpoint. Add a few assertions. Commit the generated files.

That approach works surprisingly well in a demo — and often becomes painful as soon as the API or test suite starts evolving.

The hard problem is not producing Karate syntax. The hard problem is producing tests that fit the way your project already works and remain useful after the first generation run.

The tempting approach

Imagine an API contract containing:

paths:
  /orders:
    post:
      operationId: createOrder
  /orders/{id}:
    get:
      operationId: getOrder

A generator can easily produce something like:

Feature: Orders API

Scenario: Create an order
  Given url baseUrl
  And path 'orders'
  And request { productId: 123, quantity: 1 }
  When method post
  Then status 201

Scenario: Get an order
  Given url baseUrl
  And path 'orders', 123
  When method get
  Then status 200

It is valid Karate. It may even pass.

But several important questions are still unanswered:

Where did baseUrl come from?
Does the project already have authentication helpers?
Is productId: 123 valid test data?
What should the response schema contain?
What happens for an invalid product?
What happens when quantity is missing?
Is the endpoint already covered by another scenario?
Does the team have naming, tagging or reusable-feature conventions?

A generated test that ignores those questions creates code, but not necessarily useful coverage.

Treat OpenAPI as a contract, not a test suite

OpenAPI tells us a lot about the API surface:

paths and HTTP methods
parameters
request schemas
response schemas
required fields
enums and formats
documented response codes
security schemes

That information is excellent input for test generation.

But the specification usually does not contain all the information needed for an executable test suite. Environment configuration, reusable authentication, realistic data creation and project conventions typically live elsewhere.

A good generator therefore needs two kinds of context:

Contract context — what the API says it accepts and returns.
Repository context — how this project already executes Karate tests.

Ignoring the second is where many generated suites become difficult to maintain.

Generate deterministic coverage first

I prefer to begin with rules that do not need an LLM.

For each operation, deterministic generation can derive candidate scenarios from the contract.

For example, a required field gives us at least two obvious cases:

quantity:
  type: integer
  minimum: 1

Positive case:

And request { productId: 123, quantity: 1 }
When method post
Then status 201

Boundary or negative candidates can be derived from the same schema:

And request { productId: 123, quantity: 0 }
When method post
Then status 400

and:

And request { productId: 123 }
When method post
Then status 400

This has an important advantage: the reasoning is explainable. We know exactly why each scenario exists.

AI can still improve the suite later, but it should not be the only mechanism deciding what needs to be tested.

Reuse the project before inventing configuration

Before generating a new feature, inspect the repository.

A Karate project may already contain:

src/test/java/
  karate-config.js
  auth/
    token.feature
  common/
    create-user.feature
  orders/
    orders.feature

If authentication already comes from karate-config.js, generated tests should use it.

If the team creates test users through a reusable feature, the generator should reuse that pattern instead of inventing credentials.

If scenarios consistently use tags such as:

@orders @smoke

new tests should follow the same convention.

Repository awareness is the difference between generating a standalone example and adding a maintainable test to an existing suite.

Detect coverage before generating more tests

Another mistake is assuming every OpenAPI operation needs a new feature file.

First compare the API contract with the existing Karate scenarios.

Think of the API surface as a matrix:

POST   /orders         covered
GET    /orders/{id}    covered
DELETE /orders/{id}    missing
PATCH  /orders/{id}    missing

Now generation becomes targeted.

Instead of producing another complete test suite, generate tests only for the gaps.

That makes the workflow:

OpenAPI
   ↓
Existing Karate suite
   ↓
Coverage mapping
   ↓
Missing operations
   ↓
Generate only what is missing

This is much more useful in a real repository than repeatedly regenerating everything.

Use AI as an enhancement layer

There are places where AI is genuinely useful.

For example:

understanding domain intent from descriptions
suggesting edge cases that are not encoded in the schema
adapting generated scenarios to an existing style
explaining a failed test
proposing a repair after an API change

But those workflows should sit on top of deterministic evidence.

A useful architecture looks like this:

OpenAPI specification
        ↓
Deterministic parser
        ↓
Coverage + schema rules
        ↓
Repository conventions
        ↓
Candidate Karate scenarios
        ↓
Optional AI enhancement
        ↓
Validation / review

The key word is optional.

You should still be able to generate, execute and analyse tests when an AI provider is unavailable.

Validate what was generated

Generation is not finished when a .feature file exists.

At minimum, generated output should be checked for:

valid Karate syntax
valid referenced configuration
duplicate scenarios
unresolved variables
incorrect paths or HTTP methods
missing required request data
assertions that are too weak

The last item deserves particular attention.

This test can pass while proving very little:

When method get
Then status 200

Where the contract supports it, stronger assertions might verify:

Then status 200
And match response.id == '#number'
And match response.status == '#string'
And match response.items == '#array'

The goal is not to maximize assertion count. It is to make sure the test can detect the failures it was created to protect against.

This is the workflow I built into Karate Test Management

I originally built Karate Test Management, an open-source VS Code extension, around test generation. As the project grew, the interesting problem became everything around generation.

The current workflow can:

generate Karate tests from OpenAPI
import Postman, HAR and GraphQL sources
index an existing Karate test library
execute features and individual scenarios
compare OpenAPI operations with existing test coverage
identify missing tests
track quality findings
analyse failures and flakiness
optionally use AI for generation and repair

The important architectural decision is that deterministic generation, execution and coverage analysis do not depend on AI.

AI is there to enhance test-engineering work, not to replace the evidence underneath it.

A better mental model

Instead of thinking:

OpenAPI → generated test files

I find this model more useful:

Discover
   ↓
Understand the contract and repository
   ↓
Measure existing coverage
   ↓
Generate missing scenarios
   ↓
Execute
   ↓
Analyse evidence
   ↓
Maintain

That turns test generation from a one-off code-generation trick into part of a sustainable API testing workflow.

If you use Karate and want to experiment with this approach, the project is open source:

GitHub: https://github.com/mov2day/KaratePlugin
VS Code Marketplace: https://marketplace.visualstudio.com/items?itemName=MuthuKumarKoodalingam.karate-test-generator

The next article in this series will look at a related problem: your Karate tests may all be green while large parts of your API are still untested.

Karate Test Generator for VS Code: Automate API Testing with AI

Muthu Kumar Koodalingam — Mon, 29 Dec 2025 15:02:11 +0000

🚀 Automate Karate API Testing with This VS Code Extension

Automate Karate API testing directly from OpenAPI specs, keep tests auto-synced with API changes, and enhance coverage using GitHub Copilot AI — saving up to 80% maintenance time.

📚 Table of Contents

Why Manual API Testing Fails in 2025
Introducing Karate API Test Generator
Key Features & Capabilities
Real-World Results & Case Studies
Quick Start Guide
FAQ

❌ Why Manual API Testing Fails in 2025

The Hidden Cost of Manual API Testing

Are you spending 40% of your sprint fixing broken API tests?

You’re not alone. Surveys show QA teams now spend more time maintaining tests than validating new features. Manual Karate DSL test writing creates four major bottlenecks.

🔥 The Manual Testing Crisis

1. Constant API Changes Break Test Suites

Every endpoint update triggers hours of test fixes. With microservices averaging 20+ API changes per sprint, maintenance becomes unsustainable.

2. Repetitive Boilerplate Crushes Productivity

Writing Given url, When method, Then status hundreds of times isn’t just boring — it’s where inconsistencies and bugs hide.

3. Invisible Coverage Gaps

Manual testing misses:

429 rate limits
500 server errors
Malformed payloads

These gaps cost real money when they reach production.

4. Team Style Inconsistency

10 developers = 10 styles.
Code reviews become style debates, not logic reviews.

📉 Real cost: Teams report spending 60–70% of QA time on test maintenance.

🧩 Meet Karate API Test Generator

A VS Code extension purpose-built to automate Karate API testing.

👉 Available on the VS Code Marketplace

What Makes It Different?

Unlike generic testing tools, Karate API Test Generator uniquely combines:

✅ Native Karate DSL support
✅ OpenAPI 3.1 compatibility
✅ GitHub Copilot AI enhancement
✅ Automatic test sync with API changes
✅ Style learning for team consistency

⚙️ Complete Feature Breakdown

🚀 1. Multi-Source Test Generation

OpenAPI Support

Supports OpenAPI 2.0, 3.0, and 3.1

# Generated from OpenAPI spec in seconds
Feature: User Management API Tests

Scenario: Create new user - Happy path
  Given url baseUrl + '/api/v1/users'
  And request { name: 'John Doe', email: 'john@example.com' }
  When method POST
  Then status 201
  And match response.id == '#number'
  And match response.email == 'john@example.com'

Automatically generates:

All HTTP methods
Path & query parameters
Request/response validation
Auth headers (Bearer, API Key, OAuth)
Examples from OpenAPI docs
Error scenarios (400–500)

Confluence Integration

Parse tables into test data
Convert flow diagrams to scenarios
Extract acceptance criteria
Link tests to requirements

Combined Generation

Merge OpenAPI contracts + business docs for full coverage of API + business logic.

🤖 2. GitHub Copilot AI Enhancement

Before AI

Scenario: Get user by ID
  Given url baseUrl + '/users/1'
  When method GET
  Then status 200

After AI

Scenario: Get user by ID - Valid request
  Given url baseUrl + '/users/1'
  And header Authorization = 'Bearer ' + authToken
  When method GET
  Then status 200
  And match response == 
    { id: 1, name: '#string', email: '#regex ^[\\w-\\.]+@([\\w-]+\\.)+[\\w-]{2,4}$' }

Scenario: Get user by ID - Invalid ID returns 404
  Given url baseUrl + '/users/999999'
  When method GET
  Then status 404

AI automatically adds:

Realistic test data
Edge cases
Security scenarios
Boundary testing
Performance considerations

🔄 3. Automatic Test Maintenance

Problem: New API field → dozens of broken tests
Solution: Auto-sync with OpenAPI specs

How it works:

Link OpenAPI spec to tests
Extension watches for changes
Smart notifications:

Update with AI
Ignore
Review diff

📊 Result: 80% reduction in maintenance time

🎨 4. Style Learning Engine

Learns from existing Karate tests:

Indentation & quoting style
Naming conventions
Background & tag usage

Applies conventions automatically to all new tests — perfect for large teams.

🧭 5. Developer-Friendly UI

No CLI required.

Inside VS Code:

Dashboard & stats
OpenAPI import & generation
Confluence integration
Auto-sync management
Template editor

🏆 Real-World Case Studies

🏃‍♂️ Startup: 3-Person Team

Results:

Test generation: 4 hours → 15 minutes
Maintenance: 60% → 10% of sprint
Coverage: 45% → 87%

“We stopped firefighting broken tests and started testing features.”
— Lead QA Engineer

🏢 Enterprise: 50+ Engineers

Results:

100% test style consistency
Code review time: 45 → 15 minutes
Coverage up 34%
3× faster onboarding

🔗 Contract Testing at Scale

Results:

~80% maintenance reduction
5× larger contract test suite
Prevented $50K+ in incidents

⚡ Quick Start: 5 Minutes

1. Install Extension

Open VS Code
Extensions → Search Karate API Test Generator
Install

2. (Recommended) Install GitHub Copilot

Enables AI enhancement
Free for students & OSS maintainers

3. Generate Tests

Method 1:
Right-click OpenAPI spec → Generate Karate Tests Now

Method 2:
Open extension UI → Import spec → Generate

✅ Tests generated in under 30 seconds

🧠 Best Practices

Run style learning first
Combine specs + docs
Review AI-enhanced tests
Enable auto-sync for stable APIs
Use shared templates

Comparison: Karate Test Generator vs. Alternatives

🛣️ Roadmap

GraphQL support
Run tests inside VS Code
Test results dashboard
Jira integration
Git-based auto-commits
SOAP + REST hybrid support

❓ FAQ

Does it support Karate 1.x & 2.x?
Yes.

Can I use it without Copilot?
Absolutely.

Will it replace QA engineers?
No — it removes boilerplate so QAs focus on value.

Is my OpenAPI spec sent externally?
No. Everything runs locally.

💰 Pricing & Support

Karate API Test Generator:
🆓 100% Free (VS Code Marketplace)

GitHub Copilot:

$10/month (Individual)
Free for students & OSS maintainers

Support:

📖 Docs
💬 Discord
🐛 GitHub Issues
✉️ Email

🧨 The Bottom Line

If you’re still writing Karate tests manually in 2025, you’re wasting:

⏱️ 5–10 hours per sprint per dev
💵 $50K+ annually per team
📉 Missing 30–40% edge cases
🔄 Fixing tests instead of shipping features

🎯 Ready to Transform Your API Testing?

Get started in 3 steps:

Install Karate API Test Generator
Open any OpenAPI spec
Right-click → Generate Tests

Then enjoy reclaiming 10+ hours per sprint 🚀