DEV Community

Muthu Kumar Koodalingam
Muthu Kumar Koodalingam

Posted on Originally published at muthukumarkoodalingam.com

Your Karate Test Failed in CI. Should AI Fix It?

#karate #apitesting #cicd #ai

Your Karate Test Failed in CI. Should AI Fix It?

A failed API test often creates an awkward handoff.

CI knows that a scenario failed. The Karate report may know the failed step, HTTP status, request and response. The source repository knows the scenario that produced it. But by the time someone investigates, that evidence is scattered across logs, artifacts and source files.

It is tempting to solve this with a simple prompt:

Here is the failing Karate feature and the error.
Fix the test.
Enter fullscreen mode Exit fullscreen mode

That is also a good way to create a test suite that slowly learns to accept broken behavior.

The interesting problem is not whether an LLM can edit Gherkin. It can. The problem is deciding what evidence is strong enough to justify a repair, how narrowly the repair is allowed to operate, and where a human must remain in control.

This article describes the architecture I use in Karate Test Management for evidence-driven CI repair.

A failing test is not proof that the test is wrong

Consider this scenario:

Scenario: Get an existing order
  Given path 'orders', orderId
  When method get
  Then status 200
  And match response.status == 'CONFIRMED'
Enter fullscreen mode Exit fullscreen mode

CI reports:

expected: 200
actual:   404
Enter fullscreen mode Exit fullscreen mode

A repair model could make the build green immediately:

- Then status 200
+ Then status 404
Enter fullscreen mode Exit fullscreen mode

That is syntactically valid and operationally dangerous.

The failure might mean:

  • the API contract changed;
  • test data was not created;
  • an environment reset removed the order;
  • authentication selected the wrong tenant;
  • the endpoint is broken;
  • the test is genuinely stale.

A repair system that starts by editing the assertion has confused observed behavior with expected behavior.

So the first rule is simple:

Treat a CI failure as evidence to investigate, not permission to weaken a test.

Normalize CI failures before asking AI anything

Different pipelines expose failures differently. GitHub Actions may provide downloadable test artifacts and logs. Jenkins may expose console output. A custom runner may already have structured JSON.

The repair layer should not care.

Normalize the useful evidence into one payload first:

{
  "source": "github-actions",
  "featurePath": "src/test/java/orders/orders.feature",
  "scenarioName": "Get an existing order",
  "scenarioLine": 18,
  "failedStep": "Then status 200",
  "errorMessage": "status code was: 404, expected: 200",
  "httpRequest": {
    "method": "GET",
    "url": "https://api.example.test/orders/9821"
  },
  "httpResponse": {
    "status": 404,
    "body": "{\"code\":\"ORDER_NOT_FOUND\"}"
  },
  "runId": "814521:1"
}
Enter fullscreen mode Exit fullscreen mode

This is much more useful than sending an entire CI log to a model.

It gives the repair workflow explicit anchors:

CI run
  ↓
failed job / step
  ↓
feature + scenario identity
  ↓
HTTP evidence
  ↓
source scenario
  ↓
repair candidate
Enter fullscreen mode Exit fullscreen mode

It also makes the process inspectable. If a proposed repair is questionable, you can see exactly which evidence produced it.

Locate the scenario precisely

Scenario identity is an underrated part of automated repair.

Imagine a repository containing:

Scenario: Get user
  ...
Enter fullscreen mode Exit fullscreen mode

in several feature files, or a feature with generated scenarios that have similar names.

A repair engine should use more than a free-text scenario name where possible. Useful coordinates include:

feature path
scenario name
scenario line
scenario tags
Enter fullscreen mode Exit fullscreen mode

The important safety property is unique identification.

If the system cannot identify exactly one source scenario, it should refuse the edit rather than guessing.

This principle generalizes beyond Karate: autonomous code changes become safer when uncertainty causes the system to stop instead of broadening its write scope.

Give the model evidence, not authority

Once the source scenario is located, AI can help reason about the failure. But the prompt should define a narrow repair boundary.

A useful repair context looks like this:

REPAIR CONTEXT
Feature: src/test/java/orders/orders.feature
Scenario: Get an existing order
Failed step: Then status 200
Error: status code was: 404, expected: 200

HTTP EVIDENCE
GET https://api.example.test/orders/9821
Response: 404
Body: {"code":"ORDER_NOT_FOUND"}

CONSTRAINT
Fix only the failed step and its immediate dependencies.
Do not change any other scenario.
Do not invent endpoints or fields not visible in the evidence.

OUTPUT
Return the complete repaired Scenario block only.
Enter fullscreen mode Exit fullscreen mode

Notice what is deliberately absent: "make the test pass."

The goal is to produce a candidate repair, not optimize for a green build at any cost.

Why the complete scenario is a useful repair unit

Returning a single replacement line sounds safer, but failures often involve a small dependency chain.

For example:

* def order = call read('classpath:helpers/create-order.feature')
* def orderId = order.id
Given path 'orders', orderId
When method get
Then status 200
Enter fullscreen mode Exit fullscreen mode

If order.id changed to order.orderId, repairing only the failed HTTP assertion cannot solve the root problem.

On the other hand, returning the entire feature file gives the model unnecessary authority over unrelated tests.

The scenario is a useful middle ground:

single line        → often insufficient context
single scenario    → useful bounded repair surface
entire feature     → unnecessarily broad write scope
repository         → far too broad
Enter fullscreen mode Exit fullscreen mode

The repair system can then replace exactly that scenario in the original feature.

Diff before apply should be the default

Suppose the model proposes:

Scenario: Get an existing order
- * def orderId = order.id
+ * def orderId = order.orderId
  Given path 'orders', orderId
  When method get
  Then status 200
Enter fullscreen mode Exit fullscreen mode

That is the moment where automation should slow down.

Show the original and candidate side by side. Let the engineer inspect whether the change preserves intent.

In Karate Test Management, CI repair defaults to a reviewable diff. Automatic application is an explicit setting rather than the default behavior. A backup can also be created before applying a repair.

That distinction matters because there are really two separate capabilities:

AI can propose a change
        ≠
AI is authorized to modify the test suite
Enter fullscreen mode Exit fullscreen mode

Keeping those permissions separate is one of the simplest guardrails for AI-assisted engineering.

Never let repair erase the oracle

The most dangerous repair is one that makes a test less capable of detecting defects.

Consider:

Then status 200
And match response ==
"""
{
  id: '#number',
  state: 'CONFIRMED'
}
"""
Enter fullscreen mode Exit fullscreen mode

A naive repair could turn this into:

Then status 200
Enter fullscreen mode Exit fullscreen mode

The build becomes green, but the test has lost most of its value.

A practical repair review should therefore ask:

  1. Did the candidate change the expected business behavior?
  2. Did it remove or weaken assertions?
  3. Did it introduce an endpoint, field or status not supported by evidence?
  4. Did it modify anything outside the failed scenario?
  5. Is the failure more likely product behavior, environment behavior or test behavior?

That final question is important. Some failures should produce no repair at all.

Separate deterministic work from AI work

Most of the CI-repair pipeline does not require an LLM.

Deterministic code can:

  • identify failed jobs;
  • download artifacts and logs;
  • extract .feature references;
  • locate scenario names and source lines;
  • detect common status/assertion failures;
  • extract HTTP methods, URLs, statuses and response bodies;
  • locate the exact scenario in the repository;
  • generate a diff;
  • enforce write boundaries.

AI becomes useful after that evidence has been assembled, where the task requires reasoning about how the scenario might need to change.

A safer architecture therefore looks like:

GitHub Actions / CI
        ↓
Failure extraction          deterministic
        ↓
Structured evidence         deterministic
        ↓
Scenario location           deterministic
        ↓
Repair proposal             AI-assisted
        ↓
Scope validation            deterministic
        ↓
Diff / review               human
        ↓
Apply
Enter fullscreen mode Exit fullscreen mode

This is a much stronger pattern than giving an agent a repository and asking it to "fix CI."

Pulling evidence from GitHub Actions

For GitHub Actions, a useful workflow is to inspect the failed run and gather the evidence already produced by the test job.

The current Karate Test Management implementation can pull workflow runs, jobs, artifacts and logs, then search those sources for a feature reference, scenario name, error and HTTP evidence.

The extraction is intentionally best-effort because CI output varies. A Karate failure might expose:

src/test/java/orders/orders.feature:18
Scenario: Get an existing order
status code was: 404, expected: 200
Enter fullscreen mode Exit fullscreen mode

and request logs may contain:

> GET https://api.example.test/orders/9821
response status: 404
{"code":"ORDER_NOT_FOUND"}
Enter fullscreen mode Exit fullscreen mode

From those fragments, the system can build the structured payload shown earlier.

If it cannot infer the feature path reliably, the correct behavior is to stop the automated repair path rather than invent a target.

What about Jenkins or GitLab?

The same architecture does not require every CI provider to expose the same API.

The normalized failure contract can represent several sources:

type FailureSource =
  | 'github-actions'
  | 'jenkins'
  | 'gitlab-ci'
  | 'generic';
Enter fullscreen mode Exit fullscreen mode

That means provider-specific adapters can evolve independently while the repair engine continues to consume one evidence shape.

A Jenkins pipeline, for example, could POST a compact failure payload after a failed Karate run instead of granting a desktop extension broad access to Jenkins itself.

This separation is useful operationally and from a security perspective.

AI repair should be optional

There is another architectural boundary worth keeping: the test suite must remain usable when AI is unavailable.

Generation, execution, coverage analysis and normal test management should not depend on a model provider. Repair is an enhancement layer.

If AI is disabled, quota is exhausted, or a provider is unavailable, the CI evidence is still useful:

Failed scenario
Failed step
Request
Response
Source location
Run identity
Enter fullscreen mode Exit fullscreen mode

An engineer can investigate from that information directly.

This prevents AI availability from becoming a new dependency in the delivery pipeline.

The bigger lesson: automate evidence before decisions

CI repair is one example of a broader pattern I increasingly use for AI-assisted test engineering.

Do not begin with:

failure → AI → code change
Enter fullscreen mode Exit fullscreen mode

Build this instead:

failure
  ↓
collect evidence
  ↓
normalize evidence
  ↓
identify exact change surface
  ↓
reason about candidate change
  ↓
validate boundaries
  ↓
review
Enter fullscreen mode Exit fullscreen mode

The AI step becomes smaller, better informed and easier to audit.

That is important for testing tools because the test suite is itself part of the safety system. An automated repair that silently weakens the oracle can be worse than a failing build.

How this fits into Karate Test Management

Karate Test Management treats CI repair as one part of a larger test lifecycle rather than a standalone "self-healing" trick.

The current project includes:

  • structured CI failure ingestion;
  • GitHub Actions run, artifact and log intake;
  • feature/scenario location;
  • HTTP evidence extraction;
  • AI-assisted scenario repair;
  • bounded scenario replacement;
  • reviewable diffs;
  • optional backups and explicit auto-apply;
  • the same provider-routing guardrails used by other AI-assisted workflows.

The principle behind all of them is the same: AI can help reason over test evidence, but deterministic controls decide what it is allowed to touch.

If you use Karate and want to inspect or experiment with the implementation:

The result I want from AI-assisted maintenance is not "tests that heal themselves." It is something less magical and much more useful: failures that arrive with enough evidence to make a small, reviewable repair possible.

Originally published at muthukumarkoodalingam.com.

Top comments (0)